igny8/docs/10-MODULES/AUTOMATION.md

# Automation Module

**Last Verified:** January 1, 2026  
**Version:** 1.3.0  
**Status:** ✅ Active  
**Backend Path:** `backend/igny8_core/business/automation/`  
**Frontend Path:** `frontend/src/pages/Automation/`

---

## Quick Reference

| What | File | Key Items |
|------|------|-----------|
| Models | `business/automation/models.py` | `AutomationConfig`, `AutomationRun` |
| Service | `business/automation/services/automation_service.py` | `AutomationService` |
| Logger | `business/automation/services/automation_logger.py` | `AutomationLogger` |
| Celery Tasks | `business/automation/tasks.py` | `run_automation_task`, `check_scheduled_automations` |
| Frontend | `pages/Automation/AutomationPage.tsx` | Main automation UI |
| Progress Bar | `components/Automation/GlobalProgressBar.tsx` | Full pipeline progress |
| Processing Card | `components/Automation/CurrentProcessingCard.tsx` | Real-time progress |

---

## Purpose

The Automation module runs the complete 7-stage content pipeline automatically:

```
Keywords → Clusters → Ideas → Tasks → Content → Image Prompts → Images → Published
```

---

## 7-Stage Pipeline

| Stage | Name | AI Function | Credit Cost |
|-------|------|-------------|-------------|
| 1 | Keywords → Clusters | `AutoClusterFunction` | Per batch |
| 2 | Clusters → Ideas | `GenerateIdeasFunction` | Per idea |
| 3 | Ideas → Tasks | None (local) | None |
| 4 | Tasks → Content | `GenerateContentFunction` | Per 100 words |
| 5 | Content → Image Prompts | `GenerateImagePromptsFunction` | Per prompt |
| 6 | Image Prompts → Images | `process_image_generation_queue` | Per image |
| 7 | Review → Published | None (auto-approve) | None |

**Note:** Stage 7 changed from "Manual Review Gate" to auto-approve and publish in v1.3.0.

---

## Data Models

### AutomationConfig

| Field | Type | Purpose |
|-------|------|---------|
| account | FK | Owner account |
| site | FK | Target site |
| enabled | Boolean | Enable/disable automation |
| frequency | CharField | daily/weekly/monthly |
| scheduled_time | TimeField | Time to run |
| stage_1_batch_size | Integer | Keywords per batch |
| stage_2_batch_size | Integer | Clusters per batch |
| stage_3_batch_size | Integer | Ideas per batch |
| stage_4_batch_size | Integer | Tasks per batch |
| stage_5_batch_size | Integer | Content per batch |
| stage_6_batch_size | Integer | Images per batch |
| within_stage_delay | Integer | Seconds between batches |
| between_stage_delay | Integer | Seconds between stages |
| last_run_at | DateTime | Last execution |
| next_run_at | DateTime | Next scheduled run |

### AutomationRun

| Field | Type | Purpose |
|-------|------|---------|
| config | FK | Parent config |
| trigger_type | CharField | manual/scheduled |
| status | CharField | running/paused/cancelled/completed/failed |
| current_stage | Integer | Current stage (1-7) |
| started_at | DateTime | Start time |
| paused_at | DateTime | Pause time (nullable) |
| resumed_at | DateTime | Resume time (nullable) |
| cancelled_at | DateTime | Cancel time (nullable) |
| completed_at | DateTime | Completion time (nullable) |
| total_credits_used | Decimal | Total credits consumed |
| **initial_snapshot** | JSON | **v1.3.0** Queue sizes at run start |
| stage_1_result | JSON | Stage 1 results |
| stage_2_result | JSON | Stage 2 results |
| stage_3_result | JSON | Stage 3 results |
| stage_4_result | JSON | Stage 4 results |
| stage_5_result | JSON | Stage 5 results |
| stage_6_result | JSON | Stage 6 results |
| stage_7_result | JSON | Stage 7 results |
| error_message | TextField | Error details (nullable) |

---

## API Endpoints

| Method | Path | Handler | Purpose |
|--------|------|---------|---------|
| GET | `/api/v1/automation/config/` | Get/create config | Get automation config |
| PUT | `/api/v1/automation/update_config/` | Update config | Update settings |
| POST | `/api/v1/automation/run_now/` | Start manual run | Start automation |
| GET | `/api/v1/automation/current_run/` | Get current run | Run status/progress |
| GET | `/api/v1/automation/pipeline_overview/` | Get pipeline | Stage status counts |
| GET | `/api/v1/automation/current_processing/` | Get processing | Live processing status |
| **GET** | `/api/v1/automation/run_progress/` | **v1.3.0** | Unified progress data |
| POST | `/api/v1/automation/pause/` | Pause run | Pause after current item |
| POST | `/api/v1/automation/resume/` | Resume run | Resume from saved stage |
| POST | `/api/v1/automation/cancel/` | Cancel run | Cancel after current item |
| GET | `/api/v1/automation/history/` | Get history | Last 20 runs |
| GET | `/api/v1/automation/logs/` | Get logs | Activity log for run |
| GET | `/api/v1/automation/estimate/` | Get estimate | Credit estimate |

**Query Parameters:** All require `?site_id=`, run-specific require `?run_id=`

### run_progress Endpoint (v1.3.0)

Returns unified progress data for frontend:
```json
{
  "run": { "run_id": "...", "status": "running", "current_stage": 3 },
  "global_progress": { "total_items": 100, "completed_items": 45, "percentage": 45 },
  "stages": [
    { "number": 1, "status": "completed", "input_count": 50, "processed_count": 50 },
    ...
  ],
  "metrics": { "credits_used": 120, "duration_seconds": 3600 },
  "initial_snapshot": { "stage_1_initial": 50, ... }
}
```

---

## Execution Flow

### Manual Run

1. User clicks "Run Now" on frontend
2. Frontend calls `POST /automation/run_now/?site_id=X`
3. Backend acquires cache lock `automation_lock_{site_id}`
4. **v1.3.0:** Captures initial snapshot with `_capture_initial_snapshot()`
5. Estimates credits required (1.2x buffer)
6. Validates balance >= estimate
7. Creates `AutomationRun` record
8. Enqueues `run_automation_task` Celery task
8. Returns run ID immediately

### Stage Execution

For each stage (1-7):

1. Check `_check_should_stop()` (paused/cancelled?)
2. Load items for processing
3. Process in batches (respecting batch_size)
4. For AI stages: Call AIEngine function
5. Wait `within_stage_delay` between batches
6. Save stage result JSON
7. Wait `between_stage_delay` before next stage

### Stage Result Fields

**Stage 1 (Clustering):**
```json
{
  "keywords_processed": 150,
  "clusters_created": 12,
  "batches_run": 3,
  "credits_used": 45,
  "time_elapsed": 120,
  "skipped": false,
  "partial": false
}
```

**Stage 2 (Ideas):**
```json
{
  "clusters_processed": 12,
  "ideas_created": 36,
  "batches_run": 2,
  "credits_used": 72
}
```

**Stage 3 (Tasks):**
```json
{
  "ideas_processed": 36,
  "tasks_created": 36,
  "batches_run": 4
}
```

**Stage 4 (Content):**
```json
{
  "tasks_processed": 36,
  "content_created": 36,
  "total_words": 54000,
  "batches_run": 6,
  "credits_used": 540
}
```

**Stage 5 (Image Prompts):**
```json
{
  "content_processed": 36,
  "prompts_created": 180,
  "batches_run": 4,
  "credits_used": 36
}
```

**Stage 6 (Images):**
```json
{
  "images_processed": 180,
  "images_generated": 180,
  "batches_run": 18
}
```

**Stage 7 (Review):**
```json
{
  "ready_for_review": 36
}
```

---

## Scheduling

**Celery Beat Task:** `check_scheduled_automations`  
**Frequency:** Hourly

**Logic:**
1. Find configs where `enabled=True`
2. Check if `next_run_at <= now`
3. Check if no active run exists
4. Start `run_automation_task` for eligible configs
5. Update `next_run_at` based on frequency

---

## Lock Mechanism

**Purpose:** Prevent concurrent runs for same site

**Key:** `automation_lock_{site_id}`  
**Storage:** Redis cache  
**Acquired:** On run start  
**Released:** On completion/failure/cancel

---

## Credit Validation

Before starting:
1. Calculate estimated credits for all stages
2. Apply 1.2x safety buffer
3. Compare with account balance
4. Reject if balance < estimate

During execution:
- Each AI stage checks credits before processing
- Deductions happen after successful AI calls
- `total_credits_used` accumulates across stages

---

## Frontend Integration

### AutomationPage Components

- **Config Panel:** Enable/disable, schedule settings
- **Pipeline Cards:** Stage-by-stage status with pending counts
- **Processing Card:** Live processing status during run
- **Control Buttons:** Run Now, Pause, Resume, Cancel
- **Activity Log:** Real-time log streaming
- **History Table:** Past 20 runs with status

### Polling

- Every ~5s while run is running/paused
- Fetches: `current_run`, `pipeline_overview`, `current_processing`
- Lighter polling when idle

---

## Common Issues

| Issue | Cause | Fix |
|-------|-------|-----|
| "Already running" error | Lock exists from previous run | Wait or check if stuck |
| Insufficient credits | Balance < 1.2x estimate | Add credits |
| Stage skipped | No items to process | Check previous stages |
| Run stuck | Worker crashed | Clear lock, restart |
| Images not generating | Stage 5 didn't create prompts | Check stage 5 result |

---

## Planned Changes

| Feature | Status | Description |
|---------|--------|-------------|
| Progress bar accuracy | 🐛 Bug | Fix progress calculation based on actual stage |
| Completed count display | 🐛 Bug | Fix count display in UI |
| Stage skip configuration | 🔜 Planned | Allow skipping certain stages |
| Notification on complete | 🔜 Planned | Email/webhook when done |