# Automation Module

**Last Verified:** January 18, 2026  
**Version:** 1.8.1  
**Status:** ✅ Active  
**Backend Path:** `backend/igny8_core/business/automation/`  
**Frontend Path:** `frontend/src/pages/Automation/`

---

## Quick Reference

| What | File | Key Items |
|------|------|-----------|
| Models | `business/automation/models.py` | `AutomationConfig`, `AutomationRun`, `DefaultAutomationConfig` |
| Service | `business/automation/services/automation_service.py` | `AutomationService` |
| Logger | `business/automation/services/automation_logger.py` | `AutomationLogger` |
| Celery Tasks | `business/automation/tasks.py` | `run_automation_task`, `check_scheduled_automations`, `check_test_triggers` |
| Publishing Tasks | `igny8_core/tasks/publishing_scheduler.py` | Scheduled publishing |
| **Unified Settings** | `api/unified_settings.py` | **v1.8.0** Consolidated settings API |
| **Default Settings API** | `api/unified_settings.py` | **v1.8.1** `DefaultSettingsAPIView` for reset |
| Frontend Overview | `pages/Automation/AutomationOverview.tsx` | **v1.8.0** Run history dashboard |
| Frontend Run Detail | `pages/Automation/AutomationRunDetail.tsx` | **v1.8.0** Detailed run view |
| Frontend Manual Run | `pages/Automation/AutomationPage.tsx` | Manual run UI |
| **Site Settings** | `pages/Sites/AIAutomationSettings.tsx` | **v1.8.0** Unified settings UI |
| Progress Bar | `components/Automation/GlobalProgressBar.tsx` | Full pipeline progress |
| Processing Card | `components/Automation/CurrentProcessingCard.tsx` | Real-time progress |
| **Scheduling Doc** | `docs/40-WORKFLOWS/AUTOMATION-AND-PUBLISHING-SCHEDULING.md` | **v1.8.1** Complete scheduling guide |

---

## Purpose

The Automation module runs the complete 7-stage content pipeline automatically:

```
Keywords → Clusters → Ideas → Tasks → Content → Image Prompts → Images → Published
```

**Settings Location (v1.8.0+):** Site Settings → Automation tab

---

## Scheduling (v1.8.1)

| Task | Schedule | Purpose |
|------|----------|---------|
| `check_scheduled_automations` | Every hour at `:05` | Check if automations should run |
| `check_test_triggers` | Every minute | Check for admin test triggers |

**How it works:**
- Users select hour (12-hour AM/PM format), stored as `HH:00` (24-hour)
- Celery compares `scheduled_hour == current_hour`
- 23-hour block prevents re-runs within same day

**Test Mode (Admin):**
- `test_mode_enabled` + `test_trigger_at` fields on `AutomationConfig`
- Allows immediate triggering without waiting for schedule
- Bypasses 23-hour blocking

---

## 7-Stage Pipeline

| Stage | Name | AI Function | Credit Cost | Can Skip |
|-------|------|-------------|-------------|----------|
| 1 | Keywords → Clusters | `AutoClusterFunction` | Per batch | ✅ |
| 2 | Clusters → Ideas | `GenerateIdeasFunction` | Per idea | ✅ |
| 3 | Ideas → Tasks | None (local) | None | ✅ |
| 4 | Tasks → Content | `GenerateContentFunction` | Per 100 words | ✅ |
| 5 | Content → Image Prompts | `GenerateImagePromptsFunction` | Per prompt | ✅ |
| 6 | Image Prompts → Images | `process_image_generation_queue` | Per image | ✅ |
| 7 | Review → Published | Publishing Scheduler | None | ✅ |

**Note:** Stage 7 uses the Publishing Scheduler with `PublishingSettings` for auto-approval and scheduling.

---

## Data Models

### AutomationConfig

| Field | Type | Purpose |
|-------|------|---------|
| account | FK | Owner account |
| site | FK | Target site |
| enabled | Boolean | Enable/disable automation |
| frequency | CharField | hourly/daily/weekly |
| scheduled_time | TimeField | Time to run |
| stage_1_batch_size | Integer | Keywords per batch |
| stage_2_batch_size | Integer | Clusters per batch |
| stage_3_batch_size | Integer | Ideas per batch |
| stage_4_batch_size | Integer | Tasks per batch |
| stage_5_batch_size | Integer | Content per batch |
| stage_6_batch_size | Integer | Images per batch |
| **stage_1_enabled** | Boolean | **v1.8.0** Enable stage 1 |
| **stage_2_enabled** | Boolean | **v1.8.0** Enable stage 2 |
| **stage_3_enabled** | Boolean | **v1.8.0** Enable stage 3 |
| **stage_4_enabled** | Boolean | **v1.8.0** Enable stage 4 |
| **stage_5_enabled** | Boolean | **v1.8.0** Enable stage 5 |
| **stage_6_enabled** | Boolean | **v1.8.0** Enable stage 6 |
| **stage_7_enabled** | Boolean | **v1.8.0** Enable stage 7 |
| **stage_X_use_testing** | Boolean | **v1.8.1** Use testing model (AI stages 1,2,4,5,6) |
| **stage_X_budget_pct** | Integer | **v1.8.1** Credit budget % (AI stages) |
| **max_keywords_per_run** | Integer | **v1.8.0** Per-run limit stage 1 |
| **max_clusters_per_run** | Integer | **v1.8.0** Per-run limit stage 2 |
| **max_ideas_per_run** | Integer | **v1.8.0** Per-run limit stage 3 |
| **max_tasks_per_run** | Integer | **v1.8.0** Per-run limit stage 4 |
| **max_content_per_run** | Integer | **v1.8.0** Per-run limit stage 5 |
| **max_images_per_run** | Integer | **v1.8.0** Per-run limit stage 6 |
| within_stage_delay | Integer | Seconds between batches |
| between_stage_delay | Integer | Seconds between stages |
| last_run_at | DateTime | Last execution |
| next_run_at | DateTime | Next scheduled run |
| **test_mode_enabled** | Boolean | **v1.8.1** Enable test mode for admin |
| **test_trigger_at** | DateTime | **v1.8.1** When to trigger test run |

### DefaultAutomationConfig (v1.8.1)

Singleton model for centralized default settings. See [AUTOMATION-AND-PUBLISHING-SCHEDULING.md](../40-WORKFLOWS/AUTOMATION-AND-PUBLISHING-SCHEDULING.md) for full schema.

| Field | Type | Purpose |
|-------|------|---------|
| is_enabled | Boolean | Default: Enable scheduled automation |
| frequency | CharField | Default frequency (daily/weekly/monthly) |
| next_scheduled_hour | Integer | Next hour to assign (auto-increments) |
| stage_X_enabled | Boolean | Default: Enable each stage |
| stage_X_batch_size | Integer | Default: Batch size per stage |
| stage_X_use_testing | Boolean | Default: Use testing model |
| stage_X_budget_pct | Integer | Default: Credit budget % |
| max_X_per_run | Integer | Default: Per-run limits |
| auto_approval_enabled | Boolean | Default: Auto-approve content |
| auto_publish_enabled | Boolean | Default: Auto-publish content |
| publish_days | JSONField | Default: Days to publish |
| publish_time_slots | JSONField | Default: Time slots to publish |

### AutomationRun

| Field | Type | Purpose |
|-------|------|---------|
| config | FK | Parent config |
| trigger_type | CharField | manual/scheduled |
| status | CharField | running/paused/cancelled/completed/failed |
| current_stage | Integer | Current stage (1-7) |
| started_at | DateTime | Start time |
| paused_at | DateTime | Pause time (nullable) |
| resumed_at | DateTime | Resume time (nullable) |
| cancelled_at | DateTime | Cancel time (nullable) |
| completed_at | DateTime | Completion time (nullable) |
| total_credits_used | Decimal | Total credits consumed |
| **initial_snapshot** | JSON | **v1.3.0** Queue sizes at run start |
| stage_1_result | JSON | Stage 1 results |
| stage_2_result | JSON | Stage 2 results |
| stage_3_result | JSON | Stage 3 results |
| stage_4_result | JSON | Stage 4 results |
| stage_5_result | JSON | Stage 5 results |
| stage_6_result | JSON | Stage 6 results |
| stage_7_result | JSON | Stage 7 results |
| error_message | TextField | Error details (nullable) |

---

## API Endpoints

| Method | Path | Handler | Purpose |
|--------|------|---------|---------|
| GET | `/api/v1/automation/config/` | Get/create config | Get automation config |
| PUT | `/api/v1/automation/update_config/` | Update config | Update settings |
| POST | `/api/v1/automation/run_now/` | Start manual run | Start automation |
| GET | `/api/v1/automation/current_run/` | Get current run | Run status/progress |
| GET | `/api/v1/automation/pipeline_overview/` | Get pipeline | Stage status counts |
| GET | `/api/v1/automation/current_processing/` | Get processing | Live processing status |
| **GET** | `/api/v1/automation/run_progress/` | **v1.3.0** | Unified progress data |
| POST | `/api/v1/automation/pause/` | Pause run | Pause after current item |
| POST | `/api/v1/automation/resume/` | Resume run | Resume from saved stage |
| POST | `/api/v1/automation/cancel/` | Cancel run | Cancel after current item |
| GET | `/api/v1/automation/history/` | Get history | Last 20 runs |
| GET | `/api/v1/automation/logs/` | Get logs | Activity log for run |
| GET | `/api/v1/automation/estimate/` | Get estimate | Credit estimate |
| **GET** | `/api/v1/integration/settings/defaults/` | **v1.8.1** | Get default settings for reset |

**Query Parameters:** All require `?site_id=`, run-specific require `?run_id=`

### Default Settings Endpoint (v1.8.1)

Returns centralized defaults from `DefaultAutomationConfig`:
```json
{
  "automation": { "enabled": false, "frequency": "daily", "time": "02:00" },
  "stages": [
    { "number": 1, "enabled": true, "batch_size": 50, "per_run_limit": 0, "use_testing": false, "budget_pct": 15 },
    ...
  ],
  "delays": { "within_stage": 3, "between_stage": 5 },
  "publishing": {
    "auto_approval_enabled": false,
    "auto_publish_enabled": false,
    "daily_publish_limit": 3,
    "publish_days": ["mon", "tue", "wed", "thu", "fri"],
    "time_slots": ["09:00", "14:00", "18:00"]
  }
}
```

### run_progress Endpoint (v1.3.0)

Returns unified progress data for frontend:
```json
{
  "run": { "run_id": "...", "status": "running", "current_stage": 3 },
  "global_progress": { "total_items": 100, "completed_items": 45, "percentage": 45 },
  "stages": [
    { "number": 1, "status": "completed", "input_count": 50, "processed_count": 50 },
    ...
  ],
  "metrics": { "credits_used": 120, "duration_seconds": 3600 },
  "initial_snapshot": { "stage_1_initial": 50, ... }
}
```

---

## Execution Flow

### Manual Run

1. User clicks "Run Now" on frontend
2. Frontend calls `POST /automation/run_now/?site_id=X`
3. Backend acquires cache lock `automation_lock_{site_id}`
4. **v1.3.0:** Captures initial snapshot with `_capture_initial_snapshot()`
5. Estimates credits required (1.2x buffer)
6. Validates balance >= estimate
7. Creates `AutomationRun` record
8. Enqueues `run_automation_task` Celery task
8. Returns run ID immediately

### Stage Execution

For each stage (1-7):

1. Check `_check_should_stop()` (paused/cancelled?)
2. Load items for processing
3. Process in batches (respecting batch_size)
4. For AI stages: Call AIEngine function
5. Wait `within_stage_delay` between batches
6. Save stage result JSON
7. Wait `between_stage_delay` before next stage

### Stage Result Fields

**Stage 1 (Clustering):**
```json
{
  "keywords_processed": 150,
  "clusters_created": 12,
  "batches_run": 3,
  "credits_used": 45,
  "time_elapsed": 120,
  "skipped": false,
  "partial": false
}
```

**Stage 2 (Ideas):**
```json
{
  "clusters_processed": 12,
  "ideas_created": 36,
  "batches_run": 2,
  "credits_used": 72
}
```

**Stage 3 (Tasks):**
```json
{
  "ideas_processed": 36,
  "tasks_created": 36,
  "batches_run": 4
}
```

**Stage 4 (Content):**
```json
{
  "tasks_processed": 36,
  "content_created": 36,
  "total_words": 54000,
  "batches_run": 6,
  "credits_used": 540
}
```

**Stage 5 (Image Prompts):**
```json
{
  "content_processed": 36,
  "prompts_created": 180,
  "batches_run": 4,
  "credits_used": 36
}
```

**Stage 6 (Images):**
```json
{
  "images_processed": 180,
  "images_generated": 180,
  "batches_run": 18
}
```

**Stage 7 (Review):**
```json
{
  "ready_for_review": 36
}
```

---

## Settings Configuration (v1.8.0)

**Location:** Site Settings → Automation tab  
**API:** `GET/PATCH /api/v1/integration/sites/{site_id}/unified-settings/`

> ⚠️ **v1.8.0 Change:** Settings are now consolidated in Site Settings. The previous standalone `/automation/settings` page has been removed.

### Settings UI Sections

1. **Schedule & Frequency Card**
   - Enable/disable toggle
   - Frequency (hourly/daily/weekly)
   - Days of week selection
   - Time slot selection
   - Next run display

2. **Capacity Card**
   - Total items per run (calculated)
   - Stage-by-stage breakdown

3. **AI Configuration Card**
   - Testing model selection (is_testing=true)
   - Live model selection (is_testing=false)
   - Image model selection

4. **Stage Configuration (Matrix)**
   - Per-stage Enable/Disable toggle
   - Per-stage batch size
   - Per-stage max items per run (**new in v1.8.0**)

5. **Help Cards**
   - Pipeline flow visualization
   - Stage descriptions

---

## Scheduling

**Celery Beat Task:** `check_scheduled_automations`  
**Frequency:** Hourly

**Logic:**
1. Find configs where `enabled=True`
2. Check if `next_run_at <= now`
3. Check if no active run exists
4. Start `run_automation_task` for eligible configs
5. Update `next_run_at` based on frequency

---

## Lock Mechanism

**Purpose:** Prevent concurrent runs for same site

**Key:** `automation_lock_{site_id}`  
**Storage:** Redis cache  
**Acquired:** On run start  
**Released:** On completion/failure/cancel

---

## Credit Validation

Before starting:
1. Calculate estimated credits for all stages
2. Apply 1.2x safety buffer
3. Compare with account balance
4. Reject if balance < estimate

During execution:
- Each AI stage checks credits before processing
- Deductions happen after successful AI calls
- `total_credits_used` accumulates across stages

---

## Frontend Integration

### AutomationOverview (v1.8.0)

**Path:** `/automation/overview`  
**Purpose:** Run history dashboard with:
- All automation runs with status
- Filter by status, date range
- Click to view detailed run information

### AutomationRunDetail (v1.8.0)

**Path:** `/automation/runs/:runId`  
**Purpose:** Detailed view of individual run with:
- Stage-by-stage progress
- Items processed per stage
- Errors and logs

### AutomationPage

**Path:** `/automation`  
**Purpose:** Manual run control with:
- **Pipeline Cards:** Stage-by-stage status with pending counts
- **Processing Card:** Live processing status during run
- **Control Buttons:** Run Now, Pause, Resume, Cancel
- **Activity Log:** Real-time log streaming

### Polling

- Every ~5s while run is running/paused
- Fetches: `current_run`, `pipeline_overview`, `current_processing`
- Lighter polling when idle

---

## Common Issues

| Issue | Cause | Fix |
|-------|-------|-----|
| "Already running" error | Lock exists from previous run | Wait or check if stuck |
| Insufficient credits | Balance < 1.2x estimate | Add credits |
| Stage skipped | No items to process | Check previous stages |
| Run stuck | Worker crashed | Clear lock, restart |
| Images not generating | Stage 5 didn't create prompts | Check stage 5 result |

---

## Changelog

| Version | Changes |
|---------|---------|
| v1.8.0 | Unified settings in Site Settings, per-run limits, skip stage functionality |
| v1.7.0 | AutomationOverview dashboard, AutomationRunDetail page |
| v1.3.2 | Stage 7 publishing scheduler integration |
| v1.3.0 | Progress tracking with initial snapshots |