igny8/automation-plan.md

# AI Automation Pipeline - Complete Implementation Plan
**Version:** 2.0  
**Date:** December 3, 2025  
**Scope:** Site-level automation orchestrating existing AI functions

---

## 🎯 CORE ARCHITECTURE DECISIONS

### Decision 1: Site-Level Automation (NO Sector)
**Rationale:**
- User manages automation per website, not per topic/sector
- Simpler UX - single site selector at top of page
- Database queries filter by `site_id` only (no sector_id filtering)
- Content naturally spans multiple sectors within a site
- One automation schedule per site (not per site/sector combination)

**Implementation:**
- Remove sector dropdown from automation page UI
- AutomationRun model: Remove sector foreign key
- AutomationConfig model: One config per site (not per site+sector)
- All stage database queries: `.filter(site=site)` (no sector filter)

---

### Decision 2: Single Global Automation Page
**Why:**
- Complete pipeline visibility in one place (Keywords → Draft Content)
- Configure one schedule for entire lifecycle
- See exactly where pipeline is stuck or running
- Cleaner UX - no jumping between module pages

**Location:** `/automation` (new route below Sites in sidebar)

---

### Decision 3: Strictly Sequential Stages (Never Parallel)
**Critical Principle:**
- Stage N+1 ONLY starts when Stage N is 100% complete
- Within each stage: process items in batches sequentially
- Hard stop between stages to verify completion
- Only ONE stage active at a time per site

**Example Flow:**
```
Stage 1 starts → processes ALL batches → completes 100%
  ↓ (trigger next)
Stage 2 starts → processes ALL batches → completes 100%
  ↓ (trigger next)
Stage 3 starts → ...
```

**Never:**
- Run stages in parallel
- Start next stage while current stage has pending items
- Skip verification between stages

---

### Decision 4: Automation Stops Before Publishing
**Manual Review Gate (Stage 7):**
- Automation ends when content reaches `status='draft'` with all images generated
- User manually reviews content quality, accuracy, brand voice
- User manually publishes via existing bulk actions on Content page
- No automated WordPress publishing (requires human oversight)

**Rationale:**
- Content quality control needed
- Publishing has real consequences (public-facing)
- Legal/compliance review may be required
- Brand voice verification essential

---

## 📊 EXISTING AI FUNCTIONS (Zero Duplication)

```
┌──────────────────────────────────────────────────────────────┐
│  🤖 AI AUTOMATION PIPELINE                                   │
│  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  │
│                                                              │
│  ⏰ SCHEDULE                                                 │
│  Next Run: Tomorrow at 2:00 AM (in 16 hours)               │
│  Frequency: [Daily ▼] at [02:00 ▼]                         │
│  Status: ● Scheduled                                        │
│                                                              │
│  [Run Now] [Pause Schedule] [Configure]                     │
│                                                              │
├──────────────────────────────────────────────────────────────┤
│  📊 PIPELINE OVERVIEW                                        │
│                                                              │
│  Keywords ──→ Clusters ──→ Ideas ──→ Tasks ──→ Content      │
│    47         pending      42       20       generating     │
│  pending      Stage 1      ready    queued   Stage 5        │
│                                                              │
│  Overall Progress: ━━━━━━━╸ 62% (Stage 5/7)                │
│  Estimated Completion: 2 hours 15 minutes                   │
│                                                              │
└──────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────┐
│  STAGE 1: Keywords → Clusters (AI)                          │
│  Status: ✓ Completed                                        │
│  • Processed: 60 keywords → 8 clusters                      │
│  • Time: 2m 30s | Credits: 12                               │
│  [View Details] [Retry Failed]                              │
└──────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────┐
│  STAGE 2: Clusters → Ideas (AI)                             │
│  Status: ✓ Completed                                        │
│  • Processed: 8 clusters → 56 ideas                         │
│  • Time: 8m 15s | Credits: 16                               │
│  [View Details]                                              │
└──────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────┐
│  STAGE 3: Ideas → Tasks (Local Queue)                       │
│  Status: ✓ Completed                                        │
│  • Processed: 42 ideas → 42 tasks                           │
│  • Time: Instant | Credits: 0                               │
│  [View Details]                                              │
└──────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────┐
│  STAGE 4: Tasks → Content (AI)                              │
│  Status: ● Processing (Task 3/20)                           │
│  • Current: "Ultimate Coffee Bean Guide" ━━━━╸ 65%         │
│  • Progress: 2 completed, 1 processing, 17 queued           │
│  • Time: 45m elapsed | Credits: 38 used                     │
│  • ETA: 1h 30m remaining                                    │
│  [View Details] [Pause Stage]                               │
└──────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────┐
│  STAGE 5: Content → Image Prompts (AI)                      │
│  Status: ⏸ Waiting (Stage 4 must complete)                  │
│  • Pending: 2 content pieces ready for prompts              │
│  • Queue: Will process when Stage 4 completes               │
│  [View Details] [Trigger Now]                               │
└──────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────┐
│  STAGE 6: Image Prompts → Generated Images (AI)             │
│  Status: ⏸ Waiting                                           │
│  • Pending: 0 prompts ready                                 │
│  [View Details]                                              │
└──────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────┐
│  STAGE 7: Content → Review (Manual Gate) 🚫 STOPS HERE      │
│  Status: ⏸ Awaiting Manual Review                           │
│  • Ready for Review: 2 content pieces                       │
│  • Note: Automation stops here. User reviews manually.      │
│  [Go to Review Page]                                         │
└──────────────────────────────────────────────────────────────┘

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📋 LIVE ACTIVITY LOG (Last 50 events)
├─ 14:23:45 - Stage 4: Started content generation for Task 3
├─ 14:24:12 - Stage 4: Writing sections (65% complete)
├─ 14:22:30 - Stage 4: Completed Task 2 → Content created
├─ 14:20:15 - Stage 4: Started content generation for Task 2
├─ 14:18:45 - Stage 4: Completed Task 1 → Content created
└─ [View Full Log]

💰 TOTAL CREDITS USED THIS RUN: 66 credits
```

**All 6 AI Functions Already Exist and Work:**

| Function | File Location | Input | Output | Credits | Status |
|----------|---------------|-------|--------|---------|--------|
| **auto_cluster** | `ai/functions/auto_cluster.py` | Keyword IDs (max 20) | Clusters created | 1 per 5 keywords | ✅ Working |
| **generate_ideas** | `ai/functions/generate_ideas.py` | Cluster IDs (max 5) | Ideas created | 2 per cluster | ✅ Working |
| **bulk_queue_to_writer** | `modules/planner/views.py` (line 1014) | Idea IDs | Tasks created | 0 (local) | ✅ Working |
| **generate_content** | `ai/functions/generate_content.py` | Task IDs (1 at a time) | Content draft | 1 per 500 words | ✅ Working |
| **generate_image_prompts** | `ai/functions/generate_image_prompts.py` | Content IDs | Image prompts | 0.5 per prompt | ✅ Working |
| **generate_images** | `ai/functions/generate_images.py` | Image prompt IDs | Generated images | 1-4 per image | ✅ Working |

---

### 🚫 WHAT AI FUNCTIONS ALREADY DO (DO NOT DUPLICATE)

**Credit Management** (Fully Automated in `ai/engine.py`):
```python
# Line 395 in AIEngine.execute():
CreditService.deduct_credits_for_operation(
    account=account,
    operation_type=self._get_operation_type(),
    amount=self._get_actual_amount(),
    ...
)
```
- ✅ Credits are AUTOMATICALLY deducted after successful save
- ✅ Credit calculation happens in `_get_actual_amount()` and `_get_operation_type()`
- ❌ Automation does NOT need to call `CreditService` manually
- ❌ Automation does NOT need to calculate credit costs

**Status Updates** (Handled Inside AI Functions):
- ✅ Keywords: `status='new'` → `status='mapped'` (in auto_cluster save_output)
- ✅ Clusters: Created with `status='new'` (in auto_cluster save_output)
- ✅ Ideas: `status='new'` → `status='queued'` (in bulk_queue_to_writer)
- ✅ Tasks: Created with `status='queued'`, → `status='completed'` (in generate_content)
- ✅ Content: Created with `status='draft'`, → `status='review'` ONLY when all images complete (ai/tasks.py line 723)
- ✅ Images: `status='pending'` → `status='generated'` (in generate_images save_output)
- ❌ Automation does NOT update these statuses directly

**Progress Tracking** (Event-Based System Already Exists):
- ✅ `StepTracker` and `ProgressTracker` emit real-time events during AI execution
- ✅ Each AI function has 6 phases: `INIT`, `PREP`, `AI_CALL`, `PARSE`, `SAVE`, `DONE`
- ✅ Phase descriptions available in function metadata: `get_metadata()`
- ❌ Automation does NOT need to poll progress every 2 seconds
- ❌ Automation listens to existing phase events via Celery task status

**Error Handling & Logging**:
- ✅ AIEngine wraps execution in try/catch, logs to `AIUsageLog`
- ✅ Failed operations rollback database changes automatically
- ❌ Automation only needs to check final task result (success/failure)

---

**Automation Service ONLY Does:**
1. **Batch Selection**: Query database for items to process (by status and site)
2. **Function Calling**: Call existing AI functions with selected item IDs
3. **Stage Sequencing**: Wait for Stage N completion before starting Stage N+1
4. **Scheduling**: Trigger automation runs on configurable schedules
5. **Aggregation**: Collect results from all batches and log totals per stage

---

## 🏗️ 7-STAGE PIPELINE ARCHITECTURE

### Sequential Stage Flow

| Stage | From | To | Function Used | Batch Size | Type |
|-------|------|-----|---------------|------------|------|
| **1** | Keywords (`status='new'`, `cluster_id=null`) | Clusters (`status='new'`) | `auto_cluster` | 20 keywords | AI |
| **2** | Clusters (`status='new'`, no ideas) | Ideas (`status='new'`) | `generate_ideas` | 1 cluster | AI |
| **3** | Ideas (`status='new'`) | Tasks (`status='queued'`) | `bulk_queue_to_writer` | 20 ideas | Local |
| **4** | Tasks (`status='queued'`) | Content (`status='draft'`) | `generate_content` | 1 task | AI |
| **5** | Content (`status='draft'`, no Images) | Images (`status='pending'` with prompts) | `generate_image_prompts` | 1 content | AI |
| **6** | Images (`status='pending'`) | Images (`status='generated'` with URLs) | `generate_images` | 1 image | AI |
| **7** | Content (`status='review'`) | Manual Review | None (gate) | N/A | Manual |

---

### Stage 1: Keywords → Clusters (AI)

**Purpose:** Group semantically similar keywords into topic clusters

**Database Query (Automation Orchestrator):**
```python
pending_keywords = Keywords.objects.filter(
    site=site,
    status='new',
    cluster__isnull=True,
    disabled=False
)
```

**Orchestration Logic (What Automation Does):**
1. **Select Batch**: Count pending keywords
   - If 0 keywords → Skip stage, log "No keywords to process"
   - If 1-20 keywords → Select all (batch_size = count)
   - If >20 keywords → Select first 20 (configurable batch_size)

2. **Call AI Function**:
   ```python
   from igny8_core.ai.functions.auto_cluster import AutoCluster
   
   result = AutoCluster().execute(
       payload={'ids': keyword_ids},
       account=account
   )
   # Returns: {'task_id': 'celery_task_abc123'}
   ```

3. **Monitor Progress**: Listen to Celery task status
   - Use existing `StepTracker` phase events (INIT → PREP → AI_CALL → PARSE → SAVE → DONE)
   - OR poll: `AsyncResult(task_id).state` until SUCCESS/FAILURE
   - Log phase progress: "AI analyzing keywords (65% complete)"

4. **Collect Results**: When task completes
   - AI function already updated Keywords.status → 'mapped'
   - AI function already created Cluster records with status='new'
   - AI function already deducted credits via AIEngine
   - Automation just logs: "Batch complete: N clusters created"

5. **Repeat**: If more keywords remain, select next batch and go to step 2

**Stage Completion Criteria:**
- All keyword batches processed (pending_keywords.count() == 0)
- No critical errors

**What AI Function Does (Already Implemented - DON'T DUPLICATE):**
- ✅ Groups keywords semantically using AI
- ✅ Creates Cluster records with `status='new'`
- ✅ Updates Keywords: `cluster_id=cluster.id`, `status='mapped'`
- ✅ Deducts credits automatically (AIEngine line 395)
- ✅ Logs to AIUsageLog
- ✅ Emits progress events via StepTracker

**Stage Result Logged:**
```json
{
  "keywords_processed": 47,
  "clusters_created": 8,
  "batches_run": 3,
  "credits_used": 10  // Read from AIUsageLog sum, not calculated
}
```

---

### Stage 2: Clusters → Ideas (AI)

**Purpose:** Generate content ideas for each cluster

**Database Query:**
```
Clusters.objects.filter(
    site=site,
    status='new',
    disabled=False
).exclude(
    ideas__isnull=False  # Has no ideas yet
)
```

**Process:**
1. Count clusters without ideas
2. If 0 → Skip stage
3. If > 0 → Process one cluster at a time (configurable batch size = 1)
4. For each cluster:
   - Log: "Generating ideas for cluster: {cluster.name}"
   - Call `IdeasService.generate_ideas(cluster_ids=[cluster.id], account)`
   - Function returns `{'task_id': 'xyz789'}`
   - Monitor via Celery task status or StepTracker events
   - Wait for completion
   - Log: "Cluster '{name}' complete: N ideas created"
5. Log stage summary

**Stage Completion Criteria:**
- All clusters processed
- Each cluster now has >=1 idea
- No errors

**Updates:**
- ContentIdeas: New records created with `status='new'`, `keyword_cluster_id=cluster.id`
- Clusters: `status='mapped'`
- Stage result: `{clusters_processed: 8, ideas_created: 56}`

**Credits:** ~16 credits (2 per cluster)

---

### Stage 3: Ideas → Tasks (Local Queue)

**Purpose:** Convert content ideas to writer tasks (local, no AI)

**Database Query:**
```
ContentIdeas.objects.filter(
    site=site,
    status='new'
)
```

**Process:**
1. Count pending ideas
2. If 0 → Skip stage
3. If > 0 → Split into batches of 20
4. For each batch:
   - Log: "Queueing batch X/Y (20 ideas)"
   - Call `bulk_queue_to_writer` view logic (NOT via HTTP, direct function call)
   - For each idea:
     - Create Tasks record with title=idea.idea_title, status='queued', cluster=idea.keyword_cluster
     - Update idea status to 'queued'
   - Log: "Batch X complete: 20 tasks created"
5. Log stage summary

**Stage Completion Criteria:**
- All batches processed
- All ideas now have `status='queued'`
- Corresponding Tasks exist with `status='queued'`
- No errors

**Updates:**
- Tasks: New records created with `status='queued'`
- ContentIdeas: `status` changed 'new' → 'queued'
- Stage result: `{ideas_processed: 56, tasks_created: 56, batches: 3}`

**Credits:** 0 (local operation)

---

### Stage 4: Tasks → Content (AI)

**Purpose:** Generate full content drafts from tasks

**Database Query (Automation Orchestrator):**
```python
pending_tasks = Tasks.objects.filter(
    site=site,
    status='queued',
    content__isnull=True  # No content generated yet
)
```

**Orchestration Logic:**
1. **Select Item**: Count queued tasks
   - If 0 → Skip stage
   - If > 0 → Select ONE task at a time (sequential processing)

2. **Call AI Function**:
   ```python
   from igny8_core.ai.functions.generate_content import GenerateContent
   
   result = GenerateContent().execute(
       payload={'ids': [task.id]},
       account=account
   )
   # Returns: {'task_id': 'celery_task_xyz789'}
   ```

3. **Monitor Progress**: Listen to Celery task status
   - Use `StepTracker` phase events for real-time updates
   - Log: "Writing sections (65% complete)" (from phase metadata)
   - Content generation takes 5-15 minutes per task

4. **Collect Results**: When task completes
   - AI function already created Content with `status='draft'`
   - AI function already updated Task.status → 'completed'
   - AI function already updated Idea.status → 'completed'
   - AI function already deducted credits based on word count
   - Automation logs: "Content created (2500 words)"

5. **Repeat**: Process next task sequentially

**Stage Completion Criteria:**
- All tasks processed (pending_tasks.count() == 0)
- Each task has linked Content record

**What AI Function Does (Already Implemented):**
- ✅ Generates article sections using AI
- ✅ Creates Content record with `status='draft'`, `task_id=task.id`
- ✅ Updates Task: `status='completed'`
- ✅ Updates linked Idea: `status='completed'`
- ✅ Deducts credits: 1 credit per 500 words (automatic)
- ✅ Logs to AIUsageLog with word count

**Stage Result Logged:**
```json
{
  "tasks_processed": 56,
  "content_created": 56,
  "total_words": 140000,
  "credits_used": 280  // From AIUsageLog, not calculated
}
```

---

### Stage 5: Content → Image Prompts (AI)

**Purpose:** Extract image prompts from content and create Images records with prompts

**CRITICAL:** There is NO separate "ImagePrompts" model. Images records ARE the prompts (with `status='pending'`) until images are generated.

**Database Query (Automation Orchestrator):**
```python
# Content that has NO Images records at all
content_without_images = Content.objects.filter(
    site=site,
    status='draft'
).annotate(
    images_count=Count('images')
).filter(
    images_count=0  # No Images records exist yet
)
```

**Orchestration Logic:**
1. **Select Item**: Count content without any Images records
   - If 0 → Skip stage
   - If > 0 → Select ONE content at a time (sequential)

2. **Call AI Function**:
   ```python
   from igny8_core.ai.functions.generate_image_prompts import GenerateImagePromptsFunction
   
   result = GenerateImagePromptsFunction().execute(
       payload={'ids': [content.id]},
       account=account
   )
   # Returns: {'task_id': 'celery_task_prompts456'}
   ```

3. **Monitor Progress**: Wait for completion

4. **Collect Results**: When task completes
   - AI function already created Images records with:
     - `status='pending'`
     - `prompt='...'` (AI-generated prompt text)
     - `image_type='featured'` or `'in_article'`
     - `content_id=content.id`
   - Content.status stays `'draft'` (unchanged)
   - Automation logs: "Content '{title}' complete: N prompts created"

5. **Repeat**: Process next content sequentially

**Stage Completion Criteria:**
- All content processed (content_without_images.count() == 0)
- Each content has >=1 Images record with `status='pending'` and prompt text

**What AI Function Does (Already Implemented):**
- ✅ Extracts featured image prompt from title/intro
- ✅ Extracts in-article prompts from H2 headings
- ✅ Creates Images records with `status='pending'`, `prompt='...'`
- ✅ Deducts credits automatically (0.5 per prompt)
- ✅ Logs to AIUsageLog

**Stage Result Logged:**
```json
{
  "content_processed": 56,
  "prompts_created": 224,
  "credits_used": 112  // From AIUsageLog
}
```

---

### Stage 6: Images (Prompts) → Generated Images (AI)

**Purpose:** Generate actual image URLs from Images records that contain prompts

**CRITICAL:** Input is Images records with `status='pending'` (these contain the prompts). Output is same Images records updated with `status='generated'` and `image_url='https://...'`

**Database Query (Automation Orchestrator):**
```python
# Images with prompts waiting to be generated
pending_images = Images.objects.filter(
    site=site,
    status='pending'  # Has prompt text, needs image URL
)
```

**Orchestration Logic:**
1. **Select Item**: Count pending Images
   - If 0 → Skip stage
   - If > 0 → Select ONE Image at a time (sequential)

2. **Call AI Function**:
   ```python
   from igny8_core.ai.functions.generate_images import GenerateImages
   
   result = GenerateImages().execute(
       payload={'image_ids': [image.id]},
       account=account
   )
   # Returns: {'task_id': 'celery_task_img789'}
   ```

3. **Monitor Progress**: Wait for completion

4. **Collect Results**: When task completes
   - AI function already called image API using the `prompt` field
   - AI function already updated Images:
     - `status='pending'` → `status='generated'`
     - `image_url='https://...'` (populated with generated image URL)
   - AI function already deducted credits (1-4 per image)
   - Automation logs: "Image generated: {image_url}"

5. **Automatic Content Status Change** (NOT done by automation):
   - After each image generation, background task checks if ALL Images for that Content are now `status='generated'`
   - When last image completes: Content.status changes `'draft'` → `'review'` (in `ai/tasks.py` line 723)
   - Automation does NOT trigger this - happens automatically

6. **Repeat**: Process next pending Image sequentially

**Stage Completion Criteria:**
- All pending Images processed (pending_images.count() == 0)
- All Images now have `image_url != null`, `status='generated'`

**What AI Function Does (Already Implemented):**
- ✅ Reads `prompt` field from Images record
- ✅ Calls image generation API (OpenAI/Runware) with prompt
- ✅ Updates Images: `image_url=generated_url`, `status='generated'`
- ✅ Deducts credits automatically (1-4 per image)
- ✅ Logs to AIUsageLog

**What Happens Automatically (ai/tasks.py:723):**
- ✅ Background task checks if all Images for a Content are `status='generated'`
- ✅ When complete: Content.status changes `'draft'` → `'review'`
- ✅ This happens OUTSIDE automation orchestrator (in Celery task)

**Stage Result Logged:**
```json
{
  "images_processed": 224,
  "images_generated": 224,
  "content_moved_to_review": 56,  // Side effect (automatic)
  "credits_used": 448  // From AIUsageLog
}
```

---

### Stage 7: Manual Review Gate (STOP)

**Purpose:** Automation ends - content automatically moved to 'review' status ready for manual review

**CRITICAL:** Content with `status='review'` was automatically set in Stage 6 when ALL images completed. Automation just counts them.

**Database Query (Automation Orchestrator):**
```python
# Content that has ALL images generated (status already changed to 'review')
ready_for_review = Content.objects.filter(
    site=site,
    status='review'  # Automatically set when all images complete
)
```

**Orchestration Logic:**
1. **Count Only**: Count content with `status='review'`
   - No processing, just counting
   - These Content records already have all Images with `status='generated'`

2. **Log Results**:
   - Log: "Automation complete. X content pieces ready for review"
   - Log: "Content IDs ready: [123, 456, 789, ...]"

3. **Mark Run Complete**:
   - AutomationRun.status = 'completed'
   - AutomationRun.completed_at = now()

4. **Send Notification** (optional):
   - Email/notification: "Your automation run completed. X content pieces ready for review"

5. **STOP**: No further automation stages

**Stage Completion Criteria:**
- Counting complete
- Automation run marked `status='completed'`

**What AI Function Does:**
- N/A - No AI function called in this stage

**Stage Result Logged:**
```json
{
  "ready_for_review": 56,
  "content_ids": [123, 456, 789, ...]
}
```

**What Happens Next (Manual - User Action):**
1. User navigates to `/writer/content` page
2. Content page shows filter: `status='review'`
3. User sees 56 content pieces with all images generated
4. User manually reviews:
   - Content quality
   - Image relevance
   - Brand voice
   - Accuracy
5. User selects multiple content → "Bulk Publish" action
6. Existing WordPress publishing workflow executes

**Why Manual Review is Required:**
- Quality control before public publishing
- Legal/compliance verification
- Brand voice consistency check
- Final accuracy confirmation

---

## 🔄 BATCH PROCESSING WITHIN STAGES

### Critical Concepts

**Batch vs Queue:**
- **Batch:** Group of items processed together in ONE AI call
- **Queue:** Total pending items waiting to be processed

**Example - Stage 1 with 47 keywords:**
```
Total Queue: 47 keywords
Batch Size: 20

Execution:
  Batch 1: Keywords 1-20 → Call auto_cluster → Wait for completion
  Batch 2: Keywords 21-40 → Call auto_cluster → Wait for completion
  Batch 3: Keywords 41-47 → Call auto_cluster → Wait for completion

Total Batches: 3
Processing: Sequential (never parallel)
```

**UI Display:**
```
Stage 1: Keywords → Clusters
Status: ● Processing
Queue: 47 keywords total
Progress: Batch 2/3 (40 processed, 7 remaining)
Current: Processing keywords 21-40
Time Elapsed: 4m 30s
Credits Used: 8
```

### Batch Completion Triggers

**Within Stage:**
- Batch completes → Immediately start next batch
- Last batch completes → Stage complete

**Between Stages:**
- Stage N completes → Trigger Stage N+1 automatically
- Hard verification: Ensure queue is empty before proceeding

**Detailed Stage Processing Queues (UI Elements):**

Each stage card should show:
1. **Total Queue Count** - How many items need processing in this stage
2. **Current Batch** - Which batch is being processed (e.g., "Batch 2/5")
3. **Processed Count** - How many items completed so far
4. **Remaining Count** - How many items left in queue
5. **Current Item** - What specific item is processing right now (for single-item batches)

**Example UI for Stage 4:**
```
┌──────────────────────────────────────────────────────────────┐
│  STAGE 4: Tasks → Content (AI)                              │
│  Status: ● Processing                                        │
│                                                              │
│  📊 QUEUE OVERVIEW:                                          │
│  ├─ Total Tasks: 56                                         │
│  ├─ Processed: 23                                           │
│  ├─ Remaining: 33                                           │
│  └─ Progress: ━━━━━━━╸━━━━━━━━━━━━ 41%                      │
│                                                              │
│  🔄 CURRENT PROCESSING:                                      │
│  ├─ Item: Task 24/56                                        │
│  ├─ Title: "Ultimate Coffee Bean Buying Guide"             │
│  ├─ Progress: Writing sections (65% complete)               │
│  └─ Time: 2m 15s elapsed                                    │
│                                                              │
│  💳 STAGE STATS:                                             │
│  ├─ Credits Used: 46                                        │
│  ├─ Time Elapsed: 1h 23m                                    │
│  └─ ETA: 1h 15m remaining                                   │
│                                                              │
│  [View Details] [Pause Stage]                               │
└──────────────────────────────────────────────────────────────┘
```

---

## 🗄️ DATABASE STRUCTURE

### New Models to Create

**AutomationRun** (tracks each automation execution)
```
Table: igny8_automation_runs
Fields:
  - id: Integer (PK)
  - run_id: String (unique, indexed) - Format: run_20251203_140523_manual
  - account_id: ForeignKey(Account)
  - site_id: ForeignKey(Site)
  - trigger_type: String - Choices: 'manual', 'scheduled'
  - status: String - Choices: 'running', 'paused', 'completed', 'failed'
  - current_stage: Integer - Current stage number (1-7)
  - started_at: DateTime
  - completed_at: DateTime (nullable)
  - total_credits_used: Integer
  - stage_1_result: JSON - {keywords_processed, clusters_created, batches}
  - stage_2_result: JSON - {clusters_processed, ideas_created}
  - stage_3_result: JSON - {ideas_processed, tasks_created}
  - stage_4_result: JSON - {tasks_processed, content_created, total_words}
  - stage_5_result: JSON - {content_processed, prompts_created}
  - stage_6_result: JSON - {prompts_processed, images_generated}
  - stage_7_result: JSON - {ready_for_review}
  - error_message: Text (nullable)

Indexes:
  - run_id (unique)
  - site_id, started_at
  - status, started_at
```

**AutomationConfig** (per-site configuration)
```
Table: igny8_automation_configs
Fields:
  - id: Integer (PK)
  - account_id: ForeignKey(Account)
  - site_id: ForeignKey(Site, unique) - ONE config per site
  - is_enabled: Boolean - Whether scheduled automation is active
  - frequency: String - Choices: 'daily', 'weekly', 'monthly'
  - scheduled_time: Time - When to run (e.g., 02:00)
  - stage_1_batch_size: Integer - Default 20 (keywords per batch)
  - stage_2_batch_size: Integer - Default 1 (clusters at a time)
  - stage_3_batch_size: Integer - Default 20 (ideas per batch)
  - stage_4_batch_size: Integer - Default 1 (tasks - sequential)
  - stage_5_batch_size: Integer - Default 1 (content at a time)
  - stage_6_batch_size: Integer - Default 1 (images - sequential)
  - last_run_at: DateTime (nullable)
  - next_run_at: DateTime (nullable) - Calculated based on frequency

Constraints:
  - Unique: site_id (one config per site)
```

### File-Based Logging Structure

**Directory Structure:**
```
logs/
└── automation/
    └── {account_id}/
        └── {site_id}/
            └── {run_id}/
                ├── automation_run.log (main activity log)
                ├── stage_1.log (keywords → clusters)
                ├── stage_2.log (clusters → ideas)
                ├── stage_3.log (ideas → tasks)
                ├── stage_4.log (tasks → content)
                ├── stage_5.log (content → prompts)
                ├── stage_6.log (prompts → images)
                └── stage_7.log (review gate)
```

**Log File Format (automation_run.log):**
```
========================================
AUTOMATION RUN: run_20251203_140523_manual
Started: 2025-12-03 14:05:23
Trigger: manual
Account: 5
Site: 12
========================================

14:05:23 - Automation started (trigger: manual)
14:05:23 - Credit check: Account has 1500 credits, estimated need: 866 credits
14:05:23 - Stage 1 starting: Keywords → Clusters
14:05:24 - Stage 1: Found 47 pending keywords
14:05:24 - Stage 1: Processing batch 1/3 (20 keywords)
14:05:25 - Stage 1: AI task queued: task_id=abc123
14:07:30 - Stage 1: Batch 1 complete - 3 clusters created
14:07:31 - Stage 1: Processing batch 2/3 (20 keywords)
[... continues ...]
```

**Stage-Specific Log (stage_1.log):**
```
========================================
STAGE 1: Keywords → Clusters (AI)
Started: 2025-12-03 14:05:23
========================================

14:05:24 - Query: Keywords.objects.filter(site=12, status='new', cluster__isnull=True)
14:05:24 - Found 47 pending keywords
14:05:24 - Batch size: 20 keywords
14:05:24 - Total batches: 3

--- Batch 1/3 ---
14:05:24 - Keyword IDs: [101, 102, 103, ..., 120]
14:05:25 - Calling ClusteringService.cluster_keywords(ids=[101..120], account=5, site_id=12)
14:05:25 - AI task queued: task_id=abc123
14:05:26 - Monitoring task status...
14:05:28 - Phase: INIT - Initializing (StepTracker event)
14:05:45 - Phase: AI_CALL - AI analyzing keywords (StepTracker event)
14:07:15 - Phase: SAVE - Creating clusters (StepTracker event)
14:07:30 - Phase: DONE - Complete
14:07:30 - Result: 3 clusters created
14:07:30 - Clusters: ["Coffee Beans", "Brewing Methods", "Coffee Equipment"]
14:07:30 - Credits used: 4 (from AIUsageLog)

--- Batch 2/3 ---
[... continues ...]

========================================
STAGE 1 COMPLETE
Total Time: 5m 30s
Processed: 47 keywords
Clusters Created: 8
Credits Used: 10
========================================
```

---

## 🔐 SAFETY MECHANISMS

### 1. Concurrency Control (Prevent Duplicate Runs)

**Problem:** User clicks "Run Now" while scheduled task is running

**Solution:** Distributed locking using Django cache

**Implementation Logic:**
```
When starting automation:
  1. Try to acquire lock: cache.add(f'automation_lock_{site.id}', 'locked', timeout=21600)
  2. If lock exists → Return error: "Automation already running for this site"
  3. If lock acquired → Proceed with run
  4. On completion/failure → Release lock: cache.delete(f'automation_lock_{site.id}')

Also check database:
  - Query AutomationRun.objects.filter(site=site, status='running').exists()
  - If exists → Error: "Another automation is running"
```

**User sees:**
- "Automation already in progress. Started at 02:00 AM, currently on Stage 4."
- Link to view current run progress

---

### 2. Credit Reservation (Prevent Mid-Run Failures)

**Problem:** Account runs out of credits during Stage 4

**Solution:** Reserve estimated credits at start, deduct as used

**Implementation Logic:**
```
Before starting:
  1. Estimate total credits needed:
     - Count keywords → estimate clustering credits
     - Count clusters → estimate ideas credits
     - Estimate content generation (assume avg word count)
     - Estimate image generation (assume 4 images per content)
  2. Check: account.credits_balance >= estimated_credits * 1.2 (20% buffer)
  3. If insufficient → Error: "Need ~866 credits, you have 500"
  4. Reserve credits: account.credits_reserved += estimated_credits
  5. As each stage completes → Deduct actual: account.credits_balance -= actual_used
  6. On completion → Release unused: account.credits_reserved -= unused

Database fields needed:
  - Account.credits_reserved (new field)
```

---

### 3. Stage Idempotency (Safe to Retry)

**Problem:** User resumes paused run, Stage 1 runs again creating duplicate clusters

**Solution:** Check if stage already completed before executing

**Implementation Logic:**
```
At start of each run_stage_N():
  1. Check AutomationRun.stage_N_result
  2. If result exists and has processed_count > 0:
     - Log: "Stage N already completed - skipping"
     - return (skip to next stage)
  3. Else: Proceed with stage execution
```

---

### 4. Celery Task Chaining (Non-Blocking Workers)

**Problem:** Synchronous execution blocks Celery worker for hours

**Solution:** Chain stages as separate Celery tasks

**Implementation Logic:**
```
Instead of:
  def start_automation():
      run_stage_1()  # blocks for 30 min
      run_stage_2()  # blocks for 45 min
      ...

Do:
  @shared_task
  def run_stage_1_task(run_id):
      service = AutomationService.from_run_id(run_id)
      service.run_stage_1()
      # Trigger next stage
      run_stage_2_task.apply_async(args=[run_id], countdown=5)

  @shared_task
  def run_stage_2_task(run_id):
      service = AutomationService.from_run_id(run_id)
      service.run_stage_2()
      run_stage_3_task.apply_async(args=[run_id], countdown=5)

Benefits:
  - Workers not blocked for hours
  - Can retry individual stages
  - Better monitoring in Celery Flower
  - Horizontal scaling possible
```

---

### 5. Pause/Resume Capability

**User Can:**
- Pause automation at any point
- Resume from where it left off

**Implementation Logic:**
```
Pause:
  - Update AutomationRun.status = 'paused'
  - Current stage completes current batch then stops
  - Celery task checks status before each batch

Resume:
  - Update AutomationRun.status = 'running'
  - Restart from current_stage
  - Use idempotency check to skip completed work
```

---

### 6. Error Handling Per Stage

**If Stage Fails:**
```
try:
    run_stage_1()
except Exception as e:
    - Log error to stage_1.log
    - Update AutomationRun:
        - status = 'failed'
        - error_message = str(e)
        - current_stage = 1 (where it failed)
    - Send notification: "Automation failed at Stage 1"
    - Stop execution (don't proceed to Stage 2)
    
User can:
    - View logs to see what went wrong
    - Fix issue (e.g., add credits)
    - Click "Resume" to retry from Stage 1
```

---

### 7. Log Cleanup (Prevent Disk Bloat)

**Problem:** After 1000 runs, logs occupy 80MB+ per site

**Solution:** Celery periodic task to delete old logs

**Implementation Logic:**
```
@shared_task
def cleanup_old_automation_logs():
    cutoff = datetime.now() - timedelta(days=90)  # Keep last 90 days
    
    old_runs = AutomationRun.objects.filter(
        started_at__lt=cutoff,
        status__in=['completed', 'failed']
    )
    
    for run in old_runs:
        log_dir = f'logs/automation/{run.account_id}/{run.site_id}/{run.run_id}/'
        shutil.rmtree(log_dir)  # Delete directory
        run.delete()  # Remove DB record

Schedule: Weekly, Monday 3 AM
```

---

## 🎨 FRONTEND DESIGN

### Page Structure: `/automation`

**Layout:**
```
┌─────────────────────────────────────────────────────────────┐
│  🤖 AI AUTOMATION PIPELINE                                  │
│  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │
│                                                             │
│                                                             │
│  ⏰ SCHEDULE                                                │
│  Next Run: Tomorrow at 2:00 AM (in 16 hours)              │
│  Frequency: [Daily ▼] at [02:00 ▼]                        │
│  Status: ● Scheduled                                       │
│                                                             │
│  [Run Now] [Pause Schedule] [Configure]                    │
│                                                             │
├─────────────────────────────────────────────────────────────┤
│  📊 PIPELINE OVERVIEW                                       │
│                                                             │
│  Keywords ──→ Clusters ──→ Ideas ──→ Tasks ──→ Content     │
│    47         8           42       20       generating     │
│  pending      new         ready    queued   Stage 5        │
│                                                             │
│  Overall Progress: ━━━━━━━╸━━━━━━━━━ 62% (Stage 5/7)     │
│  Estimated Completion: 2 hours 15 minutes                  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

[STAGE 1 CARD - completed state]
[STAGE 2 CARD - completed state]
[STAGE 3 CARD - completed state]
[STAGE 4 CARD - running state with queue details]
[STAGE 5 CARD - waiting state]
[STAGE 6 CARD - waiting state]
[STAGE 7 CARD - gate state]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📋 LIVE ACTIVITY LOG (Last 50 events)
├─ 14:23:45 - Stage 4: Started content generation for Task 3
├─ 14:24:12 - Stage 4: Writing sections (65% complete)
├─ 14:22:30 - Stage 4: Completed Task 2 → Content created
├─ 14:20:15 - Stage 4: Started content generation for Task 2
└─ [View Full Log]

💰 TOTAL CREDITS USED THIS RUN: 66 credits
```

**Components:**

**StageCard.tsx** - Individual stage display component
- Props: stageNumber, stageName, status, queueData, result
- Shows: Status badge, queue overview, progress bar, stats
- Actions: "View Details", "Pause", "Retry Failed"

**ActivityLog.tsx** - Live activity feed component
- Props: runId
- Fetches: `/api/v1/automation/activity_log/{runId}` every 3 seconds
- Shows: Timestamped log entries, color-coded by type (info/success/error)

**ConfigModal.tsx** - Schedule configuration modal
- Fields: Frequency dropdown, Time picker, Batch sizes (advanced)
- Saves to: AutomationConfig model via `/api/v1/automation/config/`

**Sidebar Menu Addition:**
```
Sites
  ├─ Site Management
  └─ Site Settings

Automation  ← NEW
  └─ Pipeline Dashboard

Planner
  ├─ Keywords
  ├─ Clusters
  └─ Ideas
```

---

### Real-Time Progress Updates

**UI Update Strategy:**
- **Frontend Polling**: Poll automation status API every 3 seconds when run is active
- **Backend Progress**: Uses event-based `StepTracker` to capture AI function phases
- When automation is `status='running'` → Poll every 3 seconds
- When `status='completed'` or `status='failed'` → Stop polling
- When `status='paused'` → Poll every 10 seconds

**How Progress Works:**
1. **AI Function Execution**: Each AI function emits phase events (INIT, PREP, AI_CALL, PARSE, SAVE, DONE)
2. **StepTracker Captures**: Progress tracker records these events with metadata
3. **Automation Logs**: Orchestrator reads from StepTracker and logs to file
4. **UI Polls**: Frontend polls automation status API to read aggregated progress
5. **Display**: UI shows current phase and completion percentage per stage

**API Endpoint:**
```
GET /api/v1/automation/current_run/?site_id=12

Response:
{
  "run": {
    "run_id": "run_20251203_140523_manual",
    "status": "running",
    "current_stage": 4,
    "started_at": "2025-12-03T14:05:23Z",
    "total_credits_used": 66,
    "stage_1_result": {"keywords_processed": 47, "clusters_created": 8},
    "stage_2_result": {"clusters_processed": 8, "ideas_created": 56},
    "stage_3_result": {"ideas_processed": 56, "tasks_created": 56},
    "stage_4_result": {"tasks_processed": 23, "tasks_total": 56},
    ...
  },
  "activity_log": [
    "14:23:45 - Stage 4: Started content generation for Task 3",
    "14:24:12 - Stage 4: Writing sections (65% complete)",
    ...
  ],
  "queues": {
    "stage_1": {"total": 0, "pending": 0},
    "stage_2": {"total": 0, "pending": 0},
    "stage_3": {"total": 0, "pending": 0},
    "stage_4": {"total": 56, "pending": 33},
    "stage_5": {"total": 23, "pending": 23},
    "stage_6": {"total": 0, "pending": 0},
    "stage_7": {"total": 0, "pending": 0}
  }
}
```

---

## 🔄 BACKEND IMPLEMENTATION FLOW

### Service Layer Architecture

**AutomationService** (core orchestrator)
- Location: `backend/igny8_core/business/automation/services/automation_service.py`
- Responsibility: Execute stages sequentially, manage run state
- Reuses: All existing AI function classes (NO duplication)

**AutomationLogger** (file logging)
- Location: `backend/igny8_core/business/automation/services/automation_logger.py`
- Responsibility: Write timestamped logs to files
- Methods: start_run(), log_stage_start(), log_stage_progress(), log_stage_complete()

**Key Service Methods:**

```
AutomationService:
  - __init__(account, site) → Initialize with site context (NO sector)
  - start_automation(trigger_type) → Main entry point
  - run_stage_1() → Keywords → Clusters
  - run_stage_2() → Clusters → Ideas
  - run_stage_3() → Ideas → Tasks
  - run_stage_4() → Tasks → Content
  - run_stage_5() → Content → Prompts
  - run_stage_6() → Prompts → Images
  - run_stage_7() → Review gate
  - pause_automation() → Pause current run
  - resume_automation() → Resume from current_stage
  - estimate_credits() → Calculate estimated credits needed

AutomationLogger:
  - start_run(account_id, site_id, trigger_type) → Create log directory, return run_id
  - log_stage_start(run_id, stage_number, stage_name, pending_count)
  - log_stage_progress(run_id, stage_number, message)
  - log_stage_complete(run_id, stage_number, processed_count, time_elapsed, credits_used)
  - log_stage_error(run_id, stage_number, error_message)
  - get_activity_log(run_id, last_n=50) → Return last N log lines
```

---

### API Endpoints to Implement

**AutomationViewSet** - Django REST Framework ViewSet
- Base URL: `/api/v1/automation/`
- Actions:

```
POST /api/v1/automation/run_now/
  - Body: {"site_id": 12}
  - Action: Trigger manual automation run
  - Returns: {"run_id": "run_...", "message": "Automation started"}

GET /api/v1/automation/current_run/?site_id=12
  - Returns: Current/latest run status, activity log, queue counts

POST /api/v1/automation/pause/
  - Body: {"run_id": "run_..."}
  - Action: Pause running automation

POST /api/v1/automation/resume/
  - Body: {"run_id": "run_..."}
  - Action: Resume paused automation

GET /api/v1/automation/config/?site_id=12
  - Returns: AutomationConfig for site

PUT /api/v1/automation/config/
  - Body: {"site_id": 12, "is_enabled": true, "frequency": "daily", "scheduled_time": "02:00"}
  - Action: Update automation schedule

GET /api/v1/automation/history/?site_id=12&page=1
  - Returns: Paginated list of past runs

GET /api/v1/automation/logs/{run_id}/
  - Returns: Full logs for a specific run (all stage files)
```

---

### Celery Tasks for Scheduling

**Periodic Task** (runs every hour)
```
@shared_task(name='check_scheduled_automations')
def check_scheduled_automations():
    """
    Runs every hour (via Celery Beat)
    Checks if any AutomationConfig needs to run
    """
    now = timezone.now()
    
    configs = AutomationConfig.objects.filter(
        is_enabled=True,
        next_run_at__lte=now
    )
    
    for config in configs:
        # Check for concurrent run
        if AutomationRun.objects.filter(site=config.site, status='running').exists():
            continue  # Skip if already running
        
        # Start automation
        run_automation_task.delay(
            account_id=config.account_id,
            site_id=config.site_id,
            trigger_type='scheduled'
        )
        
        # Calculate next run time
        if config.frequency == 'daily':
            config.next_run_at = now + timedelta(days=1)
        elif config.frequency == 'weekly':
            config.next_run_at = now + timedelta(weeks=1)
        elif config.frequency == 'monthly':
            config.next_run_at = now + timedelta(days=30)
        
        config.last_run_at = now
        config.save()

Schedule in celery.py:
  app.conf.beat_schedule['check-scheduled-automations'] = {
      'task': 'check_scheduled_automations',
      'schedule': crontab(minute=0),  # Every hour on the hour
  }
```

**Stage Task Chain**
```
@shared_task
def run_automation_task(account_id, site_id, trigger_type):
    """
    Main automation task - chains individual stage tasks
    """
    service = AutomationService(account_id, site_id)
    run_id = service.start_automation(trigger_type)
    
    # Chain stages as separate tasks for non-blocking execution
    chain(
        run_stage_1.si(run_id),
        run_stage_2.si(run_id),
        run_stage_3.si(run_id),
        run_stage_4.si(run_id),
        run_stage_5.si(run_id),
        run_stage_6.si(run_id),
        run_stage_7.si(run_id),
    ).apply_async()

@shared_task
def run_stage_1(run_id):
    service = AutomationService.from_run_id(run_id)
    service.run_stage_1()
    return run_id  # Pass to next task

@shared_task
def run_stage_2(run_id):
    service = AutomationService.from_run_id(run_id)
    service.run_stage_2()
    return run_id

[... similar for stages 3-7 ...]
```

---

## 🧪 TESTING STRATEGY

### Unit Tests

**Test AutomationService:**
- test_estimate_credits_calculation()
- test_stage_1_processes_batches_correctly()
- test_stage_completion_triggers_next_stage()
- test_pause_stops_after_current_batch()
- test_resume_from_paused_state()
- test_idempotency_skips_completed_stages()

**Test AutomationLogger:**
- test_creates_log_directory_structure()
- test_writes_timestamped_log_entries()
- test_get_activity_log_returns_last_n_lines()

### Integration Tests

**Test Full Pipeline:**
```
def test_full_automation_pipeline():
    # Setup: Create 10 keywords
    keywords = KeywordFactory.create_batch(10, site=site)
    
    # Execute
    service = AutomationService(account, site)
    result = service.start_automation(trigger_type='manual')
    
    # Assert Stage 1
    assert result['stage_1_result']['keywords_processed'] == 10
    assert result['stage_1_result']['clusters_created'] > 0
    
    # Assert Stage 2
    assert result['stage_2_result']['ideas_created'] > 0
    
    # Assert Stage 3
    assert result['stage_3_result']['tasks_created'] > 0
    
    # Assert Stage 4
    assert result['stage_4_result']['content_created'] > 0
    
    # Assert Stage 5
    assert result['stage_5_result']['prompts_created'] > 0
    
    # Assert Stage 6
    assert result['stage_6_result']['images_generated'] > 0
    
    # Assert final state
    assert result['status'] == 'completed'
    assert AutomationRun.objects.get(run_id=result['run_id']).status == 'completed'
```

**Test Error Scenarios:**
- test_insufficient_credits_prevents_start()
- test_concurrent_run_prevented()
- test_stage_failure_stops_pipeline()
- test_rollback_on_error()

---

## 📋 IMPLEMENTATION CHECKLIST

### Phase 1: Database & Models (Week 1)
- [ ] Create `automation` app directory structure
- [ ] Define AutomationRun model with all stage_result JSON fields
- [ ] Define AutomationConfig model (one per site, NO sector)
- [ ] Create migrations
- [ ] Test model creation and queries

### Phase 2: Logging Service (Week 1)
- [ ] Create AutomationLogger class
- [ ] Implement start_run() with log directory creation
- [ ] Implement log_stage_start(), log_stage_progress(), log_stage_complete()
- [ ] Implement get_activity_log()
- [ ] Test file logging manually

### Phase 3: Core Automation Service (Week 2)
- [ ] Create AutomationService class
- [ ] Implement estimate_credits()
- [ ] Implement start_automation() with credit check
- [ ] Implement run_stage_1() calling ClusteringService
- [ ] Test Stage 1 in isolation with real keywords
- [ ] Implement run_stage_2() calling IdeasService
- [ ] Test Stage 2 in isolation
- [ ] Implement run_stage_3() calling bulk_queue_to_writer logic
- [ ] Implement run_stage_4() calling GenerateContentFunction
- [ ] Implement run_stage_5() calling GenerateImagePromptsFunction
- [ ] Implement run_stage_6() calling GenerateImagesFunction
- [ ] Implement run_stage_7() review gate (count only)
- [ ] Implement pause_automation() and resume_automation()

### Phase 4: API Endpoints (Week 3)
- [ ] Create AutomationViewSet
- [ ] Implement run_now() action
- [ ] Implement current_run() action
- [ ] Implement pause() and resume() actions
- [ ] Implement config GET/PUT actions
- [ ] Implement history() action
- [ ] Implement logs() action
- [ ] Add URL routing in api_urls.py
- [ ] Test all endpoints with Postman/curl

### Phase 5: Celery Tasks & Scheduling (Week 3)
- [ ] Create check_scheduled_automations periodic task
- [ ] Create run_automation_task
- [ ] Create stage task chain (run_stage_1, run_stage_2, etc.)
- [ ] Register tasks in celery.py
- [ ] Add Celery Beat schedule
- [ ] Test scheduled execution

### Phase 6: Frontend Components (Week 4)
- [ ] Create /automation route in React Router
- [ ] Create Dashboard.tsx page component
- [ ] Create StageCard.tsx with queue display
- [ ] Create ActivityLog.tsx with 3-second polling
- [ ] Create ConfigModal.tsx for schedule settings
- [ ] Add "Automation" to sidebar menu (below Sites)
- [ ] Implement "Run Now" button
- [ ] Implement "Pause" and "Resume" buttons
- [ ] Test full UI flow

### Phase 7: Safety & Polish (Week 5)
- [ ] Implement distributed locking (prevent concurrent runs)
- [ ] Implement credit reservation system
- [ ] Implement stage idempotency checks
- [ ] Implement error handling and rollback
- [ ] Create cleanup_old_automation_logs task
- [ ] Add email/notification on completion/failure
- [ ] Load testing with 100+ keywords
- [ ] UI polish and responsiveness
- [ ] Documentation update

---

## 🚀 POST-LAUNCH ENHANCEMENTS

### Future Features (Phase 8+)
- **Conditional Stages:** Skip stages if no data (e.g., skip Stage 1 if no keywords)
- **Parallel Task Processing:** Process multiple tasks simultaneously in Stage 4 (with worker limits)
- **Smart Scheduling:** Avoid peak hours, optimize for cost
- **A/B Testing:** Test different prompts, compare results
- **Content Quality Scoring:** Auto-reject low-quality AI content
- **WordPress Auto-Publish:** With approval workflow and staging
- **Analytics Integration:** Track content performance post-publish
- **Social Media Auto-Post:** Share published content to social channels

---

## 📖 USER DOCUMENTATION

### How to Use Automation

**1. Configure Schedule:**
- Navigate to Automation page
- Click "Configure" button
- Set frequency (Daily/Weekly/Monthly)
- Set time (e.g., 2:00 AM)
- Optionally adjust batch sizes (advanced)
- Click "Save"

**2. Manual Run:**
- Click "Run Now" button
- Monitor progress in real-time
- View activity log for details

**3. Review Content:**
- Wait for automation to complete (or check next morning if scheduled)
- Navigate to Writer → Content page
- Filter by "Draft" status with images generated
- Review content quality
- Select multiple → Bulk Publish

**4. Monitor History:**
- View past runs in History tab
- Click run to view detailed logs
- See credits used per run

---

## ✅ SUCCESS CRITERIA

**Automation is successful if:**
- ✅ Runs without manual intervention from Keywords → Draft Content
- ✅ Processes 100+ keywords without errors
- ✅ Respects credit limits (pre-check + reservation)
- ✅ Stops at review gate (doesn't auto-publish)
- ✅ Completes within estimated time (6-12 hours for 100 keywords)
- ✅ UI shows real-time progress accurately
- ✅ Logs are detailed and troubleshoot-able
- ✅ Can pause/resume without data loss
- ✅ Scheduled runs trigger correctly
- ✅ No duplicate runs occur
- ✅ Reuses ALL existing AI functions (zero duplication)

---

**END OF COMPLETE IMPLEMENTATION PLAN**

This plan ensures a safe, modular, observable, and maintainable automation system that orchestrates the existing IGNY8 AI functions into a fully automated content pipeline.