# IGNY8 Automation System - Detailed Task List for AI Agent

## CRITICAL ANALYSIS

Based on the documentation and current implementation status, I've identified significant issues with the automation system and legacy SiteBuilder references that need systematic resolution.

---

## SECTION 1: AUTO CLUSTER AI FUNCTION ISSUES

### Task 1.1: Fix Cluster Status Assignment in AutoClusterFunction
**Problem:** Clusters created by automation are assigned `status='active'` instead of proper status progression.

**Current Behavior (from code analysis):**
- Line 95 in `auto_cluster.py`: `Clusters.objects.get_or_create()` uses `defaults={'status': 'active'}`
- This bypasses the intended status flow: `new` → `mapped`

**Required Changes:**
1. Modify `save_output()` method in `AutoClusterFunction` (backend/igny8_core/ai/functions/auto_cluster.py)
2. Change cluster creation to use `status='new'` initially
3. After keyword mapping completes, update cluster status to `status='mapped'`
4. Ensure status progression aligns with automation pipeline expectations

**Verification Steps:**
- Create test with 10 keywords
- Run Stage 1 via automation
- Verify clusters have `status='new'` → then `status='mapped'`
- Confirm automation Stage 2 correctly queries for these clusters

---

### Task 1.2: Investigate Keywords Not Mapping to Clusters
**Problem:** Keywords processed through automation are not getting their `cluster_id` assigned correctly.

**Diagnostic Steps Required:**
1. Add detailed logging in `save_output()` method around line 100-110
2. Log each keyword's ID, text, and whether cluster match was found
3. Check if case-insensitive matching is working: `keyword_obj.keyword.lower() in [k.lower() for k in cluster_keywords]`
4. Verify `Keywords.seed_keyword.keyword` field is populated correctly
5. Check if AI response format matches expected structure: `{clusters: [{name, keywords: [], description}]}`

**Root Cause Analysis:**
- Examine parsed AI response structure
- Verify keyword text extraction from `Keywords` model via `seed_keyword` relationship
- Confirm database transaction commits are occurring
- Check for any filtering conditions excluding keywords

**Fix Implementation:**
1. If mismatch in keyword text format, normalize both sides before comparison
2. If seed_keyword relationship issue, fetch keyword text correctly
3. Add fallback matching using keyword ID if text matching fails
4. Ensure `keyword_obj.save()` is called within transaction

---

### Task 1.3: Distinguish Between Manual vs Automation Execution
**Problem:** AutoClusterFunction behaves the same whether called manually or via automation.

**Required Implementation:**
1. Add `execution_context` parameter to function payload: `{'ids': [...], 'execution_context': 'automation' | 'manual'}`
2. Pass context from `AutomationService.run_stage_1()` when calling AI function
3. In `save_output()`, adjust status assignments based on context:
   - Manual: Keep current behavior (`status='active'`)
   - Automation: Use progressive status (`'new'` → `'mapped'`)
4. Document context parameter in function metadata

---

## SECTION 2: LEGACY SITEBUILDER/BLUEPRINT REMOVAL

### Task 2.1: Database Models Cleanup
**Files to Remove Completely:**
1. `backend/igny8_core/business/site_building/models.py` - Already stubbed, remove entirely
2. Migration already exists: `0002_remove_blueprint_models.py` - Verify it ran successfully

**Database Verification:**
1. Connect to production database
2. Run SQL: `SELECT tablename FROM pg_tables WHERE tablename LIKE '%blueprint%' OR tablename LIKE '%site_building%';`
3. Expected result: No tables (already dropped)
4. If tables exist, manually run DROP TABLE commands from migration

**Foreign Key Cleanup:**
1. Check `igny8_deployment_records` table - verify `site_blueprint_id` column removed
2. Check `igny8_publishing_records` table - verify `site_blueprint_id` column removed
3. Confirm indexes dropped: `igny8_publishing_recor_site_blueprint_id_des_b7c4e5f8_idx`

---

### Task 2.2: Backend Code References Removal

**Phase 2.2.1: Remove Stub Models**
- **File:** `backend/igny8_core/business/site_building/models.py`
- **Action:** Delete entire file
- **Reason:** Contains only stub classes (`SiteBlueprint`, `PageBlueprint`, `SiteBlueprintCluster`, `SiteBlueprintTaxonomy`) with no functionality

**Phase 2.2.2: Remove Entire site_building App**
- **Directory:** `backend/igny8_core/business/site_building/`
- **Action:** Delete entire directory
- **Reason:** All functionality deprecated, no active code

**Files to Delete:**
1. `services/structure_generation_service.py` - Calls deprecated AI function
2. `services/page_generation_service.py` - References PageBlueprint
3. `services/taxonomy_service.py` - Uses SiteBlueprintTaxonomy
4. `services/file_management_service.py` - SiteBuilder file management
5. `tests/` - All test files reference removed models
6. `admin.py` - Already commented out
7. `migrations/` - Keep for database history, but app removal makes them inert

**Phase 2.2.3: Remove site_builder Module**
- **Directory:** `backend/igny8_core/modules/site_builder.backup/`
- **Action:** Delete entire directory (already marked `.backup`)
- **Contains:** Deprecated API endpoints, serializers, views for blueprint management

---

### Task 2.3: Settings Configuration Cleanup

**File:** `backend/igny8_core/settings.py`

**Changes:**
1. Line 56: Already commented out - Remove comment entirely
2. Line 61: Already commented out - Remove comment entirely
3. Verify `INSTALLED_APPS` list is clean

**Verification:**
- Run `python manage.py check` - Should pass
- Run `python manage.py migrate --plan` - Should show no pending site_building migrations

---

### Task 2.4: URL Routing Cleanup

**File:** `backend/igny8_core/urls.py`

**Changes:**
1. Line 42: Comment already exists - Remove comment entirely
2. Verify no routing to `site-builder/` endpoints exists

**Verification:**
- Run Django server
- Attempt to access `/api/v1/site-builder/blueprints/` - Should return 404
- Check API root `/api/v1/` - Should not list site-builder endpoints

---

### Task 2.5: AI Function Removal

**File:** `backend/igny8_core/ai/functions/generate_page_content.py`

**Problem:** This AI function depends on `PageBlueprint` model which no longer exists.

**Action Required:**
1. **DELETE FILE:** `generate_page_content.py` (21 references to PageBlueprint)
2. **UPDATE:** `backend/igny8_core/ai/registry.py` - Remove lazy loader registration
3. **UPDATE:** `backend/igny8_core/ai/engine.py` - Remove from operation type mappings (line 599)

**Verification:**
- Search codebase for `generate_page_content` function calls
- Ensure no active code relies on this function
- Confirm AI function registry no longer lists it

---

### Task 2.6: Backend Import Statement Cleanup

**Files with Import Statements to Update:**

1. **backend/igny8_core/business/integration/services/content_sync_service.py**
   - Lines 378, 488: `from igny8_core.business.site_building.models import SiteBlueprint`
   - **Action:** Remove import, remove dependent code blocks (lines 382-388, 491-496)
   - **Alternative:** Service should use `ContentTaxonomy` directly (post-migration model)

2. **backend/igny8_core/business/integration/services/sync_health_service.py**
   - Line 335: `from igny8_core.business.site_building.models import SiteBlueprint, SiteBlueprintTaxonomy`
   - **Action:** Remove import, refactor taxonomy checks to use `ContentTaxonomy`

3. **backend/igny8_core/business/publishing/services/adapters/sites_renderer_adapter.py**
   - Line 15: `from igny8_core.business.site_building.models import SiteBlueprint`
   - **Action:** Entire adapter is deprecated - DELETE FILE
   - **Reason:** Designed to deploy SiteBlueprint instances, no longer applicable

4. **backend/igny8_core/business/publishing/services/deployment_readiness_service.py**
   - Line 10: `from igny8_core.business.site_building.models import SiteBlueprint`
   - **Action:** DELETE FILE or refactor to remove blueprint checks
   - **Reason:** Service checks blueprint readiness for deployment

5. **backend/igny8_core/business/publishing/services/deployment_service.py**
   - Line 10: `from igny8_core.business.site_building.models import SiteBlueprint`
   - **Action:** Remove blueprint-specific deployment methods

---

### Task 2.7: Frontend Files Removal

**Phase 2.7.1: Remove Type Definitions**
- **File:** `frontend/src/types/siteBuilder.ts`
- **Action:** Delete file entirely
- **References:** Used in store and components

**Phase 2.7.2: Remove API Service**
- **File:** `frontend/src/services/siteBuilder.api.ts`
- **Action:** Delete file
- **Contains:** API methods for blueprint CRUD operations

**Phase 2.7.3: Remove Pages**
- **Directory:** `frontend/src/pages/Sites/`
- **Files to Review:**
  - `Editor.tsx` - Uses PageBlueprint, SiteBlueprint types (lines 15-36)
  - `PageManager.tsx` - Fetches blueprints (lines 126-137)
  - `DeploymentPanel.tsx` - Blueprint deployment UI (46 references)

**Action for Pages:**
1. If pages ONLY deal with blueprints - DELETE
2. If pages have mixed functionality - REFACTOR to remove blueprint code
3. Likely DELETE: `Editor.tsx`, `DeploymentPanel.tsx`
4. Likely REFACTOR: `Dashboard.tsx` (remove blueprint widget)

**Phase 2.7.4: Remove Store**
- **File:** `frontend/src/store/siteDefinitionStore.ts`
- **Action:** Review dependencies, likely DELETE
- **Alternative:** If used for non-blueprint purposes, refactor to remove PageBlueprint types

**Phase 2.7.5: Remove Components**
- **File:** `frontend/src/components/sites/SiteProgressWidget.tsx`
- **Action:** DELETE if blueprint-specific
- **Uses:** `blueprintId` prop, calls `fetchSiteProgress(blueprintId)`

---

### Task 2.8: Frontend Import and Reference Cleanup

**Files Requiring Updates:**

1. **frontend/src/services/api.ts**
   - Lines 2302-2532: Multiple blueprint-related functions
   - **Action:** Remove these function exports:
     - `fetchDeploymentReadiness`
     - `createSiteBlueprint`, `updateSiteBlueprint`
     - `attachClustersToBlueprint`, `detachClustersFromBlueprint`
     - `fetchBlueprintsTaxonomies`, `createBlueprintTaxonomy`
     - `importBlueprintsTaxonomies`
     - `updatePageBlueprint`, `regeneratePageBlueprint`

2. **frontend/src/pages/Planner/Dashboard.tsx**
   - Lines 30-31: Commented imports
   - **Action:** Remove commented lines entirely

3. **frontend/src/config/pages/tasks.config.tsx**
   - Lines 110-111: Logic for `[Site Builder]` task title formatting
   - **Action:** Remove special handling, update title display logic

---

### Task 2.9: Sites Renderer Cleanup

**File:** `sites/src/loaders/loadSiteDefinition.ts`

**Current Behavior (Lines 103-159):**
- API load fails → Falls back to blueprint endpoint
- Transforms blueprint to site definition format

**Required Changes:**
1. Remove fallback to blueprint endpoint (lines 116-127)
2. Remove `transformBlueprintToSiteDefinition` function (lines 137-159)
3. If API fails, return proper error instead of fallback
4. Update error messages to remove blueprint references

---

### Task 2.10: Documentation Cleanup

**Files to Remove:**
1. `docs/igny8-pp/TAXONOMY/QUICK-REFERENCE-TAXONOMY.md` - References SiteBuilder removal
2. Update `docs/tech-stack/00-SYSTEM-ARCHITECTURE-MASTER-REFERENCE.md`:
   - Remove "Site Blueprints" from feature list (line 45)
   - Remove `site_builder/` from architecture diagrams (lines 179-180)
   - Remove SiteBuilder from system overview (line 1517)

**Files to Update:**
1. `docs/igny8-pp/01-IGNY8-REST-API-COMPLETE-REFERENCE.md`:
   - Remove entire section: "Site Blueprints" (lines 1201-1230)
   - Remove "Page Blueprints" section (lines 1230-1247)
   - Update deployment endpoints to remove blueprint references

2. `docs/igny8-pp/02-PLANNER-WRITER-WORKFLOW-TECHNICAL-GUIDE.md`:
   - Remove SiteBlueprintTaxonomy references (lines 114, 151)

---

### Task 2.11: Test Files Cleanup

**Backend Tests:**
1. DELETE: `backend/igny8_core/ai/tests/test_generate_site_structure_function.py`
2. DELETE: `backend/igny8_core/business/site_building/tests/` (entire directory)
3. DELETE: `backend/igny8_core/business/publishing/tests/test_deployment_service.py`
4. DELETE: `backend/igny8_core/business/publishing/tests/test_publisher_service.py`
5. DELETE: `backend/igny8_core/business/publishing/tests/test_adapters.py`

**Frontend Tests:**
1. DELETE: `frontend/src/__tests__/sites/BulkGeneration.test.tsx`
2. UPDATE: `frontend/src/__tests__/sites/PromptManagement.test.tsx`:
   - Remove site_structure_generation prompt type checks (lines 27-28)
3. UPDATE: `frontend/src/__tests__/sites/SiteManagement.test.tsx`:
   - Remove `[Site Builder]` task filtering logic (lines 50-51)

---

### Task 2.12: Database Migration Verification

**Critical Checks:**
1. Verify `0002_remove_blueprint_models.py` migration applied in all environments
2. Check for orphaned data:
   - Query for any `Tasks` with `taxonomy_id` pointing to deleted SiteBlueprintTaxonomy
   - Query for any `ContentIdeas` with old taxonomy foreign keys
3. If orphaned data found, create data migration to:
   - Set taxonomy foreign keys to NULL
   - Or migrate to ContentTaxonomy if mapping exists

**SQL Verification Queries:**
```sql
-- Check for blueprint tables (should return empty)
SELECT tablename FROM pg_tables 
WHERE tablename LIKE '%blueprint%' OR tablename LIKE '%site_building%';

-- Check for foreign key constraints (should return empty)
SELECT constraint_name FROM information_schema.table_constraints 
WHERE constraint_name LIKE '%blueprint%';

-- Check for orphaned taxonomy references
SELECT COUNT(*) FROM igny8_tasks WHERE taxonomy_id IS NOT NULL;
SELECT COUNT(*) FROM igny8_content_ideas WHERE taxonomy_id IS NOT NULL;
```

---

## SECTION 3: AUTOMATION PAGE UI IMPROVEMENTS

### Task 3.1: Stage Card Visual Redesign

**Current Issues:**
- Icons too large, taking excessive space
- Stage names not clearly associated with stage numbers
- Inconsistent visual hierarchy

**Required Changes:**
1. **Reduce Icon Size:**
   - Current: Large colored square icons
   - New: Smaller icons (32x32px instead of current size)
   - Position: Top-left of card, not centered

2. **Restructure Stage Header:**
   - Move stage name directly below "Stage N" text
   - Format: "Stage 1" (bold) / "Keywords → Clusters" (regular weight, smaller font)
   - Remove redundant text repetition

3. **Status Badge Positioning:**
   - Move from separate row to same line as stage number
   - Right-align badge next to stage number

**Layout Example (No Code):**
```
┌─────────────────────────────┐
│ [Icon] Stage 1     [Ready]  │
│        Keywords → Clusters   │
│                              │
│ Total Queue:            7    │
│ Processed:              0    │
│ Remaining:              7    │
└─────────────────────────────┘
```

---

### Task 3.2: Add Progress Bars to Stage Cards

**Implementation Requirements:**
1. **Individual Stage Progress Bar:**
   - Display below queue metrics
   - Calculate: `(Processed / Total Queue) * 100`
   - Visual: Colored bar matching stage color
   - Show percentage label

2. **Overall Pipeline Progress Bar:**
   - Large bar above all stage cards
   - Calculate: `(Sum of Processed Items Across All Stages) / (Sum of Total Queue Across All Stages) * 100`
   - Display current stage indicator: "Stage 4/7"
   - Show estimated completion time

3. **Progress Bar States:**
   - Empty (0%): Gray/outline only
   - In Progress (1-99%): Animated gradient
   - Complete (100%): Solid color, checkmark icon

---

### Task 3.3: Add Total Metrics Cards Above Pipeline

**New Component: MetricsSummary Cards**

**Cards to Display (Row above pipeline overview):**
1. **Keywords Card:**
   - Total: Count from database
   - Processed: Keywords with `status='mapped'`
   - Pending: Keywords with `status='new'`

2. **Clusters Card:**
   - Total: All clusters for site
   - Processed: Clusters with ideas generated
   - Pending: Clusters without ideas

3. **Ideas Card:**
   - Total: All ideas for site
   - Processed: Ideas converted to tasks (`status='in_progress'`)
   - Pending: Ideas with `status='new'`

4. **Content Card:**
   - Total: All content for site
   - Processed: Content with `status='draft'` + all images generated
   - Pending: Content without images or in generation

5. **Images Card:**
   - Total: All image records for site content
   - Processed: Images with `status='generated'`
   - Pending: Images with `status='pending'`

**Card Layout:**
- Width: Equal distribution across row
- Display: Icon, Title, Total/Processed/Pending counts
- Color: Match stage colors for visual consistency

---

### Task 3.4: Pipeline Status Card Redesign

**Current:** Wide row with text "Pipeline Status - Ready to run | 22 items pending"

**New Design Requirements:**
1. **Convert to Centered Card:**
   - Position: Above stage cards, below metrics summary
   - Width: Narrower than full width, centered
   - Style: Elevated/shadowed for emphasis

2. **Content Structure:**
   - Large status indicator (icon + text)
   - Prominent pending items count
   - Quick action buttons (Run Now, Pause, Configure)

3. **Status Visual States:**
   - Ready: Green pulse animation
   - Running: Blue animated progress
   - Paused: Yellow warning icon
   - Failed: Red alert icon

---

### Task 3.5: Remove/Compact Header Elements

**Changes to Automation Page Header:**

1. **Remove "Pipeline Overview" Section:**
   - Delete heading: "📊 Pipeline Overview"
   - Delete subtitle: "Complete view of automation pipeline status and pending items"
   - Reason: Redundant with visible pipeline cards

2. **Compact Schedule Panel:**
   - Current: Large panel with heading, status row, action buttons
   - New: Single compact row
   - Layout: `[Status Badge] | [Schedule Text] | [Last Run] | [Estimated Credits] | [Configure Button] | [Run Now Button]`
   - Remove empty space and excessive padding

---

### Task 3.6: AI Request Delays Implementation

**Problem:** Rapid sequential AI requests may hit rate limits or overload AI service.

**Required Changes:**

1. **Within-Stage Delay (between batches):**
   - Location: `AutomationService` class methods
   - Add delay after each batch completion before processing next batch
   - Configurable: 3-5 seconds (default 3 seconds)
   - Implementation point: After each AI function call completes in stage loop

2. **Between-Stage Delay:**
   - Add delay after stage completion before triggering next stage
   - Configurable: 5-10 seconds (default 5 seconds)
   - Implementation point: After `_execute_stage()` returns before incrementing `current_stage`

3. **Configuration:**
   - Add to `AutomationConfig` model: `within_stage_delay` (integer, seconds)
   - Add to `AutomationConfig` model: `between_stage_delay` (integer, seconds)
   - Expose in Configure modal for user adjustment

4. **Logging:**
   - Log delay start: "Waiting 3 seconds before next batch..."
   - Log delay end: "Delay complete, resuming processing"

---

## SECTION 4: AUTOMATION STAGE PROCESSING FIXES

### Task 4.1: Verify Stage Sequential Processing Logic

**Problem:** Pipeline not following strict sequential stage completion before moving to next stage.

**Analysis Required:**
1. Review `AutomationService.start_automation()` method
2. Verify stage loop waits for 100% completion before `current_stage += 1`
3. Check for any parallel execution logic that bypasses sequential flow

**Verification Steps:**
1. Each stage method (`run_stage_1()` to `run_stage_7()`) must return ONLY after ALL batches processed
2. Stage N+1 should NOT start if Stage N has `pending > 0`
3. Add explicit completion check before stage transition

**Required Fixes:**
- Add validation: Before starting Stage N, verify Stage N-1 has 0 pending items
- If pending items found, log warning and halt automation
- Return error status with message: "Stage N-1 incomplete, cannot proceed to Stage N"

---

### Task 4.2: Fix Batch Size Configuration Reading

**Problem:** Manual "Run Now" only processes 5 keywords instead of respecting configured batch size (20).

**Root Cause Analysis:**
1. Check if `run_stage_1()` reads from `AutomationConfig.stage_1_batch_size`
2. Verify query limit: `Keywords.objects.filter(...)[:batch_size]` uses correct variable
3. Confirm configuration loaded at automation start: `config = AutomationConfig.objects.get(site=self.site)`

**Expected Behavior:**
- If queue has 7 keywords and batch_size = 20: Process all 7 (not limit to 5)
- If queue has 47 keywords and batch_size = 20: Process 20, then next batch of 20, then final 7
- Batch size should be dynamic based on queue size: `min(queue_count, batch_size)`

**Fix Implementation:**
1. Ensure configuration loaded once at automation start
2. Pass batch_size to each stage method
3. Update query to use: `[:min(pending_count, batch_size)]`
4. Log batch selection: "Processing batch 1/3: 20 keywords"

---

### Task 4.3: Fix Stage 4 Processing Not Completing Full Queue

**Problem:** Stage 4 (Tasks → Content) not processing all tasks before moving to Stage 5.

**Investigation Steps:**
1. Check `run_stage_4()` implementation in `AutomationService`
2. Verify loop structure: Does it process tasks one-by-one until queue empty?
3. Look for premature loop exit conditions

**Expected Logic:**
```
While tasks with status='pending' exist:
    1. Get next task
    2. Call generate_content AI function
    3. Wait for completion
    4. Verify Content created
    5. Check if more pending tasks exist
    6. If yes, continue loop
    7. If no, return stage complete
```

**Common Issues to Check:**
- Loop exits after first task instead of continuing
- No loop at all - only processes one batch
- Error handling breaks loop prematurely

---

### Task 4.4: Fix Stage 5 Not Triggering (Image Prompts Generation)

**Problem:** Automation exits after Stage 4 without generating image prompts.

**Analysis Required:**
1. Verify Stage 4 completion status set correctly
2. Check if Stage 5 start condition is met
3. Review database query in `run_stage_5()`:
   - Query: Content with `status='draft'` AND `images_count=0`
   - Verify Content records created in Stage 4 have correct status

**Potential Issues:**
1. Content created with status other than 'draft'
2. Images count annotation incorrect (should use `annotate(images_count=Count('images'))`)
3. Stage handover logic doesn't trigger Stage 5

**Fix Steps:**
1. Verify Content model save in Stage 4 sets `status='draft'`
2. Ensure Stage 5 query matches Content records from Stage 4
3. Add logging: "Stage 5: Found X content pieces without images"
4. If X > 0, process; if X = 0, skip stage gracefully

---

### Task 4.5: Add Stage Handover Validation

**New Logic Required Between Each Stage:**

1. **Pre-Stage Validation:**
   - Before starting Stage N, check Stage N-1 completion:
     - Query pending items for Stage N-1
     - If pending > 0: Log error, halt automation
     - If pending = 0: Log success, proceed

2. **Post-Stage Validation:**
   - After Stage N completes, verify:
     - All input items processed
     - Expected output items created
     - No errors in stage result
   - Log validation result before moving to Stage N+1

3. **Validation Logging:**
   - Stage 1 → Stage 2: "Verified: 0 keywords pending, 8 clusters created"
   - Stage 2 → Stage 3: "Verified: 0 clusters pending, 56 ideas created"
   - Stage 3 → Stage 4: "Verified: 0 ideas pending, 56 tasks created"
   - Stage 4 → Stage 5: "Verified: 0 tasks pending, 56 content pieces created"
   - Stage 5 → Stage 6: "Verified: 0 content without images, 224 prompts created"
   - Stage 6 → Stage 7: "Verified: 0 pending images, 224 images generated"

---

### Task 4.6: Implement Dynamic Batch Size Logic

**Problem:** Fixed batch sizes don't adapt to actual queue sizes.

**Required Smart Batch Logic:**

1. **For Stages 1, 2, 3, 5:**
   - If `queue_count <= batch_size`: Process ALL items in one batch
   - If `queue_count > batch_size`: Split into batches

2. **For Stage 4 (Tasks → Content):**
   - Always process one task at a time (sequential)
   - Reason: Content generation is expensive, better control
   - Batch size config for Stage 4 can be deprecated

3. **For Stage 6 (Images):**
   - Process one image at a time (current behavior)
   - Reason: Image generation has rate limits

**Configuration Update:**
- Stage 1-3, 5: Batch size applies
- Stage 4, 6: Batch size ignored (always 1)
- Update Configure modal to clarify batch size usage per stage

---

## SECTION 5: STAGE CARD LAYOUT RESTRUCTURE

### Task 5.1: Add Missing Stage 5 Card (Content → Image Prompts)

**Problem:** Current UI combines Stages 3 & 4 into one card, Stage 5 missing.

**Required Change:**
- Create separate card for Stage 5
- Display: "Content → Image Prompts"
- Queue metrics: Content without images (not total content)
- Show progress bar for prompt extraction

---

### Task 5.2: Separate Stages 3 & 4 into Individual Cards

**Current:** One card shows "Ideas → Tasks → Content" with nested metrics.

**New Structure:**
1. **Stage 3 Card:** "Ideas → Tasks"
   - Total Queue: Ideas with `status='new'`
   - Processed: Ideas converted to tasks
   - Progress: Task creation count

2. **Stage 4 Card:** "Tasks → Content"
   - Total Queue: Tasks with `status='pending'`
   - Processed: Tasks with `status='completed'`
   - Progress: Content generation count

---

### Task 5.3: Restructure Stage Card Rows

**New Layout Requirements:**

**Row 1 (Stages 1-4):**
- Stage 1: Keywords → Clusters
- Stage 2: Clusters → Ideas
- Stage 3: Ideas → Tasks
- Stage 4: Tasks → Content

**Row 2 (Stages 5-8):**
- Stage 5: Content → Image Prompts
- Stage 6: Image Prompts → Images
- Stage 7: Review Gate (with action buttons)
- Stage 8: Status Summary (new informational card)

**Responsive Behavior:**
- Desktop: 4 cards per row
- Tablet: 2 cards per row
- Mobile: 1 card per row (vertical stack)

---

### Task 5.4: Design Stage 7 Card (Review Gate)

**Unique Requirements:**
1. **Visual Distinction:**
   - Different color scheme (amber/orange warning color)
   - Icon: Stop sign or review icon
   - Border: Dashed or highlighted

2. **Content:**
   - Title: "Manual Review Gate"
   - Status: "Automation Stops Here"
   - Count: Number of content pieces ready for review
   - Two buttons:
     - "Go to Review Page" (navigates to Writer Content page filtered by status='review')
     - "Publish Without Review" (disabled initially, placeholder for future feature)

3. **Button Behavior:**
   - Review button: Active when count > 0
   - Publish button: Disabled with tooltip "Coming soon"

---

### Task 5.5: Design Stage 8 Card (Status Summary)

**New Informational Card:**

**Purpose:** Display current automation run status without queue processing.

**Content:**
1. **Title:** "Current Status"
2. **Large Status Icon:** Based on run status
   - Running: Animated spinner
   - Completed: Checkmark
   - Failed: X icon
   - Paused: Pause icon

3. **Metrics Display:**
   - Run ID
   - Started at timestamp
   - Current stage indicator
   - Total credits used
   - Completion percentage

4. **Visual Style:**
   - No queue metrics
   - No action buttons
   - Read-only information display
   - Distinct styling (different background color, no hover effects)

---

### Task 5.6: Adjust Card Width for New Layout

**Current:** Stage cards likely using equal width across full viewport.

**New Requirements:**
- Row 1 (4 cards): Each card 23% width (with 2% gap)
- Row 2 (4 cards): Same width distribution
- Stage 8 card: Can be wider or styled differently as summary card

**Implementation Considerations:**
- Use CSS Grid or Flexbox for responsive layout
- Ensure consistent spacing between cards
- Maintain card aspect ratio for visual balance

---

## SECTION 6: ADDITIONAL ENHANCEMENTS

### Task 6.1: Add Credit Usage Tracking per Stage

**Value Addition:** Real-time visibility into credit consumption.

**Implementation:**
1. Track credits used at end of each stage in `stage_N_result` JSON field
2. Display in stage card: "Credits Used: X"
3. Add running total in overall pipeline progress bar
4. Compare estimated vs actual credits used
5. Alert if actual exceeds estimated by >20%

---

### Task 6.2: Add Estimated Completion Time per Stage

**Value Addition:** Predictable automation runtime for planning.

**Implementation:**
1. Calculate average time per item based on historical runs
2. Estimate: `Remaining Items * Average Time per Item`
3. Display in stage card: "ETA: 45 minutes"
4. Update dynamically as items process
5. Store metrics in database for accuracy improvement over time

---


### Task 6.3: Add Error Rate Monitoring

**Value Addition:** Proactive issue detection.

**Implementation:**
1. Track error count per stage
2. Display: "Errors: X (Y%)"
3. Highlight stages with >5% error rate
4. Add "View Errors" button to navigate to error log
5. Set up alerts for error rate spikes

---

### Task 6.4: Add Stage Completion Percentage

**Value Addition:** Clear progress visualization.

**Implementation:**
1. Calculate: `(Completed Items / Total Items) * 100`
2. Display as progress bar in stage card
3. Color code:
   - Green: >75%
   - Yellow: 25-75%
   - Red: <25%
4. Animate progress bar during active stages
5. Show exact percentage in text format

---

### Task 6.5: Add Stage Start/End Timestamps

**Value Addition:** Audit trail for automation runs.

**Implementation:**
1. Store start/end timestamps in `stage_N_result`
2. Display in stage card: "Started: 10:30 AM | Ended: 11:15 AM"
3