fixes

FIxes-and-refactor

@@ -0,0 +1,838 @@
+# IGNY8 Automation System - Detailed Task List for AI Agent
+
+## CRITICAL ANALYSIS
+
+Based on the documentation and current implementation status, I've identified significant issues with the automation system and legacy SiteBuilder references that need systematic resolution.
+
+---
+
+## SECTION 1: AUTO CLUSTER AI FUNCTION ISSUES
+
+### Task 1.1: Fix Cluster Status Assignment in AutoClusterFunction
+**Problem:** Clusters created by automation are assigned `status='active'` instead of proper status progression.
+
+**Current Behavior (from code analysis):**
+- Line 95 in `auto_cluster.py`: `Clusters.objects.get_or_create()` uses `defaults={'status': 'active'}`
+- This bypasses the intended status flow: `new` → `mapped`
+
+**Required Changes:**
+1. Modify `save_output()` method in `AutoClusterFunction` (backend/igny8_core/ai/functions/auto_cluster.py)
+2. Change cluster creation to use `status='new'` initially
+3. After keyword mapping completes, update cluster status to `status='mapped'`
+4. Ensure status progression aligns with automation pipeline expectations
+
+**Verification Steps:**
+- Create test with 10 keywords
+- Run Stage 1 via automation
+- Verify clusters have `status='new'` → then `status='mapped'`
+- Confirm automation Stage 2 correctly queries for these clusters
+
+---
+
+### Task 1.2: Investigate Keywords Not Mapping to Clusters
+**Problem:** Keywords processed through automation are not getting their `cluster_id` assigned correctly.
+
+**Diagnostic Steps Required:**
+1. Add detailed logging in `save_output()` method around line 100-110
+2. Log each keyword's ID, text, and whether cluster match was found
+3. Check if case-insensitive matching is working: `keyword_obj.keyword.lower() in [k.lower() for k in cluster_keywords]`
+4. Verify `Keywords.seed_keyword.keyword` field is populated correctly
+5. Check if AI response format matches expected structure: `{clusters: [{name, keywords: [], description}]}`
+
+**Root Cause Analysis:**
+- Examine parsed AI response structure
+- Verify keyword text extraction from `Keywords` model via `seed_keyword` relationship
+- Confirm database transaction commits are occurring
+- Check for any filtering conditions excluding keywords
+
+**Fix Implementation:**
+1. If mismatch in keyword text format, normalize both sides before comparison
+2. If seed_keyword relationship issue, fetch keyword text correctly
+3. Add fallback matching using keyword ID if text matching fails
+4. Ensure `keyword_obj.save()` is called within transaction
+
+---
+
+### Task 1.3: Distinguish Between Manual vs Automation Execution
+**Problem:** AutoClusterFunction behaves the same whether called manually or via automation.
+
+**Required Implementation:**
+1. Add `execution_context` parameter to function payload: `{'ids': [...], 'execution_context': 'automation' | 'manual'}`
+2. Pass context from `AutomationService.run_stage_1()` when calling AI function
+3. In `save_output()`, adjust status assignments based on context:
+   - Manual: Keep current behavior (`status='active'`)
+   - Automation: Use progressive status (`'new'` → `'mapped'`)
+4. Document context parameter in function metadata
+
+---
+
+## SECTION 2: LEGACY SITEBUILDER/BLUEPRINT REMOVAL
+
+### Task 2.1: Database Models Cleanup
+**Files to Remove Completely:**
+1. `backend/igny8_core/business/site_building/models.py` - Already stubbed, remove entirely
+2. Migration already exists: `0002_remove_blueprint_models.py` - Verify it ran successfully
+
+**Database Verification:**
+1. Connect to production database
+2. Run SQL: `SELECT tablename FROM pg_tables WHERE tablename LIKE '%blueprint%' OR tablename LIKE '%site_building%';`
+3. Expected result: No tables (already dropped)
+4. If tables exist, manually run DROP TABLE commands from migration
+
+**Foreign Key Cleanup:**
+1. Check `igny8_deployment_records` table - verify `site_blueprint_id` column removed
+2. Check `igny8_publishing_records` table - verify `site_blueprint_id` column removed
+3. Confirm indexes dropped: `igny8_publishing_recor_site_blueprint_id_des_b7c4e5f8_idx`
+
+---
+
+### Task 2.2: Backend Code References Removal
+
+**Phase 2.2.1: Remove Stub Models**
+- **File:** `backend/igny8_core/business/site_building/models.py`
+- **Action:** Delete entire file
+- **Reason:** Contains only stub classes (`SiteBlueprint`, `PageBlueprint`, `SiteBlueprintCluster`, `SiteBlueprintTaxonomy`) with no functionality
+
+**Phase 2.2.2: Remove Entire site_building App**
+- **Directory:** `backend/igny8_core/business/site_building/`
+- **Action:** Delete entire directory
+- **Reason:** All functionality deprecated, no active code
+
+**Files to Delete:**
+1. `services/structure_generation_service.py` - Calls deprecated AI function
+2. `services/page_generation_service.py` - References PageBlueprint
+3. `services/taxonomy_service.py` - Uses SiteBlueprintTaxonomy
+4. `services/file_management_service.py` - SiteBuilder file management
+5. `tests/` - All test files reference removed models
+6. `admin.py` - Already commented out
+7. `migrations/` - Keep for database history, but app removal makes them inert
+
+**Phase 2.2.3: Remove site_builder Module**
+- **Directory:** `backend/igny8_core/modules/site_builder.backup/`
+- **Action:** Delete entire directory (already marked `.backup`)
+- **Contains:** Deprecated API endpoints, serializers, views for blueprint management
+
+---
+
+### Task 2.3: Settings Configuration Cleanup
+
+**File:** `backend/igny8_core/settings.py`
+
+**Changes:**
+1. Line 56: Already commented out - Remove comment entirely
+2. Line 61: Already commented out - Remove comment entirely
+3. Verify `INSTALLED_APPS` list is clean
+
+**Verification:**
+- Run `python manage.py check` - Should pass
+- Run `python manage.py migrate --plan` - Should show no pending site_building migrations
+
+---
+
+### Task 2.4: URL Routing Cleanup
+
+**File:** `backend/igny8_core/urls.py`
+
+**Changes:**
+1. Line 42: Comment already exists - Remove comment entirely
+2. Verify no routing to `site-builder/` endpoints exists
+
+**Verification:**
+- Run Django server
+- Attempt to access `/api/v1/site-builder/blueprints/` - Should return 404
+- Check API root `/api/v1/` - Should not list site-builder endpoints
+
+---
+
+### Task 2.5: AI Function Removal
+
+**File:** `backend/igny8_core/ai/functions/generate_page_content.py`
+
+**Problem:** This AI function depends on `PageBlueprint` model which no longer exists.
+
+**Action Required:**
+1. **DELETE FILE:** `generate_page_content.py` (21 references to PageBlueprint)
+2. **UPDATE:** `backend/igny8_core/ai/registry.py` - Remove lazy loader registration
+3. **UPDATE:** `backend/igny8_core/ai/engine.py` - Remove from operation type mappings (line 599)
+
+**Verification:**
+- Search codebase for `generate_page_content` function calls
+- Ensure no active code relies on this function
+- Confirm AI function registry no longer lists it
+
+---
+
+### Task 2.6: Backend Import Statement Cleanup
+
+**Files with Import Statements to Update:**
+
+1. **backend/igny8_core/business/integration/services/content_sync_service.py**
+   - Lines 378, 488: `from igny8_core.business.site_building.models import SiteBlueprint`
+   - **Action:** Remove import, remove dependent code blocks (lines 382-388, 491-496)
+   - **Alternative:** Service should use `ContentTaxonomy` directly (post-migration model)
+
+2. **backend/igny8_core/business/integration/services/sync_health_service.py**
+   - Line 335: `from igny8_core.business.site_building.models import SiteBlueprint, SiteBlueprintTaxonomy`
+   - **Action:** Remove import, refactor taxonomy checks to use `ContentTaxonomy`
+
+3. **backend/igny8_core/business/publishing/services/adapters/sites_renderer_adapter.py**
+   - Line 15: `from igny8_core.business.site_building.models import SiteBlueprint`
+   - **Action:** Entire adapter is deprecated - DELETE FILE
+   - **Reason:** Designed to deploy SiteBlueprint instances, no longer applicable
+
+4. **backend/igny8_core/business/publishing/services/deployment_readiness_service.py**
+   - Line 10: `from igny8_core.business.site_building.models import SiteBlueprint`
+   - **Action:** DELETE FILE or refactor to remove blueprint checks
+   - **Reason:** Service checks blueprint readiness for deployment
+
+5. **backend/igny8_core/business/publishing/services/deployment_service.py**
+   - Line 10: `from igny8_core.business.site_building.models import SiteBlueprint`
+   - **Action:** Remove blueprint-specific deployment methods
+
+---
+
+### Task 2.7: Frontend Files Removal
+
+**Phase 2.7.1: Remove Type Definitions**
+- **File:** `frontend/src/types/siteBuilder.ts`
+- **Action:** Delete file entirely
+- **References:** Used in store and components
+
+**Phase 2.7.2: Remove API Service**
+- **File:** `frontend/src/services/siteBuilder.api.ts`
+- **Action:** Delete file
+- **Contains:** API methods for blueprint CRUD operations
+
+**Phase 2.7.3: Remove Pages**
+- **Directory:** `frontend/src/pages/Sites/`
+- **Files to Review:**
+  - `Editor.tsx` - Uses PageBlueprint, SiteBlueprint types (lines 15-36)
+  - `PageManager.tsx` - Fetches blueprints (lines 126-137)
+  - `DeploymentPanel.tsx` - Blueprint deployment UI (46 references)
+
+**Action for Pages:**
+1. If pages ONLY deal with blueprints - DELETE
+2. If pages have mixed functionality - REFACTOR to remove blueprint code
+3. Likely DELETE: `Editor.tsx`, `DeploymentPanel.tsx`
+4. Likely REFACTOR: `Dashboard.tsx` (remove blueprint widget)
+
+**Phase 2.7.4: Remove Store**
+- **File:** `frontend/src/store/siteDefinitionStore.ts`
+- **Action:** Review dependencies, likely DELETE
+- **Alternative:** If used for non-blueprint purposes, refactor to remove PageBlueprint types
+
+**Phase 2.7.5: Remove Components**
+- **File:** `frontend/src/components/sites/SiteProgressWidget.tsx`
+- **Action:** DELETE if blueprint-specific
+- **Uses:** `blueprintId` prop, calls `fetchSiteProgress(blueprintId)`
+
+---
+
+### Task 2.8: Frontend Import and Reference Cleanup
+
+**Files Requiring Updates:**
+
+1. **frontend/src/services/api.ts**
+   - Lines 2302-2532: Multiple blueprint-related functions
+   - **Action:** Remove these function exports:
+     - `fetchDeploymentReadiness`
+     - `createSiteBlueprint`, `updateSiteBlueprint`
+     - `attachClustersToBlueprint`, `detachClustersFromBlueprint`
+     - `fetchBlueprintsTaxonomies`, `createBlueprintTaxonomy`
+     - `importBlueprintsTaxonomies`
+     - `updatePageBlueprint`, `regeneratePageBlueprint`
+
+2. **frontend/src/pages/Planner/Dashboard.tsx**
+   - Lines 30-31: Commented imports
+   - **Action:** Remove commented lines entirely
+
+3. **frontend/src/config/pages/tasks.config.tsx**
+   - Lines 110-111: Logic for `[Site Builder]` task title formatting
+   - **Action:** Remove special handling, update title display logic
+
+---
+
+### Task 2.9: Sites Renderer Cleanup
+
+**File:** `sites/src/loaders/loadSiteDefinition.ts`
+
+**Current Behavior (Lines 103-159):**
+- API load fails → Falls back to blueprint endpoint
+- Transforms blueprint to site definition format
+
+**Required Changes:**
+1. Remove fallback to blueprint endpoint (lines 116-127)
+2. Remove `transformBlueprintToSiteDefinition` function (lines 137-159)
+3. If API fails, return proper error instead of fallback
+4. Update error messages to remove blueprint references
+
+---
+
+### Task 2.10: Documentation Cleanup
+
+**Files to Remove:**
+1. `docs/igny8-pp/TAXONOMY/QUICK-REFERENCE-TAXONOMY.md` - References SiteBuilder removal
+2. Update `docs/tech-stack/00-SYSTEM-ARCHITECTURE-MASTER-REFERENCE.md`:
+   - Remove "Site Blueprints" from feature list (line 45)
+   - Remove `site_builder/` from architecture diagrams (lines 179-180)
+   - Remove SiteBuilder from system overview (line 1517)
+
+**Files to Update:**
+1. `docs/igny8-pp/01-IGNY8-REST-API-COMPLETE-REFERENCE.md`:
+   - Remove entire section: "Site Blueprints" (lines 1201-1230)
+   - Remove "Page Blueprints" section (lines 1230-1247)
+   - Update deployment endpoints to remove blueprint references
+
+2. `docs/igny8-pp/02-PLANNER-WRITER-WORKFLOW-TECHNICAL-GUIDE.md`:
+   - Remove SiteBlueprintTaxonomy references (lines 114, 151)
+
+---
+
+### Task 2.11: Test Files Cleanup
+
+**Backend Tests:**
+1. DELETE: `backend/igny8_core/ai/tests/test_generate_site_structure_function.py`
+2. DELETE: `backend/igny8_core/business/site_building/tests/` (entire directory)
+3. DELETE: `backend/igny8_core/business/publishing/tests/test_deployment_service.py`
+4. DELETE: `backend/igny8_core/business/publishing/tests/test_publisher_service.py`
+5. DELETE: `backend/igny8_core/business/publishing/tests/test_adapters.py`
+
+**Frontend Tests:**
+1. DELETE: `frontend/src/__tests__/sites/BulkGeneration.test.tsx`
+2. UPDATE: `frontend/src/__tests__/sites/PromptManagement.test.tsx`:
+   - Remove site_structure_generation prompt type checks (lines 27-28)
+3. UPDATE: `frontend/src/__tests__/sites/SiteManagement.test.tsx`:
+   - Remove `[Site Builder]` task filtering logic (lines 50-51)
+
+---
+
+### Task 2.12: Database Migration Verification
+
+**Critical Checks:**
+1. Verify `0002_remove_blueprint_models.py` migration applied in all environments
+2. Check for orphaned data:
+   - Query for any `Tasks` with `taxonomy_id` pointing to deleted SiteBlueprintTaxonomy
+   - Query for any `ContentIdeas` with old taxonomy foreign keys
+3. If orphaned data found, create data migration to:
+   - Set taxonomy foreign keys to NULL
+   - Or migrate to ContentTaxonomy if mapping exists
+
+**SQL Verification Queries:**
+```sql
+-- Check for blueprint tables (should return empty)
+SELECT tablename FROM pg_tables 
+WHERE tablename LIKE '%blueprint%' OR tablename LIKE '%site_building%';
+
+-- Check for foreign key constraints (should return empty)
+SELECT constraint_name FROM information_schema.table_constraints 
+WHERE constraint_name LIKE '%blueprint%';
+
+-- Check for orphaned taxonomy references
+SELECT COUNT(*) FROM igny8_tasks WHERE taxonomy_id IS NOT NULL;
+SELECT COUNT(*) FROM igny8_content_ideas WHERE taxonomy_id IS NOT NULL;
+```
+
+---
+
+## SECTION 3: AUTOMATION PAGE UI IMPROVEMENTS
+
+### Task 3.1: Stage Card Visual Redesign
+
+**Current Issues:**
+- Icons too large, taking excessive space
+- Stage names not clearly associated with stage numbers
+- Inconsistent visual hierarchy
+
+**Required Changes:**
+1. **Reduce Icon Size:**
+   - Current: Large colored square icons
+   - New: Smaller icons (32x32px instead of current size)
+   - Position: Top-left of card, not centered
+
+2. **Restructure Stage Header:**
+   - Move stage name directly below "Stage N" text
+   - Format: "Stage 1" (bold) / "Keywords → Clusters" (regular weight, smaller font)
+   - Remove redundant text repetition
+
+3. **Status Badge Positioning:**
+   - Move from separate row to same line as stage number
+   - Right-align badge next to stage number
+
+**Layout Example (No Code):**
+```
+┌─────────────────────────────┐
+│ [Icon] Stage 1     [Ready]  │
+│        Keywords → Clusters   │
+│                              │
+│ Total Queue:            7    │
+│ Processed:              0    │
+│ Remaining:              7    │
+└─────────────────────────────┘
+```
+
+---
+
+### Task 3.2: Add Progress Bars to Stage Cards
+
+**Implementation Requirements:**
+1. **Individual Stage Progress Bar:**
+   - Display below queue metrics
+   - Calculate: `(Processed / Total Queue) * 100`
+   - Visual: Colored bar matching stage color
+   - Show percentage label
+
+2. **Overall Pipeline Progress Bar:**
+   - Large bar above all stage cards
+   - Calculate: `(Sum of Processed Items Across All Stages) / (Sum of Total Queue Across All Stages) * 100`
+   - Display current stage indicator: "Stage 4/7"
+   - Show estimated completion time
+
+3. **Progress Bar States:**
+   - Empty (0%): Gray/outline only
+   - In Progress (1-99%): Animated gradient
+   - Complete (100%): Solid color, checkmark icon
+
+---
+
+### Task 3.3: Add Total Metrics Cards Above Pipeline
+
+**New Component: MetricsSummary Cards**
+
+**Cards to Display (Row above pipeline overview):**
+1. **Keywords Card:**
+   - Total: Count from database
+   - Processed: Keywords with `status='mapped'`
+   - Pending: Keywords with `status='new'`
+
+2. **Clusters Card:**
+   - Total: All clusters for site
+   - Processed: Clusters with ideas generated
+   - Pending: Clusters without ideas
+
+3. **Ideas Card:**
+   - Total: All ideas for site
+   - Processed: Ideas converted to tasks (`status='in_progress'`)
+   - Pending: Ideas with `status='new'`
+
+4. **Content Card:**
+   - Total: All content for site
+   - Processed: Content with `status='draft'` + all images generated
+   - Pending: Content without images or in generation
+
+5. **Images Card:**
+   - Total: All image records for site content
+   - Processed: Images with `status='generated'`
+   - Pending: Images with `status='pending'`
+
+**Card Layout:**
+- Width: Equal distribution across row
+- Display: Icon, Title, Total/Processed/Pending counts
+- Color: Match stage colors for visual consistency
+
+---
+
+### Task 3.4: Pipeline Status Card Redesign
+
+**Current:** Wide row with text "Pipeline Status - Ready to run | 22 items pending"
+
+**New Design Requirements:**
+1. **Convert to Centered Card:**
+   - Position: Above stage cards, below metrics summary
+   - Width: Narrower than full width, centered
+   - Style: Elevated/shadowed for emphasis
+
+2. **Content Structure:**
+   - Large status indicator (icon + text)
+   - Prominent pending items count
+   - Quick action buttons (Run Now, Pause, Configure)
+
+3. **Status Visual States:**
+   - Ready: Green pulse animation
+   - Running: Blue animated progress
+   - Paused: Yellow warning icon
+   - Failed: Red alert icon
+
+---
+
+### Task 3.5: Remove/Compact Header Elements
+
+**Changes to Automation Page Header:**
+
+1. **Remove "Pipeline Overview" Section:**
+   - Delete heading: "📊 Pipeline Overview"
+   - Delete subtitle: "Complete view of automation pipeline status and pending items"
+   - Reason: Redundant with visible pipeline cards
+
+2. **Compact Schedule Panel:**
+   - Current: Large panel with heading, status row, action buttons
+   - New: Single compact row
+   - Layout: `[Status Badge] | [Schedule Text] | [Last Run] | [Estimated Credits] | [Configure Button] | [Run Now Button]`
+   - Remove empty space and excessive padding
+
+---
+
+### Task 3.6: AI Request Delays Implementation
+
+**Problem:** Rapid sequential AI requests may hit rate limits or overload AI service.
+
+**Required Changes:**
+
+1. **Within-Stage Delay (between batches):**
+   - Location: `AutomationService` class methods
+   - Add delay after each batch completion before processing next batch
+   - Configurable: 3-5 seconds (default 3 seconds)
+   - Implementation point: After each AI function call completes in stage loop
+
+2. **Between-Stage Delay:**
+   - Add delay after stage completion before triggering next stage
+   - Configurable: 5-10 seconds (default 5 seconds)
+   - Implementation point: After `_execute_stage()` returns before incrementing `current_stage`
+
+3. **Configuration:**
+   - Add to `AutomationConfig` model: `within_stage_delay` (integer, seconds)
+   - Add to `AutomationConfig` model: `between_stage_delay` (integer, seconds)
+   - Expose in Configure modal for user adjustment
+
+4. **Logging:**
+   - Log delay start: "Waiting 3 seconds before next batch..."
+   - Log delay end: "Delay complete, resuming processing"
+
+---
+
+## SECTION 4: AUTOMATION STAGE PROCESSING FIXES
+
+### Task 4.1: Verify Stage Sequential Processing Logic
+
+**Problem:** Pipeline not following strict sequential stage completion before moving to next stage.
+
+**Analysis Required:**
+1. Review `AutomationService.start_automation()` method
+2. Verify stage loop waits for 100% completion before `current_stage += 1`
+3. Check for any parallel execution logic that bypasses sequential flow
+
+**Verification Steps:**
+1. Each stage method (`run_stage_1()` to `run_stage_7()`) must return ONLY after ALL batches processed
+2. Stage N+1 should NOT start if Stage N has `pending > 0`
+3. Add explicit completion check before stage transition
+
+**Required Fixes:**
+- Add validation: Before starting Stage N, verify Stage N-1 has 0 pending items
+- If pending items found, log warning and halt automation
+- Return error status with message: "Stage N-1 incomplete, cannot proceed to Stage N"
+
+---
+
+### Task 4.2: Fix Batch Size Configuration Reading
+
+**Problem:** Manual "Run Now" only processes 5 keywords instead of respecting configured batch size (20).
+
+**Root Cause Analysis:**
+1. Check if `run_stage_1()` reads from `AutomationConfig.stage_1_batch_size`
+2. Verify query limit: `Keywords.objects.filter(...)[:batch_size]` uses correct variable
+3. Confirm configuration loaded at automation start: `config = AutomationConfig.objects.get(site=self.site)`
+
+**Expected Behavior:**
+- If queue has 7 keywords and batch_size = 20: Process all 7 (not limit to 5)
+- If queue has 47 keywords and batch_size = 20: Process 20, then next batch of 20, then final 7
+- Batch size should be dynamic based on queue size: `min(queue_count, batch_size)`
+
+**Fix Implementation:**
+1. Ensure configuration loaded once at automation start
+2. Pass batch_size to each stage method
+3. Update query to use: `[:min(pending_count, batch_size)]`
+4. Log batch selection: "Processing batch 1/3: 20 keywords"
+
+---
+
+### Task 4.3: Fix Stage 4 Processing Not Completing Full Queue
+
+**Problem:** Stage 4 (Tasks → Content) not processing all tasks before moving to Stage 5.
+
+**Investigation Steps:**
+1. Check `run_stage_4()` implementation in `AutomationService`
+2. Verify loop structure: Does it process tasks one-by-one until queue empty?
+3. Look for premature loop exit conditions
+
+**Expected Logic:**
+```
+While tasks with status='pending' exist:
+    1. Get next task
+    2. Call generate_content AI function
+    3. Wait for completion
+    4. Verify Content created
+    5. Check if more pending tasks exist
+    6. If yes, continue loop
+    7. If no, return stage complete
+```
+
+**Common Issues to Check:**
+- Loop exits after first task instead of continuing
+- No loop at all - only processes one batch
+- Error handling breaks loop prematurely
+
+---
+
+### Task 4.4: Fix Stage 5 Not Triggering (Image Prompts Generation)
+
+**Problem:** Automation exits after Stage 4 without generating image prompts.
+
+**Analysis Required:**
+1. Verify Stage 4 completion status set correctly
+2. Check if Stage 5 start condition is met
+3. Review database query in `run_stage_5()`:
+   - Query: Content with `status='draft'` AND `images_count=0`
+   - Verify Content records created in Stage 4 have correct status
+
+**Potential Issues:**
+1. Content created with status other than 'draft'
+2. Images count annotation incorrect (should use `annotate(images_count=Count('images'))`)
+3. Stage handover logic doesn't trigger Stage 5
+
+**Fix Steps:**
+1. Verify Content model save in Stage 4 sets `status='draft'`
+2. Ensure Stage 5 query matches Content records from Stage 4
+3. Add logging: "Stage 5: Found X content pieces without images"
+4. If X > 0, process; if X = 0, skip stage gracefully
+
+---
+
+### Task 4.5: Add Stage Handover Validation
+
+**New Logic Required Between Each Stage:**
+
+1. **Pre-Stage Validation:**
+   - Before starting Stage N, check Stage N-1 completion:
+     - Query pending items for Stage N-1
+     - If pending > 0: Log error, halt automation
+     - If pending = 0: Log success, proceed
+
+2. **Post-Stage Validation:**
+   - After Stage N completes, verify:
+     - All input items processed
+     - Expected output items created
+     - No errors in stage result
+   - Log validation result before moving to Stage N+1
+
+3. **Validation Logging:**
+   - Stage 1 → Stage 2: "Verified: 0 keywords pending, 8 clusters created"
+   - Stage 2 → Stage 3: "Verified: 0 clusters pending, 56 ideas created"
+   - Stage 3 → Stage 4: "Verified: 0 ideas pending, 56 tasks created"
+   - Stage 4 → Stage 5: "Verified: 0 tasks pending, 56 content pieces created"
+   - Stage 5 → Stage 6: "Verified: 0 content without images, 224 prompts created"
+   - Stage 6 → Stage 7: "Verified: 0 pending images, 224 images generated"
+
+---
+
+### Task 4.6: Implement Dynamic Batch Size Logic
+
+**Problem:** Fixed batch sizes don't adapt to actual queue sizes.
+
+**Required Smart Batch Logic:**
+
+1. **For Stages 1, 2, 3, 5:**
+   - If `queue_count <= batch_size`: Process ALL items in one batch
+   - If `queue_count > batch_size`: Split into batches
+
+2. **For Stage 4 (Tasks → Content):**
+   - Always process one task at a time (sequential)
+   - Reason: Content generation is expensive, better control
+   - Batch size config for Stage 4 can be deprecated
+
+3. **For Stage 6 (Images):**
+   - Process one image at a time (current behavior)
+   - Reason: Image generation has rate limits
+
+**Configuration Update:**
+- Stage 1-3, 5: Batch size applies
+- Stage 4, 6: Batch size ignored (always 1)
+- Update Configure modal to clarify batch size usage per stage
+
+---
+
+## SECTION 5: STAGE CARD LAYOUT RESTRUCTURE
+
+### Task 5.1: Add Missing Stage 5 Card (Content → Image Prompts)
+
+**Problem:** Current UI combines Stages 3 & 4 into one card, Stage 5 missing.
+
+**Required Change:**
+- Create separate card for Stage 5
+- Display: "Content → Image Prompts"
+- Queue metrics: Content without images (not total content)
+- Show progress bar for prompt extraction
+
+---
+
+### Task 5.2: Separate Stages 3 & 4 into Individual Cards
+
+**Current:** One card shows "Ideas → Tasks → Content" with nested metrics.
+
+**New Structure:**
+1. **Stage 3 Card:** "Ideas → Tasks"
+   - Total Queue: Ideas with `status='new'`
+   - Processed: Ideas converted to tasks
+   - Progress: Task creation count
+
+2. **Stage 4 Card:** "Tasks → Content"
+   - Total Queue: Tasks with `status='pending'`
+   - Processed: Tasks with `status='completed'`
+   - Progress: Content generation count
+
+---
+
+### Task 5.3: Restructure Stage Card Rows
+
+**New Layout Requirements:**
+
+**Row 1 (Stages 1-4):**
+- Stage 1: Keywords → Clusters
+- Stage 2: Clusters → Ideas
+- Stage 3: Ideas → Tasks
+- Stage 4: Tasks → Content
+
+**Row 2 (Stages 5-8):**
+- Stage 5: Content → Image Prompts
+- Stage 6: Image Prompts → Images
+- Stage 7: Review Gate (with action buttons)
+- Stage 8: Status Summary (new informational card)
+
+**Responsive Behavior:**
+- Desktop: 4 cards per row
+- Tablet: 2 cards per row
+- Mobile: 1 card per row (vertical stack)
+
+---
+
+### Task 5.4: Design Stage 7 Card (Review Gate)
+
+**Unique Requirements:**
+1. **Visual Distinction:**
+   - Different color scheme (amber/orange warning color)
+   - Icon: Stop sign or review icon
+   - Border: Dashed or highlighted
+
+2. **Content:**
+   - Title: "Manual Review Gate"
+   - Status: "Automation Stops Here"
+   - Count: Number of content pieces ready for review
+   - Two buttons:
+     - "Go to Review Page" (navigates to Writer Content page filtered by status='review')
+     - "Publish Without Review" (disabled initially, placeholder for future feature)
+
+3. **Button Behavior:**
+   - Review button: Active when count > 0
+   - Publish button: Disabled with tooltip "Coming soon"
+
+---
+
+### Task 5.5: Design Stage 8 Card (Status Summary)
+
+**New Informational Card:**
+
+**Purpose:** Display current automation run status without queue processing.
+
+**Content:**
+1. **Title:** "Current Status"
+2. **Large Status Icon:** Based on run status
+   - Running: Animated spinner
+   - Completed: Checkmark
+   - Failed: X icon
+   - Paused: Pause icon
+
+3. **Metrics Display:**
+   - Run ID
+   - Started at timestamp
+   - Current stage indicator
+   - Total credits used
+   - Completion percentage
+
+4. **Visual Style:**
+   - No queue metrics
+   - No action buttons
+   - Read-only information display
+   - Distinct styling (different background color, no hover effects)
+
+---
+
+### Task 5.6: Adjust Card Width for New Layout
+
+**Current:** Stage cards likely using equal width across full viewport.
+
+**New Requirements:**
+- Row 1 (4 cards): Each card 23% width (with 2% gap)
+- Row 2 (4 cards): Same width distribution
+- Stage 8 card: Can be wider or styled differently as summary card
+
+**Implementation Considerations:**
+- Use CSS Grid or Flexbox for responsive layout
+- Ensure consistent spacing between cards
+- Maintain card aspect ratio for visual balance
+
+---
+
+## SECTION 6: ADDITIONAL ENHANCEMENTS
+
+### Task 6.1: Add Credit Usage Tracking per Stage
+
+**Value Addition:** Real-time visibility into credit consumption.
+
+**Implementation:**
+1. Track credits used at end of each stage in `stage_N_result` JSON field
+2. Display in stage card: "Credits Used: X"
+3. Add running total in overall pipeline progress bar
+4. Compare estimated vs actual credits used
+5. Alert if actual exceeds estimated by >20%
+
+---
+
+### Task 6.2: Add Estimated Completion Time per Stage
+
+**Value Addition:** Predictable automation runtime for planning.
+
+**Implementation:**
+1. Calculate average time per item based on historical runs
+2. Estimate: `Remaining Items * Average Time per Item`
+3. Display in stage card: "ETA: 45 minutes"
+4. Update dynamically as items process
+5. Store metrics in database for accuracy improvement over time
+
+---
+
+
+### Task 6.3: Add Error Rate Monitoring
+
+**Value Addition:** Proactive issue detection.
+
+**Implementation:**
+1. Track error count per stage
+2. Display: "Errors: X (Y%)"
+3. Highlight stages with >5% error rate
+4. Add "View Errors" button to navigate to error log
+5. Set up alerts for error rate spikes
+
+---
+
+### Task 6.4: Add Stage Completion Percentage
+
+**Value Addition:** Clear progress visualization.
+
+**Implementation:**
+1. Calculate: `(Completed Items / Total Items) * 100`
+2. Display as progress bar in stage card
+3. Color code:
+   - Green: >75%
+   - Yellow: 25-75%
+   - Red: <25%
+4. Animate progress bar during active stages
+5. Show exact percentage in text format
+
+---
+
+### Task 6.5: Add Stage Start/End Timestamps
+
+**Value Addition:** Audit trail for automation runs.
+
+**Implementation:**
+1. Store start/end timestamps in `stage_N_result`
+2. Display in stage card: "Started: 10:30 AM | Ended: 11:15 AM"
+3