From 544741fbe608cb78d01050da21dac4903b9fa595 Mon Sep 17 00:00:00 2001 From: "IGNY8 VPS (Salman)" Date: Wed, 3 Dec 2025 14:03:08 +0000 Subject: [PATCH] fixes --- FIxes-and-refactor | 838 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 838 insertions(+) create mode 100644 FIxes-and-refactor diff --git a/FIxes-and-refactor b/FIxes-and-refactor new file mode 100644 index 00000000..d972eff6 --- /dev/null +++ b/FIxes-and-refactor @@ -0,0 +1,838 @@ +# IGNY8 Automation System - Detailed Task List for AI Agent + +## CRITICAL ANALYSIS + +Based on the documentation and current implementation status, I've identified significant issues with the automation system and legacy SiteBuilder references that need systematic resolution. + +--- + +## SECTION 1: AUTO CLUSTER AI FUNCTION ISSUES + +### Task 1.1: Fix Cluster Status Assignment in AutoClusterFunction +**Problem:** Clusters created by automation are assigned `status='active'` instead of proper status progression. + +**Current Behavior (from code analysis):** +- Line 95 in `auto_cluster.py`: `Clusters.objects.get_or_create()` uses `defaults={'status': 'active'}` +- This bypasses the intended status flow: `new` → `mapped` + +**Required Changes:** +1. Modify `save_output()` method in `AutoClusterFunction` (backend/igny8_core/ai/functions/auto_cluster.py) +2. Change cluster creation to use `status='new'` initially +3. After keyword mapping completes, update cluster status to `status='mapped'` +4. Ensure status progression aligns with automation pipeline expectations + +**Verification Steps:** +- Create test with 10 keywords +- Run Stage 1 via automation +- Verify clusters have `status='new'` → then `status='mapped'` +- Confirm automation Stage 2 correctly queries for these clusters + +--- + +### Task 1.2: Investigate Keywords Not Mapping to Clusters +**Problem:** Keywords processed through automation are not getting their `cluster_id` assigned correctly. + +**Diagnostic Steps Required:** +1. Add detailed logging in `save_output()` method around line 100-110 +2. Log each keyword's ID, text, and whether cluster match was found +3. Check if case-insensitive matching is working: `keyword_obj.keyword.lower() in [k.lower() for k in cluster_keywords]` +4. Verify `Keywords.seed_keyword.keyword` field is populated correctly +5. Check if AI response format matches expected structure: `{clusters: [{name, keywords: [], description}]}` + +**Root Cause Analysis:** +- Examine parsed AI response structure +- Verify keyword text extraction from `Keywords` model via `seed_keyword` relationship +- Confirm database transaction commits are occurring +- Check for any filtering conditions excluding keywords + +**Fix Implementation:** +1. If mismatch in keyword text format, normalize both sides before comparison +2. If seed_keyword relationship issue, fetch keyword text correctly +3. Add fallback matching using keyword ID if text matching fails +4. Ensure `keyword_obj.save()` is called within transaction + +--- + +### Task 1.3: Distinguish Between Manual vs Automation Execution +**Problem:** AutoClusterFunction behaves the same whether called manually or via automation. + +**Required Implementation:** +1. Add `execution_context` parameter to function payload: `{'ids': [...], 'execution_context': 'automation' | 'manual'}` +2. Pass context from `AutomationService.run_stage_1()` when calling AI function +3. In `save_output()`, adjust status assignments based on context: + - Manual: Keep current behavior (`status='active'`) + - Automation: Use progressive status (`'new'` → `'mapped'`) +4. Document context parameter in function metadata + +--- + +## SECTION 2: LEGACY SITEBUILDER/BLUEPRINT REMOVAL + +### Task 2.1: Database Models Cleanup +**Files to Remove Completely:** +1. `backend/igny8_core/business/site_building/models.py` - Already stubbed, remove entirely +2. Migration already exists: `0002_remove_blueprint_models.py` - Verify it ran successfully + +**Database Verification:** +1. Connect to production database +2. Run SQL: `SELECT tablename FROM pg_tables WHERE tablename LIKE '%blueprint%' OR tablename LIKE '%site_building%';` +3. Expected result: No tables (already dropped) +4. If tables exist, manually run DROP TABLE commands from migration + +**Foreign Key Cleanup:** +1. Check `igny8_deployment_records` table - verify `site_blueprint_id` column removed +2. Check `igny8_publishing_records` table - verify `site_blueprint_id` column removed +3. Confirm indexes dropped: `igny8_publishing_recor_site_blueprint_id_des_b7c4e5f8_idx` + +--- + +### Task 2.2: Backend Code References Removal + +**Phase 2.2.1: Remove Stub Models** +- **File:** `backend/igny8_core/business/site_building/models.py` +- **Action:** Delete entire file +- **Reason:** Contains only stub classes (`SiteBlueprint`, `PageBlueprint`, `SiteBlueprintCluster`, `SiteBlueprintTaxonomy`) with no functionality + +**Phase 2.2.2: Remove Entire site_building App** +- **Directory:** `backend/igny8_core/business/site_building/` +- **Action:** Delete entire directory +- **Reason:** All functionality deprecated, no active code + +**Files to Delete:** +1. `services/structure_generation_service.py` - Calls deprecated AI function +2. `services/page_generation_service.py` - References PageBlueprint +3. `services/taxonomy_service.py` - Uses SiteBlueprintTaxonomy +4. `services/file_management_service.py` - SiteBuilder file management +5. `tests/` - All test files reference removed models +6. `admin.py` - Already commented out +7. `migrations/` - Keep for database history, but app removal makes them inert + +**Phase 2.2.3: Remove site_builder Module** +- **Directory:** `backend/igny8_core/modules/site_builder.backup/` +- **Action:** Delete entire directory (already marked `.backup`) +- **Contains:** Deprecated API endpoints, serializers, views for blueprint management + +--- + +### Task 2.3: Settings Configuration Cleanup + +**File:** `backend/igny8_core/settings.py` + +**Changes:** +1. Line 56: Already commented out - Remove comment entirely +2. Line 61: Already commented out - Remove comment entirely +3. Verify `INSTALLED_APPS` list is clean + +**Verification:** +- Run `python manage.py check` - Should pass +- Run `python manage.py migrate --plan` - Should show no pending site_building migrations + +--- + +### Task 2.4: URL Routing Cleanup + +**File:** `backend/igny8_core/urls.py` + +**Changes:** +1. Line 42: Comment already exists - Remove comment entirely +2. Verify no routing to `site-builder/` endpoints exists + +**Verification:** +- Run Django server +- Attempt to access `/api/v1/site-builder/blueprints/` - Should return 404 +- Check API root `/api/v1/` - Should not list site-builder endpoints + +--- + +### Task 2.5: AI Function Removal + +**File:** `backend/igny8_core/ai/functions/generate_page_content.py` + +**Problem:** This AI function depends on `PageBlueprint` model which no longer exists. + +**Action Required:** +1. **DELETE FILE:** `generate_page_content.py` (21 references to PageBlueprint) +2. **UPDATE:** `backend/igny8_core/ai/registry.py` - Remove lazy loader registration +3. **UPDATE:** `backend/igny8_core/ai/engine.py` - Remove from operation type mappings (line 599) + +**Verification:** +- Search codebase for `generate_page_content` function calls +- Ensure no active code relies on this function +- Confirm AI function registry no longer lists it + +--- + +### Task 2.6: Backend Import Statement Cleanup + +**Files with Import Statements to Update:** + +1. **backend/igny8_core/business/integration/services/content_sync_service.py** + - Lines 378, 488: `from igny8_core.business.site_building.models import SiteBlueprint` + - **Action:** Remove import, remove dependent code blocks (lines 382-388, 491-496) + - **Alternative:** Service should use `ContentTaxonomy` directly (post-migration model) + +2. **backend/igny8_core/business/integration/services/sync_health_service.py** + - Line 335: `from igny8_core.business.site_building.models import SiteBlueprint, SiteBlueprintTaxonomy` + - **Action:** Remove import, refactor taxonomy checks to use `ContentTaxonomy` + +3. **backend/igny8_core/business/publishing/services/adapters/sites_renderer_adapter.py** + - Line 15: `from igny8_core.business.site_building.models import SiteBlueprint` + - **Action:** Entire adapter is deprecated - DELETE FILE + - **Reason:** Designed to deploy SiteBlueprint instances, no longer applicable + +4. **backend/igny8_core/business/publishing/services/deployment_readiness_service.py** + - Line 10: `from igny8_core.business.site_building.models import SiteBlueprint` + - **Action:** DELETE FILE or refactor to remove blueprint checks + - **Reason:** Service checks blueprint readiness for deployment + +5. **backend/igny8_core/business/publishing/services/deployment_service.py** + - Line 10: `from igny8_core.business.site_building.models import SiteBlueprint` + - **Action:** Remove blueprint-specific deployment methods + +--- + +### Task 2.7: Frontend Files Removal + +**Phase 2.7.1: Remove Type Definitions** +- **File:** `frontend/src/types/siteBuilder.ts` +- **Action:** Delete file entirely +- **References:** Used in store and components + +**Phase 2.7.2: Remove API Service** +- **File:** `frontend/src/services/siteBuilder.api.ts` +- **Action:** Delete file +- **Contains:** API methods for blueprint CRUD operations + +**Phase 2.7.3: Remove Pages** +- **Directory:** `frontend/src/pages/Sites/` +- **Files to Review:** + - `Editor.tsx` - Uses PageBlueprint, SiteBlueprint types (lines 15-36) + - `PageManager.tsx` - Fetches blueprints (lines 126-137) + - `DeploymentPanel.tsx` - Blueprint deployment UI (46 references) + +**Action for Pages:** +1. If pages ONLY deal with blueprints - DELETE +2. If pages have mixed functionality - REFACTOR to remove blueprint code +3. Likely DELETE: `Editor.tsx`, `DeploymentPanel.tsx` +4. Likely REFACTOR: `Dashboard.tsx` (remove blueprint widget) + +**Phase 2.7.4: Remove Store** +- **File:** `frontend/src/store/siteDefinitionStore.ts` +- **Action:** Review dependencies, likely DELETE +- **Alternative:** If used for non-blueprint purposes, refactor to remove PageBlueprint types + +**Phase 2.7.5: Remove Components** +- **File:** `frontend/src/components/sites/SiteProgressWidget.tsx` +- **Action:** DELETE if blueprint-specific +- **Uses:** `blueprintId` prop, calls `fetchSiteProgress(blueprintId)` + +--- + +### Task 2.8: Frontend Import and Reference Cleanup + +**Files Requiring Updates:** + +1. **frontend/src/services/api.ts** + - Lines 2302-2532: Multiple blueprint-related functions + - **Action:** Remove these function exports: + - `fetchDeploymentReadiness` + - `createSiteBlueprint`, `updateSiteBlueprint` + - `attachClustersToBlueprint`, `detachClustersFromBlueprint` + - `fetchBlueprintsTaxonomies`, `createBlueprintTaxonomy` + - `importBlueprintsTaxonomies` + - `updatePageBlueprint`, `regeneratePageBlueprint` + +2. **frontend/src/pages/Planner/Dashboard.tsx** + - Lines 30-31: Commented imports + - **Action:** Remove commented lines entirely + +3. **frontend/src/config/pages/tasks.config.tsx** + - Lines 110-111: Logic for `[Site Builder]` task title formatting + - **Action:** Remove special handling, update title display logic + +--- + +### Task 2.9: Sites Renderer Cleanup + +**File:** `sites/src/loaders/loadSiteDefinition.ts` + +**Current Behavior (Lines 103-159):** +- API load fails → Falls back to blueprint endpoint +- Transforms blueprint to site definition format + +**Required Changes:** +1. Remove fallback to blueprint endpoint (lines 116-127) +2. Remove `transformBlueprintToSiteDefinition` function (lines 137-159) +3. If API fails, return proper error instead of fallback +4. Update error messages to remove blueprint references + +--- + +### Task 2.10: Documentation Cleanup + +**Files to Remove:** +1. `docs/igny8-pp/TAXONOMY/QUICK-REFERENCE-TAXONOMY.md` - References SiteBuilder removal +2. Update `docs/tech-stack/00-SYSTEM-ARCHITECTURE-MASTER-REFERENCE.md`: + - Remove "Site Blueprints" from feature list (line 45) + - Remove `site_builder/` from architecture diagrams (lines 179-180) + - Remove SiteBuilder from system overview (line 1517) + +**Files to Update:** +1. `docs/igny8-pp/01-IGNY8-REST-API-COMPLETE-REFERENCE.md`: + - Remove entire section: "Site Blueprints" (lines 1201-1230) + - Remove "Page Blueprints" section (lines 1230-1247) + - Update deployment endpoints to remove blueprint references + +2. `docs/igny8-pp/02-PLANNER-WRITER-WORKFLOW-TECHNICAL-GUIDE.md`: + - Remove SiteBlueprintTaxonomy references (lines 114, 151) + +--- + +### Task 2.11: Test Files Cleanup + +**Backend Tests:** +1. DELETE: `backend/igny8_core/ai/tests/test_generate_site_structure_function.py` +2. DELETE: `backend/igny8_core/business/site_building/tests/` (entire directory) +3. DELETE: `backend/igny8_core/business/publishing/tests/test_deployment_service.py` +4. DELETE: `backend/igny8_core/business/publishing/tests/test_publisher_service.py` +5. DELETE: `backend/igny8_core/business/publishing/tests/test_adapters.py` + +**Frontend Tests:** +1. DELETE: `frontend/src/__tests__/sites/BulkGeneration.test.tsx` +2. UPDATE: `frontend/src/__tests__/sites/PromptManagement.test.tsx`: + - Remove site_structure_generation prompt type checks (lines 27-28) +3. UPDATE: `frontend/src/__tests__/sites/SiteManagement.test.tsx`: + - Remove `[Site Builder]` task filtering logic (lines 50-51) + +--- + +### Task 2.12: Database Migration Verification + +**Critical Checks:** +1. Verify `0002_remove_blueprint_models.py` migration applied in all environments +2. Check for orphaned data: + - Query for any `Tasks` with `taxonomy_id` pointing to deleted SiteBlueprintTaxonomy + - Query for any `ContentIdeas` with old taxonomy foreign keys +3. If orphaned data found, create data migration to: + - Set taxonomy foreign keys to NULL + - Or migrate to ContentTaxonomy if mapping exists + +**SQL Verification Queries:** +```sql +-- Check for blueprint tables (should return empty) +SELECT tablename FROM pg_tables +WHERE tablename LIKE '%blueprint%' OR tablename LIKE '%site_building%'; + +-- Check for foreign key constraints (should return empty) +SELECT constraint_name FROM information_schema.table_constraints +WHERE constraint_name LIKE '%blueprint%'; + +-- Check for orphaned taxonomy references +SELECT COUNT(*) FROM igny8_tasks WHERE taxonomy_id IS NOT NULL; +SELECT COUNT(*) FROM igny8_content_ideas WHERE taxonomy_id IS NOT NULL; +``` + +--- + +## SECTION 3: AUTOMATION PAGE UI IMPROVEMENTS + +### Task 3.1: Stage Card Visual Redesign + +**Current Issues:** +- Icons too large, taking excessive space +- Stage names not clearly associated with stage numbers +- Inconsistent visual hierarchy + +**Required Changes:** +1. **Reduce Icon Size:** + - Current: Large colored square icons + - New: Smaller icons (32x32px instead of current size) + - Position: Top-left of card, not centered + +2. **Restructure Stage Header:** + - Move stage name directly below "Stage N" text + - Format: "Stage 1" (bold) / "Keywords → Clusters" (regular weight, smaller font) + - Remove redundant text repetition + +3. **Status Badge Positioning:** + - Move from separate row to same line as stage number + - Right-align badge next to stage number + +**Layout Example (No Code):** +``` +┌─────────────────────────────┐ +│ [Icon] Stage 1 [Ready] │ +│ Keywords → Clusters │ +│ │ +│ Total Queue: 7 │ +│ Processed: 0 │ +│ Remaining: 7 │ +└─────────────────────────────┘ +``` + +--- + +### Task 3.2: Add Progress Bars to Stage Cards + +**Implementation Requirements:** +1. **Individual Stage Progress Bar:** + - Display below queue metrics + - Calculate: `(Processed / Total Queue) * 100` + - Visual: Colored bar matching stage color + - Show percentage label + +2. **Overall Pipeline Progress Bar:** + - Large bar above all stage cards + - Calculate: `(Sum of Processed Items Across All Stages) / (Sum of Total Queue Across All Stages) * 100` + - Display current stage indicator: "Stage 4/7" + - Show estimated completion time + +3. **Progress Bar States:** + - Empty (0%): Gray/outline only + - In Progress (1-99%): Animated gradient + - Complete (100%): Solid color, checkmark icon + +--- + +### Task 3.3: Add Total Metrics Cards Above Pipeline + +**New Component: MetricsSummary Cards** + +**Cards to Display (Row above pipeline overview):** +1. **Keywords Card:** + - Total: Count from database + - Processed: Keywords with `status='mapped'` + - Pending: Keywords with `status='new'` + +2. **Clusters Card:** + - Total: All clusters for site + - Processed: Clusters with ideas generated + - Pending: Clusters without ideas + +3. **Ideas Card:** + - Total: All ideas for site + - Processed: Ideas converted to tasks (`status='in_progress'`) + - Pending: Ideas with `status='new'` + +4. **Content Card:** + - Total: All content for site + - Processed: Content with `status='draft'` + all images generated + - Pending: Content without images or in generation + +5. **Images Card:** + - Total: All image records for site content + - Processed: Images with `status='generated'` + - Pending: Images with `status='pending'` + +**Card Layout:** +- Width: Equal distribution across row +- Display: Icon, Title, Total/Processed/Pending counts +- Color: Match stage colors for visual consistency + +--- + +### Task 3.4: Pipeline Status Card Redesign + +**Current:** Wide row with text "Pipeline Status - Ready to run | 22 items pending" + +**New Design Requirements:** +1. **Convert to Centered Card:** + - Position: Above stage cards, below metrics summary + - Width: Narrower than full width, centered + - Style: Elevated/shadowed for emphasis + +2. **Content Structure:** + - Large status indicator (icon + text) + - Prominent pending items count + - Quick action buttons (Run Now, Pause, Configure) + +3. **Status Visual States:** + - Ready: Green pulse animation + - Running: Blue animated progress + - Paused: Yellow warning icon + - Failed: Red alert icon + +--- + +### Task 3.5: Remove/Compact Header Elements + +**Changes to Automation Page Header:** + +1. **Remove "Pipeline Overview" Section:** + - Delete heading: "📊 Pipeline Overview" + - Delete subtitle: "Complete view of automation pipeline status and pending items" + - Reason: Redundant with visible pipeline cards + +2. **Compact Schedule Panel:** + - Current: Large panel with heading, status row, action buttons + - New: Single compact row + - Layout: `[Status Badge] | [Schedule Text] | [Last Run] | [Estimated Credits] | [Configure Button] | [Run Now Button]` + - Remove empty space and excessive padding + +--- + +### Task 3.6: AI Request Delays Implementation + +**Problem:** Rapid sequential AI requests may hit rate limits or overload AI service. + +**Required Changes:** + +1. **Within-Stage Delay (between batches):** + - Location: `AutomationService` class methods + - Add delay after each batch completion before processing next batch + - Configurable: 3-5 seconds (default 3 seconds) + - Implementation point: After each AI function call completes in stage loop + +2. **Between-Stage Delay:** + - Add delay after stage completion before triggering next stage + - Configurable: 5-10 seconds (default 5 seconds) + - Implementation point: After `_execute_stage()` returns before incrementing `current_stage` + +3. **Configuration:** + - Add to `AutomationConfig` model: `within_stage_delay` (integer, seconds) + - Add to `AutomationConfig` model: `between_stage_delay` (integer, seconds) + - Expose in Configure modal for user adjustment + +4. **Logging:** + - Log delay start: "Waiting 3 seconds before next batch..." + - Log delay end: "Delay complete, resuming processing" + +--- + +## SECTION 4: AUTOMATION STAGE PROCESSING FIXES + +### Task 4.1: Verify Stage Sequential Processing Logic + +**Problem:** Pipeline not following strict sequential stage completion before moving to next stage. + +**Analysis Required:** +1. Review `AutomationService.start_automation()` method +2. Verify stage loop waits for 100% completion before `current_stage += 1` +3. Check for any parallel execution logic that bypasses sequential flow + +**Verification Steps:** +1. Each stage method (`run_stage_1()` to `run_stage_7()`) must return ONLY after ALL batches processed +2. Stage N+1 should NOT start if Stage N has `pending > 0` +3. Add explicit completion check before stage transition + +**Required Fixes:** +- Add validation: Before starting Stage N, verify Stage N-1 has 0 pending items +- If pending items found, log warning and halt automation +- Return error status with message: "Stage N-1 incomplete, cannot proceed to Stage N" + +--- + +### Task 4.2: Fix Batch Size Configuration Reading + +**Problem:** Manual "Run Now" only processes 5 keywords instead of respecting configured batch size (20). + +**Root Cause Analysis:** +1. Check if `run_stage_1()` reads from `AutomationConfig.stage_1_batch_size` +2. Verify query limit: `Keywords.objects.filter(...)[:batch_size]` uses correct variable +3. Confirm configuration loaded at automation start: `config = AutomationConfig.objects.get(site=self.site)` + +**Expected Behavior:** +- If queue has 7 keywords and batch_size = 20: Process all 7 (not limit to 5) +- If queue has 47 keywords and batch_size = 20: Process 20, then next batch of 20, then final 7 +- Batch size should be dynamic based on queue size: `min(queue_count, batch_size)` + +**Fix Implementation:** +1. Ensure configuration loaded once at automation start +2. Pass batch_size to each stage method +3. Update query to use: `[:min(pending_count, batch_size)]` +4. Log batch selection: "Processing batch 1/3: 20 keywords" + +--- + +### Task 4.3: Fix Stage 4 Processing Not Completing Full Queue + +**Problem:** Stage 4 (Tasks → Content) not processing all tasks before moving to Stage 5. + +**Investigation Steps:** +1. Check `run_stage_4()` implementation in `AutomationService` +2. Verify loop structure: Does it process tasks one-by-one until queue empty? +3. Look for premature loop exit conditions + +**Expected Logic:** +``` +While tasks with status='pending' exist: + 1. Get next task + 2. Call generate_content AI function + 3. Wait for completion + 4. Verify Content created + 5. Check if more pending tasks exist + 6. If yes, continue loop + 7. If no, return stage complete +``` + +**Common Issues to Check:** +- Loop exits after first task instead of continuing +- No loop at all - only processes one batch +- Error handling breaks loop prematurely + +--- + +### Task 4.4: Fix Stage 5 Not Triggering (Image Prompts Generation) + +**Problem:** Automation exits after Stage 4 without generating image prompts. + +**Analysis Required:** +1. Verify Stage 4 completion status set correctly +2. Check if Stage 5 start condition is met +3. Review database query in `run_stage_5()`: + - Query: Content with `status='draft'` AND `images_count=0` + - Verify Content records created in Stage 4 have correct status + +**Potential Issues:** +1. Content created with status other than 'draft' +2. Images count annotation incorrect (should use `annotate(images_count=Count('images'))`) +3. Stage handover logic doesn't trigger Stage 5 + +**Fix Steps:** +1. Verify Content model save in Stage 4 sets `status='draft'` +2. Ensure Stage 5 query matches Content records from Stage 4 +3. Add logging: "Stage 5: Found X content pieces without images" +4. If X > 0, process; if X = 0, skip stage gracefully + +--- + +### Task 4.5: Add Stage Handover Validation + +**New Logic Required Between Each Stage:** + +1. **Pre-Stage Validation:** + - Before starting Stage N, check Stage N-1 completion: + - Query pending items for Stage N-1 + - If pending > 0: Log error, halt automation + - If pending = 0: Log success, proceed + +2. **Post-Stage Validation:** + - After Stage N completes, verify: + - All input items processed + - Expected output items created + - No errors in stage result + - Log validation result before moving to Stage N+1 + +3. **Validation Logging:** + - Stage 1 → Stage 2: "Verified: 0 keywords pending, 8 clusters created" + - Stage 2 → Stage 3: "Verified: 0 clusters pending, 56 ideas created" + - Stage 3 → Stage 4: "Verified: 0 ideas pending, 56 tasks created" + - Stage 4 → Stage 5: "Verified: 0 tasks pending, 56 content pieces created" + - Stage 5 → Stage 6: "Verified: 0 content without images, 224 prompts created" + - Stage 6 → Stage 7: "Verified: 0 pending images, 224 images generated" + +--- + +### Task 4.6: Implement Dynamic Batch Size Logic + +**Problem:** Fixed batch sizes don't adapt to actual queue sizes. + +**Required Smart Batch Logic:** + +1. **For Stages 1, 2, 3, 5:** + - If `queue_count <= batch_size`: Process ALL items in one batch + - If `queue_count > batch_size`: Split into batches + +2. **For Stage 4 (Tasks → Content):** + - Always process one task at a time (sequential) + - Reason: Content generation is expensive, better control + - Batch size config for Stage 4 can be deprecated + +3. **For Stage 6 (Images):** + - Process one image at a time (current behavior) + - Reason: Image generation has rate limits + +**Configuration Update:** +- Stage 1-3, 5: Batch size applies +- Stage 4, 6: Batch size ignored (always 1) +- Update Configure modal to clarify batch size usage per stage + +--- + +## SECTION 5: STAGE CARD LAYOUT RESTRUCTURE + +### Task 5.1: Add Missing Stage 5 Card (Content → Image Prompts) + +**Problem:** Current UI combines Stages 3 & 4 into one card, Stage 5 missing. + +**Required Change:** +- Create separate card for Stage 5 +- Display: "Content → Image Prompts" +- Queue metrics: Content without images (not total content) +- Show progress bar for prompt extraction + +--- + +### Task 5.2: Separate Stages 3 & 4 into Individual Cards + +**Current:** One card shows "Ideas → Tasks → Content" with nested metrics. + +**New Structure:** +1. **Stage 3 Card:** "Ideas → Tasks" + - Total Queue: Ideas with `status='new'` + - Processed: Ideas converted to tasks + - Progress: Task creation count + +2. **Stage 4 Card:** "Tasks → Content" + - Total Queue: Tasks with `status='pending'` + - Processed: Tasks with `status='completed'` + - Progress: Content generation count + +--- + +### Task 5.3: Restructure Stage Card Rows + +**New Layout Requirements:** + +**Row 1 (Stages 1-4):** +- Stage 1: Keywords → Clusters +- Stage 2: Clusters → Ideas +- Stage 3: Ideas → Tasks +- Stage 4: Tasks → Content + +**Row 2 (Stages 5-8):** +- Stage 5: Content → Image Prompts +- Stage 6: Image Prompts → Images +- Stage 7: Review Gate (with action buttons) +- Stage 8: Status Summary (new informational card) + +**Responsive Behavior:** +- Desktop: 4 cards per row +- Tablet: 2 cards per row +- Mobile: 1 card per row (vertical stack) + +--- + +### Task 5.4: Design Stage 7 Card (Review Gate) + +**Unique Requirements:** +1. **Visual Distinction:** + - Different color scheme (amber/orange warning color) + - Icon: Stop sign or review icon + - Border: Dashed or highlighted + +2. **Content:** + - Title: "Manual Review Gate" + - Status: "Automation Stops Here" + - Count: Number of content pieces ready for review + - Two buttons: + - "Go to Review Page" (navigates to Writer Content page filtered by status='review') + - "Publish Without Review" (disabled initially, placeholder for future feature) + +3. **Button Behavior:** + - Review button: Active when count > 0 + - Publish button: Disabled with tooltip "Coming soon" + +--- + +### Task 5.5: Design Stage 8 Card (Status Summary) + +**New Informational Card:** + +**Purpose:** Display current automation run status without queue processing. + +**Content:** +1. **Title:** "Current Status" +2. **Large Status Icon:** Based on run status + - Running: Animated spinner + - Completed: Checkmark + - Failed: X icon + - Paused: Pause icon + +3. **Metrics Display:** + - Run ID + - Started at timestamp + - Current stage indicator + - Total credits used + - Completion percentage + +4. **Visual Style:** + - No queue metrics + - No action buttons + - Read-only information display + - Distinct styling (different background color, no hover effects) + +--- + +### Task 5.6: Adjust Card Width for New Layout + +**Current:** Stage cards likely using equal width across full viewport. + +**New Requirements:** +- Row 1 (4 cards): Each card 23% width (with 2% gap) +- Row 2 (4 cards): Same width distribution +- Stage 8 card: Can be wider or styled differently as summary card + +**Implementation Considerations:** +- Use CSS Grid or Flexbox for responsive layout +- Ensure consistent spacing between cards +- Maintain card aspect ratio for visual balance + +--- + +## SECTION 6: ADDITIONAL ENHANCEMENTS + +### Task 6.1: Add Credit Usage Tracking per Stage + +**Value Addition:** Real-time visibility into credit consumption. + +**Implementation:** +1. Track credits used at end of each stage in `stage_N_result` JSON field +2. Display in stage card: "Credits Used: X" +3. Add running total in overall pipeline progress bar +4. Compare estimated vs actual credits used +5. Alert if actual exceeds estimated by >20% + +--- + +### Task 6.2: Add Estimated Completion Time per Stage + +**Value Addition:** Predictable automation runtime for planning. + +**Implementation:** +1. Calculate average time per item based on historical runs +2. Estimate: `Remaining Items * Average Time per Item` +3. Display in stage card: "ETA: 45 minutes" +4. Update dynamically as items process +5. Store metrics in database for accuracy improvement over time + +--- + + +### Task 6.3: Add Error Rate Monitoring + +**Value Addition:** Proactive issue detection. + +**Implementation:** +1. Track error count per stage +2. Display: "Errors: X (Y%)" +3. Highlight stages with >5% error rate +4. Add "View Errors" button to navigate to error log +5. Set up alerts for error rate spikes + +--- + +### Task 6.4: Add Stage Completion Percentage + +**Value Addition:** Clear progress visualization. + +**Implementation:** +1. Calculate: `(Completed Items / Total Items) * 100` +2. Display as progress bar in stage card +3. Color code: + - Green: >75% + - Yellow: 25-75% + - Red: <25% +4. Animate progress bar during active stages +5. Show exact percentage in text format + +--- + +### Task 6.5: Add Stage Start/End Timestamps + +**Value Addition:** Audit trail for automation runs. + +**Implementation:** +1. Store start/end timestamps in `stage_N_result` +2. Display in stage card: "Started: 10:30 AM | Ended: 11:15 AM" +3