igny8/docs/PRE-LAUNCH/ITEM-1-STATUS-MODULES.md

Item 1: Status Modules for AI Planner and Writer

Priority: Critical
Target: Production Launch
Last Updated: December 11, 2025

---

Overview

Standardize and fix status tracking across Planner and Writer modules to ensure consistent, accurate, and real-time progress updates for all AI operations. This includes backend status management, frontend progress displays, WebSocket/polling systems, and error handling.

---

Current Implementation Analysis

Backend Status System

Status Fields in Models

Location: backend/igny8_core/business/planning/models.py and backend/igny8_core/business/content/models.py

Model	Current Status Choices	Issues
Keywords	new, mapped	Limited states, no processing states
Clusters	new, mapped	No active/processing states
ContentIdeas	new, queued, completed	Missing processing, failed states
Tasks	queued, completed	Missing intermediate states (processing, failed, cancelled)
Content	draft, review, published	Separate from generation status
AutomationRun	running, paused, cancelled, completed, failed	Well-defined but isolated

Key Problem: Status choices are inconsistent across models. Some models track generation status, others track editorial status, causing confusion.

---

AI Function Progress Tracking

Location: backend/igny8_core/ai/tracker.py, backend/igny8_core/ai/engine.py

Progress Phases

All AI functions emit standard progress phases:

Phase	Default Label	Percentage Range	Purpose
INIT	Initializing...	0-10%	Validation, setup
PREP	Preparing data...	10-20%	Data loading, preparation
AI_CALL	Processing with AI...	20-80%	AI model execution
PARSE	Parsing response...	80-90%	Response validation, parsing
SAVE	Saving results...	90-100%	Database operations
DONE	Complete!	100%	Final state

Implementation:

• StepTracker class emits step-level events
• ProgressTracker class manages Celery task state updates
• AIEngine orchestrates phase transitions

Issues:

• Phase messages are generic and don't reflect actual counts
• Percentage calculations sometimes inconsistent
• No clear error state handling in phases

---

Frontend Progress Modal System

Location: frontend/src/hooks/useProgressModal.ts, frontend/src/components/common/ProgressModal.tsx

Current Modal Implementation

Hook: useProgressModal

• Manages modal state, task polling, progress updates
• Polls backend every 1 second for task status
• Maps backend phases to user-friendly messages
• Tracks step logs for debugging

Modal Component: ProgressModal

• Displays progress bar with percentage
• Shows current phase and message
• Lists step-by-step progress visually
• Provides cancel/close actions

Step Mapping Logic

Function: getStepInfo() in useProgressModal.ts

Maps backend phases to:

• User-friendly messages with dynamic counts
• Adjusted percentages for smooth progression
• Function-specific labels (clustering vs. ideas vs. content)

Problems Identified:

1. Inaccurate counts in messages: Extracts counts from messages using regex, often missing or incorrect
2. Generic phase labels: Messages don't always reflect what's actually happening
3. Inconsistent percentage progression: Jumps or stalls during AI_CALL phase
4. Status text not aligned: Labels in modal don't match backend phase names

---

Progress Display by Page

Page	Location	Progress Display	Issues
Keywords	frontend/src/pages/Planner/Keywords.tsx	ProgressModal, AI logs	Clustering progress text inaccurate
Clusters	frontend/src/pages/Planner/Clusters.tsx	ProgressModal only	No cluster generation function exists
Ideas	frontend/src/pages/Planner/Ideas.tsx	ProgressModal, reload on complete	Text shows generic "preparing clusters"
Tasks	frontend/src/pages/Writer/Tasks.tsx	ProgressModal, AI logs	Content generation text inaccurate
Content	frontend/src/pages/Writer/Content.tsx	ProgressModal	Missing real-time updates
Images	frontend/src/pages/Writer/Images.tsx	ImageQueueModal	Uses separate queue-based system

---

Known Bugs and Issues

From Pre-Launch-Pending.md Known Bugs section:

Manual AI Function Run Modals

Affected Pages: Keywords, Clusters, Ideas, Tasks

Issues:

• Progress modal texts are not accurate for specific functions
• Dynamic variables not properly populated with counts
• Generic messages instead of function-specific ones

Example:

• Clustering: Should say "Mapping 45 keywords into clusters" but shows "Preparing data..."
• Ideas: Should say "Generating ideas for 3 clusters" but shows generic text

---

Automation Wizard Progress

Page: Automation (frontend/src/pages/Automation/AutomationWizard.tsx)

Issues:

• Wrong queue items displayed
• Missing queue items for certain stages
• Progress bar doesn't progress properly
• Total in queue and processed counts buggy across stages
• Stage cards: Real-time metrics not optimized

---

WebSocket vs Polling

Current Implementation: Polling-based

Location: useProgressModal.ts - polls task status every 1 second

Note: Code mentions WebSockets but current implementation uses REST API polling via /api/v1/... endpoints

Improvement Needed: Consider WebSocket implementation for real-time updates without polling overhead

---

Required Changes

A. Backend Status Standardization

1. Unified Status Enum

Create: backend/igny8_core/common/status_enums.py

Define standard status choices for all modules:

GENERATION_STATUS_CHOICES = [
    ('pending', 'Pending'),
    ('queued', 'Queued'),
    ('processing', 'Processing'),
    ('completed', 'Completed'),
    ('failed', 'Failed'),
    ('cancelled', 'Cancelled'),
]

CONTENT_STATUS_CHOICES = [
    ('draft', 'Draft'),
    ('review', 'Review'),
    ('published', 'Published'),
    ('archived', 'Archived'),
]

Action: Add generation_status field to models that undergo AI processing:

• Keywords (for extraction/clustering operations)
• Clusters (for cluster generation, if implemented)
• ContentIdeas (for idea generation)
• Tasks (for content generation)
• Content (separate from editorial status)

2. Enhanced Progress Details

Update: backend/igny8_core/ai/tracker.py

Modifications:

• StepTracker.update_step() should include:
  • items_total: Total count of items being processed
  • items_processed: Current count of items processed
  • current_item_name: Name/title of current item
  • estimated_remaining_seconds: Estimated time remaining

Example Progress Payload:

{
  "phase": "AI_CALL",
  "message": "Clustering 45 keywords",
  "percentage": 45,
  "items_total": 45,
  "items_processed": 20,
  "current_item": "best seo tools",
  "estimated_seconds_remaining": 120
}

3. Function-Specific Progress Messages

Update: Each AI function's phase messages

Files to modify:

• backend/igny8_core/ai/functions/auto_cluster.py
• backend/igny8_core/ai/functions/generate_ideas.py
• backend/igny8_core/ai/functions/generate_content.py
• backend/igny8_core/ai/functions/generate_image_prompts.py
• backend/igny8_core/ai/functions/generate_images.py

Changes per function:

Function	Phase	Accurate Message Template
auto_cluster	PREP	"Loading {count} keywords for clustering"
auto_cluster	AI_CALL	"Analyzing keyword relationships ({processed}/{total})"
auto_cluster	SAVE	"Creating {cluster_count} clusters"
generate_ideas	PREP	"Preparing {count} cluster(s) for idea generation"
generate_ideas	AI_CALL	"Generating ideas for cluster: {cluster_name}"
generate_ideas	SAVE	"Created {idea_count} content ideas"
generate_content	PREP	"Preparing task: {task_title}"
generate_content	AI_CALL	"Writing {word_count}-word article"
generate_content	SAVE	"Saving article: {title}"
generate_image_prompts	PREP	"Mapping content for {count} image prompts"
generate_image_prompts	AI_CALL	"Writing featured image prompt"
generate_image_prompts	PARSE	"Writing {count} in-article image prompts"
generate_images	PREP	"Preparing {count} images for generation"
generate_images	AI_CALL	"Generating image {current}/{total}: {prompt}"

Implementation: Use self.tracker.update_step() with dynamic values from actual data

---

B. Frontend Progress Display Improvements

1. Update ProgressModal Messages

File: frontend/src/components/common/ProgressModal.tsx

Function: getSuccessMessage()

Current Issues:

• Generic success messages
• Count extraction from logs is unreliable
• Doesn't distinguish between function types clearly

Required Changes:

Function	Current Success Message	Improved Success Message
Clustering	"Clustering complete"	"45 keywords mapped into 8 clusters"
Ideas	"Content ideas created successfully"	"Generated 12 content ideas for 3 clusters"
Content	"Article drafted successfully"	"1,245-word article drafted successfully"
Image Prompts	Generic text	"1 Featured Image + 4 In-article Image Prompts ready"
Images	"Images generated successfully"	"5 images generated successfully"

Implementation: Extract counts from backend details field, not from message strings

---

2. Improve Step Mapping Logic

File: frontend/src/hooks/useProgressModal.ts

Function: getStepInfo()

Problems:

• Complex regex-based count extraction from messages
• Percentage calculation doesn't match backend
• Messages manually constructed instead of using backend messages

Solution:

• Rely on backend details field for counts:

  const { items_total, items_processed, current_item } = progress.details;
  

• Use backend message directly when it contains proper counts
• Only enhance message format, don't reconstruct it

Example Refactor:

// OLD: Complex regex extraction
const keywordCount = extractNumber(/(\d+)\s+keyword/i, message);

// NEW: Use backend details
const percentage = details.percentage;
const message = details.message; // Already has counts
const itemName = details.current_item || '';

---

3. Real-Time Updates Per Page

Pages to update: Keywords, Ideas, Tasks, Content

Current behavior: Modal closes, then data reloads

Improved behavior:

• Show inline status badge in table rows during processing
• Update table row in real-time as status changes
• Auto-reload specific rows instead of entire table
• Show notification when task completes in background

Implementation approach:

• Subscribe to task updates while modal is open
• Update table state with new status
• Add status badge column to tables showing:
  • 🔄 Processing (animated)
  • ✅ Completed
  • ❌ Failed
  • ⏸️ Paused

---

C. Automation Wizard Progress Fixes

File: frontend/src/pages/Automation/AutomationWizard.tsx

Issues identified:

1. Wrong queue items displayed
2. Missing queue items
3. Progress bar inaccurate
4. Stage counts buggy

Required fixes:

1. Stage Progress Cards

Current location: Stage cards display in wizard

Problems:

• Counts mismatch between backend and frontend
• "In Queue" vs "Processed" numbers don't add up
• Real-time updates missing

Fix:

• Pull stage results from AutomationRun.stage_X_result JSON fields
• Display accurate counts:

  Stage 1: Clustering
  - Keywords Loaded: 45
  - Clusters Created: 8
  - Status: Completed
  

2. Queue Item Display

Fix queue item tracking:

• Each stage should show items being processed
• Items should appear in queue before processing
• Items should move to "Completed" list after processing
• Failed items should show in "Failed" section with retry option

Data structure per stage:

{
  "pending": ["item1", "item2"],
  "processing": ["item3"],
  "completed": ["item4", "item5"],
  "failed": [{"item": "item6", "error": "..."}]
}

3. Progress Bar Calculation

Fix:

• Base percentage on actual completed stages
• Within-stage percentage based on items processed
• Formula: (completed_stages * 100 / 7) + (current_stage_progress / 7)

---

D. Error State Handling

Backend Error Logging

Location: backend/igny8_core/ai/engine.py

Current: Errors logged but status not always propagated

Improvement:

• Always set task state to FAILURE on error
• Include detailed error in state metadata
• Log error with context (function, input data, trace)

Error payload structure:

{
  "status": "error",
  "error_type": "InsufficientCreditsError",
  "error_message": "Insufficient credits. Required: 45, Available: 20",
  "phase": "INIT",
  "timestamp": "2025-12-11T10:30:00Z"
}

Frontend Error Display

Files: All progress modals

Current: Generic "Task failed" message

Improvement:

• Show specific error type
• Display user-friendly error message
• Provide actionable next steps
• Show "Retry" button for recoverable errors

Error message mapping:

Error Type	User Message	Action
InsufficientCreditsError	"Not enough credits. You need X credits."	"Purchase Credits" button
ValidationError	"Invalid data: {details}"	"Fix Data" button
AIProviderError	"AI service temporarily unavailable"	"Retry" button
TimeoutError	"Request timed out. Try again."	"Retry" button

---

QA Testing Checklist

Manual AI Function Tests

Test each function on each page:

Page	Function	Test Case	Expected Result
Keywords	Clustering	Select 50 keywords, run clustering	Modal shows "Clustering 50 keywords", creates clusters, shows "50 keywords mapped into X clusters"
Ideas	Generate Ideas	Select 3 clusters, generate ideas	Modal shows "Generating ideas for 3 clusters", creates ideas, shows "Generated X ideas for 3 clusters"
Tasks	Generate Content	Select 1 task, generate content	Modal shows "Writing {word_count}-word article", creates content, shows "{word_count}-word article drafted"
Tasks	Generate Image Prompts	Select 1 content, generate prompts	Modal shows "Mapping content for X prompts", shows "1 Featured + X In-article prompts ready"
Images	Generate Images	Select content with prompts, generate	ImageQueueModal shows each image generating, completes all

Edge Case Tests

Scenario	Expected Behavior
Task cancelled mid-process	Status updates to "cancelled", modal shows cancellation message
Task fails due to insufficient credits	Error modal with credit purchase link
Task fails due to validation error	Error modal with specific validation issue
Multiple tasks running simultaneously	Each has independent progress tracking
Network disconnection during task	Polling resumes when reconnected, catches up with status
Task completes while modal is closed	Table updates status automatically, notification shown

Automation Wizard Tests

Stage	Test Case	Expected Result
Stage 1	Run clustering on 100 keywords	Shows batch progress, creates clusters, displays accurate counts
Stage 2	Generate ideas for 10 clusters	Shows cluster-by-cluster progress, creates ideas
Stage 3	Convert ideas to tasks	Shows conversion progress, creates tasks
Stage 4	Generate content for tasks	Shows content generation per task
Stage 5	Generate image prompts	Shows prompt creation progress
Stage 6	Generate images	Shows image generation progress
Stage 7	Final review	Shows completed content ready for publishing
Pause/Resume	Pause at stage 3, resume	Continues from where it left off
Cancel	Cancel at stage 4	Stops processing, shows cancelled status

---

Implementation Priority

Phase 1: Backend Fixes (Week 1)

1. ✅ Create unified status enums
2. ✅ Add generation_status fields to models
3. ✅ Update AI function progress messages with accurate counts
4. ✅ Enhance tracker to include item counts and current item

Phase 2: Frontend Display (Week 1-2)

1. ✅ Update ProgressModal success messages
2. ✅ Refactor step mapping logic to use backend details
3. ✅ Add real-time status badges to table rows
4. ✅ Improve error message display

Phase 3: Automation Wizard (Week 2)

1. ✅ Fix stage progress cards
2. ✅ Fix queue item tracking
3. ✅ Fix progress bar calculation
4. ✅ Add pause/resume/cancel controls

Phase 4: QA & Polish (Week 2-3)

1. ✅ Run all manual tests
2. ✅ Run all edge case tests
3. ✅ Run automation wizard tests
4. ✅ Fix any discovered issues
5. ✅ Performance optimization

---

Success Metrics

• ✅ All progress modals show accurate counts and messages
• ✅ Progress percentages match actual work completed
• ✅ Automation wizard shows correct queue states
• ✅ Error messages are specific and actionable
• ✅ Real-time updates work consistently
• ✅ No user-reported confusion about task status
• ✅ All QA test cases pass

---

Related Files Reference

Backend

• backend/igny8_core/business/planning/models.py - Keywords, Clusters, ContentIdeas models
• backend/igny8_core/business/content/models.py - Tasks, Content models
• backend/igny8_core/business/automation/models.py - AutomationRun model
• backend/igny8_core/ai/tracker.py - ProgressTracker, StepTracker
• backend/igny8_core/ai/engine.py - AIEngine orchestration
• backend/igny8_core/ai/functions/*.py - All AI function implementations

Frontend

• frontend/src/hooks/useProgressModal.ts - Progress modal hook
• frontend/src/components/common/ProgressModal.tsx - Progress modal component
• frontend/src/components/common/ImageQueueModal.tsx - Image generation modal
• frontend/src/pages/Planner/Keywords.tsx - Keywords page with clustering
• frontend/src/pages/Planner/Ideas.tsx - Ideas page with idea generation
• frontend/src/pages/Writer/Tasks.tsx - Tasks page with content generation
• frontend/src/pages/Writer/Content.tsx - Content page
• frontend/src/pages/Writer/Images.tsx - Images page
• frontend/src/pages/Automation/AutomationWizard.tsx - Automation wizard

---

Notes

• Consider WebSocket implementation for production to reduce polling overhead
• Status badges in tables should be subtle and not distract from main content
• Error messages should prioritize user understanding over technical accuracy
• Progress messages should be encouraging and informative
• Real-time updates should not cause UI flickering or performance issues