igny8/docs/PRE-LAUNCH/ITEM-1-STATUS-MODULES.md

# Item 1: Status Modules for AI Planner and Writer

**Priority:** Critical  
**Target:** Production Launch  
**Last Updated:** December 11, 2025

---

## Overview

Standardize and fix status tracking across Planner and Writer modules to ensure consistent, accurate, and real-time progress updates for all AI operations. This includes backend status management, frontend progress displays, WebSocket/polling systems, and error handling.

---

## Current Implementation Analysis

### Backend Status System

#### Status Fields in Models

**Location:** `backend/igny8_core/business/planning/models.py` and `backend/igny8_core/business/content/models.py`

| Model | Current Status Choices | Issues |
|-------|----------------------|---------|
| **Keywords** | `new`, `mapped` | Limited states, no processing states |
| **Clusters** | `new`, `mapped` | No active/processing states |
| **ContentIdeas** | `new`, `queued`, `completed` | Missing `processing`, `failed` states |
| **Tasks** | `queued`, `completed` | Missing intermediate states (processing, failed, cancelled) |
| **Content** | `draft`, `review`, `published` | Separate from generation status |
| **AutomationRun** | `running`, `paused`, `cancelled`, `completed`, `failed` | Well-defined but isolated |

**Key Problem:** Status choices are inconsistent across models. Some models track generation status, others track editorial status, causing confusion.

---

### AI Function Progress Tracking

**Location:** `backend/igny8_core/ai/tracker.py`, `backend/igny8_core/ai/engine.py`

#### Progress Phases

All AI functions emit standard progress phases:

| Phase | Default Label | Percentage Range | Purpose |
|-------|--------------|------------------|---------|
| `INIT` | Initializing... | 0-10% | Validation, setup |
| `PREP` | Preparing data... | 10-20% | Data loading, preparation |
| `AI_CALL` | Processing with AI... | 20-80% | AI model execution |
| `PARSE` | Parsing response... | 80-90% | Response validation, parsing |
| `SAVE` | Saving results... | 90-100% | Database operations |
| `DONE` | Complete! | 100% | Final state |

**Implementation:** 
- `StepTracker` class emits step-level events
- `ProgressTracker` class manages Celery task state updates
- `AIEngine` orchestrates phase transitions

**Issues:**
- Phase messages are generic and don't reflect actual counts
- Percentage calculations sometimes inconsistent
- No clear error state handling in phases

---

### Frontend Progress Modal System

**Location:** `frontend/src/hooks/useProgressModal.ts`, `frontend/src/components/common/ProgressModal.tsx`

#### Current Modal Implementation

**Hook:** `useProgressModal`
- Manages modal state, task polling, progress updates
- Polls backend every 1 second for task status
- Maps backend phases to user-friendly messages
- Tracks step logs for debugging

**Modal Component:** `ProgressModal`
- Displays progress bar with percentage
- Shows current phase and message
- Lists step-by-step progress visually
- Provides cancel/close actions

#### Step Mapping Logic

**Function:** `getStepInfo()` in `useProgressModal.ts`

Maps backend phases to:
- User-friendly messages with dynamic counts
- Adjusted percentages for smooth progression
- Function-specific labels (clustering vs. ideas vs. content)

**Problems Identified:**
1. **Inaccurate counts in messages**: Extracts counts from messages using regex, often missing or incorrect
2. **Generic phase labels**: Messages don't always reflect what's actually happening
3. **Inconsistent percentage progression**: Jumps or stalls during AI_CALL phase
4. **Status text not aligned**: Labels in modal don't match backend phase names

---

### Progress Display by Page

| Page | Location | Progress Display | Issues |
|------|----------|------------------|--------|
| Keywords | `frontend/src/pages/Planner/Keywords.tsx` | ProgressModal, AI logs | Clustering progress text inaccurate |
| Clusters | `frontend/src/pages/Planner/Clusters.tsx` | ProgressModal only | No cluster generation function exists |
| Ideas | `frontend/src/pages/Planner/Ideas.tsx` | ProgressModal, reload on complete | Text shows generic "preparing clusters" |
| Tasks | `frontend/src/pages/Writer/Tasks.tsx` | ProgressModal, AI logs | Content generation text inaccurate |
| Content | `frontend/src/pages/Writer/Content.tsx` | ProgressModal | Missing real-time updates |
| Images | `frontend/src/pages/Writer/Images.tsx` | ImageQueueModal | Uses separate queue-based system |

---

### Known Bugs and Issues

From `Pre-Launch-Pending.md` Known Bugs section:

#### Manual AI Function Run Modals

**Affected Pages:** Keywords, Clusters, Ideas, Tasks

**Issues:**
- Progress modal texts are not accurate for specific functions
- Dynamic variables not properly populated with counts
- Generic messages instead of function-specific ones

**Example:**
- Clustering: Should say "Mapping 45 keywords into clusters" but shows "Preparing data..."
- Ideas: Should say "Generating ideas for 3 clusters" but shows generic text

---

#### Automation Wizard Progress

**Page:** Automation (`frontend/src/pages/Automation/AutomationWizard.tsx`)

**Issues:**
- Wrong queue items displayed
- Missing queue items for certain stages
- Progress bar doesn't progress properly
- Total in queue and processed counts buggy across stages
- Stage cards: Real-time metrics not optimized

---

### WebSocket vs Polling

**Current Implementation:** Polling-based

**Location:** `useProgressModal.ts` - polls task status every 1 second

**Note:** Code mentions WebSockets but current implementation uses REST API polling via `/api/v1/...` endpoints

**Improvement Needed:** Consider WebSocket implementation for real-time updates without polling overhead

---

## Required Changes

### A. Backend Status Standardization

#### 1. Unified Status Enum

**Create:** `backend/igny8_core/common/status_enums.py`

Define standard status choices for all modules:

```
GENERATION_STATUS_CHOICES = [
    ('pending', 'Pending'),
    ('queued', 'Queued'),
    ('processing', 'Processing'),
    ('completed', 'Completed'),
    ('failed', 'Failed'),
    ('cancelled', 'Cancelled'),
]

CONTENT_STATUS_CHOICES = [
    ('draft', 'Draft'),
    ('review', 'Review'),
    ('published', 'Published'),
    ('archived', 'Archived'),
]
```

**Action:** Add `generation_status` field to models that undergo AI processing:
- Keywords (for extraction/clustering operations)
- Clusters (for cluster generation, if implemented)
- ContentIdeas (for idea generation)
- Tasks (for content generation)
- Content (separate from editorial status)

#### 2. Enhanced Progress Details

**Update:** `backend/igny8_core/ai/tracker.py`

**Modifications:**
- `StepTracker.update_step()` should include:
  - `items_total`: Total count of items being processed
  - `items_processed`: Current count of items processed
  - `current_item_name`: Name/title of current item
  - `estimated_remaining_seconds`: Estimated time remaining

**Example Progress Payload:**
```json
{
  "phase": "AI_CALL",
  "message": "Clustering 45 keywords",
  "percentage": 45,
  "items_total": 45,
  "items_processed": 20,
  "current_item": "best seo tools",
  "estimated_seconds_remaining": 120
}
```

#### 3. Function-Specific Progress Messages

**Update:** Each AI function's phase messages

**Files to modify:**
- `backend/igny8_core/ai/functions/auto_cluster.py`
- `backend/igny8_core/ai/functions/generate_ideas.py`
- `backend/igny8_core/ai/functions/generate_content.py`
- `backend/igny8_core/ai/functions/generate_image_prompts.py`
- `backend/igny8_core/ai/functions/generate_images.py`

**Changes per function:**

| Function | Phase | Accurate Message Template |
|----------|-------|---------------------------|
| auto_cluster | PREP | "Loading {count} keywords for clustering" |
| auto_cluster | AI_CALL | "Analyzing keyword relationships ({processed}/{total})" |
| auto_cluster | SAVE | "Creating {cluster_count} clusters" |
| generate_ideas | PREP | "Preparing {count} cluster(s) for idea generation" |
| generate_ideas | AI_CALL | "Generating ideas for cluster: {cluster_name}" |
| generate_ideas | SAVE | "Created {idea_count} content ideas" |
| generate_content | PREP | "Preparing task: {task_title}" |
| generate_content | AI_CALL | "Writing {word_count}-word article" |
| generate_content | SAVE | "Saving article: {title}" |
| generate_image_prompts | PREP | "Mapping content for {count} image prompts" |
| generate_image_prompts | AI_CALL | "Writing featured image prompt" |
| generate_image_prompts | PARSE | "Writing {count} in-article image prompts" |
| generate_images | PREP | "Preparing {count} images for generation" |
| generate_images | AI_CALL | "Generating image {current}/{total}: {prompt}" |

**Implementation:** Use `self.tracker.update_step()` with dynamic values from actual data

---

### B. Frontend Progress Display Improvements

#### 1. Update ProgressModal Messages

**File:** `frontend/src/components/common/ProgressModal.tsx`

**Function:** `getSuccessMessage()`

**Current Issues:**
- Generic success messages
- Count extraction from logs is unreliable
- Doesn't distinguish between function types clearly

**Required Changes:**
| Function | Current Success Message | Improved Success Message |
|----------|------------------------|--------------------------|
| Clustering | "Clustering complete" | "45 keywords mapped into 8 clusters" |
| Ideas | "Content ideas created successfully" | "Generated 12 content ideas for 3 clusters" |
| Content | "Article drafted successfully" | "1,245-word article drafted successfully" |
| Image Prompts | Generic text | "1 Featured Image + 4 In-article Image Prompts ready" |
| Images | "Images generated successfully" | "5 images generated successfully" |

**Implementation:** Extract counts from backend `details` field, not from message strings

---

#### 2. Improve Step Mapping Logic

**File:** `frontend/src/hooks/useProgressModal.ts`

**Function:** `getStepInfo()`

**Problems:**
- Complex regex-based count extraction from messages
- Percentage calculation doesn't match backend
- Messages manually constructed instead of using backend messages

**Solution:**
- **Rely on backend `details` field** for counts:
  ```typescript
  const { items_total, items_processed, current_item } = progress.details;
  ```
- Use backend message directly when it contains proper counts
- Only enhance message format, don't reconstruct it

**Example Refactor:**
```typescript
// OLD: Complex regex extraction
const keywordCount = extractNumber(/(\d+)\s+keyword/i, message);

// NEW: Use backend details
const percentage = details.percentage;
const message = details.message; // Already has counts
const itemName = details.current_item || '';
```

---

#### 3. Real-Time Updates Per Page

**Pages to update:** Keywords, Ideas, Tasks, Content

**Current behavior:** Modal closes, then data reloads

**Improved behavior:**
- Show inline status badge in table rows during processing
- Update table row in real-time as status changes
- Auto-reload specific rows instead of entire table
- Show notification when task completes in background

**Implementation approach:**
- Subscribe to task updates while modal is open
- Update table state with new status
- Add `status` badge column to tables showing:
  - 🔄 Processing (animated)
  - ✅ Completed
  - ❌ Failed
  - ⏸️ Paused

---

### C. Automation Wizard Progress Fixes

**File:** `frontend/src/pages/Automation/AutomationWizard.tsx`

**Issues identified:**
1. Wrong queue items displayed
2. Missing queue items
3. Progress bar inaccurate
4. Stage counts buggy

**Required fixes:**

#### 1. Stage Progress Cards

**Current location:** Stage cards display in wizard

**Problems:**
- Counts mismatch between backend and frontend
- "In Queue" vs "Processed" numbers don't add up
- Real-time updates missing

**Fix:**
- Pull stage results from `AutomationRun.stage_X_result` JSON fields
- Display accurate counts:
  ```
  Stage 1: Clustering
  - Keywords Loaded: 45
  - Clusters Created: 8
  - Status: Completed
  ```

#### 2. Queue Item Display

**Fix queue item tracking:**
- Each stage should show items being processed
- Items should appear in queue before processing
- Items should move to "Completed" list after processing
- Failed items should show in "Failed" section with retry option

**Data structure per stage:**
```json
{
  "pending": ["item1", "item2"],
  "processing": ["item3"],
  "completed": ["item4", "item5"],
  "failed": [{"item": "item6", "error": "..."}]
}
```

#### 3. Progress Bar Calculation

**Fix:** 
- Base percentage on actual completed stages
- Within-stage percentage based on items processed
- Formula: `(completed_stages * 100 / 7) + (current_stage_progress / 7)`

---

### D. Error State Handling

#### Backend Error Logging

**Location:** `backend/igny8_core/ai/engine.py`

**Current:** Errors logged but status not always propagated

**Improvement:**
- Always set task state to `FAILURE` on error
- Include detailed error in state metadata
- Log error with context (function, input data, trace)

**Error payload structure:**
```json
{
  "status": "error",
  "error_type": "InsufficientCreditsError",
  "error_message": "Insufficient credits. Required: 45, Available: 20",
  "phase": "INIT",
  "timestamp": "2025-12-11T10:30:00Z"
}
```

#### Frontend Error Display

**Files:** All progress modals

**Current:** Generic "Task failed" message

**Improvement:**
- Show specific error type
- Display user-friendly error message
- Provide actionable next steps
- Show "Retry" button for recoverable errors

**Error message mapping:**

| Error Type | User Message | Action |
|------------|--------------|--------|
| InsufficientCreditsError | "Not enough credits. You need X credits." | "Purchase Credits" button |
| ValidationError | "Invalid data: {details}" | "Fix Data" button |
| AIProviderError | "AI service temporarily unavailable" | "Retry" button |
| TimeoutError | "Request timed out. Try again." | "Retry" button |

---

## QA Testing Checklist

### Manual AI Function Tests

Test each function on each page:

| Page | Function | Test Case | Expected Result |
|------|----------|-----------|-----------------|
| Keywords | Clustering | Select 50 keywords, run clustering | Modal shows "Clustering 50 keywords", creates clusters, shows "50 keywords mapped into X clusters" |
| Ideas | Generate Ideas | Select 3 clusters, generate ideas | Modal shows "Generating ideas for 3 clusters", creates ideas, shows "Generated X ideas for 3 clusters" |
| Tasks | Generate Content | Select 1 task, generate content | Modal shows "Writing {word_count}-word article", creates content, shows "{word_count}-word article drafted" |
| Tasks | Generate Image Prompts | Select 1 content, generate prompts | Modal shows "Mapping content for X prompts", shows "1 Featured + X In-article prompts ready" |
| Images | Generate Images | Select content with prompts, generate | ImageQueueModal shows each image generating, completes all |

### Edge Case Tests

| Scenario | Expected Behavior |
|----------|-------------------|
| Task cancelled mid-process | Status updates to "cancelled", modal shows cancellation message |
| Task fails due to insufficient credits | Error modal with credit purchase link |
| Task fails due to validation error | Error modal with specific validation issue |
| Multiple tasks running simultaneously | Each has independent progress tracking |
| Network disconnection during task | Polling resumes when reconnected, catches up with status |
| Task completes while modal is closed | Table updates status automatically, notification shown |

### Automation Wizard Tests

| Stage | Test Case | Expected Result |
|-------|-----------|-----------------|
| Stage 1 | Run clustering on 100 keywords | Shows batch progress, creates clusters, displays accurate counts |
| Stage 2 | Generate ideas for 10 clusters | Shows cluster-by-cluster progress, creates ideas |
| Stage 3 | Convert ideas to tasks | Shows conversion progress, creates tasks |
| Stage 4 | Generate content for tasks | Shows content generation per task |
| Stage 5 | Generate image prompts | Shows prompt creation progress |
| Stage 6 | Generate images | Shows image generation progress |
| Stage 7 | Final review | Shows completed content ready for publishing |
| Pause/Resume | Pause at stage 3, resume | Continues from where it left off |
| Cancel | Cancel at stage 4 | Stops processing, shows cancelled status |

---

## Implementation Priority

### Phase 1: Backend Fixes (Week 1)
1. ✅ Create unified status enums
2. ✅ Add `generation_status` fields to models
3. ✅ Update AI function progress messages with accurate counts
4. ✅ Enhance tracker to include item counts and current item

### Phase 2: Frontend Display (Week 1-2)
1. ✅ Update ProgressModal success messages
2. ✅ Refactor step mapping logic to use backend details
3. ✅ Add real-time status badges to table rows
4. ✅ Improve error message display

### Phase 3: Automation Wizard (Week 2)
1. ✅ Fix stage progress cards
2. ✅ Fix queue item tracking
3. ✅ Fix progress bar calculation
4. ✅ Add pause/resume/cancel controls

### Phase 4: QA & Polish (Week 2-3)
1. ✅ Run all manual tests
2. ✅ Run all edge case tests
3. ✅ Run automation wizard tests
4. ✅ Fix any discovered issues
5. ✅ Performance optimization

---

## Success Metrics

- ✅ All progress modals show accurate counts and messages
- ✅ Progress percentages match actual work completed
- ✅ Automation wizard shows correct queue states
- ✅ Error messages are specific and actionable
- ✅ Real-time updates work consistently
- ✅ No user-reported confusion about task status
- ✅ All QA test cases pass

---

## Related Files Reference

### Backend
- `backend/igny8_core/business/planning/models.py` - Keywords, Clusters, ContentIdeas models
- `backend/igny8_core/business/content/models.py` - Tasks, Content models
- `backend/igny8_core/business/automation/models.py` - AutomationRun model
- `backend/igny8_core/ai/tracker.py` - ProgressTracker, StepTracker
- `backend/igny8_core/ai/engine.py` - AIEngine orchestration
- `backend/igny8_core/ai/functions/*.py` - All AI function implementations

### Frontend
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook
- `frontend/src/components/common/ProgressModal.tsx` - Progress modal component
- `frontend/src/components/common/ImageQueueModal.tsx` - Image generation modal
- `frontend/src/pages/Planner/Keywords.tsx` - Keywords page with clustering
- `frontend/src/pages/Planner/Ideas.tsx` - Ideas page with idea generation
- `frontend/src/pages/Writer/Tasks.tsx` - Tasks page with content generation
- `frontend/src/pages/Writer/Content.tsx` - Content page
- `frontend/src/pages/Writer/Images.tsx` - Images page
- `frontend/src/pages/Automation/AutomationWizard.tsx` - Automation wizard

---

## Notes

- Consider WebSocket implementation for production to reduce polling overhead
- Status badges in tables should be subtle and not distract from main content
- Error messages should prioritize user understanding over technical accuracy
- Progress messages should be encouraging and informative
- Real-time updates should not cause UI flickering or performance issues