pre-launch-final mods-docs

2025-12-11 07:20:21 +00:00
parent 20fdd3b295
commit a736bc3d34
8 changed files with 5464 additions and 0 deletions
--- a/docs/PRE-LAUNCH/ITEM-1-STATUS-MODULES.md
+++ b/docs/PRE-LAUNCH/ITEM-1-STATUS-MODULES.md
@@ -0,0 +1,547 @@
+# Item 1: Status Modules for AI Planner and Writer
+
+**Priority:** Critical  
+**Target:** Production Launch  
+**Last Updated:** December 11, 2025
+
+---
+
+## Overview
+
+Standardize and fix status tracking across Planner and Writer modules to ensure consistent, accurate, and real-time progress updates for all AI operations. This includes backend status management, frontend progress displays, WebSocket/polling systems, and error handling.
+
+---
+
+## Current Implementation Analysis
+
+### Backend Status System
+
+#### Status Fields in Models
+
+**Location:** `backend/igny8_core/business/planning/models.py` and `backend/igny8_core/business/content/models.py`
+
+| Model | Current Status Choices | Issues |
+|-------|----------------------|---------|
+| **Keywords** | `new`, `mapped` | Limited states, no processing states |
+| **Clusters** | `new`, `mapped` | No active/processing states |
+| **ContentIdeas** | `new`, `queued`, `completed` | Missing `processing`, `failed` states |
+| **Tasks** | `queued`, `completed` | Missing intermediate states (processing, failed, cancelled) |
+| **Content** | `draft`, `review`, `published` | Separate from generation status |
+| **AutomationRun** | `running`, `paused`, `cancelled`, `completed`, `failed` | Well-defined but isolated |
+
+**Key Problem:** Status choices are inconsistent across models. Some models track generation status, others track editorial status, causing confusion.
+
+---
+
+### AI Function Progress Tracking
+
+**Location:** `backend/igny8_core/ai/tracker.py`, `backend/igny8_core/ai/engine.py`
+
+#### Progress Phases
+
+All AI functions emit standard progress phases:
+
+| Phase | Default Label | Percentage Range | Purpose |
+|-------|--------------|------------------|---------|
+| `INIT` | Initializing... | 0-10% | Validation, setup |
+| `PREP` | Preparing data... | 10-20% | Data loading, preparation |
+| `AI_CALL` | Processing with AI... | 20-80% | AI model execution |
+| `PARSE` | Parsing response... | 80-90% | Response validation, parsing |
+| `SAVE` | Saving results... | 90-100% | Database operations |
+| `DONE` | Complete! | 100% | Final state |
+
+**Implementation:** 
+- `StepTracker` class emits step-level events
+- `ProgressTracker` class manages Celery task state updates
+- `AIEngine` orchestrates phase transitions
+
+**Issues:**
+- Phase messages are generic and don't reflect actual counts
+- Percentage calculations sometimes inconsistent
+- No clear error state handling in phases
+
+---
+
+### Frontend Progress Modal System
+
+**Location:** `frontend/src/hooks/useProgressModal.ts`, `frontend/src/components/common/ProgressModal.tsx`
+
+#### Current Modal Implementation
+
+**Hook:** `useProgressModal`
+- Manages modal state, task polling, progress updates
+- Polls backend every 1 second for task status
+- Maps backend phases to user-friendly messages
+- Tracks step logs for debugging
+
+**Modal Component:** `ProgressModal`
+- Displays progress bar with percentage
+- Shows current phase and message
+- Lists step-by-step progress visually
+- Provides cancel/close actions
+
+#### Step Mapping Logic
+
+**Function:** `getStepInfo()` in `useProgressModal.ts`
+
+Maps backend phases to:
+- User-friendly messages with dynamic counts
+- Adjusted percentages for smooth progression
+- Function-specific labels (clustering vs. ideas vs. content)
+
+**Problems Identified:**
+1. **Inaccurate counts in messages**: Extracts counts from messages using regex, often missing or incorrect
+2. **Generic phase labels**: Messages don't always reflect what's actually happening
+3. **Inconsistent percentage progression**: Jumps or stalls during AI_CALL phase
+4. **Status text not aligned**: Labels in modal don't match backend phase names
+
+---
+
+### Progress Display by Page
+
+| Page | Location | Progress Display | Issues |
+|------|----------|------------------|--------|
+| Keywords | `frontend/src/pages/Planner/Keywords.tsx` | ProgressModal, AI logs | Clustering progress text inaccurate |
+| Clusters | `frontend/src/pages/Planner/Clusters.tsx` | ProgressModal only | No cluster generation function exists |
+| Ideas | `frontend/src/pages/Planner/Ideas.tsx` | ProgressModal, reload on complete | Text shows generic "preparing clusters" |
+| Tasks | `frontend/src/pages/Writer/Tasks.tsx` | ProgressModal, AI logs | Content generation text inaccurate |
+| Content | `frontend/src/pages/Writer/Content.tsx` | ProgressModal | Missing real-time updates |
+| Images | `frontend/src/pages/Writer/Images.tsx` | ImageQueueModal | Uses separate queue-based system |
+
+---
+
+### Known Bugs and Issues
+
+From `Pre-Launch-Pending.md` Known Bugs section:
+
+#### Manual AI Function Run Modals
+
+**Affected Pages:** Keywords, Clusters, Ideas, Tasks
+
+**Issues:**
+- Progress modal texts are not accurate for specific functions
+- Dynamic variables not properly populated with counts
+- Generic messages instead of function-specific ones
+
+**Example:**
+- Clustering: Should say "Mapping 45 keywords into clusters" but shows "Preparing data..."
+- Ideas: Should say "Generating ideas for 3 clusters" but shows generic text
+
+---
+
+#### Automation Wizard Progress
+
+**Page:** Automation (`frontend/src/pages/Automation/AutomationWizard.tsx`)
+
+**Issues:**
+- Wrong queue items displayed
+- Missing queue items for certain stages
+- Progress bar doesn't progress properly
+- Total in queue and processed counts buggy across stages
+- Stage cards: Real-time metrics not optimized
+
+---
+
+### WebSocket vs Polling
+
+**Current Implementation:** Polling-based
+
+**Location:** `useProgressModal.ts` - polls task status every 1 second
+
+**Note:** Code mentions WebSockets but current implementation uses REST API polling via `/api/v1/...` endpoints
+
+**Improvement Needed:** Consider WebSocket implementation for real-time updates without polling overhead
+
+---
+
+## Required Changes
+
+### A. Backend Status Standardization
+
+#### 1. Unified Status Enum
+
+**Create:** `backend/igny8_core/common/status_enums.py`
+
+Define standard status choices for all modules:
+
+```
+GENERATION_STATUS_CHOICES = [
+    ('pending', 'Pending'),
+    ('queued', 'Queued'),
+    ('processing', 'Processing'),
+    ('completed', 'Completed'),
+    ('failed', 'Failed'),
+    ('cancelled', 'Cancelled'),
+]
+
+CONTENT_STATUS_CHOICES = [
+    ('draft', 'Draft'),
+    ('review', 'Review'),
+    ('published', 'Published'),
+    ('archived', 'Archived'),
+]
+```
+
+**Action:** Add `generation_status` field to models that undergo AI processing:
+- Keywords (for extraction/clustering operations)
+- Clusters (for cluster generation, if implemented)
+- ContentIdeas (for idea generation)
+- Tasks (for content generation)
+- Content (separate from editorial status)
+
+#### 2. Enhanced Progress Details
+
+**Update:** `backend/igny8_core/ai/tracker.py`
+
+**Modifications:**
+- `StepTracker.update_step()` should include:
+  - `items_total`: Total count of items being processed
+  - `items_processed`: Current count of items processed
+  - `current_item_name`: Name/title of current item
+  - `estimated_remaining_seconds`: Estimated time remaining
+
+**Example Progress Payload:**
+```json
+{
+  "phase": "AI_CALL",
+  "message": "Clustering 45 keywords",
+  "percentage": 45,
+  "items_total": 45,
+  "items_processed": 20,
+  "current_item": "best seo tools",
+  "estimated_seconds_remaining": 120
+}
+```
+
+#### 3. Function-Specific Progress Messages
+
+**Update:** Each AI function's phase messages
+
+**Files to modify:**
+- `backend/igny8_core/ai/functions/auto_cluster.py`
+- `backend/igny8_core/ai/functions/generate_ideas.py`
+- `backend/igny8_core/ai/functions/generate_content.py`
+- `backend/igny8_core/ai/functions/generate_image_prompts.py`
+- `backend/igny8_core/ai/functions/generate_images.py`
+
+**Changes per function:**
+
+| Function | Phase | Accurate Message Template |
+|----------|-------|---------------------------|
+| auto_cluster | PREP | "Loading {count} keywords for clustering" |
+| auto_cluster | AI_CALL | "Analyzing keyword relationships ({processed}/{total})" |
+| auto_cluster | SAVE | "Creating {cluster_count} clusters" |
+| generate_ideas | PREP | "Preparing {count} cluster(s) for idea generation" |
+| generate_ideas | AI_CALL | "Generating ideas for cluster: {cluster_name}" |
+| generate_ideas | SAVE | "Created {idea_count} content ideas" |
+| generate_content | PREP | "Preparing task: {task_title}" |
+| generate_content | AI_CALL | "Writing {word_count}-word article" |
+| generate_content | SAVE | "Saving article: {title}" |
+| generate_image_prompts | PREP | "Mapping content for {count} image prompts" |
+| generate_image_prompts | AI_CALL | "Writing featured image prompt" |
+| generate_image_prompts | PARSE | "Writing {count} in-article image prompts" |
+| generate_images | PREP | "Preparing {count} images for generation" |
+| generate_images | AI_CALL | "Generating image {current}/{total}: {prompt}" |
+
+**Implementation:** Use `self.tracker.update_step()` with dynamic values from actual data
+
+---
+
+### B. Frontend Progress Display Improvements
+
+#### 1. Update ProgressModal Messages
+
+**File:** `frontend/src/components/common/ProgressModal.tsx`
+
+**Function:** `getSuccessMessage()`
+
+**Current Issues:**
+- Generic success messages
+- Count extraction from logs is unreliable
+- Doesn't distinguish between function types clearly
+
+**Required Changes:**
+| Function | Current Success Message | Improved Success Message |
+|----------|------------------------|--------------------------|
+| Clustering | "Clustering complete" | "45 keywords mapped into 8 clusters" |
+| Ideas | "Content ideas created successfully" | "Generated 12 content ideas for 3 clusters" |
+| Content | "Article drafted successfully" | "1,245-word article drafted successfully" |
+| Image Prompts | Generic text | "1 Featured Image + 4 In-article Image Prompts ready" |
+| Images | "Images generated successfully" | "5 images generated successfully" |
+
+**Implementation:** Extract counts from backend `details` field, not from message strings
+
+---
+
+#### 2. Improve Step Mapping Logic
+
+**File:** `frontend/src/hooks/useProgressModal.ts`
+
+**Function:** `getStepInfo()`
+
+**Problems:**
+- Complex regex-based count extraction from messages
+- Percentage calculation doesn't match backend
+- Messages manually constructed instead of using backend messages
+
+**Solution:**
+- **Rely on backend `details` field** for counts:
+  ```typescript
+  const { items_total, items_processed, current_item } = progress.details;
+  ```
+- Use backend message directly when it contains proper counts
+- Only enhance message format, don't reconstruct it
+
+**Example Refactor:**
+```typescript
+// OLD: Complex regex extraction
+const keywordCount = extractNumber(/(\d+)\s+keyword/i, message);
+
+// NEW: Use backend details
+const percentage = details.percentage;
+const message = details.message; // Already has counts
+const itemName = details.current_item || '';
+```
+
+---
+
+#### 3. Real-Time Updates Per Page
+
+**Pages to update:** Keywords, Ideas, Tasks, Content
+
+**Current behavior:** Modal closes, then data reloads
+
+**Improved behavior:**
+- Show inline status badge in table rows during processing
+- Update table row in real-time as status changes
+- Auto-reload specific rows instead of entire table
+- Show notification when task completes in background
+
+**Implementation approach:**
+- Subscribe to task updates while modal is open
+- Update table state with new status
+- Add `status` badge column to tables showing:
+  - 🔄 Processing (animated)
+  - ✅ Completed
+  - ❌ Failed
+  - ⏸️ Paused
+
+---
+
+### C. Automation Wizard Progress Fixes
+
+**File:** `frontend/src/pages/Automation/AutomationWizard.tsx`
+
+**Issues identified:**
+1. Wrong queue items displayed
+2. Missing queue items
+3. Progress bar inaccurate
+4. Stage counts buggy
+
+**Required fixes:**
+
+#### 1. Stage Progress Cards
+
+**Current location:** Stage cards display in wizard
+
+**Problems:**
+- Counts mismatch between backend and frontend
+- "In Queue" vs "Processed" numbers don't add up
+- Real-time updates missing
+
+**Fix:**
+- Pull stage results from `AutomationRun.stage_X_result` JSON fields
+- Display accurate counts:
+  ```
+  Stage 1: Clustering
+  - Keywords Loaded: 45
+  - Clusters Created: 8
+  - Status: Completed
+  ```
+
+#### 2. Queue Item Display
+
+**Fix queue item tracking:**
+- Each stage should show items being processed
+- Items should appear in queue before processing
+- Items should move to "Completed" list after processing
+- Failed items should show in "Failed" section with retry option
+
+**Data structure per stage:**
+```json
+{
+  "pending": ["item1", "item2"],
+  "processing": ["item3"],
+  "completed": ["item4", "item5"],
+  "failed": [{"item": "item6", "error": "..."}]
+}
+```
+
+#### 3. Progress Bar Calculation
+
+**Fix:** 
+- Base percentage on actual completed stages
+- Within-stage percentage based on items processed
+- Formula: `(completed_stages * 100 / 7) + (current_stage_progress / 7)`
+
+---
+
+### D. Error State Handling
+
+#### Backend Error Logging
+
+**Location:** `backend/igny8_core/ai/engine.py`
+
+**Current:** Errors logged but status not always propagated
+
+**Improvement:**
+- Always set task state to `FAILURE` on error
+- Include detailed error in state metadata
+- Log error with context (function, input data, trace)
+
+**Error payload structure:**
+```json
+{
+  "status": "error",
+  "error_type": "InsufficientCreditsError",
+  "error_message": "Insufficient credits. Required: 45, Available: 20",
+  "phase": "INIT",
+  "timestamp": "2025-12-11T10:30:00Z"
+}
+```
+
+#### Frontend Error Display
+
+**Files:** All progress modals
+
+**Current:** Generic "Task failed" message
+
+**Improvement:**
+- Show specific error type
+- Display user-friendly error message
+- Provide actionable next steps
+- Show "Retry" button for recoverable errors
+
+**Error message mapping:**
+
+| Error Type | User Message | Action |
+|------------|--------------|--------|
+| InsufficientCreditsError | "Not enough credits. You need X credits." | "Purchase Credits" button |
+| ValidationError | "Invalid data: {details}" | "Fix Data" button |
+| AIProviderError | "AI service temporarily unavailable" | "Retry" button |
+| TimeoutError | "Request timed out. Try again." | "Retry" button |
+
+---
+
+## QA Testing Checklist
+
+### Manual AI Function Tests
+
+Test each function on each page:
+
+| Page | Function | Test Case | Expected Result |
+|------|----------|-----------|-----------------|
+| Keywords | Clustering | Select 50 keywords, run clustering | Modal shows "Clustering 50 keywords", creates clusters, shows "50 keywords mapped into X clusters" |
+| Ideas | Generate Ideas | Select 3 clusters, generate ideas | Modal shows "Generating ideas for 3 clusters", creates ideas, shows "Generated X ideas for 3 clusters" |
+| Tasks | Generate Content | Select 1 task, generate content | Modal shows "Writing {word_count}-word article", creates content, shows "{word_count}-word article drafted" |
+| Tasks | Generate Image Prompts | Select 1 content, generate prompts | Modal shows "Mapping content for X prompts", shows "1 Featured + X In-article prompts ready" |
+| Images | Generate Images | Select content with prompts, generate | ImageQueueModal shows each image generating, completes all |
+
+### Edge Case Tests
+
+| Scenario | Expected Behavior |
+|----------|-------------------|
+| Task cancelled mid-process | Status updates to "cancelled", modal shows cancellation message |
+| Task fails due to insufficient credits | Error modal with credit purchase link |
+| Task fails due to validation error | Error modal with specific validation issue |
+| Multiple tasks running simultaneously | Each has independent progress tracking |
+| Network disconnection during task | Polling resumes when reconnected, catches up with status |
+| Task completes while modal is closed | Table updates status automatically, notification shown |
+
+### Automation Wizard Tests
+
+| Stage | Test Case | Expected Result |
+|-------|-----------|-----------------|
+| Stage 1 | Run clustering on 100 keywords | Shows batch progress, creates clusters, displays accurate counts |
+| Stage 2 | Generate ideas for 10 clusters | Shows cluster-by-cluster progress, creates ideas |
+| Stage 3 | Convert ideas to tasks | Shows conversion progress, creates tasks |
+| Stage 4 | Generate content for tasks | Shows content generation per task |
+| Stage 5 | Generate image prompts | Shows prompt creation progress |
+| Stage 6 | Generate images | Shows image generation progress |
+| Stage 7 | Final review | Shows completed content ready for publishing |
+| Pause/Resume | Pause at stage 3, resume | Continues from where it left off |
+| Cancel | Cancel at stage 4 | Stops processing, shows cancelled status |
+
+---
+
+## Implementation Priority
+
+### Phase 1: Backend Fixes (Week 1)
+1. ✅ Create unified status enums
+2. ✅ Add `generation_status` fields to models
+3. ✅ Update AI function progress messages with accurate counts
+4. ✅ Enhance tracker to include item counts and current item
+
+### Phase 2: Frontend Display (Week 1-2)
+1. ✅ Update ProgressModal success messages
+2. ✅ Refactor step mapping logic to use backend details
+3. ✅ Add real-time status badges to table rows
+4. ✅ Improve error message display
+
+### Phase 3: Automation Wizard (Week 2)
+1. ✅ Fix stage progress cards
+2. ✅ Fix queue item tracking
+3. ✅ Fix progress bar calculation
+4. ✅ Add pause/resume/cancel controls
+
+### Phase 4: QA & Polish (Week 2-3)
+1. ✅ Run all manual tests
+2. ✅ Run all edge case tests
+3. ✅ Run automation wizard tests
+4. ✅ Fix any discovered issues
+5. ✅ Performance optimization
+
+---
+
+## Success Metrics
+
+- ✅ All progress modals show accurate counts and messages
+- ✅ Progress percentages match actual work completed
+- ✅ Automation wizard shows correct queue states
+- ✅ Error messages are specific and actionable
+- ✅ Real-time updates work consistently
+- ✅ No user-reported confusion about task status
+- ✅ All QA test cases pass
+
+---
+
+## Related Files Reference
+
+### Backend
+- `backend/igny8_core/business/planning/models.py` - Keywords, Clusters, ContentIdeas models
+- `backend/igny8_core/business/content/models.py` - Tasks, Content models
+- `backend/igny8_core/business/automation/models.py` - AutomationRun model
+- `backend/igny8_core/ai/tracker.py` - ProgressTracker, StepTracker
+- `backend/igny8_core/ai/engine.py` - AIEngine orchestration
+- `backend/igny8_core/ai/functions/*.py` - All AI function implementations
+
+### Frontend
+- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook
+- `frontend/src/components/common/ProgressModal.tsx` - Progress modal component
+- `frontend/src/components/common/ImageQueueModal.tsx` - Image generation modal
+- `frontend/src/pages/Planner/Keywords.tsx` - Keywords page with clustering
+- `frontend/src/pages/Planner/Ideas.tsx` - Ideas page with idea generation
+- `frontend/src/pages/Writer/Tasks.tsx` - Tasks page with content generation
+- `frontend/src/pages/Writer/Content.tsx` - Content page
+- `frontend/src/pages/Writer/Images.tsx` - Images page
+- `frontend/src/pages/Automation/AutomationWizard.tsx` - Automation wizard
+
+---
+
+## Notes
+
+- Consider WebSocket implementation for production to reduce polling overhead
+- Status badges in tables should be subtle and not distract from main content
+- Error messages should prioritize user understanding over technical accuracy
+- Progress messages should be encouraging and informative
+- Real-time updates should not cause UI flickering or performance issues