salman/igny8
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

asdsad

Browse Source
This commit is contained in:
IGNY8 VPS (Salman)
2025-12-15 03:30:32 +00:00
parent 8162b6ae92
commit c23698f7f8
5 changed files with 7 additions and 2461 deletions
Download Patch File Download Diff File Expand all files Collapse all files

456
IMPLEMENTATION-AUDIT-AND-ACTION-PLAN.md
View File

@@ -1,456 +0,0 @@
# Implementation Audit - Reality Check
**Based on Actual Codebase Analysis**
**Date:** December 15, 2025
**Status:** Code-Verified Action Plan
**Purpose:** Ground implementation in what actually exists, not assumptions
---
## Executive Summary
### What Actually Works (Code-Verified)
✅ **PageHeader Component** (`frontend/src/components/common/PageHeader.tsx`, 170 lines)
- Handles site/sector selectors, badges, navigation tabs, last updated timestamp
- Used consistently across all Planner/Writer pages
- **No changes needed**
✅ **ProgressModal Component** (`frontend/src/components/common/ProgressModal.tsx`, 822 lines)
- Shows step-by-step progress for AI operations
- Has visual step indicators, handles all AI functions
- Extracts success messages with counts
- **Works well, but has fragile regex parsing**
✅ **useProgressModal Hook** (`frontend/src/hooks/useProgressModal.ts`, 815 lines)
- Polls Celery task status every 2 seconds
- Has elaborate `getStepInfo()` function (146 lines) that uses regex to extract counts
- **Problem**: Relies on message format, not structured data
✅ **Backend ProgressTracker** (`backend/igny8_core/ai/tracker.py`, 347 lines)
- Has `update(phase, percentage, message, current, total, current_item, meta)` method
- **Supports structured data via `meta` dict**
- **Problem**: AI functions don't actually use `meta` to pass structured details
✅ **Model Status Fields Work Fine**
- Keywords: `status` = `('new', 'New'), ('mapped', 'Mapped')`
- Clusters: `status` = `('new', 'New'), ('mapped', 'Mapped')`
- ContentIdeas: `status` = `('new', 'New'), ('queued', 'Queued'), ('completed', 'Completed')`
- Tasks: `status` = `('queued', 'Queued'), ('completed', 'Completed')`
- Content: `status` = `('draft', 'Draft'), ('review', 'Review'), ('published', 'Published')`
- **These work for workflow state tracking - don't need to change**
### The Actual Problems
❌ **Problem 1: Fragile Progress Data Extraction**
- Frontend uses regex to parse counts from messages: `/(\d+)\s+keyword/i`, `/(\d+)\s+cluster/i`
- **146 lines** of extraction logic in `getStepInfo()` function
- Breaks if message wording changes
- **Solution**: Backend should pass structured details, frontend reads directly
❌ **Problem 2: Missing Workflow Guidance**
- Users don't know where they are in the workflow (no "Step 2 of 3")
- No contextual help for first-time users
- No quick metrics summary visible
- **Solution**: Add StepBanner, HelperNotification, MetricsPanel components
❌ **Problem 3: Automation Progress Parsing Issues**
- AutomationPage tries to parse `stage_X_result` JSON
- Queue item display incorrect
- Progress bar doesn't reflect actual progress
- **Solution**: Fix stage result structure and parsing logic
---
## Action Plan (Reality-Based)
### Phase 1: Backend - Emit Structured Progress Details (Week 1)
**Goal**: Make AI functions pass structured data instead of just messages
**Files to Update:**
1. `backend/igny8_core/ai/functions/auto_cluster.py` (347 lines)
2. `backend/igny8_core/ai/functions/generate_ideas.py`
3. `backend/igny8_core/ai/functions/generate_content.py`
4. `backend/igny8_core/ai/functions/generate_image_prompts.py`
5. `backend/igny8_core/ai/functions/generate_images.py`
**Change Pattern:**
**Before:**
```python
tracker.update(
phase='SAVE',
percentage=90,
message=f"Creating {len(clusters)} clusters"
)
```
**After:**
```python
tracker.update(
phase='SAVE',
percentage=90,
message=f"Creating {len(clusters)} clusters",
meta={
'items_total': len(keywords),
'items_processed': processed_count,
'items_created': len(clusters),
'current_item_name': current_cluster.name if current_cluster else None
}
)
```
**Effort:** ~3-4 hours per function = 15-20 hours total
---
### Phase 2: Frontend - Simplify Progress Modal (Week 1-2)
**Goal**: Remove regex parsing, read structured details directly
**Files to Update:**
1. `frontend/src/hooks/useProgressModal.ts` (815 lines)
- **Remove**: Lines 96-242 (146 lines of regex extraction)
- **Add**: Simple `details` accessor from `state.meta.details`
2. `frontend/src/components/common/ProgressModal.tsx` (822 lines)
- **Simplify**: `getSuccessMessage()` to read from backend
- **Remove**: Regex patterns for extracting counts
**Before (useProgressModal.ts, ~146 lines):**
```typescript
const getStepInfo = (stepName: string, message: string, allSteps: any[]) => {
// Extract keyword count
let keywordCount = extractNumber(/(\d+)\s+keyword/i, message);
// ... 140 more lines of extraction logic
};
```
**After (~20 lines):**
```typescript
const getStepInfo = (state: any): ProgressDisplay => {
const details = state.meta?.details || {};
return {
percentage: state.meta?.percentage || 0,
message: state.meta?.message || 'Processing...',
itemsTotal: details.items_total,
itemsProcessed: details.items_processed,
itemsCreated: details.items_created,
currentItemName: details.current_item_name,
};
};
```
**Effort:** ~6-8 hours
---
### Phase 3: Create Missing UX Components (Week 2)
#### 3.1 StepBanner Component
**File:** `frontend/src/components/workflow/StepBanner.tsx` (NEW)
**Purpose:** Show "Step 2 of 3: Organize Clusters" with clickable progress
**Interface:**
```typescript
interface StepBannerProps {
currentStep: number;
totalSteps: number;
steps: Array<{
label: string;
href: string;
completed: boolean;
}>;
}
```
**Usage:**
```tsx
// In Clusters.tsx
<StepBanner
currentStep={2}
totalSteps={3}
steps={[
{ label: 'Extract Keywords', href: '/planner/keywords', completed: true },
{ label: 'Organize Clusters', href: '/planner/clusters', completed: false },
{ label: 'Generate Ideas', href: '/planner/ideas', completed: false },
]}
/>
```
**Effort:** ~4 hours
---
#### 3.2 HelperNotification Component
**File:** `frontend/src/components/helper/HelperNotification.tsx` (NEW)
**Purpose:** Dismissible contextual help (stored in localStorage)
**Interface:**
```typescript
interface HelperNotificationProps {
type: 'welcome' | 'info' | 'tip' | 'warning' | 'success';
title: string;
message: string;
actions?: Array<{ label: string; onClick?: () => void; }>;
dismissible?: boolean;
pageKey: string; // For localStorage persistence
}
```
**Storage:** `localStorage.setItem('helper_dismissed_keywords', 'true')`
**Usage:**
```tsx
// In Keywords.tsx
<HelperNotification
type="welcome"
title="Welcome to Keywords"
message="Import keywords or add manually to get started"
pageKey="keywords"
actions={[
{ label: 'Import CSV', onClick: handleImport },
{ label: 'Learn More', onClick: () => window.open('/docs/keywords') }
]}
/>
```
**Effort:** ~4 hours
---
#### 3.3 MetricsPanel Component
**File:** `frontend/src/components/dashboard/MetricsPanel.tsx` (NEW)
**Purpose:** Collapsible metrics summary (alternative to ModuleMetricsFooter)
**Interface:**
```typescript
interface MetricsPanelProps {
title: string;
metrics: Array<{
label: string;
value: string | number;
subtitle?: string;
tooltip?: string;
}>;
collapsible?: boolean;
defaultCollapsed?: boolean;
}
```
**Usage:**
```tsx
// In Keywords.tsx
<MetricsPanel
title="Keyword Metrics"
metrics={[
{ label: 'Total', value: totalCount, subtitle: 'All keywords' },
{ label: 'New', value: newCount, subtitle: 'Unmapped' },
{ label: 'Mapped', value: mappedCount, subtitle: 'In clusters' },
]}
collapsible={true}
/>
```
**Effort:** ~6 hours
---
### Phase 4: Update Pages with New Components (Week 2-3)
#### Planner Module
**Keywords.tsx** (998 lines)
- Add StepBanner (currentStep=1, totalSteps=3)
- Add HelperNotification for first-time guidance
- Add MetricsPanel (or keep existing ModuleMetricsFooter)
- **Effort:** 2-3 hours
**Clusters.tsx**
- Add StepBanner (currentStep=2, totalSteps=3)
- Add HelperNotification
- Add MetricsPanel
- **Effort:** 2-3 hours
**Ideas.tsx**
- Add StepBanner (currentStep=3, totalSteps=3)
- Add HelperNotification
- Add MetricsPanel
- **Effort:** 2-3 hours
#### Writer Module
**Tasks.tsx**
- Add StepBanner (currentStep=1, totalSteps=3)
- Add HelperNotification
- Add MetricsPanel
- **Effort:** 2-3 hours
**Content.tsx**
- Add StepBanner (currentStep=2, totalSteps=3)
- Add HelperNotification
- Add MetricsPanel
- **Effort:** 2-3 hours
**Images.tsx**
- Add StepBanner (currentStep=3, totalSteps=3)
- Add HelperNotification
- Add MetricsPanel
- **Effort:** 2-3 hours
**Total Page Updates:** 12-18 hours
---
### Phase 5: Fix Automation Progress Issues (Week 3)
**File:** `frontend/src/pages/Automation/AutomationPage.tsx` (995 lines)
**Current Issues:**
1. Stage progress cards show wrong/missing queue items
2. Progress bar calculation inaccurate
3. Stage metrics don't add up
**Root Cause:** Stage result JSON structure not being parsed correctly
**Fix:**
1. Add HelperNotification for automation guidance
2. Fix stage result parsing logic
3. Fix progress bar calculation: `(completed_stages * 100 / 7) + (current_stage_progress / 7)`
4. Ensure stage metrics accurately reflect backend counts
**Effort:** 8-10 hours
---
## Total Effort Estimate
| Phase | Description | Hours | Duration |
|-------|-------------|-------|----------|
| Phase 1 | Backend structured details | 15-20h | Week 1 |
| Phase 2 | Frontend simplification | 6-8h | Week 1-2 |
| Phase 3 | New UX components | 14h | Week 2 |
| Phase 4 | Update pages | 12-18h | Week 2-3 |
| Phase 5 | Fix automation | 8-10h | Week 3 |
| **TOTAL** | **55-70 hours** | **~2-3 weeks** |
---
## What NOT To Do
❌ **Don't add `generation_status` field to models**
- Current status fields work fine for workflow tracking
- No need for DB migrations
❌ **Don't create status enum files**
- Models already have STATUS_CHOICES inline
- No need for centralized enums
❌ **Don't refactor ProgressModal completely**
- It works well, just needs to read structured data instead of parsing messages
- Keep existing step visualization logic
❌ **Don't change Celery task structure**
- Task state updates work fine
- Just need to pass more structured data in `meta`
---
## Testing Strategy
### Phase 1-2 Testing (Progress System)
**Test Each AI Function:**
1. Keywords → Clustering (auto_cluster.py)
- Verify `details.items_total`, `details.items_processed`, `details.items_created` appear in state
- Verify frontend displays counts without regex
- **Expected:** "50 keywords processed → 8 clusters created"
2. Clusters → Ideas (generate_ideas.py)
- Verify cluster names in `details.current_item_name`
- Verify idea counts
- **Expected:** "3 clusters processed → 12 ideas created"
3. Tasks → Content (generate_content.py)
- Verify task title and word count in details
- **Expected:** "Task: How to Train Your Dog → 1500 words generated"
4. Content → Image Prompts (generate_image_prompts.py)
- Verify prompt counts (featured + in-article)
- **Expected:** "1 featured + 5 in-article prompts created"
5. Image Prompts → Images (generate_images.py)
- Verify image generation progress
- **Expected:** "Generating image 3/6"
### Phase 3-4 Testing (UX Components)
**Test Each New Component:**
1. **StepBanner**
- Verify workflow position shown correctly
- Verify completed steps clickable, navigate correctly
- Verify current step highlighted
2. **HelperNotification**
- Verify dismissal persists in localStorage
- Verify actions work
- Verify doesn't show again after dismissal
3. **MetricsPanel**
- Verify collapse/expand works
- Verify tooltips show on hover
- Verify metrics update in real-time
### Phase 5 Testing (Automation)
**Test Automation Pipeline:**
1. Start automation run
2. Verify each stage shows accurate pending/processing/completed counts
3. Verify progress bar advances correctly
4. Pause and verify state persists
5. Resume and verify continues from correct stage
6. Verify stage result JSON contains all expected fields
---
## Success Criteria
### Backend
✅ All AI functions emit structured details via `tracker.update(meta={...})`
✅ Details include: `items_total`, `items_processed`, `items_created`, `current_item_name`
✅ Celery task state includes details in `meta.details`
### Frontend
✅ useProgressModal reads details directly from `state.meta.details`
✅ No regex parsing in useProgressModal or ProgressModal
✅ Progress messages show accurate counts
✅ All new UX components created and working
### UX
✅ Users see workflow step via StepBanner
✅ Users get contextual help via HelperNotification
✅ Users see metrics summary via MetricsPanel
✅ Automation progress accurate and transparent
---
## Implementation Notes
1. **Backwards Compatibility**: New `meta.details` is additive, doesn't break existing code
2. **No DB Changes**: All changes are in application layer (tracker, frontend)
3. **localStorage Management**: Consider adding "Reset all helpers" in settings
4. **Performance**: MetricsPanel uses existing data, no extra API calls
5. **Dark Mode**: All new components must have dark mode styles
6. **Accessibility**: All new components must support keyboard navigation
---
**End of Reality-Based Action Plan**

7
PENDING-ISSUES.md
View File

@@ -13,3 +13,10 @@
**Status:** Fix implemented by calling `admin_site.each_context()` in `Igny8ModelAdmin._inject_sidebar_context()` to get full Unfold branding context. Needs testing verification. **Status:** Fix implemented by calling `admin_site.each_context()` in `Igny8ModelAdmin._inject_sidebar_context()` to get full Unfold branding context. Needs testing verification.
--- ---
## 🔴 AI FUunctions progress modals texts and counts to be fixed
## 🔴 AUTOAMTION queue when run manualy completed count to be fixed, and progress abr to be imrpoved and fixed based on actual stage and all other data have bugs
## 🔴 Improve the metrics below tbale son all module pages, with actiionable metrics and steps
## 🔴 Align prompts with teh strategy

547
docs/PRE-LAUNCH/ITEM-1-STATUS-MODULES.md
View File

@@ -1,547 +0,0 @@
# Item 1: Status Modules for AI Planner and Writer
**Priority:** Critical
**Target:** Production Launch
**Last Updated:** December 11, 2025
---
## Overview
Standardize and fix status tracking across Planner and Writer modules to ensure consistent, accurate, and real-time progress updates for all AI operations. This includes backend status management, frontend progress displays, WebSocket/polling systems, and error handling.
---
## Current Implementation Analysis
### Backend Status System
#### Status Fields in Models
**Location:** `backend/igny8_core/business/planning/models.py` and `backend/igny8_core/business/content/models.py`
| Model | Current Status Choices | Issues |
|-------|----------------------|---------|
| **Keywords** | `new`, `mapped` | Limited states, no processing states |
| **Clusters** | `new`, `mapped` | No active/processing states |
| **ContentIdeas** | `new`, `queued`, `completed` | Missing `processing`, `failed` states |
| **Tasks** | `queued`, `completed` | Missing intermediate states (processing, failed, cancelled) |
| **Content** | `draft`, `review`, `published` | Separate from generation status |
| **AutomationRun** | `running`, `paused`, `cancelled`, `completed`, `failed` | Well-defined but isolated |
**Key Problem:** Status choices are inconsistent across models. Some models track generation status, others track editorial status, causing confusion.
---
### AI Function Progress Tracking
**Location:** `backend/igny8_core/ai/tracker.py`, `backend/igny8_core/ai/engine.py`
#### Progress Phases
All AI functions emit standard progress phases:
| Phase | Default Label | Percentage Range | Purpose |
|-------|--------------|------------------|---------|
| `INIT` | Initializing... | 0-10% | Validation, setup |
| `PREP` | Preparing data... | 10-20% | Data loading, preparation |
| `AI_CALL` | Processing with AI... | 20-80% | AI model execution |
| `PARSE` | Parsing response... | 80-90% | Response validation, parsing |
| `SAVE` | Saving results... | 90-100% | Database operations |
| `DONE` | Complete! | 100% | Final state |
**Implementation:**
- `StepTracker` class emits step-level events
- `ProgressTracker` class manages Celery task state updates
- `AIEngine` orchestrates phase transitions
**Issues:**
- Phase messages are generic and don't reflect actual counts
- Percentage calculations sometimes inconsistent
- No clear error state handling in phases
---
### Frontend Progress Modal System
**Location:** `frontend/src/hooks/useProgressModal.ts`, `frontend/src/components/common/ProgressModal.tsx`
#### Current Modal Implementation
**Hook:** `useProgressModal`
- Manages modal state, task polling, progress updates
- Polls backend every 1 second for task status
- Maps backend phases to user-friendly messages
- Tracks step logs for debugging
**Modal Component:** `ProgressModal`
- Displays progress bar with percentage
- Shows current phase and message
- Lists step-by-step progress visually
- Provides cancel/close actions
#### Step Mapping Logic
**Function:** `getStepInfo()` in `useProgressModal.ts`
Maps backend phases to:
- User-friendly messages with dynamic counts
- Adjusted percentages for smooth progression
- Function-specific labels (clustering vs. ideas vs. content)
**Problems Identified:**
1. **Inaccurate counts in messages**: Extracts counts from messages using regex, often missing or incorrect
2. **Generic phase labels**: Messages don't always reflect what's actually happening
3. **Inconsistent percentage progression**: Jumps or stalls during AI_CALL phase
4. **Status text not aligned**: Labels in modal don't match backend phase names
---
### Progress Display by Page
| Page | Location | Progress Display | Issues |
|------|----------|------------------|--------|
| Keywords | `frontend/src/pages/Planner/Keywords.tsx` | ProgressModal, AI logs | Clustering progress text inaccurate |
| Clusters | `frontend/src/pages/Planner/Clusters.tsx` | ProgressModal only | No cluster generation function exists |
| Ideas | `frontend/src/pages/Planner/Ideas.tsx` | ProgressModal, reload on complete | Text shows generic "preparing clusters" |
| Tasks | `frontend/src/pages/Writer/Tasks.tsx` | ProgressModal, AI logs | Content generation text inaccurate |
| Content | `frontend/src/pages/Writer/Content.tsx` | ProgressModal | Missing real-time updates |
| Images | `frontend/src/pages/Writer/Images.tsx` | ImageQueueModal | Uses separate queue-based system |
---
### Known Bugs and Issues
From `Pre-Launch-Pending.md` Known Bugs section:
#### Manual AI Function Run Modals
**Affected Pages:** Keywords, Clusters, Ideas, Tasks
**Issues:**
- Progress modal texts are not accurate for specific functions
- Dynamic variables not properly populated with counts
- Generic messages instead of function-specific ones
**Example:**
- Clustering: Should say "Mapping 45 keywords into clusters" but shows "Preparing data..."
- Ideas: Should say "Generating ideas for 3 clusters" but shows generic text
---
#### Automation Wizard Progress
**Page:** Automation (`frontend/src/pages/Automation/AutomationWizard.tsx`)
**Issues:**
- Wrong queue items displayed
- Missing queue items for certain stages
- Progress bar doesn't progress properly
- Total in queue and processed counts buggy across stages
- Stage cards: Real-time metrics not optimized
---
### WebSocket vs Polling
**Current Implementation:** Polling-based
**Location:** `useProgressModal.ts` - polls task status every 1 second
**Note:** Code mentions WebSockets but current implementation uses REST API polling via `/api/v1/...` endpoints
**Improvement Needed:** Consider WebSocket implementation for real-time updates without polling overhead
---
## Required Changes
### A. Backend Status Standardization
#### 1. Unified Status Enum
**Create:** `backend/igny8_core/common/status_enums.py`
Define standard status choices for all modules:
```
GENERATION_STATUS_CHOICES = [
('pending', 'Pending'),
('queued', 'Queued'),
('processing', 'Processing'),
('completed', 'Completed'),
('failed', 'Failed'),
('cancelled', 'Cancelled'),
]
CONTENT_STATUS_CHOICES = [
('draft', 'Draft'),
('review', 'Review'),
('published', 'Published'),
('archived', 'Archived'),
]
```
**Action:** Add `generation_status` field to models that undergo AI processing:
- Keywords (for extraction/clustering operations)
- Clusters (for cluster generation, if implemented)
- ContentIdeas (for idea generation)
- Tasks (for content generation)
- Content (separate from editorial status)
#### 2. Enhanced Progress Details
**Update:** `backend/igny8_core/ai/tracker.py`
**Modifications:**
- `StepTracker.update_step()` should include:
- `items_total`: Total count of items being processed
- `items_processed`: Current count of items processed
- `current_item_name`: Name/title of current item
- `estimated_remaining_seconds`: Estimated time remaining
**Example Progress Payload:**
```json
{
"phase": "AI_CALL",
"message": "Clustering 45 keywords",
"percentage": 45,
"items_total": 45,
"items_processed": 20,
"current_item": "best seo tools",
"estimated_seconds_remaining": 120
}
```
#### 3. Function-Specific Progress Messages
**Update:** Each AI function's phase messages
**Files to modify:**
- `backend/igny8_core/ai/functions/auto_cluster.py`
- `backend/igny8_core/ai/functions/generate_ideas.py`
- `backend/igny8_core/ai/functions/generate_content.py`
- `backend/igny8_core/ai/functions/generate_image_prompts.py`
- `backend/igny8_core/ai/functions/generate_images.py`
**Changes per function:**
| Function | Phase | Accurate Message Template |
|----------|-------|---------------------------|
| auto_cluster | PREP | "Loading {count} keywords for clustering" |
| auto_cluster | AI_CALL | "Analyzing keyword relationships ({processed}/{total})" |
| auto_cluster | SAVE | "Creating {cluster_count} clusters" |
| generate_ideas | PREP | "Preparing {count} cluster(s) for idea generation" |
| generate_ideas | AI_CALL | "Generating ideas for cluster: {cluster_name}" |
| generate_ideas | SAVE | "Created {idea_count} content ideas" |
| generate_content | PREP | "Preparing task: {task_title}" |
| generate_content | AI_CALL | "Writing {word_count}-word article" |
| generate_content | SAVE | "Saving article: {title}" |
| generate_image_prompts | PREP | "Mapping content for {count} image prompts" |
| generate_image_prompts | AI_CALL | "Writing featured image prompt" |
| generate_image_prompts | PARSE | "Writing {count} in-article image prompts" |
| generate_images | PREP | "Preparing {count} images for generation" |
| generate_images | AI_CALL | "Generating image {current}/{total}: {prompt}" |
**Implementation:** Use `self.tracker.update_step()` with dynamic values from actual data
---
### B. Frontend Progress Display Improvements
#### 1. Update ProgressModal Messages
**File:** `frontend/src/components/common/ProgressModal.tsx`
**Function:** `getSuccessMessage()`
**Current Issues:**
- Generic success messages
- Count extraction from logs is unreliable
- Doesn't distinguish between function types clearly
**Required Changes:**
| Function | Current Success Message | Improved Success Message |
|----------|------------------------|--------------------------|
| Clustering | "Clustering complete" | "45 keywords mapped into 8 clusters" |
| Ideas | "Content ideas created successfully" | "Generated 12 content ideas for 3 clusters" |
| Content | "Article drafted successfully" | "1,245-word article drafted successfully" |
| Image Prompts | Generic text | "1 Featured Image + 4 In-article Image Prompts ready" |
| Images | "Images generated successfully" | "5 images generated successfully" |
**Implementation:** Extract counts from backend `details` field, not from message strings
---
#### 2. Improve Step Mapping Logic
**File:** `frontend/src/hooks/useProgressModal.ts`
**Function:** `getStepInfo()`
**Problems:**
- Complex regex-based count extraction from messages
- Percentage calculation doesn't match backend
- Messages manually constructed instead of using backend messages
**Solution:**
- **Rely on backend `details` field** for counts:
```typescript
const { items_total, items_processed, current_item } = progress.details;
```
- Use backend message directly when it contains proper counts
- Only enhance message format, don't reconstruct it
**Example Refactor:**
```typescript
// OLD: Complex regex extraction
const keywordCount = extractNumber(/(\d+)\s+keyword/i, message);
// NEW: Use backend details
const percentage = details.percentage;
const message = details.message; // Already has counts
const itemName = details.current_item || '';
```
---
#### 3. Real-Time Updates Per Page
**Pages to update:** Keywords, Ideas, Tasks, Content
**Current behavior:** Modal closes, then data reloads
**Improved behavior:**
- Show inline status badge in table rows during processing
- Update table row in real-time as status changes
- Auto-reload specific rows instead of entire table
- Show notification when task completes in background
**Implementation approach:**
- Subscribe to task updates while modal is open
- Update table state with new status
- Add `status` badge column to tables showing:
- 🔄 Processing (animated)
- ✅ Completed
- ❌ Failed
- ⏸️ Paused
---
### C. Automation Wizard Progress Fixes
**File:** `frontend/src/pages/Automation/AutomationWizard.tsx`
**Issues identified:**
1. Wrong queue items displayed
2. Missing queue items
3. Progress bar inaccurate
4. Stage counts buggy
**Required fixes:**
#### 1. Stage Progress Cards
**Current location:** Stage cards display in wizard
**Problems:**
- Counts mismatch between backend and frontend
- "In Queue" vs "Processed" numbers don't add up
- Real-time updates missing
**Fix:**
- Pull stage results from `AutomationRun.stage_X_result` JSON fields
- Display accurate counts:
```
Stage 1: Clustering
- Keywords Loaded: 45
- Clusters Created: 8
- Status: Completed
```
#### 2. Queue Item Display
**Fix queue item tracking:**
- Each stage should show items being processed
- Items should appear in queue before processing
- Items should move to "Completed" list after processing
- Failed items should show in "Failed" section with retry option
**Data structure per stage:**
```json
{
"pending": ["item1", "item2"],
"processing": ["item3"],
"completed": ["item4", "item5"],
"failed": [{"item": "item6", "error": "..."}]
}
```
#### 3. Progress Bar Calculation
**Fix:**
- Base percentage on actual completed stages
- Within-stage percentage based on items processed
- Formula: `(completed_stages * 100 / 7) + (current_stage_progress / 7)`
---
### D. Error State Handling
#### Backend Error Logging
**Location:** `backend/igny8_core/ai/engine.py`
**Current:** Errors logged but status not always propagated
**Improvement:**
- Always set task state to `FAILURE` on error
- Include detailed error in state metadata
- Log error with context (function, input data, trace)
**Error payload structure:**
```json
{
"status": "error",
"error_type": "InsufficientCreditsError",
"error_message": "Insufficient credits. Required: 45, Available: 20",
"phase": "INIT",
"timestamp": "2025-12-11T10:30:00Z"
}
```
#### Frontend Error Display
**Files:** All progress modals
**Current:** Generic "Task failed" message
**Improvement:**
- Show specific error type
- Display user-friendly error message
- Provide actionable next steps
- Show "Retry" button for recoverable errors
**Error message mapping:**
| Error Type | User Message | Action |
|------------|--------------|--------|
| InsufficientCreditsError | "Not enough credits. You need X credits." | "Purchase Credits" button |
| ValidationError | "Invalid data: {details}" | "Fix Data" button |
| AIProviderError | "AI service temporarily unavailable" | "Retry" button |
| TimeoutError | "Request timed out. Try again." | "Retry" button |
---
## QA Testing Checklist
### Manual AI Function Tests
Test each function on each page:
| Page | Function | Test Case | Expected Result |
|------|----------|-----------|-----------------|
| Keywords | Clustering | Select 50 keywords, run clustering | Modal shows "Clustering 50 keywords", creates clusters, shows "50 keywords mapped into X clusters" |
| Ideas | Generate Ideas | Select 3 clusters, generate ideas | Modal shows "Generating ideas for 3 clusters", creates ideas, shows "Generated X ideas for 3 clusters" |
| Tasks | Generate Content | Select 1 task, generate content | Modal shows "Writing {word_count}-word article", creates content, shows "{word_count}-word article drafted" |
| Tasks | Generate Image Prompts | Select 1 content, generate prompts | Modal shows "Mapping content for X prompts", shows "1 Featured + X In-article prompts ready" |
| Images | Generate Images | Select content with prompts, generate | ImageQueueModal shows each image generating, completes all |
### Edge Case Tests
| Scenario | Expected Behavior |
|----------|-------------------|
| Task cancelled mid-process | Status updates to "cancelled", modal shows cancellation message |
| Task fails due to insufficient credits | Error modal with credit purchase link |
| Task fails due to validation error | Error modal with specific validation issue |
| Multiple tasks running simultaneously | Each has independent progress tracking |
| Network disconnection during task | Polling resumes when reconnected, catches up with status |
| Task completes while modal is closed | Table updates status automatically, notification shown |
### Automation Wizard Tests
| Stage | Test Case | Expected Result |
|-------|-----------|-----------------|
| Stage 1 | Run clustering on 100 keywords | Shows batch progress, creates clusters, displays accurate counts |
| Stage 2 | Generate ideas for 10 clusters | Shows cluster-by-cluster progress, creates ideas |
| Stage 3 | Convert ideas to tasks | Shows conversion progress, creates tasks |
| Stage 4 | Generate content for tasks | Shows content generation per task |
| Stage 5 | Generate image prompts | Shows prompt creation progress |
| Stage 6 | Generate images | Shows image generation progress |
| Stage 7 | Final review | Shows completed content ready for publishing |
| Pause/Resume | Pause at stage 3, resume | Continues from where it left off |
| Cancel | Cancel at stage 4 | Stops processing, shows cancelled status |
---
## Implementation Priority
### Phase 1: Backend Fixes (Week 1)
1. ✅ Create unified status enums
2. ✅ Add `generation_status` fields to models
3. ✅ Update AI function progress messages with accurate counts
4. ✅ Enhance tracker to include item counts and current item
### Phase 2: Frontend Display (Week 1-2)
1. ✅ Update ProgressModal success messages
2. ✅ Refactor step mapping logic to use backend details
3. ✅ Add real-time status badges to table rows
4. ✅ Improve error message display
### Phase 3: Automation Wizard (Week 2)
1. ✅ Fix stage progress cards
2. ✅ Fix queue item tracking
3. ✅ Fix progress bar calculation
4. ✅ Add pause/resume/cancel controls
### Phase 4: QA & Polish (Week 2-3)
1. ✅ Run all manual tests
2. ✅ Run all edge case tests
3. ✅ Run automation wizard tests
4. ✅ Fix any discovered issues
5. ✅ Performance optimization
---
## Success Metrics
- ✅ All progress modals show accurate counts and messages
- ✅ Progress percentages match actual work completed
- ✅ Automation wizard shows correct queue states
- ✅ Error messages are specific and actionable
- ✅ Real-time updates work consistently
- ✅ No user-reported confusion about task status
- ✅ All QA test cases pass
---
## Related Files Reference
### Backend
- `backend/igny8_core/business/planning/models.py` - Keywords, Clusters, ContentIdeas models
- `backend/igny8_core/business/content/models.py` - Tasks, Content models
- `backend/igny8_core/business/automation/models.py` - AutomationRun model
- `backend/igny8_core/ai/tracker.py` - ProgressTracker, StepTracker
- `backend/igny8_core/ai/engine.py` - AIEngine orchestration
- `backend/igny8_core/ai/functions/*.py` - All AI function implementations
### Frontend
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook
- `frontend/src/components/common/ProgressModal.tsx` - Progress modal component
- `frontend/src/components/common/ImageQueueModal.tsx` - Image generation modal
- `frontend/src/pages/Planner/Keywords.tsx` - Keywords page with clustering
- `frontend/src/pages/Planner/Ideas.tsx` - Ideas page with idea generation
- `frontend/src/pages/Writer/Tasks.tsx` - Tasks page with content generation
- `frontend/src/pages/Writer/Content.tsx` - Content page
- `frontend/src/pages/Writer/Images.tsx` - Images page
- `frontend/src/pages/Automation/AutomationWizard.tsx` - Automation wizard
---
## Notes
- Consider WebSocket implementation for production to reduce polling overhead
- Status badges in tables should be subtle and not distract from main content
- Error messages should prioritize user understanding over technical accuracy
- Progress messages should be encouraging and informative
- Real-time updates should not cause UI flickering or performance issues

713
docs/PRE-LAUNCH/ITEM-3-PROMPT-OPTIMIZATION.md
View File

@@ -1,713 +0,0 @@
# Item 3: Prompt Improvement and Model Optimization
**Priority:** High
**Target:** Production Launch
**Last Updated:** December 11, 2025
---
## Overview
Redesign and optimize all AI prompts for clustering, idea generation, content generation, and image prompt extraction to achieve:
- Extreme accuracy and consistent outputs
- Faster processing with optimized token usage
- Correct word count adherence (500, 1000, 1500 words)
- Improved clustering quality and idea relevance
- Better image prompt clarity and relevance
---
## Current Prompt System Architecture
### Prompt Registry
**Location:** `backend/igny8_core/ai/prompts.py`
**Class:** `PromptRegistry`
**Hierarchy** (resolution order):
1. Task-level `prompt_override` (if exists on specific task)
2. Database prompt from `AIPrompt` model (account-specific)
3. Default fallback from `PromptRegistry.DEFAULT_PROMPTS`
**Storage:**
- Default prompts: Hardcoded in `prompts.py`
- Account overrides: `system_aiprompt` database table
- Task overrides: `prompt_override` field on task object
---
## Current Prompts Analysis
### 1. Clustering Prompt
**Function:** `auto_cluster`
**File:** `backend/igny8_core/ai/functions/auto_cluster.py`
**Prompt Key:** `'clustering'`
#### Current Prompt Structure
**Approach:** Semantic strategist + intent-driven clustering
**Key Instructions:**
- Return single JSON with "clusters" array
- Each cluster: name, description, keywords[]
- Multi-dimensional grouping (intent, use-case, function, persona, context)
- Model real search behavior and user journeys
- Avoid superficial groupings and duplicates
- 3-10 keywords per cluster
**Strengths:**
✅ Clear JSON output format
✅ Detailed grouping logic with dimensions
✅ Emphasis on semantic strength over keyword matching
✅ User journey modeling (Problem → Solution, General → Specific)
**Issues:**
❌ Very long prompt (~400+ tokens) - may confuse model
❌ No examples provided - model must guess formatting
❌ Doesn't specify what to do with outliers explicitly
❌ No guidance on cluster count (outputs variable)
❌ Description length not constrained
**Real-World Performance Issues:**
- Sometimes creates too many small clusters (1-2 keywords each)
- Inconsistent cluster naming convention
- Descriptions sometimes generic ("Keywords related to...")
---
### 2. Idea Generation Prompt
**Function:** `generate_ideas`
**File:** `backend/igny8_core/ai/functions/generate_ideas.py`
**Prompt Key:** `'ideas'`
#### Current Prompt Structure
**Approach:** SEO-optimized content ideas + outlines
**Key Instructions:**
- Input: Clusters + Keywords
- Output: JSON "ideas" array
- 1 cluster_hub + 2-4 supporting ideas per cluster
- Fields: title, description, content_type, content_structure, cluster_id, estimated_word_count, covered_keywords
- Outline format: intro (hook + 2 paragraphs), 5-8 H2 sections with 2-3 H3s each
- Content mixing: paragraphs, lists, tables, blockquotes
- No bullets/lists at start
- Professional tone, no generic phrasing
**Strengths:**
✅ Detailed outline structure
✅ Content mixing guidance (lists, tables, blockquotes)
✅ Clear JSON format
✅ Tone guidelines
**Issues:**
❌ Very complex prompt (600+ tokens)
❌ Outline format too prescriptive (might limit creativity)
❌ No examples provided
❌ Estimated word count often inaccurate (too high or too low)
❌ "hook" guidance unclear (what makes a good hook?)
❌ Content structure validation not enforced
**Real-World Performance Issues:**
- Generated ideas sometimes too similar within cluster
- Outlines don't always respect structure types (e.g., "review" vs "guide")
- covered_keywords field sometimes empty or incorrect
- cluster_hub vs supporting ideas distinction unclear
---
### 3. Content Generation Prompt
**Function:** `generate_content`
**File:** `backend/igny8_core/ai/functions/generate_content.py`
**Prompt Key:** `'content_generation'`
#### Current Prompt Structure
**Approach:** Editorial content strategist
**Key Instructions:**
- Output: JSON {title, content (HTML)}
- Introduction: 1 italic hook (30-40 words) + 2 paragraphs (50-60 words each), no headings
- H2 sections: 5-8 total, 250-300 words each
- Section format: 2 narrative paragraphs → list/table → optional closing paragraph → 2-3 subsections
- Vary list/table types
- Never start section with list/table
- Tone: professional, no passive voice, no generic intros
- Keyword usage: natural in title, intro, headings
**Strengths:**
✅ Detailed structure guidance
✅ Strong tone/style rules
✅ HTML output format
✅ Keyword integration guidance
**Issues:**
❌ **Word count not mentioned in prompt** - critical flaw
❌ No guidance on 500 vs 1000 vs 1500 word versions
❌ Hook word count (30-40) + paragraph counts (50-60 × 2) don't scale proportionally
❌ Section word count (250-300) doesn't adapt to total target
❌ No example output
❌ Content structure (article vs guide vs review) not clearly differentiated
❌ Table column guidance missing (what columns? how many?)
**Real-World Performance Issues:**
- **Output length wildly inconsistent** (generates 800 words when asked for 1500)
- Introductions sometimes have headings despite instructions
- Lists appear at start of sections
- Table structure unclear (random columns)
- Doesn't adapt content density to word count
---
### 4. Image Prompt Extraction
**Function:** `generate_image_prompts`
**File:** `backend/igny8_core/ai/functions/generate_image_prompts.py`
**Prompt Key:** `'image_prompt_extraction'`
#### Current Prompt Structure
**Approach:** Extract visual descriptions from article
**Key Instructions:**
- Input: article title + content
- Output: JSON {featured_prompt, in_article_prompts[]}
- Extract featured image (main topic)
- Extract up to {max_images} in-article images
- Each prompt detailed for image generation (visual elements, style, mood, composition)
**Strengths:**
✅ Clear structure
✅ Separates featured vs in-article
✅ Emphasizes detail in descriptions
**Issues:**
❌ No guidance on what makes a good image prompt
❌ No style/mood specifications
❌ Doesn't specify where in article to place images
❌ No examples
❌ "Detailed enough" is subjective
**Real-World Performance Issues:**
- Prompts sometimes too generic ("Image of a person using a laptop")
- No context from article content (extracts irrelevant visuals)
- Featured image prompt sometimes identical to in-article prompt
- No guidance on image diversity (all similar)
---
### 5. Image Generation Template
**Prompt Key:** `'image_prompt_template'`
#### Current Template
**Approach:** Template-based prompt assembly
**Format:**
```
Create a high-quality {image_type} image... "{post_title}"... {image_prompt}...
Focus on realistic, well-composed scene... lifestyle/editorial web content...
Avoid text, watermarks, logos... **not blurry.**
```
**Issues:**
❌ {image_type} not always populated
❌ "high-quality" and "not blurry" redundant/unclear
❌ No style guidance (photographic, illustration, 3D, etc.)
❌ No aspect ratio specification
---
## Required Improvements
### A. Clustering Prompt Redesign
#### Goals
- Reduce prompt length by 30-40%
- Add 2-3 concrete examples
- Enforce consistent cluster count (5-15 clusters ideal)
- Standardize cluster naming (title case, descriptive)
- Limit description to 20-30 words
#### Proposed Structure
**Section 1: Role & Task** (50 tokens)
- Clear, concise role definition
- Task: group keywords into intent-driven clusters
**Section 2: Output Format with Example** (100 tokens)
- JSON structure
- Show 1 complete example cluster
- Specify exact field requirements
**Section 3: Clustering Rules** (150 tokens)
- List 5-7 key rules (bullet format)
- Keyword-first approach
- Intent dimensions (brief)
- Quality thresholds (3-10 keywords per cluster)
- No duplicates
**Section 4: Quality Checklist** (50 tokens)
- Checklist of 4-5 validation points
- Model self-validates before output
**Total:** ~350 tokens (vs current ~420)
#### Example Output Format to Include
```json
{
"clusters": [
{
"name": "Organic Bedding Benefits",
"description": "Health, eco-friendly, and comfort aspects of organic cotton bedding materials",
"keywords": ["organic sheets", "eco-friendly bedding", "chemical-free cotton", "hypoallergenic sheets", "sustainable bedding"]
}
]
}
```
---
### B. Idea Generation Prompt Redesign
#### Goals
- Simplify outline structure (less prescriptive)
- Add examples of cluster_hub vs supporting ideas
- Better covered_keywords extraction
- Adaptive word count estimation
- Content structure differentiation
#### Proposed Structure
**Section 1: Role & Objective** (40 tokens)
- SEO content strategist
- Task: generate content ideas from clusters
**Section 2: Output Format with Examples** (150 tokens)
- Show 1 cluster_hub example
- Show 1 supporting idea example
- Highlight key differences
**Section 3: Idea Generation Rules** (100 tokens)
- 1 cluster_hub (comprehensive, authoritative)
- 2-4 supporting ideas (specific angles)
- Word count: 1500-2200 for hubs, 1000-1500 for supporting
- covered_keywords: extract from cluster keywords
**Section 4: Outline Guidance** (100 tokens)
- Simplified: Intro + 5-8 sections + Conclusion
- Section types by content_structure:
- article: narrative + data
- guide: step-by-step + tips
- review: pros/cons + comparison
- listicle: numbered + categories
- comparison: side-by-side + verdict
**Total:** ~390 tokens (vs current ~610)
---
### C. Content Generation Prompt Redesign
**Most Critical Improvement:** Word Count Adherence
#### Goals
- **Primary:** Generate exact word count (±5% tolerance)
- Scale structure proportionally to word count
- Differentiate content structures clearly
- Improve HTML quality and consistency
- Better keyword integration
#### Proposed Adaptive Word Count System
**Word Count Targets:**
- 500 words: Short-form (5 sections × 80 words + intro/outro 60 words)
- 1000 words: Standard (6 sections × 140 words + intro/outro 120 words)
- 1500 words: Long-form (7 sections × 180 words + intro/outro 180 words)
**Prompt Variable Replacement:**
Before sending to AI, calculate:
- `{TARGET_WORD_COUNT}` - from task.word_count
- `{INTRO_WORDS}` - 60 / 120 / 180 based on target
- `{SECTION_COUNT}` - 5 / 6 / 7 based on target
- `{SECTION_WORDS}` - 80 / 140 / 180 based on target
- `{HOOK_WORDS}` - 25 / 35 / 45 based on target
#### Proposed Structure
**Section 1: Role & Objective** (30 tokens)
```
You are an editorial content writer. Generate a {TARGET_WORD_COUNT}-word article...
```
**Section 2: Word Count Requirements** (80 tokens)
```
CRITICAL: The content must be exactly {TARGET_WORD_COUNT} words (±5% tolerance).
Structure breakdown:
- Introduction: {INTRO_WORDS} words total
- Hook (italic): {HOOK_WORDS} words
- Paragraphs: 2 × ~{INTRO_WORDS/2} words each
- Main Sections: {SECTION_COUNT} H2 sections
- Each section: {SECTION_WORDS} words
- Conclusion: 60 words
Word count validation: Count words in final output and adjust if needed.
```
**Section 3: Content Flow & HTML** (120 tokens)
- Detailed structure per section
- HTML tag usage (<p>, <h2>, <h3>, <ul>, <ol>, <table>)
- Formatting rules
**Section 4: Style & Quality** (80 tokens)
- Tone guidance
- Keyword usage
- Avoid generic phrases
- Examples of good vs bad openings
**Section 5: Content Structure Types** (90 tokens)
- article: {structure description}
- guide: {structure description}
- review: {structure description}
- comparison: {structure description}
- listicle: {structure description}
- cluster_hub: {structure description}
**Section 6: Output Format with Example** (100 tokens)
- JSON structure
- Show abbreviated example with proper HTML
**Total:** ~500 tokens (vs current ~550, but much more precise)
---
### D. Image Prompt Improvements
#### Goals
- Generate visually diverse prompts
- Better context from article content
- Specify image placement guidelines
- Improve prompt detail and clarity
#### Proposed Extraction Prompt Structure
**Section 1: Task & Context** (50 tokens)
```
Extract image prompts from this article for visual content placement.
Article: {title}
Content: {content}
Required: 1 featured + {max_images} in-article images
```
**Section 2: Image Types & Guidelines** (100 tokens)
```
Featured Image:
- Hero visual representing article's main theme
- Broad, engaging, high-quality
- Should work at large sizes (1200×630+)
In-Article Images (place strategically):
1. After introduction
2. Mid-article (before major H2 sections)
3. Supporting specific concepts or examples
4. Before conclusion
Each prompt must describe:
- Subject & composition
- Visual style (photographic, minimal, editorial)
- Mood & lighting
- Color palette suggestions
- Avoid: text, logos, faces (unless relevant)
```
**Section 3: Prompt Quality Rules** (80 tokens)
- Be specific and descriptive (not generic)
- Include scene details, angles, perspective
- Specify lighting, time of day if relevant
- Mention style references
- Ensure diversity across all images
- No duplicate concepts
**Section 4: Output Format** (50 tokens)
- JSON structure
- Show example with good vs bad prompts
#### Proposed Template Prompt Improvement
Replace current template with:
```
A {style} photograph for "{post_title}". {image_prompt}.
Composition: {composition_hint}. Lighting: {lighting_hint}.
Mood: {mood}. Style: clean, modern, editorial web content.
No text, watermarks, or logos.
```
Where:
- {style} - photographic, minimalist, lifestyle, etc.
- {composition_hint} - center-framed, rule-of-thirds, wide-angle, etc.
- {lighting_hint} - natural daylight, soft indoor, dramatic, etc.
- {mood} - professional, warm, energetic, calm, etc.
---
## Implementation Plan
### Phase 1: Clustering Prompt (Week 1)
**Tasks:**
1. ✅ Draft new clustering prompt with examples
2. ✅ Test with sample keyword sets (20, 50, 100 keywords)
3. ✅ Compare outputs: old vs new
4. ✅ Validate cluster quality (manual review)
5. ✅ Update `PromptRegistry.DEFAULT_PROMPTS['clustering']`
6. ✅ Deploy and monitor
**Success Criteria:**
- Consistent cluster count (5-15)
- No single-keyword clusters
- Clear, descriptive names
- Concise descriptions (20-30 words)
- 95%+ of keywords clustered
---
### Phase 2: Idea Generation Prompt (Week 1-2)
**Tasks:**
1. ✅ Draft new ideas prompt with examples
2. ✅ Test with 5-10 clusters
3. ✅ Validate cluster_hub vs supporting idea distinction
4. ✅ Check covered_keywords accuracy
5. ✅ Verify content_structure alignment
6. ✅ Update `PromptRegistry.DEFAULT_PROMPTS['ideas']`
7. ✅ Deploy and monitor
**Success Criteria:**
- Clear distinction between hub and supporting ideas
- Accurate covered_keywords extraction
- Appropriate word count estimates
- Outlines match content_structure type
- No duplicate ideas within cluster
---
### Phase 3: Content Generation Prompt (Week 2)
**Tasks:**
1. ✅ Draft new content prompt with word count logic
2. ✅ Implement dynamic variable replacement in `build_prompt()`
3. ✅ Test with 500, 1000, 1500 word targets
4. ✅ Validate actual word counts (automated counting)
5. ✅ Test all content_structure types
6. ✅ Verify HTML quality and consistency
7. ✅ Update `PromptRegistry.DEFAULT_PROMPTS['content_generation']`
8. ✅ Deploy and monitor
**Code Change Required:**
**File:** `backend/igny8_core/ai/functions/generate_content.py`
**Method:** `build_prompt()`
**Add word count calculation:**
```python
def build_prompt(self, data: Any, account=None) -> str:
task = data if not isinstance(data, list) else data[0]
# Calculate adaptive word count parameters
target_words = task.word_count or 1000
if target_words <= 600:
intro_words = 60
section_count = 5
section_words = 80
hook_words = 25
elif target_words <= 1200:
intro_words = 120
section_count = 6
section_words = 140
hook_words = 35
else:
intro_words = 180
section_count = 7
section_words = 180
hook_words = 45
# Get prompt and replace variables
prompt = PromptRegistry.get_prompt(
function_name='generate_content',
account=account,
task=task,
context={
'TARGET_WORD_COUNT': target_words,
'INTRO_WORDS': intro_words,
'SECTION_COUNT': section_count,
'SECTION_WORDS': section_words,
'HOOK_WORDS': hook_words,
# ... existing context
}
)
return prompt
```
**Success Criteria:**
- 95%+ of generated content within ±5% of target word count
- HTML structure consistent
- Content structure types clearly differentiated
- Keyword integration natural
- No sections starting with lists
---
### Phase 4: Image Prompt Improvements (Week 2-3)
**Tasks:**
1. ✅ Draft new extraction prompt with placement guidelines
2. ✅ Draft new template prompt with style variables
3. ✅ Test with 10 sample articles
4. ✅ Validate image diversity and relevance
5. ✅ Update both prompts in registry
6. ✅ Update `GenerateImagePromptsFunction` to use new template
7. ✅ Deploy and monitor
**Success Criteria:**
- No duplicate image concepts in same article
- Prompts are specific and detailed
- Featured image distinct from in-article images
- Image placement logically distributed
- Generated images relevant to content
---
## Prompt Versioning & Testing
### Version Control
**Recommendation:** Store prompt versions in database for A/B testing
**Schema:**
```python
class AIPromptVersion(models.Model):
prompt_type = CharField(choices=PROMPT_TYPE_CHOICES)
version = IntegerField()
prompt_value = TextField()
is_active = BooleanField(default=False)
created_at = DateTimeField(auto_now_add=True)
performance_metrics = JSONField(default=dict) # Track success rates
```
**Process:**
1. Test new prompt version alongside current
2. Compare outputs on same inputs
3. Measure quality metrics (manual + automated)
4. Gradually roll out if better
5. Keep old version as fallback
---
### Automated Quality Metrics
**Implement automated checks:**
| Metric | Check | Threshold |
|--------|-------|-----------|
| Word Count Accuracy | `abs(actual - target) / target` | < 0.05 (±5%) |
| HTML Validity | Parse with BeautifulSoup | 100% valid |
| Keyword Presence | Count keyword mentions | ≥ 3 for primary |
| Structure Compliance | Check H2/H3 hierarchy | Valid structure |
| Cluster Count | Number of clusters | 5-15 |
| Cluster Size | Keywords per cluster | 3-10 |
| No Duplicates | Keyword appears once | 100% unique |
**Log results:**
- Track per prompt version
- Identify patterns in failures
- Use for prompt iteration
---
## Model Selection & Optimization
### Current Models
**Location:** `backend/igny8_core/ai/settings.py`
**Default Models per Function:**
- Clustering: GPT-4 (expensive but accurate)
- Ideas: GPT-4 (creative)
- Content: GPT-4 (quality)
- Image Prompts: GPT-3.5-turbo (simpler task)
- Images: DALL-E 3 / Runware
### Optimization Opportunities
**Cost vs Quality Tradeoffs:**
| Function | Current | Alternative | Cost Savings | Quality Impact |
|----------|---------|-------------|--------------|----------------|
| Clustering | GPT-4 | GPT-4-turbo | 50% | Minimal |
| Ideas | GPT-4 | GPT-4-turbo | 50% | Minimal |
| Content | GPT-4 | GPT-4-turbo | 50% | Test required |
| Image Prompts | GPT-3.5 | Keep | - | - |
**Recommendation:** Test GPT-4-turbo for all text generation tasks
- Faster response time
- 50% cost reduction
- Similar quality for structured outputs
---
## Success Metrics
- ✅ Word count accuracy: 95%+ within ±5%
- ✅ Clustering quality: No single-keyword clusters
- ✅ Idea generation: Clear hub vs supporting distinction
- ✅ HTML validity: 100%
- ✅ Keyword integration: Natural, not stuffed
- ✅ Image prompt diversity: No duplicates
- ✅ User satisfaction: Fewer manual edits needed
- ✅ Processing time: <10s for 1000-word article
- ✅ Credit cost: 30% reduction with model optimization
---
## Related Files Reference
### Backend
- `backend/igny8_core/ai/prompts.py` - Prompt registry and defaults
- `backend/igny8_core/ai/functions/auto_cluster.py` - Clustering function
- `backend/igny8_core/ai/functions/generate_ideas.py` - Ideas function
- `backend/igny8_core/ai/functions/generate_content.py` - Content function
- `backend/igny8_core/ai/functions/generate_image_prompts.py` - Image prompts
- `backend/igny8_core/ai/settings.py` - Model configuration
- `backend/igny8_core/modules/system/models.py` - AIPrompt model
### Testing
- Create test suite: `backend/igny8_core/ai/tests/test_prompts.py`
- Test fixtures with sample inputs
- Automated quality validation
- Performance benchmarks
---
## Notes
- All prompt changes should be tested on real data first
- Keep old prompts in version history for rollback
- Monitor user feedback on content quality
- Consider user-customizable prompt templates (advanced feature)
- Document prompt engineering best practices for team
- SAG clustering prompt (mentioned in original doc) to be handled separately as specialized architecture

745
docs/PRE-LAUNCH/ITEM-4-PAGE-FLOW-UX.md
View File

@@ -1,745 +0,0 @@
# Item 4: Page Flow and Workflow UX Improvements
**Priority:** High
**Target:** Production Launch
**Last Updated:** December 11, 2025
---
## Overview
Improve overall navigation, page structure, workflow clarity, and helper systems across all main user-facing areas including Setup, Workflow (Planner/Writer), Account, and Settings. This encompasses page flow standardization, helper notifications, tooltips, metrics panels, and notification systems.
---
## Current Implementation Analysis
### Main Menu Groups
**Location:** `frontend/src/layout/AppLayout.tsx` and navigation components
**Current Groups:**
1. **Setup** - Sites, Sectors, Integrations setup
2. **Workflow** - Planner (Keywords, Clusters, Ideas) and Writer (Tasks, Content, Images)
3. **Account** - Billing, Team, Profile
4. **Settings** - System settings, prompts, preferences
**Issues:**
- Inconsistent page header patterns across groups
- Some pages missing breadcrumbs or context
- Linear workflows (Planner → Writer) not clearly guided
- Cross-navigation between related pages unclear
---
### Page Header Component
**Location:** `frontend/src/components/common/PageHeader.tsx`
**Current Features:**
- Page title
- Last updated timestamp
- Site/Sector selector (can be hidden)
- Optional refresh button
- Optional badge
- Optional navigation tabs
**Strengths:**
✅ Standardized across modules
✅ Site/sector context management
✅ Auto-loads sectors when site changes
**Missing:**
❌ No breadcrumbs for deeper navigation
❌ No contextual help link
❌ No step indicators for workflows
❌ No quick action buttons section
---
### Module Navigation Tabs
**Location:** `frontend/src/components/navigation/ModuleNavigationTabs.tsx`
**Current Usage:**
- Planner pages: Keywords | Clusters | Ideas | Mapping
- Writer pages: Tasks | Content | Images
**Strengths:**
✅ Consistent tab design
✅ Active tab highlighting
✅ Responsive layout
**Missing:**
❌ No tab progress indicators
❌ No disabled state for incomplete prerequisites
❌ No tooltips explaining each tab
---
### Helper Systems
**Current State:**
- No systematic helper notifications
- No onboarding tooltips
- No inline guidance text
- No contextual help system
**What's Needed:**
- Welcome messages for first-time page visits
- Tooltips for icons and complex actions
- Inline help text under form fields
- Step-by-step workflow banners
- Contextual "Learn more" links
---
### Metrics Display
**Current Locations:**
- Dashboard: Overview metrics
- Individual pages: Table counts, filters
- Footer: Module-specific metrics (some pages only)
**Component:** `frontend/src/components/dashboard/ModuleMetricsFooter.tsx`
**Issues:**
- Inconsistent placement (some pages have footer metrics, some don't)
- No collapsible panel option
- Metrics not updated in real-time
- No metric explanations (tooltips)
---
## Required Improvements
### A. Standardize Page Structure
#### Universal Page Template
All workflow pages should follow this structure:
```tsx
<div className="page-container">
{/* 1. Page Header */}
<PageHeader
title="Page Title"
badge={badge}
navigation={<ModuleNavigationTabs tabs={tabs} />}
/>
{/* 2. Step Banner (if workflow page) */}
{isWorkflowPage && <StepBanner currentStep={2} totalSteps={5} />}
{/* 3. Helper Notification (first visit or contextual) */}
{showHelper && <HelperNotification message="..." />}
{/* 4. Page Actions Bar */}
<PageActionsBar
primaryAction={primaryAction}
secondaryActions={secondaryActions}
filters={filters}
/>
{/* 5. Main Content */}
<MainContent>
{/* Tables, forms, cards, etc. */}
</MainContent>
{/* 6. Bottom Metrics Panel (collapsible) */}
<MetricsPanel metrics={metrics} collapsible={true} />
</div>
```
---
### B. New Components to Create
#### 1. StepBanner Component
**File:** `frontend/src/components/workflow/StepBanner.tsx` (NEW)
**Purpose:** Show current step in multi-step workflows
**Props:**
```typescript
interface StepBannerProps {
currentStep: number;
totalSteps: number;
steps: Array<{
label: string;
href: string;
completed: boolean;
}>;
onStepClick?: (stepIndex: number) => void;
}
```
**UI Design:**
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2 of 5: Organize into Clusters
1. Extract Keywords ✓ → 2. Cluster Keywords ● → 3. Generate Ideas → 4. Create Content → 5. Publish
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```
**States:**
- ✓ Completed (green, clickable)
- ● Current (blue, active)
- Pending (gray, disabled or clickable)
**Usage:**
```tsx
<StepBanner
currentStep={2}
totalSteps={5}
steps={[
{ label: 'Extract Keywords', href: '/planner/keywords', completed: true },
{ label: 'Cluster Keywords', href: '/planner/clusters', completed: false },
{ label: 'Generate Ideas', href: '/planner/ideas', completed: false },
{ label: 'Create Content', href: '/writer/tasks', completed: false },
{ label: 'Publish', href: '/writer/content', completed: false },
]}
/>
```
---
#### 2. HelperNotification Component
**File:** `frontend/src/components/helper/HelperNotification.tsx` (NEW)
**Purpose:** Display contextual guidance and tips
**Types:**
- **Welcome**: First-time page visit
- **Info**: General information
- **Tip**: Best practice suggestion
- **Warning**: Potential issue
- **Success**: Achievement/milestone
**Props:**
```typescript
interface HelperNotificationProps {
type: 'welcome' | 'info' | 'tip' | 'warning' | 'success';
title: string;
message: string;
actions?: Array<{
label: string;
href?: string;
onClick?: () => void;
}>;
dismissible?: boolean;
onDismiss?: () => void;
}
```
**UI Design:**
```
┌─────────────────────────────────────────────────┐
│ 💡 Welcome to Keywords │
│ │
│ This is where you extract and manage SEO │
│ keywords. Start by adding keywords manually or │
│ importing from a CSV file. │
│ │
│ [Import CSV] [Learn More] [Got it] [×] │
└─────────────────────────────────────────────────┘
```
**Persistence:**
- Store dismissed state in localStorage per page
- Key: `helper_dismissed_{page_slug}`
- Option to show "Show hints again" in settings
---
#### 3. Tooltip Component Enhancement
**File:** `frontend/src/components/ui/Tooltip.tsx` (enhance existing)
**Add:**
- Keyboard accessibility (ESC to close, tab navigation)
- Delay before show (300ms hover)
- Max width constraints
- Arrow positioning
- Rich content support (not just text)
**Usage Examples:**
```tsx
{/* Simple text tooltip */}
<Tooltip content="Click to cluster all keywords">
<button>Cluster</button>
</Tooltip>
{/* Rich content tooltip */}
<Tooltip content={
<div>
<strong>Clustering</strong>
<p>Groups keywords by semantic similarity</p>
<a href="/help/clustering">Learn more →</a>
</div>
}>
<InfoIcon />
</Tooltip>
```
---
#### 4. InlineGuidance Component
**File:** `frontend/src/components/helper/InlineGuidance.tsx` (NEW)
**Purpose:** Small help text under form fields
**Props:**
```typescript
interface InlineGuidanceProps {
text: string;
type?: 'info' | 'warning' | 'error';
icon?: ReactNode;
}
```
**UI Design:**
```
[Input Field: Enter keyword]
ℹ️ Use lowercase, no special characters. Separate multiple keywords with commas.
```
**Usage:**
```tsx
<FormField label="Keywords">
<input type="text" placeholder="Enter keyword" />
<InlineGuidance
text="Use lowercase, no special characters. Separate multiple keywords with commas."
/>
</FormField>
```
---
#### 5. Enhanced MetricsPanel Component
**File:** `frontend/src/components/dashboard/ModuleMetricsPanel.tsx` (NEW)
**Improvements over current MetricsFooter:**
- Collapsible/expandable
- Fixed to bottom or inline at page bottom
- Real-time updates
- Tooltips explaining each metric
- Click to filter/drill-down
- Export metrics option
**Props:**
```typescript
interface MetricsPanelProps {
title: string;
metrics: Array<{
label: string;
value: string | number;
subtitle?: string;
icon?: ReactNode;
color?: string;
tooltip?: string;
onClick?: () => void;
}>;
collapsible?: boolean;
defaultCollapsed?: boolean;
position?: 'fixed' | 'relative';
}
```
**UI Design (Expanded):**
```
┌─────────────────────────────────────────────────────────┐
│ ▼ Keyword Metrics [Collapse] × │
├─────────────────────────────────────────────────────────┤
│ Total: 450 │ New: 120 │ Mapped: 330 │ Vol: 125K │
│ ℹ️ All saved │ ℹ️ Unmapped │ ℹ️ In cluster │ ℹ️ Search │
│ │ │ │ volume │
└─────────────────────────────────────────────────────────┘
```
**UI Design (Collapsed):**
```
▶ Keyword Metrics: 450 total, 120 new, 330 mapped
```
---
### C. Navigation Improvements
#### 1. Add Breadcrumbs
**Component:** `frontend/src/components/navigation/Breadcrumbs.tsx` (NEW)
**Usage:**
```tsx
<Breadcrumbs
items={[
{ label: 'Setup', href: '/setup' },
{ label: 'Sites', href: '/setup/sites' },
{ label: 'Site Settings' }, // Current page (no href)
]}
/>
```
**Display:**
```
Home > Setup > Sites > Site Settings
```
**Rules:**
- Show for pages 3+ levels deep
- Current page not clickable
- Auto-hide on mobile if > 3 items
---
#### 2. Cross-Navigation Links
**Add "Next Step" buttons between related pages:**
**Examples:**
| Current Page | Next Step Button | Action |
|--------------|------------------|---------|
| Keywords | "Cluster Keywords →" | Navigate to Clusters page |
| Clusters | "Generate Ideas →" | Navigate to Ideas page |
| Ideas | "Create Tasks →" | Convert ideas to tasks, navigate to Tasks |
| Tasks | "Generate Content →" | Trigger content generation |
| Content | "Generate Images →" | Navigate to Images page |
**Implementation:**
```tsx
<PageActionsBar
primaryAction={{
label: 'Cluster Keywords',
icon: <ArrowRightIcon />,
onClick: () => navigate('/planner/clusters'),
variant: 'primary'
}}
/>
```
---
#### 3. Workflow Progress Indicator (Global)
**Location:** Header or sidebar
**Display current workflow stage:**
```
Planner Workflow: Step 2/3
├─ Keywords ✓
├─ Clusters ● (in progress)
└─ Ideas
Writer Workflow: Not started
```
**Click to navigate to any completed or current step**
---
### D. Settings Reorganization
#### Current Problems
- Settings scattered across multiple locations
- Inconsistent categories
- Hard to find specific settings
#### Proposed Structure
**New Layout:** `frontend/src/pages/Settings/`
```
Settings/
├── index.tsx (Settings hub with cards)
├── General.tsx
├── Publishing.tsx
├── Pagination.tsx
├── Branding.tsx
├── WorkflowDefaults.tsx
├── Integrations.tsx
├── Notifications.tsx
├── Account.tsx
└── Billing.tsx (redirect to /billing)
```
**Settings Categories:**
| Category | Subcategories | What It Contains |
|----------|---------------|------------------|
| **General** | Account basics | Display name, timezone, language, date format |
| **Publishing** | WordPress settings | Connection, publishing defaults, SEO rules, auto-publish options |
| **Pagination** | List controls | Items per page for each module (keywords, clusters, ideas, etc.) |
| **Branding** | Appearance | Logo, colors, custom CSS (if applicable) |
| **Workflow Defaults** | AI settings | Default word count, tone, structure, author profile |
| **Integrations** | External services | API keys, OAuth connections, webhooks |
| **Notifications** | Alert preferences | Email alerts, in-app notifications, notification types |
| **Account & Billing** | Subscription | Plan details, usage, credits, invoices, team members |
**Settings Hub (Landing Page):**
```tsx
<SettingsHub>
<SettingsCard
icon={<GeneralIcon />}
title="General"
description="Basic account settings and preferences"
href="/settings/general"
/>
<SettingsCard
icon={<PublishIcon />}
title="Publishing"
description="WordPress connection and publishing rules"
href="/settings/publishing"
/>
{/* ... more cards */}
</SettingsHub>
```
---
### E. Notification System
#### Notification Types
| Type | Use Case | Display | Action |
|------|----------|---------|--------|
| **Progress** | Task started, queued, processing | Blue info icon | View progress modal |
| **Success** | Task completed, export success | Green check icon | View result |
| **Failure** | API error, validation failed | Red error icon | View error details, retry |
| **Action Required** | Missing settings, incomplete setup | Orange warning icon | Navigate to fix |
#### Notification Dropdown
**Location:** Header, bell icon with badge count
**Features:**
- List notifications chronologically
- Read/unread states
- Mark all as read
- Filter by type
- Clear all button
- Links to relevant pages
**Example Notification:**
```
┌─────────────────────────────────────────────┐
│ 🔵 Content Generation Started │
│ "Best SEO Tools 2024" - 2 minutes ago │
│ [View Progress] │
├─────────────────────────────────────────────┤
│ ✅ Clustering Completed │
│ 45 keywords → 8 clusters - 10 min ago │
│ [View Clusters] │
├─────────────────────────────────────────────┤
│ ❌ Image Generation Failed │
│ Insufficient credits - 1 hour ago │
│ [Purchase Credits] [Dismiss] │
└─────────────────────────────────────────────┘
```
#### Implementation
**Backend:**
- Create `Notification` model with account FK
- API endpoints: list, mark read, clear
- Emit notifications on key events
**Frontend:**
- `useNotifications()` hook
- `NotificationDropdown` component
- Auto-refresh every 30 seconds
- WebSocket support (future enhancement)
---
### F. Header Metrics Enhancement
**Current:** Basic stats in dashboard
**Improved:** Always-visible header metrics
**Location:** Right side of header (next to user menu)
**Display:**
```
[Icon] 3 In Progress | [Icon] 5 Completed | [Icon] 2,450 Credits | [Bell] 7
```
**Click to expand:**
- In Progress: Show list of running tasks
- Completed: Recent completions
- Credits: Usage breakdown, purchase link
- Bell: Notification dropdown
---
## Page-Specific Improvements
### Planner Pages
#### Keywords Page
**Add:**
- ✅ Helper: "Import keywords or add manually to start"
- ✅ Tooltip on "Cluster" button: "Group keywords by semantic similarity"
- ✅ Inline guidance on import: "CSV format: keyword, volume, difficulty"
- ✅ Next step button: "Cluster Keywords →"
- ✅ Metrics panel: Total, New, Mapped, Avg Volume
#### Clusters Page
**Add:**
- ✅ Helper: "Clusters organize keywords into topic groups"
- ✅ Step banner: Step 2/3 in Planner workflow
- ✅ Tooltip on cluster cards: Show keyword count, volume
- ✅ Next step: "Generate Ideas →"
- ✅ Metrics panel: Clusters, Avg size, Total volume
#### Ideas Page
**Add:**
- ✅ Helper: "Content ideas are generated from your clusters"
- ✅ Step banner: Step 3/3 in Planner workflow
- ✅ Content structure badge with tooltip
- ✅ Next step: "Create Tasks →" (converts to tasks)
- ✅ Metrics panel: Ideas, Cluster hubs, Word count estimates
---
### Writer Pages
#### Tasks Page
**Add:**
- ✅ Helper: "Tasks are content generation jobs queued for AI"
- ✅ Step banner: Step 1/3 in Writer workflow
- ✅ Status badges with tooltips
- ✅ Next step: "Generate Content →"
- ✅ Metrics panel: Total tasks, Queued, Completed, Total words
#### Content Page
**Add:**
- ✅ Helper: "Review and edit AI-generated content before publishing"
- ✅ Step banner: Step 2/3 in Writer workflow
- ✅ Status badges (draft/review/published)
- ✅ Next step: "Generate Images →"
- ✅ Metrics panel: Total content, Drafts, Published, Total words
#### Images Page
**Add:**
- ✅ Helper: "Generate images for your content from AI prompts"
- ✅ Step banner: Step 3/3 in Writer workflow
- ✅ Image preview on hover
- ✅ Next step: "Publish to WordPress →"
- ✅ Metrics panel: Total images, Generated, Published
---
### Automation Page
**Add:**
- ✅ Helper: "Automation runs the entire workflow automatically"
- ✅ Stage progress cards with clear metrics
- ✅ Pause/Resume/Cancel controls
- ✅ Real-time stage updates
- ✅ Estimated completion time
- ✅ Credit consumption display
---
## Implementation Plan
### Phase 1: New Components (Week 1)
1. ✅ Create StepBanner component
2. ✅ Create HelperNotification component
3. ✅ Enhance Tooltip component
4. ✅ Create InlineGuidance component
5. ✅ Create MetricsPanel component
### Phase 2: Navigation (Week 1-2)
1. ✅ Create Breadcrumbs component
2. ✅ Add cross-navigation buttons
3. ✅ Implement workflow progress indicator
4. ✅ Update ModuleNavigationTabs with tooltips
### Phase 3: Settings Reorganization (Week 2)
1. ✅ Create new Settings pages structure
2. ✅ Migrate existing settings
3. ✅ Create Settings hub landing page
4. ✅ Add category cards
### Phase 4: Notification System (Week 2-3)
1. ✅ Create Notification model (backend)
2. ✅ Create notification endpoints
3. ✅ Create NotificationDropdown component
4. ✅ Integrate with existing workflows
5. ✅ Add bell icon with badge to header
### Phase 5: Page Updates (Week 3)
1. ✅ Update all Planner pages with new components
2. ✅ Update all Writer pages with new components
3. ✅ Update Automation page
4. ✅ Add helpers, tooltips, metrics to each page
---
## Success Metrics
- ✅ All workflow pages have step banners
- ✅ All pages have helper notifications on first visit
- ✅ All icons and actions have tooltips
- ✅ All form fields have inline guidance
- ✅ Settings are organized and easy to find
- ✅ Users receive notifications for all key events
- ✅ Cross-navigation reduces clicks to complete workflows
- ✅ Metrics panels show real-time data
- ✅ User feedback indicates clearer navigation
---
## Related Files Reference
### Components (NEW)
- `frontend/src/components/workflow/StepBanner.tsx`
- `frontend/src/components/helper/HelperNotification.tsx`
- `frontend/src/components/helper/InlineGuidance.tsx`
- `frontend/src/components/navigation/Breadcrumbs.tsx`
- `frontend/src/components/dashboard/MetricsPanel.tsx`
- `frontend/src/components/notifications/NotificationDropdown.tsx`
### Components (Enhance)
- `frontend/src/components/common/PageHeader.tsx`
- `frontend/src/components/navigation/ModuleNavigationTabs.tsx`
- `frontend/src/components/ui/Tooltip.tsx`
### Pages (Update)
- All `frontend/src/pages/Planner/*.tsx`
- All `frontend/src/pages/Writer/*.tsx`
- `frontend/src/pages/Automation/*.tsx`
- `frontend/src/pages/Settings/*.tsx` (reorganize)
### Backend (NEW)
- `backend/igny8_core/modules/system/models.py` - Add Notification model
- `backend/igny8_core/modules/system/views.py` - Notification endpoints
---
## Notes
- Helper notifications should be non-intrusive and dismissible
- Tooltips should be brief (2-3 sentences max)
- Metrics should update in real-time or near-real-time
- Settings should be searchable (future enhancement)
- Consider onboarding tour for new users (future phase)
- All guidance text should be editable by admin (future enhancement)