docs

docs/automation/auto-cluster-validation-fix-plan.md

@@ -0,0 +1,753 @@
+# Auto-Cluster Validation Fix Plan
+
+**Date:** December 4, 2025  
+**Status:** Design Phase  
+**Priority:** MEDIUM
+
+---
+
+## 🎯 OBJECTIVE
+
+Add validation to prevent auto-cluster from running with less than 5 keywords, and ensure both manual auto-cluster and automation pipeline use the same shared validation logic to maintain consistency.
+
+---
+
+## 🔍 CURRENT STATE ANALYSIS
+
+### Current Behavior
+
+**Auto-Cluster Function:**
+- Located in: `backend/igny8_core/ai/functions/auto_cluster.py`
+- No minimum keyword validation
+- Accepts any number of keywords (even 1)
+- May produce poor quality clusters with insufficient data
+
+**Automation Pipeline:**
+- Located in: `backend/igny8_core/business/automation/services/automation_service.py`
+- Uses auto-cluster in Stage 1
+- No pre-check for minimum keywords
+- May waste credits on insufficient data
+
+### Problems
+
+1. ❌ **No Minimum Check:** Auto-cluster runs with 1-4 keywords
+2. ❌ **Poor Results:** AI cannot create meaningful clusters with < 5 keywords
+3. ❌ **Wasted Credits:** Charges credits for insufficient analysis
+4. ❌ **Inconsistent Validation:** No shared validation between manual and automation
+5. ❌ **User Confusion:** Error occurs during processing, not at selection
+
+---
+
+## ✅ PROPOSED SOLUTION
+
+### Validation Strategy
+
+**Single Source of Truth:**
+- Create one validation function
+- Use it in both auto-cluster function AND automation pipeline
+- Consistent error messages
+- No code duplication
+
+**Error Behavior:**
+- **Manual Auto-Cluster:** Return error before API call
+- **Automation Pipeline:** Skip Stage 1 with warning in logs
+
+---
+
+## 📋 IMPLEMENTATION PLAN
+
+### Step 1: Create Shared Validation Module
+
+**New File:** `backend/igny8_core/ai/validators/cluster_validators.py`
+
+```python
+"""
+Cluster-specific validators
+Shared between auto-cluster function and automation pipeline
+"""
+import logging
+from typing import Dict, List
+
+logger = logging.getLogger(__name__)
+
+
+def validate_minimum_keywords(
+    keyword_ids: List[int],
+    account=None,
+    min_required: int = 5
+) -> Dict:
+    """
+    Validate that sufficient keywords are available for clustering
+    
+    Args:
+        keyword_ids: List of keyword IDs to cluster
+        account: Account object for filtering
+        min_required: Minimum number of keywords required (default: 5)
+    
+    Returns:
+        Dict with 'valid' (bool) and 'error' (str) or 'count' (int)
+    """
+    from igny8_core.modules.planner.models import Keywords
+    
+    # Build queryset
+    queryset = Keywords.objects.filter(id__in=keyword_ids, status='new')
+    
+    if account:
+        queryset = queryset.filter(account=account)
+    
+    # Count available keywords
+    count = queryset.count()
+    
+    # Validate minimum
+    if count < min_required:
+        return {
+            'valid': False,
+            'error': f'Insufficient keywords for clustering. Need at least {min_required} keywords, but only {count} available.',
+            'count': count,
+            'required': min_required
+        }
+    
+    return {
+        'valid': True,
+        'count': count,
+        'required': min_required
+    }
+
+
+def validate_keyword_selection(
+    selected_ids: List[int],
+    available_count: int,
+    min_required: int = 5
+) -> Dict:
+    """
+    Validate keyword selection (for frontend validation)
+    
+    Args:
+        selected_ids: List of selected keyword IDs
+        available_count: Total count of available keywords
+        min_required: Minimum required
+    
+    Returns:
+        Dict with validation result
+    """
+    selected_count = len(selected_ids)
+    
+    # Check if any keywords selected
+    if selected_count == 0:
+        return {
+            'valid': False,
+            'error': 'No keywords selected',
+            'type': 'NO_SELECTION'
+        }
+    
+    # Check if enough selected
+    if selected_count < min_required:
+        return {
+            'valid': False,
+            'error': f'Please select at least {min_required} keywords. Currently selected: {selected_count}',
+            'type': 'INSUFFICIENT_SELECTION',
+            'selected': selected_count,
+            'required': min_required
+        }
+    
+    # Check if enough available (even if not all selected)
+    if available_count < min_required:
+        return {
+            'valid': False,
+            'error': f'Not enough keywords available. Need at least {min_required} keywords, but only {available_count} exist.',
+            'type': 'INSUFFICIENT_AVAILABLE',
+            'available': available_count,
+            'required': min_required
+        }
+    
+    return {
+        'valid': True,
+        'selected': selected_count,
+        'available': available_count,
+        'required': min_required
+    }
+```
+
+### Step 2: Update Auto-Cluster Function
+
+**File:** `backend/igny8_core/ai/functions/auto_cluster.py`
+
+**Add import:**
+```python
+from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
+```
+
+**Update validate() method:**
+```python
+def validate(self, payload: dict, account=None) -> Dict:
+    """Validate keyword IDs and minimum count"""
+    result = super().validate(payload, account)
+    if not result['valid']:
+        return result
+    
+    keyword_ids = payload.get('keyword_ids', [])
+    
+    if not keyword_ids:
+        return {'valid': False, 'error': 'No keyword IDs provided'}
+    
+    # NEW: Validate minimum keywords using shared validator
+    min_validation = validate_minimum_keywords(
+        keyword_ids=keyword_ids,
+        account=account,
+        min_required=5  # Configurable constant
+    )
+    
+    if not min_validation['valid']:
+        # Log the validation failure
+        logger.warning(
+            f"[AutoCluster] Validation failed: {min_validation['error']}"
+        )
+        return min_validation
+    
+    # Log successful validation
+    logger.info(
+        f"[AutoCluster] Validation passed: {min_validation['count']} keywords available (min: {min_validation['required']})"
+    )
+    
+    return {'valid': True}
+```
+
+### Step 3: Update Automation Pipeline
+
+**File:** `backend/igny8_core/business/automation/services/automation_service.py`
+
+**Add import:**
+```python
+from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
+```
+
+**Update run_stage_1() method:**
+```python
+def run_stage_1(self):
+    """Stage 1: Keywords → Clusters (AI)"""
+    stage_number = 1
+    stage_name = "Keywords → Clusters (AI)"
+    start_time = time.time()
+    
+    # Query pending keywords
+    pending_keywords = Keywords.objects.filter(
+        site=self.site,
+        status='new'
+    )
+    
+    total_count = pending_keywords.count()
+    
+    # NEW: Pre-stage validation for minimum keywords
+    keyword_ids = list(pending_keywords.values_list('id', flat=True))
+    
+    min_validation = validate_minimum_keywords(
+        keyword_ids=keyword_ids,
+        account=self.account,
+        min_required=5
+    )
+    
+    if not min_validation['valid']:
+        # Log validation failure
+        self.logger.log_stage_start(
+            self.run.run_id, self.account.id, self.site.id,
+            stage_number, stage_name, total_count
+        )
+        
+        error_msg = min_validation['error']
+        self.logger.log_stage_error(
+            self.run.run_id, self.account.id, self.site.id,
+            stage_number, error_msg
+        )
+        
+        # Skip stage with proper result
+        self.run.stage_1_result = {
+            'keywords_processed': 0,
+            'clusters_created': 0,
+            'skipped': True,
+            'skip_reason': error_msg,
+            'credits_used': 0
+        }
+        self.run.current_stage = 2
+        self.run.save()
+        
+        logger.warning(f"[AutomationService] Stage 1 skipped: {error_msg}")
+        return
+    
+    # Log stage start
+    self.logger.log_stage_start(
+        self.run.run_id, self.account.id, self.site.id,
+        stage_number, stage_name, total_count
+    )
+    
+    # ... rest of existing stage logic ...
+```
+
+### Step 4: Update API Endpoint
+
+**File:** `backend/igny8_core/modules/planner/views.py` (KeywordsViewSet)
+
+**Update auto_cluster action:**
+```python
+@action(detail=False, methods=['post'], url_path='auto_cluster', url_name='auto_cluster')
+def auto_cluster(self, request):
+    """Auto-cluster keywords using AI"""
+    from igny8_core.ai.tasks import run_ai_task
+    from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
+    
+    account = getattr(request, 'account', None)
+    keyword_ids = request.data.get('ids', [])
+    
+    if not keyword_ids:
+        return error_response(
+            error='No keyword IDs provided',
+            status_code=status.HTTP_400_BAD_REQUEST,
+            request=request
+        )
+    
+    # NEW: Validate minimum keywords BEFORE queuing task
+    validation = validate_minimum_keywords(
+        keyword_ids=keyword_ids,
+        account=account,
+        min_required=5
+    )
+    
+    if not validation['valid']:
+        return error_response(
+            error=validation['error'],
+            status_code=status.HTTP_400_BAD_REQUEST,
+            request=request,
+            extra_data={
+                'count': validation.get('count'),
+                'required': validation.get('required')
+            }
+        )
+    
+    # Validation passed - proceed with clustering
+    account_id = account.id if account else None
+    
+    try:
+        if hasattr(run_ai_task, 'delay'):
+            task = run_ai_task.delay(
+                function_name='auto_cluster',
+                payload={'keyword_ids': keyword_ids},
+                account_id=account_id
+            )
+            return success_response(
+                data={'task_id': str(task.id)},
+                message=f'Auto-cluster started with {validation["count"]} keywords',
+                request=request
+            )
+        else:
+            # Synchronous fallback
+            result = run_ai_task(
+                function_name='auto_cluster',
+                payload={'keyword_ids': keyword_ids},
+                account_id=account_id
+            )
+            return success_response(data=result, request=request)
+            
+    except Exception as e:
+        logger.error(f"Failed to start auto-cluster: {e}", exc_info=True)
+        return error_response(
+            error=f'Failed to start clustering: {str(e)}',
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            request=request
+        )
+```
+
+### Step 5: Add Frontend Validation (Optional but Recommended)
+
+**File:** `frontend/src/pages/Planner/Keywords.tsx`
+
+**Update handleAutoCluster function:**
+```typescript
+const handleAutoCluster = async () => {
+  try {
+    const selectedIds = selectedKeywords.map(k => k.id);
+    
+    // Frontend validation (pre-check before API call)
+    if (selectedIds.length < 5) {
+      toast.error(
+        `Please select at least 5 keywords for auto-clustering. Currently selected: ${selectedIds.length}`,
+        { duration: 5000 }
+      );
+      return;
+    }
+    
+    // Check total available
+    const availableCount = keywords.filter(k => k.status === 'new').length;
+    if (availableCount < 5) {
+      toast.error(
+        `Not enough keywords available. Need at least 5 keywords, but only ${availableCount} exist.`,
+        { duration: 5000 }
+      );
+      return;
+    }
+    
+    // Proceed with API call
+    const result = await autoClusterKeywords(selectedIds);
+    
+    if (result.task_id) {
+      toast.success(`Auto-cluster started with ${selectedIds.length} keywords`);
+      setTaskId(result.task_id);
+    } else {
+      toast.error('Failed to start auto-cluster');
+    }
+    
+  } catch (error: any) {
+    // Backend validation error (in case frontend check was bypassed)
+    const errorMsg = error.response?.data?.error || error.message;
+    toast.error(errorMsg);
+  }
+};
+```
+
+---
+
+## 🗂️ FILE STRUCTURE
+
+### New Files
+```
+backend/igny8_core/ai/validators/
+├── __init__.py
+└── cluster_validators.py  (NEW)
+```
+
+### Modified Files
+```
+backend/igny8_core/ai/functions/auto_cluster.py
+backend/igny8_core/business/automation/services/automation_service.py
+backend/igny8_core/modules/planner/views.py
+frontend/src/pages/Planner/Keywords.tsx
+```
+
+---
+
+## 🧪 TESTING PLAN
+
+### Unit Tests
+
+**File:** `backend/igny8_core/ai/validators/tests/test_cluster_validators.py`
+
+```python
+import pytest
+from django.test import TestCase
+from igny8_core.ai.validators.cluster_validators import (
+    validate_minimum_keywords,
+    validate_keyword_selection
+)
+from igny8_core.modules.planner.models import Keywords
+from igny8_core.auth.models import Account, Site
+
+
+class ClusterValidatorsTestCase(TestCase):
+    def setUp(self):
+        self.account = Account.objects.create(name='Test Account')
+        self.site = Site.objects.create(name='Test Site', account=self.account)
+    
+    def test_validate_minimum_keywords_success(self):
+        """Test with sufficient keywords (>= 5)"""
+        # Create 10 keywords
+        keyword_ids = []
+        for i in range(10):
+            kw = Keywords.objects.create(
+                keyword=f'keyword {i}',
+                status='new',
+                account=self.account,
+                site=self.site
+            )
+            keyword_ids.append(kw.id)
+        
+        result = validate_minimum_keywords(keyword_ids, self.account)
+        
+        assert result['valid'] is True
+        assert result['count'] == 10
+        assert result['required'] == 5
+    
+    def test_validate_minimum_keywords_failure(self):
+        """Test with insufficient keywords (< 5)"""
+        # Create only 3 keywords
+        keyword_ids = []
+        for i in range(3):
+            kw = Keywords.objects.create(
+                keyword=f'keyword {i}',
+                status='new',
+                account=self.account,
+                site=self.site
+            )
+            keyword_ids.append(kw.id)
+        
+        result = validate_minimum_keywords(keyword_ids, self.account)
+        
+        assert result['valid'] is False
+        assert 'Insufficient keywords' in result['error']
+        assert result['count'] == 3
+        assert result['required'] == 5
+    
+    def test_validate_minimum_keywords_edge_case_exactly_5(self):
+        """Test with exactly 5 keywords (boundary)"""
+        keyword_ids = []
+        for i in range(5):
+            kw = Keywords.objects.create(
+                keyword=f'keyword {i}',
+                status='new',
+                account=self.account,
+                site=self.site
+            )
+            keyword_ids.append(kw.id)
+        
+        result = validate_minimum_keywords(keyword_ids, self.account)
+        
+        assert result['valid'] is True
+        assert result['count'] == 5
+    
+    def test_validate_keyword_selection_insufficient(self):
+        """Test frontend selection validation"""
+        result = validate_keyword_selection(
+            selected_ids=[1, 2, 3],  # Only 3
+            available_count=10,
+            min_required=5
+        )
+        
+        assert result['valid'] is False
+        assert result['type'] == 'INSUFFICIENT_SELECTION'
+        assert result['selected'] == 3
+        assert result['required'] == 5
+```
+
+### Integration Tests
+
+```python
+class AutoClusterIntegrationTestCase(TestCase):
+    def test_auto_cluster_with_insufficient_keywords(self):
+        """Test auto-cluster endpoint rejects < 5 keywords"""
+        # Create only 3 keywords
+        keyword_ids = self._create_keywords(3)
+        
+        response = self.client.post(
+            '/api/planner/keywords/auto_cluster/',
+            data={'ids': keyword_ids},
+            HTTP_AUTHORIZATION=f'Bearer {self.token}'
+        )
+        
+        assert response.status_code == 400
+        assert 'Insufficient keywords' in response.json()['error']
+    
+    def test_automation_skips_stage_1_with_insufficient_keywords(self):
+        """Test automation skips Stage 1 if < 5 keywords"""
+        # Create only 2 keywords
+        self._create_keywords(2)
+        
+        # Start automation
+        run_id = self.automation_service.start_automation('manual')
+        
+        # Verify Stage 1 was skipped
+        run = AutomationRun.objects.get(run_id=run_id)
+        assert run.stage_1_result['skipped'] is True
+        assert 'Insufficient keywords' in run.stage_1_result['skip_reason']
+        assert run.current_stage == 2  # Moved to next stage
+```
+
+### Manual Test Cases
+
+- [ ] **Test 1:** Try auto-cluster with 0 keywords selected
+  - Expected: Error message "No keywords selected"
+  
+- [ ] **Test 2:** Try auto-cluster with 3 keywords selected
+  - Expected: Error message "Please select at least 5 keywords. Currently selected: 3"
+  
+- [ ] **Test 3:** Try auto-cluster with exactly 5 keywords
+  - Expected: Success, clustering starts
+  
+- [ ] **Test 4:** Run automation with 2 keywords in site
+  - Expected: Stage 1 skipped with warning in logs
+  
+- [ ] **Test 5:** Run automation with 10 keywords in site
+  - Expected: Stage 1 runs normally
+
+---
+
+## 📊 ERROR MESSAGES
+
+### Frontend (User-Facing)
+
+**No Selection:**
+```
+❌ No keywords selected
+Please select keywords to cluster.
+```
+
+**Insufficient Selection:**
+```
+❌ Please select at least 5 keywords for auto-clustering
+Currently selected: 3 keywords
+You need at least 5 keywords to create meaningful clusters.
+```
+
+**Insufficient Available:**
+```
+❌ Not enough keywords available
+Need at least 5 keywords, but only 2 exist.
+Add more keywords before running auto-cluster.
+```
+
+### Backend (Logs)
+
+**Validation Failed:**
+```
+[AutoCluster] Validation failed: Insufficient keywords for clustering. Need at least 5 keywords, but only 3 available.
+```
+
+**Validation Passed:**
+```
+[AutoCluster] Validation passed: 15 keywords available (min: 5)
+```
+
+**Automation Stage Skipped:**
+```
+[AutomationService] Stage 1 skipped: Insufficient keywords for clustering. Need at least 5 keywords, but only 2 available.
+```
+
+---
+
+## 🎯 CONFIGURATION
+
+### Constants File
+
+**File:** `backend/igny8_core/ai/constants.py` (or create if doesn't exist)
+
+```python
+"""
+AI Function Configuration Constants
+"""
+
+# Cluster Configuration
+MIN_KEYWORDS_FOR_CLUSTERING = 5  # Minimum keywords needed for meaningful clusters
+OPTIMAL_KEYWORDS_FOR_CLUSTERING = 20  # Recommended for best results
+
+# Other AI limits...
+```
+
+**Usage in validators:**
+```python
+from igny8_core.ai.constants import MIN_KEYWORDS_FOR_CLUSTERING
+
+def validate_minimum_keywords(keyword_ids, account=None):
+    min_required = MIN_KEYWORDS_FOR_CLUSTERING
+    # ... validation logic
+```
+
+---
+
+## 🔄 SHARED VALIDATION PATTERN
+
+### Why This Approach Works
+
+**✅ Single Source of Truth:**
+- One function: `validate_minimum_keywords()`
+- Used by both auto-cluster function and automation
+- Update in one place applies everywhere
+
+**✅ Consistent Behavior:**
+- Same error messages
+- Same validation logic
+- Same minimum requirements
+
+**✅ Easy to Maintain:**
+- Want to change minimum from 5 to 10? Change one constant
+- Want to add new validation? Add to one function
+- Want to test? Test one module
+
+**✅ No Code Duplication:**
+- DRY principle followed
+- Reduces bugs from inconsistency
+- Easier code review
+
+### Pattern for Future Validators
+
+```python
+# backend/igny8_core/ai/validators/content_validators.py
+
+def validate_minimum_content_length(content_text: str, min_words: int = 100):
+    """
+    Shared validator for content minimum length
+    Used by: GenerateContentFunction, Automation Stage 4, Content creation
+    """
+    word_count = len(content_text.split())
+    
+    if word_count < min_words:
+        return {
+            'valid': False,
+            'error': f'Content too short. Minimum {min_words} words required, got {word_count}.'
+        }
+    
+    return {'valid': True, 'word_count': word_count}
+```
+
+---
+
+## 🚀 IMPLEMENTATION STEPS
+
+### Phase 1: Create Validator (Day 1)
+- [ ] Create `cluster_validators.py`
+- [ ] Implement `validate_minimum_keywords()`
+- [ ] Implement `validate_keyword_selection()`
+- [ ] Write unit tests
+
+### Phase 2: Integrate Backend (Day 1)
+- [ ] Update `AutoClusterFunction.validate()`
+- [ ] Update `AutomationService.run_stage_1()`
+- [ ] Update `KeywordsViewSet.auto_cluster()`
+- [ ] Write integration tests
+
+### Phase 3: Frontend (Day 2)
+- [ ] Add frontend validation in Keywords page
+- [ ] Add user-friendly error messages
+- [ ] Test error scenarios
+
+### Phase 4: Testing & Deployment (Day 2)
+- [ ] Run all tests
+- [ ] Manual QA testing
+- [ ] Deploy to production
+- [ ] Monitor first few auto-cluster runs
+
+---
+
+## 🎯 SUCCESS CRITERIA
+
+✅ Auto-cluster returns error if < 5 keywords selected  
+✅ Automation skips Stage 1 if < 5 keywords available  
+✅ Both use same validation function (no duplication)  
+✅ Clear error messages guide users  
+✅ Frontend validation provides instant feedback  
+✅ Backend validation catches edge cases  
+✅ All tests pass  
+✅ No regression in existing functionality
+
+---
+
+## 📈 FUTURE ENHANCEMENTS
+
+### V2 Features
+
+1. **Configurable Minimum:**
+   - Allow admin to set minimum via settings
+   - Default: 5, Range: 3-20
+
+2. **Quality Scoring:**
+   - Show quality indicator based on keyword count
+   - 5-10: "Fair", 11-20: "Good", 21+: "Excellent"
+
+3. **Smart Recommendations:**
+   - "You have 4 keywords. Add 1 more for best results"
+   - "15 keywords selected. Good for clustering!"
+
+4. **Batch Size Validation:**
+   - Warn if too many keywords selected (> 100)
+   - Suggest splitting into multiple runs
+
+---
+
+## END OF PLAN
+
+This plan ensures robust, consistent validation for auto-cluster across all entry points (manual and automation) using shared, well-tested validation logic.

docs/automation/automation-progress-ux-improvement-plan.md

@@ -0,0 +1,725 @@
+# Automation Progress UX Improvement Plan
+
+**Date:** December 4, 2025  
+**Status:** Design Phase  
+**Priority:** MEDIUM
+
+---
+
+## 🎯 OBJECTIVE
+
+Improve the automation progress tracking UX to show **real-time processing status** for currently processing items, making it easier for users to understand what's happening during automation runs.
+
+---
+
+## 🔍 CURRENT STATE ANALYSIS
+
+### Current Behavior
+
+**What Users See Now:**
+1. A "Current State" card that shows the stage being processed
+2. Stage number and status (e.g., "Stage 3: Ideas → Tasks")
+3. **BUT:** No visibility into which specific records are being processed
+4. **Problem:** User only knows when a full stage completes
+
+**Example Current Experience:**
+```
+┌─────────────────────────────────────┐
+│ Current State: Stage 2              │
+│ Clusters → Ideas (AI)               │
+│                                     │
+│ Status: Processing                  │
+└─────────────────────────────────────┘
+
+[User waits... no updates until stage completes]
+```
+
+### User Pain Points
+
+1. ❌ **No Record-Level Progress:** Can't see which keywords/ideas/content are being processed
+2. ❌ **No Queue Visibility:** Don't know what's coming up next
+3. ❌ **No Item Count Progress:** "Processing 15 of 50 keywords..." is missing
+4. ❌ **Card Position:** Current state card is at bottom, requires scrolling
+5. ❌ **No Percentage Progress:** Just a spinner, no quantitative feedback
+
+---
+
+## ✅ PROPOSED SOLUTION
+
+### New Design Concept
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ 🔄 AUTOMATION IN PROGRESS                                                   │
+│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67%          │
+│                                                                             │
+│ Stage 2: Clusters → Ideas (AI)                                             │
+│Column 1                                                                             │
+│ Currently Processing:                                                       │
+│ • "Best SEO tools for small business" (Cluster #42)                        │
+│                                                                        Column 2     │
+│ Up Next:                                                                    │
+│ • "Content marketing automation platforms"                                 │
+│ • "AI-powered content creation tools"                                      │
+│                                                                        Sinngle row centered     │
+│ Progress: 34/50 clusters processed                                         │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+[STAGES SECTION BELOW - All 7 stages in grid view]
+```
+
+---
+
+## 📐 DETAILED DESIGN SPECIFICATIONS
+
+### 1. Card Repositioning
+
+**Move from:** Bottom of page (after stages)  
+**Move to:** Top of page (above stages section)  
+**Layout:** Max width 1200px horizontal card  
+**Visibility:** Only shown when `currentRun?.status === 'running'`
+
+### 2. Card Structure
+
+#### Header Section
+- **Left:** Large stage number icon (animated pulse)
+- **Center:** Stage name + type badge (AI/Local/Manual)
+- **Right:** Percentage complete (calculated from processed/total)
+
+#### Progress Bar
+- **Type:** Animated linear progress bar
+- **Colors:**
+  - Blue for active stage
+  - Green for completed
+  - Gray for pending
+- **Updates:** Refresh every 3-5 seconds via polling
+
+#### Currently Processing Section
+- **For Keywords Stage:**
+  ```
+  Currently Processing:
+  • "keyword 1"
+  • "keyword 2"
+  • "keyword 3"
+  
+  + 47 more in queue
+  ```
+
+- **For Ideas Stage:**
+  ```
+  Currently Processing:
+  • "10 Ways to Improve SEO Rankings"
+  
+  Up Next:
+  • "Content Marketing Best Practices 2025"
+  • "AI Tools for Content Writers"
+  ```
+
+- **For Content Stage:**
+  ```
+  Currently Processing:
+  • "How to Use ChatGPT for Content Creation" (2,500 words)
+  
+  Up Next:
+  • "Best AI Image Generators in 2025"
+  ```
+
+#### Record Counter
+```
+Progress: [current]/[total] [items] processed
+Example: Progress: 15/50 keywords processed
+```
+
+### 3. Refresh Strategy
+
+**Polling Approach:**
+```typescript
+// Poll every 3 seconds while automation is running
+useEffect(() => {
+  if (currentRun?.status === 'running') {
+    const interval = setInterval(() => {
+      // Refresh ONLY the current processing data
+      fetchCurrentProcessingState();
+    }, 3000);
+    
+    return () => clearInterval(interval);
+  }
+}, [currentRun]);
+```
+
+**Partial Refresh:**
+- Only refresh the "Currently Processing" component
+- Don't reload entire page
+- Don't re-fetch stage cards
+- Smooth transition (no flickering)
+
+---
+
+## 🗄️ BACKEND CHANGES REQUIRED
+
+### New API Endpoint
+
+**URL:** `GET /api/automation/current_processing/`  
+**Params:** `?site_id={id}&run_id={run_id}`
+
+**Response Format:**
+```json
+{
+  "run_id": "abc123",
+  "current_stage": 2,
+  "stage_name": "Clusters → Ideas",
+  "stage_type": "AI",
+  "total_items": 50,
+  "processed_items": 34,
+  "percentage": 68,
+  "currently_processing": [
+    {
+      "id": 42,
+      "title": "Best SEO tools for small business",
+      "type": "cluster"
+    }
+  ],
+  "up_next": [
+    {
+      "id": 43,
+      "title": "Content marketing automation platforms",
+      "type": "cluster"
+    },
+    {
+      "id": 44,
+      "title": "AI-powered content creation tools",
+      "type": "cluster"
+    }
+  ],
+  "remaining_count": 16
+}
+```
+
+### Implementation in AutomationService
+
+**File:** `backend/igny8_core/business/automation/services/automation_service.py`
+
+**Add method:**
+```python
+def get_current_processing_state(self) -> dict:
+    """
+    Get real-time processing state for current automation run
+    """
+    if not self.run or self.run.status != 'running':
+        return None
+    
+    stage = self.run.current_stage
+    
+    # Get stage-specific data
+    if stage == 1:  # Keywords → Clusters
+        queue = Keywords.objects.filter(
+            site=self.site, status='new'
+        ).order_by('id')
+        
+        return {
+            'stage_number': 1,
+            'stage_name': 'Keywords → Clusters',
+            'stage_type': 'AI',
+            'total_items': queue.count() + self._get_processed_count(stage),
+            'processed_items': self._get_processed_count(stage),
+            'currently_processing': self._get_current_items(queue, 3),
+            'up_next': self._get_next_items(queue, 2, skip=3),
+        }
+    
+    elif stage == 2:  # Clusters → Ideas
+        queue = Clusters.objects.filter(
+            site=self.site, status='new', disabled=False
+        ).order_by('id')
+        
+        return {
+            'stage_number': 2,
+            'stage_name': 'Clusters → Ideas',
+            'stage_type': 'AI',
+            'total_items': queue.count() + self._get_processed_count(stage),
+            'processed_items': self._get_processed_count(stage),
+            'currently_processing': self._get_current_items(queue, 1),
+            'up_next': self._get_next_items(queue, 2, skip=1),
+        }
+    
+    # ... similar for stages 3-6
+    
+def _get_processed_count(self, stage: int) -> int:
+    """Get count of items processed in current stage"""
+    result_key = f'stage_{stage}_result'
+    result = getattr(self.run, result_key, {})
+    
+    # Extract appropriate count from result
+    if stage == 1:
+        return result.get('keywords_processed', 0)
+    elif stage == 2:
+        return result.get('clusters_processed', 0)
+    # ... etc
+    
+def _get_current_items(self, queryset, count: int) -> list:
+    """Get currently processing items"""
+    items = queryset[:count]
+    return [
+        {
+            'id': item.id,
+            'title': getattr(item, 'keyword', None) or 
+                     getattr(item, 'cluster_name', None) or
+                     getattr(item, 'idea_title', None) or
+                     getattr(item, 'title', None),
+            'type': queryset.model.__name__.lower()
+        }
+        for item in items
+    ]
+```
+
+### Add View in AutomationViewSet
+
+**File:** `backend/igny8_core/business/automation/views.py`
+
+```python
+@action(detail=False, methods=['get'], url_path='current_processing')
+def current_processing(self, request):
+    """Get current processing state for active automation run"""
+    site_id = request.GET.get('site_id')
+    run_id = request.GET.get('run_id')
+    
+    if not site_id or not run_id:
+        return error_response(
+            error='site_id and run_id required',
+            status_code=400,
+            request=request
+        )
+    
+    try:
+        run = AutomationRun.objects.get(run_id=run_id, site_id=site_id)
+        
+        if run.status != 'running':
+            return success_response(data=None, request=request)
+        
+        service = AutomationService.from_run_id(run_id)
+        state = service.get_current_processing_state()
+        
+        return success_response(data=state, request=request)
+        
+    except AutomationRun.DoesNotExist:
+        return error_response(
+            error='Run not found',
+            status_code=404,
+            request=request
+        )
+```
+
+---
+
+## 🎨 FRONTEND CHANGES REQUIRED
+
+### 1. New Component: CurrentProcessingCard
+
+**File:** `frontend/src/components/Automation/CurrentProcessingCard.tsx`
+
+```typescript
+interface CurrentProcessingCardProps {
+  runId: string;
+  siteId: number;
+  currentStage: number;
+  onComplete?: () => void;
+}
+
+const CurrentProcessingCard: React.FC<CurrentProcessingCardProps> = ({
+  runId,
+  siteId,
+  currentStage,
+  onComplete
+}) => {
+  const [processingState, setProcessingState] = useState<ProcessingState | null>(null);
+  
+  // Poll every 3 seconds
+  useEffect(() => {
+    const fetchState = async () => {
+      const state = await automationService.getCurrentProcessing(siteId, runId);
+      setProcessingState(state);
+      
+      // If stage completed, trigger refresh
+      if (state && state.processed_items === state.total_items) {
+        onComplete?.();
+      }
+    };
+    
+    fetchState();
+    const interval = setInterval(fetchState, 3000);
+    
+    return () => clearInterval(interval);
+  }, [siteId, runId]);
+  
+  if (!processingState) return null;
+  
+  const percentage = Math.round(
+    (processingState.processed_items / processingState.total_items) * 100
+  );
+  
+  return (
+    <div className="bg-blue-50 dark:bg-blue-900/20 border-2 border-blue-500 rounded-lg p-6 mb-6">
+      {/* Header */}
+      <div className="flex items-center justify-between mb-4">
+        <div className="flex items-center gap-3">
+          <div className="animate-pulse">
+            <BoltIcon className="w-8 h-8 text-blue-600" />
+          </div>
+          <div>
+            <h2 className="text-2xl font-bold text-gray-900 dark:text-white">
+              Automation In Progress
+            </h2>
+            <p className="text-sm text-gray-600 dark:text-gray-400">
+              Stage {currentStage}: {processingState.stage_name}
+              <span className="ml-2 px-2 py-0.5 bg-blue-100 dark:bg-blue-900 text-blue-700 dark:text-blue-300 rounded text-xs">
+                {processingState.stage_type}
+              </span>
+            </p>
+          </div>
+        </div>
+        <div className="text-right">
+          <div className="text-4xl font-bold text-blue-600">{percentage}%</div>
+          <div className="text-sm text-gray-600 dark:text-gray-400">
+            {processingState.processed_items}/{processingState.total_items} processed
+          </div>
+        </div>
+      </div>
+      
+      {/* Progress Bar */}
+      <div className="mb-6">
+        <div className="w-full bg-gray-200 dark:bg-gray-700 rounded-full h-3">
+          <div
+            className="bg-blue-600 h-3 rounded-full transition-all duration-500"
+            style={{ width: `${percentage}%` }}
+          />
+        </div>
+      </div>
+      
+      {/* Currently Processing */}
+      <div className="grid grid-cols-1 md:grid-cols-2 gap-4">
+        <div>
+          <h3 className="text-sm font-semibold text-gray-700 dark:text-gray-300 mb-2">
+            Currently Processing:
+          </h3>
+          <div className="space-y-1">
+            {processingState.currently_processing.map((item, idx) => (
+              <div key={idx} className="flex items-start gap-2 text-sm">
+                <span className="text-blue-600 mt-1">•</span>
+                <span className="text-gray-800 dark:text-gray-200 font-medium">
+                  {item.title}
+                </span>
+              </div>
+            ))}
+          </div>
+        </div>
+        
+        <div>
+          <h3 className="text-sm font-semibold text-gray-700 dark:text-gray-300 mb-2">
+            Up Next:
+          </h3>
+          <div className="space-y-1">
+            {processingState.up_next.map((item, idx) => (
+              <div key={idx} className="flex items-start gap-2 text-sm">
+                <span className="text-gray-400 mt-1">•</span>
+                <span className="text-gray-600 dark:text-gray-400">
+                  {item.title}
+                </span>
+              </div>
+            ))}
+            {processingState.remaining_count > processingState.up_next.length && (
+              <div className="text-xs text-gray-500 mt-2">
+                + {processingState.remaining_count - processingState.up_next.length} more in queue
+              </div>
+            )}
+          </div>
+        </div>
+      </div>
+    </div>
+  );
+};
+```
+
+### 2. Update AutomationPage.tsx
+
+**File:** `frontend/src/pages/Automation/AutomationPage.tsx`
+
+```typescript
+// Add new import
+import CurrentProcessingCard from '../../components/Automation/CurrentProcessingCard';
+
+// In the component
+return (
+  <div className="p-6">
+    <PageMeta title="Automation" description="AI automation pipeline" />
+    
+    {/* Current Processing Card - MOVE TO TOP */}
+    {currentRun?.status === 'running' && (
+      <CurrentProcessingCard
+        runId={currentRun.run_id}
+        siteId={selectedSite.id}
+        currentStage={currentRun.current_stage}
+        onComplete={() => {
+          // Refresh full page metrics when stage completes
+          loadAutomationData();
+        }}
+      />
+    )}
+    
+    {/* Metrics Cards */}
+    <div className="grid grid-cols-1 md:grid-cols-3 gap-6 mb-6">
+      {/* ... existing metrics ... */}
+    </div>
+    
+    {/* Stages Section */}
+    <ComponentCard>
+      <h2 className="text-xl font-semibold mb-4">Pipeline Stages</h2>
+      {/* ... existing stages ... */}
+    </ComponentCard>
+    
+    {/* Rest of the page ... */}
+  </div>
+);
+```
+
+### 3. Add Service Method
+
+**File:** `frontend/src/services/automationService.ts`
+
+```typescript
+export interface ProcessingState {
+  run_id: string;
+  current_stage: number;
+  stage_name: string;
+  stage_type: 'AI' | 'Local' | 'Manual';
+  total_items: number;
+  processed_items: number;
+  percentage: number;
+  currently_processing: Array<{
+    id: number;
+    title: string;
+    type: string;
+  }>;
+  up_next: Array<{
+    id: number;
+    title: string;
+    type: string;
+  }>;
+  remaining_count: number;
+}
+
+// Add to automationService
+getCurrentProcessing: async (
+  siteId: number,
+  runId: string
+): Promise<ProcessingState | null> => {
+  return fetchAPI(
+    buildUrl('/current_processing/', { site_id: siteId, run_id: runId })
+  );
+},
+```
+
+---
+
+## 🧪 TESTING PLAN
+
+### Unit Tests
+
+- [ ] Test `get_current_processing_state()` for each stage
+- [ ] Test `_get_processed_count()` calculation
+- [ ] Test `_get_current_items()` formatting
+- [ ] Test API endpoint with various run states
+
+### Integration Tests
+
+- [ ] Test polling updates every 3 seconds
+- [ ] Test stage completion triggers full refresh
+- [ ] Test card disappears when automation completes
+- [ ] Test with 0 items (edge case)
+- [ ] Test with 1000+ items (performance)
+
+### Visual/UX Tests
+
+- [ ] Card positioned at top of page
+- [ ] Progress bar animates smoothly
+- [ ] Record names display correctly
+- [ ] Responsive design (mobile/tablet/desktop)
+- [ ] Dark mode support
+- [ ] Loading states
+- [ ] Error states
+
+---
+
+## 📊 STAGE-SPECIFIC DISPLAY FORMATS
+
+### Stage 1: Keywords → Clusters
+
+```
+Currently Processing:
+• "best seo tools"
+• "content marketing platforms"
+• "ai writing assistants"
+
++ 47 more keywords in queue
+
+Progress: 3/50 keywords processed
+```
+
+### Stage 2: Clusters → Ideas
+
+```
+Currently Processing:
+• "SEO Tools and Software" (Cluster #12)
+
+Up Next:
+• "Content Marketing Strategies"
+• "AI Content Generation"
+
+Progress: 12/25 clusters processed
+```
+
+### Stage 3: Ideas → Tasks
+
+```
+Currently Processing:
+• "10 Best SEO Tools for 2025"
+
+Up Next:
+• "How to Create Content with AI"
+• "Content Marketing ROI Calculator"
+
+Progress: 8/30 ideas processed
+```
+
+### Stage 4: Tasks → Content
+
+```
+Currently Processing:
+• "Ultimate Guide to SEO in 2025" (2,500 words)
+
+Up Next:
+• "AI Content Creation Best Practices"
+
+Progress: 5/15 tasks processed
+```
+
+### Stage 5: Content → Image Prompts
+
+```
+Currently Processing:
+• "How to Use ChatGPT for Content" (Extracting 3 image prompts)
+
+Up Next:
+• "Best AI Image Generators 2025"
+
+Progress: 10/15 content pieces processed
+```
+
+### Stage 6: Image Prompts → Images
+
+```
+Currently Processing:
+• Featured image for "SEO Guide 2025"
+
+Up Next:
+• In-article image #1 for "SEO Guide 2025"
+• In-article image #2 for "SEO Guide 2025"
+
+Progress: 15/45 images generated
+```
+
+### Stage 7: Manual Review Gate
+
+```
+Automation Complete! ✅
+
+Ready for Review:
+• "Ultimate Guide to SEO in 2025"
+• "AI Content Creation Best Practices"
+• "Best Image Generators 2025"
+
++ 12 more content pieces
+
+Total: 15 content pieces ready for review
+```
+
+---
+
+## 🎯 SUCCESS METRICS
+
+### User Experience
+
+✅ Users can see **exactly what's being processed** at any moment  
+✅ Users know **what's coming up next** in the queue  
+✅ Users can estimate **remaining time** based on progress  
+✅ Users get **quantitative feedback** (percentage, counts)  
+✅ Users see **smooth, non-disruptive updates** (no page flicker)
+
+### Technical
+
+✅ Polling interval: 3 seconds (balance between freshness and load)  
+✅ API response time: < 200ms  
+✅ Component re-render: Only the processing card, not entire page  
+✅ Memory usage: No memory leaks from polling  
+✅ Error handling: Graceful degradation if API fails
+
+---
+
+## 🚀 IMPLEMENTATION PHASES
+
+### Phase 1: Backend (1-2 days)
+- [ ] Implement `get_current_processing_state()` method
+- [ ] Add `/current_processing/` API endpoint
+- [ ] Test with all 7 stages
+- [ ] Add unit tests
+
+### Phase 2: Frontend (2-3 days)
+- [ ] Create `CurrentProcessingCard` component
+- [ ] Add polling logic with cleanup
+- [ ] Style with Tailwind (match existing design system)
+- [ ] Add dark mode support
+- [ ] Integrate into `AutomationPage`
+
+### Phase 3: Testing & Refinement (1-2 days)
+- [ ] Integration testing
+- [ ] Performance testing
+- [ ] UX testing
+- [ ] Bug fixes
+
+### Phase 4: Deployment
+- [ ] Deploy backend changes
+- [ ] Deploy frontend changes
+- [ ] Monitor first automation runs
+- [ ] Collect user feedback
+
+---
+
+## 🔄 FUTURE ENHANCEMENTS
+
+### V2 Features (Post-MVP)
+
+1. **Estimated Time Remaining:**
+   ```
+   Progress: 15/50 keywords processed
+   Estimated time remaining: ~8 minutes
+   ```
+
+2. **Stage-Level Progress Bar:**
+   - Each stage shows its own mini progress bar
+   - Visual indicator of which stages are complete
+
+3. **Click to View Details:**
+   - Click on a record name to see modal with details
+   - Preview generated content/images
+
+4. **Pause/Resume from Card:**
+   - Add pause button directly in the card
+   - Quick action without scrolling
+
+5. **Export Processing Log:**
+   - Download real-time processing log
+   - CSV of all processed items with timestamps
+
+---
+
+## END OF PLAN
+
+This plan provides a comprehensive UX improvement for automation progress tracking, making the process transparent and user-friendly while maintaining system performance.

docs/automation/automation-stage-6-image-generation-fix.md

@@ -0,0 +1,403 @@
+# Automation Stage 6 - Image Generation Fix Plan
+
+**Date:** December 4, 2025  
+**Status:** Analysis Complete - Implementation Required  
+**Priority:** HIGH
+
+---
+
+## 🔍 PROBLEM IDENTIFICATION
+
+### Current Issue
+
+Stage 6 of the automation pipeline (Image Prompts → Generated Images) is **NOT running correctly**. The issue stems from using the wrong AI function for image generation.
+
+### Root Cause Analysis
+
+**Current Implementation (INCORRECT):**
+```python
+# File: backend/igny8_core/business/automation/services/automation_service.py
+# Line ~935
+
+engine = AIEngine(account=self.account)
+result = engine.execute(
+    fn=GenerateImagesFunction(),
+    payload={'image_ids': [image.id]}  # ❌ WRONG
+)
+```
+
+**Why It Fails:**
+
+1. `GenerateImagesFunction()` expects:
+   - Input: `{'ids': [task_ids]}`  (Task IDs, NOT Image IDs)
+   - Purpose: Extract prompts from Tasks and generate images for tasks
+   - Use case: When you have Tasks with content but no images
+
+2. Automation Stage 6 has:
+   - Input: Images records with `status='pending'` (already have prompts)
+   - Purpose: Generate actual image URLs from existing prompts
+   - Context: Images were created in Stage 5 by `GenerateImagePromptsFunction`
+
+### How Other Stages Work Correctly
+
+**Stage 1:** Keywords → Clusters
+```python
+engine.execute(
+    fn=AutoClusterFunction(),
+    payload={'keyword_ids': keyword_ids}  # ✅ Correct
+)
+```
+
+**Stage 2:** Clusters → Ideas
+```python
+engine.execute(
+    fn=GenerateIdeasFunction(),
+    payload={'cluster_ids': cluster_ids}  # ✅ Correct
+)
+```
+
+**Stage 4:** Tasks → Content
+```python
+engine.execute(
+    fn=GenerateContentFunction(),
+    payload={'ids': task_ids}  # ✅ Correct
+)
+```
+
+**Stage 5:** Content → Image Prompts
+```python
+engine.execute(
+    fn=GenerateImagePromptsFunction(),
+    payload={'ids': content_ids}  # ✅ Correct
+)
+```
+
+**Stage 6:** Image Prompts → Images (BROKEN)
+```python
+# Currently uses GenerateImagesFunction (WRONG)
+# Should use process_image_generation_queue (CORRECT)
+```
+
+---
+
+## ✅ CORRECT SOLUTION
+
+### The Right Approach
+
+**Use `process_image_generation_queue` Celery task** - This is the same approach used by:
+1. Writer/Images page (`/writer/images/generate_images/` endpoint)
+2. Manual image generation from prompts
+
+**Evidence from Working Code:**
+
+```python
+# File: backend/igny8_core/modules/writer/views.py
+# ImagesViewSet.generate_images()
+
+from igny8_core.ai.tasks import process_image_generation_queue
+
+task = process_image_generation_queue.delay(
+    image_ids=image_ids,      # ✅ Accepts image_ids
+    account_id=account_id,
+    content_id=content_id
+)
+```
+
+**What `process_image_generation_queue` Does:**
+
+1. ✅ Accepts `image_ids` (list of Image record IDs)
+2. ✅ Each Image record already has a `prompt` field (populated by Stage 5)
+3. ✅ Generates images sequentially with progress tracking
+4. ✅ Updates Images records: `status='pending'` → `status='generated'`
+5. ✅ Downloads and saves images locally
+6. ✅ Automatically handles credits deduction
+7. ✅ Supports multiple providers (OpenAI, Runware)
+8. ✅ Handles errors gracefully (continues on failure)
+
+---
+
+## 📋 IMPLEMENTATION PLAN
+
+### Changes Required
+
+**File:** `backend/igny8_core/business/automation/services/automation_service.py`
+
+**Location:** `run_stage_6()` method (lines 874-1022)
+
+### Step 1: Import the Correct Task
+
+**Current:**
+```python
+from igny8_core.ai.functions.generate_images import GenerateImagesFunction
+```
+
+**Add:**
+```python
+from igny8_core.ai.tasks import process_image_generation_queue
+```
+
+### Step 2: Modify Stage 6 Logic
+
+**Replace this block (lines ~920-945):**
+
+```python
+# INCORRECT - Delete this
+for idx, image in enumerate(image_list, 1):
+    try:
+        content_title = image.content.title if image.content else 'Unknown'
+        self.logger.log_stage_progress(
+            self.run.run_id, self.account.id, self.site.id,
+            stage_number, f"Generating image {idx}/{total_images}: {image.image_type} for '{content_title}'"
+        )
+        
+        # Call AI function via AIEngine
+        engine = AIEngine(account=self.account)
+        result = engine.execute(
+            fn=GenerateImagesFunction(),
+            payload={'image_ids': [image.id]}  # ❌ WRONG
+        )
+        
+        # Monitor task
+        task_id = result.get('task_id')
+        if task_id:
+            self._wait_for_task(task_id, stage_number, f"Image for '{content_title}'", continue_on_error=True)
+        
+        images_processed += 1
+```
+
+**With this:**
+
+```python
+# CORRECT - Use process_image_generation_queue
+for idx, image in enumerate(image_list, 1):
+    try:
+        content_title = image.content.title if image.content else 'Unknown'
+        self.logger.log_stage_progress(
+            self.run.run_id, self.account.id, self.site.id,
+            stage_number, f"Generating image {idx}/{total_images}: {image.image_type} for '{content_title}'"
+        )
+        
+        # Call process_image_generation_queue directly (same as Writer/Images page)
+        from igny8_core.ai.tasks import process_image_generation_queue
+        
+        # Queue the task
+        if hasattr(process_image_generation_queue, 'delay'):
+            task = process_image_generation_queue.delay(
+                image_ids=[image.id],
+                account_id=self.account.id,
+                content_id=image.content.id if image.content else None
+            )
+            task_id = str(task.id)
+        else:
+            # Fallback for testing (synchronous)
+            result = process_image_generation_queue(
+                image_ids=[image.id],
+                account_id=self.account.id,
+                content_id=image.content.id if image.content else None
+            )
+            task_id = None
+        
+        # Monitor task (if async)
+        if task_id:
+            self._wait_for_task(task_id, stage_number, f"Image for '{content_title}'", continue_on_error=True)
+        
+        images_processed += 1
+```
+
+### Step 3: Update Logging
+
+The logging structure remains the same, just update the log messages to reflect the correct process:
+
+```python
+self.logger.log_stage_progress(
+    self.run.run_id, self.account.id, self.site.id,
+    stage_number, f"Image generation task queued for '{content_title}' ({images_processed}/{total_images})"
+)
+```
+
+### Step 4: No Changes Needed For
+
+✅ Stage 5 (Image Prompt Extraction) - Already correct  
+✅ Images table structure - Already has all required fields  
+✅ Progress tracking - Already implemented in `process_image_generation_queue`  
+✅ Credits deduction - Automatic in `process_image_generation_queue`  
+✅ Error handling - Built into the task with `continue_on_error=True`
+
+---
+
+## 🔄 HOW IT WORKS (CORRECTED FLOW)
+
+### Stage 5: Content → Image Prompts
+
+```
+Input:  Content (status='draft', no images)
+AI:     GenerateImagePromptsFunction
+Output: Images (status='pending', prompt='...')
+```
+
+### Stage 6: Image Prompts → Generated Images (FIXED)
+
+```
+Input:  Images (status='pending', has prompt)
+Task:   process_image_generation_queue (Celery task)
+AI:     Calls OpenAI/Runware API with prompt
+Output: Images (status='generated', image_url='https://...', image_path='/path/to/file')
+```
+
+### What Happens in process_image_generation_queue
+
+1. **Load Image Record:**
+   - Get Image by ID
+   - Read existing `prompt` field (created in Stage 5)
+   - Get Content for template formatting
+
+2. **Format Prompt:**
+   - Use image_prompt_template from PromptRegistry
+   - Format: `"Create a {image_type} image for '{post_title}'. Prompt: {image_prompt}"`
+   - Handle model-specific limits (DALL-E 3: 4000 chars, DALL-E 2: 1000 chars)
+
+3. **Generate Image:**
+   - Call `AICore.generate_image()`
+   - Uses configured provider (OpenAI/Runware)
+   - Uses configured model (dall-e-3, runware:97@1, etc.)
+   - Respects image size settings
+
+4. **Download & Save:**
+   - Download image from URL
+   - Save to `/data/app/igny8/frontend/public/images/ai-images/`
+   - Update Image record with both `image_url` and `image_path`
+
+5. **Update Status:**
+   - `status='pending'` → `status='generated'`
+   - Triggers automatic Content status update (if all images generated)
+
+6. **Deduct Credits:**
+   - Automatic via `AICore` credit system
+   - Records in `AIUsageLog`
+
+---
+
+## 🧪 TESTING CHECKLIST
+
+### Pre-Deployment Tests
+
+- [ ] **Unit Test:** Verify `process_image_generation_queue` works with single image
+- [ ] **Integration Test:** Run Stage 6 with 3-5 pending images
+- [ ] **Error Handling:** Test with invalid image ID
+- [ ] **Credits:** Verify credits are deducted correctly
+- [ ] **Multi-Provider:** Test with both OpenAI and Runware
+
+### Post-Deployment Validation
+
+- [ ] **Full Pipeline:** Run Automation from Stage 1 → Stage 7
+- [ ] **Verify Stage 5 Output:** Images created with `status='pending'` and prompts
+- [ ] **Verify Stage 6 Output:** Images updated to `status='generated'` with URLs
+- [ ] **Check Downloads:** Images saved to `/data/app/igny8/frontend/public/images/ai-images/`
+- [ ] **Monitor Logs:** Review automation logs for Stage 6 completion
+- [ ] **Credits Report:** Confirm Stage 6 credits recorded in automation results
+
+### Success Criteria
+
+✅ Stage 6 completes without errors  
+✅ All pending images get generated  
+✅ Images are downloaded and accessible  
+✅ Content status automatically updates when all images generated  
+✅ Credits are properly deducted and logged  
+✅ Automation proceeds to Stage 7 (Manual Review Gate)
+
+---
+
+## 📊 COMPARISON: BEFORE vs AFTER
+
+### BEFORE (Broken)
+
+```python
+# ❌ WRONG APPROACH
+GenerateImagesFunction()
+- Expects: task_ids
+- Purpose: Extract prompts from Tasks
+- Problem: Doesn't work with Images that already have prompts
+```
+
+**Result:** Stage 6 fails, images never generated
+
+### AFTER (Fixed)
+
+```python
+# ✅ CORRECT APPROACH
+process_image_generation_queue()
+- Accepts: image_ids
+- Purpose: Generate images from existing prompts
+- Works with: Images (status='pending' with prompts)
+```
+
+**Result:** Stage 6 succeeds, images generated sequentially with progress tracking
+
+---
+
+## 🔒 SAFETY & ROLLBACK
+
+### Backup Plan
+
+If the fix causes issues:
+
+1. **Rollback Code:**
+   - Git revert the automation_service.py changes
+   - Automation still works for Stages 1-5
+
+2. **Manual Workaround:**
+   - Users can manually generate images from Writer/Images page
+   - This uses the same `process_image_generation_queue` task
+
+3. **No Data Loss:**
+   - Stage 5 already created Images with prompts
+   - These remain in database and can be processed anytime
+
+---
+
+## 📝 IMPLEMENTATION STEPS
+
+1. **Update Code:** Modify `run_stage_6()` as documented above
+2. **Test Locally:** Run automation with test data
+3. **Code Review:** Verify changes match working Writer/Images implementation
+4. **Deploy:** Push to production
+5. **Monitor:** Watch first automation run for Stage 6 completion
+6. **Validate:** Check images generated and credits deducted
+
+---
+
+## 🎯 EXPECTED OUTCOME
+
+After implementing this fix:
+
+✅ **Stage 6 will work correctly** - Images generate from prompts  
+✅ **Consistent with manual flow** - Same logic as Writer/Images page  
+✅ **Proper credits tracking** - Automated deduction via AICore  
+✅ **Sequential processing** - One image at a time with progress  
+✅ **Error resilience** - Continues on failure, logs errors  
+✅ **Full pipeline completion** - Automation flows from Stage 1 → Stage 7  
+
+---
+
+## 🔗 RELATED FUNCTIONS
+
+### Keep These Functions (Working Correctly)
+
+- `GenerateImagePromptsFunction` - Stage 5 ✅
+- `AutoClusterFunction` - Stage 1 ✅
+- `GenerateIdeasFunction` - Stage 2 ✅
+- `GenerateContentFunction` - Stage 4 ✅
+
+### Use This Task for Stage 6
+
+- `process_image_generation_queue` - Celery task for Images → Generated Images ✅
+
+### DO NOT USE in Automation
+
+- `GenerateImagesFunction` - For Tasks, not for Images with existing prompts ❌
+
+---
+
+## END OF PLAN
+
+This plan provides a clear, actionable fix for Automation Stage 6 image generation, aligning it with the working manual image generation flow used throughout the application.