docs

docs/automation/auto-cluster-validation-fix-plan.md

@@ -0,0 +1,753 @@
+# Auto-Cluster Validation Fix Plan
+
+**Date:** December 4, 2025  
+**Status:** Design Phase  
+**Priority:** MEDIUM
+
+---
+
+## 🎯 OBJECTIVE
+
+Add validation to prevent auto-cluster from running with less than 5 keywords, and ensure both manual auto-cluster and automation pipeline use the same shared validation logic to maintain consistency.
+
+---
+
+## 🔍 CURRENT STATE ANALYSIS
+
+### Current Behavior
+
+**Auto-Cluster Function:**
+- Located in: `backend/igny8_core/ai/functions/auto_cluster.py`
+- No minimum keyword validation
+- Accepts any number of keywords (even 1)
+- May produce poor quality clusters with insufficient data
+
+**Automation Pipeline:**
+- Located in: `backend/igny8_core/business/automation/services/automation_service.py`
+- Uses auto-cluster in Stage 1
+- No pre-check for minimum keywords
+- May waste credits on insufficient data
+
+### Problems
+
+1. ❌ **No Minimum Check:** Auto-cluster runs with 1-4 keywords
+2. ❌ **Poor Results:** AI cannot create meaningful clusters with < 5 keywords
+3. ❌ **Wasted Credits:** Charges credits for insufficient analysis
+4. ❌ **Inconsistent Validation:** No shared validation between manual and automation
+5. ❌ **User Confusion:** Error occurs during processing, not at selection
+
+---
+
+## ✅ PROPOSED SOLUTION
+
+### Validation Strategy
+
+**Single Source of Truth:**
+- Create one validation function
+- Use it in both auto-cluster function AND automation pipeline
+- Consistent error messages
+- No code duplication
+
+**Error Behavior:**
+- **Manual Auto-Cluster:** Return error before API call
+- **Automation Pipeline:** Skip Stage 1 with warning in logs
+
+---
+
+## 📋 IMPLEMENTATION PLAN
+
+### Step 1: Create Shared Validation Module
+
+**New File:** `backend/igny8_core/ai/validators/cluster_validators.py`
+
+```python
+"""
+Cluster-specific validators
+Shared between auto-cluster function and automation pipeline
+"""
+import logging
+from typing import Dict, List
+
+logger = logging.getLogger(__name__)
+
+
+def validate_minimum_keywords(
+    keyword_ids: List[int],
+    account=None,
+    min_required: int = 5
+) -> Dict:
+    """
+    Validate that sufficient keywords are available for clustering
+    
+    Args:
+        keyword_ids: List of keyword IDs to cluster
+        account: Account object for filtering
+        min_required: Minimum number of keywords required (default: 5)
+    
+    Returns:
+        Dict with 'valid' (bool) and 'error' (str) or 'count' (int)
+    """
+    from igny8_core.modules.planner.models import Keywords
+    
+    # Build queryset
+    queryset = Keywords.objects.filter(id__in=keyword_ids, status='new')
+    
+    if account:
+        queryset = queryset.filter(account=account)
+    
+    # Count available keywords
+    count = queryset.count()
+    
+    # Validate minimum
+    if count < min_required:
+        return {
+            'valid': False,
+            'error': f'Insufficient keywords for clustering. Need at least {min_required} keywords, but only {count} available.',
+            'count': count,
+            'required': min_required
+        }
+    
+    return {
+        'valid': True,
+        'count': count,
+        'required': min_required
+    }
+
+
+def validate_keyword_selection(
+    selected_ids: List[int],
+    available_count: int,
+    min_required: int = 5
+) -> Dict:
+    """
+    Validate keyword selection (for frontend validation)
+    
+    Args:
+        selected_ids: List of selected keyword IDs
+        available_count: Total count of available keywords
+        min_required: Minimum required
+    
+    Returns:
+        Dict with validation result
+    """
+    selected_count = len(selected_ids)
+    
+    # Check if any keywords selected
+    if selected_count == 0:
+        return {
+            'valid': False,
+            'error': 'No keywords selected',
+            'type': 'NO_SELECTION'
+        }
+    
+    # Check if enough selected
+    if selected_count < min_required:
+        return {
+            'valid': False,
+            'error': f'Please select at least {min_required} keywords. Currently selected: {selected_count}',
+            'type': 'INSUFFICIENT_SELECTION',
+            'selected': selected_count,
+            'required': min_required
+        }
+    
+    # Check if enough available (even if not all selected)
+    if available_count < min_required:
+        return {
+            'valid': False,
+            'error': f'Not enough keywords available. Need at least {min_required} keywords, but only {available_count} exist.',
+            'type': 'INSUFFICIENT_AVAILABLE',
+            'available': available_count,
+            'required': min_required
+        }
+    
+    return {
+        'valid': True,
+        'selected': selected_count,
+        'available': available_count,
+        'required': min_required
+    }
+```
+
+### Step 2: Update Auto-Cluster Function
+
+**File:** `backend/igny8_core/ai/functions/auto_cluster.py`
+
+**Add import:**
+```python
+from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
+```
+
+**Update validate() method:**
+```python
+def validate(self, payload: dict, account=None) -> Dict:
+    """Validate keyword IDs and minimum count"""
+    result = super().validate(payload, account)
+    if not result['valid']:
+        return result
+    
+    keyword_ids = payload.get('keyword_ids', [])
+    
+    if not keyword_ids:
+        return {'valid': False, 'error': 'No keyword IDs provided'}
+    
+    # NEW: Validate minimum keywords using shared validator
+    min_validation = validate_minimum_keywords(
+        keyword_ids=keyword_ids,
+        account=account,
+        min_required=5  # Configurable constant
+    )
+    
+    if not min_validation['valid']:
+        # Log the validation failure
+        logger.warning(
+            f"[AutoCluster] Validation failed: {min_validation['error']}"
+        )
+        return min_validation
+    
+    # Log successful validation
+    logger.info(
+        f"[AutoCluster] Validation passed: {min_validation['count']} keywords available (min: {min_validation['required']})"
+    )
+    
+    return {'valid': True}
+```
+
+### Step 3: Update Automation Pipeline
+
+**File:** `backend/igny8_core/business/automation/services/automation_service.py`
+
+**Add import:**
+```python
+from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
+```
+
+**Update run_stage_1() method:**
+```python
+def run_stage_1(self):
+    """Stage 1: Keywords → Clusters (AI)"""
+    stage_number = 1
+    stage_name = "Keywords → Clusters (AI)"
+    start_time = time.time()
+    
+    # Query pending keywords
+    pending_keywords = Keywords.objects.filter(
+        site=self.site,
+        status='new'
+    )
+    
+    total_count = pending_keywords.count()
+    
+    # NEW: Pre-stage validation for minimum keywords
+    keyword_ids = list(pending_keywords.values_list('id', flat=True))
+    
+    min_validation = validate_minimum_keywords(
+        keyword_ids=keyword_ids,
+        account=self.account,
+        min_required=5
+    )
+    
+    if not min_validation['valid']:
+        # Log validation failure
+        self.logger.log_stage_start(
+            self.run.run_id, self.account.id, self.site.id,
+            stage_number, stage_name, total_count
+        )
+        
+        error_msg = min_validation['error']
+        self.logger.log_stage_error(
+            self.run.run_id, self.account.id, self.site.id,
+            stage_number, error_msg
+        )
+        
+        # Skip stage with proper result
+        self.run.stage_1_result = {
+            'keywords_processed': 0,
+            'clusters_created': 0,
+            'skipped': True,
+            'skip_reason': error_msg,
+            'credits_used': 0
+        }
+        self.run.current_stage = 2
+        self.run.save()
+        
+        logger.warning(f"[AutomationService] Stage 1 skipped: {error_msg}")
+        return
+    
+    # Log stage start
+    self.logger.log_stage_start(
+        self.run.run_id, self.account.id, self.site.id,
+        stage_number, stage_name, total_count
+    )
+    
+    # ... rest of existing stage logic ...
+```
+
+### Step 4: Update API Endpoint
+
+**File:** `backend/igny8_core/modules/planner/views.py` (KeywordsViewSet)
+
+**Update auto_cluster action:**
+```python
+@action(detail=False, methods=['post'], url_path='auto_cluster', url_name='auto_cluster')
+def auto_cluster(self, request):
+    """Auto-cluster keywords using AI"""
+    from igny8_core.ai.tasks import run_ai_task
+    from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
+    
+    account = getattr(request, 'account', None)
+    keyword_ids = request.data.get('ids', [])
+    
+    if not keyword_ids:
+        return error_response(
+            error='No keyword IDs provided',
+            status_code=status.HTTP_400_BAD_REQUEST,
+            request=request
+        )
+    
+    # NEW: Validate minimum keywords BEFORE queuing task
+    validation = validate_minimum_keywords(
+        keyword_ids=keyword_ids,
+        account=account,
+        min_required=5
+    )
+    
+    if not validation['valid']:
+        return error_response(
+            error=validation['error'],
+            status_code=status.HTTP_400_BAD_REQUEST,
+            request=request,
+            extra_data={
+                'count': validation.get('count'),
+                'required': validation.get('required')
+            }
+        )
+    
+    # Validation passed - proceed with clustering
+    account_id = account.id if account else None
+    
+    try:
+        if hasattr(run_ai_task, 'delay'):
+            task = run_ai_task.delay(
+                function_name='auto_cluster',
+                payload={'keyword_ids': keyword_ids},
+                account_id=account_id
+            )
+            return success_response(
+                data={'task_id': str(task.id)},
+                message=f'Auto-cluster started with {validation["count"]} keywords',
+                request=request
+            )
+        else:
+            # Synchronous fallback
+            result = run_ai_task(
+                function_name='auto_cluster',
+                payload={'keyword_ids': keyword_ids},
+                account_id=account_id
+            )
+            return success_response(data=result, request=request)
+            
+    except Exception as e:
+        logger.error(f"Failed to start auto-cluster: {e}", exc_info=True)
+        return error_response(
+            error=f'Failed to start clustering: {str(e)}',
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            request=request
+        )
+```
+
+### Step 5: Add Frontend Validation (Optional but Recommended)
+
+**File:** `frontend/src/pages/Planner/Keywords.tsx`
+
+**Update handleAutoCluster function:**
+```typescript
+const handleAutoCluster = async () => {
+  try {
+    const selectedIds = selectedKeywords.map(k => k.id);
+    
+    // Frontend validation (pre-check before API call)
+    if (selectedIds.length < 5) {
+      toast.error(
+        `Please select at least 5 keywords for auto-clustering. Currently selected: ${selectedIds.length}`,
+        { duration: 5000 }
+      );
+      return;
+    }
+    
+    // Check total available
+    const availableCount = keywords.filter(k => k.status === 'new').length;
+    if (availableCount < 5) {
+      toast.error(
+        `Not enough keywords available. Need at least 5 keywords, but only ${availableCount} exist.`,
+        { duration: 5000 }
+      );
+      return;
+    }
+    
+    // Proceed with API call
+    const result = await autoClusterKeywords(selectedIds);
+    
+    if (result.task_id) {
+      toast.success(`Auto-cluster started with ${selectedIds.length} keywords`);
+      setTaskId(result.task_id);
+    } else {
+      toast.error('Failed to start auto-cluster');
+    }
+    
+  } catch (error: any) {
+    // Backend validation error (in case frontend check was bypassed)
+    const errorMsg = error.response?.data?.error || error.message;
+    toast.error(errorMsg);
+  }
+};
+```
+
+---
+
+## 🗂️ FILE STRUCTURE
+
+### New Files
+```
+backend/igny8_core/ai/validators/
+├── __init__.py
+└── cluster_validators.py  (NEW)
+```
+
+### Modified Files
+```
+backend/igny8_core/ai/functions/auto_cluster.py
+backend/igny8_core/business/automation/services/automation_service.py
+backend/igny8_core/modules/planner/views.py
+frontend/src/pages/Planner/Keywords.tsx
+```
+
+---
+
+## 🧪 TESTING PLAN
+
+### Unit Tests
+
+**File:** `backend/igny8_core/ai/validators/tests/test_cluster_validators.py`
+
+```python
+import pytest
+from django.test import TestCase
+from igny8_core.ai.validators.cluster_validators import (
+    validate_minimum_keywords,
+    validate_keyword_selection
+)
+from igny8_core.modules.planner.models import Keywords
+from igny8_core.auth.models import Account, Site
+
+
+class ClusterValidatorsTestCase(TestCase):
+    def setUp(self):
+        self.account = Account.objects.create(name='Test Account')
+        self.site = Site.objects.create(name='Test Site', account=self.account)
+    
+    def test_validate_minimum_keywords_success(self):
+        """Test with sufficient keywords (>= 5)"""
+        # Create 10 keywords
+        keyword_ids = []
+        for i in range(10):
+            kw = Keywords.objects.create(
+                keyword=f'keyword {i}',
+                status='new',
+                account=self.account,
+                site=self.site
+            )
+            keyword_ids.append(kw.id)
+        
+        result = validate_minimum_keywords(keyword_ids, self.account)
+        
+        assert result['valid'] is True
+        assert result['count'] == 10
+        assert result['required'] == 5
+    
+    def test_validate_minimum_keywords_failure(self):
+        """Test with insufficient keywords (< 5)"""
+        # Create only 3 keywords
+        keyword_ids = []
+        for i in range(3):
+            kw = Keywords.objects.create(
+                keyword=f'keyword {i}',
+                status='new',
+                account=self.account,
+                site=self.site
+            )
+            keyword_ids.append(kw.id)
+        
+        result = validate_minimum_keywords(keyword_ids, self.account)
+        
+        assert result['valid'] is False
+        assert 'Insufficient keywords' in result['error']
+        assert result['count'] == 3
+        assert result['required'] == 5
+    
+    def test_validate_minimum_keywords_edge_case_exactly_5(self):
+        """Test with exactly 5 keywords (boundary)"""
+        keyword_ids = []
+        for i in range(5):
+            kw = Keywords.objects.create(
+                keyword=f'keyword {i}',
+                status='new',
+                account=self.account,
+                site=self.site
+            )
+            keyword_ids.append(kw.id)
+        
+        result = validate_minimum_keywords(keyword_ids, self.account)
+        
+        assert result['valid'] is True
+        assert result['count'] == 5
+    
+    def test_validate_keyword_selection_insufficient(self):
+        """Test frontend selection validation"""
+        result = validate_keyword_selection(
+            selected_ids=[1, 2, 3],  # Only 3
+            available_count=10,
+            min_required=5
+        )
+        
+        assert result['valid'] is False
+        assert result['type'] == 'INSUFFICIENT_SELECTION'
+        assert result['selected'] == 3
+        assert result['required'] == 5
+```
+
+### Integration Tests
+
+```python
+class AutoClusterIntegrationTestCase(TestCase):
+    def test_auto_cluster_with_insufficient_keywords(self):
+        """Test auto-cluster endpoint rejects < 5 keywords"""
+        # Create only 3 keywords
+        keyword_ids = self._create_keywords(3)
+        
+        response = self.client.post(
+            '/api/planner/keywords/auto_cluster/',
+            data={'ids': keyword_ids},
+            HTTP_AUTHORIZATION=f'Bearer {self.token}'
+        )
+        
+        assert response.status_code == 400
+        assert 'Insufficient keywords' in response.json()['error']
+    
+    def test_automation_skips_stage_1_with_insufficient_keywords(self):
+        """Test automation skips Stage 1 if < 5 keywords"""
+        # Create only 2 keywords
+        self._create_keywords(2)
+        
+        # Start automation
+        run_id = self.automation_service.start_automation('manual')
+        
+        # Verify Stage 1 was skipped
+        run = AutomationRun.objects.get(run_id=run_id)
+        assert run.stage_1_result['skipped'] is True
+        assert 'Insufficient keywords' in run.stage_1_result['skip_reason']
+        assert run.current_stage == 2  # Moved to next stage
+```
+
+### Manual Test Cases
+
+- [ ] **Test 1:** Try auto-cluster with 0 keywords selected
+  - Expected: Error message "No keywords selected"
+  
+- [ ] **Test 2:** Try auto-cluster with 3 keywords selected
+  - Expected: Error message "Please select at least 5 keywords. Currently selected: 3"
+  
+- [ ] **Test 3:** Try auto-cluster with exactly 5 keywords
+  - Expected: Success, clustering starts
+  
+- [ ] **Test 4:** Run automation with 2 keywords in site
+  - Expected: Stage 1 skipped with warning in logs
+  
+- [ ] **Test 5:** Run automation with 10 keywords in site
+  - Expected: Stage 1 runs normally
+
+---
+
+## 📊 ERROR MESSAGES
+
+### Frontend (User-Facing)
+
+**No Selection:**
+```
+❌ No keywords selected
+Please select keywords to cluster.
+```
+
+**Insufficient Selection:**
+```
+❌ Please select at least 5 keywords for auto-clustering
+Currently selected: 3 keywords
+You need at least 5 keywords to create meaningful clusters.
+```
+
+**Insufficient Available:**
+```
+❌ Not enough keywords available
+Need at least 5 keywords, but only 2 exist.
+Add more keywords before running auto-cluster.
+```
+
+### Backend (Logs)
+
+**Validation Failed:**
+```
+[AutoCluster] Validation failed: Insufficient keywords for clustering. Need at least 5 keywords, but only 3 available.
+```
+
+**Validation Passed:**
+```
+[AutoCluster] Validation passed: 15 keywords available (min: 5)
+```
+
+**Automation Stage Skipped:**
+```
+[AutomationService] Stage 1 skipped: Insufficient keywords for clustering. Need at least 5 keywords, but only 2 available.
+```
+
+---
+
+## 🎯 CONFIGURATION
+
+### Constants File
+
+**File:** `backend/igny8_core/ai/constants.py` (or create if doesn't exist)
+
+```python
+"""
+AI Function Configuration Constants
+"""
+
+# Cluster Configuration
+MIN_KEYWORDS_FOR_CLUSTERING = 5  # Minimum keywords needed for meaningful clusters
+OPTIMAL_KEYWORDS_FOR_CLUSTERING = 20  # Recommended for best results
+
+# Other AI limits...
+```
+
+**Usage in validators:**
+```python
+from igny8_core.ai.constants import MIN_KEYWORDS_FOR_CLUSTERING
+
+def validate_minimum_keywords(keyword_ids, account=None):
+    min_required = MIN_KEYWORDS_FOR_CLUSTERING
+    # ... validation logic
+```
+
+---
+
+## 🔄 SHARED VALIDATION PATTERN
+
+### Why This Approach Works
+
+**✅ Single Source of Truth:**
+- One function: `validate_minimum_keywords()`
+- Used by both auto-cluster function and automation
+- Update in one place applies everywhere
+
+**✅ Consistent Behavior:**
+- Same error messages
+- Same validation logic
+- Same minimum requirements
+
+**✅ Easy to Maintain:**
+- Want to change minimum from 5 to 10? Change one constant
+- Want to add new validation? Add to one function
+- Want to test? Test one module
+
+**✅ No Code Duplication:**
+- DRY principle followed
+- Reduces bugs from inconsistency
+- Easier code review
+
+### Pattern for Future Validators
+
+```python
+# backend/igny8_core/ai/validators/content_validators.py
+
+def validate_minimum_content_length(content_text: str, min_words: int = 100):
+    """
+    Shared validator for content minimum length
+    Used by: GenerateContentFunction, Automation Stage 4, Content creation
+    """
+    word_count = len(content_text.split())
+    
+    if word_count < min_words:
+        return {
+            'valid': False,
+            'error': f'Content too short. Minimum {min_words} words required, got {word_count}.'
+        }
+    
+    return {'valid': True, 'word_count': word_count}
+```
+
+---
+
+## 🚀 IMPLEMENTATION STEPS
+
+### Phase 1: Create Validator (Day 1)
+- [ ] Create `cluster_validators.py`
+- [ ] Implement `validate_minimum_keywords()`
+- [ ] Implement `validate_keyword_selection()`
+- [ ] Write unit tests
+
+### Phase 2: Integrate Backend (Day 1)
+- [ ] Update `AutoClusterFunction.validate()`
+- [ ] Update `AutomationService.run_stage_1()`
+- [ ] Update `KeywordsViewSet.auto_cluster()`
+- [ ] Write integration tests
+
+### Phase 3: Frontend (Day 2)
+- [ ] Add frontend validation in Keywords page
+- [ ] Add user-friendly error messages
+- [ ] Test error scenarios
+
+### Phase 4: Testing & Deployment (Day 2)
+- [ ] Run all tests
+- [ ] Manual QA testing
+- [ ] Deploy to production
+- [ ] Monitor first few auto-cluster runs
+
+---
+
+## 🎯 SUCCESS CRITERIA
+
+✅ Auto-cluster returns error if < 5 keywords selected  
+✅ Automation skips Stage 1 if < 5 keywords available  
+✅ Both use same validation function (no duplication)  
+✅ Clear error messages guide users  
+✅ Frontend validation provides instant feedback  
+✅ Backend validation catches edge cases  
+✅ All tests pass  
+✅ No regression in existing functionality
+
+---
+
+## 📈 FUTURE ENHANCEMENTS
+
+### V2 Features
+
+1. **Configurable Minimum:**
+   - Allow admin to set minimum via settings
+   - Default: 5, Range: 3-20
+
+2. **Quality Scoring:**
+   - Show quality indicator based on keyword count
+   - 5-10: "Fair", 11-20: "Good", 21+: "Excellent"
+
+3. **Smart Recommendations:**
+   - "You have 4 keywords. Add 1 more for best results"
+   - "15 keywords selected. Good for clustering!"
+
+4. **Batch Size Validation:**
+   - Warn if too many keywords selected (> 100)
+   - Suggest splitting into multiple runs
+
+---
+
+## END OF PLAN
+
+This plan ensures robust, consistent validation for auto-cluster across all entry points (manual and automation) using shared, well-tested validation logic.

docs/automation/automation-progress-ux-improvement-plan.md

@@ -0,0 +1,725 @@
+# Automation Progress UX Improvement Plan
+
+**Date:** December 4, 2025  
+**Status:** Design Phase  
+**Priority:** MEDIUM
+
+---
+
+## 🎯 OBJECTIVE
+
+Improve the automation progress tracking UX to show **real-time processing status** for currently processing items, making it easier for users to understand what's happening during automation runs.
+
+---
+
+## 🔍 CURRENT STATE ANALYSIS
+
+### Current Behavior
+
+**What Users See Now:**
+1. A "Current State" card that shows the stage being processed
+2. Stage number and status (e.g., "Stage 3: Ideas → Tasks")
+3. **BUT:** No visibility into which specific records are being processed
+4. **Problem:** User only knows when a full stage completes
+
+**Example Current Experience:**
+```
+┌─────────────────────────────────────┐
+│ Current State: Stage 2              │
+│ Clusters → Ideas (AI)               │
+│                                     │
+│ Status: Processing                  │
+└─────────────────────────────────────┘
+
+[User waits... no updates until stage completes]
+```
+
+### User Pain Points
+
+1. ❌ **No Record-Level Progress:** Can't see which keywords/ideas/content are being processed
+2. ❌ **No Queue Visibility:** Don't know what's coming up next
+3. ❌ **No Item Count Progress:** "Processing 15 of 50 keywords..." is missing
+4. ❌ **Card Position:** Current state card is at bottom, requires scrolling
+5. ❌ **No Percentage Progress:** Just a spinner, no quantitative feedback
+
+---
+
+## ✅ PROPOSED SOLUTION
+
+### New Design Concept
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ 🔄 AUTOMATION IN PROGRESS                                                   │
+│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67%          │
+│                                                                             │
+│ Stage 2: Clusters → Ideas (AI)                                             │
+│Column 1                                                                             │
+│ Currently Processing:                                                       │
+│ • "Best SEO tools for small business" (Cluster #42)                        │
+│                                                                        Column 2     │
+│ Up Next:                                                                    │
+│ • "Content marketing automation platforms"                                 │
+│ • "AI-powered content creation tools"                                      │
+│                                                                        Sinngle row centered     │
+│ Progress: 34/50 clusters processed                                         │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+[STAGES SECTION BELOW - All 7 stages in grid view]
+```
+
+---
+
+## 📐 DETAILED DESIGN SPECIFICATIONS
+
+### 1. Card Repositioning
+
+**Move from:** Bottom of page (after stages)  
+**Move to:** Top of page (above stages section)  
+**Layout:** Max width 1200px horizontal card  
+**Visibility:** Only shown when `currentRun?.status === 'running'`
+
+### 2. Card Structure
+
+#### Header Section
+- **Left:** Large stage number icon (animated pulse)
+- **Center:** Stage name + type badge (AI/Local/Manual)
+- **Right:** Percentage complete (calculated from processed/total)
+
+#### Progress Bar
+- **Type:** Animated linear progress bar
+- **Colors:**
+  - Blue for active stage
+  - Green for completed
+  - Gray for pending
+- **Updates:** Refresh every 3-5 seconds via polling
+
+#### Currently Processing Section
+- **For Keywords Stage:**
+  ```
+  Currently Processing:
+  • "keyword 1"
+  • "keyword 2"
+  • "keyword 3"
+  
+  + 47 more in queue
+  ```
+
+- **For Ideas Stage:**
+  ```
+  Currently Processing:
+  • "10 Ways to Improve SEO Rankings"
+  
+  Up Next:
+  • "Content Marketing Best Practices 2025"
+  • "AI Tools for Content Writers"
+  ```
+
+- **For Content Stage:**
+  ```
+  Currently Processing:
+  • "How to Use ChatGPT for Content Creation" (2,500 words)
+  
+  Up Next:
+  • "Best AI Image Generators in 2025"
+  ```
+
+#### Record Counter
+```
+Progress: [current]/[total] [items] processed
+Example: Progress: 15/50 keywords processed
+```
+
+### 3. Refresh Strategy
+
+**Polling Approach:**
+```typescript
+// Poll every 3 seconds while automation is running
+useEffect(() => {
+  if (currentRun?.status === 'running') {
+    const interval = setInterval(() => {
+      // Refresh ONLY the current processing data
+      fetchCurrentProcessingState();
+    }, 3000);
+    
+    return () => clearInterval(interval);
+  }
+}, [currentRun]);
+```
+
+**Partial Refresh:**
+- Only refresh the "Currently Processing" component
+- Don't reload entire page
+- Don't re-fetch stage cards
+- Smooth transition (no flickering)
+
+---
+
+## 🗄️ BACKEND CHANGES REQUIRED
+
+### New API Endpoint
+
+**URL:** `GET /api/automation/current_processing/`  
+**Params:** `?site_id={id}&run_id={run_id}`
+
+**Response Format:**
+```json
+{
+  "run_id": "abc123",
+  "current_stage": 2,
+  "stage_name": "Clusters → Ideas",
+  "stage_type": "AI",
+  "total_items": 50,
+  "processed_items": 34,
+  "percentage": 68,
+  "currently_processing": [
+    {
+      "id": 42,
+      "title": "Best SEO tools for small business",
+      "type": "cluster"
+    }
+  ],
+  "up_next": [
+    {
+      "id": 43,
+      "title": "Content marketing automation platforms",
+      "type": "cluster"
+    },
+    {
+      "id": 44,
+      "title": "AI-powered content creation tools",
+      "type": "cluster"
+    }
+  ],
+  "remaining_count": 16
+}
+```
+
+### Implementation in AutomationService
+
+**File:** `backend/igny8_core/business/automation/services/automation_service.py`
+
+**Add method:**
+```python
+def get_current_processing_state(self) -> dict:
+    """
+    Get real-time processing state for current automation run
+    """
+    if not self.run or self.run.status != 'running':
+        return None
+    
+    stage = self.run.current_stage
+    
+    # Get stage-specific data
+    if stage == 1:  # Keywords → Clusters
+        queue = Keywords.objects.filter(
+            site=self.site, status='new'
+        ).order_by('id')
+        
+        return {
+            'stage_number': 1,
+            'stage_name': 'Keywords → Clusters',
+            'stage_type': 'AI',
+            'total_items': queue.count() + self._get_processed_count(stage),
+            'processed_items': self._get_processed_count(stage),
+            'currently_processing': self._get_current_items(queue, 3),
+            'up_next': self._get_next_items(queue, 2, skip=3),
+        }
+    
+    elif stage == 2:  # Clusters → Ideas
+        queue = Clusters.objects.filter(
+            site=self.site, status='new', disabled=False
+        ).order_by('id')
+        
+        return {
+            'stage_number': 2,
+            'stage_name': 'Clusters → Ideas',
+            'stage_type': 'AI',
+            'total_items': queue.count() + self._get_processed_count(stage),
+            'processed_items': self._get_processed_count(stage),
+            'currently_processing': self._get_current_items(queue, 1),
+            'up_next': self._get_next_items(queue, 2, skip=1),
+        }
+    
+    # ... similar for stages 3-6
+    
+def _get_processed_count(self, stage: int) -> int:
+    """Get count of items processed in current stage"""
+    result_key = f'stage_{stage}_result'
+    result = getattr(self.run, result_key, {})
+    
+    # Extract appropriate count from result
+    if stage == 1:
+        return result.get('keywords_processed', 0)
+    elif stage == 2:
+        return result.get('clusters_processed', 0)
+    # ... etc
+    
+def _get_current_items(self, queryset, count: int) -> list:
+    """Get currently processing items"""
+    items = queryset[:count]
+    return [
+        {
+            'id': item.id,
+            'title': getattr(item, 'keyword', None) or 
+                     getattr(item, 'cluster_name', None) or
+                     getattr(item, 'idea_title', None) or
+                     getattr(item, 'title', None),
+            'type': queryset.model.__name__.lower()
+        }
+        for item in items
+    ]
+```
+
+### Add View in AutomationViewSet
+
+**File:** `backend/igny8_core/business/automation/views.py`
+
+```python
+@action(detail=False, methods=['get'], url_path='current_processing')
+def current_processing(self, request):
+    """Get current processing state for active automation run"""
+    site_id = request.GET.get('site_id')
+    run_id = request.GET.get('run_id')
+    
+    if not site_id or not run_id:
+        return error_response(
+            error='site_id and run_id required',
+            status_code=400,
+            request=request
+        )
+    
+    try:
+        run = AutomationRun.objects.get(run_id=run_id, site_id=site_id)
+        
+        if run.status != 'running':
+            return success_response(data=None, request=request)
+        
+        service = AutomationService.from_run_id(run_id)
+        state = service.get_current_processing_state()
+        
+        return success_response(data=state, request=request)
+        
+    except AutomationRun.DoesNotExist:
+        return error_response(
+            error='Run not found',
+            status_code=404,
+            request=request
+        )
+```
+
+---
+
+## 🎨 FRONTEND CHANGES REQUIRED
+
+### 1. New Component: CurrentProcessingCard
+
+**File:** `frontend/src/components/Automation/CurrentProcessingCard.tsx`
+
+```typescript
+interface CurrentProcessingCardProps {
+  runId: string;
+  siteId: number;
+  currentStage: number;
+  onComplete?: () => void;
+}
+
+const CurrentProcessingCard: React.FC<CurrentProcessingCardProps> = ({
+  runId,
+  siteId,
+  currentStage,
+  onComplete
+}) => {
+  const [processingState, setProcessingState] = useState<ProcessingState | null>(null);
+  
+  // Poll every 3 seconds
+  useEffect(() => {
+    const fetchState = async () => {
+      const state = await automationService.getCurrentProcessing(siteId, runId);
+      setProcessingState(state);
+      
+      // If stage completed, trigger refresh
+      if (state && state.processed_items === state.total_items) {
+        onComplete?.();
+      }
+    };
+    
+    fetchState();
+    const interval = setInterval(fetchState, 3000);
+    
+    return () => clearInterval(interval);
+  }, [siteId, runId]);
+  
+  if (!processingState) return null;
+  
+  const percentage = Math.round(
+    (processingState.processed_items / processingState.total_items) * 100
+  );
+  
+  return (
+    <div className="bg-blue-50 dark:bg-blue-900/20 border-2 border-blue-500 rounded-lg p-6 mb-6">
+      {/* Header */}
+      <div className="flex items-center justify-between mb-4">
+        <div className="flex items-center gap-3">
+          <div className="animate-pulse">
+            <BoltIcon className="w-8 h-8 text-blue-600" />
+          </div>
+          <div>
+            <h2 className="text-2xl font-bold text-gray-900 dark:text-white">
+              Automation In Progress
+            </h2>
+            <p className="text-sm text-gray-600 dark:text-gray-400">
+              Stage {currentStage}: {processingState.stage_name}
+              <span className="ml-2 px-2 py-0.5 bg-blue-100 dark:bg-blue-900 text-blue-700 dark:text-blue-300 rounded text-xs">
+                {processingState.stage_type}
+              </span>
+            </p>
+          </div>
+        </div>
+        <div className="text-right">
+          <div className="text-4xl font-bold text-blue-600">{percentage}%</div>
+          <div className="text-sm text-gray-600 dark:text-gray-400">
+            {processingState.processed_items}/{processingState.total_items} processed
+          </div>
+        </div>
+      </div>
+      
+      {/* Progress Bar */}
+      <div className="mb-6">
+        <div className="w-full bg-gray-200 dark:bg-gray-700 rounded-full h-3">
+          <div
+            className="bg-blue-600 h-3 rounded-full transition-all duration-500"
+            style={{ width: `${percentage}%` }}
+          />
+        </div>
+      </div>
+      
+      {/* Currently Processing */}
+      <div className="grid grid-cols-1 md:grid-cols-2 gap-4">
+        <div>
+          <h3 className="text-sm font-semibold text-gray-700 dark:text-gray-300 mb-2">
+            Currently Processing:
+          </h3>
+          <div className="space-y-1">
+            {processingState.currently_processing.map((item, idx) => (
+              <div key={idx} className="flex items-start gap-2 text-sm">
+                <span className="text-blue-600 mt-1">•</span>
+                <span className="text-gray-800 dark:text-gray-200 font-medium">
+                  {item.title}
+                </span>
+              </div>
+            ))}
+          </div>
+        </div>
+        
+        <div>
+          <h3 className="text-sm font-semibold text-gray-700 dark:text-gray-300 mb-2">
+            Up Next:
+          </h3>
+          <div className="space-y-1">
+            {processingState.up_next.map((item, idx) => (
+              <div key={idx} className="flex items-start gap-2 text-sm">
+                <span className="text-gray-400 mt-1">•</span>
+                <span className="text-gray-600 dark:text-gray-400">
+                  {item.title}
+                </span>
+              </div>
+            ))}
+            {processingState.remaining_count > processingState.up_next.length && (
+              <div className="text-xs text-gray-500 mt-2">
+                + {processingState.remaining_count - processingState.up_next.length} more in queue
+              </div>
+            )}
+          </div>
+        </div>
+      </div>
+    </div>
+  );
+};
+```
+
+### 2. Update AutomationPage.tsx
+
+**File:** `frontend/src/pages/Automation/AutomationPage.tsx`
+
+```typescript
+// Add new import
+import CurrentProcessingCard from '../../components/Automation/CurrentProcessingCard';
+
+// In the component
+return (
+  <div className="p-6">
+    <PageMeta title="Automation" description="AI automation pipeline" />
+    
+    {/* Current Processing Card - MOVE TO TOP */}
+    {currentRun?.status === 'running' && (
+      <CurrentProcessingCard
+        runId={currentRun.run_id}
+        siteId={selectedSite.id}
+        currentStage={currentRun.current_stage}
+        onComplete={() => {
+          // Refresh full page metrics when stage completes
+          loadAutomationData();
+        }}
+      />
+    )}
+    
+    {/* Metrics Cards */}
+    <div className="grid grid-cols-1 md:grid-cols-3 gap-6 mb-6">
+      {/* ... existing metrics ... */}
+    </div>
+    
+    {/* Stages Section */}
+    <ComponentCard>
+      <h2 className="text-xl font-semibold mb-4">Pipeline Stages</h2>
+      {/* ... existing stages ... */}
+    </ComponentCard>
+    
+    {/* Rest of the page ... */}
+  </div>
+);
+```
+
+### 3. Add Service Method
+
+**File:** `frontend/src/services/automationService.ts`
+
+```typescript
+export interface ProcessingState {
+  run_id: string;
+  current_stage: number;
+  stage_name: string;
+  stage_type: 'AI' | 'Local' | 'Manual';
+  total_items: number;
+  processed_items: number;
+  percentage: number;
+  currently_processing: Array<{
+    id: number;
+    title: string;
+    type: string;
+  }>;
+  up_next: Array<{
+    id: number;
+    title: string;
+    type: string;
+  }>;
+  remaining_count: number;
+}
+
+// Add to automationService
+getCurrentProcessing: async (
+  siteId: number,
+  runId: string
+): Promise<ProcessingState | null> => {
+  return fetchAPI(
+    buildUrl('/current_processing/', { site_id: siteId, run_id: runId })
+  );
+},
+```
+
+---
+
+## 🧪 TESTING PLAN
+
+### Unit Tests
+
+- [ ] Test `get_current_processing_state()` for each stage
+- [ ] Test `_get_processed_count()` calculation
+- [ ] Test `_get_current_items()` formatting
+- [ ] Test API endpoint with various run states
+
+### Integration Tests
+
+- [ ] Test polling updates every 3 seconds
+- [ ] Test stage completion triggers full refresh
+- [ ] Test card disappears when automation completes
+- [ ] Test with 0 items (edge case)
+- [ ] Test with 1000+ items (performance)
+
+### Visual/UX Tests
+
+- [ ] Card positioned at top of page
+- [ ] Progress bar animates smoothly
+- [ ] Record names display correctly
+- [ ] Responsive design (mobile/tablet/desktop)
+- [ ] Dark mode support
+- [ ] Loading states
+- [ ] Error states
+
+---
+
+## 📊 STAGE-SPECIFIC DISPLAY FORMATS
+
+### Stage 1: Keywords → Clusters
+
+```
+Currently Processing:
+• "best seo tools"
+• "content marketing platforms"
+• "ai writing assistants"
+
++ 47 more keywords in queue
+
+Progress: 3/50 keywords processed
+```
+
+### Stage 2: Clusters → Ideas
+
+```
+Currently Processing:
+• "SEO Tools and Software" (Cluster #12)
+
+Up Next:
+• "Content Marketing Strategies"
+• "AI Content Generation"
+
+Progress: 12/25 clusters processed
+```
+
+### Stage 3: Ideas → Tasks
+
+```
+Currently Processing:
+• "10 Best SEO Tools for 2025"
+
+Up Next:
+• "How to Create Content with AI"
+• "Content Marketing ROI Calculator"
+
+Progress: 8/30 ideas processed
+```
+
+### Stage 4: Tasks → Content
+
+```
+Currently Processing:
+• "Ultimate Guide to SEO in 2025" (2,500 words)
+
+Up Next:
+• "AI Content Creation Best Practices"
+
+Progress: 5/15 tasks processed
+```
+
+### Stage 5: Content → Image Prompts
+
+```
+Currently Processing:
+• "How to Use ChatGPT for Content" (Extracting 3 image prompts)
+
+Up Next:
+• "Best AI Image Generators 2025"
+
+Progress: 10/15 content pieces processed
+```
+
+### Stage 6: Image Prompts → Images
+
+```
+Currently Processing:
+• Featured image for "SEO Guide 2025"
+
+Up Next:
+• In-article image #1 for "SEO Guide 2025"
+• In-article image #2 for "SEO Guide 2025"
+
+Progress: 15/45 images generated
+```
+
+### Stage 7: Manual Review Gate
+
+```
+Automation Complete! ✅
+
+Ready for Review:
+• "Ultimate Guide to SEO in 2025"
+• "AI Content Creation Best Practices"
+• "Best Image Generators 2025"
+
++ 12 more content pieces
+
+Total: 15 content pieces ready for review
+```
+
+---
+
+## 🎯 SUCCESS METRICS
+
+### User Experience
+
+✅ Users can see **exactly what's being processed** at any moment  
+✅ Users know **what's coming up next** in the queue  
+✅ Users can estimate **remaining time** based on progress  
+✅ Users get **quantitative feedback** (percentage, counts)  
+✅ Users see **smooth, non-disruptive updates** (no page flicker)
+
+### Technical
+
+✅ Polling interval: 3 seconds (balance between freshness and load)  
+✅ API response time: < 200ms  
+✅ Component re-render: Only the processing card, not entire page  
+✅ Memory usage: No memory leaks from polling  
+✅ Error handling: Graceful degradation if API fails
+
+---
+
+## 🚀 IMPLEMENTATION PHASES
+
+### Phase 1: Backend (1-2 days)
+- [ ] Implement `get_current_processing_state()` method
+- [ ] Add `/current_processing/` API endpoint
+- [ ] Test with all 7 stages
+- [ ] Add unit tests
+
+### Phase 2: Frontend (2-3 days)
+- [ ] Create `CurrentProcessingCard` component
+- [ ] Add polling logic with cleanup
+- [ ] Style with Tailwind (match existing design system)
+- [ ] Add dark mode support
+- [ ] Integrate into `AutomationPage`
+
+### Phase 3: Testing & Refinement (1-2 days)
+- [ ] Integration testing
+- [ ] Performance testing
+- [ ] UX testing
+- [ ] Bug fixes
+
+### Phase 4: Deployment
+- [ ] Deploy backend changes
+- [ ] Deploy frontend changes
+- [ ] Monitor first automation runs
+- [ ] Collect user feedback
+
+---
+
+## 🔄 FUTURE ENHANCEMENTS
+
+### V2 Features (Post-MVP)
+
+1. **Estimated Time Remaining:**
+   ```
+   Progress: 15/50 keywords processed
+   Estimated time remaining: ~8 minutes
+   ```
+
+2. **Stage-Level Progress Bar:**
+   - Each stage shows its own mini progress bar
+   - Visual indicator of which stages are complete
+
+3. **Click to View Details:**
+   - Click on a record name to see modal with details
+   - Preview generated content/images
+
+4. **Pause/Resume from Card:**
+   - Add pause button directly in the card
+   - Quick action without scrolling
+
+5. **Export Processing Log:**
+   - Download real-time processing log
+   - CSV of all processed items with timestamps
+
+---
+
+## END OF PLAN
+
+This plan provides a comprehensive UX improvement for automation progress tracking, making the process transparent and user-friendly while maintaining system performance.

docs/automation/automation-stage-6-image-generation-fix.md

@@ -0,0 +1,403 @@
+# Automation Stage 6 - Image Generation Fix Plan
+
+**Date:** December 4, 2025  
+**Status:** Analysis Complete - Implementation Required  
+**Priority:** HIGH
+
+---
+
+## 🔍 PROBLEM IDENTIFICATION
+
+### Current Issue
+
+Stage 6 of the automation pipeline (Image Prompts → Generated Images) is **NOT running correctly**. The issue stems from using the wrong AI function for image generation.
+
+### Root Cause Analysis
+
+**Current Implementation (INCORRECT):**
+```python
+# File: backend/igny8_core/business/automation/services/automation_service.py
+# Line ~935
+
+engine = AIEngine(account=self.account)
+result = engine.execute(
+    fn=GenerateImagesFunction(),
+    payload={'image_ids': [image.id]}  # ❌ WRONG
+)
+```
+
+**Why It Fails:**
+
+1. `GenerateImagesFunction()` expects:
+   - Input: `{'ids': [task_ids]}`  (Task IDs, NOT Image IDs)
+   - Purpose: Extract prompts from Tasks and generate images for tasks
+   - Use case: When you have Tasks with content but no images
+
+2. Automation Stage 6 has:
+   - Input: Images records with `status='pending'` (already have prompts)
+   - Purpose: Generate actual image URLs from existing prompts
+   - Context: Images were created in Stage 5 by `GenerateImagePromptsFunction`
+
+### How Other Stages Work Correctly
+
+**Stage 1:** Keywords → Clusters
+```python
+engine.execute(
+    fn=AutoClusterFunction(),
+    payload={'keyword_ids': keyword_ids}  # ✅ Correct
+)
+```
+
+**Stage 2:** Clusters → Ideas
+```python
+engine.execute(
+    fn=GenerateIdeasFunction(),
+    payload={'cluster_ids': cluster_ids}  # ✅ Correct
+)
+```
+
+**Stage 4:** Tasks → Content
+```python
+engine.execute(
+    fn=GenerateContentFunction(),
+    payload={'ids': task_ids}  # ✅ Correct
+)
+```
+
+**Stage 5:** Content → Image Prompts
+```python
+engine.execute(
+    fn=GenerateImagePromptsFunction(),
+    payload={'ids': content_ids}  # ✅ Correct
+)
+```
+
+**Stage 6:** Image Prompts → Images (BROKEN)
+```python
+# Currently uses GenerateImagesFunction (WRONG)
+# Should use process_image_generation_queue (CORRECT)
+```
+
+---
+
+## ✅ CORRECT SOLUTION
+
+### The Right Approach
+
+**Use `process_image_generation_queue` Celery task** - This is the same approach used by:
+1. Writer/Images page (`/writer/images/generate_images/` endpoint)
+2. Manual image generation from prompts
+
+**Evidence from Working Code:**
+
+```python
+# File: backend/igny8_core/modules/writer/views.py
+# ImagesViewSet.generate_images()
+
+from igny8_core.ai.tasks import process_image_generation_queue
+
+task = process_image_generation_queue.delay(
+    image_ids=image_ids,      # ✅ Accepts image_ids
+    account_id=account_id,
+    content_id=content_id
+)
+```
+
+**What `process_image_generation_queue` Does:**
+
+1. ✅ Accepts `image_ids` (list of Image record IDs)
+2. ✅ Each Image record already has a `prompt` field (populated by Stage 5)
+3. ✅ Generates images sequentially with progress tracking
+4. ✅ Updates Images records: `status='pending'` → `status='generated'`
+5. ✅ Downloads and saves images locally
+6. ✅ Automatically handles credits deduction
+7. ✅ Supports multiple providers (OpenAI, Runware)
+8. ✅ Handles errors gracefully (continues on failure)
+
+---
+
+## 📋 IMPLEMENTATION PLAN
+
+### Changes Required
+
+**File:** `backend/igny8_core/business/automation/services/automation_service.py`
+
+**Location:** `run_stage_6()` method (lines 874-1022)
+
+### Step 1: Import the Correct Task
+
+**Current:**
+```python
+from igny8_core.ai.functions.generate_images import GenerateImagesFunction
+```
+
+**Add:**
+```python
+from igny8_core.ai.tasks import process_image_generation_queue
+```
+
+### Step 2: Modify Stage 6 Logic
+
+**Replace this block (lines ~920-945):**
+
+```python
+# INCORRECT - Delete this
+for idx, image in enumerate(image_list, 1):
+    try:
+        content_title = image.content.title if image.content else 'Unknown'
+        self.logger.log_stage_progress(
+            self.run.run_id, self.account.id, self.site.id,
+            stage_number, f"Generating image {idx}/{total_images}: {image.image_type} for '{content_title}'"
+        )
+        
+        # Call AI function via AIEngine
+        engine = AIEngine(account=self.account)
+        result = engine.execute(
+            fn=GenerateImagesFunction(),
+            payload={'image_ids': [image.id]}  # ❌ WRONG
+        )
+        
+        # Monitor task
+        task_id = result.get('task_id')
+        if task_id:
+            self._wait_for_task(task_id, stage_number, f"Image for '{content_title}'", continue_on_error=True)
+        
+        images_processed += 1
+```
+
+**With this:**
+
+```python
+# CORRECT - Use process_image_generation_queue
+for idx, image in enumerate(image_list, 1):
+    try:
+        content_title = image.content.title if image.content else 'Unknown'
+        self.logger.log_stage_progress(
+            self.run.run_id, self.account.id, self.site.id,
+            stage_number, f"Generating image {idx}/{total_images}: {image.image_type} for '{content_title}'"
+        )
+        
+        # Call process_image_generation_queue directly (same as Writer/Images page)
+        from igny8_core.ai.tasks import process_image_generation_queue
+        
+        # Queue the task
+        if hasattr(process_image_generation_queue, 'delay'):
+            task = process_image_generation_queue.delay(
+                image_ids=[image.id],
+                account_id=self.account.id,
+                content_id=image.content.id if image.content else None
+            )
+            task_id = str(task.id)
+        else:
+            # Fallback for testing (synchronous)
+            result = process_image_generation_queue(
+                image_ids=[image.id],
+                account_id=self.account.id,
+                content_id=image.content.id if image.content else None
+            )
+            task_id = None
+        
+        # Monitor task (if async)
+        if task_id:
+            self._wait_for_task(task_id, stage_number, f"Image for '{content_title}'", continue_on_error=True)
+        
+        images_processed += 1
+```
+
+### Step 3: Update Logging
+
+The logging structure remains the same, just update the log messages to reflect the correct process:
+
+```python
+self.logger.log_stage_progress(
+    self.run.run_id, self.account.id, self.site.id,
+    stage_number, f"Image generation task queued for '{content_title}' ({images_processed}/{total_images})"
+)
+```
+
+### Step 4: No Changes Needed For
+
+✅ Stage 5 (Image Prompt Extraction) - Already correct  
+✅ Images table structure - Already has all required fields  
+✅ Progress tracking - Already implemented in `process_image_generation_queue`  
+✅ Credits deduction - Automatic in `process_image_generation_queue`  
+✅ Error handling - Built into the task with `continue_on_error=True`
+
+---
+
+## 🔄 HOW IT WORKS (CORRECTED FLOW)
+
+### Stage 5: Content → Image Prompts
+
+```
+Input:  Content (status='draft', no images)
+AI:     GenerateImagePromptsFunction
+Output: Images (status='pending', prompt='...')
+```
+
+### Stage 6: Image Prompts → Generated Images (FIXED)
+
+```
+Input:  Images (status='pending', has prompt)
+Task:   process_image_generation_queue (Celery task)
+AI:     Calls OpenAI/Runware API with prompt
+Output: Images (status='generated', image_url='https://...', image_path='/path/to/file')
+```
+
+### What Happens in process_image_generation_queue
+
+1. **Load Image Record:**
+   - Get Image by ID
+   - Read existing `prompt` field (created in Stage 5)
+   - Get Content for template formatting
+
+2. **Format Prompt:**
+   - Use image_prompt_template from PromptRegistry
+   - Format: `"Create a {image_type} image for '{post_title}'. Prompt: {image_prompt}"`
+   - Handle model-specific limits (DALL-E 3: 4000 chars, DALL-E 2: 1000 chars)
+
+3. **Generate Image:**
+   - Call `AICore.generate_image()`
+   - Uses configured provider (OpenAI/Runware)
+   - Uses configured model (dall-e-3, runware:97@1, etc.)
+   - Respects image size settings
+
+4. **Download & Save:**
+   - Download image from URL
+   - Save to `/data/app/igny8/frontend/public/images/ai-images/`
+   - Update Image record with both `image_url` and `image_path`
+
+5. **Update Status:**
+   - `status='pending'` → `status='generated'`
+   - Triggers automatic Content status update (if all images generated)
+
+6. **Deduct Credits:**
+   - Automatic via `AICore` credit system
+   - Records in `AIUsageLog`
+
+---
+
+## 🧪 TESTING CHECKLIST
+
+### Pre-Deployment Tests
+
+- [ ] **Unit Test:** Verify `process_image_generation_queue` works with single image
+- [ ] **Integration Test:** Run Stage 6 with 3-5 pending images
+- [ ] **Error Handling:** Test with invalid image ID
+- [ ] **Credits:** Verify credits are deducted correctly
+- [ ] **Multi-Provider:** Test with both OpenAI and Runware
+
+### Post-Deployment Validation
+
+- [ ] **Full Pipeline:** Run Automation from Stage 1 → Stage 7
+- [ ] **Verify Stage 5 Output:** Images created with `status='pending'` and prompts
+- [ ] **Verify Stage 6 Output:** Images updated to `status='generated'` with URLs
+- [ ] **Check Downloads:** Images saved to `/data/app/igny8/frontend/public/images/ai-images/`
+- [ ] **Monitor Logs:** Review automation logs for Stage 6 completion
+- [ ] **Credits Report:** Confirm Stage 6 credits recorded in automation results
+
+### Success Criteria
+
+✅ Stage 6 completes without errors  
+✅ All pending images get generated  
+✅ Images are downloaded and accessible  
+✅ Content status automatically updates when all images generated  
+✅ Credits are properly deducted and logged  
+✅ Automation proceeds to Stage 7 (Manual Review Gate)
+
+---
+
+## 📊 COMPARISON: BEFORE vs AFTER
+
+### BEFORE (Broken)
+
+```python
+# ❌ WRONG APPROACH
+GenerateImagesFunction()
+- Expects: task_ids
+- Purpose: Extract prompts from Tasks
+- Problem: Doesn't work with Images that already have prompts
+```
+
+**Result:** Stage 6 fails, images never generated
+
+### AFTER (Fixed)
+
+```python
+# ✅ CORRECT APPROACH
+process_image_generation_queue()
+- Accepts: image_ids
+- Purpose: Generate images from existing prompts
+- Works with: Images (status='pending' with prompts)
+```
+
+**Result:** Stage 6 succeeds, images generated sequentially with progress tracking
+
+---
+
+## 🔒 SAFETY & ROLLBACK
+
+### Backup Plan
+
+If the fix causes issues:
+
+1. **Rollback Code:**
+   - Git revert the automation_service.py changes
+   - Automation still works for Stages 1-5
+
+2. **Manual Workaround:**
+   - Users can manually generate images from Writer/Images page
+   - This uses the same `process_image_generation_queue` task
+
+3. **No Data Loss:**
+   - Stage 5 already created Images with prompts
+   - These remain in database and can be processed anytime
+
+---
+
+## 📝 IMPLEMENTATION STEPS
+
+1. **Update Code:** Modify `run_stage_6()` as documented above
+2. **Test Locally:** Run automation with test data
+3. **Code Review:** Verify changes match working Writer/Images implementation
+4. **Deploy:** Push to production
+5. **Monitor:** Watch first automation run for Stage 6 completion
+6. **Validate:** Check images generated and credits deducted
+
+---
+
+## 🎯 EXPECTED OUTCOME
+
+After implementing this fix:
+
+✅ **Stage 6 will work correctly** - Images generate from prompts  
+✅ **Consistent with manual flow** - Same logic as Writer/Images page  
+✅ **Proper credits tracking** - Automated deduction via AICore  
+✅ **Sequential processing** - One image at a time with progress  
+✅ **Error resilience** - Continues on failure, logs errors  
+✅ **Full pipeline completion** - Automation flows from Stage 1 → Stage 7  
+
+---
+
+## 🔗 RELATED FUNCTIONS
+
+### Keep These Functions (Working Correctly)
+
+- `GenerateImagePromptsFunction` - Stage 5 ✅
+- `AutoClusterFunction` - Stage 1 ✅
+- `GenerateIdeasFunction` - Stage 2 ✅
+- `GenerateContentFunction` - Stage 4 ✅
+
+### Use This Task for Stage 6
+
+- `process_image_generation_queue` - Celery task for Images → Generated Images ✅
+
+### DO NOT USE in Automation
+
+- `GenerateImagesFunction` - For Tasks, not for Images with existing prompts ❌
+
+---
+
+## END OF PLAN
+
+This plan provides a clear, actionable fix for Automation Stage 6 image generation, aligning it with the working manual image generation flow used throughout the application.

docs/billing/credits-system-audit-and-improvement-plan.md

@@ -0,0 +1,858 @@
+# Credits System - Complete Audit and Improvement Plan
+
+**Date:** December 4, 2025  
+**Status:** Audit Complete - Awaiting Implementation  
+**Priority:** HIGH
+
+---
+
+## 📋 EXECUTIVE SUMMARY
+
+This document provides a comprehensive audit of the IGNY8 credits system, identifies gaps and potential issues, and proposes improvements including backend admin configuration for credit costs per function.
+
+**Current State:** ✅ Working  
+**Areas for Improvement:** Configuration Management, Billing Integration, Admin UI, Reporting
+
+---
+
+## 🔍 SYSTEM ARCHITECTURE AUDIT
+
+### Current Credit System Components
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    CREDITS SYSTEM                            │
+├─────────────────────────────────────────────────────────────┤
+│                                                              │
+│  1. Account Model (credits field)                          │
+│  2. CreditTransaction Model (history)                      │
+│  3. CreditUsageLog Model (detailed usage)                  │
+│  4. CreditService (business logic)                         │
+│  5. CREDIT_COSTS (hardcoded constants)                     │
+│  6. Credit API Endpoints                                    │
+│  7. Frontend Dashboard                                      │
+│                                                              │
+└─────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 🗄️ DATABASE MODELS AUDIT
+
+### 1. Account Model (`auth.Account`)
+
+**Location:** `backend/igny8_core/auth/models.py`
+
+**Credits Field:**
+```python
+credits = models.IntegerField(default=0, help_text="Current credit balance")
+```
+
+**Status:** ✅ Working  
+**Findings:**
+- Simple integer field for balance
+- No soft delete or archive mechanism
+- No credit expiration tracking
+- No overdraft protection
+
+**Recommendations:**
+- ✅ Keep simple design (no changes needed)
+- Add `credits_expires_at` field for subscription credits
+- Add `bonus_credits` field separate from subscription credits
+
+---
+
+### 2. CreditTransaction Model
+
+**Location:** `backend/igny8_core/business/billing/models.py`
+
+**Current Structure:**
+```python
+class CreditTransaction(AccountBaseModel):
+    TRANSACTION_TYPE_CHOICES = [
+        ('purchase', 'Purchase'),
+        ('subscription', 'Subscription Renewal'),
+        ('refund', 'Refund'),
+        ('deduction', 'Usage Deduction'),
+        ('adjustment', 'Manual Adjustment'),
+    ]
+    
+    transaction_type = models.CharField(max_length=20, choices=TRANSACTION_TYPE_CHOICES)
+    amount = models.IntegerField()  # + for add, - for deduct
+    balance_after = models.IntegerField()
+    description = models.CharField(max_length=255)
+    metadata = models.JSONField(default=dict)
+    created_at = models.DateTimeField(auto_now_add=True)
+```
+
+**Status:** ✅ Working  
+**Findings:**
+- Comprehensive transaction types
+- Good metadata for context
+- Proper indexing on account/type/date
+- Missing: invoice_id FK, payment_method
+
+**Recommendations:**
+- Add `invoice_id` FK (for billing integration)
+- Add `payment_method` field ('stripe', 'manual', 'free')
+- Add `status` field ('pending', 'completed', 'failed', 'refunded')
+- Add `external_transaction_id` for Stripe payment IDs
+
+---
+
+### 3. CreditUsageLog Model
+
+**Location:** `backend/igny8_core/business/billing/models.py`
+
+**Current Structure:**
+```python
+class CreditUsageLog(AccountBaseModel):
+    OPERATION_TYPE_CHOICES = [
+        ('clustering', 'Clustering'),
+        ('idea_generation', 'Idea Generation'),
+        ('content_generation', 'Content Generation'),
+        ('image_prompt_extraction', 'Image Prompt Extraction'),
+        ('image_generation', 'Image Generation'),
+        ('linking', 'Content Linking'),
+        ('optimization', 'Content Optimization'),
+        ('site_structure_generation', 'Site Structure Generation'),
+        ('site_page_generation', 'Site Page Generation'),
+    ]
+    
+    operation_type = models.CharField(max_length=50, choices=OPERATION_TYPE_CHOICES)
+    credits_used = models.IntegerField()
+    cost_usd = models.DecimalField(max_digits=10, decimal_places=2, null=True)
+    model_used = models.CharField(max_length=100, blank=True)
+    tokens_input = models.IntegerField(null=True)
+    tokens_output = models.IntegerField(null=True)
+    related_object_type = models.CharField(max_length=50, blank=True)
+    related_object_id = models.IntegerField(null=True)
+    metadata = models.JSONField(default=dict)
+    created_at = models.DateTimeField(auto_now_add=True)
+```
+
+**Status:** ✅ Working  
+**Findings:**
+- Excellent detail tracking (model, tokens, cost)
+- Good related object tracking
+- Proper operation type choices
+- Missing: site/sector isolation, duration tracking
+
+**Recommendations:**
+- Add `site` FK for multi-tenant filtering
+- Add `sector` FK for isolation
+- Add `duration_seconds` field (API call time)
+- Add `success` boolean field (track failures)
+- Add `error_message` field for failed operations
+
+---
+
+## 💳 CREDIT COST CONFIGURATION AUDIT
+
+### Current Implementation: Hardcoded Constants
+
+**Location:** `backend/igny8_core/business/billing/constants.py`
+
+```python
+CREDIT_COSTS = {
+    'clustering': 10,                    # Per clustering request
+    'idea_generation': 15,               # Per cluster → ideas request
+    'content_generation': 1,             # Per 100 words
+    'image_prompt_extraction': 2,        # Per content piece
+    'image_generation': 5,               # Per image
+    'linking': 8,                        # Per content piece
+    'optimization': 1,                   # Per 200 words
+    'site_structure_generation': 50,     # Per site blueprint
+    'site_page_generation': 20,          # Per page
+}
+```
+
+**Status:** ⚠️ Working but NOT configurable  
+**Problems:**
+1. ❌ **Hardcoded values** - Requires code deployment to change
+2. ❌ **No admin UI** - Cannot adjust costs without developer
+3. ❌ **No versioning** - Cannot track cost changes over time
+4. ❌ **No A/B testing** - Cannot test different pricing
+5. ❌ **No per-account pricing** - All accounts same cost
+6. ❌ **No promotional pricing** - Cannot offer discounts
+
+---
+
+## 💰 BILLING & INVOICING GAPS
+
+### Current State
+
+**✅ Working:**
+- Credit deduction on AI operations
+- Credit transaction logging
+- Credit balance API
+- Credit usage API
+- Monthly credit replenishment (Celery task)
+
+**❌ Missing (NOT Implemented):**
+1. **Invoice Generation** - No Invoice model or PDF generation
+2. **Payment Processing** - No Stripe/PayPal integration
+3. **Subscription Management** - No recurring billing
+4. **Purchase Credits** - No one-time credit purchase flow
+5. **Refund Processing** - No refund workflow
+6. **Payment History** - No payment records
+7. **Tax Calculation** - No tax/VAT handling
+8. **Billing Address** - No billing info storage
+
+---
+
+## 🎯 PROPOSED SOLUTION: Backend Admin Configuration
+
+### New Model: CreditCostConfig
+
+**Purpose:** Make credit costs configurable from Django Admin
+
+**Location:** `backend/igny8_core/modules/billing/models.py`
+
+```python
+class CreditCostConfig(models.Model):
+    """
+    Configurable credit costs per AI function
+    Admin-editable alternative to hardcoded constants
+    """
+    # Operation identification
+    operation_type = models.CharField(
+        max_length=50,
+        unique=True,
+        choices=CreditUsageLog.OPERATION_TYPE_CHOICES,
+        help_text="AI operation type"
+    )
+    
+    # Cost configuration
+    credits_cost = models.IntegerField(
+        validators=[MinValueValidator(0)],
+        help_text="Credits required for this operation"
+    )
+    
+    # Unit of measurement
+    unit = models.CharField(
+        max_length=50,
+        default='per_request',
+        choices=[
+            ('per_request', 'Per Request'),
+            ('per_100_words', 'Per 100 Words'),
+            ('per_200_words', 'Per 200 Words'),
+            ('per_item', 'Per Item'),
+            ('per_image', 'Per Image'),
+        ],
+        help_text="What the cost applies to"
+    )
+    
+    # Metadata
+    display_name = models.CharField(max_length=100, help_text="Human-readable name")
+    description = models.TextField(blank=True, help_text="What this operation does")
+    
+    # Status
+    is_active = models.BooleanField(default=True, help_text="Enable/disable this operation")
+    
+    # Audit fields
+    created_at = models.DateTimeField(auto_now_add=True)
+    updated_at = models.DateTimeField(auto_now=True)
+    updated_by = models.ForeignKey(
+        'auth.User',
+        null=True,
+        blank=True,
+        on_delete=models.SET_NULL,
+        help_text="Admin who last updated"
+    )
+    
+    # Change tracking
+    previous_cost = models.IntegerField(
+        null=True,
+        blank=True,
+        help_text="Cost before last update (for audit trail)"
+    )
+    
+    class Meta:
+        db_table = 'igny8_credit_cost_config'
+        verbose_name = 'Credit Cost Configuration'
+        verbose_name_plural = 'Credit Cost Configurations'
+        ordering = ['operation_type']
+    
+    def __str__(self):
+        return f"{self.display_name} - {self.credits_cost} credits {self.unit}"
+    
+    def save(self, *args, **kwargs):
+        # Track cost changes
+        if self.pk:
+            old = CreditCostConfig.objects.get(pk=self.pk)
+            if old.credits_cost != self.credits_cost:
+                self.previous_cost = old.credits_cost
+        super().save(*args, **kwargs)
+```
+
+---
+
+### Django Admin Configuration
+
+**Location:** `backend/igny8_core/modules/billing/admin.py`
+
+```python
+from django.contrib import admin
+from django.utils.html import format_html
+from .models import CreditCostConfig
+
+@admin.register(CreditCostConfig)
+class CreditCostConfigAdmin(admin.ModelAdmin):
+    list_display = [
+        'operation_type',
+        'display_name',
+        'credits_cost_display',
+        'unit',
+        'is_active',
+        'cost_change_indicator',
+        'updated_at',
+        'updated_by'
+    ]
+    
+    list_filter = ['is_active', 'unit', 'updated_at']
+    search_fields = ['operation_type', 'display_name', 'description']
+    
+    fieldsets = (
+        ('Operation', {
+            'fields': ('operation_type', 'display_name', 'description')
+        }),
+        ('Cost Configuration', {
+            'fields': ('credits_cost', 'unit', 'is_active')
+        }),
+        ('Audit Trail', {
+            'fields': ('previous_cost', 'updated_by', 'created_at', 'updated_at'),
+            'classes': ('collapse',)
+        }),
+    )
+    
+    readonly_fields = ['created_at', 'updated_at', 'previous_cost']
+    
+    def credits_cost_display(self, obj):
+        """Show cost with color coding"""
+        if obj.credits_cost >= 20:
+            color = 'red'
+        elif obj.credits_cost >= 10:
+            color = 'orange'
+        else:
+            color = 'green'
+        return format_html(
+            '<span style="color: {}; font-weight: bold;">{} credits</span>',
+            color,
+            obj.credits_cost
+        )
+    credits_cost_display.short_description = 'Cost'
+    
+    def cost_change_indicator(self, obj):
+        """Show if cost changed recently"""
+        if obj.previous_cost is not None:
+            if obj.credits_cost > obj.previous_cost:
+                icon = '📈'  # Increased
+                color = 'red'
+            elif obj.credits_cost < obj.previous_cost:
+                icon = '📉'  # Decreased
+                color = 'green'
+            else:
+                icon = '➡️'  # Same
+                color = 'gray'
+            
+            return format_html(
+                '{} <span style="color: {};">({} → {})</span>',
+                icon,
+                color,
+                obj.previous_cost,
+                obj.credits_cost
+            )
+        return '—'
+    cost_change_indicator.short_description = 'Recent Change'
+    
+    def save_model(self, request, obj, form, change):
+        """Track who made the change"""
+        obj.updated_by = request.user
+        super().save_model(request, obj, form, change)
+```
+
+---
+
+### Updated CreditService to Use Database
+
+**Location:** `backend/igny8_core/business/billing/services/credit_service.py`
+
+```python
+class CreditService:
+    """Service for managing credits"""
+    
+    @staticmethod
+    def get_credit_cost(operation_type, amount=None):
+        """
+        Get credit cost for an operation.
+        Now checks database config first, falls back to constants.
+        
+        Args:
+            operation_type: Type of operation
+            amount: Optional amount (word count, image count, etc.)
+        
+        Returns:
+            int: Credit cost
+        """
+        # Try to get from database config first
+        try:
+            from igny8_core.modules.billing.models import CreditCostConfig
+            
+            config = CreditCostConfig.objects.filter(
+                operation_type=operation_type,
+                is_active=True
+            ).first()
+            
+            if config:
+                base_cost = config.credits_cost
+                
+                # Apply unit-based calculation
+                if config.unit == 'per_100_words' and amount:
+                    return max(1, (amount // 100)) * base_cost
+                elif config.unit == 'per_200_words' and amount:
+                    return max(1, (amount // 200)) * base_cost
+                elif config.unit in ['per_item', 'per_image'] and amount:
+                    return amount * base_cost
+                else:
+                    return base_cost
+        
+        except Exception as e:
+            logger.warning(f"Failed to get cost from database, using constants: {e}")
+        
+        # Fallback to hardcoded constants
+        from igny8_core.business.billing.constants import CREDIT_COSTS
+        
+        base_cost = CREDIT_COSTS.get(operation_type, 1)
+        
+        # Apply multipliers for word-based operations
+        if operation_type == 'content_generation' and amount:
+            return max(1, (amount // 100))  # 1 credit per 100 words
+        elif operation_type == 'optimization' and amount:
+            return max(1, (amount // 200))  # 1 credit per 200 words
+        elif operation_type in ['image_generation'] and amount:
+            return amount * base_cost
+        else:
+            return base_cost
+```
+
+---
+
+## 📊 ADMIN UI - CREDIT COST CONFIGURATION
+
+### Management Command: Initialize Credit Costs
+
+**Location:** `backend/igny8_core/modules/billing/management/commands/init_credit_costs.py`
+
+```python
+from django.core.management.base import BaseCommand
+from igny8_core.modules.billing.models import CreditCostConfig
+from igny8_core.business.billing.constants import CREDIT_COSTS
+
+class Command(BaseCommand):
+    help = 'Initialize credit cost configurations from constants'
+    
+    def handle(self, *args, **options):
+        """Migrate hardcoded costs to database"""
+        
+        operation_metadata = {
+            'clustering': {
+                'display_name': 'Auto Clustering',
+                'description': 'Group keywords into semantic clusters using AI',
+                'unit': 'per_request'
+            },
+            'idea_generation': {
+                'display_name': 'Idea Generation',
+                'description': 'Generate content ideas from keyword clusters',
+                'unit': 'per_request'
+            },
+            'content_generation': {
+                'display_name': 'Content Generation',
+                'description': 'Generate article content using AI',
+                'unit': 'per_100_words'
+            },
+            'image_prompt_extraction': {
+                'display_name': 'Image Prompt Extraction',
+                'description': 'Extract image prompts from content',
+                'unit': 'per_request'
+            },
+            'image_generation': {
+                'display_name': 'Image Generation',
+                'description': 'Generate images using AI (DALL-E, Runware)',
+                'unit': 'per_image'
+            },
+            'linking': {
+                'display_name': 'Content Linking',
+                'description': 'Generate internal links between content',
+                'unit': 'per_request'
+            },
+            'optimization': {
+                'display_name': 'Content Optimization',
+                'description': 'Optimize content for SEO',
+                'unit': 'per_200_words'
+            },
+            'site_structure_generation': {
+                'display_name': 'Site Structure Generation',
+                'description': 'Generate complete site blueprint',
+                'unit': 'per_request'
+            },
+            'site_page_generation': {
+                'display_name': 'Site Page Generation',
+                'description': 'Generate site pages from blueprint',
+                'unit': 'per_item'
+            },
+        }
+        
+        created_count = 0
+        updated_count = 0
+        
+        for operation_type, cost in CREDIT_COSTS.items():
+            # Skip legacy aliases
+            if operation_type in ['ideas', 'content', 'images', 'reparse']:
+                continue
+            
+            metadata = operation_metadata.get(operation_type, {})
+            
+            config, created = CreditCostConfig.objects.get_or_create(
+                operation_type=operation_type,
+                defaults={
+                    'credits_cost': cost,
+                    'display_name': metadata.get('display_name', operation_type.replace('_', ' ').title()),
+                    'description': metadata.get('description', ''),
+                    'unit': metadata.get('unit', 'per_request'),
+                    'is_active': True
+                }
+            )
+            
+            if created:
+                created_count += 1
+                self.stdout.write(
+                    self.style.SUCCESS(f'✅ Created: {config.display_name} - {cost} credits')
+                )
+            else:
+                updated_count += 1
+                self.stdout.write(
+                    self.style.WARNING(f'⚠️  Already exists: {config.display_name}')
+                )
+        
+        self.stdout.write(
+            self.style.SUCCESS(f'\n✅ Complete: {created_count} created, {updated_count} already existed')
+        )
+```
+
+**Run command:**
+```bash
+python manage.py init_credit_costs
+```
+
+---
+
+## 🔍 POTENTIAL ISSUES & FIXES
+
+### Issue 1: Race Conditions in Credit Deduction
+
+**Problem:**
+```python
+# Current code (simplified)
+if account.credits < required:
+    raise InsufficientCreditsError()
+account.credits -= required  # Race condition here!
+account.save()
+```
+
+**Risk:** Two requests could both check balance and deduct simultaneously
+
+**Fix:** Use database-level atomic update
+```python
+from django.db.models import F
+
+@transaction.atomic
+def deduct_credits(account, amount):
+    # Atomic update with check
+    updated = Account.objects.filter(
+        id=account.id,
+        credits__gte=amount  # Check in database
+    ).update(
+        credits=F('credits') - amount
+    )
+    
+    if updated == 0:
+        raise InsufficientCreditsError()
+    
+    account.refresh_from_db()
+    return account.credits
+```
+
+---
+
+### Issue 2: Negative Credit Balance
+
+**Problem:** No hard constraint preventing negative credits
+
+**Fix 1:** Database constraint
+```python
+# Migration
+operations = [
+    migrations.AddConstraint(
+        model_name='account',
+        constraint=models.CheckConstraint(
+            check=models.Q(credits__gte=0),
+            name='credits_non_negative'
+        ),
+    ),
+]
+```
+
+**Fix 2:** Service layer validation (current approach - OK)
+
+---
+
+### Issue 3: Missing Credit Expiration
+
+**Problem:** Subscription credits never expire
+
+**Fix:** Add expiration tracking
+```python
+# Account model
+credits_expires_at = models.DateTimeField(null=True, blank=True)
+
+# Celery task (daily)
+@shared_task
+def expire_credits():
+    """Expire old credits"""
+    expired = Account.objects.filter(
+        credits_expires_at__lt=timezone.now(),
+        credits__gt=0
+    )
+    
+    for account in expired:
+        # Transfer to expired_credits field or log
+        account.credits = 0
+        account.save()
+```
+
+---
+
+### Issue 4: No Usage Analytics
+
+**Problem:** Hard to analyze which functions cost most credits
+
+**Fix:** Add aggregation views
+```python
+# Backend
+@action(detail=False, methods=['get'])
+def cost_breakdown(self, request):
+    """Get credit cost breakdown by operation"""
+    from django.db.models import Sum
+    
+    breakdown = CreditUsageLog.objects.filter(
+        account=request.account
+    ).values('operation_type').annotate(
+        total_credits=Sum('credits_used'),
+        count=Count('id')
+    ).order_by('-total_credits')
+    
+    return Response(breakdown)
+```
+
+---
+
+### Issue 5: No Budget Alerts
+
+**Problem:** Users can run out of credits unexpectedly
+
+**Fix:** Add threshold alerts
+```python
+# After each deduction
+def check_low_balance(account):
+    if account.credits < 100:  # Configurable threshold
+        send_low_balance_email(account)
+        
+    if account.credits < 50:
+        send_critical_balance_email(account)
+```
+
+---
+
+## 📈 FUTURE ENHANCEMENTS
+
+### Phase 1: Billing Integration (Priority: HIGH)
+
+**Models to Add:**
+1. `Invoice` - Store invoice records
+2. `Payment` - Track payments
+3. `Subscription` - Recurring billing
+4. `CreditPackage` - One-time credit purchases
+
+**Integrations:**
+- Stripe for payments
+- PDF generation for invoices
+- Email notifications
+
+---
+
+### Phase 2: Advanced Pricing (Priority: MEDIUM)
+
+**Features:**
+1. **Volume Discounts**
+   - 1000+ credits/month: 10% discount
+   - 5000+ credits/month: 20% discount
+
+2. **Per-Account Pricing**
+   - Enterprise accounts: Custom pricing
+   - Trial accounts: Limited operations
+
+3. **Promotional Codes**
+   - Discount codes
+   - Free credit grants
+
+4. **Credit Bundles**
+   - Starter: 500 credits
+   - Pro: 2000 credits
+   - Enterprise: 10000 credits
+
+---
+
+### Phase 3: Usage Analytics Dashboard (Priority: MEDIUM)
+
+**Features:**
+1. **Cost Breakdown Charts**
+   - By operation type
+   - By time period
+   - By site/sector
+
+2. **Trend Analysis**
+   - Daily/weekly/monthly usage
+   - Forecasting
+   - Budget alerts
+
+3. **Comparison Reports**
+   - Compare accounts
+   - Compare time periods
+   - Benchmark against averages
+
+---
+
+## 🧪 TESTING CHECKLIST
+
+### Unit Tests Required
+
+- [ ] Test CreditCostConfig model creation
+- [ ] Test CreditService with database config
+- [ ] Test fallback to constants
+- [ ] Test atomic credit deduction
+- [ ] Test negative balance prevention
+- [ ] Test cost calculation with units
+
+### Integration Tests Required
+
+- [ ] Test full credit deduction flow
+- [ ] Test monthly replenishment
+- [ ] Test admin UI operations
+- [ ] Test concurrent deductions (race conditions)
+- [ ] Test cost changes propagate correctly
+
+### Manual Testing Required
+
+- [ ] Create credit config in Django Admin
+- [ ] Update cost and verify in logs
+- [ ] Deactivate operation and verify rejection
+- [ ] Test with different units (per 100 words, per image)
+- [ ] Verify audit trail (previous_cost, updated_by)
+
+---
+
+## 📋 IMPLEMENTATION ROADMAP
+
+### Week 1: Database Configuration
+- [ ] Create `CreditCostConfig` model
+- [ ] Create migration
+- [ ] Create Django Admin
+- [ ] Create init_credit_costs command
+- [ ] Update CreditService to use database
+
+### Week 2: Testing & Refinement
+- [ ] Write unit tests
+- [ ] Write integration tests
+- [ ] Manual QA testing
+- [ ] Fix race conditions
+- [ ] Add constraints
+
+### Week 3: Documentation & Deployment
+- [ ] Update API documentation
+- [ ] Create admin user guide
+- [ ] Deploy to staging
+- [ ] User acceptance testing
+- [ ] Deploy to production
+
+### Week 4: Monitoring & Optimization
+- [ ] Monitor cost changes
+- [ ] Analyze usage patterns
+- [ ] Optimize slow queries
+- [ ] Plan Phase 2 features
+
+---
+
+## 🎯 SUCCESS CRITERIA
+
+✅ **Backend Admin:** Credits configurable via Django Admin  
+✅ **No Code Deploys:** Cost changes don't require deployment  
+✅ **Audit Trail:** Track who changed costs and when  
+✅ **Backward Compatible:** Existing code continues to work  
+✅ **Performance:** No regression in credit deduction speed  
+✅ **Data Integrity:** No race conditions or negative balances  
+✅ **Testing:** 100% test coverage for critical paths
+
+---
+
+## 📊 CREDITS SYSTEM FLOWCHART
+
+```mermaid
+graph TD
+    A[AI Operation Request] --> B{Check CreditCostConfig}
+    B -->|Found| C[Get Cost from Database]
+    B -->|Not Found| D[Get Cost from Constants]
+    C --> E[Calculate Total Cost]
+    D --> E
+    E --> F{Sufficient Credits?}
+    F -->|Yes| G[Atomic Deduct Credits]
+    F -->|No| H[Raise InsufficientCreditsError]
+    G --> I[Create CreditTransaction]
+    I --> J[Create CreditUsageLog]
+    J --> K[Return Success]
+    H --> L[Return Error]
+```
+
+---
+
+## 🔐 SECURITY CONSIDERATIONS
+
+### Credit Manipulation Prevention
+
+1. **No Client-Side Credit Calculation**
+   - All calculations server-side
+   - Credits never exposed in API responses
+
+2. **Atomic Transactions**
+   - Use database transactions
+   - Prevent race conditions
+
+3. **Audit Logging**
+   - Log all credit changes
+   - Track who/when/why
+
+4. **Rate Limiting**
+   - Prevent credit abuse
+   - Throttle expensive operations
+
+5. **Admin Permissions**
+   - Only superusers can modify costs
+   - Track all admin changes
+
+---
+
+## END OF AUDIT
+
+This comprehensive audit identifies all aspects of the credits system, proposes a database-driven configuration approach, and provides a clear roadmap for implementation. The system is currently working well but lacks flexibility for cost adjustments without code deployments.
+
+**Recommendation:** Implement the CreditCostConfig model in Phase 1 to enable admin-configurable costs.