audits

2025-11-10 23:04:12 +05:00
parent 6c9da30b68
commit ecbab4d380
2 changed files with 1018 additions and 0 deletions
--- a/IGNY8_AI_AUDIT_PLAN.md
+++ b/IGNY8_AI_AUDIT_PLAN.md
@@ -0,0 +1,247 @@
+# IGNY8 AI System Audit — Execution Plan
+
+## Objective
+Perform a complete structural and functional audit of the IGNY8 AI subsystem exactly as it exists, without any modifications, renaming, or assumptions. Document all findings in a baseline report.
+
+## Scope
+
+### Primary Directory: `backend/igny8_core/ai/`
+**Core AI Files (15 files):**
+- `__init__.py` - Package initialization and exports
+- `admin.py` - Django admin configuration for AI models
+- `ai_core.py` - Core AI functionality
+- `apps.py` - Django app configuration
+- `base.py` - Base classes or utilities
+- `constants.py` - AI-related constants
+- `engine.py` - AI engine implementation
+- `models.py` - Database models for AI entities
+- `processor.py` - AI processing logic
+- `prompts.py` - Prompt templates and management
+- `registry.py` - Function/component registry
+- `settings.py` - AI-specific settings
+- `tasks.py` - Celery task definitions
+- `tracker.py` - Progress tracking and state management
+- `types.py` - Type definitions and schemas
+- `validators.py` - Validation logic
+
+**AI Functions Subdirectory (5 files):**
+- `functions/__init__.py` - Function package exports
+- `functions/auto_cluster.py` - Automatic clustering functionality
+- `functions/generate_content.py` - Content generation logic
+- `functions/generate_ideas.py` - Idea generation logic
+- `functions/generate_images.py` - Image generation logic
+
+### Related Directories
+
+**`backend/igny8_core/utils/` (4 files):**
+- `ai_processor.py` - AI processing utilities
+- `content_normalizer.py` - Content normalization utilities
+- `queue_manager.py` - Queue management utilities
+- `wordpress.py` - WordPress integration utilities
+
+**`backend/igny8_core/modules/` (AI-related files):**
+- `planner/tasks.py` - Planner module Celery tasks
+- `writer/tasks.py` - Writer module Celery tasks
+- `system/models.py` - System models (may contain AI settings)
+- `system/settings_models.py` - Settings models
+- `system/settings_views.py` - Settings views
+- `system/views.py` - System views
+- `system/utils.py` - System utilities
+
+**Configuration Files:**
+- `backend/igny8_core/celery.py` - Celery configuration and task registration
+- `backend/igny8_core/settings.py` - Django settings (AI configuration loading)
+
+## Audit Methodology
+
+### Phase 1: File Inventory and Initial Reading
+1. Read all files in `backend/igny8_core/ai/` directory
+2. Read all files in `backend/igny8_core/ai/functions/` directory
+3. Read AI-related files in `backend/igny8_core/utils/`
+4. Read AI-related task files in `backend/igny8_core/modules/`
+5. Read configuration and integration files
+
+### Phase 2: Function and Class Analysis
+1. Extract all function definitions with:
+   - Function name
+   - Parameters and types
+   - Return values
+   - Docstrings/documentation
+   - Decorators (especially Celery tasks)
+2. Extract all class definitions with:
+   - Class name
+   - Inheritance hierarchy
+   - Methods and their purposes
+   - Class-level attributes
+3. Identify call sites for each function/class method
+
+### Phase 3: Dependency Mapping
+1. Map import relationships:
+   - Which files import from which files
+   - External dependencies (libraries, Django, Celery)
+   - Circular dependencies (if any)
+2. Create dependency graph/table showing:
+   - Direct imports
+   - Indirect dependencies
+   - Shared utilities
+
+### Phase 4: System Flow Analysis
+1. Trace request flow:
+   - Frontend API endpoints → Views/Serializers
+   - Views → Celery tasks
+   - Celery tasks → AI functions
+   - AI functions → External APIs/Models
+   - Results → Database storage
+   - Results → Response to frontend
+2. Document:
+   - Entry points (API endpoints, admin actions, management commands)
+   - Task queue flow (Celery task registration and execution)
+   - State management (tracker, progress updates)
+   - Error handling paths
+   - Logging and debug output
+
+### Phase 5: Integration Points Analysis
+1. **Celery Integration:**
+   - Task registration in `celery.py`
+   - Task decorators and configurations
+   - Task routing and queues
+   - Async execution patterns
+
+2. **Database Integration:**
+   - Models used by AI subsystem
+   - Model relationships
+   - Data persistence patterns
+   - Query patterns
+
+3. **Frontend Integration:**
+   - API endpoints that trigger AI tasks
+   - Serializers for AI data
+   - Response formats
+   - WebSocket/SSE for progress updates (if any)
+
+4. **Configuration Integration:**
+   - Settings loading (Django settings, environment variables)
+   - Model/provider configuration
+   - API key management
+   - Feature flags or switches
+
+5. **Debug Panel Integration:**
+   - Debug logging mechanisms
+   - Progress tracking
+   - State inspection tools
+
+### Phase 6: Redundancy and Pattern Identification
+1. Identify:
+   - Duplicated code blocks
+   - Similar functions with slight variations
+   - Repeated patterns that could indicate consolidation opportunities
+   - Unused or dead code
+   - Overlapping responsibilities
+
+2. Document patterns:
+   - Common error handling approaches
+   - Repeated validation logic
+   - Similar processing pipelines
+   - Shared utility patterns
+
+### Phase 7: Documentation Compilation
+Create structured document with sections:
+1. **Current File Inventory** - List all files with brief role descriptions
+2. **Function Inventory** - Comprehensive list of all functions with descriptions
+3. **Class Inventory** - All classes and their purposes
+4. **Dependency Graph/Table** - Import relationships and dependencies
+5. **System Flow Description** - End-to-end flow documentation
+6. **Integration Points** - Detailed integration documentation
+7. **Identified Redundancies** - Patterns and duplications found
+8. **Summary of Potential Consolidation Areas** - Observations only (no refactoring proposals)
+
+## Execution Rules
+
+### Strict Guidelines:
+- ✅ **DO:** Read all code exactly as written
+- ✅ **DO:** Document what exists without modification
+- ✅ **DO:** Label any assumptions explicitly
+- ✅ **DO:** Trace actual code paths, not theoretical ones
+- ✅ **DO:** Include line numbers and file paths for references
+
+### Prohibited Actions:
+- ❌ **DON'T:** Rename anything
+- ❌ **DON'T:** Merge or consolidate code
+- ❌ **DON'T:** Propose new architecture
+- ❌ **DON'T:** Suggest simplifications
+- ❌ **DON'T:** Make any code changes
+- ❌ **DON'T:** Create new files (except the audit document)
+- ❌ **DON'T:** Assume functionality without reading code
+
+## Deliverable
+
+**Document Title:** `IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md`
+
+**Structure:**
+```markdown
+# IGNY8 AI System Audit — Current Structure & Flow Mapping (Baseline Report)
+
+## Executive Summary
+[Brief overview of findings]
+
+## 1. Current File Inventory
+[Complete list with descriptions]
+
+## 2. Function Inventory
+[All functions documented]
+
+## 3. Class Inventory
+[All classes documented]
+
+## 4. Dependency Graph/Table
+[Import relationships]
+
+## 5. System Flow Description
+[End-to-end flows]
+
+## 6. Integration Points
+[Celery, Database, Frontend, Configuration, Debug]
+
+## 7. Identified Redundancies or Repetition
+[Patterns found]
+
+## 8. Summary of Potential Consolidation Areas
+[Observations only]
+
+## 9. Assumptions Made
+[Any assumptions explicitly labeled]
+
+## 10. Appendix
+[Additional details, code snippets, etc.]
+```
+
+## Execution Checklist
+
+- [ ] Phase 1: Read all AI core files
+- [ ] Phase 1: Read all AI function files
+- [ ] Phase 1: Read all utility files
+- [ ] Phase 1: Read all module task files
+- [ ] Phase 1: Read configuration files
+- [ ] Phase 2: Extract and document all functions
+- [ ] Phase 2: Extract and document all classes
+- [ ] Phase 3: Map all import dependencies
+- [ ] Phase 4: Trace system flows
+- [ ] Phase 5: Document integration points
+- [ ] Phase 6: Identify redundancies
+- [ ] Phase 7: Compile final audit document
+
+## Estimated File Count
+- **AI Core Files:** 15 files
+- **AI Functions:** 5 files
+- **Utilities:** 4 files
+- **Module Tasks:** 2 files
+- **System Module:** ~5 files
+- **Configuration:** 2 files
+- **Total:** ~33 files to analyze
+
+## Notes
+- This is a discovery phase only
+- All findings must be based on actual code
+- No refactoring or improvements will be proposed
+- The goal is to understand the current state completely
+
--- a/IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md
+++ b/IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md
@@ -0,0 +1,771 @@
+# IGNY8 AI System Audit — Current Structure & Flow Mapping (Baseline Report)
+
+**Date:** 2024-12-19  
+**Scope:** Complete structural and functional audit of the IGNY8 AI subsystem  
+**Methodology:** Code analysis without modifications or assumptions
+
+---
+
+## Executive Summary
+
+The IGNY8 AI subsystem is a comprehensive framework for AI-powered content operations including keyword clustering, idea generation, content generation, and image generation. The system uses a unified framework architecture with a centralized execution engine, but also maintains legacy code paths for backward compatibility.
+
+**Key Findings:**
+- **20 core AI files** in `backend/igny8_core/ai/`
+- **4 AI function implementations** following BaseAIFunction pattern
+- **Dual execution paths:** New unified framework (`run_ai_task` → `AIEngine`) and legacy paths (direct task calls)
+- **Centralized AI request handling** via `AICore.run_ai_request()`
+- **Comprehensive tracking** via `StepTracker`, `ProgressTracker`, `CostTracker`, and `ConsoleStepTracker`
+- **Multiple prompt management systems:** `PromptRegistry` (new) and direct database queries (legacy)
+
+---
+
+## 1. Current File Inventory
+
+### 1.1 Core AI Framework (`backend/igny8_core/ai/`)
+
+| File | Lines | Purpose |
+|------|-------|---------|
+| `__init__.py` | 73 | Package exports - exposes main classes and functions |
+| `admin.py` | 60 | Django admin configuration for `AITaskLog` model |
+| `ai_core.py` | 756 | Centralized AI request handler - `AICore` class with `run_ai_request()` and `generate_image()` |
+| `apps.py` | 21 | Django app configuration |
+| `base.py` | 95 | Abstract base class `BaseAIFunction` - defines function interface |
+| `constants.py` | 42 | Model pricing, valid models, configuration constants |
+| `engine.py` | 375 | Central orchestrator `AIEngine` - manages function lifecycle |
+| `models.py` | 52 | Database model `AITaskLog` for unified logging |
+| `processor.py` | 76 | **DEPRECATED** wrapper around `AICore` for backward compatibility |
+| `prompts.py` | 432 | `PromptRegistry` - centralized prompt management with hierarchical resolution |
+| `registry.py` | 97 | Function registry with lazy loading - `register_function()`, `get_function()` |
+| `settings.py` | 117 | Model configurations per function - `MODEL_CONFIG`, `get_model_config()` |
+| `tasks.py` | 131 | Unified Celery task entrypoint `run_ai_task()` - single entry point for all AI functions |
+| `tracker.py` | 348 | Progress tracking utilities - `StepTracker`, `ProgressTracker`, `CostTracker`, `ConsoleStepTracker` |
+| `types.py` | 44 | Type definitions - `StepLog`, `ProgressState`, `AITaskResult` dataclasses |
+| `validators.py` | 187 | Validation functions - `validate_ids()`, `validate_keywords_exist()`, etc. |
+
+### 1.2 AI Function Implementations (`backend/igny8_core/ai/functions/`)
+
+| File | Lines | Purpose |
+|------|-------|---------|
+| `__init__.py` | 18 | Function package exports |
+| `auto_cluster.py` | 330 | `AutoClusterFunction` - Groups keywords into semantic clusters |
+| `generate_content.py` | 388 | `GenerateContentFunction` + `generate_content_core()` - Generates article content |
+| `generate_ideas.py` | 335 | `GenerateIdeasFunction` + `generate_ideas_core()` - Generates content ideas from clusters |
+| `generate_images.py` | 279 | `GenerateImagesFunction` + `generate_images_core()` - Generates images for tasks |
+
+### 1.3 Utility Files (`backend/igny8_core/utils/`)
+
+| File | Lines | Purpose |
+|------|-------|---------|
+| `ai_processor.py` | 1407 | **LEGACY** Unified AI interface - `AIProcessor` class with OpenAI/Runware support. Contains duplicate constants and methods. |
+| `content_normalizer.py` | 273 | Content normalization - converts AI responses to HTML format |
+| `queue_manager.py` | 90 | Queue abstraction (currently placeholder, not fully implemented) |
+| `wordpress.py` | (not read) | WordPress integration utilities |
+
+### 1.4 Module Task Files
+
+| File | Lines | Purpose |
+|------|-------|---------|
+| `modules/planner/tasks.py` | 736 | **DEPRECATED** Legacy clustering task `auto_cluster_keywords_task()` - uses old `AIProcessor` |
+| `modules/writer/tasks.py` | 1156 | Legacy content/image generation tasks - `auto_generate_content_task()`, `auto_generate_images_task()` |
+
+### 1.5 Configuration Files
+
+| File | Purpose |
+|------|---------|
+| `celery.py` | Celery app configuration - auto-discovers tasks from all Django apps |
+| `modules/system/models.py` | `AIPrompt`, `IntegrationSettings` models for AI configuration |
+
+---
+
+## 2. Function Inventory
+
+### 2.1 Core Framework Functions
+
+#### `AICore` (ai_core.py)
+- **`__init__(account=None)`** - Initialize with account context, loads API keys and model from `IntegrationSettings`
+- **`_load_account_settings()`** - Loads OpenAI/Runware API keys and model from `IntegrationSettings` or Django settings
+- **`get_api_key(integration_type='openai')`** - Returns API key for integration type
+- **`get_model(integration_type='openai')`** - Returns model name for integration type
+- **`run_ai_request(prompt, model=None, max_tokens=4000, temperature=0.7, response_format=None, api_key=None, function_name='ai_request', function_id=None, tracker=None)`** - **CENTRAL METHOD** - Handles all OpenAI text generation requests with console logging
+- **`extract_json(response_text)`** - Extracts JSON from response text (handles markdown code blocks)
+- **`generate_image(prompt, provider='openai', model=None, size='1024x1024', n=1, api_key=None, negative_prompt=None, function_name='generate_image')`** - Generates images via OpenAI DALL-E or Runware
+- **`_generate_image_openai(...)`** - Internal method for OpenAI image generation
+- **`_generate_image_runware(...)`** - Internal method for Runware image generation
+- **`calculate_cost(model, input_tokens, output_tokens, model_type='text')`** - Calculates API cost
+- **`call_openai(...)`** - **LEGACY** - Redirects to `run_ai_request()`
+
+#### `AIEngine` (engine.py)
+- **`__init__(celery_task=None, account=None)`** - Initialize with Celery task and account
+- **`execute(fn: BaseAIFunction, payload: dict)`** - **CENTRAL ORCHESTRATOR** - Unified execution pipeline:
+  - Phase 1: INIT (0-10%) - Validation
+  - Phase 2: PREP (10-25%) - Data loading & prompt building
+  - Phase 3: AI_CALL (25-70%) - API call to provider
+  - Phase 4: PARSE (70-85%) - Response parsing
+  - Phase 5: SAVE (85-98%) - Database operations
+  - Phase 6: DONE (98-100%) - Finalization
+- **`_handle_error(error, fn=None, exc_info=False)`** - Centralized error handling
+- **`_log_to_database(fn, payload, parsed, save_result, error=None)`** - Logs to `AITaskLog` model
+- **`_calculate_credits_for_clustering(keyword_count, tokens, cost)`** - Calculates credits for clustering operations
+
+#### `PromptRegistry` (prompts.py)
+- **`get_prompt(function_name, account=None, task=None, context=None)`** - Hierarchical prompt resolution:
+  1. Task-level `prompt_override` (if exists)
+  2. DB prompt for (account, function)
+  3. Default fallback from registry
+- **`_render_prompt(prompt_template, context)`** - Renders template with `[IGNY8_*]` placeholders and `{variable}` format
+- **`get_image_prompt_template(account=None)`** - Gets image prompt template
+- **`get_negative_prompt(account=None)`** - Gets negative prompt
+
+#### `BaseAIFunction` (base.py) - Abstract Interface
+- **`get_name()`** - Returns function name (abstract)
+- **`get_metadata()`** - Returns function metadata (display name, description, phases)
+- **`validate(payload, account=None)`** - Validates input payload (default: checks for 'ids')
+- **`get_max_items()`** - Returns max items limit (optional)
+- **`prepare(payload, account=None)`** - Loads and prepares data (abstract)
+- **`build_prompt(data, account=None)`** - Builds AI prompt (abstract)
+- **`get_model(account=None)`** - Returns model override (optional)
+- **`parse_response(response, step_tracker=None)`** - Parses AI response (abstract)
+- **`save_output(parsed, original_data, account=None, progress_tracker=None, step_tracker=None)`** - Saves results to database (abstract)
+
+### 2.2 AI Function Implementations
+
+#### `AutoClusterFunction` (functions/auto_cluster.py)
+- **`get_name()`** - Returns `'auto_cluster'`
+- **`validate(payload, account=None)`** - Validates keyword IDs exist
+- **`prepare(payload, account=None)`** - Loads keywords with relationships
+- **`build_prompt(data, account=None)`** - Builds clustering prompt using `PromptRegistry`
+- **`parse_response(response, step_tracker=None)`** - Parses JSON cluster data
+- **`save_output(parsed, original_data, account=None, ...)`** - Creates/updates clusters and assigns keywords
+
+#### `GenerateIdeasFunction` (functions/generate_ideas.py)
+- **`get_name()`** - Returns `'generate_ideas'`
+- **`validate(payload, account=None)`** - Validates cluster IDs exist
+- **`prepare(payload, account=None)`** - Loads clusters with keywords
+- **`build_prompt(data, account=None)`** - Builds ideas generation prompt
+- **`parse_response(response, step_tracker=None)`** - Parses JSON ideas data
+- **`save_output(parsed, original_data, account=None, ...)`** - Creates `ContentIdeas` records
+
+#### `GenerateContentFunction` (functions/generate_content.py)
+- **`get_name()`** - Returns `'generate_content'`
+- **`validate(payload, account=None)`** - Validates task IDs exist
+- **`prepare(payload, account=None)`** - Loads tasks with relationships
+- **`build_prompt(data, account=None)`** - Builds content generation prompt
+- **`parse_response(response, step_tracker=None)`** - Parses JSON or plain text content
+- **`save_output(parsed, original_data, account=None, ...)`** - Saves content to `Content` model
+
+#### `GenerateImagesFunction` (functions/generate_images.py)
+- **`get_name()`** - Returns `'generate_images'`
+- **`validate(payload, account=None)`** - Validates task IDs exist
+- **`prepare(payload, account=None)`** - Loads tasks and image settings
+- **`build_prompt(data, account=None)`** - Extracts image prompts from task content (calls AI)
+- **`parse_response(response, step_tracker=None)`** - Returns parsed response (already parsed)
+- **`save_output(parsed, original_data, account=None, ...)`** - Creates `Images` records
+
+### 2.3 Tracking Functions
+
+#### `StepTracker` (tracker.py)
+- **`add_request_step(step_name, status='success', message='', error=None, duration=None)`** - Adds request step
+- **`add_response_step(step_name, status='success', message='', error=None, duration=None)`** - Adds response step
+- **`get_meta()`** - Returns metadata dict with request/response steps
+
+#### `ProgressTracker` (tracker.py)
+- **`update(phase, percentage, message, current=None, total=None, current_item=None, meta=None)`** - Updates Celery task state
+- **`set_phase(phase, percentage, message, meta=None)`** - Sets progress phase
+- **`complete(message='Task complete!', meta=None)`** - Marks task as complete
+- **`error(error_message, meta=None)`** - Marks task as failed
+- **`get_duration()`** - Returns elapsed time in milliseconds
+
+#### `ConsoleStepTracker` (tracker.py)
+- **`init(message='Task started')`** - Logs initialization
+- **`prep(message)`** - Logs preparation phase
+- **`ai_call(message)`** - Logs AI call phase
+- **`parse(message)`** - Logs parsing phase
+- **`save(message)`** - Logs save phase
+- **`done(message='Execution completed')`** - Logs completion
+- **`error(error_type, message, exception=None)`** - Logs error
+- **`retry(attempt, max_attempts, reason='')`** - Logs retry
+- **`timeout(timeout_seconds)`** - Logs timeout
+- **`rate_limit(retry_after)`** - Logs rate limit
+- **`malformed_json(details='')`** - Logs JSON parsing error
+
+#### `CostTracker` (tracker.py)
+- **`record(function_name, cost, tokens, model=None)`** - Records API call cost
+- **`get_total()`** - Returns total cost
+- **`get_total_tokens()`** - Returns total tokens
+- **`get_operations()`** - Returns all operations list
+
+### 2.4 Celery Tasks
+
+#### `run_ai_task` (ai/tasks.py)
+- **`run_ai_task(self, function_name: str, payload: dict, account_id: int = None)`** - **UNIFIED ENTRYPOINT** - Dynamically loads and executes AI functions via `AIEngine`
+
+#### Legacy Tasks (modules/*/tasks.py)
+- **`auto_cluster_keywords_task`** (planner/tasks.py) - **DEPRECATED** - Uses old `AIProcessor`
+- **`auto_generate_content_task`** (writer/tasks.py) - Uses `AIProcessor` directly (not via framework)
+- **`auto_generate_images_task`** (writer/tasks.py) - Uses `AIProcessor` directly
+
+### 2.5 Legacy Functions (ai_processor.py)
+
+#### `AIProcessor` - **LEGACY/DEPRECATED**
+- **`_call_openai(prompt, model=None, max_tokens=4000, temperature=0.7, response_format=None, api_key=None, function_id=None, response_steps=None)`** - Internal OpenAI API caller
+- **`_extract_json_from_response(response_text)`** - JSON extraction (duplicate of `AICore.extract_json()`)
+- **`generate_content(prompt, model=None, max_tokens=4000, temperature=0.7, **kwargs)`** - Generates text content
+- **`extract_image_prompts(content, title, max_images=3, account=None)`** - Extracts image prompts from content
+- **`check_moderation(text, api_key=None)`** - Checks content moderation
+- **`generate_image(prompt, provider='openai', model=None, size='1024x1024', n=1, api_key=None, **kwargs)`** - Generates images
+- **`cluster_keywords(keywords, sector_name=None, account=None, response_steps=None, progress_callback=None, tracker=None, **kwargs)`** - **DEPRECATED** - Clusters keywords (old method)
+- **`generate_ideas(clusters, account=None, **kwargs)`** - Generates ideas (old method)
+- **`get_prompt(prompt_type, account=None)`** - Gets prompt from database (old method)
+- **`estimate_cost(operation, tokens_or_prompt, model=None)`** - Estimates cost (not implemented)
+
+---
+
+## 3. Class Inventory
+
+### 3.1 Core Classes
+
+| Class | File | Purpose | Inheritance |
+|-------|------|---------|-------------|
+| `AICore` | ai_core.py | Centralized AI request handler | - |
+| `AIEngine` | engine.py | Central orchestrator for AI functions | - |
+| `BaseAIFunction` | base.py | Abstract base for all AI functions | ABC |
+| `PromptRegistry` | prompts.py | Centralized prompt management | - |
+| `StepTracker` | tracker.py | Tracks request/response steps | - |
+| `ProgressTracker` | tracker.py | Tracks Celery progress updates | - |
+| `CostTracker` | tracker.py | Tracks API costs and tokens | - |
+| `ConsoleStepTracker` | tracker.py | Console-based step logging | - |
+| `AITaskLog` | models.py | Database model for AI task logging | AccountBaseModel |
+| `AIProcessor` | utils/ai_processor.py | **LEGACY** Unified AI interface | - |
+
+### 3.2 Function Classes
+
+| Class | File | Purpose | Inheritance |
+|-------|------|---------|-------------|
+| `AutoClusterFunction` | functions/auto_cluster.py | Keyword clustering | BaseAIFunction |
+| `GenerateIdeasFunction` | functions/generate_ideas.py | Idea generation | BaseAIFunction |
+| `GenerateContentFunction` | functions/generate_content.py | Content generation | BaseAIFunction |
+| `GenerateImagesFunction` | functions/generate_images.py | Image generation | BaseAIFunction |
+
+### 3.3 Data Classes
+
+| Class | File | Purpose |
+|-------|------|---------|
+| `StepLog` | types.py | Single step in request/response tracking |
+| `ProgressState` | types.py | Progress state for AI tasks |
+| `AITaskResult` | types.py | Result from AI function execution |
+
+---
+
+## 4. Dependency Graph/Table
+
+### 4.1 Import Relationships
+
+```
+ai/__init__.py
+  ├─> registry.py (register_function, get_function, list_functions)
+  ├─> engine.py (AIEngine)
+  ├─> base.py (BaseAIFunction)
+  ├─> ai_core.py (AICore)
+  ├─> validators.py (all validators)
+  ├─> constants.py (all constants)
+  ├─> prompts.py (PromptRegistry, get_prompt)
+  └─> settings.py (MODEL_CONFIG, get_model_config, etc.)
+
+ai/tasks.py
+  ├─> engine.py (AIEngine)
+  └─> registry.py (get_function_instance)
+
+ai/engine.py
+  ├─> base.py (BaseAIFunction)
+  ├─> tracker.py (StepTracker, ProgressTracker, CostTracker, ConsoleStepTracker)
+  ├─> ai_core.py (AICore)
+  └─> settings.py (get_model_config)
+
+ai/ai_core.py
+  ├─> constants.py (MODEL_RATES, IMAGE_MODEL_RATES, etc.)
+  └─> tracker.py (ConsoleStepTracker)
+
+ai/functions/auto_cluster.py
+  ├─> base.py (BaseAIFunction)
+  ├─> ai_core.py (AICore)
+  ├─> prompts.py (PromptRegistry)
+  └─> settings.py (get_model_config)
+
+ai/functions/generate_content.py
+  ├─> base.py (BaseAIFunction)
+  ├─> ai_core.py (AICore)
+  ├─> prompts.py (PromptRegistry)
+  └─> settings.py (get_model_config)
+
+ai/functions/generate_ideas.py
+  ├─> base.py (BaseAIFunction)
+  ├─> ai_core.py (AICore)
+  ├─> prompts.py (PromptRegistry)
+  └─> settings.py (get_model_config)
+
+ai/functions/generate_images.py
+  ├─> base.py (BaseAIFunction)
+  ├─> ai_core.py (AICore)
+  ├─> prompts.py (PromptRegistry)
+  └─> settings.py (get_model_config)
+
+utils/ai_processor.py
+  ├─> modules/system/models.py (IntegrationSettings)
+  └─> modules/system/utils.py (get_prompt_value, get_default_prompt)
+
+modules/planner/tasks.py
+  └─> utils/ai_processor.py (AIProcessor) [DEPRECATED PATH]
+
+modules/writer/tasks.py
+  ├─> utils/ai_processor.py (AIProcessor) [LEGACY PATH]
+  └─> ai/functions/generate_content.py (generate_content_core)
+```
+
+### 4.2 External Dependencies
+
+| Dependency | Used By | Purpose |
+|------------|---------|---------|
+| `django` | All files | Django ORM, models, settings |
+| `celery` | tasks.py, engine.py | Async task execution |
+| `requests` | ai_core.py, ai_processor.py | HTTP requests to OpenAI/Runware APIs |
+| `json` | Multiple files | JSON parsing |
+| `re` | ai_core.py, ai_processor.py, content_normalizer.py | Regex for JSON extraction |
+| `logging` | All files | Logging |
+| `time` | tracker.py, ai_core.py | Timing and duration tracking |
+| `bs4` (BeautifulSoup) | content_normalizer.py | HTML parsing (optional) |
+
+### 4.3 Database Models Dependencies
+
+| Model | Used By | Purpose |
+|-------|---------|---------|
+| `AITaskLog` | engine.py | Unified AI task logging |
+| `IntegrationSettings` | ai_core.py, ai_processor.py, settings.py | API keys and model configuration |
+| `AIPrompt` | prompts.py | Custom prompt templates |
+| `Keywords` | auto_cluster.py, validators.py | Keyword data |
+| `Clusters` | auto_cluster.py, generate_ideas.py | Cluster data |
+| `ContentIdeas` | generate_ideas.py | Content ideas |
+| `Tasks` | generate_content.py, generate_images.py | Writer tasks |
+| `Content` | generate_content.py | Generated content |
+| `Images` | generate_images.py | Generated images |
+
+---
+
+## 5. System Flow Description
+
+### 5.1 New Unified Framework Flow (Recommended Path)
+
+```
+Frontend API Call
+  ↓
+ViewSet Action (e.g., planner/views.py::auto_cluster)
+  ↓
+run_ai_task.delay(function_name='auto_cluster', payload={ids: [...]}, account_id=123)
+  ↓
+Celery Worker: run_ai_task (ai/tasks.py)
+  ├─> Load Account
+  ├─> get_function_instance('auto_cluster') → AutoClusterFunction
+  └─> AIEngine.execute(AutoClusterFunction, payload)
+       ├─> Phase 1: INIT (0-10%)
+       │   └─> fn.validate(payload, account)
+       ├─> Phase 2: PREP (10-25%)
+       │   ├─> fn.prepare(payload, account) → Load keywords
+       │   └─> fn.build_prompt(data, account) → PromptRegistry.get_prompt()
+       ├─> Phase 3: AI_CALL (25-70%)
+       │   ├─> AICore.run_ai_request(prompt, model, ...)
+       │   │   ├─> Load API key from IntegrationSettings
+       │   │   ├─> Validate model
+       │   │   ├─> Build OpenAI request
+       │   │   ├─> Send HTTP request
+       │   │   ├─> Parse response
+       │   │   └─> Calculate cost
+       │   └─> Track cost via CostTracker
+       ├─> Phase 4: PARSE (70-85%)
+       │   └─> fn.parse_response(response_content, step_tracker)
+       ├─> Phase 5: SAVE (85-98%)
+       │   └─> fn.save_output(parsed, original_data, account, ...)
+       │       └─> Database transaction: Create/update clusters
+       └─> Phase 6: DONE (98-100%)
+           ├─> Log to AITaskLog
+           └─> Return result dict
+  ↓
+Celery Task State Update (SUCCESS/FAILURE)
+  ↓
+Frontend Polls Task Status
+  ↓
+Progress Modal Displays Steps
+```
+
+### 5.2 Legacy Content Generation Flow (Still Active)
+
+```
+Frontend API Call
+  ↓
+ViewSet Action (writer/views.py::auto_generate_content)
+  ↓
+auto_generate_content_task.delay(task_ids, account_id)
+  ↓
+Celery Worker: auto_generate_content_task (modules/writer/tasks.py)
+  ├─> Load Tasks from database
+  ├─> For each task:
+  │   ├─> Load prompt template (get_prompt_value)
+  │   ├─> Format prompt with task data
+  │   ├─> AIProcessor.generate_content(prompt)
+  │   │   └─> AIProcessor._call_openai() [DUPLICATE OF AICore.run_ai_request]
+  │   ├─> Parse response (GenerateContentFunction.parse_response)
+  │   └─> Save content (GenerateContentFunction.save_output)
+  └─> Return result
+```
+
+### 5.3 Legacy Clustering Flow (Deprecated)
+
+```
+Frontend API Call
+  ↓
+ViewSet Action (planner/views.py::auto_cluster) [OLD PATH]
+  ↓
+_auto_cluster_keywords_core() (modules/planner/tasks.py)
+  ├─> Load keywords
+  ├─> AIProcessor.cluster_keywords() [DEPRECATED]
+  │   └─> AIProcessor._call_openai() [DUPLICATE]
+  ├─> Parse clusters
+  └─> Save to database
+```
+
+### 5.4 Image Generation Flow
+
+```
+Frontend API Call
+  ↓
+ViewSet Action (writer/views.py::auto_generate_images)
+  ↓
+auto_generate_images_task.delay(task_ids, account_id)
+  ↓
+Celery Worker: auto_generate_images_task (modules/writer/tasks.py)
+  ├─> Load tasks
+  ├─> For each task:
+  │   ├─> Extract image prompts (AIProcessor.extract_image_prompts)
+  │   │   └─> Calls AI to extract prompts from content
+  │   ├─> Generate featured image (AIProcessor.generate_image)
+  │   ├─> Generate desktop images (if enabled)
+  │   └─> Generate mobile images (if enabled)
+  └─> Save Images records
+```
+
+---
+
+## 6. Integration Points
+
+### 6.1 Celery Integration
+
+**Task Registration:**
+- `celery.py` uses `app.autodiscover_tasks()` to auto-discover tasks from all Django apps
+- Tasks are registered via `@shared_task` decorator
+
+**Registered Tasks:**
+1. `ai.tasks.run_ai_task` - Unified entrypoint (NEW)
+2. `planner.tasks.auto_cluster_keywords_task` - **DEPRECATED**
+3. `writer.tasks.auto_generate_content_task` - Legacy (still active)
+4. `writer.tasks.auto_generate_images_task` - Legacy (still active)
+
+**Task State Management:**
+- `ProgressTracker.update()` calls `task.update_state(state='PROGRESS', meta={...})`
+- Progress metadata includes: `phase`, `percentage`, `message`, `request_steps`, `response_steps`
+- Final states: `SUCCESS`, `FAILURE`
+
+### 6.2 Database Integration
+
+**Models:**
+- `AITaskLog` - Unified logging table (used by `AIEngine._log_to_database()`)
+- `IntegrationSettings` - API keys and model configuration (used by `AICore._load_account_settings()`)
+- `AIPrompt` - Custom prompt templates (used by `PromptRegistry.get_prompt()`)
+
+**Data Models:**
+- `Keywords` - Input for clustering
+- `Clusters` - Output of clustering
+- `ContentIdeas` - Output of idea generation
+- `Tasks` - Input for content/image generation
+- `Content` - Output of content generation
+- `Images` - Output of image generation
+
+### 6.3 Frontend API Integration
+
+**Endpoints:**
+1. `POST /v1/planner/keywords/auto_cluster/` → `planner.views.ClusterViewSet.auto_cluster()`
+   - Calls `run_ai_task.delay(function_name='auto_cluster', ...)`
+
+2. `POST /v1/planner/clusters/auto_generate_ideas/` → `planner.views.ClusterViewSet.auto_generate_ideas()`
+   - Calls `run_ai_task.delay(function_name='auto_generate_ideas', ...)`
+
+3. `POST /v1/writer/tasks/auto_generate_content/` → `writer.views.TasksViewSet.auto_generate_content()`
+   - Calls `auto_generate_content_task.delay(...)` [LEGACY PATH]
+
+4. `POST /v1/writer/tasks/auto_generate_images/` → `writer.views.TasksViewSet.auto_generate_images()`
+   - Calls `auto_generate_images_task.delay(...)` [LEGACY PATH]
+
+**Response Format:**
+- Success: `{success: true, task_id: "...", message: "..."}`
+- Error: `{success: false, error: "..."}`
+
+**Progress Tracking:**
+- Frontend polls Celery task status via `GET /v1/system/tasks/{task_id}/status/`
+- Progress modal displays `request_steps` and `response_steps` from task meta
+
+### 6.4 Configuration Integration
+
+**Settings Loading Hierarchy:**
+1. **Account-level** (`IntegrationSettings` model):
+   - API keys: `IntegrationSettings.config['apiKey']`
+   - Model: `IntegrationSettings.config['model']`
+   - Image settings: `IntegrationSettings.config` (for image_generation type)
+
+2. **Django Settings** (fallback):
+   - `OPENAI_API_KEY`
+   - `RUNWARE_API_KEY`
+   - `DEFAULT_AI_MODEL`
+
+3. **Function-level** (`ai/settings.py`):
+   - `MODEL_CONFIG[function_name]` - Default model, max_tokens, temperature per function
+
+**Prompt Loading Hierarchy:**
+1. Task-level override: `task.prompt_override` (if exists)
+2. Account-level: `AIPrompt` model (account, prompt_type)
+3. Default: `PromptRegistry.DEFAULT_PROMPTS[prompt_type]`
+
+### 6.5 Debug Panel Integration
+
+**Console Logging:**
+- `ConsoleStepTracker` logs to stdout/stderr (only if `DEBUG_MODE=True`)
+- Logs include timestamps, phases, messages, errors
+
+**Step Tracking:**
+- `StepTracker` maintains `request_steps` and `response_steps` arrays
+- Steps include: `stepNumber`, `stepName`, `status`, `message`, `duration`, `error`
+- Steps are included in Celery task meta and displayed in progress modal
+
+**Database Logging:**
+- `AITaskLog` records all AI task executions
+- Fields: `task_id`, `function_name`, `phase`, `status`, `cost`, `tokens`, `request_steps`, `response_steps`, `error`, `payload`, `result`
+
+---
+
+## 7. Identified Redundancies or Repetition
+
+### 7.1 Duplicate Constants
+
+**Location 1:** `ai/constants.py`
+- `MODEL_RATES`, `IMAGE_MODEL_RATES`, `VALID_OPENAI_IMAGE_MODELS`, `VALID_SIZES_BY_MODEL`, `DEFAULT_AI_MODEL`, `JSON_MODE_MODELS`
+
+**Location 2:** `utils/ai_processor.py` (lines 18-44)
+- **EXACT DUPLICATE** of all constants from `constants.py`
+
+**Impact:** Constants are defined in two places, risking inconsistency.
+
+### 7.2 Duplicate JSON Extraction Logic
+
+**Location 1:** `ai/ai_core.py::extract_json()` (lines 391-429)
+- Handles markdown code blocks, multiline JSON, direct JSON
+
+**Location 2:** `utils/ai_processor.py::_extract_json_from_response()` (lines 342-449)
+- **MORE COMPREHENSIVE** - handles more edge cases, balanced brace matching
+
+**Impact:** Two implementations with different capabilities. `ai_processor.py` version is more robust.
+
+### 7.3 Duplicate OpenAI API Calling Logic
+
+**Location 1:** `ai/ai_core.py::run_ai_request()` (lines 106-389)
+- Centralized method with console logging via `ConsoleStepTracker`
+- Handles validation, model selection, cost calculation
+- Returns standardized dict format
+
+**Location 2:** `utils/ai_processor.py::_call_openai()` (lines 125-340)
+- **SIMILAR LOGIC** but with `response_steps` parameter instead of `tracker`
+- Less comprehensive error handling
+- Different return format
+
+**Impact:** Two code paths for the same operation. New code should use `AICore.run_ai_request()`.
+
+### 7.4 Duplicate Image Generation Logic
+
+**Location 1:** `ai/ai_core.py::generate_image()` + `_generate_image_openai()` + `_generate_image_runware()` (lines 431-728)
+- Uses `print()` statements for logging (inconsistent with console tracker)
+
+**Location 2:** `utils/ai_processor.py::generate_image()` (lines 667-1043)
+- **MORE COMPREHENSIVE** - extensive logging, better error handling
+- Handles Runware authentication flow
+
+**Impact:** Two implementations. `ai_processor.py` version has better logging.
+
+### 7.5 Duplicate Prompt Loading Logic
+
+**Location 1:** `ai/prompts.py::PromptRegistry.get_prompt()` (lines 280-333)
+- Hierarchical resolution: task override → DB prompt → default
+- Supports `[IGNY8_*]` placeholders and `{variable}` format
+
+**Location 2:** `utils/ai_processor.py::get_prompt()` (lines 1044-1057)
+- Simple database lookup via `modules/system/utils.get_prompt_value()`
+- No hierarchical resolution
+
+**Location 3:** Direct calls in `modules/writer/tasks.py` (lines 343, 959, 964)
+- Uses `get_prompt_value()` and `get_default_prompt()` directly
+
+**Impact:** Three different ways to load prompts. New code should use `PromptRegistry`.
+
+### 7.6 Duplicate Model Configuration Logic
+
+**Location 1:** `ai/settings.py::get_model_config()` (lines 49-97)
+- Reads from `IntegrationSettings` if account provided
+- Falls back to `MODEL_CONFIG` defaults
+
+**Location 2:** `ai/ai_core.py::_load_account_settings()` (lines 46-90)
+- Reads model from `IntegrationSettings` directly
+- Similar logic but embedded in `AICore.__init__()`
+
+**Location 3:** `utils/ai_processor.py::_get_model()` (lines 98-123)
+- Reads model from `IntegrationSettings` directly
+- Similar logic but embedded in `AIProcessor.__init__()`
+
+**Impact:** Model loading logic duplicated in three places.
+
+### 7.7 Duplicate API Key Loading Logic
+
+**Location 1:** `ai/ai_core.py::_load_account_settings()` (lines 46-90)
+- Loads OpenAI and Runware keys from `IntegrationSettings`
+
+**Location 2:** `utils/ai_processor.py::_get_api_key()` (lines 73-96)
+- **EXACT SAME LOGIC** for loading API keys
+
+**Impact:** Identical code in two places.
+
+### 7.8 Repeated Error Handling Patterns
+
+**Pattern:** Multiple files have similar try/except blocks for:
+- API request errors
+- JSON parsing errors
+- Database errors
+- Validation errors
+
+**Impact:** Error handling is not centralized, making it harder to maintain consistent error messages and logging.
+
+### 7.9 Repeated Progress Update Patterns
+
+**Pattern:** Multiple places manually build progress update dicts:
+- `modules/writer/tasks.py` (lines 62-73, 220-231, etc.)
+- `modules/planner/tasks.py` (lines 59-71, 203-215, etc.)
+- `ai/engine.py` (lines 57, 79, 141, etc.)
+
+**Impact:** Progress update format is not standardized, though `ProgressTracker` exists to handle this.
+
+---
+
+## 8. Summary of Potential Consolidation Areas
+
+### 8.1 Constants Consolidation
+
+**Observation:** Model rates, valid models, and configuration constants are duplicated between `ai/constants.py` and `utils/ai_processor.py`.
+
+**Potential Action:** Remove constants from `ai_processor.py` and import from `constants.py`. However, `ai_processor.py` is marked as legacy, so this may not be necessary if it's being phased out.
+
+### 8.2 JSON Extraction Consolidation
+
+**Observation:** Two JSON extraction methods exist with different capabilities. `ai_processor.py::_extract_json_from_response()` is more comprehensive.
+
+**Potential Action:** Enhance `AICore.extract_json()` with logic from `ai_processor.py`, or create a shared utility function.
+
+### 8.3 API Request Consolidation
+
+**Observation:** `AICore.run_ai_request()` and `AIProcessor._call_openai()` perform the same operation with different interfaces.
+
+**Potential Action:** All new code should use `AICore.run_ai_request()`. Legacy code in `ai_processor.py` can remain for backward compatibility but should be marked as deprecated.
+
+### 8.4 Image Generation Consolidation
+
+**Observation:** Two image generation implementations exist. `ai_processor.py` version has better logging.
+
+**Potential Action:** Enhance `AICore.generate_image()` with logging improvements from `ai_processor.py`, or migrate all code to use `AICore.generate_image()`.
+
+### 8.5 Prompt Loading Consolidation
+
+**Observation:** Three different methods exist for loading prompts: `PromptRegistry.get_prompt()`, `AIProcessor.get_prompt()`, and direct `get_prompt_value()` calls.
+
+**Potential Action:** Migrate all code to use `PromptRegistry.get_prompt()` for consistency. Update legacy code paths gradually.
+
+### 8.6 Model/API Key Loading Consolidation
+
+**Observation:** Model and API key loading logic is duplicated in `AICore`, `AIProcessor`, and `settings.py`.
+
+**Potential Action:** Create shared utility functions for loading settings from `IntegrationSettings`, used by all classes.
+
+### 8.7 Error Handling Consolidation
+
+**Observation:** Error handling patterns are repeated across multiple files.
+
+**Potential Action:** Create centralized error handling utilities or enhance `AIEngine._handle_error()` to be more reusable.
+
+### 8.8 Progress Tracking Consolidation
+
+**Observation:** Some code manually builds progress update dicts instead of using `ProgressTracker`.
+
+**Potential Action:** Migrate all progress updates to use `ProgressTracker.update()` for consistency.
+
+### 8.9 Legacy Code Path Elimination
+
+**Observation:** Multiple execution paths exist:
+- New: `run_ai_task` → `AIEngine` → `BaseAIFunction` implementations
+- Legacy: Direct `AIProcessor` calls in `modules/*/tasks.py`
+
+**Potential Action:** Gradually migrate all legacy tasks to use the new framework. Mark legacy code as deprecated.
+
+---
+
+## 9. Assumptions Made
+
+1. **File `wordpress.py`** was not read - assumed to be unrelated to AI processing based on name.
+
+2. **Frontend code** was partially analyzed via search results - full frontend audit not performed.
+
+3. **Database migrations** were not analyzed - assumed to be standard Django migrations.
+
+4. **Test files** were not analyzed - `ai/tests/test_run.py` exists but was not read.
+
+5. **Settings file** (`backend/igny8_core/settings.py`) was not read - assumed to contain standard Django settings.
+
+6. **System module utilities** (`modules/system/utils.py`) were referenced but not fully read - assumed to contain `get_prompt_value()` and `get_default_prompt()` functions.
+
+---
+
+## 10. Appendix
+
+### 10.1 File Line Counts
+
+| Directory | Files | Total Lines |
+|-----------|-------|-------------|
+| `ai/` | 15 | ~2,500 |
+| `ai/functions/` | 5 | ~1,300 |
+| `utils/` (AI-related) | 3 | ~1,800 |
+| `modules/planner/tasks.py` | 1 | 736 |
+| `modules/writer/tasks.py` | 1 | 1,156 |
+| **Total** | **25** | **~7,500** |
+
+### 10.2 Function Count Summary
+
+- **Core Framework Functions:** ~30
+- **AI Function Implementations:** 4 classes × ~6 methods = ~24 methods
+- **Tracking Functions:** ~20
+- **Legacy Functions:** ~15
+- **Celery Tasks:** 4
+- **Total:** ~90 functions/methods
+
+### 10.3 Key Design Patterns
+
+1. **Template Method Pattern:** `BaseAIFunction` defines algorithm skeleton, subclasses implement steps
+2. **Registry Pattern:** `FunctionRegistry` for dynamic function discovery
+3. **Factory Pattern:** `get_function_instance()` creates function instances
+4. **Strategy Pattern:** Different AI functions implement same interface
+5. **Observer Pattern:** `ProgressTracker` updates Celery task state
+6. **Facade Pattern:** `AICore` provides simplified interface to OpenAI/Runware APIs
+
+---
+
+**End of Report**
+