audits

2025-11-10 23:04:12 +05:00
parent 6c9da30b68
commit ecbab4d380
2 changed files with 1018 additions and 0 deletions
--- a/IGNY8_AI_AUDIT_PLAN.md
+++ b/IGNY8_AI_AUDIT_PLAN.md
@@ -0,0 +1,247 @@
 # IGNY8 AI System Audit — Execution Plan
 ## Objective
 Perform a complete structural and functional audit of the IGNY8 AI subsystem exactly as it exists, without any modifications, renaming, or assumptions. Document all findings in a baseline report.
 ## Scope
 ### Primary Directory: `backend/igny8_core/ai/`
 **Core AI Files (15 files):**
 - `__init__.py` - Package initialization and exports
 - `admin.py` - Django admin configuration for AI models
 - `ai_core.py` - Core AI functionality
 - `apps.py` - Django app configuration
 - `base.py` - Base classes or utilities
 - `constants.py` - AI-related constants
 - `engine.py` - AI engine implementation
 - `models.py` - Database models for AI entities
 - `processor.py` - AI processing logic
 - `prompts.py` - Prompt templates and management
 - `registry.py` - Function/component registry
 - `settings.py` - AI-specific settings
 - `tasks.py` - Celery task definitions
 - `tracker.py` - Progress tracking and state management
 - `types.py` - Type definitions and schemas
 - `validators.py` - Validation logic
 **AI Functions Subdirectory (5 files):**
 - `functions/__init__.py` - Function package exports
 - `functions/auto_cluster.py` - Automatic clustering functionality
 - `functions/generate_content.py` - Content generation logic
 - `functions/generate_ideas.py` - Idea generation logic
 - `functions/generate_images.py` - Image generation logic
 ### Related Directories
 **`backend/igny8_core/utils/` (4 files):**
 - `ai_processor.py` - AI processing utilities
 - `content_normalizer.py` - Content normalization utilities
 - `queue_manager.py` - Queue management utilities
 - `wordpress.py` - WordPress integration utilities
 **`backend/igny8_core/modules/` (AI-related files):**
 - `planner/tasks.py` - Planner module Celery tasks
 - `writer/tasks.py` - Writer module Celery tasks
 - `system/models.py` - System models (may contain AI settings)
 - `system/settings_models.py` - Settings models
 - `system/settings_views.py` - Settings views
 - `system/views.py` - System views
 - `system/utils.py` - System utilities
 **Configuration Files:**
 - `backend/igny8_core/celery.py` - Celery configuration and task registration
 - `backend/igny8_core/settings.py` - Django settings (AI configuration loading)
 ## Audit Methodology
 ### Phase 1: File Inventory and Initial Reading
 1. Read all files in `backend/igny8_core/ai/` directory
 2. Read all files in `backend/igny8_core/ai/functions/` directory
 3. Read AI-related files in `backend/igny8_core/utils/`
 4. Read AI-related task files in `backend/igny8_core/modules/`
 5. Read configuration and integration files
 ### Phase 2: Function and Class Analysis
 1. Extract all function definitions with:
   - Function name
   - Parameters and types
   - Return values
   - Docstrings/documentation
   - Decorators (especially Celery tasks)
 2. Extract all class definitions with:
   - Class name
   - Inheritance hierarchy
   - Methods and their purposes
   - Class-level attributes
 3. Identify call sites for each function/class method
 ### Phase 3: Dependency Mapping
 1. Map import relationships:
   - Which files import from which files
   - External dependencies (libraries, Django, Celery)
   - Circular dependencies (if any)
 2. Create dependency graph/table showing:
   - Direct imports
   - Indirect dependencies
   - Shared utilities
 ### Phase 4: System Flow Analysis
 1. Trace request flow:
   - Frontend API endpoints → Views/Serializers
   - Views → Celery tasks
   - Celery tasks → AI functions
   - AI functions → External APIs/Models
   - Results → Database storage
   - Results → Response to frontend
 2. Document:
   - Entry points (API endpoints, admin actions, management commands)
   - Task queue flow (Celery task registration and execution)
   - State management (tracker, progress updates)
   - Error handling paths
   - Logging and debug output
 ### Phase 5: Integration Points Analysis
 1. **Celery Integration:**
   - Task registration in `celery.py`
   - Task decorators and configurations
   - Task routing and queues
   - Async execution patterns
 2. **Database Integration:**
   - Models used by AI subsystem
   - Model relationships
   - Data persistence patterns
   - Query patterns
 3. **Frontend Integration:**
   - API endpoints that trigger AI tasks
   - Serializers for AI data
   - Response formats
   - WebSocket/SSE for progress updates (if any)
 4. **Configuration Integration:**
   - Settings loading (Django settings, environment variables)
   - Model/provider configuration
   - API key management
   - Feature flags or switches
 5. **Debug Panel Integration:**
   - Debug logging mechanisms
   - Progress tracking
   - State inspection tools
 ### Phase 6: Redundancy and Pattern Identification
 1. Identify:
   - Duplicated code blocks
   - Similar functions with slight variations
   - Repeated patterns that could indicate consolidation opportunities
   - Unused or dead code
   - Overlapping responsibilities
 2. Document patterns:
   - Common error handling approaches
   - Repeated validation logic
   - Similar processing pipelines
   - Shared utility patterns
 ### Phase 7: Documentation Compilation
 Create structured document with sections:
 1. **Current File Inventory** - List all files with brief role descriptions
 2. **Function Inventory** - Comprehensive list of all functions with descriptions
 3. **Class Inventory** - All classes and their purposes
 4. **Dependency Graph/Table** - Import relationships and dependencies
 5. **System Flow Description** - End-to-end flow documentation
 6. **Integration Points** - Detailed integration documentation
 7. **Identified Redundancies** - Patterns and duplications found
 8. **Summary of Potential Consolidation Areas** - Observations only (no refactoring proposals)
 ## Execution Rules
 ### Strict Guidelines:
 - ✅ **DO:** Read all code exactly as written
 - ✅ **DO:** Document what exists without modification
 - ✅ **DO:** Label any assumptions explicitly
 - ✅ **DO:** Trace actual code paths, not theoretical ones
 - ✅ **DO:** Include line numbers and file paths for references
 ### Prohibited Actions:
 - ❌ **DON'T:** Rename anything
 - ❌ **DON'T:** Merge or consolidate code
 - ❌ **DON'T:** Propose new architecture
 - ❌ **DON'T:** Suggest simplifications
 - ❌ **DON'T:** Make any code changes
 - ❌ **DON'T:** Create new files (except the audit document)
 - ❌ **DON'T:** Assume functionality without reading code
 ## Deliverable
 **Document Title:** `IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md`
 **Structure:**
 ```markdown
 # IGNY8 AI System Audit — Current Structure & Flow Mapping (Baseline Report)
 ## Executive Summary
 [Brief overview of findings]
 ## 1. Current File Inventory
 [Complete list with descriptions]
 ## 2. Function Inventory
 [All functions documented]
 ## 3. Class Inventory
 [All classes documented]
 ## 4. Dependency Graph/Table
 [Import relationships]
 ## 5. System Flow Description
 [End-to-end flows]
 ## 6. Integration Points
 [Celery, Database, Frontend, Configuration, Debug]
 ## 7. Identified Redundancies or Repetition
 [Patterns found]
 ## 8. Summary of Potential Consolidation Areas
 [Observations only]
 ## 9. Assumptions Made
 [Any assumptions explicitly labeled]
 ## 10. Appendix
 [Additional details, code snippets, etc.]
 ```
 ## Execution Checklist
 - [ ] Phase 1: Read all AI core files
 - [ ] Phase 1: Read all AI function files
 - [ ] Phase 1: Read all utility files
 - [ ] Phase 1: Read all module task files
 - [ ] Phase 1: Read configuration files
 - [ ] Phase 2: Extract and document all functions
 - [ ] Phase 2: Extract and document all classes
 - [ ] Phase 3: Map all import dependencies
 - [ ] Phase 4: Trace system flows
 - [ ] Phase 5: Document integration points
 - [ ] Phase 6: Identify redundancies
 - [ ] Phase 7: Compile final audit document
 ## Estimated File Count
 - **AI Core Files:** 15 files
 - **AI Functions:** 5 files
 - **Utilities:** 4 files
 - **Module Tasks:** 2 files
 - **System Module:** ~5 files
 - **Configuration:** 2 files
 - **Total:** ~33 files to analyze
 ## Notes
 - This is a discovery phase only
 - All findings must be based on actual code
 - No refactoring or improvements will be proposed
 - The goal is to understand the current state completely
--- a/IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md
+++ b/IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md
@@ -0,0 +1,771 @@
 # IGNY8 AI System Audit — Current Structure & Flow Mapping (Baseline Report)
 **Date:** 2024-12-19  
 **Scope:** Complete structural and functional audit of the IGNY8 AI subsystem  
 **Methodology:** Code analysis without modifications or assumptions
 ---
 ## Executive Summary
 The IGNY8 AI subsystem is a comprehensive framework for AI-powered content operations including keyword clustering, idea generation, content generation, and image generation. The system uses a unified framework architecture with a centralized execution engine, but also maintains legacy code paths for backward compatibility.
 **Key Findings:**
 - **20 core AI files** in `backend/igny8_core/ai/`
 - **4 AI function implementations** following BaseAIFunction pattern
 - **Dual execution paths:** New unified framework (`run_ai_task` → `AIEngine`) and legacy paths (direct task calls)
 - **Centralized AI request handling** via `AICore.run_ai_request()`
 - **Comprehensive tracking** via `StepTracker`, `ProgressTracker`, `CostTracker`, and `ConsoleStepTracker`
 - **Multiple prompt management systems:** `PromptRegistry` (new) and direct database queries (legacy)
 ---
 ## 1. Current File Inventory
 ### 1.1 Core AI Framework (`backend/igny8_core/ai/`)
 | File | Lines | Purpose |
 |------|-------|---------|
 | `__init__.py` | 73 | Package exports - exposes main classes and functions |
 | `admin.py` | 60 | Django admin configuration for `AITaskLog` model |
 | `ai_core.py` | 756 | Centralized AI request handler - `AICore` class with `run_ai_request()` and `generate_image()` |
 | `apps.py` | 21 | Django app configuration |
 | `base.py` | 95 | Abstract base class `BaseAIFunction` - defines function interface |
 | `constants.py` | 42 | Model pricing, valid models, configuration constants |
 | `engine.py` | 375 | Central orchestrator `AIEngine` - manages function lifecycle |
 | `models.py` | 52 | Database model `AITaskLog` for unified logging |
 | `processor.py` | 76 | **DEPRECATED** wrapper around `AICore` for backward compatibility |
 | `prompts.py` | 432 | `PromptRegistry` - centralized prompt management with hierarchical resolution |
 | `registry.py` | 97 | Function registry with lazy loading - `register_function()`, `get_function()` |
 | `settings.py` | 117 | Model configurations per function - `MODEL_CONFIG`, `get_model_config()` |
 | `tasks.py` | 131 | Unified Celery task entrypoint `run_ai_task()` - single entry point for all AI functions |
 | `tracker.py` | 348 | Progress tracking utilities - `StepTracker`, `ProgressTracker`, `CostTracker`, `ConsoleStepTracker` |
 | `types.py` | 44 | Type definitions - `StepLog`, `ProgressState`, `AITaskResult` dataclasses |
 | `validators.py` | 187 | Validation functions - `validate_ids()`, `validate_keywords_exist()`, etc. |
 ### 1.2 AI Function Implementations (`backend/igny8_core/ai/functions/`)
 | File | Lines | Purpose |
 |------|-------|---------|
 | `__init__.py` | 18 | Function package exports |
 | `auto_cluster.py` | 330 | `AutoClusterFunction` - Groups keywords into semantic clusters |
 | `generate_content.py` | 388 | `GenerateContentFunction` + `generate_content_core()` - Generates article content |
 | `generate_ideas.py` | 335 | `GenerateIdeasFunction` + `generate_ideas_core()` - Generates content ideas from clusters |
 | `generate_images.py` | 279 | `GenerateImagesFunction` + `generate_images_core()` - Generates images for tasks |
 ### 1.3 Utility Files (`backend/igny8_core/utils/`)
 | File | Lines | Purpose |
 |------|-------|---------|
 | `ai_processor.py` | 1407 | **LEGACY** Unified AI interface - `AIProcessor` class with OpenAI/Runware support. Contains duplicate constants and methods. |
 | `content_normalizer.py` | 273 | Content normalization - converts AI responses to HTML format |
 | `queue_manager.py` | 90 | Queue abstraction (currently placeholder, not fully implemented) |
 | `wordpress.py` | (not read) | WordPress integration utilities |
 ### 1.4 Module Task Files
 | File | Lines | Purpose |
 |------|-------|---------|
 | `modules/planner/tasks.py` | 736 | **DEPRECATED** Legacy clustering task `auto_cluster_keywords_task()` - uses old `AIProcessor` |
 | `modules/writer/tasks.py` | 1156 | Legacy content/image generation tasks - `auto_generate_content_task()`, `auto_generate_images_task()` |
 ### 1.5 Configuration Files
 | File | Purpose |
 |------|---------|
 | `celery.py` | Celery app configuration - auto-discovers tasks from all Django apps |
 | `modules/system/models.py` | `AIPrompt`, `IntegrationSettings` models for AI configuration |
 ---
 ## 2. Function Inventory
 ### 2.1 Core Framework Functions
 #### `AICore` (ai_core.py)
 - **`__init__(account=None)`** - Initialize with account context, loads API keys and model from `IntegrationSettings`
 - **`_load_account_settings()`** - Loads OpenAI/Runware API keys and model from `IntegrationSettings` or Django settings
 - **`get_api_key(integration_type='openai')`** - Returns API key for integration type
 - **`get_model(integration_type='openai')`** - Returns model name for integration type
 - **`run_ai_request(prompt, model=None, max_tokens=4000, temperature=0.7, response_format=None, api_key=None, function_name='ai_request', function_id=None, tracker=None)`** - **CENTRAL METHOD** - Handles all OpenAI text generation requests with console logging
 - **`extract_json(response_text)`** - Extracts JSON from response text (handles markdown code blocks)
 - **`generate_image(prompt, provider='openai', model=None, size='1024x1024', n=1, api_key=None, negative_prompt=None, function_name='generate_image')`** - Generates images via OpenAI DALL-E or Runware
 - **`_generate_image_openai(...)`** - Internal method for OpenAI image generation
 - **`_generate_image_runware(...)`** - Internal method for Runware image generation
 - **`calculate_cost(model, input_tokens, output_tokens, model_type='text')`** - Calculates API cost
 - **`call_openai(...)`** - **LEGACY** - Redirects to `run_ai_request()`
 #### `AIEngine` (engine.py)
 - **`__init__(celery_task=None, account=None)`** - Initialize with Celery task and account
 - **`execute(fn: BaseAIFunction, payload: dict)`** - **CENTRAL ORCHESTRATOR** - Unified execution pipeline:
  - Phase 1: INIT (0-10%) - Validation
  - Phase 2: PREP (10-25%) - Data loading & prompt building
  - Phase 3: AI_CALL (25-70%) - API call to provider
  - Phase 4: PARSE (70-85%) - Response parsing
  - Phase 5: SAVE (85-98%) - Database operations
  - Phase 6: DONE (98-100%) - Finalization
 - **`_handle_error(error, fn=None, exc_info=False)`** - Centralized error handling
 - **`_log_to_database(fn, payload, parsed, save_result, error=None)`** - Logs to `AITaskLog` model
 - **`_calculate_credits_for_clustering(keyword_count, tokens, cost)`** - Calculates credits for clustering operations
 #### `PromptRegistry` (prompts.py)
 - **`get_prompt(function_name, account=None, task=None, context=None)`** - Hierarchical prompt resolution:
  1. Task-level `prompt_override` (if exists)
  2. DB prompt for (account, function)
  3. Default fallback from registry
 - **`_render_prompt(prompt_template, context)`** - Renders template with `[IGNY8_*]` placeholders and `{variable}` format
 - **`get_image_prompt_template(account=None)`** - Gets image prompt template
 - **`get_negative_prompt(account=None)`** - Gets negative prompt
 #### `BaseAIFunction` (base.py) - Abstract Interface
 - **`get_name()`** - Returns function name (abstract)
 - **`get_metadata()`** - Returns function metadata (display name, description, phases)
 - **`validate(payload, account=None)`** - Validates input payload (default: checks for 'ids')
 - **`get_max_items()`** - Returns max items limit (optional)
 - **`prepare(payload, account=None)`** - Loads and prepares data (abstract)
 - **`build_prompt(data, account=None)`** - Builds AI prompt (abstract)
 - **`get_model(account=None)`** - Returns model override (optional)
 - **`parse_response(response, step_tracker=None)`** - Parses AI response (abstract)
 - **`save_output(parsed, original_data, account=None, progress_tracker=None, step_tracker=None)`** - Saves results to database (abstract)
 ### 2.2 AI Function Implementations
 #### `AutoClusterFunction` (functions/auto_cluster.py)
 - **`get_name()`** - Returns `'auto_cluster'`
 - **`validate(payload, account=None)`** - Validates keyword IDs exist
 - **`prepare(payload, account=None)`** - Loads keywords with relationships
 - **`build_prompt(data, account=None)`** - Builds clustering prompt using `PromptRegistry`
 - **`parse_response(response, step_tracker=None)`** - Parses JSON cluster data
 - **`save_output(parsed, original_data, account=None, ...)`** - Creates/updates clusters and assigns keywords
 #### `GenerateIdeasFunction` (functions/generate_ideas.py)
 - **`get_name()`** - Returns `'generate_ideas'`
 - **`validate(payload, account=None)`** - Validates cluster IDs exist
 - **`prepare(payload, account=None)`** - Loads clusters with keywords
 - **`build_prompt(data, account=None)`** - Builds ideas generation prompt
 - **`parse_response(response, step_tracker=None)`** - Parses JSON ideas data
 - **`save_output(parsed, original_data, account=None, ...)`** - Creates `ContentIdeas` records
 #### `GenerateContentFunction` (functions/generate_content.py)
 - **`get_name()`** - Returns `'generate_content'`
 - **`validate(payload, account=None)`** - Validates task IDs exist
 - **`prepare(payload, account=None)`** - Loads tasks with relationships
 - **`build_prompt(data, account=None)`** - Builds content generation prompt
 - **`parse_response(response, step_tracker=None)`** - Parses JSON or plain text content
 - **`save_output(parsed, original_data, account=None, ...)`** - Saves content to `Content` model
 #### `GenerateImagesFunction` (functions/generate_images.py)
 - **`get_name()`** - Returns `'generate_images'`
 - **`validate(payload, account=None)`** - Validates task IDs exist
 - **`prepare(payload, account=None)`** - Loads tasks and image settings
 - **`build_prompt(data, account=None)`** - Extracts image prompts from task content (calls AI)
 - **`parse_response(response, step_tracker=None)`** - Returns parsed response (already parsed)
 - **`save_output(parsed, original_data, account=None, ...)`** - Creates `Images` records
 ### 2.3 Tracking Functions
 #### `StepTracker` (tracker.py)
 - **`add_request_step(step_name, status='success', message='', error=None, duration=None)`** - Adds request step
 - **`add_response_step(step_name, status='success', message='', error=None, duration=None)`** - Adds response step
 - **`get_meta()`** - Returns metadata dict with request/response steps
 #### `ProgressTracker` (tracker.py)
 - **`update(phase, percentage, message, current=None, total=None, current_item=None, meta=None)`** - Updates Celery task state
 - **`set_phase(phase, percentage, message, meta=None)`** - Sets progress phase
 - **`complete(message='Task complete!', meta=None)`** - Marks task as complete
 - **`error(error_message, meta=None)`** - Marks task as failed
 - **`get_duration()`** - Returns elapsed time in milliseconds
 #### `ConsoleStepTracker` (tracker.py)
 - **`init(message='Task started')`** - Logs initialization
 - **`prep(message)`** - Logs preparation phase
 - **`ai_call(message)`** - Logs AI call phase
 - **`parse(message)`** - Logs parsing phase
 - **`save(message)`** - Logs save phase
 - **`done(message='Execution completed')`** - Logs completion
 - **`error(error_type, message, exception=None)`** - Logs error
 - **`retry(attempt, max_attempts, reason='')`** - Logs retry
 - **`timeout(timeout_seconds)`** - Logs timeout
 - **`rate_limit(retry_after)`** - Logs rate limit
 - **`malformed_json(details='')`** - Logs JSON parsing error
 #### `CostTracker` (tracker.py)
 - **`record(function_name, cost, tokens, model=None)`** - Records API call cost
 - **`get_total()`** - Returns total cost
 - **`get_total_tokens()`** - Returns total tokens
 - **`get_operations()`** - Returns all operations list
 ### 2.4 Celery Tasks
 #### `run_ai_task` (ai/tasks.py)
 - **`run_ai_task(self, function_name: str, payload: dict, account_id: int = None)`** - **UNIFIED ENTRYPOINT** - Dynamically loads and executes AI functions via `AIEngine`
 #### Legacy Tasks (modules/*/tasks.py)
 - **`auto_cluster_keywords_task`** (planner/tasks.py) - **DEPRECATED** - Uses old `AIProcessor`
 - **`auto_generate_content_task`** (writer/tasks.py) - Uses `AIProcessor` directly (not via framework)
 - **`auto_generate_images_task`** (writer/tasks.py) - Uses `AIProcessor` directly
 ### 2.5 Legacy Functions (ai_processor.py)
 #### `AIProcessor` - **LEGACY/DEPRECATED**
 - **`_call_openai(prompt, model=None, max_tokens=4000, temperature=0.7, response_format=None, api_key=None, function_id=None, response_steps=None)`** - Internal OpenAI API caller
 - **`_extract_json_from_response(response_text)`** - JSON extraction (duplicate of `AICore.extract_json()`)
 - **`generate_content(prompt, model=None, max_tokens=4000, temperature=0.7, **kwargs)`** - Generates text content
 - **`extract_image_prompts(content, title, max_images=3, account=None)`** - Extracts image prompts from content
 - **`check_moderation(text, api_key=None)`** - Checks content moderation
 - **`generate_image(prompt, provider='openai', model=None, size='1024x1024', n=1, api_key=None, **kwargs)`** - Generates images
 - **`cluster_keywords(keywords, sector_name=None, account=None, response_steps=None, progress_callback=None, tracker=None, **kwargs)`** - **DEPRECATED** - Clusters keywords (old method)
 - **`generate_ideas(clusters, account=None, **kwargs)`** - Generates ideas (old method)
 - **`get_prompt(prompt_type, account=None)`** - Gets prompt from database (old method)
 - **`estimate_cost(operation, tokens_or_prompt, model=None)`** - Estimates cost (not implemented)
 ---
 ## 3. Class Inventory
 ### 3.1 Core Classes
 | Class | File | Purpose | Inheritance |
 |-------|------|---------|-------------|
 | `AICore` | ai_core.py | Centralized AI request handler | - |
 | `AIEngine` | engine.py | Central orchestrator for AI functions | - |
 | `BaseAIFunction` | base.py | Abstract base for all AI functions | ABC |
 | `PromptRegistry` | prompts.py | Centralized prompt management | - |
 | `StepTracker` | tracker.py | Tracks request/response steps | - |
 | `ProgressTracker` | tracker.py | Tracks Celery progress updates | - |
 | `CostTracker` | tracker.py | Tracks API costs and tokens | - |
 | `ConsoleStepTracker` | tracker.py | Console-based step logging | - |
 | `AITaskLog` | models.py | Database model for AI task logging | AccountBaseModel |
 | `AIProcessor` | utils/ai_processor.py | **LEGACY** Unified AI interface | - |
 ### 3.2 Function Classes
 | Class | File | Purpose | Inheritance |
 |-------|------|---------|-------------|
 | `AutoClusterFunction` | functions/auto_cluster.py | Keyword clustering | BaseAIFunction |
 | `GenerateIdeasFunction` | functions/generate_ideas.py | Idea generation | BaseAIFunction |
 | `GenerateContentFunction` | functions/generate_content.py | Content generation | BaseAIFunction |
 | `GenerateImagesFunction` | functions/generate_images.py | Image generation | BaseAIFunction |
 ### 3.3 Data Classes
 | Class | File | Purpose |
 |-------|------|---------|
 | `StepLog` | types.py | Single step in request/response tracking |
 | `ProgressState` | types.py | Progress state for AI tasks |
 | `AITaskResult` | types.py | Result from AI function execution |
 ---
 ## 4. Dependency Graph/Table
 ### 4.1 Import Relationships
 ```
 ai/__init__.py
  ├─> registry.py (register_function, get_function, list_functions)
  ├─> engine.py (AIEngine)
  ├─> base.py (BaseAIFunction)
  ├─> ai_core.py (AICore)
  ├─> validators.py (all validators)
  ├─> constants.py (all constants)
  ├─> prompts.py (PromptRegistry, get_prompt)
  └─> settings.py (MODEL_CONFIG, get_model_config, etc.)
 ai/tasks.py
  ├─> engine.py (AIEngine)
  └─> registry.py (get_function_instance)
 ai/engine.py
  ├─> base.py (BaseAIFunction)
  ├─> tracker.py (StepTracker, ProgressTracker, CostTracker, ConsoleStepTracker)
  ├─> ai_core.py (AICore)
  └─> settings.py (get_model_config)
 ai/ai_core.py
  ├─> constants.py (MODEL_RATES, IMAGE_MODEL_RATES, etc.)
  └─> tracker.py (ConsoleStepTracker)
 ai/functions/auto_cluster.py
  ├─> base.py (BaseAIFunction)
  ├─> ai_core.py (AICore)
  ├─> prompts.py (PromptRegistry)
  └─> settings.py (get_model_config)
 ai/functions/generate_content.py
  ├─> base.py (BaseAIFunction)
  ├─> ai_core.py (AICore)
  ├─> prompts.py (PromptRegistry)
  └─> settings.py (get_model_config)
 ai/functions/generate_ideas.py
  ├─> base.py (BaseAIFunction)
  ├─> ai_core.py (AICore)
  ├─> prompts.py (PromptRegistry)
  └─> settings.py (get_model_config)
 ai/functions/generate_images.py
  ├─> base.py (BaseAIFunction)
  ├─> ai_core.py (AICore)
  ├─> prompts.py (PromptRegistry)
  └─> settings.py (get_model_config)
 utils/ai_processor.py
  ├─> modules/system/models.py (IntegrationSettings)
  └─> modules/system/utils.py (get_prompt_value, get_default_prompt)
 modules/planner/tasks.py
  └─> utils/ai_processor.py (AIProcessor) [DEPRECATED PATH]
 modules/writer/tasks.py
  ├─> utils/ai_processor.py (AIProcessor) [LEGACY PATH]
  └─> ai/functions/generate_content.py (generate_content_core)
 ```
 ### 4.2 External Dependencies
 | Dependency | Used By | Purpose |
 |------------|---------|---------|
 | `django` | All files | Django ORM, models, settings |
 | `celery` | tasks.py, engine.py | Async task execution |
 | `requests` | ai_core.py, ai_processor.py | HTTP requests to OpenAI/Runware APIs |
 | `json` | Multiple files | JSON parsing |
 | `re` | ai_core.py, ai_processor.py, content_normalizer.py | Regex for JSON extraction |
 | `logging` | All files | Logging |
 | `time` | tracker.py, ai_core.py | Timing and duration tracking |
 | `bs4` (BeautifulSoup) | content_normalizer.py | HTML parsing (optional) |
 ### 4.3 Database Models Dependencies
 | Model | Used By | Purpose |
 |-------|---------|---------|
 | `AITaskLog` | engine.py | Unified AI task logging |
 | `IntegrationSettings` | ai_core.py, ai_processor.py, settings.py | API keys and model configuration |
 | `AIPrompt` | prompts.py | Custom prompt templates |
 | `Keywords` | auto_cluster.py, validators.py | Keyword data |
 | `Clusters` | auto_cluster.py, generate_ideas.py | Cluster data |
 | `ContentIdeas` | generate_ideas.py | Content ideas |
 | `Tasks` | generate_content.py, generate_images.py | Writer tasks |
 | `Content` | generate_content.py | Generated content |
 | `Images` | generate_images.py | Generated images |
 ---
 ## 5. System Flow Description
 ### 5.1 New Unified Framework Flow (Recommended Path)
 ```
 Frontend API Call
  ↓
 ViewSet Action (e.g., planner/views.py::auto_cluster)
  ↓
 run_ai_task.delay(function_name='auto_cluster', payload={ids: [...]}, account_id=123)
  ↓
 Celery Worker: run_ai_task (ai/tasks.py)
  ├─> Load Account
  ├─> get_function_instance('auto_cluster') → AutoClusterFunction
  └─> AIEngine.execute(AutoClusterFunction, payload)
       ├─> Phase 1: INIT (0-10%)
       │   └─> fn.validate(payload, account)
       ├─> Phase 2: PREP (10-25%)
       │   ├─> fn.prepare(payload, account) → Load keywords
       │   └─> fn.build_prompt(data, account) → PromptRegistry.get_prompt()
       ├─> Phase 3: AI_CALL (25-70%)
       │   ├─> AICore.run_ai_request(prompt, model, ...)
       │   │   ├─> Load API key from IntegrationSettings
       │   │   ├─> Validate model
       │   │   ├─> Build OpenAI request
       │   │   ├─> Send HTTP request
       │   │   ├─> Parse response
       │   │   └─> Calculate cost
       │   └─> Track cost via CostTracker
       ├─> Phase 4: PARSE (70-85%)
       │   └─> fn.parse_response(response_content, step_tracker)
       ├─> Phase 5: SAVE (85-98%)
       │   └─> fn.save_output(parsed, original_data, account, ...)
       │       └─> Database transaction: Create/update clusters
       └─> Phase 6: DONE (98-100%)
           ├─> Log to AITaskLog
           └─> Return result dict
  ↓
 Celery Task State Update (SUCCESS/FAILURE)
  ↓
 Frontend Polls Task Status
  ↓
 Progress Modal Displays Steps
 ```
 ### 5.2 Legacy Content Generation Flow (Still Active)
 ```
 Frontend API Call
  ↓
 ViewSet Action (writer/views.py::auto_generate_content)
  ↓
 auto_generate_content_task.delay(task_ids, account_id)
  ↓
 Celery Worker: auto_generate_content_task (modules/writer/tasks.py)
  ├─> Load Tasks from database
  ├─> For each task:
  │   ├─> Load prompt template (get_prompt_value)
  │   ├─> Format prompt with task data
  │   ├─> AIProcessor.generate_content(prompt)
  │   │   └─> AIProcessor._call_openai() [DUPLICATE OF AICore.run_ai_request]
  │   ├─> Parse response (GenerateContentFunction.parse_response)
  │   └─> Save content (GenerateContentFunction.save_output)
  └─> Return result
 ```
 ### 5.3 Legacy Clustering Flow (Deprecated)
 ```
 Frontend API Call
  ↓
 ViewSet Action (planner/views.py::auto_cluster) [OLD PATH]
  ↓
 _auto_cluster_keywords_core() (modules/planner/tasks.py)
  ├─> Load keywords
  ├─> AIProcessor.cluster_keywords() [DEPRECATED]
  │   └─> AIProcessor._call_openai() [DUPLICATE]
  ├─> Parse clusters
  └─> Save to database
 ```
 ### 5.4 Image Generation Flow
 ```
 Frontend API Call
  ↓
 ViewSet Action (writer/views.py::auto_generate_images)
  ↓
 auto_generate_images_task.delay(task_ids, account_id)
  ↓
 Celery Worker: auto_generate_images_task (modules/writer/tasks.py)
  ├─> Load tasks
  ├─> For each task:
  │   ├─> Extract image prompts (AIProcessor.extract_image_prompts)
  │   │   └─> Calls AI to extract prompts from content
  │   ├─> Generate featured image (AIProcessor.generate_image)
  │   ├─> Generate desktop images (if enabled)
  │   └─> Generate mobile images (if enabled)
  └─> Save Images records
 ```
 ---
 ## 6. Integration Points
 ### 6.1 Celery Integration
 **Task Registration:**
 - `celery.py` uses `app.autodiscover_tasks()` to auto-discover tasks from all Django apps
 - Tasks are registered via `@shared_task` decorator
 **Registered Tasks:**
 1. `ai.tasks.run_ai_task` - Unified entrypoint (NEW)
 2. `planner.tasks.auto_cluster_keywords_task` - **DEPRECATED**
 3. `writer.tasks.auto_generate_content_task` - Legacy (still active)
 4. `writer.tasks.auto_generate_images_task` - Legacy (still active)
 **Task State Management:**
 - `ProgressTracker.update()` calls `task.update_state(state='PROGRESS', meta={...})`
 - Progress metadata includes: `phase`, `percentage`, `message`, `request_steps`, `response_steps`
 - Final states: `SUCCESS`, `FAILURE`
 ### 6.2 Database Integration
 **Models:**
 - `AITaskLog` - Unified logging table (used by `AIEngine._log_to_database()`)
 - `IntegrationSettings` - API keys and model configuration (used by `AICore._load_account_settings()`)
 - `AIPrompt` - Custom prompt templates (used by `PromptRegistry.get_prompt()`)
 **Data Models:**
 - `Keywords` - Input for clustering
 - `Clusters` - Output of clustering
 - `ContentIdeas` - Output of idea generation
 - `Tasks` - Input for content/image generation
 - `Content` - Output of content generation
 - `Images` - Output of image generation
 ### 6.3 Frontend API Integration
 **Endpoints:**
 1. `POST /v1/planner/keywords/auto_cluster/` → `planner.views.ClusterViewSet.auto_cluster()`
   - Calls `run_ai_task.delay(function_name='auto_cluster', ...)`
 2. `POST /v1/planner/clusters/auto_generate_ideas/` → `planner.views.ClusterViewSet.auto_generate_ideas()`
   - Calls `run_ai_task.delay(function_name='auto_generate_ideas', ...)`
 3. `POST /v1/writer/tasks/auto_generate_content/` → `writer.views.TasksViewSet.auto_generate_content()`
   - Calls `auto_generate_content_task.delay(...)` [LEGACY PATH]
 4. `POST /v1/writer/tasks/auto_generate_images/` → `writer.views.TasksViewSet.auto_generate_images()`
   - Calls `auto_generate_images_task.delay(...)` [LEGACY PATH]
 **Response Format:**
 - Success: `{success: true, task_id: "...", message: "..."}`
 - Error: `{success: false, error: "..."}`
 **Progress Tracking:**
 - Frontend polls Celery task status via `GET /v1/system/tasks/{task_id}/status/`
 - Progress modal displays `request_steps` and `response_steps` from task meta
 ### 6.4 Configuration Integration
 **Settings Loading Hierarchy:**
 1. **Account-level** (`IntegrationSettings` model):
   - API keys: `IntegrationSettings.config['apiKey']`
   - Model: `IntegrationSettings.config['model']`
   - Image settings: `IntegrationSettings.config` (for image_generation type)
 2. **Django Settings** (fallback):
   - `OPENAI_API_KEY`
   - `RUNWARE_API_KEY`
   - `DEFAULT_AI_MODEL`
 3. **Function-level** (`ai/settings.py`):
   - `MODEL_CONFIG[function_name]` - Default model, max_tokens, temperature per function
 **Prompt Loading Hierarchy:**
 1. Task-level override: `task.prompt_override` (if exists)
 2. Account-level: `AIPrompt` model (account, prompt_type)
 3. Default: `PromptRegistry.DEFAULT_PROMPTS[prompt_type]`
 ### 6.5 Debug Panel Integration
 **Console Logging:**
 - `ConsoleStepTracker` logs to stdout/stderr (only if `DEBUG_MODE=True`)
 - Logs include timestamps, phases, messages, errors
 **Step Tracking:**
 - `StepTracker` maintains `request_steps` and `response_steps` arrays
 - Steps include: `stepNumber`, `stepName`, `status`, `message`, `duration`, `error`
 - Steps are included in Celery task meta and displayed in progress modal
 **Database Logging:**
 - `AITaskLog` records all AI task executions
 - Fields: `task_id`, `function_name`, `phase`, `status`, `cost`, `tokens`, `request_steps`, `response_steps`, `error`, `payload`, `result`
 ---
 ## 7. Identified Redundancies or Repetition
 ### 7.1 Duplicate Constants
 **Location 1:** `ai/constants.py`
 - `MODEL_RATES`, `IMAGE_MODEL_RATES`, `VALID_OPENAI_IMAGE_MODELS`, `VALID_SIZES_BY_MODEL`, `DEFAULT_AI_MODEL`, `JSON_MODE_MODELS`
 **Location 2:** `utils/ai_processor.py` (lines 18-44)
 - **EXACT DUPLICATE** of all constants from `constants.py`
 **Impact:** Constants are defined in two places, risking inconsistency.
 ### 7.2 Duplicate JSON Extraction Logic
 **Location 1:** `ai/ai_core.py::extract_json()` (lines 391-429)
 - Handles markdown code blocks, multiline JSON, direct JSON
 **Location 2:** `utils/ai_processor.py::_extract_json_from_response()` (lines 342-449)
 - **MORE COMPREHENSIVE** - handles more edge cases, balanced brace matching
 **Impact:** Two implementations with different capabilities. `ai_processor.py` version is more robust.
 ### 7.3 Duplicate OpenAI API Calling Logic
 **Location 1:** `ai/ai_core.py::run_ai_request()` (lines 106-389)
 - Centralized method with console logging via `ConsoleStepTracker`
 - Handles validation, model selection, cost calculation
 - Returns standardized dict format
 **Location 2:** `utils/ai_processor.py::_call_openai()` (lines 125-340)
 - **SIMILAR LOGIC** but with `response_steps` parameter instead of `tracker`
 - Less comprehensive error handling
 - Different return format
 **Impact:** Two code paths for the same operation. New code should use `AICore.run_ai_request()`.
 ### 7.4 Duplicate Image Generation Logic
 **Location 1:** `ai/ai_core.py::generate_image()` + `_generate_image_openai()` + `_generate_image_runware()` (lines 431-728)
 - Uses `print()` statements for logging (inconsistent with console tracker)
 **Location 2:** `utils/ai_processor.py::generate_image()` (lines 667-1043)
 - **MORE COMPREHENSIVE** - extensive logging, better error handling
 - Handles Runware authentication flow
 **Impact:** Two implementations. `ai_processor.py` version has better logging.
 ### 7.5 Duplicate Prompt Loading Logic
 **Location 1:** `ai/prompts.py::PromptRegistry.get_prompt()` (lines 280-333)
 - Hierarchical resolution: task override → DB prompt → default
 - Supports `[IGNY8_*]` placeholders and `{variable}` format
 **Location 2:** `utils/ai_processor.py::get_prompt()` (lines 1044-1057)
 - Simple database lookup via `modules/system/utils.get_prompt_value()`
 - No hierarchical resolution
 **Location 3:** Direct calls in `modules/writer/tasks.py` (lines 343, 959, 964)
 - Uses `get_prompt_value()` and `get_default_prompt()` directly
 **Impact:** Three different ways to load prompts. New code should use `PromptRegistry`.
 ### 7.6 Duplicate Model Configuration Logic
 **Location 1:** `ai/settings.py::get_model_config()` (lines 49-97)
 - Reads from `IntegrationSettings` if account provided
 - Falls back to `MODEL_CONFIG` defaults
 **Location 2:** `ai/ai_core.py::_load_account_settings()` (lines 46-90)
 - Reads model from `IntegrationSettings` directly
 - Similar logic but embedded in `AICore.__init__()`
 **Location 3:** `utils/ai_processor.py::_get_model()` (lines 98-123)
 - Reads model from `IntegrationSettings` directly
 - Similar logic but embedded in `AIProcessor.__init__()`
 **Impact:** Model loading logic duplicated in three places.
 ### 7.7 Duplicate API Key Loading Logic
 **Location 1:** `ai/ai_core.py::_load_account_settings()` (lines 46-90)
 - Loads OpenAI and Runware keys from `IntegrationSettings`
 **Location 2:** `utils/ai_processor.py::_get_api_key()` (lines 73-96)
 - **EXACT SAME LOGIC** for loading API keys
 **Impact:** Identical code in two places.
 ### 7.8 Repeated Error Handling Patterns
 **Pattern:** Multiple files have similar try/except blocks for:
 - API request errors
 - JSON parsing errors
 - Database errors
 - Validation errors
 **Impact:** Error handling is not centralized, making it harder to maintain consistent error messages and logging.
 ### 7.9 Repeated Progress Update Patterns
 **Pattern:** Multiple places manually build progress update dicts:
 - `modules/writer/tasks.py` (lines 62-73, 220-231, etc.)
 - `modules/planner/tasks.py` (lines 59-71, 203-215, etc.)
 - `ai/engine.py` (lines 57, 79, 141, etc.)
 **Impact:** Progress update format is not standardized, though `ProgressTracker` exists to handle this.
 ---
 ## 8. Summary of Potential Consolidation Areas
 ### 8.1 Constants Consolidation
 **Observation:** Model rates, valid models, and configuration constants are duplicated between `ai/constants.py` and `utils/ai_processor.py`.
 **Potential Action:** Remove constants from `ai_processor.py` and import from `constants.py`. However, `ai_processor.py` is marked as legacy, so this may not be necessary if it's being phased out.
 ### 8.2 JSON Extraction Consolidation
 **Observation:** Two JSON extraction methods exist with different capabilities. `ai_processor.py::_extract_json_from_response()` is more comprehensive.
 **Potential Action:** Enhance `AICore.extract_json()` with logic from `ai_processor.py`, or create a shared utility function.
 ### 8.3 API Request Consolidation
 **Observation:** `AICore.run_ai_request()` and `AIProcessor._call_openai()` perform the same operation with different interfaces.
 **Potential Action:** All new code should use `AICore.run_ai_request()`. Legacy code in `ai_processor.py` can remain for backward compatibility but should be marked as deprecated.
 ### 8.4 Image Generation Consolidation
 **Observation:** Two image generation implementations exist. `ai_processor.py` version has better logging.
 **Potential Action:** Enhance `AICore.generate_image()` with logging improvements from `ai_processor.py`, or migrate all code to use `AICore.generate_image()`.
 ### 8.5 Prompt Loading Consolidation
 **Observation:** Three different methods exist for loading prompts: `PromptRegistry.get_prompt()`, `AIProcessor.get_prompt()`, and direct `get_prompt_value()` calls.
 **Potential Action:** Migrate all code to use `PromptRegistry.get_prompt()` for consistency. Update legacy code paths gradually.
 ### 8.6 Model/API Key Loading Consolidation
 **Observation:** Model and API key loading logic is duplicated in `AICore`, `AIProcessor`, and `settings.py`.
 **Potential Action:** Create shared utility functions for loading settings from `IntegrationSettings`, used by all classes.
 ### 8.7 Error Handling Consolidation
 **Observation:** Error handling patterns are repeated across multiple files.
 **Potential Action:** Create centralized error handling utilities or enhance `AIEngine._handle_error()` to be more reusable.
 ### 8.8 Progress Tracking Consolidation
 **Observation:** Some code manually builds progress update dicts instead of using `ProgressTracker`.
 **Potential Action:** Migrate all progress updates to use `ProgressTracker.update()` for consistency.
 ### 8.9 Legacy Code Path Elimination
 **Observation:** Multiple execution paths exist:
 - New: `run_ai_task` → `AIEngine` → `BaseAIFunction` implementations
 - Legacy: Direct `AIProcessor` calls in `modules/*/tasks.py`
 **Potential Action:** Gradually migrate all legacy tasks to use the new framework. Mark legacy code as deprecated.
 ---
 ## 9. Assumptions Made
 1. **File `wordpress.py`** was not read - assumed to be unrelated to AI processing based on name.
 2. **Frontend code** was partially analyzed via search results - full frontend audit not performed.
 3. **Database migrations** were not analyzed - assumed to be standard Django migrations.
 4. **Test files** were not analyzed - `ai/tests/test_run.py` exists but was not read.
 5. **Settings file** (`backend/igny8_core/settings.py`) was not read - assumed to contain standard Django settings.
 6. **System module utilities** (`modules/system/utils.py`) were referenced but not fully read - assumed to contain `get_prompt_value()` and `get_default_prompt()` functions.
 ---
 ## 10. Appendix
 ### 10.1 File Line Counts
 | Directory | Files | Total Lines |
 |-----------|-------|-------------|
 | `ai/` | 15 | ~2,500 |
 | `ai/functions/` | 5 | ~1,300 |
 | `utils/` (AI-related) | 3 | ~1,800 |
 | `modules/planner/tasks.py` | 1 | 736 |
 | `modules/writer/tasks.py` | 1 | 1,156 |
 | **Total** | **25** | **~7,500** |
 ### 10.2 Function Count Summary
 - **Core Framework Functions:** ~30
 - **AI Function Implementations:** 4 classes × ~6 methods = ~24 methods
 - **Tracking Functions:** ~20
 - **Legacy Functions:** ~15
 - **Celery Tasks:** 4
 - **Total:** ~90 functions/methods
 ### 10.3 Key Design Patterns
 1. **Template Method Pattern:** `BaseAIFunction` defines algorithm skeleton, subclasses implement steps
 2. **Registry Pattern:** `FunctionRegistry` for dynamic function discovery
 3. **Factory Pattern:** `get_function_instance()` creates function instances
 4. **Strategy Pattern:** Different AI functions implement same interface
 5. **Observer Pattern:** `ProgressTracker` updates Celery task state
 6. **Facade Pattern:** `AICore` provides simplified interface to OpenAI/Runware APIs
 ---
 **End of Report**