From ecbab4d380e10c8d846ae2353f2920b35c4a113e Mon Sep 17 00:00:00 2001 From: Desktop Date: Mon, 10 Nov 2025 23:04:12 +0500 Subject: [PATCH] audits --- IGNY8_AI_AUDIT_PLAN.md | 247 ++++++++ IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md | 771 +++++++++++++++++++++++ 2 files changed, 1018 insertions(+) create mode 100644 IGNY8_AI_AUDIT_PLAN.md create mode 100644 IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md diff --git a/IGNY8_AI_AUDIT_PLAN.md b/IGNY8_AI_AUDIT_PLAN.md new file mode 100644 index 00000000..84edefe2 --- /dev/null +++ b/IGNY8_AI_AUDIT_PLAN.md @@ -0,0 +1,247 @@ +# IGNY8 AI System Audit — Execution Plan + +## Objective +Perform a complete structural and functional audit of the IGNY8 AI subsystem exactly as it exists, without any modifications, renaming, or assumptions. Document all findings in a baseline report. + +## Scope + +### Primary Directory: `backend/igny8_core/ai/` +**Core AI Files (15 files):** +- `__init__.py` - Package initialization and exports +- `admin.py` - Django admin configuration for AI models +- `ai_core.py` - Core AI functionality +- `apps.py` - Django app configuration +- `base.py` - Base classes or utilities +- `constants.py` - AI-related constants +- `engine.py` - AI engine implementation +- `models.py` - Database models for AI entities +- `processor.py` - AI processing logic +- `prompts.py` - Prompt templates and management +- `registry.py` - Function/component registry +- `settings.py` - AI-specific settings +- `tasks.py` - Celery task definitions +- `tracker.py` - Progress tracking and state management +- `types.py` - Type definitions and schemas +- `validators.py` - Validation logic + +**AI Functions Subdirectory (5 files):** +- `functions/__init__.py` - Function package exports +- `functions/auto_cluster.py` - Automatic clustering functionality +- `functions/generate_content.py` - Content generation logic +- `functions/generate_ideas.py` - Idea generation logic +- `functions/generate_images.py` - Image generation logic + +### Related Directories + +**`backend/igny8_core/utils/` (4 files):** +- `ai_processor.py` - AI processing utilities +- `content_normalizer.py` - Content normalization utilities +- `queue_manager.py` - Queue management utilities +- `wordpress.py` - WordPress integration utilities + +**`backend/igny8_core/modules/` (AI-related files):** +- `planner/tasks.py` - Planner module Celery tasks +- `writer/tasks.py` - Writer module Celery tasks +- `system/models.py` - System models (may contain AI settings) +- `system/settings_models.py` - Settings models +- `system/settings_views.py` - Settings views +- `system/views.py` - System views +- `system/utils.py` - System utilities + +**Configuration Files:** +- `backend/igny8_core/celery.py` - Celery configuration and task registration +- `backend/igny8_core/settings.py` - Django settings (AI configuration loading) + +## Audit Methodology + +### Phase 1: File Inventory and Initial Reading +1. Read all files in `backend/igny8_core/ai/` directory +2. Read all files in `backend/igny8_core/ai/functions/` directory +3. Read AI-related files in `backend/igny8_core/utils/` +4. Read AI-related task files in `backend/igny8_core/modules/` +5. Read configuration and integration files + +### Phase 2: Function and Class Analysis +1. Extract all function definitions with: + - Function name + - Parameters and types + - Return values + - Docstrings/documentation + - Decorators (especially Celery tasks) +2. Extract all class definitions with: + - Class name + - Inheritance hierarchy + - Methods and their purposes + - Class-level attributes +3. Identify call sites for each function/class method + +### Phase 3: Dependency Mapping +1. Map import relationships: + - Which files import from which files + - External dependencies (libraries, Django, Celery) + - Circular dependencies (if any) +2. Create dependency graph/table showing: + - Direct imports + - Indirect dependencies + - Shared utilities + +### Phase 4: System Flow Analysis +1. Trace request flow: + - Frontend API endpoints → Views/Serializers + - Views → Celery tasks + - Celery tasks → AI functions + - AI functions → External APIs/Models + - Results → Database storage + - Results → Response to frontend +2. Document: + - Entry points (API endpoints, admin actions, management commands) + - Task queue flow (Celery task registration and execution) + - State management (tracker, progress updates) + - Error handling paths + - Logging and debug output + +### Phase 5: Integration Points Analysis +1. **Celery Integration:** + - Task registration in `celery.py` + - Task decorators and configurations + - Task routing and queues + - Async execution patterns + +2. **Database Integration:** + - Models used by AI subsystem + - Model relationships + - Data persistence patterns + - Query patterns + +3. **Frontend Integration:** + - API endpoints that trigger AI tasks + - Serializers for AI data + - Response formats + - WebSocket/SSE for progress updates (if any) + +4. **Configuration Integration:** + - Settings loading (Django settings, environment variables) + - Model/provider configuration + - API key management + - Feature flags or switches + +5. **Debug Panel Integration:** + - Debug logging mechanisms + - Progress tracking + - State inspection tools + +### Phase 6: Redundancy and Pattern Identification +1. Identify: + - Duplicated code blocks + - Similar functions with slight variations + - Repeated patterns that could indicate consolidation opportunities + - Unused or dead code + - Overlapping responsibilities + +2. Document patterns: + - Common error handling approaches + - Repeated validation logic + - Similar processing pipelines + - Shared utility patterns + +### Phase 7: Documentation Compilation +Create structured document with sections: +1. **Current File Inventory** - List all files with brief role descriptions +2. **Function Inventory** - Comprehensive list of all functions with descriptions +3. **Class Inventory** - All classes and their purposes +4. **Dependency Graph/Table** - Import relationships and dependencies +5. **System Flow Description** - End-to-end flow documentation +6. **Integration Points** - Detailed integration documentation +7. **Identified Redundancies** - Patterns and duplications found +8. **Summary of Potential Consolidation Areas** - Observations only (no refactoring proposals) + +## Execution Rules + +### Strict Guidelines: +- ✅ **DO:** Read all code exactly as written +- ✅ **DO:** Document what exists without modification +- ✅ **DO:** Label any assumptions explicitly +- ✅ **DO:** Trace actual code paths, not theoretical ones +- ✅ **DO:** Include line numbers and file paths for references + +### Prohibited Actions: +- ❌ **DON'T:** Rename anything +- ❌ **DON'T:** Merge or consolidate code +- ❌ **DON'T:** Propose new architecture +- ❌ **DON'T:** Suggest simplifications +- ❌ **DON'T:** Make any code changes +- ❌ **DON'T:** Create new files (except the audit document) +- ❌ **DON'T:** Assume functionality without reading code + +## Deliverable + +**Document Title:** `IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md` + +**Structure:** +```markdown +# IGNY8 AI System Audit — Current Structure & Flow Mapping (Baseline Report) + +## Executive Summary +[Brief overview of findings] + +## 1. Current File Inventory +[Complete list with descriptions] + +## 2. Function Inventory +[All functions documented] + +## 3. Class Inventory +[All classes documented] + +## 4. Dependency Graph/Table +[Import relationships] + +## 5. System Flow Description +[End-to-end flows] + +## 6. Integration Points +[Celery, Database, Frontend, Configuration, Debug] + +## 7. Identified Redundancies or Repetition +[Patterns found] + +## 8. Summary of Potential Consolidation Areas +[Observations only] + +## 9. Assumptions Made +[Any assumptions explicitly labeled] + +## 10. Appendix +[Additional details, code snippets, etc.] +``` + +## Execution Checklist + +- [ ] Phase 1: Read all AI core files +- [ ] Phase 1: Read all AI function files +- [ ] Phase 1: Read all utility files +- [ ] Phase 1: Read all module task files +- [ ] Phase 1: Read configuration files +- [ ] Phase 2: Extract and document all functions +- [ ] Phase 2: Extract and document all classes +- [ ] Phase 3: Map all import dependencies +- [ ] Phase 4: Trace system flows +- [ ] Phase 5: Document integration points +- [ ] Phase 6: Identify redundancies +- [ ] Phase 7: Compile final audit document + +## Estimated File Count +- **AI Core Files:** 15 files +- **AI Functions:** 5 files +- **Utilities:** 4 files +- **Module Tasks:** 2 files +- **System Module:** ~5 files +- **Configuration:** 2 files +- **Total:** ~33 files to analyze + +## Notes +- This is a discovery phase only +- All findings must be based on actual code +- No refactoring or improvements will be proposed +- The goal is to understand the current state completely + diff --git a/IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md b/IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md new file mode 100644 index 00000000..6bdbcf58 --- /dev/null +++ b/IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md @@ -0,0 +1,771 @@ +# IGNY8 AI System Audit — Current Structure & Flow Mapping (Baseline Report) + +**Date:** 2024-12-19 +**Scope:** Complete structural and functional audit of the IGNY8 AI subsystem +**Methodology:** Code analysis without modifications or assumptions + +--- + +## Executive Summary + +The IGNY8 AI subsystem is a comprehensive framework for AI-powered content operations including keyword clustering, idea generation, content generation, and image generation. The system uses a unified framework architecture with a centralized execution engine, but also maintains legacy code paths for backward compatibility. + +**Key Findings:** +- **20 core AI files** in `backend/igny8_core/ai/` +- **4 AI function implementations** following BaseAIFunction pattern +- **Dual execution paths:** New unified framework (`run_ai_task` → `AIEngine`) and legacy paths (direct task calls) +- **Centralized AI request handling** via `AICore.run_ai_request()` +- **Comprehensive tracking** via `StepTracker`, `ProgressTracker`, `CostTracker`, and `ConsoleStepTracker` +- **Multiple prompt management systems:** `PromptRegistry` (new) and direct database queries (legacy) + +--- + +## 1. Current File Inventory + +### 1.1 Core AI Framework (`backend/igny8_core/ai/`) + +| File | Lines | Purpose | +|------|-------|---------| +| `__init__.py` | 73 | Package exports - exposes main classes and functions | +| `admin.py` | 60 | Django admin configuration for `AITaskLog` model | +| `ai_core.py` | 756 | Centralized AI request handler - `AICore` class with `run_ai_request()` and `generate_image()` | +| `apps.py` | 21 | Django app configuration | +| `base.py` | 95 | Abstract base class `BaseAIFunction` - defines function interface | +| `constants.py` | 42 | Model pricing, valid models, configuration constants | +| `engine.py` | 375 | Central orchestrator `AIEngine` - manages function lifecycle | +| `models.py` | 52 | Database model `AITaskLog` for unified logging | +| `processor.py` | 76 | **DEPRECATED** wrapper around `AICore` for backward compatibility | +| `prompts.py` | 432 | `PromptRegistry` - centralized prompt management with hierarchical resolution | +| `registry.py` | 97 | Function registry with lazy loading - `register_function()`, `get_function()` | +| `settings.py` | 117 | Model configurations per function - `MODEL_CONFIG`, `get_model_config()` | +| `tasks.py` | 131 | Unified Celery task entrypoint `run_ai_task()` - single entry point for all AI functions | +| `tracker.py` | 348 | Progress tracking utilities - `StepTracker`, `ProgressTracker`, `CostTracker`, `ConsoleStepTracker` | +| `types.py` | 44 | Type definitions - `StepLog`, `ProgressState`, `AITaskResult` dataclasses | +| `validators.py` | 187 | Validation functions - `validate_ids()`, `validate_keywords_exist()`, etc. | + +### 1.2 AI Function Implementations (`backend/igny8_core/ai/functions/`) + +| File | Lines | Purpose | +|------|-------|---------| +| `__init__.py` | 18 | Function package exports | +| `auto_cluster.py` | 330 | `AutoClusterFunction` - Groups keywords into semantic clusters | +| `generate_content.py` | 388 | `GenerateContentFunction` + `generate_content_core()` - Generates article content | +| `generate_ideas.py` | 335 | `GenerateIdeasFunction` + `generate_ideas_core()` - Generates content ideas from clusters | +| `generate_images.py` | 279 | `GenerateImagesFunction` + `generate_images_core()` - Generates images for tasks | + +### 1.3 Utility Files (`backend/igny8_core/utils/`) + +| File | Lines | Purpose | +|------|-------|---------| +| `ai_processor.py` | 1407 | **LEGACY** Unified AI interface - `AIProcessor` class with OpenAI/Runware support. Contains duplicate constants and methods. | +| `content_normalizer.py` | 273 | Content normalization - converts AI responses to HTML format | +| `queue_manager.py` | 90 | Queue abstraction (currently placeholder, not fully implemented) | +| `wordpress.py` | (not read) | WordPress integration utilities | + +### 1.4 Module Task Files + +| File | Lines | Purpose | +|------|-------|---------| +| `modules/planner/tasks.py` | 736 | **DEPRECATED** Legacy clustering task `auto_cluster_keywords_task()` - uses old `AIProcessor` | +| `modules/writer/tasks.py` | 1156 | Legacy content/image generation tasks - `auto_generate_content_task()`, `auto_generate_images_task()` | + +### 1.5 Configuration Files + +| File | Purpose | +|------|---------| +| `celery.py` | Celery app configuration - auto-discovers tasks from all Django apps | +| `modules/system/models.py` | `AIPrompt`, `IntegrationSettings` models for AI configuration | + +--- + +## 2. Function Inventory + +### 2.1 Core Framework Functions + +#### `AICore` (ai_core.py) +- **`__init__(account=None)`** - Initialize with account context, loads API keys and model from `IntegrationSettings` +- **`_load_account_settings()`** - Loads OpenAI/Runware API keys and model from `IntegrationSettings` or Django settings +- **`get_api_key(integration_type='openai')`** - Returns API key for integration type +- **`get_model(integration_type='openai')`** - Returns model name for integration type +- **`run_ai_request(prompt, model=None, max_tokens=4000, temperature=0.7, response_format=None, api_key=None, function_name='ai_request', function_id=None, tracker=None)`** - **CENTRAL METHOD** - Handles all OpenAI text generation requests with console logging +- **`extract_json(response_text)`** - Extracts JSON from response text (handles markdown code blocks) +- **`generate_image(prompt, provider='openai', model=None, size='1024x1024', n=1, api_key=None, negative_prompt=None, function_name='generate_image')`** - Generates images via OpenAI DALL-E or Runware +- **`_generate_image_openai(...)`** - Internal method for OpenAI image generation +- **`_generate_image_runware(...)`** - Internal method for Runware image generation +- **`calculate_cost(model, input_tokens, output_tokens, model_type='text')`** - Calculates API cost +- **`call_openai(...)`** - **LEGACY** - Redirects to `run_ai_request()` + +#### `AIEngine` (engine.py) +- **`__init__(celery_task=None, account=None)`** - Initialize with Celery task and account +- **`execute(fn: BaseAIFunction, payload: dict)`** - **CENTRAL ORCHESTRATOR** - Unified execution pipeline: + - Phase 1: INIT (0-10%) - Validation + - Phase 2: PREP (10-25%) - Data loading & prompt building + - Phase 3: AI_CALL (25-70%) - API call to provider + - Phase 4: PARSE (70-85%) - Response parsing + - Phase 5: SAVE (85-98%) - Database operations + - Phase 6: DONE (98-100%) - Finalization +- **`_handle_error(error, fn=None, exc_info=False)`** - Centralized error handling +- **`_log_to_database(fn, payload, parsed, save_result, error=None)`** - Logs to `AITaskLog` model +- **`_calculate_credits_for_clustering(keyword_count, tokens, cost)`** - Calculates credits for clustering operations + +#### `PromptRegistry` (prompts.py) +- **`get_prompt(function_name, account=None, task=None, context=None)`** - Hierarchical prompt resolution: + 1. Task-level `prompt_override` (if exists) + 2. DB prompt for (account, function) + 3. Default fallback from registry +- **`_render_prompt(prompt_template, context)`** - Renders template with `[IGNY8_*]` placeholders and `{variable}` format +- **`get_image_prompt_template(account=None)`** - Gets image prompt template +- **`get_negative_prompt(account=None)`** - Gets negative prompt + +#### `BaseAIFunction` (base.py) - Abstract Interface +- **`get_name()`** - Returns function name (abstract) +- **`get_metadata()`** - Returns function metadata (display name, description, phases) +- **`validate(payload, account=None)`** - Validates input payload (default: checks for 'ids') +- **`get_max_items()`** - Returns max items limit (optional) +- **`prepare(payload, account=None)`** - Loads and prepares data (abstract) +- **`build_prompt(data, account=None)`** - Builds AI prompt (abstract) +- **`get_model(account=None)`** - Returns model override (optional) +- **`parse_response(response, step_tracker=None)`** - Parses AI response (abstract) +- **`save_output(parsed, original_data, account=None, progress_tracker=None, step_tracker=None)`** - Saves results to database (abstract) + +### 2.2 AI Function Implementations + +#### `AutoClusterFunction` (functions/auto_cluster.py) +- **`get_name()`** - Returns `'auto_cluster'` +- **`validate(payload, account=None)`** - Validates keyword IDs exist +- **`prepare(payload, account=None)`** - Loads keywords with relationships +- **`build_prompt(data, account=None)`** - Builds clustering prompt using `PromptRegistry` +- **`parse_response(response, step_tracker=None)`** - Parses JSON cluster data +- **`save_output(parsed, original_data, account=None, ...)`** - Creates/updates clusters and assigns keywords + +#### `GenerateIdeasFunction` (functions/generate_ideas.py) +- **`get_name()`** - Returns `'generate_ideas'` +- **`validate(payload, account=None)`** - Validates cluster IDs exist +- **`prepare(payload, account=None)`** - Loads clusters with keywords +- **`build_prompt(data, account=None)`** - Builds ideas generation prompt +- **`parse_response(response, step_tracker=None)`** - Parses JSON ideas data +- **`save_output(parsed, original_data, account=None, ...)`** - Creates `ContentIdeas` records + +#### `GenerateContentFunction` (functions/generate_content.py) +- **`get_name()`** - Returns `'generate_content'` +- **`validate(payload, account=None)`** - Validates task IDs exist +- **`prepare(payload, account=None)`** - Loads tasks with relationships +- **`build_prompt(data, account=None)`** - Builds content generation prompt +- **`parse_response(response, step_tracker=None)`** - Parses JSON or plain text content +- **`save_output(parsed, original_data, account=None, ...)`** - Saves content to `Content` model + +#### `GenerateImagesFunction` (functions/generate_images.py) +- **`get_name()`** - Returns `'generate_images'` +- **`validate(payload, account=None)`** - Validates task IDs exist +- **`prepare(payload, account=None)`** - Loads tasks and image settings +- **`build_prompt(data, account=None)`** - Extracts image prompts from task content (calls AI) +- **`parse_response(response, step_tracker=None)`** - Returns parsed response (already parsed) +- **`save_output(parsed, original_data, account=None, ...)`** - Creates `Images` records + +### 2.3 Tracking Functions + +#### `StepTracker` (tracker.py) +- **`add_request_step(step_name, status='success', message='', error=None, duration=None)`** - Adds request step +- **`add_response_step(step_name, status='success', message='', error=None, duration=None)`** - Adds response step +- **`get_meta()`** - Returns metadata dict with request/response steps + +#### `ProgressTracker` (tracker.py) +- **`update(phase, percentage, message, current=None, total=None, current_item=None, meta=None)`** - Updates Celery task state +- **`set_phase(phase, percentage, message, meta=None)`** - Sets progress phase +- **`complete(message='Task complete!', meta=None)`** - Marks task as complete +- **`error(error_message, meta=None)`** - Marks task as failed +- **`get_duration()`** - Returns elapsed time in milliseconds + +#### `ConsoleStepTracker` (tracker.py) +- **`init(message='Task started')`** - Logs initialization +- **`prep(message)`** - Logs preparation phase +- **`ai_call(message)`** - Logs AI call phase +- **`parse(message)`** - Logs parsing phase +- **`save(message)`** - Logs save phase +- **`done(message='Execution completed')`** - Logs completion +- **`error(error_type, message, exception=None)`** - Logs error +- **`retry(attempt, max_attempts, reason='')`** - Logs retry +- **`timeout(timeout_seconds)`** - Logs timeout +- **`rate_limit(retry_after)`** - Logs rate limit +- **`malformed_json(details='')`** - Logs JSON parsing error + +#### `CostTracker` (tracker.py) +- **`record(function_name, cost, tokens, model=None)`** - Records API call cost +- **`get_total()`** - Returns total cost +- **`get_total_tokens()`** - Returns total tokens +- **`get_operations()`** - Returns all operations list + +### 2.4 Celery Tasks + +#### `run_ai_task` (ai/tasks.py) +- **`run_ai_task(self, function_name: str, payload: dict, account_id: int = None)`** - **UNIFIED ENTRYPOINT** - Dynamically loads and executes AI functions via `AIEngine` + +#### Legacy Tasks (modules/*/tasks.py) +- **`auto_cluster_keywords_task`** (planner/tasks.py) - **DEPRECATED** - Uses old `AIProcessor` +- **`auto_generate_content_task`** (writer/tasks.py) - Uses `AIProcessor` directly (not via framework) +- **`auto_generate_images_task`** (writer/tasks.py) - Uses `AIProcessor` directly + +### 2.5 Legacy Functions (ai_processor.py) + +#### `AIProcessor` - **LEGACY/DEPRECATED** +- **`_call_openai(prompt, model=None, max_tokens=4000, temperature=0.7, response_format=None, api_key=None, function_id=None, response_steps=None)`** - Internal OpenAI API caller +- **`_extract_json_from_response(response_text)`** - JSON extraction (duplicate of `AICore.extract_json()`) +- **`generate_content(prompt, model=None, max_tokens=4000, temperature=0.7, **kwargs)`** - Generates text content +- **`extract_image_prompts(content, title, max_images=3, account=None)`** - Extracts image prompts from content +- **`check_moderation(text, api_key=None)`** - Checks content moderation +- **`generate_image(prompt, provider='openai', model=None, size='1024x1024', n=1, api_key=None, **kwargs)`** - Generates images +- **`cluster_keywords(keywords, sector_name=None, account=None, response_steps=None, progress_callback=None, tracker=None, **kwargs)`** - **DEPRECATED** - Clusters keywords (old method) +- **`generate_ideas(clusters, account=None, **kwargs)`** - Generates ideas (old method) +- **`get_prompt(prompt_type, account=None)`** - Gets prompt from database (old method) +- **`estimate_cost(operation, tokens_or_prompt, model=None)`** - Estimates cost (not implemented) + +--- + +## 3. Class Inventory + +### 3.1 Core Classes + +| Class | File | Purpose | Inheritance | +|-------|------|---------|-------------| +| `AICore` | ai_core.py | Centralized AI request handler | - | +| `AIEngine` | engine.py | Central orchestrator for AI functions | - | +| `BaseAIFunction` | base.py | Abstract base for all AI functions | ABC | +| `PromptRegistry` | prompts.py | Centralized prompt management | - | +| `StepTracker` | tracker.py | Tracks request/response steps | - | +| `ProgressTracker` | tracker.py | Tracks Celery progress updates | - | +| `CostTracker` | tracker.py | Tracks API costs and tokens | - | +| `ConsoleStepTracker` | tracker.py | Console-based step logging | - | +| `AITaskLog` | models.py | Database model for AI task logging | AccountBaseModel | +| `AIProcessor` | utils/ai_processor.py | **LEGACY** Unified AI interface | - | + +### 3.2 Function Classes + +| Class | File | Purpose | Inheritance | +|-------|------|---------|-------------| +| `AutoClusterFunction` | functions/auto_cluster.py | Keyword clustering | BaseAIFunction | +| `GenerateIdeasFunction` | functions/generate_ideas.py | Idea generation | BaseAIFunction | +| `GenerateContentFunction` | functions/generate_content.py | Content generation | BaseAIFunction | +| `GenerateImagesFunction` | functions/generate_images.py | Image generation | BaseAIFunction | + +### 3.3 Data Classes + +| Class | File | Purpose | +|-------|------|---------| +| `StepLog` | types.py | Single step in request/response tracking | +| `ProgressState` | types.py | Progress state for AI tasks | +| `AITaskResult` | types.py | Result from AI function execution | + +--- + +## 4. Dependency Graph/Table + +### 4.1 Import Relationships + +``` +ai/__init__.py + ├─> registry.py (register_function, get_function, list_functions) + ├─> engine.py (AIEngine) + ├─> base.py (BaseAIFunction) + ├─> ai_core.py (AICore) + ├─> validators.py (all validators) + ├─> constants.py (all constants) + ├─> prompts.py (PromptRegistry, get_prompt) + └─> settings.py (MODEL_CONFIG, get_model_config, etc.) + +ai/tasks.py + ├─> engine.py (AIEngine) + └─> registry.py (get_function_instance) + +ai/engine.py + ├─> base.py (BaseAIFunction) + ├─> tracker.py (StepTracker, ProgressTracker, CostTracker, ConsoleStepTracker) + ├─> ai_core.py (AICore) + └─> settings.py (get_model_config) + +ai/ai_core.py + ├─> constants.py (MODEL_RATES, IMAGE_MODEL_RATES, etc.) + └─> tracker.py (ConsoleStepTracker) + +ai/functions/auto_cluster.py + ├─> base.py (BaseAIFunction) + ├─> ai_core.py (AICore) + ├─> prompts.py (PromptRegistry) + └─> settings.py (get_model_config) + +ai/functions/generate_content.py + ├─> base.py (BaseAIFunction) + ├─> ai_core.py (AICore) + ├─> prompts.py (PromptRegistry) + └─> settings.py (get_model_config) + +ai/functions/generate_ideas.py + ├─> base.py (BaseAIFunction) + ├─> ai_core.py (AICore) + ├─> prompts.py (PromptRegistry) + └─> settings.py (get_model_config) + +ai/functions/generate_images.py + ├─> base.py (BaseAIFunction) + ├─> ai_core.py (AICore) + ├─> prompts.py (PromptRegistry) + └─> settings.py (get_model_config) + +utils/ai_processor.py + ├─> modules/system/models.py (IntegrationSettings) + └─> modules/system/utils.py (get_prompt_value, get_default_prompt) + +modules/planner/tasks.py + └─> utils/ai_processor.py (AIProcessor) [DEPRECATED PATH] + +modules/writer/tasks.py + ├─> utils/ai_processor.py (AIProcessor) [LEGACY PATH] + └─> ai/functions/generate_content.py (generate_content_core) +``` + +### 4.2 External Dependencies + +| Dependency | Used By | Purpose | +|------------|---------|---------| +| `django` | All files | Django ORM, models, settings | +| `celery` | tasks.py, engine.py | Async task execution | +| `requests` | ai_core.py, ai_processor.py | HTTP requests to OpenAI/Runware APIs | +| `json` | Multiple files | JSON parsing | +| `re` | ai_core.py, ai_processor.py, content_normalizer.py | Regex for JSON extraction | +| `logging` | All files | Logging | +| `time` | tracker.py, ai_core.py | Timing and duration tracking | +| `bs4` (BeautifulSoup) | content_normalizer.py | HTML parsing (optional) | + +### 4.3 Database Models Dependencies + +| Model | Used By | Purpose | +|-------|---------|---------| +| `AITaskLog` | engine.py | Unified AI task logging | +| `IntegrationSettings` | ai_core.py, ai_processor.py, settings.py | API keys and model configuration | +| `AIPrompt` | prompts.py | Custom prompt templates | +| `Keywords` | auto_cluster.py, validators.py | Keyword data | +| `Clusters` | auto_cluster.py, generate_ideas.py | Cluster data | +| `ContentIdeas` | generate_ideas.py | Content ideas | +| `Tasks` | generate_content.py, generate_images.py | Writer tasks | +| `Content` | generate_content.py | Generated content | +| `Images` | generate_images.py | Generated images | + +--- + +## 5. System Flow Description + +### 5.1 New Unified Framework Flow (Recommended Path) + +``` +Frontend API Call + ↓ +ViewSet Action (e.g., planner/views.py::auto_cluster) + ↓ +run_ai_task.delay(function_name='auto_cluster', payload={ids: [...]}, account_id=123) + ↓ +Celery Worker: run_ai_task (ai/tasks.py) + ├─> Load Account + ├─> get_function_instance('auto_cluster') → AutoClusterFunction + └─> AIEngine.execute(AutoClusterFunction, payload) + ├─> Phase 1: INIT (0-10%) + │ └─> fn.validate(payload, account) + ├─> Phase 2: PREP (10-25%) + │ ├─> fn.prepare(payload, account) → Load keywords + │ └─> fn.build_prompt(data, account) → PromptRegistry.get_prompt() + ├─> Phase 3: AI_CALL (25-70%) + │ ├─> AICore.run_ai_request(prompt, model, ...) + │ │ ├─> Load API key from IntegrationSettings + │ │ ├─> Validate model + │ │ ├─> Build OpenAI request + │ │ ├─> Send HTTP request + │ │ ├─> Parse response + │ │ └─> Calculate cost + │ └─> Track cost via CostTracker + ├─> Phase 4: PARSE (70-85%) + │ └─> fn.parse_response(response_content, step_tracker) + ├─> Phase 5: SAVE (85-98%) + │ └─> fn.save_output(parsed, original_data, account, ...) + │ └─> Database transaction: Create/update clusters + └─> Phase 6: DONE (98-100%) + ├─> Log to AITaskLog + └─> Return result dict + ↓ +Celery Task State Update (SUCCESS/FAILURE) + ↓ +Frontend Polls Task Status + ↓ +Progress Modal Displays Steps +``` + +### 5.2 Legacy Content Generation Flow (Still Active) + +``` +Frontend API Call + ↓ +ViewSet Action (writer/views.py::auto_generate_content) + ↓ +auto_generate_content_task.delay(task_ids, account_id) + ↓ +Celery Worker: auto_generate_content_task (modules/writer/tasks.py) + ├─> Load Tasks from database + ├─> For each task: + │ ├─> Load prompt template (get_prompt_value) + │ ├─> Format prompt with task data + │ ├─> AIProcessor.generate_content(prompt) + │ │ └─> AIProcessor._call_openai() [DUPLICATE OF AICore.run_ai_request] + │ ├─> Parse response (GenerateContentFunction.parse_response) + │ └─> Save content (GenerateContentFunction.save_output) + └─> Return result +``` + +### 5.3 Legacy Clustering Flow (Deprecated) + +``` +Frontend API Call + ↓ +ViewSet Action (planner/views.py::auto_cluster) [OLD PATH] + ↓ +_auto_cluster_keywords_core() (modules/planner/tasks.py) + ├─> Load keywords + ├─> AIProcessor.cluster_keywords() [DEPRECATED] + │ └─> AIProcessor._call_openai() [DUPLICATE] + ├─> Parse clusters + └─> Save to database +``` + +### 5.4 Image Generation Flow + +``` +Frontend API Call + ↓ +ViewSet Action (writer/views.py::auto_generate_images) + ↓ +auto_generate_images_task.delay(task_ids, account_id) + ↓ +Celery Worker: auto_generate_images_task (modules/writer/tasks.py) + ├─> Load tasks + ├─> For each task: + │ ├─> Extract image prompts (AIProcessor.extract_image_prompts) + │ │ └─> Calls AI to extract prompts from content + │ ├─> Generate featured image (AIProcessor.generate_image) + │ ├─> Generate desktop images (if enabled) + │ └─> Generate mobile images (if enabled) + └─> Save Images records +``` + +--- + +## 6. Integration Points + +### 6.1 Celery Integration + +**Task Registration:** +- `celery.py` uses `app.autodiscover_tasks()` to auto-discover tasks from all Django apps +- Tasks are registered via `@shared_task` decorator + +**Registered Tasks:** +1. `ai.tasks.run_ai_task` - Unified entrypoint (NEW) +2. `planner.tasks.auto_cluster_keywords_task` - **DEPRECATED** +3. `writer.tasks.auto_generate_content_task` - Legacy (still active) +4. `writer.tasks.auto_generate_images_task` - Legacy (still active) + +**Task State Management:** +- `ProgressTracker.update()` calls `task.update_state(state='PROGRESS', meta={...})` +- Progress metadata includes: `phase`, `percentage`, `message`, `request_steps`, `response_steps` +- Final states: `SUCCESS`, `FAILURE` + +### 6.2 Database Integration + +**Models:** +- `AITaskLog` - Unified logging table (used by `AIEngine._log_to_database()`) +- `IntegrationSettings` - API keys and model configuration (used by `AICore._load_account_settings()`) +- `AIPrompt` - Custom prompt templates (used by `PromptRegistry.get_prompt()`) + +**Data Models:** +- `Keywords` - Input for clustering +- `Clusters` - Output of clustering +- `ContentIdeas` - Output of idea generation +- `Tasks` - Input for content/image generation +- `Content` - Output of content generation +- `Images` - Output of image generation + +### 6.3 Frontend API Integration + +**Endpoints:** +1. `POST /v1/planner/keywords/auto_cluster/` → `planner.views.ClusterViewSet.auto_cluster()` + - Calls `run_ai_task.delay(function_name='auto_cluster', ...)` + +2. `POST /v1/planner/clusters/auto_generate_ideas/` → `planner.views.ClusterViewSet.auto_generate_ideas()` + - Calls `run_ai_task.delay(function_name='auto_generate_ideas', ...)` + +3. `POST /v1/writer/tasks/auto_generate_content/` → `writer.views.TasksViewSet.auto_generate_content()` + - Calls `auto_generate_content_task.delay(...)` [LEGACY PATH] + +4. `POST /v1/writer/tasks/auto_generate_images/` → `writer.views.TasksViewSet.auto_generate_images()` + - Calls `auto_generate_images_task.delay(...)` [LEGACY PATH] + +**Response Format:** +- Success: `{success: true, task_id: "...", message: "..."}` +- Error: `{success: false, error: "..."}` + +**Progress Tracking:** +- Frontend polls Celery task status via `GET /v1/system/tasks/{task_id}/status/` +- Progress modal displays `request_steps` and `response_steps` from task meta + +### 6.4 Configuration Integration + +**Settings Loading Hierarchy:** +1. **Account-level** (`IntegrationSettings` model): + - API keys: `IntegrationSettings.config['apiKey']` + - Model: `IntegrationSettings.config['model']` + - Image settings: `IntegrationSettings.config` (for image_generation type) + +2. **Django Settings** (fallback): + - `OPENAI_API_KEY` + - `RUNWARE_API_KEY` + - `DEFAULT_AI_MODEL` + +3. **Function-level** (`ai/settings.py`): + - `MODEL_CONFIG[function_name]` - Default model, max_tokens, temperature per function + +**Prompt Loading Hierarchy:** +1. Task-level override: `task.prompt_override` (if exists) +2. Account-level: `AIPrompt` model (account, prompt_type) +3. Default: `PromptRegistry.DEFAULT_PROMPTS[prompt_type]` + +### 6.5 Debug Panel Integration + +**Console Logging:** +- `ConsoleStepTracker` logs to stdout/stderr (only if `DEBUG_MODE=True`) +- Logs include timestamps, phases, messages, errors + +**Step Tracking:** +- `StepTracker` maintains `request_steps` and `response_steps` arrays +- Steps include: `stepNumber`, `stepName`, `status`, `message`, `duration`, `error` +- Steps are included in Celery task meta and displayed in progress modal + +**Database Logging:** +- `AITaskLog` records all AI task executions +- Fields: `task_id`, `function_name`, `phase`, `status`, `cost`, `tokens`, `request_steps`, `response_steps`, `error`, `payload`, `result` + +--- + +## 7. Identified Redundancies or Repetition + +### 7.1 Duplicate Constants + +**Location 1:** `ai/constants.py` +- `MODEL_RATES`, `IMAGE_MODEL_RATES`, `VALID_OPENAI_IMAGE_MODELS`, `VALID_SIZES_BY_MODEL`, `DEFAULT_AI_MODEL`, `JSON_MODE_MODELS` + +**Location 2:** `utils/ai_processor.py` (lines 18-44) +- **EXACT DUPLICATE** of all constants from `constants.py` + +**Impact:** Constants are defined in two places, risking inconsistency. + +### 7.2 Duplicate JSON Extraction Logic + +**Location 1:** `ai/ai_core.py::extract_json()` (lines 391-429) +- Handles markdown code blocks, multiline JSON, direct JSON + +**Location 2:** `utils/ai_processor.py::_extract_json_from_response()` (lines 342-449) +- **MORE COMPREHENSIVE** - handles more edge cases, balanced brace matching + +**Impact:** Two implementations with different capabilities. `ai_processor.py` version is more robust. + +### 7.3 Duplicate OpenAI API Calling Logic + +**Location 1:** `ai/ai_core.py::run_ai_request()` (lines 106-389) +- Centralized method with console logging via `ConsoleStepTracker` +- Handles validation, model selection, cost calculation +- Returns standardized dict format + +**Location 2:** `utils/ai_processor.py::_call_openai()` (lines 125-340) +- **SIMILAR LOGIC** but with `response_steps` parameter instead of `tracker` +- Less comprehensive error handling +- Different return format + +**Impact:** Two code paths for the same operation. New code should use `AICore.run_ai_request()`. + +### 7.4 Duplicate Image Generation Logic + +**Location 1:** `ai/ai_core.py::generate_image()` + `_generate_image_openai()` + `_generate_image_runware()` (lines 431-728) +- Uses `print()` statements for logging (inconsistent with console tracker) + +**Location 2:** `utils/ai_processor.py::generate_image()` (lines 667-1043) +- **MORE COMPREHENSIVE** - extensive logging, better error handling +- Handles Runware authentication flow + +**Impact:** Two implementations. `ai_processor.py` version has better logging. + +### 7.5 Duplicate Prompt Loading Logic + +**Location 1:** `ai/prompts.py::PromptRegistry.get_prompt()` (lines 280-333) +- Hierarchical resolution: task override → DB prompt → default +- Supports `[IGNY8_*]` placeholders and `{variable}` format + +**Location 2:** `utils/ai_processor.py::get_prompt()` (lines 1044-1057) +- Simple database lookup via `modules/system/utils.get_prompt_value()` +- No hierarchical resolution + +**Location 3:** Direct calls in `modules/writer/tasks.py` (lines 343, 959, 964) +- Uses `get_prompt_value()` and `get_default_prompt()` directly + +**Impact:** Three different ways to load prompts. New code should use `PromptRegistry`. + +### 7.6 Duplicate Model Configuration Logic + +**Location 1:** `ai/settings.py::get_model_config()` (lines 49-97) +- Reads from `IntegrationSettings` if account provided +- Falls back to `MODEL_CONFIG` defaults + +**Location 2:** `ai/ai_core.py::_load_account_settings()` (lines 46-90) +- Reads model from `IntegrationSettings` directly +- Similar logic but embedded in `AICore.__init__()` + +**Location 3:** `utils/ai_processor.py::_get_model()` (lines 98-123) +- Reads model from `IntegrationSettings` directly +- Similar logic but embedded in `AIProcessor.__init__()` + +**Impact:** Model loading logic duplicated in three places. + +### 7.7 Duplicate API Key Loading Logic + +**Location 1:** `ai/ai_core.py::_load_account_settings()` (lines 46-90) +- Loads OpenAI and Runware keys from `IntegrationSettings` + +**Location 2:** `utils/ai_processor.py::_get_api_key()` (lines 73-96) +- **EXACT SAME LOGIC** for loading API keys + +**Impact:** Identical code in two places. + +### 7.8 Repeated Error Handling Patterns + +**Pattern:** Multiple files have similar try/except blocks for: +- API request errors +- JSON parsing errors +- Database errors +- Validation errors + +**Impact:** Error handling is not centralized, making it harder to maintain consistent error messages and logging. + +### 7.9 Repeated Progress Update Patterns + +**Pattern:** Multiple places manually build progress update dicts: +- `modules/writer/tasks.py` (lines 62-73, 220-231, etc.) +- `modules/planner/tasks.py` (lines 59-71, 203-215, etc.) +- `ai/engine.py` (lines 57, 79, 141, etc.) + +**Impact:** Progress update format is not standardized, though `ProgressTracker` exists to handle this. + +--- + +## 8. Summary of Potential Consolidation Areas + +### 8.1 Constants Consolidation + +**Observation:** Model rates, valid models, and configuration constants are duplicated between `ai/constants.py` and `utils/ai_processor.py`. + +**Potential Action:** Remove constants from `ai_processor.py` and import from `constants.py`. However, `ai_processor.py` is marked as legacy, so this may not be necessary if it's being phased out. + +### 8.2 JSON Extraction Consolidation + +**Observation:** Two JSON extraction methods exist with different capabilities. `ai_processor.py::_extract_json_from_response()` is more comprehensive. + +**Potential Action:** Enhance `AICore.extract_json()` with logic from `ai_processor.py`, or create a shared utility function. + +### 8.3 API Request Consolidation + +**Observation:** `AICore.run_ai_request()` and `AIProcessor._call_openai()` perform the same operation with different interfaces. + +**Potential Action:** All new code should use `AICore.run_ai_request()`. Legacy code in `ai_processor.py` can remain for backward compatibility but should be marked as deprecated. + +### 8.4 Image Generation Consolidation + +**Observation:** Two image generation implementations exist. `ai_processor.py` version has better logging. + +**Potential Action:** Enhance `AICore.generate_image()` with logging improvements from `ai_processor.py`, or migrate all code to use `AICore.generate_image()`. + +### 8.5 Prompt Loading Consolidation + +**Observation:** Three different methods exist for loading prompts: `PromptRegistry.get_prompt()`, `AIProcessor.get_prompt()`, and direct `get_prompt_value()` calls. + +**Potential Action:** Migrate all code to use `PromptRegistry.get_prompt()` for consistency. Update legacy code paths gradually. + +### 8.6 Model/API Key Loading Consolidation + +**Observation:** Model and API key loading logic is duplicated in `AICore`, `AIProcessor`, and `settings.py`. + +**Potential Action:** Create shared utility functions for loading settings from `IntegrationSettings`, used by all classes. + +### 8.7 Error Handling Consolidation + +**Observation:** Error handling patterns are repeated across multiple files. + +**Potential Action:** Create centralized error handling utilities or enhance `AIEngine._handle_error()` to be more reusable. + +### 8.8 Progress Tracking Consolidation + +**Observation:** Some code manually builds progress update dicts instead of using `ProgressTracker`. + +**Potential Action:** Migrate all progress updates to use `ProgressTracker.update()` for consistency. + +### 8.9 Legacy Code Path Elimination + +**Observation:** Multiple execution paths exist: +- New: `run_ai_task` → `AIEngine` → `BaseAIFunction` implementations +- Legacy: Direct `AIProcessor` calls in `modules/*/tasks.py` + +**Potential Action:** Gradually migrate all legacy tasks to use the new framework. Mark legacy code as deprecated. + +--- + +## 9. Assumptions Made + +1. **File `wordpress.py`** was not read - assumed to be unrelated to AI processing based on name. + +2. **Frontend code** was partially analyzed via search results - full frontend audit not performed. + +3. **Database migrations** were not analyzed - assumed to be standard Django migrations. + +4. **Test files** were not analyzed - `ai/tests/test_run.py` exists but was not read. + +5. **Settings file** (`backend/igny8_core/settings.py`) was not read - assumed to contain standard Django settings. + +6. **System module utilities** (`modules/system/utils.py`) were referenced but not fully read - assumed to contain `get_prompt_value()` and `get_default_prompt()` functions. + +--- + +## 10. Appendix + +### 10.1 File Line Counts + +| Directory | Files | Total Lines | +|-----------|-------|-------------| +| `ai/` | 15 | ~2,500 | +| `ai/functions/` | 5 | ~1,300 | +| `utils/` (AI-related) | 3 | ~1,800 | +| `modules/planner/tasks.py` | 1 | 736 | +| `modules/writer/tasks.py` | 1 | 1,156 | +| **Total** | **25** | **~7,500** | + +### 10.2 Function Count Summary + +- **Core Framework Functions:** ~30 +- **AI Function Implementations:** 4 classes × ~6 methods = ~24 methods +- **Tracking Functions:** ~20 +- **Legacy Functions:** ~15 +- **Celery Tasks:** 4 +- **Total:** ~90 functions/methods + +### 10.3 Key Design Patterns + +1. **Template Method Pattern:** `BaseAIFunction` defines algorithm skeleton, subclasses implement steps +2. **Registry Pattern:** `FunctionRegistry` for dynamic function discovery +3. **Factory Pattern:** `get_function_instance()` creates function instances +4. **Strategy Pattern:** Different AI functions implement same interface +5. **Observer Pattern:** `ProgressTracker` updates Celery task state +6. **Facade Pattern:** `AICore` provides simplified interface to OpenAI/Runware APIs + +--- + +**End of Report** +