salman/igny8
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
727c999413798856c91796892999d9c670104346
igny8/IGNY8_AI_SYSTEM_AUDIT_BASELINE_REPORT.md
Desktop ecbab4d380 audits
2025-11-10 23:04:12 +05:00

34 KiB
Raw Blame History

IGNY8 AI System Audit — Current Structure & Flow Mapping (Baseline Report)

Date: 2024-12-19
Scope: Complete structural and functional audit of the IGNY8 AI subsystem
Methodology: Code analysis without modifications or assumptions

Executive Summary

The IGNY8 AI subsystem is a comprehensive framework for AI-powered content operations including keyword clustering, idea generation, content generation, and image generation. The system uses a unified framework architecture with a centralized execution engine, but also maintains legacy code paths for backward compatibility.

Key Findings:

  • 20 core AI files in backend/igny8_core/ai/
  • 4 AI function implementations following BaseAIFunction pattern
  • Dual execution paths: New unified framework (run_ai_task → AIEngine) and legacy paths (direct task calls)
  • Centralized AI request handling via AICore.run_ai_request()
  • Comprehensive tracking via StepTracker, ProgressTracker, CostTracker, and ConsoleStepTracker
  • Multiple prompt management systems: PromptRegistry (new) and direct database queries (legacy)

1. Current File Inventory

1.1 Core AI Framework (backend/igny8_core/ai/)

File Lines Purpose
__init__.py 73 Package exports - exposes main classes and functions
admin.py 60 Django admin configuration for AITaskLog model
ai_core.py 756 Centralized AI request handler - AICore class with run_ai_request() and generate_image()
apps.py 21 Django app configuration
base.py 95 Abstract base class BaseAIFunction - defines function interface
constants.py 42 Model pricing, valid models, configuration constants
engine.py 375 Central orchestrator AIEngine - manages function lifecycle
models.py 52 Database model AITaskLog for unified logging
processor.py 76 DEPRECATED wrapper around AICore for backward compatibility
prompts.py 432 PromptRegistry - centralized prompt management with hierarchical resolution
registry.py 97 Function registry with lazy loading - register_function(), get_function()
settings.py 117 Model configurations per function - MODEL_CONFIG, get_model_config()
tasks.py 131 Unified Celery task entrypoint run_ai_task() - single entry point for all AI functions
tracker.py 348 Progress tracking utilities - StepTracker, ProgressTracker, CostTracker, ConsoleStepTracker
types.py 44 Type definitions - StepLog, ProgressState, AITaskResult dataclasses
validators.py 187 Validation functions - validate_ids(), validate_keywords_exist(), etc.

1.2 AI Function Implementations (backend/igny8_core/ai/functions/)

File Lines Purpose
__init__.py 18 Function package exports
auto_cluster.py 330 AutoClusterFunction - Groups keywords into semantic clusters
generate_content.py 388 GenerateContentFunction + generate_content_core() - Generates article content
generate_ideas.py 335 GenerateIdeasFunction + generate_ideas_core() - Generates content ideas from clusters
generate_images.py 279 GenerateImagesFunction + generate_images_core() - Generates images for tasks

1.3 Utility Files (backend/igny8_core/utils/)

File Lines Purpose
ai_processor.py 1407 LEGACY Unified AI interface - AIProcessor class with OpenAI/Runware support. Contains duplicate constants and methods.
content_normalizer.py 273 Content normalization - converts AI responses to HTML format
queue_manager.py 90 Queue abstraction (currently placeholder, not fully implemented)
wordpress.py (not read) WordPress integration utilities

1.4 Module Task Files

File Lines Purpose
modules/planner/tasks.py 736 DEPRECATED Legacy clustering task auto_cluster_keywords_task() - uses old AIProcessor
modules/writer/tasks.py 1156 Legacy content/image generation tasks - auto_generate_content_task(), auto_generate_images_task()

1.5 Configuration Files

File Purpose
celery.py Celery app configuration - auto-discovers tasks from all Django apps
modules/system/models.py AIPrompt, IntegrationSettings models for AI configuration

2. Function Inventory

2.1 Core Framework Functions

AICore (ai_core.py)

  • __init__(account=None) - Initialize with account context, loads API keys and model from IntegrationSettings
  • _load_account_settings() - Loads OpenAI/Runware API keys and model from IntegrationSettings or Django settings
  • get_api_key(integration_type='openai') - Returns API key for integration type
  • get_model(integration_type='openai') - Returns model name for integration type
  • run_ai_request(prompt, model=None, max_tokens=4000, temperature=0.7, response_format=None, api_key=None, function_name='ai_request', function_id=None, tracker=None) - CENTRAL METHOD - Handles all OpenAI text generation requests with console logging
  • extract_json(response_text) - Extracts JSON from response text (handles markdown code blocks)
  • generate_image(prompt, provider='openai', model=None, size='1024x1024', n=1, api_key=None, negative_prompt=None, function_name='generate_image') - Generates images via OpenAI DALL-E or Runware
  • _generate_image_openai(...) - Internal method for OpenAI image generation
  • _generate_image_runware(...) - Internal method for Runware image generation
  • calculate_cost(model, input_tokens, output_tokens, model_type='text') - Calculates API cost
  • call_openai(...) - LEGACY - Redirects to run_ai_request()

AIEngine (engine.py)

  • __init__(celery_task=None, account=None) - Initialize with Celery task and account
  • execute(fn: BaseAIFunction, payload: dict) - CENTRAL ORCHESTRATOR - Unified execution pipeline:
    • Phase 1: INIT (0-10%) - Validation
    • Phase 2: PREP (10-25%) - Data loading & prompt building
    • Phase 3: AI_CALL (25-70%) - API call to provider
    • Phase 4: PARSE (70-85%) - Response parsing
    • Phase 5: SAVE (85-98%) - Database operations
    • Phase 6: DONE (98-100%) - Finalization
  • _handle_error(error, fn=None, exc_info=False) - Centralized error handling
  • _log_to_database(fn, payload, parsed, save_result, error=None) - Logs to AITaskLog model
  • _calculate_credits_for_clustering(keyword_count, tokens, cost) - Calculates credits for clustering operations

PromptRegistry (prompts.py)

  • get_prompt(function_name, account=None, task=None, context=None) - Hierarchical prompt resolution:
    1. Task-level prompt_override (if exists)
    2. DB prompt for (account, function)
    3. Default fallback from registry
  • _render_prompt(prompt_template, context) - Renders template with [IGNY8_*] placeholders and {variable} format
  • get_image_prompt_template(account=None) - Gets image prompt template
  • get_negative_prompt(account=None) - Gets negative prompt

BaseAIFunction (base.py) - Abstract Interface

  • get_name() - Returns function name (abstract)
  • get_metadata() - Returns function metadata (display name, description, phases)
  • validate(payload, account=None) - Validates input payload (default: checks for 'ids')
  • get_max_items() - Returns max items limit (optional)
  • prepare(payload, account=None) - Loads and prepares data (abstract)
  • build_prompt(data, account=None) - Builds AI prompt (abstract)
  • get_model(account=None) - Returns model override (optional)
  • parse_response(response, step_tracker=None) - Parses AI response (abstract)
  • save_output(parsed, original_data, account=None, progress_tracker=None, step_tracker=None) - Saves results to database (abstract)

2.2 AI Function Implementations

AutoClusterFunction (functions/auto_cluster.py)

  • get_name() - Returns 'auto_cluster'
  • validate(payload, account=None) - Validates keyword IDs exist
  • prepare(payload, account=None) - Loads keywords with relationships
  • build_prompt(data, account=None) - Builds clustering prompt using PromptRegistry
  • parse_response(response, step_tracker=None) - Parses JSON cluster data
  • save_output(parsed, original_data, account=None, ...) - Creates/updates clusters and assigns keywords

GenerateIdeasFunction (functions/generate_ideas.py)

  • get_name() - Returns 'generate_ideas'
  • validate(payload, account=None) - Validates cluster IDs exist
  • prepare(payload, account=None) - Loads clusters with keywords
  • build_prompt(data, account=None) - Builds ideas generation prompt
  • parse_response(response, step_tracker=None) - Parses JSON ideas data
  • save_output(parsed, original_data, account=None, ...) - Creates ContentIdeas records

GenerateContentFunction (functions/generate_content.py)

  • get_name() - Returns 'generate_content'
  • validate(payload, account=None) - Validates task IDs exist
  • prepare(payload, account=None) - Loads tasks with relationships
  • build_prompt(data, account=None) - Builds content generation prompt
  • parse_response(response, step_tracker=None) - Parses JSON or plain text content
  • save_output(parsed, original_data, account=None, ...) - Saves content to Content model

GenerateImagesFunction (functions/generate_images.py)

  • get_name() - Returns 'generate_images'
  • validate(payload, account=None) - Validates task IDs exist
  • prepare(payload, account=None) - Loads tasks and image settings
  • build_prompt(data, account=None) - Extracts image prompts from task content (calls AI)
  • parse_response(response, step_tracker=None) - Returns parsed response (already parsed)
  • save_output(parsed, original_data, account=None, ...) - Creates Images records

2.3 Tracking Functions

StepTracker (tracker.py)

  • add_request_step(step_name, status='success', message='', error=None, duration=None) - Adds request step
  • add_response_step(step_name, status='success', message='', error=None, duration=None) - Adds response step
  • get_meta() - Returns metadata dict with request/response steps

ProgressTracker (tracker.py)

  • update(phase, percentage, message, current=None, total=None, current_item=None, meta=None) - Updates Celery task state
  • set_phase(phase, percentage, message, meta=None) - Sets progress phase
  • complete(message='Task complete!', meta=None) - Marks task as complete
  • error(error_message, meta=None) - Marks task as failed
  • get_duration() - Returns elapsed time in milliseconds

ConsoleStepTracker (tracker.py)

  • init(message='Task started') - Logs initialization
  • prep(message) - Logs preparation phase
  • ai_call(message) - Logs AI call phase
  • parse(message) - Logs parsing phase
  • save(message) - Logs save phase
  • done(message='Execution completed') - Logs completion
  • error(error_type, message, exception=None) - Logs error
  • retry(attempt, max_attempts, reason='') - Logs retry
  • timeout(timeout_seconds) - Logs timeout
  • rate_limit(retry_after) - Logs rate limit
  • malformed_json(details='') - Logs JSON parsing error

CostTracker (tracker.py)

  • record(function_name, cost, tokens, model=None) - Records API call cost
  • get_total() - Returns total cost
  • get_total_tokens() - Returns total tokens
  • get_operations() - Returns all operations list

2.4 Celery Tasks

run_ai_task (ai/tasks.py)

  • run_ai_task(self, function_name: str, payload: dict, account_id: int = None) - UNIFIED ENTRYPOINT - Dynamically loads and executes AI functions via AIEngine

Legacy Tasks (modules/*/tasks.py)

  • auto_cluster_keywords_task (planner/tasks.py) - DEPRECATED - Uses old AIProcessor
  • auto_generate_content_task (writer/tasks.py) - Uses AIProcessor directly (not via framework)
  • auto_generate_images_task (writer/tasks.py) - Uses AIProcessor directly

2.5 Legacy Functions (ai_processor.py)

AIProcessor - LEGACY/DEPRECATED

  • _call_openai(prompt, model=None, max_tokens=4000, temperature=0.7, response_format=None, api_key=None, function_id=None, response_steps=None) - Internal OpenAI API caller
  • _extract_json_from_response(response_text) - JSON extraction (duplicate of AICore.extract_json())
  • generate_content(prompt, model=None, max_tokens=4000, temperature=0.7, **kwargs) - Generates text content
  • extract_image_prompts(content, title, max_images=3, account=None) - Extracts image prompts from content
  • check_moderation(text, api_key=None) - Checks content moderation
  • generate_image(prompt, provider='openai', model=None, size='1024x1024', n=1, api_key=None, **kwargs) - Generates images
  • cluster_keywords(keywords, sector_name=None, account=None, response_steps=None, progress_callback=None, tracker=None, **kwargs) - DEPRECATED - Clusters keywords (old method)
  • generate_ideas(clusters, account=None, **kwargs) - Generates ideas (old method)
  • get_prompt(prompt_type, account=None) - Gets prompt from database (old method)
  • estimate_cost(operation, tokens_or_prompt, model=None) - Estimates cost (not implemented)

3. Class Inventory

3.1 Core Classes

Class File Purpose Inheritance
AICore ai_core.py Centralized AI request handler -
AIEngine engine.py Central orchestrator for AI functions -
BaseAIFunction base.py Abstract base for all AI functions ABC
PromptRegistry prompts.py Centralized prompt management -
StepTracker tracker.py Tracks request/response steps -
ProgressTracker tracker.py Tracks Celery progress updates -
CostTracker tracker.py Tracks API costs and tokens -
ConsoleStepTracker tracker.py Console-based step logging -
AITaskLog models.py Database model for AI task logging AccountBaseModel
AIProcessor utils/ai_processor.py LEGACY Unified AI interface -

3.2 Function Classes

Class File Purpose Inheritance
AutoClusterFunction functions/auto_cluster.py Keyword clustering BaseAIFunction
GenerateIdeasFunction functions/generate_ideas.py Idea generation BaseAIFunction
GenerateContentFunction functions/generate_content.py Content generation BaseAIFunction
GenerateImagesFunction functions/generate_images.py Image generation BaseAIFunction

3.3 Data Classes

Class File Purpose
StepLog types.py Single step in request/response tracking
ProgressState types.py Progress state for AI tasks
AITaskResult types.py Result from AI function execution

4. Dependency Graph/Table

4.1 Import Relationships

ai/__init__.py
  ├─> registry.py (register_function, get_function, list_functions)
  ├─> engine.py (AIEngine)
  ├─> base.py (BaseAIFunction)
  ├─> ai_core.py (AICore)
  ├─> validators.py (all validators)
  ├─> constants.py (all constants)
  ├─> prompts.py (PromptRegistry, get_prompt)
  └─> settings.py (MODEL_CONFIG, get_model_config, etc.)

ai/tasks.py
  ├─> engine.py (AIEngine)
  └─> registry.py (get_function_instance)

ai/engine.py
  ├─> base.py (BaseAIFunction)
  ├─> tracker.py (StepTracker, ProgressTracker, CostTracker, ConsoleStepTracker)
  ├─> ai_core.py (AICore)
  └─> settings.py (get_model_config)

ai/ai_core.py
  ├─> constants.py (MODEL_RATES, IMAGE_MODEL_RATES, etc.)
  └─> tracker.py (ConsoleStepTracker)

ai/functions/auto_cluster.py
  ├─> base.py (BaseAIFunction)
  ├─> ai_core.py (AICore)
  ├─> prompts.py (PromptRegistry)
  └─> settings.py (get_model_config)

ai/functions/generate_content.py
  ├─> base.py (BaseAIFunction)
  ├─> ai_core.py (AICore)
  ├─> prompts.py (PromptRegistry)
  └─> settings.py (get_model_config)

ai/functions/generate_ideas.py
  ├─> base.py (BaseAIFunction)
  ├─> ai_core.py (AICore)
  ├─> prompts.py (PromptRegistry)
  └─> settings.py (get_model_config)

ai/functions/generate_images.py
  ├─> base.py (BaseAIFunction)
  ├─> ai_core.py (AICore)
  ├─> prompts.py (PromptRegistry)
  └─> settings.py (get_model_config)

utils/ai_processor.py
  ├─> modules/system/models.py (IntegrationSettings)
  └─> modules/system/utils.py (get_prompt_value, get_default_prompt)

modules/planner/tasks.py
  └─> utils/ai_processor.py (AIProcessor) [DEPRECATED PATH]

modules/writer/tasks.py
  ├─> utils/ai_processor.py (AIProcessor) [LEGACY PATH]
  └─> ai/functions/generate_content.py (generate_content_core)

4.2 External Dependencies

Dependency Used By Purpose
django All files Django ORM, models, settings
celery tasks.py, engine.py Async task execution
requests ai_core.py, ai_processor.py HTTP requests to OpenAI/Runware APIs
json Multiple files JSON parsing
re ai_core.py, ai_processor.py, content_normalizer.py Regex for JSON extraction
logging All files Logging
time tracker.py, ai_core.py Timing and duration tracking
bs4 (BeautifulSoup) content_normalizer.py HTML parsing (optional)

4.3 Database Models Dependencies

Model Used By Purpose
AITaskLog engine.py Unified AI task logging
IntegrationSettings ai_core.py, ai_processor.py, settings.py API keys and model configuration
AIPrompt prompts.py Custom prompt templates
Keywords auto_cluster.py, validators.py Keyword data
Clusters auto_cluster.py, generate_ideas.py Cluster data
ContentIdeas generate_ideas.py Content ideas
Tasks generate_content.py, generate_images.py Writer tasks
Content generate_content.py Generated content
Images generate_images.py Generated images

5. System Flow Description

5.1 New Unified Framework Flow (Recommended Path)

Frontend API Call
  ↓
ViewSet Action (e.g., planner/views.py::auto_cluster)
  ↓
run_ai_task.delay(function_name='auto_cluster', payload={ids: [...]}, account_id=123)
  ↓
Celery Worker: run_ai_task (ai/tasks.py)
  ├─> Load Account
  ├─> get_function_instance('auto_cluster') → AutoClusterFunction
  └─> AIEngine.execute(AutoClusterFunction, payload)
       ├─> Phase 1: INIT (0-10%)
       │   └─> fn.validate(payload, account)
       ├─> Phase 2: PREP (10-25%)
       │   ├─> fn.prepare(payload, account) → Load keywords
       │   └─> fn.build_prompt(data, account) → PromptRegistry.get_prompt()
       ├─> Phase 3: AI_CALL (25-70%)
       │   ├─> AICore.run_ai_request(prompt, model, ...)
       │   │   ├─> Load API key from IntegrationSettings
       │   │   ├─> Validate model
       │   │   ├─> Build OpenAI request
       │   │   ├─> Send HTTP request
       │   │   ├─> Parse response
       │   │   └─> Calculate cost
       │   └─> Track cost via CostTracker
       ├─> Phase 4: PARSE (70-85%)
       │   └─> fn.parse_response(response_content, step_tracker)
       ├─> Phase 5: SAVE (85-98%)
       │   └─> fn.save_output(parsed, original_data, account, ...)
       │       └─> Database transaction: Create/update clusters
       └─> Phase 6: DONE (98-100%)
           ├─> Log to AITaskLog
           └─> Return result dict
  ↓
Celery Task State Update (SUCCESS/FAILURE)
  ↓
Frontend Polls Task Status
  ↓
Progress Modal Displays Steps

5.2 Legacy Content Generation Flow (Still Active)

Frontend API Call
  ↓
ViewSet Action (writer/views.py::auto_generate_content)
  ↓
auto_generate_content_task.delay(task_ids, account_id)
  ↓
Celery Worker: auto_generate_content_task (modules/writer/tasks.py)
  ├─> Load Tasks from database
  ├─> For each task:
  │   ├─> Load prompt template (get_prompt_value)
  │   ├─> Format prompt with task data
  │   ├─> AIProcessor.generate_content(prompt)
  │   │   └─> AIProcessor._call_openai() [DUPLICATE OF AICore.run_ai_request]
  │   ├─> Parse response (GenerateContentFunction.parse_response)
  │   └─> Save content (GenerateContentFunction.save_output)
  └─> Return result

5.3 Legacy Clustering Flow (Deprecated)

Frontend API Call
  ↓
ViewSet Action (planner/views.py::auto_cluster) [OLD PATH]
  ↓
_auto_cluster_keywords_core() (modules/planner/tasks.py)
  ├─> Load keywords
  ├─> AIProcessor.cluster_keywords() [DEPRECATED]
  │   └─> AIProcessor._call_openai() [DUPLICATE]
  ├─> Parse clusters
  └─> Save to database

5.4 Image Generation Flow

Frontend API Call
  ↓
ViewSet Action (writer/views.py::auto_generate_images)
  ↓
auto_generate_images_task.delay(task_ids, account_id)
  ↓
Celery Worker: auto_generate_images_task (modules/writer/tasks.py)
  ├─> Load tasks
  ├─> For each task:
  │   ├─> Extract image prompts (AIProcessor.extract_image_prompts)
  │   │   └─> Calls AI to extract prompts from content
  │   ├─> Generate featured image (AIProcessor.generate_image)
  │   ├─> Generate desktop images (if enabled)
  │   └─> Generate mobile images (if enabled)
  └─> Save Images records

6. Integration Points

6.1 Celery Integration

Task Registration:

  • celery.py uses app.autodiscover_tasks() to auto-discover tasks from all Django apps
  • Tasks are registered via @shared_task decorator

Registered Tasks:

  1. ai.tasks.run_ai_task - Unified entrypoint (NEW)
  2. planner.tasks.auto_cluster_keywords_task - DEPRECATED
  3. writer.tasks.auto_generate_content_task - Legacy (still active)
  4. writer.tasks.auto_generate_images_task - Legacy (still active)

Task State Management:

  • ProgressTracker.update() calls task.update_state(state='PROGRESS', meta={...})
  • Progress metadata includes: phase, percentage, message, request_steps, response_steps
  • Final states: SUCCESS, FAILURE

6.2 Database Integration

Models:

  • AITaskLog - Unified logging table (used by AIEngine._log_to_database())
  • IntegrationSettings - API keys and model configuration (used by AICore._load_account_settings())
  • AIPrompt - Custom prompt templates (used by PromptRegistry.get_prompt())

Data Models:

  • Keywords - Input for clustering
  • Clusters - Output of clustering
  • ContentIdeas - Output of idea generation
  • Tasks - Input for content/image generation
  • Content - Output of content generation
  • Images - Output of image generation

6.3 Frontend API Integration

Endpoints:

  1. POST /v1/planner/keywords/auto_cluster/ → planner.views.ClusterViewSet.auto_cluster()

    • Calls run_ai_task.delay(function_name='auto_cluster', ...)

  2. POST /v1/planner/clusters/auto_generate_ideas/ → planner.views.ClusterViewSet.auto_generate_ideas()

    • Calls run_ai_task.delay(function_name='auto_generate_ideas', ...)

  3. POST /v1/writer/tasks/auto_generate_content/ → writer.views.TasksViewSet.auto_generate_content()

    • Calls auto_generate_content_task.delay(...) [LEGACY PATH]

  4. POST /v1/writer/tasks/auto_generate_images/ → writer.views.TasksViewSet.auto_generate_images()

    • Calls auto_generate_images_task.delay(...) [LEGACY PATH]

Response Format:

  • Success: {success: true, task_id: "...", message: "..."}
  • Error: {success: false, error: "..."}

Progress Tracking:

  • Frontend polls Celery task status via GET /v1/system/tasks/{task_id}/status/
  • Progress modal displays request_steps and response_steps from task meta

6.4 Configuration Integration

Settings Loading Hierarchy:

  1. Account-level (IntegrationSettings model):

    • API keys: IntegrationSettings.config['apiKey']
    • Model: IntegrationSettings.config['model']
    • Image settings: IntegrationSettings.config (for image_generation type)

  2. Django Settings (fallback):

    • OPENAI_API_KEY
    • RUNWARE_API_KEY
    • DEFAULT_AI_MODEL

  3. Function-level (ai/settings.py):

    • MODEL_CONFIG[function_name] - Default model, max_tokens, temperature per function

Prompt Loading Hierarchy:

  1. Task-level override: task.prompt_override (if exists)
  2. Account-level: AIPrompt model (account, prompt_type)
  3. Default: PromptRegistry.DEFAULT_PROMPTS[prompt_type]

6.5 Debug Panel Integration

Console Logging:

  • ConsoleStepTracker logs to stdout/stderr (only if DEBUG_MODE=True)
  • Logs include timestamps, phases, messages, errors

Step Tracking:

  • StepTracker maintains request_steps and response_steps arrays
  • Steps include: stepNumber, stepName, status, message, duration, error
  • Steps are included in Celery task meta and displayed in progress modal

Database Logging:

  • AITaskLog records all AI task executions
  • Fields: task_id, function_name, phase, status, cost, tokens, request_steps, response_steps, error, payload, result

7. Identified Redundancies or Repetition

7.1 Duplicate Constants

Location 1: ai/constants.py

  • MODEL_RATES, IMAGE_MODEL_RATES, VALID_OPENAI_IMAGE_MODELS, VALID_SIZES_BY_MODEL, DEFAULT_AI_MODEL, JSON_MODE_MODELS

Location 2: utils/ai_processor.py (lines 18-44)

  • EXACT DUPLICATE of all constants from constants.py

Impact: Constants are defined in two places, risking inconsistency.

7.2 Duplicate JSON Extraction Logic

Location 1: ai/ai_core.py::extract_json() (lines 391-429)

  • Handles markdown code blocks, multiline JSON, direct JSON

Location 2: utils/ai_processor.py::_extract_json_from_response() (lines 342-449)

  • MORE COMPREHENSIVE - handles more edge cases, balanced brace matching

Impact: Two implementations with different capabilities. ai_processor.py version is more robust.

7.3 Duplicate OpenAI API Calling Logic

Location 1: ai/ai_core.py::run_ai_request() (lines 106-389)

  • Centralized method with console logging via ConsoleStepTracker
  • Handles validation, model selection, cost calculation
  • Returns standardized dict format

Location 2: utils/ai_processor.py::_call_openai() (lines 125-340)

  • SIMILAR LOGIC but with response_steps parameter instead of tracker
  • Less comprehensive error handling
  • Different return format

Impact: Two code paths for the same operation. New code should use AICore.run_ai_request().

7.4 Duplicate Image Generation Logic

Location 1: ai/ai_core.py::generate_image() + _generate_image_openai() + _generate_image_runware() (lines 431-728)

  • Uses print() statements for logging (inconsistent with console tracker)

Location 2: utils/ai_processor.py::generate_image() (lines 667-1043)

  • MORE COMPREHENSIVE - extensive logging, better error handling
  • Handles Runware authentication flow

Impact: Two implementations. ai_processor.py version has better logging.

7.5 Duplicate Prompt Loading Logic

Location 1: ai/prompts.py::PromptRegistry.get_prompt() (lines 280-333)

  • Hierarchical resolution: task override → DB prompt → default
  • Supports [IGNY8_*] placeholders and {variable} format

Location 2: utils/ai_processor.py::get_prompt() (lines 1044-1057)

  • Simple database lookup via modules/system/utils.get_prompt_value()
  • No hierarchical resolution

Location 3: Direct calls in modules/writer/tasks.py (lines 343, 959, 964)

  • Uses get_prompt_value() and get_default_prompt() directly

Impact: Three different ways to load prompts. New code should use PromptRegistry.

7.6 Duplicate Model Configuration Logic

Location 1: ai/settings.py::get_model_config() (lines 49-97)

  • Reads from IntegrationSettings if account provided
  • Falls back to MODEL_CONFIG defaults

Location 2: ai/ai_core.py::_load_account_settings() (lines 46-90)

  • Reads model from IntegrationSettings directly
  • Similar logic but embedded in AICore.__init__()

Location 3: utils/ai_processor.py::_get_model() (lines 98-123)

  • Reads model from IntegrationSettings directly
  • Similar logic but embedded in AIProcessor.__init__()

Impact: Model loading logic duplicated in three places.

7.7 Duplicate API Key Loading Logic

Location 1: ai/ai_core.py::_load_account_settings() (lines 46-90)

  • Loads OpenAI and Runware keys from IntegrationSettings

Location 2: utils/ai_processor.py::_get_api_key() (lines 73-96)

  • EXACT SAME LOGIC for loading API keys

Impact: Identical code in two places.

7.8 Repeated Error Handling Patterns

Pattern: Multiple files have similar try/except blocks for:

  • API request errors
  • JSON parsing errors
  • Database errors
  • Validation errors

Impact: Error handling is not centralized, making it harder to maintain consistent error messages and logging.

7.9 Repeated Progress Update Patterns

Pattern: Multiple places manually build progress update dicts:

  • modules/writer/tasks.py (lines 62-73, 220-231, etc.)
  • modules/planner/tasks.py (lines 59-71, 203-215, etc.)
  • ai/engine.py (lines 57, 79, 141, etc.)

Impact: Progress update format is not standardized, though ProgressTracker exists to handle this.

8. Summary of Potential Consolidation Areas

8.1 Constants Consolidation

Observation: Model rates, valid models, and configuration constants are duplicated between ai/constants.py and utils/ai_processor.py.

Potential Action: Remove constants from ai_processor.py and import from constants.py. However, ai_processor.py is marked as legacy, so this may not be necessary if it's being phased out.

8.2 JSON Extraction Consolidation

Observation: Two JSON extraction methods exist with different capabilities. ai_processor.py::_extract_json_from_response() is more comprehensive.

Potential Action: Enhance AICore.extract_json() with logic from ai_processor.py, or create a shared utility function.

8.3 API Request Consolidation

Observation: AICore.run_ai_request() and AIProcessor._call_openai() perform the same operation with different interfaces.

Potential Action: All new code should use AICore.run_ai_request(). Legacy code in ai_processor.py can remain for backward compatibility but should be marked as deprecated.

8.4 Image Generation Consolidation

Observation: Two image generation implementations exist. ai_processor.py version has better logging.

Potential Action: Enhance AICore.generate_image() with logging improvements from ai_processor.py, or migrate all code to use AICore.generate_image().

8.5 Prompt Loading Consolidation

Observation: Three different methods exist for loading prompts: PromptRegistry.get_prompt(), AIProcessor.get_prompt(), and direct get_prompt_value() calls.

Potential Action: Migrate all code to use PromptRegistry.get_prompt() for consistency. Update legacy code paths gradually.

8.6 Model/API Key Loading Consolidation

Observation: Model and API key loading logic is duplicated in AICore, AIProcessor, and settings.py.

Potential Action: Create shared utility functions for loading settings from IntegrationSettings, used by all classes.

8.7 Error Handling Consolidation

Observation: Error handling patterns are repeated across multiple files.

Potential Action: Create centralized error handling utilities or enhance AIEngine._handle_error() to be more reusable.

8.8 Progress Tracking Consolidation

Observation: Some code manually builds progress update dicts instead of using ProgressTracker.

Potential Action: Migrate all progress updates to use ProgressTracker.update() for consistency.

8.9 Legacy Code Path Elimination

Observation: Multiple execution paths exist:

  • New: run_ai_task → AIEngine → BaseAIFunction implementations
  • Legacy: Direct AIProcessor calls in modules/*/tasks.py

Potential Action: Gradually migrate all legacy tasks to use the new framework. Mark legacy code as deprecated.

9. Assumptions Made

  1. File wordpress.py was not read - assumed to be unrelated to AI processing based on name.

  2. Frontend code was partially analyzed via search results - full frontend audit not performed.

  3. Database migrations were not analyzed - assumed to be standard Django migrations.

  4. Test files were not analyzed - ai/tests/test_run.py exists but was not read.

  5. Settings file (backend/igny8_core/settings.py) was not read - assumed to contain standard Django settings.

  6. System module utilities (modules/system/utils.py) were referenced but not fully read - assumed to contain get_prompt_value() and get_default_prompt() functions.

10. Appendix

10.1 File Line Counts

Directory Files Total Lines
ai/ 15 ~2,500
ai/functions/ 5 ~1,300
utils/ (AI-related) 3 ~1,800
modules/planner/tasks.py 1 736
modules/writer/tasks.py 1 1,156
Total 25 ~7,500

10.2 Function Count Summary

  • Core Framework Functions: ~30
  • AI Function Implementations: 4 classes × ~6 methods = ~24 methods
  • Tracking Functions: ~20
  • Legacy Functions: ~15
  • Celery Tasks: 4
  • Total: ~90 functions/methods

10.3 Key Design Patterns

  1. Template Method Pattern: BaseAIFunction defines algorithm skeleton, subclasses implement steps
  2. Registry Pattern: FunctionRegistry for dynamic function discovery
  3. Factory Pattern: get_function_instance() creates function instances
  4. Strategy Pattern: Different AI functions implement same interface
  5. Observer Pattern: ProgressTracker updates Celery task state
  6. Facade Pattern: AICore provides simplified interface to OpenAI/Runware APIs

End of Report

Reference in New Issue View Git Blame Copy Permalink