129 lines
18 KiB
Markdown
129 lines
18 KiB
Markdown
# AI-Related Elements Across Codebase
|
|
|
|
Complete table of all AI-related logic, configs, prompts, validations, retries, logging, response parsing, etc.
|
|
|
|
| Type | Name / Key | File | Function / Class | Line No | Notes |
|
|
|------|------------|------|------------------|---------|-------|
|
|
| **🧠 AI Core Functions** |
|
|
| AI Core | `_auto_cluster_keywords_core` | `backend/igny8_core/modules/planner/tasks.py` | `_auto_cluster_keywords_core()` | 26 | Core clustering logic (legacy, used by old tasks) |
|
|
| AI Core | `_generate_single_idea_core` | `backend/igny8_core/modules/planner/tasks.py` | `_generate_single_idea_core()` | 1047 | Core idea generation logic (legacy) |
|
|
| AI Core | `auto_generate_content_task` | `backend/igny8_core/modules/writer/tasks.py` | `auto_generate_content_task()` | 27 | Celery task for content generation (legacy) |
|
|
| AI Core | `AutoClusterFunction` | `backend/igny8_core/ai/functions/auto_cluster.py` | `AutoClusterFunction` class | 14 | New framework function for clustering |
|
|
| AI Core | `cluster_keywords` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor.cluster_keywords()` | 1080 | Legacy clustering method |
|
|
| AI Core | `generate_ideas` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor.generate_ideas()` | 1280 | Legacy idea generation method |
|
|
| AI Core | `generate_content` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor.generate_content()` | 446 | Legacy content generation method |
|
|
| AI Core | `generate_image` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor.generate_image()` | 656 | Image generation (OpenAI DALL-E / Runware) |
|
|
| AI Core | `generate_image` | `backend/igny8_core/ai/processor.py` | `AIProcessor.generate_image()` | 61 | Framework wrapper for image generation |
|
|
| AI Core | `run_ai_task` | `backend/igny8_core/ai/tasks.py` | `run_ai_task()` | 13 | Unified Celery entrypoint for all AI functions |
|
|
| AI Core | `AIEngine.execute` | `backend/igny8_core/ai/engine.py` | `AIEngine.execute()` | 26 | Central orchestrator for all AI functions |
|
|
| **🔁 Retry / Model Fallback Logic** |
|
|
| Retry | `_call_openai` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor._call_openai()` | 125 | Main OpenAI API call method with error handling |
|
|
| Retry | `_get_api_key` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor._get_api_key()` | 73 | Gets API key from IntegrationSettings or Django settings (fallback) |
|
|
| Retry | `_get_model` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor._get_model()` | 98 | Gets model from IntegrationSettings or Django settings (fallback) |
|
|
| Retry | `max_retries=3` | `backend/igny8_core/ai/tasks.py` | `@shared_task(bind=True, max_retries=3)` | 12 | Celery task retry configuration |
|
|
| Retry | `max_retries=3` | `backend/igny8_core/modules/writer/tasks.py` | `@shared_task(bind=True, max_retries=3)` | 26 | Content generation task retry configuration |
|
|
| Retry | Error handling | `backend/igny8_core/utils/ai_processor.py` | `_call_openai()` | 191-305 | HTTP error handling, JSON parsing errors, timeout handling |
|
|
| Retry | Fallback to Django settings | `backend/igny8_core/utils/ai_processor.py` | `_get_api_key()` | 91-95 | Falls back to `OPENAI_API_KEY` or `RUNWARE_API_KEY` from settings |
|
|
| Retry | Fallback to default model | `backend/igny8_core/utils/ai_processor.py` | `_get_model()` | 120-123 | Falls back to `DEFAULT_AI_MODEL` from settings (default: 'gpt-4.1') |
|
|
| **🧱 Prompt Sources** |
|
|
| Prompt | `clustering` | `backend/igny8_core/modules/system/utils.py` | `get_default_prompt('clustering')` | 10-30 | Hardcoded default clustering prompt template |
|
|
| Prompt | `ideas` | `backend/igny8_core/modules/system/utils.py` | `get_default_prompt('ideas')` | 32-55 | Hardcoded default ideas generation prompt template |
|
|
| Prompt | `content_generation` | `backend/igny8_core/modules/system/utils.py` | `get_default_prompt('content_generation')` | 57-75 | Hardcoded default content generation prompt template |
|
|
| Prompt | `image_prompt_extraction` | `backend/igny8_core/modules/system/utils.py` | `get_default_prompt('image_prompt_extraction')` | 77-98 | Hardcoded default image prompt extraction template |
|
|
| Prompt | `image_prompt_template` | `backend/igny8_core/modules/system/utils.py` | `get_default_prompt('image_prompt_template')` | 100 | Hardcoded default image prompt template |
|
|
| Prompt | `negative_prompt` | `backend/igny8_core/modules/system/utils.py` | `get_default_prompt('negative_prompt')` | 102 | Hardcoded default negative prompt |
|
|
| Prompt | `get_prompt_value` | `backend/igny8_core/modules/system/utils.py` | `get_prompt_value()` | 108 | Retrieves prompt from DB (AIPrompt model) or returns default |
|
|
| Prompt | `AIPrompt` model | `backend/igny8_core/modules/system/models.py` | `AIPrompt` class | 13 | Database model storing account-specific prompts |
|
|
| Prompt | `prompt_type` | `backend/igny8_core/modules/system/models.py` | `AIPrompt.prompt_type` | 25 | Choices: clustering, ideas, content_generation, image_prompt_extraction, image_prompt_template, negative_prompt |
|
|
| Prompt | `build_prompt` | `backend/igny8_core/ai/functions/auto_cluster.py` | `AutoClusterFunction.build_prompt()` | 117 | Builds clustering prompt using `get_prompt_value()` |
|
|
| Prompt | `[IGNY8_KEYWORDS]` | `backend/igny8_core/ai/functions/auto_cluster.py` | `build_prompt()` | 131 | Placeholder replaced with keyword list |
|
|
| Prompt | `[IGNY8_CLUSTERS]` | `backend/igny8_core/modules/system/utils.py` | `get_default_prompt('ideas')` | 34 | Placeholder for ideas prompt |
|
|
| Prompt | `[IGNY8_IDEA]` | `backend/igny8_core/modules/system/utils.py` | `get_default_prompt('content_generation')` | 60 | Placeholder for content generation prompt |
|
|
| **⚠️ Validation / Limits** |
|
|
| Validation | `validate` | `backend/igny8_core/ai/base.py` | `BaseAIFunction.validate()` | 34 | Base validation: checks for 'ids' array, max_items limit |
|
|
| Validation | `validate` | `backend/igny8_core/ai/functions/auto_cluster.py` | `AutoClusterFunction.validate()` | 37 | Custom validation: checks keywords exist, plan limits |
|
|
| Validation | `get_max_items` | `backend/igny8_core/ai/functions/auto_cluster.py` | `AutoClusterFunction.get_max_items()` | 34 | Returns 20 (max keywords per cluster operation) |
|
|
| Validation | `check_credits` | `backend/igny8_core/modules/billing/services.py` | `CreditService.check_credits()` | 16 | Checks if account has enough credits |
|
|
| Validation | Daily cluster limit | `backend/igny8_core/ai/functions/auto_cluster.py` | `validate()` | 59-71 | Checks `plan.daily_cluster_limit` against clusters created today |
|
|
| Validation | Max clusters limit | `backend/igny8_core/ai/functions/auto_cluster.py` | `validate()` | 73-79 | Checks `plan.max_clusters` against total clusters |
|
|
| Validation | `VALID_OPENAI_IMAGE_MODELS` | `backend/igny8_core/utils/ai_processor.py` | Constant | 34 | Set: {'dall-e-3', 'dall-e-2'} |
|
|
| Validation | `VALID_SIZES_BY_MODEL` | `backend/igny8_core/utils/ai_processor.py` | Constant | 41 | Dict mapping models to valid sizes |
|
|
| Validation | Model validation | `backend/igny8_core/utils/ai_processor.py` | `generate_image()` | 704-708 | Validates model is in VALID_OPENAI_IMAGE_MODELS |
|
|
| Validation | Size validation | `backend/igny8_core/utils/ai_processor.py` | `generate_image()` | 719-724 | Validates size is valid for selected model |
|
|
| Validation | Model rate validation | `backend/igny8_core/utils/ai_processor.py` | `_get_model()` | 112 | Validates model is in MODEL_RATES before using |
|
|
| **🪵 AI Debug Steps** |
|
|
| Debug Step | `addRequestStep` | `backend/igny8_core/ai/tracker.py` | `StepTracker.add_request_step()` | 21 | Adds request phase step (INIT, PREP, SAVE, DONE) |
|
|
| Debug Step | `addResponseStep` | `backend/igny8_core/ai/tracker.py` | `StepTracker.add_response_step()` | 45 | Adds response phase step (AI_CALL, PARSE) |
|
|
| Debug Step | `request_steps` | `backend/igny8_core/ai/tracker.py` | `StepTracker.request_steps` | 17 | List of request steps |
|
|
| Debug Step | `response_steps` | `backend/igny8_core/ai/tracker.py` | `StepTracker.response_steps` | 18 | List of response steps |
|
|
| Debug Step | `request_steps` | `backend/igny8_core/modules/planner/tasks.py` | `_auto_cluster_keywords_core()` | 37 | Legacy request steps tracking |
|
|
| Debug Step | `response_steps` | `backend/igny8_core/modules/planner/tasks.py` | `_auto_cluster_keywords_core()` | 38 | Legacy response steps tracking |
|
|
| Debug Step | `meta['request_steps']` | `backend/igny8_core/modules/system/integration_views.py` | `task_progress()` | 936-937 | Extracts request_steps from Celery task meta |
|
|
| Debug Step | `meta['response_steps']` | `backend/igny8_core/modules/system/integration_views.py` | `task_progress()` | 938-939 | Extracts response_steps from Celery task meta |
|
|
| Debug Step | `request_steps` | `backend/igny8_core/ai/models.py` | `AITaskLog.request_steps` | 31 | JSONField storing request steps in database |
|
|
| Debug Step | `response_steps` | `backend/igny8_core/ai/models.py` | `AITaskLog.response_steps` | 32 | JSONField storing response steps in database |
|
|
| Debug Step | `get_meta()` | `backend/igny8_core/ai/tracker.py` | `StepTracker.get_meta()` | 69 | Returns dict with request_steps and response_steps |
|
|
| Debug Step | Step injection | `backend/igny8_core/ai/engine.py` | `execute()` | 47, 60, 68, 125, 138, 174 | Adds steps at each phase (INIT, PREP, AI_CALL, PARSE, SAVE, DONE) |
|
|
| Debug Step | Step counter | `backend/igny8_core/ai/tracker.py` | `StepTracker.step_counter` | 19 | Auto-increments for sequential step numbers |
|
|
| **🧾 Model Config / Selection** |
|
|
| Model Config | `default_model` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor.__init__()` | 67 | Set from `_get_model()` or defaults to 'gpt-4.1' |
|
|
| Model Config | `_get_model` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor._get_model()` | 98 | Gets model from IntegrationSettings.config['model'] or Django settings |
|
|
| Model Config | `MODEL_RATES` | `backend/igny8_core/utils/ai_processor.py` | Constant | 19 | Dict: {'gpt-4.1': {...}, 'gpt-4o-mini': {...}, 'gpt-4o': {...}} |
|
|
| Model Config | `IMAGE_MODEL_RATES` | `backend/igny8_core/utils/ai_processor.py` | Constant | 26 | Dict: {'dall-e-3': 0.040, 'dall-e-2': 0.020, ...} |
|
|
| Model Config | `DEFAULT_AI_MODEL` | `backend/igny8_core/utils/ai_processor.py` | Django setting | 121 | Default: 'gpt-4.1' if not set |
|
|
| Model Config | `get_model` | `backend/igny8_core/ai/base.py` | `BaseAIFunction.get_model()` | 70 | Override to specify model (defaults to account's default) |
|
|
| Model Config | `model` parameter | `backend/igny8_core/ai/processor.py` | `call()` | 22 | Model can be passed to AIProcessor.call() |
|
|
| Model Config | `json_models` | `backend/igny8_core/ai/processor.py` | `call()` | 40 | List: ['gpt-4o', 'gpt-4o-mini', 'gpt-4-turbo-preview'] (supports JSON mode) |
|
|
| Model Config | `IntegrationSettings.config['model']` | `backend/igny8_core/modules/system/models.py` | `IntegrationSettings.config` | 56 | JSONField storing model name per account |
|
|
| Model Config | Model selection | `backend/igny8_core/ai/engine.py` | `execute()` | 65 | Gets model via `fn.get_model(self.account)` |
|
|
| **📦 Request/Response Structuring** |
|
|
| Request/Response | `response_format` | `backend/igny8_core/utils/ai_processor.py` | `_call_openai()` | 131 | Optional dict for JSON mode: {'type': 'json_object'} |
|
|
| Request/Response | `response_format` | `backend/igny8_core/ai/processor.py` | `call()` | 25 | Optional dict for JSON mode |
|
|
| Request/Response | JSON mode auto-enable | `backend/igny8_core/ai/processor.py` | `call()` | 41-42 | Auto-enables JSON mode for json_models if not specified |
|
|
| Request/Response | `_extract_json_from_response` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor._extract_json_from_response()` | 334 | Extracts JSON from response (handles markdown code blocks, multiline) |
|
|
| Request/Response | `extract_json` | `backend/igny8_core/ai/processor.py` | `AIProcessor.extract_json()` | 57 | Wrapper for `_extract_json_from_response()` |
|
|
| Request/Response | `parse_response` | `backend/igny8_core/ai/functions/auto_cluster.py` | `AutoClusterFunction.parse_response()` | 158 | Parses AI response into cluster data structure |
|
|
| Request/Response | `parse_response` | `backend/igny8_core/ai/base.py` | `BaseAIFunction.parse_response()` | 75 | Abstract method for parsing AI response |
|
|
| Request/Response | Request body format | `backend/igny8_core/utils/ai_processor.py` | `_call_openai()` | 171-183 | Format: {'model': str, 'messages': [...], 'temperature': float, 'max_tokens': int, 'response_format': dict} |
|
|
| Request/Response | Response format | `backend/igny8_core/utils/ai_processor.py` | `_call_openai()` | 215-329 | Returns: {'content': str, 'input_tokens': int, 'output_tokens': int, 'total_tokens': int, 'model': str, 'cost': float, 'error': str, 'api_id': str} |
|
|
| Request/Response | JSON parsing fallback | `backend/igny8_core/ai/functions/auto_cluster.py` | `parse_response()` | 173-176 | Falls back to `extract_json()` if direct JSON parse fails |
|
|
| **📍 Paths / Constants** |
|
|
| Path/Constant | `OPENAI_API_KEY` | `backend/igny8_core/utils/ai_processor.py` | Django setting | 93 | Fallback API key from Django settings |
|
|
| Path/Constant | `RUNWARE_API_KEY` | `backend/igny8_core/utils/ai_processor.py` | Django setting | 95 | Fallback API key from Django settings |
|
|
| Path/Constant | OpenAI API URL | `backend/igny8_core/utils/ai_processor.py` | `_call_openai()` | 163 | Hardcoded: 'https://api.openai.com/v1/chat/completions' |
|
|
| Path/Constant | OpenAI Images URL | `backend/igny8_core/utils/ai_processor.py` | `generate_image()` | 735 | Hardcoded: 'https://api.openai.com/v1/images/generations' |
|
|
| Path/Constant | Runware API URL | `backend/igny8_core/utils/ai_processor.py` | `generate_image()` | 844 | Hardcoded: 'https://api.runware.ai/v1' |
|
|
| Path/Constant | `IntegrationSettings.config['apiKey']` | `backend/igny8_core/modules/system/models.py` | `IntegrationSettings.config` | 56 | JSONField storing API key per account |
|
|
| Path/Constant | Prompt type choices | `backend/igny8_core/modules/system/models.py` | `AIPrompt.PROMPT_TYPE_CHOICES` | 16 | ['clustering', 'ideas', 'content_generation', 'image_prompt_extraction', 'image_prompt_template', 'negative_prompt'] |
|
|
| Path/Constant | Integration type choices | `backend/igny8_core/modules/system/models.py` | `IntegrationSettings.INTEGRATION_TYPE_CHOICES` | 48 | ['openai', 'runware', 'gsc', 'image_generation'] |
|
|
| Path/Constant | `get_function_instance` | `backend/igny8_core/ai/registry.py` | `get_function_instance()` | - | Gets AI function instance from registry |
|
|
| Path/Constant | `auto_cluster` | `backend/igny8_core/ai/registry.py` | `register_lazy_function()` | 69 | Registered function name for clustering |
|
|
| **💰 Cost Tracking** |
|
|
| Cost | `CostTracker` | `backend/igny8_core/ai/tracker.py` | `CostTracker` class | 193 | Tracks API costs and token usage |
|
|
| Cost | `record` | `backend/igny8_core/ai/tracker.py` | `CostTracker.record()` | 201 | Records cost, tokens, model for an operation |
|
|
| Cost | Cost calculation | `backend/igny8_core/utils/ai_processor.py` | `_call_openai()` | 277-295 | Calculates cost from MODEL_RATES based on input/output tokens |
|
|
| Cost | Image cost calculation | `backend/igny8_core/utils/ai_processor.py` | `generate_image()` | 790 | Calculates cost from IMAGE_MODEL_RATES * n images |
|
|
| Cost | Credit logging | `backend/igny8_core/ai/engine.py` | `execute()` | 142-171 | Logs credit usage to CreditUsageLog after successful save |
|
|
| Cost | `_calculate_credits_for_clustering` | `backend/igny8_core/ai/engine.py` | `_calculate_credits_for_clustering()` | 257 | Calculates credits used (from plan config or fallback formula) |
|
|
| **📊 Progress Tracking** |
|
|
| Progress | `ProgressTracker` | `backend/igny8_core/ai/tracker.py` | `ProgressTracker` class | 77 | Tracks progress updates for AI tasks |
|
|
| Progress | `update` | `backend/igny8_core/ai/tracker.py` | `ProgressTracker.update()` | 89 | Updates Celery task state with progress |
|
|
| Progress | Phase percentages | `backend/igny8_core/ai/engine.py` | `execute()` | 30-36 | INIT (0-10%), PREP (10-25%), AI_CALL (25-70%), PARSE (70-85%), SAVE (85-98%), DONE (98-100%) |
|
|
| Progress | `update_ai_progress` | `backend/igny8_core/ai/tracker.py` | `ProgressTracker.update_ai_progress()` | 184 | Callback for AI processor progress updates |
|
|
| **🗄️ Database Logging** |
|
|
| Database | `AITaskLog` | `backend/igny8_core/ai/models.py` | `AITaskLog` model | 8 | Unified logging table for all AI tasks |
|
|
| Database | `_log_to_database` | `backend/igny8_core/ai/engine.py` | `AIEngine._log_to_database()` | 220 | Logs task execution to AITaskLog table |
|
|
| Database | Task logging | `backend/igny8_core/ai/engine.py` | `execute()` | 178 | Called after successful execution to log to database |
|
|
| **🔄 Celery Integration** |
|
|
| Celery | `run_ai_task` | `backend/igny8_core/ai/tasks.py` | `run_ai_task()` | 13 | Unified Celery task entrypoint |
|
|
| Celery | Task state updates | `backend/igny8_core/ai/tracker.py` | `ProgressTracker.update()` | 124-131 | Updates Celery task state via `task.update_state()` |
|
|
| Celery | Meta injection | `backend/igny8_core/modules/system/integration_views.py` | `task_progress()` | 936-991 | Extracts request_steps/response_steps from task meta and returns to frontend |
|
|
| Celery | Error state | `backend/igny8_core/ai/tasks.py` | `run_ai_task()` | 68-84 | Updates task state to FAILURE on error |
|
|
| **🔧 Utility Functions** |
|
|
| Utility | `get_default_prompt` | `backend/igny8_core/modules/system/utils.py` | `get_default_prompt()` | 7 | Returns hardcoded default prompt by type |
|
|
| Utility | `get_prompt_value` | `backend/igny8_core/modules/system/utils.py` | `get_prompt_value()` | 108 | Gets prompt from DB or returns default |
|
|
| Utility | `check_moderation` | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor.check_moderation()` | 594 | Checks content against OpenAI moderation API |
|
|
| Utility | `prepare` | `backend/igny8_core/ai/functions/auto_cluster.py` | `AutoClusterFunction.prepare()` | 85 | Loads keywords with relationships |
|
|
| Utility | `save_output` | `backend/igny8_core/ai/functions/auto_cluster.py` | `AutoClusterFunction.save_output()` | 205 | Saves clusters to database with transaction |
|
|
|