igny8/docs/ActiveDocs/04-AI-FUNCTIONS.md

IGNY8 AI Functions Documentation

Version: 1.0
Last Updated: 2025-01-XX
Purpose: Complete documentation of all AI functions in the IGNY8 system, their workflows, dependencies, and integration points.

---

Table of Contents

1. AI Functions Overview
2. AIProcessor Class
3. Planner Module AI Functions
4. Writer Module AI Functions
5. System Module AI Functions
6. Progress Tracking
7. Error Handling
8. Dependencies

---

AI Functions Overview

The IGNY8 system includes 8 primary AI functions across 3 modules:

1. Planner Module (2 functions):

   • Auto Cluster Keywords (auto_cluster)
   • Auto Generate Ideas (auto_generate_ideas)

2. Writer Module (3 functions):

   • Auto Generate Content (auto_generate_content)
   • Auto Generate Images (auto_generate_images)
   • Extract Image Prompts (extract_image_prompts)

3. System Module (3 functions):

   • Test OpenAI Integration (test_openai)
   • Test Runware Integration (test_runware)
   • Generate Image (Test) (generate_image)

All AI functions use the unified AIProcessor class as the core interface for AI operations.

---

AIProcessor Class

Location: backend/igny8_core/utils/ai_processor.py

Purpose: Unified AI interface for all AI operations.

Initialization

Method: AIProcessor.__init__()

Parameters:

• account (optional): Account object for loading account-specific settings

Dependencies:

• IntegrationSettings model (for API keys and model configuration)
• Django settings (fallback for API keys)

Initializes:

• self.openai_api_key: Loaded from IntegrationSettings or Django settings
• self.runware_api_key: Loaded from IntegrationSettings or Django settings
• self.default_model: Loaded from IntegrationSettings or Django settings
• self.model_rates: Model pricing rates (per 1M tokens)
• self.image_model_rates: Image model pricing rates (per image)

Core Methods

cluster_keywords()

Purpose: Cluster keywords using AI-based semantic similarity.

Parameters:

• keywords: List of keyword dictionaries
• sector_name: Optional sector name for context
• account: Optional account object
• response_steps: Optional list for step tracking
• progress_callback: Optional callback function

Dependencies:

• AIPrompt model (for clustering prompt template)
• get_default_prompt() utility function
• _call_openai() method
• _extract_json_from_response() method

Returns: Dictionary with clusters array and response_steps

generate_ideas()

Purpose: Generate content ideas for clusters using AI.

Parameters:

• clusters: List of cluster dictionaries
• account: Optional account object

Dependencies:

• AIPrompt model (for ideas prompt template)
• get_default_prompt() utility function
• _call_openai() method
• _extract_json_from_response() method

Returns: Dictionary with ideas array

generate_content()

Purpose: Generate text content using OpenAI GPT models.

Parameters:

• prompt: The prompt text
• model: Optional model name (defaults to self.default_model)
• max_tokens: Maximum tokens to generate
• temperature: Sampling temperature

Dependencies:

• _call_openai() method

Returns: Dictionary with content, tokens_used, model, cost, error

extract_image_prompts()

Purpose: Extract image prompts from content using AI.

Parameters:

• content: Article content text
• title: Article title
• max_images: Maximum number of images to extract prompts for
• account: Optional account object

Dependencies:

• AIPrompt model (for image prompt extraction template)
• get_default_prompt() utility function
• _call_openai() method
• _extract_json_from_response() method

Returns: Dictionary with featured_prompt and in_article_prompts array

generate_image()

Purpose: Generate images using OpenAI DALL-E or Runware.

Parameters:

• prompt: Image generation prompt
• model: Image model name (dall-e-3, dall-e-2, or runware model)
• size: Image size (e.g., "1024x1024")
• n: Number of images to generate
• account: Optional account object

Dependencies:

• IntegrationSettings model (for API keys)
• OpenAI API or Runware API

Returns: Dictionary with image_url, revised_prompt, cost, provider, model

Internal Methods

_call_openai()

Purpose: Make HTTP request to OpenAI API.

Parameters:

• prompt: The prompt text
• model: Model name
• max_tokens: Maximum tokens
• temperature: Sampling temperature
• response_format: Optional JSON response format
• response_steps: Optional list for step tracking

Dependencies:

• requests library
• OpenAI API endpoint

Returns: Dictionary with content, input_tokens, output_tokens, total_tokens, model, cost, error, api_id

_extract_json_from_response()

Purpose: Extract JSON from OpenAI response (handles various formats).

Parameters:

• response_text: Raw response text from OpenAI

Dependencies:

• json library
• re library (for regex patterns)

Returns: Parsed JSON dictionary or None

_get_api_key()

Purpose: Get API key from IntegrationSettings or Django settings.

Parameters:

• integration_type: Type of integration ('openai' or 'runware')
• account: Optional account object

Dependencies:

• IntegrationSettings model
• Django settings

Returns: API key string or None

_get_model()

Purpose: Get default model from IntegrationSettings or Django settings.

Parameters:

• integration_type: Type of integration ('openai')
• account: Optional account object

Dependencies:

• IntegrationSettings model
• Django settings

Returns: Model name string or default

get_prompt()

Purpose: Get prompt template from AIPrompt model or default.

Parameters:

• prompt_type: Type of prompt ('clustering', 'ideas', 'content_generation', etc.)
• account: Optional account object

Dependencies:

• AIPrompt model
• get_default_prompt() utility function

Returns: Prompt template string

---

Planner Module AI Functions

1. Auto Cluster Keywords

Function Name: auto_cluster

API Endpoint: POST /v1/planner/keywords/auto_cluster/

ViewSet Action: KeywordViewSet.auto_cluster()

Celery Task: auto_cluster_keywords_task()

Core Function: _auto_cluster_keywords_core()

AIProcessor Method: AIProcessor.cluster_keywords()

Purpose: Automatically group related keywords into semantic clusters using AI.

Complete Workflow (16 Steps)

Request Phase (Steps 1-7):

1. API Endpoint Validation: Validate keyword IDs array
2. Account/Site Extraction: Extract account and site from request
3. Task Queuing: Queue Celery task
4. Keyword Loading & Validation: Load keywords from database
5. Relationship Validation: Validate account, site, sector relationships
6. AIProcessor Creation: Create AIProcessor instance
7. AI Call Preparation: Format keywords for AI processing

AI Processing Phase (Steps 8-12): 8. HTTP Response Validation: Validate OpenAI API response 9. Content Extraction: Extract content from response 10. Cost Calculation: Calculate API cost 11. JSON Extraction & Parsing: Extract and parse JSON from response 12. Cluster Data Validation: Validate cluster data structure

Database Phase (Steps 13-16): 13. Database Transaction Start: Begin atomic transaction 14. Cluster Creation/Update: Create or update clusters 15. Keyword Matching & Assignment: Match keywords to clusters 16. Metrics Recalculation & Commit: Recalculate cluster metrics and commit

Dependencies

Models:

• Keywords model
• Clusters model
• Account model
• Site model
• Sector model
• IntegrationSettings model (for OpenAI API key)
• AIPrompt model (for clustering prompt template)
• Plan model (for plan limits)

Functions:

• KeywordViewSet.auto_cluster() (API endpoint)
• auto_cluster_keywords_task() (Celery task)
• _auto_cluster_keywords_core() (core function)
• AIProcessor.cluster_keywords() (AI processing)

Files:

• backend/igny8_core/modules/planner/views.py
• backend/igny8_core/modules/planner/tasks.py
• backend/igny8_core/utils/ai_processor.py

2. Auto Generate Ideas

Function Name: auto_generate_ideas

API Endpoint: POST /v1/planner/clusters/auto_generate_ideas/

ViewSet Action: ClusterViewSet.auto_generate_ideas()

Celery Task: auto_generate_ideas_task()

Core Function: _generate_single_idea_core()

AIProcessor Method: AIProcessor.generate_ideas()

Purpose: Generate content ideas for keyword clusters using AI.

Workflow

1. API Endpoint: Validate cluster IDs, extract account
2. Task Queuing: Queue Celery task
3. Cluster Loading: Load clusters from database
4. AI Processing: Call AIProcessor.generate_ideas()
5. Idea Creation: Create ContentIdeas records

Dependencies

Models:

• Clusters model
• ContentIdeas model
• Keywords model
• Account model
• Site model
• Sector model
• IntegrationSettings model (for OpenAI API key)
• AIPrompt model (for ideas prompt template)
• Plan model (for plan limits)

Functions:

• ClusterViewSet.auto_generate_ideas() (API endpoint)
• auto_generate_ideas_task() (Celery task)
• _generate_single_idea_core() (core function)
• AIProcessor.generate_ideas() (AI processing)

Files:

• backend/igny8_core/modules/planner/views.py
• backend/igny8_core/modules/planner/tasks.py
• backend/igny8_core/utils/ai_processor.py

---

Writer Module AI Functions

3. Auto Generate Content

Function Name: auto_generate_content

API Endpoint: POST /v1/writer/tasks/auto_generate_content/

ViewSet Action: TasksViewSet.auto_generate_content()

Celery Task: auto_generate_content_task()

AIProcessor Method: AIProcessor.generate_content()

Purpose: Generate complete blog post/article content using AI.

Workflow

1. API Endpoint: Validate task IDs, extract account
2. Task Queuing: Queue Celery task
3. Task Loading: Load tasks with relationships
4. Content Generation: Call AIProcessor.generate_content() for each task
5. Content Saving: Save generated content to Content model

Dependencies

Models:

• Tasks model
• Content model
• ContentIdeas model
• Clusters model
• Keywords model
• Account model
• Site model
• Sector model
• IntegrationSettings model (for OpenAI API key)
• AIPrompt model (for content generation prompt template)
• AuthorProfile model (for writing style)
• Plan model (for plan limits)

Functions:

• TasksViewSet.auto_generate_content() (API endpoint)
• auto_generate_content_task() (Celery task)
• AIProcessor.generate_content() (AI processing)

Files:

• backend/igny8_core/modules/writer/views.py
• backend/igny8_core/modules/writer/tasks.py
• backend/igny8_core/utils/ai_processor.py

4. Auto Generate Images

Function Name: auto_generate_images

API Endpoint: POST /v1/writer/tasks/auto_generate_images/

ViewSet Action: TasksViewSet.auto_generate_images()

Celery Task: auto_generate_images_task()

AIProcessor Methods: AIProcessor.extract_image_prompts(), AIProcessor.generate_image()

Purpose: Generate images for content tasks using AI (OpenAI DALL-E or Runware).

Workflow

1. API Endpoint: Validate task IDs, extract account
2. Task Queuing: Queue Celery task
3. Task Loading: Load tasks with content
4. Image Prompt Extraction: Call AIProcessor.extract_image_prompts()
5. Image Generation: Call AIProcessor.generate_image() for each prompt
6. Image Saving: Save generated images to Images model

Dependencies

Models:

• Tasks model
• Images model
• Content model
• Account model
• IntegrationSettings model (for OpenAI/Runware API keys)
• AIPrompt model (for image prompt templates)
• Plan model (for plan limits and image model choices)

Functions:

• TasksViewSet.auto_generate_images() (API endpoint)
• auto_generate_images_task() (Celery task)
• AIProcessor.extract_image_prompts() (prompt extraction)
• AIProcessor.generate_image() (image generation)

Files:

• backend/igny8_core/modules/writer/views.py
• backend/igny8_core/modules/writer/tasks.py
• backend/igny8_core/utils/ai_processor.py

5. Extract Image Prompts

Function Name: extract_image_prompts

AIProcessor Method: AIProcessor.extract_image_prompts()

Purpose: Extract image prompts from generated content for image generation.

Note: This is typically called internally by auto_generate_images, but can be called directly.

Dependencies

Models:

• Content model
• AIPrompt model (for image prompt extraction template)
• IntegrationSettings model (for OpenAI API key)

Functions:

• AIProcessor.extract_image_prompts() (AI processing)

Files:

• backend/igny8_core/utils/ai_processor.py

---

System Module AI Functions

6. Test OpenAI Integration

Function Name: test_openai

API Endpoint: POST /v1/system/settings/integrations/{id}/test_openai/

ViewSet Action: IntegrationSettingsViewSet.test_openai()

Purpose: Test OpenAI API connection and validate API key.

Workflow

1. Get API Key: From request or saved IntegrationSettings
2. Test Connection: Call OpenAI API (/v1/models or /v1/chat/completions)
3. Return Result: Success or error message

Dependencies

Models:

• IntegrationSettings model
• Account model

Functions:

• IntegrationSettingsViewSet.test_openai() (API endpoint)
• AIProcessor class (for API key loading)

Files:

• backend/igny8_core/modules/system/integration_views.py
• backend/igny8_core/utils/ai_processor.py

7. Test Runware Integration

Function Name: test_runware

API Endpoint: POST /v1/system/settings/integrations/{id}/test_runware/

ViewSet Action: IntegrationSettingsViewSet.test_runware()

Purpose: Test Runware API connection and validate API key.

Workflow

1. Get API Key: From request or saved IntegrationSettings
2. Test Connection: Call Runware API (/v1)
3. Return Result: Success or error message

Dependencies

Models:

• IntegrationSettings model
• Account model

Functions:

• IntegrationSettingsViewSet.test_runware() (API endpoint)
• AIProcessor class (for API key loading)

Files:

• backend/igny8_core/modules/system/integration_views.py
• backend/igny8_core/utils/ai_processor.py

8. Generate Image (Test)

Function Name: generate_image

API Endpoint: POST /v1/system/settings/integrations/{id}/generate_image/

ViewSet Action: IntegrationSettingsViewSet.generate_image()

Purpose: Test image generation with OpenAI DALL-E or Runware.

Workflow

1. Get Configuration: From request or saved IntegrationSettings
2. Generate Image: Call AIProcessor.generate_image()
3. Return Result: Image URL or error message

Dependencies

Models:

• IntegrationSettings model
• Account model

Functions:

• IntegrationSettingsViewSet.generate_image() (API endpoint)
• AIProcessor.generate_image() (image generation)

Files:

• backend/igny8_core/modules/system/integration_views.py
• backend/igny8_core/utils/ai_processor.py

---

Progress Tracking

Task Progress Endpoint

Endpoint: GET /v1/system/settings/task_progress/{task_id}/

ViewSet Action: IntegrationSettingsViewSet.task_progress()

Purpose: Get Celery task status and progress.

Response Format:

{
  "state": "PROGRESS",
  "meta": {
    "current": 2,
    "total": 10,
    "percentage": 20,
    "message": "Processing...",
    "request_steps": [...],
    "response_steps": [...]
  }
}

States:

• PENDING: Task is queued but not started
• PROGRESS: Task is in progress
• SUCCESS: Task completed successfully
• FAILURE: Task failed with error

Frontend Integration

Hook: useProgressModal

Functionality:

• Polls task progress endpoint every 2 seconds
• Displays progress percentage and message
• Shows step-by-step logs (request_steps and response_steps)
• Auto-closes on completion

Store: aiRequestLogsStore

Functionality:

• Stores AI request/response logs
• Tracks step-by-step execution
• Displays logs in UI

---

Error Handling

Error Types

1. API Errors: OpenAI/Runware API errors
2. Validation Errors: Input validation failures
3. Database Errors: Database operation failures
4. Connection Errors: Celery/Redis connection issues

Error Propagation

1. API Endpoint: Catches errors, returns error response with request_steps
2. Celery Task: Catches errors, updates task state to FAILURE with error details
3. Core Function: Catches errors, logs to request_steps/response_steps
4. AIProcessor: Catches errors, returns error in result dictionary

Error Logging

• All errors logged to request_steps or response_steps
• Error details include: error type, message, function name, step number
• Frontend displays errors in AI Request/Response Logs section

---

Dependencies

Database Models

• Account: Account isolation
• Site, Sector: Site/sector hierarchy
• Keywords, Clusters, ContentIdeas: Planner data
• Tasks, Content, Images: Writer data
• IntegrationSettings: API keys and configuration
• AIPrompt: Prompt templates
• Plan: Plan limits

External Services

• OpenAI API: GPT models for text generation
• OpenAI DALL-E: Image generation
• Runware API: Alternative image generation

Utilities

• get_default_prompt(): Default prompt templates
• AIProcessor: Unified AI interface
• Celery: Asynchronous task processing
• Redis: Celery broker

---

Summary

The IGNY8 AI functions system provides:

1. Unified Interface: Single AIProcessor class for all AI operations
2. Account Isolation: All AI functions respect account boundaries
3. Progress Tracking: Real-time progress updates with step-by-step logs
4. Error Handling: Comprehensive error logging and propagation
5. Flexible Configuration: Account-specific API keys and prompts
6. Asynchronous Processing: Celery tasks for long-running operations
7. Multiple Providers: Support for OpenAI and Runware
8. Complete Workflows: End-to-end workflows from API to database

All AI functions integrate seamlessly with the account/site/user/plan architecture and provide detailed logging for debugging and monitoring.