salman/igny8
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

reference plugin and image gen analysis

Browse Source
This commit is contained in:
Desktop
2025-11-11 21:16:37 +05:00
parent fedf415646
commit f4d62448cf
111 changed files with 49595 additions and 3988 deletions
Download Patch File Download Diff File Expand all files Collapse all files

271
docs/AI-FUNCTION-FILES.md
View File

@@ -1,271 +0,0 @@
# AI Function Related Files
This document lists all files containing code related to:
- Auto Cluster Keywords
- Auto Generate Ideas
- Auto Generate Content
- Auto Generate Images
---
## Backend Files
### Auto Cluster Keywords
#### Core Implementation
- `backend/igny8_core/ai/functions/auto_cluster.py` - **Main AI function implementation** (BaseAIFunction)
- `backend/igny8_core/ai/base.py` - Base AI function class
- `backend/igny8_core/ai/engine.py` - AI engine orchestrator
- `backend/igny8_core/ai/processor.py` - AI processor wrapper
- `backend/igny8_core/ai/tasks.py` - Unified Celery task entrypoint
- `backend/igny8_core/ai/registry.py` - Function registry
- `backend/igny8_core/ai/tracker.py` - Progress and cost tracking
#### API Endpoints & Views
- `backend/igny8_core/modules/planner/views.py` - **KeywordViewSet.auto_cluster()** action
- `backend/igny8_core/modules/planner/urls.py` - URL routing
#### Celery Tasks (Legacy/Alternative)
- `backend/igny8_core/modules/planner/tasks.py` - **auto_cluster_keywords_task()** (legacy implementation)
#### AI Processor
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.cluster_keywords()** method
#### Models
- `backend/igny8_core/modules/planner/models.py` - Keywords, Clusters models
#### Serializers
- `backend/igny8_core/modules/planner/serializers.py` - Keyword, Cluster serializers
- `backend/igny8_core/modules/planner/cluster_serializers.py` - Cluster-specific serializers
#### System Integration
- `backend/igny8_core/modules/system/schemas.py` - Schema definitions
- `backend/igny8_core/modules/system/utils.py` - Prompt loading utilities
---
### Auto Generate Ideas
#### Core Implementation
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.generate_ideas()** method
#### API Endpoints & Views
- `backend/igny8_core/modules/planner/views.py` - **ClusterViewSet.auto_generate_ideas()** action
- `backend/igny8_core/modules/planner/urls.py` - URL routing
#### Celery Tasks
- `backend/igny8_core/modules/planner/tasks.py` - **auto_generate_ideas_task()** and **generate_single_idea_core()**
#### Models
- `backend/igny8_core/modules/planner/models.py` - Clusters, ContentIdeas models
#### Serializers
- `backend/igny8_core/modules/planner/serializers.py` - Cluster, ContentIdeas serializers
#### System Integration
- `backend/igny8_core/modules/system/utils.py` - Prompt loading utilities
---
### Auto Generate Content
#### Core Implementation
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.generate_content()** method
#### API Endpoints & Views
- `backend/igny8_core/modules/writer/views.py` - **TasksViewSet.auto_generate_content()** action
- `backend/igny8_core/modules/writer/urls.py` - URL routing
#### Celery Tasks
- `backend/igny8_core/modules/writer/tasks.py` - **auto_generate_content_task()**
#### Models
- `backend/igny8_core/modules/writer/models.py` - Tasks, Content models
#### Serializers
- `backend/igny8_core/modules/writer/serializers.py` - Task, Content serializers
#### System Integration
- `backend/igny8_core/modules/system/schemas.py` - Schema definitions
- `backend/igny8_core/modules/system/utils.py` - Prompt loading utilities
---
### Auto Generate Images
#### Core Implementation
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.extract_image_prompts()** and **AIProcessor.generate_image()** methods
#### API Endpoints & Views
- `backend/igny8_core/modules/writer/views.py` - **TasksViewSet.auto_generate_images()** action
- `backend/igny8_core/modules/writer/urls.py` - URL routing
#### Celery Tasks
- `backend/igny8_core/modules/writer/tasks.py` - **auto_generate_images_task()**
#### Models
- `backend/igny8_core/modules/writer/models.py` - Tasks, Images models
#### Serializers
- `backend/igny8_core/modules/writer/serializers.py` - Task, Images serializers
#### System Integration
- `backend/igny8_core/modules/system/schemas.py` - Schema definitions
- `backend/igny8_core/modules/system/integration_views.py` - Integration settings for image generation
---
## Frontend Files
### Auto Cluster Keywords
#### Pages
- `frontend/src/pages/Planner/Keywords.tsx` - **Main page component** with auto cluster functionality
#### API Services
- `frontend/src/services/api.ts` - **autoClusterKeywords()** function
#### Configuration
- `frontend/src/config/pages/keywords.config.tsx` - Page configuration
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
#### State Management
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
#### Hooks
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
---
### Auto Generate Ideas
#### Pages
- `frontend/src/pages/Planner/Clusters.tsx` - **Main page component** with auto generate ideas functionality
#### API Services
- `frontend/src/services/api.ts` - **autoGenerateIdeas()** function
#### Configuration
- `frontend/src/config/pages/clusters.config.tsx` - Page configuration
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
#### State Management
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
#### Hooks
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
---
### Auto Generate Content
#### Pages
- `frontend/src/pages/Writer/Tasks.tsx` - **Main page component** with auto generate content functionality
#### API Services
- `frontend/src/services/api.ts` - **autoGenerateContent()** function
#### Configuration
- `frontend/src/config/pages/tasks.config.tsx` - Page configuration
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
#### State Management
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
#### Hooks
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
---
### Auto Generate Images
#### Pages
- `frontend/src/pages/Writer/Tasks.tsx` - **Main page component** with auto generate images functionality
#### API Services
- `frontend/src/services/api.ts` - **autoGenerateImages()** function
#### Configuration
- `frontend/src/config/pages/tasks.config.tsx` - Page configuration
- `frontend/src/config/pages/images.config.tsx` - Image-related configuration
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
#### State Management
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
#### Hooks
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
---
## Shared/Common Files
### AI Framework (Backend)
- `backend/igny8_core/ai/__init__.py` - AI module initialization and auto-registration
- `backend/igny8_core/ai/models.py` - AITaskLog model for unified logging
- `backend/igny8_core/ai/types.py` - Shared dataclasses and types
- `backend/igny8_core/ai/admin.py` - Admin interface for AITaskLog
### Progress Tracking (Backend)
- `backend/igny8_core/modules/system/integration_views.py` - **task_progress()** endpoint
### Progress Tracking (Frontend)
- `frontend/src/components/common/ProgressModal.tsx` - Progress modal component
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook
### Common Components (Frontend)
- `frontend/src/components/common/FormModal.tsx` - Form modal used in AI function pages
---
## Summary by Function
### Auto Cluster Keywords
**Backend**: 15 files
**Frontend**: 5 files
**Total**: 20 files
### Auto Generate Ideas
**Backend**: 7 files
**Frontend**: 5 files
**Total**: 12 files
### Auto Generate Content
**Backend**: 7 files
**Frontend**: 5 files
**Total**: 12 files
### Auto Generate Images
**Backend**: 7 files
**Frontend**: 5 files
**Total**: 12 files
### Shared/Common
**Backend**: 4 files
**Frontend**: 3 files
**Total**: 7 files
---
## Key Files (Most Important)
### Backend Core Files
1. `backend/igny8_core/ai/functions/auto_cluster.py` - Auto cluster AI function
2. `backend/igny8_core/utils/ai_processor.py` - Unified AI processor (all functions)
3. `backend/igny8_core/modules/planner/views.py` - Auto cluster & ideas API endpoints
4. `backend/igny8_core/modules/writer/views.py` - Content & images API endpoints
5. `backend/igny8_core/modules/planner/tasks.py` - Planner Celery tasks
6. `backend/igny8_core/modules/writer/tasks.py` - Writer Celery tasks
7. `backend/igny8_core/ai/tasks.py` - Unified AI task entrypoint
### Frontend Core Files
1. `frontend/src/pages/Planner/Keywords.tsx` - Auto cluster UI
2. `frontend/src/pages/Planner/Clusters.tsx` - Auto generate ideas UI
3. `frontend/src/pages/Writer/Tasks.tsx` - Content & images generation UI
4. `frontend/src/services/api.ts` - All API functions
5. `frontend/src/hooks/useProgressModal.ts` - Progress tracking hook
---
**Last Updated**: 2025-01-XX

274
docs/AI-NEW-DESIGN/07-AI-FRAMEWORK.md
View File

@@ -1,274 +0,0 @@
# IGNY8 AI Framework Documentation
**Version:** 1.0
**Last Updated:** 2025-01-XX
**Purpose:** Complete documentation of the unified AI framework architecture.
---
## Overview
The IGNY8 AI Framework provides a unified, consistent architecture for all AI functions. It eliminates code duplication, standardizes progress tracking, and provides a single interface for all AI operations.
### Key Benefits
- **90% Code Reduction**: Functions are now ~100 lines instead of ~600
- **Consistent UX**: All functions use the same progress modal and tracking
- **Unified Logging**: Single `AITaskLog` table for all AI operations
- **Easy Extension**: Add new functions by creating one class
- **Better Debugging**: Detailed step-by-step tracking for all operations
---
## Architecture
### Directory Structure
```
igny8_core/ai/
├── __init__.py # Auto-registers all functions
├── apps.py # Django app configuration
├── admin.py # Admin interface for AITaskLog
├── base.py # BaseAIFunction abstract class
├── engine.py # AIEngine orchestrator
├── processor.py # AIProcessor wrapper
├── registry.py # Function registry
├── tracker.py # StepTracker, ProgressTracker, CostTracker
├── tasks.py # Unified Celery task entrypoint
├── types.py # Shared dataclasses
├── models.py # AITaskLog model
└── functions/ # Function implementations
├── __init__.py
└── auto_cluster.py # Auto cluster function
```
---
## Core Components
### 1. BaseAIFunction
Abstract base class that all AI functions inherit from.
**Methods to implement:**
- `get_name()`: Return function name
- `prepare()`: Load and prepare data
- `build_prompt()`: Build AI prompt
- `parse_response()`: Parse AI response
- `save_output()`: Save results to database
**Optional overrides:**
- `validate()`: Custom validation
- `get_max_items()`: Set item limit
- `get_model()`: Specify AI model
- `get_metadata()`: Function metadata
### 2. AIEngine
Central orchestrator that manages the execution pipeline.
**Phases:**
- INIT (0-10%): Validation & setup
- PREP (10-25%): Data loading & prompt building
- AI_CALL (25-60%): API call to provider
- PARSE (60-80%): Response parsing
- SAVE (80-95%): Database operations
- DONE (95-100%): Finalization
### 3. Function Registry
Dynamic function discovery system.
**Usage:**
```python
from igny8_core.ai.registry import register_function, get_function
# Register function
register_function('auto_cluster', AutoClusterFunction)
# Get function
fn = get_function('auto_cluster')
```
### 4. Unified Celery Task
Single entrypoint for all AI functions.
**Endpoint:** `run_ai_task(function_name, payload, account_id)`
**Example:**
```python
from igny8_core.ai.tasks import run_ai_task
task = run_ai_task.delay(
function_name='auto_cluster',
payload={'ids': [1, 2, 3], 'sector_id': 1},
account_id=1
)
```
---
## Function Implementation Example
### Auto Cluster Function
```python
from igny8_core.ai.base import BaseAIFunction
class AutoClusterFunction(BaseAIFunction):
def get_name(self) -> str:
return 'auto_cluster'
def get_max_items(self) -> int:
return 20
def prepare(self, payload: dict, account=None) -> Dict:
# Load keywords
ids = payload.get('ids', [])
keywords = Keywords.objects.filter(id__in=ids)
return {'keywords': keywords, ...}
def build_prompt(self, data: Dict, account=None) -> str:
# Build clustering prompt
return prompt_template.replace('[IGNY8_KEYWORDS]', keywords_text)
def parse_response(self, response: str, step_tracker=None) -> List[Dict]:
# Parse AI response
return clusters
def save_output(self, parsed, original_data, account, progress_tracker) -> Dict:
# Save clusters to database
return {'clusters_created': 5, 'keywords_updated': 20}
```
---
## API Endpoint Example
### Before (Old): ~300 lines
### After (New): ~50 lines
```python
@action(detail=False, methods=['post'], url_path='auto_cluster')
def auto_cluster(self, request):
from igny8_core.ai.tasks import run_ai_task
account = getattr(request, 'account', None)
account_id = account.id if account else None
payload = {
'ids': request.data.get('ids', []),
'sector_id': request.data.get('sector_id')
}
task = run_ai_task.delay(
function_name='auto_cluster',
payload=payload,
account_id=account_id
)
return Response({
'success': True,
'task_id': str(task.id),
'message': 'Clustering started'
})
```
---
## Progress Tracking
### Unified Progress Endpoint
**URL:** `/api/v1/system/settings/task_progress/<task_id>/`
**Response:**
```json
{
"state": "PROGRESS",
"meta": {
"phase": "AI_CALL",
"percentage": 45,
"message": "Analyzing keyword relationships...",
"request_steps": [...],
"response_steps": [...],
"cost": 0.000123,
"tokens": 1500
}
}
```
### Frontend Integration
All AI functions use the same progress modal:
- Single `useProgressModal` hook
- Unified progress endpoint
- Consistent phase labels
- Step-by-step logs
---
## Database Logging
### AITaskLog Model
Unified logging table for all AI operations.
**Fields:**
- `task_id`: Celery task ID
- `function_name`: Function name
- `account`: Account (required)
- `phase`: Current phase
- `status`: success/error/pending
- `cost`: API cost
- `tokens`: Token usage
- `request_steps`: Request step logs
- `response_steps`: Response step logs
- `error`: Error message (if any)
---
## Migration Guide
### Migrating Existing Functions
1. Create function class inheriting `BaseAIFunction`
2. Implement required methods
3. Register function in `ai/__init__.py`
4. Update API endpoint to use `run_ai_task`
5. Test and remove old code
### Example Migration
**Old code:**
```python
@action(...)
def auto_cluster(self, request):
# 300 lines of code
```
**New code:**
```python
@action(...)
def auto_cluster(self, request):
# 20 lines using framework
```
---
## Summary
The AI Framework provides:
1. **Unified Architecture**: Single framework for all AI functions
2. **Code Reduction**: 90% less code per function
3. **Consistent UX**: Same progress modal for all functions
4. **Better Debugging**: Detailed step tracking
5. **Easy Extension**: Add functions quickly
6. **Unified Logging**: Single log table
7. **Cost Tracking**: Automatic cost calculation
This architecture ensures maintainability, consistency, and extensibility while dramatically reducing code duplication.

191
docs/AI-NEW-DESIGN/STAGE1-REORGANIZATION-COMPLETE.md
View File

@@ -1,191 +0,0 @@
# Stage 1 - AI Folder Structure & Functional Split - COMPLETE ✅
## Summary
Successfully reorganized the AI backend into a clean, modular structure where every AI function lives inside its own file within `/ai/functions/`.
## ✅ Completed Deliverables
### 1. Folder Structure Created
```
backend/igny8_core/ai/
├── functions/
│ ├── __init__.py ✅
│ ├── auto_cluster.py ✅
│ ├── generate_ideas.py ✅
│ ├── generate_content.py ✅
│ └── generate_images.py ✅
├── ai_core.py ✅ (Shared operations)
├── validators.py ✅ (Consolidated validation)
├── constants.py ✅ (Model pricing, valid models)
├── engine.py ✅ (Updated to use AICore)
├── tracker.py ✅ (Existing)
├── base.py ✅ (Existing)
├── processor.py ✅ (Existing wrapper)
├── registry.py ✅ (Updated with new functions)
└── __init__.py ✅ (Updated exports)
```
### 2. Shared Modules Created
#### `ai_core.py`
- **Purpose**: Shared operations for all AI functions
- **Features**:
- API call construction (`call_openai`)
- Model selection (`get_model`, `get_api_key`)
- Response parsing (`extract_json`)
- Image generation (`generate_image`)
- Cost calculation (`calculate_cost`)
- **Status**: ✅ Complete
#### `validators.py`
- **Purpose**: Consolidated validation logic
- **Functions**:
- `validate_ids()` - Base ID validation
- `validate_keywords_exist()` - Keyword existence check
- `validate_cluster_limits()` - Plan limit checks
- `validate_cluster_exists()` - Cluster existence
- `validate_tasks_exist()` - Task existence
- `validate_api_key()` - API key validation
- `validate_model()` - Model validation
- `validate_image_size()` - Image size validation
- **Status**: ✅ Complete
#### `constants.py`
- **Purpose**: AI-related constants
- **Constants**:
- `MODEL_RATES` - Text model pricing
- `IMAGE_MODEL_RATES` - Image model pricing
- `VALID_OPENAI_IMAGE_MODELS` - Valid image models
- `VALID_SIZES_BY_MODEL` - Valid sizes per model
- `DEFAULT_AI_MODEL` - Default model name
- `JSON_MODE_MODELS` - Models supporting JSON mode
- **Status**: ✅ Complete
### 3. Function Files Created
#### `functions/auto_cluster.py`
- **Status**: ✅ Updated to use new validators and AICore
- **Changes**:
- Uses `validate_ids()`, `validate_keywords_exist()`, `validate_cluster_limits()` from validators
- Uses `AICore.extract_json()` for JSON parsing
- Maintains backward compatibility
#### `functions/generate_ideas.py`
- **Status**: ✅ Created
- **Features**:
- `GenerateIdeasFunction` class (BaseAIFunction)
- `generate_ideas_core()` legacy function for backward compatibility
- Uses AICore for API calls
- Uses validators for validation
#### `functions/generate_content.py`
- **Status**: ✅ Created
- **Features**:
- `GenerateContentFunction` class (BaseAIFunction)
- `generate_content_core()` legacy function for backward compatibility
- Uses AICore for API calls
- Uses validators for validation
#### `functions/generate_images.py`
- **Status**: ✅ Created
- **Features**:
- `GenerateImagesFunction` class (BaseAIFunction)
- `generate_images_core()` legacy function for backward compatibility
- Uses AICore for image generation
- Uses validators for validation
### 4. Import Paths Updated
#### Updated Files:
- ✅ `modules/planner/views.py` - Uses `generate_ideas_core` from new location
- ✅ `modules/planner/tasks.py` - Imports `generate_ideas_core` from new location
- ✅ `modules/writer/tasks.py` - Imports `generate_content_core` and `generate_images_core` from new locations
- ✅ `ai/engine.py` - Uses `AICore` instead of `AIProcessor`
- ✅ `ai/functions/auto_cluster.py` - Uses new validators and AICore
- ✅ `ai/registry.py` - Registered all new functions
- ✅ `ai/__init__.py` - Exports all new modules
### 5. Dependencies Verified
#### No Circular Dependencies ✅
- Functions depend on: `ai_core`, `validators`, `constants`, `base`
- `ai_core` depends on: `utils.ai_processor` (legacy, will be refactored later)
- `validators` depends on: `constants`, models
- `engine` depends on: `ai_core`, `base`, `tracker`
- All imports are clean and modular
#### Modular Structure ✅
- Each function file is self-contained
- Shared logic in `ai_core.py`
- Validation logic in `validators.py`
- Constants in `constants.py`
- No scattered or duplicated logic
## 📋 File Structure Details
### Core AI Modules
| File | Purpose | Dependencies |
|------|---------|--------------|
| `ai_core.py` | Shared AI operations | `utils.ai_processor` (legacy) |
| `validators.py` | All validation logic | `constants`, models |
| `constants.py` | AI constants | None |
| `engine.py` | Execution orchestrator | `ai_core`, `base`, `tracker` |
| `base.py` | Base function class | None |
| `tracker.py` | Progress/step tracking | None |
| `registry.py` | Function registry | `base`, function modules |
### Function Files
| File | Function Class | Legacy Function | Status |
|------|----------------|-----------------|--------|
| `auto_cluster.py` | `AutoClusterFunction` | N/A (uses engine) | ✅ Updated |
| `generate_ideas.py` | `GenerateIdeasFunction` | `generate_ideas_core()` | ✅ Created |
| `generate_content.py` | `GenerateContentFunction` | `generate_content_core()` | ✅ Created |
| `generate_images.py` | `GenerateImagesFunction` | `generate_images_core()` | ✅ Created |
## 🔄 Import Path Changes
### Old Imports (Still work, but deprecated)
```python
from igny8_core.utils.ai_processor import AIProcessor
from igny8_core.modules.planner.tasks import _generate_single_idea_core
```
### New Imports (Recommended)
```python
from igny8_core.ai.functions.generate_ideas import generate_ideas_core
from igny8_core.ai.functions.generate_content import generate_content_core
from igny8_core.ai.functions.generate_images import generate_images_core
from igny8_core.ai.ai_core import AICore
from igny8_core.ai.validators import validate_ids, validate_cluster_limits
from igny8_core.ai.constants import MODEL_RATES, DEFAULT_AI_MODEL
```
## ✅ Verification Checklist
- [x] All function files created in `ai/functions/`
- [x] Shared modules (`ai_core`, `validators`, `constants`) created
- [x] No circular dependencies
- [x] All imports updated in views and tasks
- [x] Functions registered in registry
- [x] `__init__.py` files updated
- [x] Backward compatibility maintained (legacy functions still work)
- [x] No linting errors
- [x] Structure matches required layout
## 🎯 Next Steps (Future Stages)
- **Stage 2**: Inject tracker into all functions
- **Stage 3**: Simplify logging
- **Stage 4**: Clean up legacy code
## 📝 Notes
- Legacy `AIProcessor` from `utils.ai_processor` is still used by `ai_core.py` as a wrapper
- This will be refactored in later stages
- All existing API endpoints continue to work
- No functional changes - only structural reorganization

220
docs/AI-NEW-DESIGN/STAGE2-EXECUTION-LOGGING-COMPLETE.md
View File

@@ -1,220 +0,0 @@
# Stage 2 - AI Execution & Logging Layer - COMPLETE ✅
## Summary
Successfully created a centralized, consistent, and traceable execution layer for all AI requests with unified request handler and clean console-based logging.
## ✅ Completed Deliverables
### 1. Centralized Execution in `ai_core.py`
#### `run_ai_request()` Method
- **Purpose**: Single entry point for all AI text generation requests
- **Features**:
- Step-by-step console logging with `print()` statements
- Standardized request payload construction
- Error handling with detailed logging
- Token counting and cost calculation
- Rate limit detection and logging
- Timeout handling
- JSON mode auto-enablement for supported models
#### Console Logging Format
```
[AI][function_name] Step 1: Preparing request...
[AI][function_name] Step 2: Using model: gpt-4o
[AI][function_name] Step 3: Auto-enabled JSON mode for gpt-4o
[AI][function_name] Step 4: Prompt length: 1234 characters
[AI][function_name] Step 5: Request payload prepared (model=gpt-4o, max_tokens=4000, temp=0.7)
[AI][function_name] Step 6: Sending request to OpenAI API...
[AI][function_name] Step 7: Received response in 2.34s (status=200)
[AI][function_name] Step 8: Received 150 tokens (input: 50, output: 100)
[AI][function_name] Step 9: Content length: 450 characters
[AI][function_name] Step 10: Cost calculated: $0.000123
[AI][function_name][Success] Request completed successfully
```
#### Error Logging Format
```
[AI][function_name][Error] OpenAI Rate Limit - waiting 60s
[AI][function_name][Error] HTTP 429 error: Rate limit exceeded (Rate limit - retry after 60s)
[AI][function_name][Error] Request timeout (60s exceeded)
[AI][function_name][Error] Failed to parse JSON response: ...
```
### 2. Image Generation with Logging
#### `generate_image()` Method
- **Purpose**: Centralized image generation with console logging
- **Features**:
- Supports OpenAI DALL-E and Runware
- Model and size validation
- Step-by-step console logging
- Error handling with detailed messages
- Cost calculation
#### Console Logging Format
```
[AI][generate_images] Step 1: Preparing image generation request...
[AI][generate_images] Provider: OpenAI
[AI][generate_images] Step 2: Using model: dall-e-3, size: 1024x1024
[AI][generate_images] Step 3: Sending request to OpenAI Images API...
[AI][generate_images] Step 4: Received response in 5.67s (status=200)
[AI][generate_images] Step 5: Image generated successfully
[AI][generate_images] Step 6: Cost: $0.0400
[AI][generate_images][Success] Image generation completed
```
### 3. Updated All Function Files
#### `functions/auto_cluster.py`
- ✅ Uses `AICore.extract_json()` for JSON parsing
- ✅ Engine calls `run_ai_request()` (via engine.py)
#### `functions/generate_ideas.py`
- ✅ Updated `generate_ideas_core()` to use `run_ai_request()`
- ✅ Console logging enabled with function name
#### `functions/generate_content.py`
- ✅ Updated `generate_content_core()` to use `run_ai_request()`
- ✅ Console logging enabled with function name
#### `functions/generate_images.py`
- ✅ Updated to use `run_ai_request()` for prompt extraction
- ✅ Updated to use `generate_image()` with logging
- ✅ Console logging enabled
### 4. Updated Engine
#### `engine.py`
- ✅ Updated to use `run_ai_request()` instead of `call_openai()`
- ✅ Passes function name for logging context
- ✅ Maintains backward compatibility
### 5. Deprecated Old Code
#### `processor.py`
- ✅ Marked as DEPRECATED
- ✅ Redirects all calls to `AICore`
- ✅ Kept for backward compatibility only
- ✅ All methods now use `AICore` internally
### 6. Edge Case Handling
#### Implemented in `run_ai_request()`:
- ✅ **API Key Validation**: Logs error if not configured
- ✅ **Prompt Length**: Logs character count
- ✅ **Rate Limits**: Detects and logs retry-after time
- ✅ **Timeouts**: Handles 60s timeout with clear error
- ✅ **JSON Parsing Errors**: Logs decode errors with context
- ✅ **Empty Responses**: Validates content exists
- ✅ **Token Overflow**: Max tokens enforced
- ✅ **Model Validation**: Auto-selects JSON mode for supported models
### 7. Standardized Request Schema
#### OpenAI Request Payload
```python
{
"model": "gpt-4o",
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.7,
"max_tokens": 4000,
"response_format": {"type": "json_object"} # Auto-enabled for supported models
}
```
#### All Functions Use Same Logic:
- Model selection (account default or override)
- JSON mode auto-enablement
- Token limits
- Temperature settings
- Error handling
### 8. Test Script Created
#### `ai/tests/test_run.py`
- ✅ Test script for all AI functions
- ✅ Tests `run_ai_request()` directly
- ✅ Tests JSON extraction
- ✅ Placeholder tests for all functions
- ✅ Can be run standalone to verify logging
## 📋 File Changes Summary
| File | Changes | Status |
|------|---------|--------|
| `ai_core.py` | Complete rewrite with `run_ai_request()` and console logging | ✅ Complete |
| `engine.py` | Updated to use `run_ai_request()` | ✅ Complete |
| `processor.py` | Marked deprecated, redirects to AICore | ✅ Complete |
| `functions/auto_cluster.py` | Uses AICore methods | ✅ Complete |
| `functions/generate_ideas.py` | Uses `run_ai_request()` | ✅ Complete |
| `functions/generate_content.py` | Uses `run_ai_request()` | ✅ Complete |
| `functions/generate_images.py` | Uses `run_ai_request()` and `generate_image()` | ✅ Complete |
| `tests/test_run.py` | Test script created | ✅ Complete |
## 🔄 Migration Path
### Old Code (Deprecated)
```python
from igny8_core.utils.ai_processor import AIProcessor
processor = AIProcessor(account=account)
result = processor._call_openai(prompt, model=model)
```
### New Code (Recommended)
```python
from igny8_core.ai.ai_core import AICore
ai_core = AICore(account=account)
result = ai_core.run_ai_request(
prompt=prompt,
model=model,
function_name='my_function'
)
```
## ✅ Verification Checklist
- [x] `run_ai_request()` created with console logging
- [x] All function files updated to use `run_ai_request()`
- [x] Engine updated to use `run_ai_request()`
- [x] Old processor code deprecated
- [x] Edge cases handled with logging
- [x] Request schema standardized
- [x] Test script created
- [x] No linting errors
- [x] Backward compatibility maintained
## 🎯 Benefits Achieved
1. **Centralized Execution**: All AI requests go through one method
2. **Consistent Logging**: Every request logs steps to console
3. **Better Debugging**: Clear step-by-step visibility
4. **Error Handling**: Comprehensive error detection and logging
5. **Reduced Duplication**: No scattered AI call logic
6. **Easy Testing**: Single point to test/mock
7. **Future Ready**: Easy to add retry logic, backoff, etc.
## 📝 Console Output Example
When running any AI function, you'll see:
```
[AI][generate_ideas] Step 1: Preparing request...
[AI][generate_ideas] Step 2: Using model: gpt-4o
[AI][generate_ideas] Step 3: Auto-enabled JSON mode for gpt-4o
[AI][generate_ideas] Step 4: Prompt length: 2345 characters
[AI][generate_ideas] Step 5: Request payload prepared (model=gpt-4o, max_tokens=4000, temp=0.7)
[AI][generate_ideas] Step 6: Sending request to OpenAI API...
[AI][generate_ideas] Step 7: Received response in 3.45s (status=200)
[AI][generate_ideas] Step 8: Received 250 tokens (input: 100, output: 150)
[AI][generate_ideas] Step 9: Content length: 600 characters
[AI][generate_ideas] Step 10: Cost calculated: $0.000250
[AI][generate_ideas][Success] Request completed successfully
```
## 🚀 Next Steps (Future Stages)
- **Stage 3**: Simplify logging (optional - console logging already implemented)
- **Stage 4**: Clean up legacy code (remove old processor completely)
- **Future**: Add retry logic, exponential backoff, request queuing

171
docs/AI-NEW-DESIGN/STAGE3-LOGGING-COMPLETE.md
View File

@@ -1,171 +0,0 @@
# Stage 3 - Clean Logging, Unified Debug Flow & Step Traceability - COMPLETE ✅
## Summary
Successfully replaced all fragmented or frontend-based debugging systems with a consistent, lightweight backend-only logging flow. All AI activity is now tracked via structured console messages with no UI panels, no Zustand state, and no silent failures.
## ✅ Completed Deliverables
### 1. ConsoleStepTracker Created
#### `tracker.py` - ConsoleStepTracker Class
- **Purpose**: Lightweight console-based step tracker for AI functions
- **Features**:
- Logs each step to console with timestamps and clear labels
- Only logs if `DEBUG_MODE` is True
- Standardized phase methods: `init()`, `prep()`, `ai_call()`, `parse()`, `save()`, `done()`
- Error logging: `error()`, `timeout()`, `rate_limit()`, `malformed_json()`
- Retry logging: `retry()`
- Duration tracking
#### Log Format
```
[HH:MM:SS] [function_name] [PHASE] message
[HH:MM:SS] [function_name] [PHASE] ✅ success message
[HH:MM:SS] [function_name] [PHASE] [ERROR] error message
[function_name] === AI Task Complete ===
```
### 2. DEBUG_MODE Constant Added
#### `constants.py`
- Added `DEBUG_MODE = True` constant
- Controls all console logging
- Can be set to `False` in production to disable verbose logging
- All print statements check `DEBUG_MODE` before logging
### 3. Integrated Tracker into AI Functions
#### `generate_ideas.py`
- ✅ Added `ConsoleStepTracker` initialization
- ✅ Logs: INIT → PREP → AI_CALL → PARSE → SAVE → DONE
- ✅ Error handling with tracker.error()
- ✅ Passes tracker to `run_ai_request()`
#### `ai_core.py`
- ✅ Updated `run_ai_request()` to accept optional tracker parameter
- ✅ All logging now uses tracker methods
- ✅ Replaced all `print()` statements with tracker calls
- ✅ Standardized error logging format
### 4. Frontend Debug Systems Deprecated
#### `TablePageTemplate.tsx`
- ✅ Commented out `AIRequestLogsSection` component
- ✅ Commented out import of `useAIRequestLogsStore`
- ✅ Added deprecation comments
#### Frontend Store (Kept for now, but unused)
- `aiRequestLogsStore.ts` - Still exists but no longer used
- All calls to `addLog`, `updateLog`, `addRequestStep`, `addResponseStep` are deprecated
### 5. Error Standardization
#### Standardized Error Format
```
[ERROR] {function_name}: {error_type} – {message}
```
#### Error Types
- `ConfigurationError` - API key not configured
- `ValidationError` - Input validation failed
- `HTTPError` - HTTP request failed
- `Timeout` - Request timeout
- `RateLimit` - Rate limit hit
- `MalformedJSON` - JSON parsing failed
- `EmptyResponse` - No content in response
- `ParseError` - Response parsing failed
- `Exception` - Unexpected exception
### 6. Example Console Output
#### Successful Execution
```
[14:23:45] [generate_ideas] [INIT] Task started
[14:23:45] [generate_ideas] [PREP] Loading account and cluster data...
[14:23:45] [generate_ideas] [PREP] Validating input...
[14:23:45] [generate_ideas] [PREP] Loading cluster with keywords...
[14:23:45] [generate_ideas] [PREP] Building prompt...
[14:23:45] [generate_ideas] [AI_CALL] Preparing request...
[14:23:45] [generate_ideas] [AI_CALL] Using model: gpt-4o
[14:23:45] [generate_ideas] [AI_CALL] Auto-enabled JSON mode for gpt-4o
[14:23:45] [generate_ideas] [AI_CALL] Prompt length: 1234 characters
[14:23:45] [generate_ideas] [AI_CALL] Request payload prepared (model=gpt-4o, max_tokens=4000, temp=0.7)
[14:23:45] [generate_ideas] [AI_CALL] Sending request to OpenAI API...
[14:23:48] [generate_ideas] [AI_CALL] Received response in 2.34s (status=200)
[14:23:48] [generate_ideas] [PARSE] Received 250 tokens (input: 100, output: 150)
[14:23:48] [generate_ideas] [PARSE] Content length: 600 characters
[14:23:48] [generate_ideas] [PARSE] Cost calculated: $0.000250
[14:23:48] [generate_ideas] [DONE] ✅ Request completed successfully (Duration: 3.12s)
[14:23:48] [generate_ideas] [PARSE] Parsing AI response...
[14:23:48] [generate_ideas] [PARSE] Parsed 1 idea(s)
[14:23:48] [generate_ideas] [SAVE] Saving idea to database...
[14:23:48] [generate_ideas] [SAVE] Saved 1 idea(s)
[14:23:48] [generate_ideas] [DONE] ✅ Idea 'My Great Idea' created successfully (Duration: 3.15s)
[generate_ideas] === AI Task Complete ===
```
#### Error Execution
```
[14:25:10] [generate_ideas] [INIT] Task started
[14:25:10] [generate_ideas] [PREP] Loading account and cluster data...
[14:25:10] [generate_ideas] [PREP] Validating input...
[14:25:10] [generate_ideas] [PREP] [ERROR] ValidationError – No cluster found
```
## 📋 File Changes Summary
| File | Changes | Status |
|------|---------|--------|
| `tracker.py` | Added `ConsoleStepTracker` class | ✅ Complete |
| `constants.py` | Added `DEBUG_MODE` constant | ✅ Complete |
| `ai_core.py` | Updated to use tracker, removed print() statements | ✅ Complete |
| `generate_ideas.py` | Integrated ConsoleStepTracker | ✅ Complete |
| `TablePageTemplate.tsx` | Commented out frontend debug UI | ✅ Complete |
## 🔄 Remaining Work
### Functions Still Need Tracker Integration
- [ ] `auto_cluster.py` - Add tracker to core function
- [ ] `generate_content.py` - Add tracker to core function
- [ ] `generate_images.py` - Add tracker to core function
### Image Generation Logging
- [ ] Update `_generate_image_openai()` to use tracker
- [ ] Update `_generate_image_runware()` to use tracker
- [ ] Replace all print() statements with tracker calls
### Frontend Cleanup
- [ ] Remove or fully comment out `AIRequestLogsSection` function body
- [ ] Remove unused imports from `api.ts` and `useProgressModal.ts`
- [ ] Optionally delete `aiRequestLogsStore.ts` (or keep for reference)
## ✅ Verification Checklist
- [x] ConsoleStepTracker created with all methods
- [x] DEBUG_MODE constant added
- [x] `run_ai_request()` updated to use tracker
- [x] `generate_ideas.py` integrated with tracker
- [x] Frontend debug UI commented out
- [x] Error logging standardized
- [ ] All function files integrated (partial)
- [ ] Image generation logging updated (pending)
- [ ] All print() statements replaced (partial)
## 🎯 Benefits Achieved
1. **Unified Logging**: All AI functions use same logging format
2. **Backend-Only**: No frontend state management needed
3. **Production Ready**: Can disable logs via DEBUG_MODE
4. **Clear Traceability**: Every step visible in console
5. **Error Visibility**: All errors clearly labeled and logged
6. **No Silent Failures**: Every failure prints its cause
## 📝 Next Steps
1. Complete tracker integration in remaining functions
2. Update image generation methods
3. Remove remaining print() statements
4. Test end-to-end with all four AI flows
5. Optionally clean up frontend debug code completely

220
docs/AI-NEW-DESIGN/STAGE4-PROMPT-REGISTRY-COMPLETE.md
View File

@@ -1,220 +0,0 @@
# Stage 4 - Prompt Registry, Model Unification, and Final Function Hooks - COMPLETE ✅
## Summary
Successfully created a centralized prompt registry system, unified model configurations, and standardized all AI function execution with clean, minimal function files.
## ✅ Completed Deliverables
### 1. Prompt Registry System Created
#### `ai/prompts.py` - PromptRegistry Class
- **Purpose**: Centralized prompt management with hierarchical resolution
- **Features**:
- Hierarchical prompt resolution:
1. Task-level `prompt_override` (if exists)
2. DB prompt for (account, function)
3. Default fallback from registry
- Supports both `.format()` style and `[IGNY8_*]` placeholder replacement
- Function-to-prompt-type mapping
- Convenience methods: `get_image_prompt_template()`, `get_negative_prompt()`
#### Prompt Resolution Priority
```python
# Priority 1: Task override
if task.prompt_override:
use task.prompt_override
# Priority 2: DB prompt
elif DB prompt for (account, function) exists:
use DB prompt
# Priority 3: Default fallback
else:
use default from registry
```
### 2. Model Configuration Centralized
#### `ai/settings.py` - MODEL_CONFIG
- **Purpose**: Centralized model configurations for all AI functions
- **Configurations**:
```python
MODEL_CONFIG = {
"auto_cluster": {
"model": "gpt-4o-mini",
"max_tokens": 3000,
"temperature": 0.7,
"response_format": {"type": "json_object"},
},
"generate_ideas": {
"model": "gpt-4.1",
"max_tokens": 4000,
"temperature": 0.7,
"response_format": {"type": "json_object"},
},
"generate_content": {
"model": "gpt-4.1",
"max_tokens": 8000,
"temperature": 0.7,
"response_format": None, # Text output
},
"generate_images": {
"model": "dall-e-3",
"size": "1024x1024",
"provider": "openai",
},
}
```
#### Helper Functions
- `get_model_config(function_name)` - Get full config
- `get_model(function_name)` - Get model name
- `get_max_tokens(function_name)` - Get max tokens
- `get_temperature(function_name)` - Get temperature
### 3. Updated All AI Functions
#### `functions/auto_cluster.py`
- ✅ Uses `PromptRegistry.get_prompt()`
- ✅ Uses `get_model_config()` for model settings
- ✅ Removed direct `get_prompt_value()` calls
#### `functions/generate_ideas.py`
- ✅ Uses `PromptRegistry.get_prompt()` with context
- ✅ Uses `get_model_config()` for model settings
- ✅ Clean prompt building with context variables
#### `functions/generate_content.py`
- ✅ Uses `PromptRegistry.get_prompt()` with task support
- ✅ Uses `get_model_config()` for model settings
- ✅ Supports task-level prompt overrides
#### `functions/generate_images.py`
- ✅ Uses `PromptRegistry.get_prompt()` for extraction
- ✅ Uses `PromptRegistry.get_image_prompt_template()`
- ✅ Uses `PromptRegistry.get_negative_prompt()`
- ✅ Uses `get_model_config()` for model settings
### 4. Updated Engine
#### `engine.py`
- ✅ Uses `get_model_config()` instead of `fn.get_model()`
- ✅ Passes model config to `run_ai_request()`
- ✅ Unified model selection across all functions
### 5. Standardized Response Format
All functions now return consistent format:
```python
{
"success": True/False,
"output": "HTML or image_url or data",
"raw": raw_response_json, # Optional
"meta": {
"word_count": 1536, # For content
"keywords": [...], # For clusters
"model_used": "gpt-4.1",
"tokens": 250,
"cost": 0.000123
},
"error": None or error_message
}
```
## 📋 File Changes Summary
| File | Changes | Status |
|------|---------|--------|
| `prompts.py` | Created PromptRegistry class | ✅ Complete |
| `settings.py` | Created MODEL_CONFIG and helpers | ✅ Complete |
| `functions/auto_cluster.py` | Updated to use registry and settings | ✅ Complete |
| `functions/generate_ideas.py` | Updated to use registry and settings | ✅ Complete |
| `functions/generate_content.py` | Updated to use registry and settings | ✅ Complete |
| `functions/generate_images.py` | Updated to use registry and settings | ✅ Complete |
| `engine.py` | Updated to use model config | ✅ Complete |
| `__init__.py` | Exported new modules | ✅ Complete |
## 🔄 Migration Path
### Old Code (Deprecated)
```python
from igny8_core.modules.system.utils import get_prompt_value, get_default_prompt
prompt_template = get_prompt_value(account, 'clustering')
prompt = prompt_template.replace('[IGNY8_KEYWORDS]', keywords_text)
```
### New Code (Recommended)
```python
from igny8_core.ai.prompts import PromptRegistry
from igny8_core.ai.settings import get_model_config
# Get prompt from registry
prompt = PromptRegistry.get_prompt(
function_name='auto_cluster',
account=account,
context={'KEYWORDS': keywords_text}
)
# Get model config
model_config = get_model_config('auto_cluster')
```
## ✅ Verification Checklist
- [x] PromptRegistry created with hierarchical resolution
- [x] MODEL_CONFIG created with all function configs
- [x] All functions updated to use registry
- [x] All functions updated to use model config
- [x] Engine updated to use model config
- [x] Response format standardized
- [x] No direct prompt utility calls in functions
- [x] Task-level overrides supported
- [x] DB prompts supported
- [x] Default fallbacks working
## 🎯 Benefits Achieved
1. **Centralized Prompts**: All prompts in one registry
2. **Hierarchical Resolution**: Task → DB → Default
3. **Model Unification**: All models configured in one place
4. **Easy Customization**: Tenant admins can override prompts
5. **Consistent Execution**: All functions use same pattern
6. **Traceability**: Prompt source clearly identifiable
7. **Minimal Functions**: Functions are clean and focused
## 📝 Prompt Source Traceability
Each prompt execution logs its source:
- `[PROMPT] Using task-level prompt override for generate_content`
- `[PROMPT] Using DB prompt for generate_ideas (account 123)`
- `[PROMPT] Using default prompt for auto_cluster`
## 🚀 Final Structure
```
/ai/
├── functions/
│ ├── auto_cluster.py ← Uses registry + settings
│ ├── generate_ideas.py ← Uses registry + settings
│ ├── generate_content.py ← Uses registry + settings
│ └── generate_images.py ← Uses registry + settings
├── prompts.py ← Prompt Registry ✅
├── settings.py ← Model Configs ✅
├── ai_core.py ← Unified execution ✅
├── engine.py ← Uses settings ✅
└── tracker.py ← Console logging ✅
```
## ✅ Expected Outcomes Achieved
- ✅ All AI executions use common format
- ✅ Prompt customization is dynamic and override-able
- ✅ No duplication across AI functions
- ✅ Every AI task has:
- ✅ Clean inputs
- ✅ Unified execution
- ✅ Standard outputs
- ✅ Clear error tracking
- ✅ Prompt traceability

749
docs/Architecture/01-ARCHITECTURE.md
View File

@@ -1,749 +0,0 @@
# IGNY8 System Architecture
**Version:** 1.0
**Last Updated:** 2025-01-XX
**Purpose:** Complete system architecture documentation covering design patterns, principles, tech stack, and structural organization.
---
## Table of Contents
1. [System Overview](#system-overview)
2. [Tech Stack](#tech-stack)
3. [Core Architecture Principles](#core-architecture-principles)
4. [Project Structure](#project-structure)
5. [Key Architectural Patterns](#key-architectural-patterns)
6. [Multi-Tenancy Architecture](#multi-tenancy-architecture)
7. [Module Organization](#module-organization)
8. [API Architecture](#api-architecture)
9. [Frontend Architecture](#frontend-architecture)
10. [Backend Architecture](#backend-architecture)
11. [Database Architecture](#database-architecture)
12. [Security Architecture](#security-architecture)
13. [Deployment Architecture](#deployment-architecture)
---
## System Overview
**IGNY8** is a full-stack SaaS platform for SEO keyword management and AI-driven content generation. The system migrated from a WordPress plugin architecture to a modern Django + React architecture, providing a scalable, multi-account platform for content planning and generation.
### Core Capabilities
- **Multi-Account SaaS Platform**: Complete account isolation with site/sector hierarchy
- **SEO Keyword Management**: Import, organize, and cluster keywords
- **AI-Powered Content Planning**: Automated keyword clustering and content idea generation
- **AI Content Generation**: Automated blog post and article generation
- **Image Generation**: AI-powered image generation for content
- **WordPress Integration**: Publish content directly to WordPress sites
- **Subscription Management**: Plan-based limits and billing integration
---
## Tech Stack
### Backend
- **Framework**: Django 5.2+ with Django REST Framework (DRF)
- **Database**: PostgreSQL
- **Task Queue**: Celery with Redis broker
- **Authentication**: JWT (JSON Web Tokens) + Session-based auth
- **API**: RESTful API with DRF ViewSets
- **Caching**: Redis
- **File Storage**: Local filesystem (configurable for S3)
### Frontend
- **Framework**: React 19 with TypeScript
- **Build Tool**: Vite
- **Styling**: Tailwind CSS
- **State Management**: Zustand
- **Routing**: React Router v6
- **HTTP Client**: Fetch API with custom wrapper
- **UI Components**: Custom component library
### Infrastructure
- **Containerization**: Docker & Docker Compose
- **Reverse Proxy**: Caddy (HTTPS on port 443)
- **Process Management**: Supervisor (for Celery workers)
- **Monitoring**: Portainer (optional)
### Development Tools
- **Backend**: Python 3.11+, pip, virtualenv
- **Frontend**: Node.js 18+, npm/yarn
- **Version Control**: Git
- **Code Quality**: ESLint, Prettier (frontend), Black, Flake8 (backend)
---
## Core Architecture Principles
### 1. Configuration-Driven Everything
**Principle**: Zero HTML/JSX duplication - All UI rendered from configuration.
- **Tables, filters, forms** all driven by config files
- **Single source of truth** - Change config, UI updates everywhere
- **Page-local config** for page-specific settings
- **Shared snippets** for reusable column/filter/action definitions
**Implementation**:
- Frontend: Config files in `/config/pages/` and `/config/snippets/`
- Backend: DRF serializers and ViewSet actions
### 2. Multi-Tenancy Foundation
**Principle**: Complete account isolation with automatic filtering.
- All models inherit `AccountBaseModel` with automatic account isolation
- All ViewSets inherit `AccountModelViewSet` with automatic account filtering
- Middleware injects account context from JWT on every request
- Site > Sector hierarchy for content organization
**Implementation**:
- Base models: `AccountBaseModel`, `SiteSectorBaseModel`
- Base ViewSets: `AccountModelViewSet`, `SiteSectorModelViewSet`
- Middleware: `AccountContextMiddleware` sets `request.account`
### 3. Template System (4 Universal Templates)
**Principle**: Reusable templates for all page types.
- **DashboardTemplate**: Module home pages (KPIs, workflow steps, charts)
- **TablePageTemplate**: CRUD table pages (Keywords, Clusters, Tasks, etc.)
- **FormPageTemplate**: Settings/form pages (Settings, Integration, etc.)
- **SystemPageTemplate**: System/admin pages (Logs, Status, Monitoring)
**Implementation**:
- Frontend: `/templates/` directory with 4 template components
- Config-driven: Templates accept config objects for customization
### 4. Unified AI Processor
**Principle**: Single interface for all AI operations.
- Single `AIProcessor` class handles all AI operations
- Manual and automated workflows use same functions
- Action-based routing: 'clustering', 'ideas', 'content_generation', 'image_generation'
- Account-specific API keys and model configuration
**Implementation**:
- Backend: `AIProcessor` class in `/utils/ai_processor.py`
- Integration: Loads API keys from `IntegrationSettings` model
### 5. Module-Based Organization
**Principle**: Clear module boundaries with shared utilities.
- Each module = Django app (`/modules/{module}/`)
- Clear module boundaries with shared utilities
- Module router pattern for subpage routing
- Consistent structure across modules
**Implementation**:
- Backend: `/modules/planner/`, `/modules/writer/`, `/modules/system/`, `/modules/billing/`
- Frontend: `/pages/Planner/`, `/pages/Writer/`, `/pages/Settings/`, etc.
---
## Project Structure
```
igny8/
├── backend/ # Django backend
│ └── igny8_core/ # Django project
│ ├── auth/ # Multi-tenancy, User, Account, Plan models
│ │ ├── models.py # Account, User, Plan, Site, Sector, Industry models
│ │ ├── views.py # Account, User, Site, Sector ViewSets
│ │ ├── serializers.py # Account, User, Plan serializers
│ │ └── urls.py # Auth module URLs
│ ├── modules/ # Feature modules
│ │ ├── planner/ # Keywords, Clusters, Ideas
│ │ │ ├── models.py # Keywords, Clusters, ContentIdeas models
│ │ │ ├── views.py # KeywordViewSet, ClusterViewSet, ContentIdeasViewSet
│ │ │ ├── tasks.py # Celery tasks for AI operations
│ │ │ ├── serializers.py # Model serializers
│ │ │ └── urls.py # Planner module URLs
│ │ ├── writer/ # Tasks, Content, Images
│ │ │ ├── models.py # Tasks, Content, Images models
│ │ │ ├── views.py # TasksViewSet
│ │ │ ├── tasks.py # Celery tasks for content/image generation
│ │ │ └── urls.py # Writer module URLs
│ │ ├── system/ # Settings, Prompts, Integration
│ │ │ ├── models.py # AIPrompt, IntegrationSettings, AuthorProfile, Strategy
│ │ │ ├── views.py # AIPromptViewSet, AuthorProfileViewSet
│ │ │ ├── integration_views.py # IntegrationSettingsViewSet, task_progress
│ │ │ ├── utils.py # Default prompts, prompt loading
│ │ │ └── urls.py # System module URLs
│ │ └── billing/ # Credits, Transactions, Usage
│ │ ├── models.py # CreditTransaction, UsageLog models
│ │ ├── views.py # Billing ViewSets
│ │ └── services.py # CreditService
│ ├── api/ # API base classes
│ │ └── base.py # AccountModelViewSet, SiteSectorModelViewSet
│ ├── utils/ # Shared utilities
│ │ ├── ai_processor.py # Unified AI interface
│ │ └── content_normalizer.py # Content processing utilities
│ ├── middleware/ # Custom middleware
│ │ ├── account.py # AccountContextMiddleware (sets request.account)
│ │ └── resource_tracker.py # ResourceTrackerMiddleware (API metrics)
│ ├── settings.py # Django settings
│ ├── urls.py # Root URL configuration
│ └── celery.py # Celery configuration
│
├── frontend/ # React frontend
│ └── src/
│ ├── pages/ # Page components
│ │ ├── Planner/ # KeywordsPage, ClustersPage, IdeasPage, Dashboard
│ │ ├── Writer/ # TasksPage, DraftsPage, PublishedPage, Dashboard
│ │ ├── Settings/ # General, Integration, Status, ImportExport
│ │ ├── Billing/ # Credits, Transactions, Usage
│ │ └── AuthPages/ # SignIn, SignUp
│ ├── templates/ # 4 master templates
│ │ ├── DashboardTemplate.tsx
│ │ ├── TablePageTemplate.tsx
│ │ ├── FormPageTemplate.tsx
│ │ └── SystemPageTemplate.tsx
│ ├── components/ # UI components
│ │ ├── layout/ # AppLayout, Sidebar, Header, Breadcrumbs
│ │ ├── table/ # DataTable, Filters, Actions, Pagination
│ │ ├── ui/ # Button, Card, Modal, Toast, etc.
│ │ └── auth/ # ProtectedRoute, Auth components
│ ├── config/ # Configuration files
│ │ ├── pages/ # Page-specific configs
│ │ │ └── keywords.config.tsx
│ │ ├── snippets/ # Shared column/filter/action definitions
│ │ │ ├── columns.snippets.ts
│ │ │ ├── filters.snippets.ts
│ │ │ └── actions.snippets.ts
│ │ └── routes.config.ts # Route configuration
│ ├── store/ # Zustand stores
│ │ ├── authStore.ts # Authentication state
│ │ ├── plannerStore.ts # Planner module state
│ │ ├── siteStore.ts # Site selection state
│ │ └── aiRequestLogsStore.ts # AI request/response logs
│ ├── services/ # API clients
│ │ └── api.ts # fetchAPI, API functions
│ ├── hooks/ # Custom React hooks
│ │ ├── useProgressModal.ts # Progress modal for long-running tasks
│ │ └── useAuth.ts # Authentication hook
│ ├── layout/ # Layout components
│ │ └── AppLayout.tsx # Main app layout wrapper
│ ├── App.tsx # Root component with routing
│ └── main.tsx # Entry point
│
└── docs/ # Documentation
└── ActiveDocs/ # Active documentation
├── 01-ARCHITECTURE.md # This file
├── 02-FRONTEND.md # Frontend documentation
├── 03-BACKEND.md # Backend documentation
├── 04-AI-FUNCTIONS.md # AI functions documentation
└── 05-ACCOUNT-USER-PLAN.md # Account/User/Plan documentation
```
---
## Key Architectural Patterns
### Configuration System (Page-Local Config + Shared Snippets)
**Rule**: Config = Page-Local, Snippets = Shared
**Structure**:
```
/pages/Planner/KeywordsPage.tsx
├── Imports snippets from /config/snippets/
├── Defines page-local tableConfig, filterConfig, actionsConfig
└── Passes config to TablePageTemplate
/config/snippets/
├── columns.snippets.ts # statusColumn, titleColumn, etc.
├── filters.snippets.ts # statusFilter, dateRangeFilter, etc.
├── actions.snippets.ts # commonActions, bulkActions, etc.
```
**Benefits**:
- Reusable components across pages
- Page-specific customization
- Single source of truth for shared definitions
### Base ViewSet Pattern
**Backend Pattern**: All ViewSets inherit from base classes for consistent behavior.
**Base Classes**:
- `AccountModelViewSet`: Automatic account filtering
- `SiteSectorModelViewSet`: Account + site/sector filtering + access control
**Benefits**:
- Consistent access control
- Automatic account isolation
- Reduced code duplication
### Celery Task Pattern
**Pattern**: Long-running operations use Celery tasks with progress tracking.
**Structure**:
1. API endpoint queues Celery task
2. Task updates progress via `update_state()`
3. Frontend polls `task_progress` endpoint
4. Progress displayed in modal
**Benefits**:
- Non-blocking API responses
- Real-time progress updates
- Scalable background processing
### AI Function Pattern
**Pattern**: All AI functions follow consistent structure.
**Structure**:
1. API Endpoint: Validates input, queues Celery task
2. Celery Task: Wraps core function with progress tracking
3. Core Function: Business logic, calls AIProcessor
4. AIProcessor: Makes API calls, returns structured data
5. Core Function: Saves results to database
**Benefits**:
- Consistent error handling
- Progress tracking
- Reusable AI interface
---
## Multi-Tenancy Architecture
### Account Isolation
**Principle**: All data is isolated by account.
**Implementation**:
- All models inherit `AccountBaseModel` (has `account` ForeignKey)
- All ViewSets inherit `AccountModelViewSet` (filters by `request.account`)
- Middleware sets `request.account` from JWT token
**Access Control**:
- Admin/Developer users: Bypass account filtering (see all accounts)
- System account users: Bypass account filtering (see all accounts)
- Regular users: Only see data from their account
### Site/Sector Hierarchy
**Structure**:
```
Account (1) ──< (N) Site
Site (1) ──< (1-5) Sector
Sector (1) ──< (N) Keywords, Clusters, ContentIdeas, Tasks
```
**Implementation**:
- Models inherit `SiteSectorBaseModel` (has `site` and `sector` ForeignKeys)
- ViewSets inherit `SiteSectorModelViewSet` (filters by accessible sites)
- User access control via `User.get_accessible_sites()`
**Site Access Control**:
- System account users: All active sites
- Developers: All active sites
- Owners/Admins: All sites in their account
- Editors/Viewers: Only sites granted via `SiteUserAccess`
---
## Module Organization
### Planner Module
**Purpose**: Keyword management and content planning.
**Models**:
- `Keywords`: Individual keywords with volume, difficulty, intent
- `Clusters`: Keyword clusters (groups of related keywords)
- `ContentIdeas`: Content ideas generated from clusters
**ViewSets**:
- `KeywordViewSet`: CRUD + `auto_cluster` action
- `ClusterViewSet`: CRUD + `auto_generate_ideas` action
- `ContentIdeasViewSet`: CRUD operations
**Tasks**:
- `auto_cluster_keywords_task`: AI-powered keyword clustering
- `auto_generate_ideas_task`: AI-powered content idea generation
### Writer Module
**Purpose**: Content generation and management.
**Models**:
- `Tasks`: Content generation tasks
- `Content`: Generated content (HTML)
- `Images`: Generated images for tasks
**ViewSets**:
- `TasksViewSet`: CRUD + `auto_generate_content`, `auto_generate_images` actions
**Tasks**:
- `auto_generate_content_task`: AI-powered content generation
- `auto_generate_images_task`: AI-powered image generation
### System Module
**Purpose**: System settings, prompts, and integrations.
**Models**:
- `AIPrompt`: AI prompt templates (clustering, ideas, content, images)
- `IntegrationSettings`: API keys and configuration (OpenAI, Runware, etc.)
- `AuthorProfile`: Writing style profiles
- `Strategy`: Content strategies per sector
**ViewSets**:
- `AIPromptViewSet`: CRUD for prompts
- `IntegrationSettingsViewSet`: CRUD + `test_openai`, `test_runware`, `generate_image`, `task_progress` actions
- `AuthorProfileViewSet`: CRUD for author profiles
- `StrategyViewSet`: CRUD for strategies
### Billing Module
**Purpose**: Credits, transactions, and usage tracking.
**Models**:
- `CreditTransaction`: Credit purchase/usage transactions
- `UsageLog`: Daily/monthly usage tracking
**ViewSets**:
- `CreditTransactionViewSet`: CRUD for transactions
- `UsageLogViewSet`: Read-only usage logs
**Services**:
- `CreditService`: Credit calculation and deduction logic
---
## API Architecture
### RESTful API Design
**Base URL**: `/api/v1/`
**Endpoint Structure**:
- `/api/v1/planner/keywords/` - Keywords CRUD
- `/api/v1/planner/keywords/auto_cluster/` - Auto-cluster action
- `/api/v1/planner/clusters/` - Clusters CRUD
- `/api/v1/planner/clusters/auto_generate_ideas/` - Auto-generate ideas action
- `/api/v1/writer/tasks/` - Tasks CRUD
- `/api/v1/writer/tasks/auto_generate_content/` - Auto-generate content action
- `/api/v1/system/settings/task_progress/{task_id}/` - Task progress polling
### Authentication
**Methods**:
- JWT (JSON Web Tokens) - Primary method
- Session-based auth - Fallback for admin
**Flow**:
1. User signs in → Backend returns JWT token
2. Frontend stores token in localStorage
3. Frontend includes token in `Authorization: Bearer {token}` header
4. Backend middleware validates token and sets `request.user` and `request.account`
### Response Format
**Success Response**:
```json
{
"success": true,
"data": { ... },
"message": "Optional message"
}
```
**Error Response**:
```json
{
"success": false,
"message": "Error message",
"errors": { ... }
}
```
### Pagination
**Format**: Page-based pagination
**Response**:
```json
{
"count": 100,
"next": "http://api.example.com/api/v1/resource/?page=2",
"previous": null,
"results": [ ... ]
}
```
---
## Frontend Architecture
### Component Hierarchy
```
App
└── AppLayout
├── Sidebar (navigation)
├── Header (user menu, notifications)
└── Main Content
└── Page Component
└── Template (DashboardTemplate, TablePageTemplate, etc.)
└── Components (DataTable, Filters, etc.)
```
### State Management
**Zustand Stores**:
- `authStore`: Authentication state (user, token, account)
- `plannerStore`: Planner module state
- `siteStore`: Selected site/sector
- `aiRequestLogsStore`: AI request/response logs
- `pageSizeStore`: Table page size preference
**Local State**: React `useState` for component-specific state
### Routing
**Structure**: React Router v6 with nested routes
**Routes**:
- `/` - Home/Dashboard
- `/planner` - Planner Dashboard
- `/planner/keywords` - Keywords page
- `/planner/clusters` - Clusters page
- `/planner/ideas` - Ideas page
- `/writer` - Writer Dashboard
- `/writer/tasks` - Tasks page
- `/settings` - Settings pages
**Protected Routes**: All routes except `/signin` and `/signup` require authentication
---
## Backend Architecture
### Model Inheritance Hierarchy
```
models.Model
└── AccountBaseModel (adds account ForeignKey)
└── SiteSectorBaseModel (adds site, sector ForeignKeys)
└── Keywords, Clusters, ContentIdeas, Tasks, etc.
```
### ViewSet Inheritance Hierarchy
```
viewsets.ModelViewSet
└── AccountModelViewSet (adds account filtering)
└── SiteSectorModelViewSet (adds site/sector filtering)
└── KeywordViewSet, ClusterViewSet, TasksViewSet, etc.
```
### Middleware Stack
1. **SecurityMiddleware**: Django security middleware
2. **SessionMiddleware**: Session management
3. **AuthenticationMiddleware**: User authentication
4. **AccountContextMiddleware**: Sets `request.account` from JWT
5. **ResourceTrackerMiddleware**: Tracks API request metrics
### Celery Task Architecture
**Broker**: Redis
**Workers**: Separate Celery worker processes
**Task Structure**:
```python
@shared_task(bind=True)
def my_task(self, ...):
# Update progress
self.update_state(state='PROGRESS', meta={...})
# Do work
result = do_work()
# Return result
return result
```
**Progress Tracking**:
- Frontend polls `/api/v1/system/settings/task_progress/{task_id}/`
- Backend returns task state and meta information
- Progress displayed in modal
---
## Database Architecture
### Core Tables
- `igny8_accounts`: Account information
- `igny8_users`: User accounts
- `igny8_plans`: Subscription plans
- `igny8_subscriptions`: Active subscriptions
- `igny8_sites`: Sites within accounts
- `igny8_sectors`: Sectors within sites
- `igny8_industries`: Global industry templates
- `igny8_industry_sectors`: Industry sector templates
### Planner Tables
- `igny8_keywords`: Keywords
- `igny8_clusters`: Keyword clusters
- `igny8_content_ideas`: Content ideas
### Writer Tables
- `igny8_tasks`: Content generation tasks
- `igny8_content`: Generated content
- `igny8_images`: Generated images
### System Tables
- `igny8_ai_prompts`: AI prompt templates
- `igny8_integration_settings`: API keys and configuration
- `igny8_author_profiles`: Writing style profiles
- `igny8_strategies`: Content strategies
### Billing Tables
- `igny8_credit_transactions`: Credit transactions
- `igny8_usage_logs`: Usage tracking
### Indexes
**Account Isolation**: All tables have indexes on `account`
**Site/Sector Filtering**: Tables with site/sector have composite indexes on `(account, site, sector)`
**Performance**: Indexes on frequently queried fields (status, created_at, etc.)
---
## Security Architecture
### Authentication
**JWT Tokens**:
- Signed with secret key
- Contains user ID and account ID
- Expires after configured time
- Stored in localStorage (frontend)
**Session Auth**:
- Fallback for admin interface
- Django session framework
### Authorization
**Role-Based Access Control (RBAC)**:
- `developer`: Full system access
- `owner`: Full account access
- `admin`: Account admin access
- `editor`: Content editing access
- `viewer`: Read-only access
**Access Control**:
- Account-level: Automatic filtering by `request.account`
- Site-level: Filtering by `user.get_accessible_sites()`
- Action-level: Permission checks in ViewSet actions
### Data Isolation
**Account Isolation**:
- All queries filtered by account
- Admin/Developer override for system accounts
**Site Access Control**:
- Users can only access granted sites
- Admin/Developer override for all sites
### API Security
**CORS**: Configured for frontend domain
**CSRF**: Enabled for session-based auth
**Rate Limiting**: (Future implementation)
**Input Validation**: DRF serializers validate all input
---
## Deployment Architecture
### Docker Compose Setup
**Services**:
- `backend`: Django application (port 8010/8011)
- `frontend`: React application (port 5173/8021)
- `db`: PostgreSQL database
- `redis`: Redis for Celery broker and caching
- `caddy`: Reverse proxy (HTTPS on port 443)
- `celery-worker`: Celery worker process
- `celery-beat`: Celery beat scheduler (optional)
### Environment Configuration
**Backend**:
- `DJANGO_SETTINGS_MODULE`: Django settings module
- `DATABASE_URL`: PostgreSQL connection string
- `REDIS_URL`: Redis connection string
- `SECRET_KEY`: Django secret key
- `OPENAI_API_KEY`: OpenAI API key (fallback)
**Frontend**:
- `VITE_API_URL`: Backend API URL
- `VITE_APP_NAME`: Application name
### Scaling Considerations
**Horizontal Scaling**:
- Multiple Celery workers
- Multiple backend instances (load balanced)
- Multiple frontend instances (static files)
**Vertical Scaling**:
- Database connection pooling
- Redis connection pooling
- Celery worker concurrency
### Monitoring
**Application Monitoring**:
- ResourceTrackerMiddleware tracks API request metrics
- Celery task monitoring via Flower (optional)
**Infrastructure Monitoring**:
- Portainer for container monitoring
- Database monitoring via PostgreSQL logs
- Redis monitoring via Redis CLI
---
## Summary
The IGNY8 architecture is built on:
1. **Configuration-Driven Design**: Zero duplication, single source of truth
2. **Multi-Tenancy Foundation**: Complete account isolation with site/sector hierarchy
3. **Template System**: 4 universal templates for all page types
4. **Unified AI Interface**: Single AIProcessor for all AI operations
5. **Module-Based Organization**: Clear boundaries with shared utilities
6. **RESTful API**: Consistent API design with DRF
7. **Modern Frontend**: React + TypeScript with Zustand state management
8. **Scalable Backend**: Django + Celery for async processing
9. **Security First**: JWT auth, RBAC, data isolation
10. **Docker Deployment**: Containerized for easy deployment and scaling
This architecture ensures scalability, maintainability, and extensibility while maintaining a clean separation of concerns across modules.

225
docs/GIT-AUTO-SYNC-ARCHITECTURE.md
View File

@@ -1,225 +0,0 @@
# Git Auto-Sync Architecture - Complete Picture
## Overview
This document explains how the automatic git synchronization works between Gitea (bare repository) and the VPS working copy.
## Architecture Components
### 1. **Bare Repository (Gitea Server)**
- **Host Path**: `/data/app/gitea/git/repositories/salman/igny8.git`
- **Container Path**: `/data/git/repositories/salman/igny8.git`
- **Type**: Bare repository (no working tree)
- **Purpose**: Stores all commits, branches, and git history
- **Access**: Served by Gitea at `https://git.igny8.com/salman/igny8.git`
### 2. **Working Copy (VPS Deployment)**
- **Host Path**: `/data/app/igny8`
- **Container Path**: `/deploy/igny8` (mounted from host)
- **Type**: Full git repository with working tree
- **Purpose**: Live application code that runs on VPS
- **Remotes**:
- `origin`: `https://git.igny8.com/salman/igny8.git` (HTTPS remote)
- `deploy`: `/data/git/repositories/salman/igny8.git` (local bare repo path)
### 3. **Docker Volume Mount**
```yaml
# From docker-compose.yml
volumes:
- ./gitea:/data # Gitea data directory
- /data/app/igny8:/deploy/igny8:rw # Mount working copy into container
```
This mount makes the VPS working copy accessible inside the Gitea container at `/deploy/igny8`.
### 4. **Post-Receive Hook**
- **Location**: `/data/app/gitea/git/repositories/salman/igny8.git/hooks/post-receive`
- **Trigger**: Automatically runs after every `git push` to Gitea
- **Execution Context**: Runs inside Gitea container, in the bare repository directory
## Complete Flow: Push to Auto-Update
### Step 1: Developer Pushes to Gitea
```bash
git push origin main
# or
git push https://git.igny8.com/salman/igny8.git main
```
### Step 2: Gitea Receives Push
- Push arrives at Gitea server
- Gitea validates and stores commits in bare repository
- Bare repo is updated: `/data/app/gitea/git/repositories/salman/igny8.git`
### Step 3: Post-Receive Hook Executes
The hook runs automatically with this context:
- **Current Directory**: `/data/git/repositories/salman/igny8.git` (bare repo)
- **Environment**: Inside Gitea container
- **Access**: Can access both bare repo and mounted working copy
### Step 4: Hook Logic Breakdown
```bash
# 1. Define paths
DEPLOY_DIR="/deploy/igny8" # Working copy (mounted from host)
SOURCE_REPO="$(pwd)" # Bare repo: /data/git/repositories/salman/igny8.git
# 2. Check if working copy is a git repository
if [ -d "$DEPLOY_DIR/.git" ]; then
# Working copy exists and is a full git repo
# 3. Set up git command with explicit paths
GIT_CMD="git --git-dir=$DEPLOY_DIR/.git --work-tree=$DEPLOY_DIR"
# 4. Add 'deploy' remote if it doesn't exist
# This points to the bare repo (no network needed, direct file access)
if ! $GIT_CMD remote get-url deploy >/dev/null 2>&1; then
$GIT_CMD remote add deploy "$SOURCE_REPO"
fi
# 5. Fetch latest commits from bare repo
# Fetches directly from file system, no HTTPS/SSH needed
$GIT_CMD fetch deploy main
# 6. Reset working tree to latest commit
# This updates all files in /data/app/igny8 to match latest commit
$GIT_CMD reset --hard remotes/deploy/main
# 7. Update origin/main tracking branch
# This is CRITICAL: tells git that local HEAD matches origin/main
# Without this, git status shows "ahead" even though files are synced
$GIT_CMD update-ref refs/remotes/origin/main HEAD
else
# First time setup: working copy doesn't exist yet
# Checkout files from bare repo to create working copy
export GIT_DIR="$SOURCE_REPO"
export GIT_WORK_TREE="$DEPLOY_DIR"
git checkout -f main
fi
```
### Step 5: Result
- **Files Updated**: All files in `/data/app/igny8` now match latest commit
- **Git Status**: Shows "up to date with origin/main" (no phantom commits)
- **Application**: If using volume mounts, containers see changes immediately
## Why This Design?
### Problem We Solved
1. **Phantom Commits**: Previously, hook updated files but not git metadata, causing git to think local was "ahead"
2. **Manual Sync Required**: Had to manually `git pull` after every push
3. **Sync Issues**: Working tree and git metadata were out of sync
### Solution
1. **Direct File System Access**: Hook uses `deploy` remote pointing to bare repo path (no network overhead)
2. **Complete Sync**: Updates both files AND git metadata (tracking branches)
3. **Automatic**: Runs on every push, no manual intervention needed
## Key Git Concepts Used
### 1. Bare Repository
- Repository without working tree
- Only contains `.git` contents (objects, refs, etc.)
- Used by servers to store code without checking out files
### 2. Working Tree
- Directory with actual files you can edit
- Has `.git` directory with repository metadata
- This is what developers work with
### 3. Remote Tracking Branches
- `refs/remotes/origin/main`: Git's record of what `origin/main` was last time we fetched
- When we update this to match HEAD, git knows we're in sync
- Without updating this, git compares HEAD to stale tracking branch → shows "ahead"
### 4. Git Reset --hard
- Moves HEAD to specified commit
- Updates working tree to match that commit
- Discards any local changes (force update)
## File System Layout
```
Host System:
├── /data/app/gitea/
│ └── git/repositories/salman/igny8.git/ (bare repo)
│ ├── objects/ (all commits, trees, blobs)
│ ├── refs/ (branches, tags)
│ └── hooks/
│ └── post-receive (auto-sync hook)
│
└── /data/app/igny8/ (working copy)
├── .git/ (git metadata)
│ ├── config (has 'origin' and 'deploy' remotes)
│ └── refs/
│ └── remotes/
│ ├── origin/main (tracking branch)
│ └── deploy/main (tracking branch)
├── backend/ (actual application files)
├── frontend/
└── ...
Inside Gitea Container:
├── /data/git/repositories/salman/igny8.git/ (same as host bare repo)
└── /deploy/igny8/ (mounted from /data/app/igny8)
└── (same contents as host /data/app/igny8)
```
## Checking Status
### On Host VPS
```bash
cd /data/app/igny8
git status # Should show "up to date with origin/main"
git log --oneline -5 # See recent commits
git remote -v # See remotes (origin + deploy)
```
### Hook Logs
```bash
docker exec gitea cat /data/gitea/log/hooks.log
```
### Manual Sync (if needed)
```bash
cd /data/app/igny8
git pull origin main # Pull from HTTPS remote
# OR
git fetch deploy main # Fetch from local bare repo
git reset --hard remotes/deploy/main
```
## Troubleshooting
### Issue: "Your branch is ahead of origin/main"
**Cause**: Hook didn't update `refs/remotes/origin/main` tracking branch
**Fix**: Hook should run `git update-ref refs/remotes/origin/main HEAD` (already in hook)
### Issue: Files not updating after push
**Check**:
1. Hook logs: `docker exec gitea cat /data/gitea/log/hooks.log`
2. Hook executable: `ls -la /data/app/gitea/git/repositories/salman/igny8.git/hooks/post-receive`
3. Mount exists: `docker exec gitea ls -la /deploy/igny8`
### Issue: Hook not running
**Check**:
1. Gitea logs: `docker logs gitea --tail 50`
2. Hook file exists and is executable
3. Push actually succeeded (check Gitea web UI)
## Summary
**The Complete Flow:**
1. Developer pushes → Gitea bare repo updated
2. Post-receive hook triggers automatically
3. Hook fetches from bare repo (via `deploy` remote)
4. Hook resets working copy to latest commit
5. Hook updates tracking branch metadata
6. VPS working copy is now in sync
7. Application containers see updated files (via volume mounts)
**Key Innovation:**
- Uses local file system path (`deploy` remote) instead of HTTPS
- Updates both files AND git metadata
- Fully automatic, no manual steps required

0
docs/Architecture/MASTER-ARCHITECTURE.md → docs/MASTER-ARCHITECTURE.md
View File

136
docs/ai-docs/AI_FILES_ANALYSIS.md Normal file
View File

@@ -0,0 +1,136 @@
# AI Folder Files Analysis
## Which Files Are Actually Needed for 3 Active Functions
### ✅ **REQUIRED FILES** (All needed for active functions)
#### 1. **tasks.py** - ✅ REQUIRED
- **Purpose:** Unified Celery task entrypoint
- **Used by:** All views (planner/views.py, writer/views.py)
- **Dependencies:** engine.py, registry.py
- **Status:** KEEP
#### 2. **engine.py** - ✅ REQUIRED
- **Purpose:** Central orchestrator for all AI functions
- **Used by:** tasks.py
- **Dependencies:** base.py, tracker.py, ai_core.py, settings.py, models.py
- **Status:** KEEP
#### 3. **base.py** - ✅ REQUIRED
- **Purpose:** Abstract base class for all AI functions
- **Used by:** All function classes (AutoClusterFunction, GenerateIdeasFunction, GenerateContentFunction)
- **Dependencies:** None
- **Status:** KEEP
#### 4. **registry.py** - ✅ REQUIRED
- **Purpose:** Function registry with lazy loading
- **Used by:** tasks.py
- **Dependencies:** base.py, function modules
- **Status:** KEEP
#### 5. **ai_core.py** - ✅ REQUIRED
- **Purpose:** Centralized AI request handler
- **Used by:** engine.py, all function classes
- **Dependencies:** constants.py, tracker.py
- **Status:** KEEP
#### 6. **prompts.py** - ✅ REQUIRED
- **Purpose:** Centralized prompt management
- **Used by:** All function classes (build_prompt methods)
- **Dependencies:** None
- **Status:** KEEP
#### 7. **settings.py** - ✅ REQUIRED
- **Purpose:** Model configurations per function
- **Used by:** engine.py
- **Dependencies:** constants.py (via ai_processor import)
- **Status:** KEEP
#### 8. **validators.py** - ✅ REQUIRED
- **Purpose:** Validation functions
- **Used by:** All function classes (validate methods)
- **Dependencies:** constants.py, models
- **Status:** KEEP
#### 9. **tracker.py** - ✅ REQUIRED
- **Purpose:** Progress tracking utilities
- **Used by:** engine.py, ai_core.py
- **Dependencies:** types.py (imported but NOT USED), constants.py
- **Status:** KEEP (but can remove types.py import)
#### 10. **constants.py** - ✅ REQUIRED
- **Purpose:** AI constants (model rates, valid models)
- **Used by:** ai_core.py, settings.py, validators.py
- **Dependencies:** None
- **Status:** KEEP
#### 11. **models.py** - ✅ REQUIRED
- **Purpose:** AITaskLog database model
- **Used by:** engine.py (for logging), admin.py
- **Dependencies:** AccountBaseModel
- **Status:** KEEP
#### 12. **apps.py** - ✅ REQUIRED
- **Purpose:** Django app configuration
- **Used by:** Django (app registration)
- **Dependencies:** admin.py
- **Status:** KEEP
#### 13. **admin.py** - ✅ REQUIRED (Optional but recommended)
- **Purpose:** Django admin interface for AITaskLog
- **Used by:** apps.py (auto-imported)
- **Dependencies:** models.py
- **Status:** KEEP (useful for debugging/admin)
---
### ❌ **UNUSED FILES** (Can be removed)
#### 1. **types.py** - ❌ NOT USED
- **Purpose:** Type definitions (StepLog, ProgressState, AITaskResult dataclasses)
- **Used by:** tracker.py (imported but never instantiated)
- **Actual usage:** None - tracker.py uses plain Dict types instead
- **Status:** **SAFE TO REMOVE**
- **Action:** Remove file and remove import from tracker.py
---
## Summary
### Files to KEEP (13 files):
1. ✅ tasks.py
2. ✅ engine.py
3. ✅ base.py
4. ✅ registry.py
5. ✅ ai_core.py
6. ✅ prompts.py
7. ✅ settings.py
8. ✅ validators.py
9. ✅ tracker.py
10. ✅ constants.py
11. ✅ models.py
12. ✅ apps.py
13. ✅ admin.py
### Files to REMOVE (1 file):
1. ❌ **types.py** - Imported but never used
---
## Action Items
1. **Remove types.py** - File is not used
2. **Remove import from tracker.py** - Remove `from igny8_core.ai.types import StepLog, ProgressState` (line 8)
---
## Verification
All other files are actively used in the execution flow:
- **tasks.py** → **registry.py** → **engine.py** → **base.py** (function classes)
- **engine.py** → **ai_core.py** → **constants.py**
- **engine.py** → **tracker.py** → **constants.py**
- **engine.py** → **settings.py** → **constants.py**
- **engine.py** → **models.py** (AITaskLog)
- **function classes** → **prompts.py**, **validators.py**
- **apps.py** → **admin.py** → **models.py**

277
docs/ai-docs/AI_FUNCTIONS_AUDIT_REPORT.md Normal file
View File

@@ -0,0 +1,277 @@
# AI Functions Deep Audit Report
## Clustering, Idea Generation, and Content Generation
**Date:** 2025-01-XX
**Scope:** Complete audit of AI functions for clustering, idea generation, and content generation to identify unused code and files safe to remove.
---
## Executive Summary
This audit identifies **legacy code, deprecated functions, and unused implementations** that are safe to remove from the codebase. The current system uses a unified AI framework through `run_ai_task` → `AIEngine` → `BaseAIFunction` implementations.
---
## 1. CURRENT ACTIVE ARCHITECTURE
### 1.1 Active Flow (What's Actually Used)
```
Frontend API Call
↓
views.py (auto_cluster/auto_generate_ideas/auto_generate_content)
↓
run_ai_task (ai/tasks.py) - Unified Celery task entrypoint
↓
AIEngine (ai/engine.py) - Orchestrator
↓
BaseAIFunction implementations:
- AutoClusterFunction (ai/functions/auto_cluster.py)
- GenerateIdeasFunction (ai/functions/generate_ideas.py)
- GenerateContentFunction (ai/functions/generate_content.py)
↓
AICore (ai/ai_core.py) - Centralized AI request handler
↓
AIProvider (OpenAI/Runware)
```
### 1.2 Active Files (KEEP)
#### Core Framework
- ✅ `backend/igny8_core/ai/tasks.py` - Unified Celery task (`run_ai_task`)
- ✅ `backend/igny8_core/ai/engine.py` - AIEngine orchestrator
- ✅ `backend/igny8_core/ai/base.py` - BaseAIFunction abstract class
- ✅ `backend/igny8_core/ai/ai_core.py` - AICore centralized handler
- ✅ `backend/igny8_core/ai/registry.py` - Function registry
- ✅ `backend/igny8_core/ai/prompts.py` - PromptRegistry
- ✅ `backend/igny8_core/ai/settings.py` - Model configurations
- ✅ `backend/igny8_core/ai/constants.py` - AI constants
- ✅ `backend/igny8_core/ai/tracker.py` - Progress tracking
- ✅ `backend/igny8_core/ai/validators.py` - Validation functions
#### Function Implementations
- ✅ `backend/igny8_core/ai/functions/auto_cluster.py` - AutoClusterFunction
- ✅ `backend/igny8_core/ai/functions/generate_ideas.py` - GenerateIdeasFunction
- ✅ `backend/igny8_core/ai/functions/generate_content.py` - GenerateContentFunction
- ✅ `backend/igny8_core/ai/functions/__init__.py` - Exports
#### API Endpoints
- ✅ `backend/igny8_core/modules/planner/views.py` - KeywordViewSet.auto_cluster(), ClusterViewSet.auto_generate_ideas()
- ✅ `backend/igny8_core/modules/writer/views.py` - TasksViewSet.auto_generate_content()
---
## 2. DEPRECATED / UNUSED CODE (SAFE TO REMOVE)
### 2.1 Legacy Wrapper Functions (NOT USED)
#### ❌ `generate_ideas_core()` - **SAFE TO REMOVE**
- **Location:** `backend/igny8_core/ai/functions/generate_ideas.py:234`
- **Status:** Legacy wrapper function for backward compatibility
- **Usage:** ❌ **NOT CALLED ANYWHERE** (only in commented test code)
- **Purpose:** Was meant for direct calls without Celery, but all calls now go through `run_ai_task`
- **Action:** **REMOVE** - Function and export from `__init__.py`
#### ❌ `generate_content_core()` - **SAFE TO REMOVE**
- **Location:** `backend/igny8_core/ai/functions/generate_content.py:303`
- **Status:** Legacy wrapper function for backward compatibility
- **Usage:** ❌ **NOT CALLED ANYWHERE** (only in commented test code)
- **Purpose:** Was meant for direct calls without Celery, but all calls now go through `run_ai_task`
- **Action:** **REMOVE** - Function and export from `__init__.py`
### 2.2 AIProcessor Deprecated Methods (NOT USED)
#### ❌ `AIProcessor.cluster_keywords()` - **SAFE TO REMOVE**
- **Location:** `backend/igny8_core/utils/ai_processor.py:1049-1282`
- **Status:** ⚠️ **DEPRECATED** (marked with deprecation warning)
- **Usage:** ❌ **NOT CALLED ANYWHERE**
- **Lines:** ~233 lines of code
- **Action:** **REMOVE** entire method
#### ❌ `AIProcessor.generate_ideas()` - **SAFE TO REMOVE**
- **Location:** `backend/igny8_core/utils/ai_processor.py:1284-1363`
- **Status:** Legacy method
- **Usage:** ❌ **NOT CALLED ANYWHERE**
- **Lines:** ~80 lines of code
- **Action:** **REMOVE** entire method
#### ❌ `AIProcessor.generate_content()` - **SAFE TO REMOVE**
- **Location:** `backend/igny8_core/utils/ai_processor.py:433-531`
- **Status:** Legacy method
- **Usage:** ❌ **NOT CALLED ANYWHERE** (only called internally by `extract_image_prompts`, which is also unused)
- **Lines:** ~98 lines of code
- **Action:** **REMOVE** entire method
#### ❌ `AIProcessor.extract_image_prompts()` - **SAFE TO REMOVE**
- **Location:** `backend/igny8_core/utils/ai_processor.py:471-580`
- **Status:** Legacy method
- **Usage:** ❌ **NOT CALLED ANYWHERE** (new framework uses `GenerateImagesFunction` which uses `AICore.run_ai_request` directly)
- **Lines:** ~110 lines of code
- **Action:** **REMOVE** entire method
**Note:** `AIProcessor.generate_image()` is **STILL USED** in `integration_views.py` for image generation, so keep the class and that method.
### 2.3 Unused Exports
#### ❌ `generate_ideas_core` export - **SAFE TO REMOVE**
- **Location:** `backend/igny8_core/ai/functions/__init__.py:5,12`
- **Action:** Remove from imports and `__all__`
#### ❌ `generate_content_core` export - **SAFE TO REMOVE**
- **Location:** `backend/igny8_core/ai/functions/__init__.py:6,14`
- **Action:** Remove from imports and `__all__`
### 2.4 Test File Cleanup
#### ⚠️ `backend/igny8_core/ai/tests/test_run.py`
- **Status:** Contains commented-out code referencing removed functions
- **Action:** Clean up commented code (lines 16, 63, 75, 126-127)
---
## 3. AIProcessor - PARTIAL CLEANUP
### 3.1 What to KEEP in AIProcessor
✅ **KEEP:**
- `AIProcessor.__init__()` - Initialization
- `AIProcessor._get_api_key()` - API key retrieval
- `AIProcessor._get_model()` - Model retrieval
- `AIProcessor._call_openai()` - OpenAI API calls (used by generate_image)
- `AIProcessor._extract_json_from_response()` - JSON extraction
- `AIProcessor.generate_image()` - **STILL USED** in `integration_views.py`
- `AIProcessor.get_prompt()` - May be used by generate_image
- Constants imports (MODEL_RATES, etc.) - Used by AICore
### 3.2 What to REMOVE from AIProcessor
❌ **REMOVE:**
- `AIProcessor.cluster_keywords()` - ~233 lines
- `AIProcessor.generate_ideas()` - ~80 lines
- `AIProcessor.generate_content()` - ~98 lines
- `AIProcessor.extract_image_prompts()` - ~110 lines
- **Total:** ~521 lines of unused code
---
## 4. FILES ALREADY DELETED (Good!)
✅ **Already Removed:**
- `backend/igny8_core/modules/planner/tasks.py` - ✅ Already deleted
- `backend/igny8_core/modules/writer/tasks.py` - ✅ Already deleted
---
## 5. SUMMARY OF REMOVALS
### 5.1 Functions to Remove
| Function | Location | Lines | Status |
|----------|----------|-------|--------|
| `generate_ideas_core()` | `ai/functions/generate_ideas.py:234` | ~100 | ❌ Remove |
| `generate_content_core()` | `ai/functions/generate_content.py:303` | ~85 | ❌ Remove |
| `AIProcessor.cluster_keywords()` | `utils/ai_processor.py:1049` | ~233 | ❌ Remove |
| `AIProcessor.generate_ideas()` | `utils/ai_processor.py:1284` | ~80 | ❌ Remove |
| `AIProcessor.generate_content()` | `utils/ai_processor.py:433` | ~98 | ❌ Remove |
| `AIProcessor.extract_image_prompts()` | `utils/ai_processor.py:471` | ~110 | ❌ Remove |
**Total Lines to Remove:** ~706 lines
### 5.2 Exports to Remove
| Export | Location |
|--------|----------|
| `generate_ideas_core` | `ai/functions/__init__.py:5,12` |
| `generate_content_core` | `ai/functions/__init__.py:6,14` |
### 5.3 Test Cleanup
| File | Action |
|------|--------|
| `ai/tests/test_run.py` | Remove commented code (lines 16, 63, 75, 126-127) |
---
## 6. VERIFICATION CHECKLIST
Before removing, verify:
- [ ] ✅ No imports of `generate_ideas_core` or `generate_content_core` anywhere
- [ ] ✅ No calls to `AIProcessor.cluster_keywords()`, `generate_ideas()`, or `generate_content()`
- [ ] ✅ All active code paths use `run_ai_task` → `AIEngine` → `BaseAIFunction`
- [ ] ✅ `AIProcessor.generate_image()` is still used (verify in `integration_views.py`)
- [ ] ✅ Constants from `ai_processor.py` are still imported by `AICore` (verify)
---
## 7. RECOMMENDED REMOVAL ORDER
1. **Phase 1: Remove Legacy Wrapper Functions**
- Remove `generate_ideas_core()` from `generate_ideas.py`
- Remove `generate_content_core()` from `generate_content.py`
- Remove exports from `__init__.py`
2. **Phase 2: Remove AIProcessor Deprecated Methods**
- Remove `AIProcessor.cluster_keywords()`
- Remove `AIProcessor.generate_ideas()`
- Remove `AIProcessor.generate_content()`
- Remove `AIProcessor.extract_image_prompts()` (depends on generate_content, so remove after)
3. **Phase 3: Cleanup Tests**
- Clean up commented code in `test_run.py`
4. **Phase 4: Verification**
- Run full test suite
- Verify all AI functions still work
- Check for any broken imports
---
## 8. IMPACT ANALYSIS
### 8.1 No Breaking Changes Expected
- ✅ All active code paths use the new framework
- ✅ No external dependencies on removed functions
- ✅ Views only use `run_ai_task` (verified)
### 8.2 Benefits
- 🎯 **~706 lines of dead code removed**
- 🎯 **Clearer codebase** - Only active code remains
- 🎯 **Easier maintenance** - No confusion about which path to use
- 🎯 **Reduced technical debt**
---
## 9. FILES TO MODIFY
### 9.1 Files to Edit
1. `backend/igny8_core/ai/functions/generate_ideas.py`
- Remove `generate_ideas_core()` function (lines 234-333)
2. `backend/igny8_core/ai/functions/generate_content.py`
- Remove `generate_content_core()` function (lines 303-386)
3. `backend/igny8_core/ai/functions/__init__.py`
- Remove `generate_ideas_core` import and export
- Remove `generate_content_core` import and export
4. `backend/igny8_core/utils/ai_processor.py`
- Remove `cluster_keywords()` method (lines 1049-1282)
- Remove `generate_ideas()` method (lines 1284-1363)
- Remove `generate_content()` method (lines 433-531)
- Remove `extract_image_prompts()` method (lines 471-580)
5. `backend/igny8_core/ai/tests/test_run.py`
- Remove commented code referencing removed functions
---
## 10. CONCLUSION
This audit identifies **~706 lines of unused legacy code** that can be safely removed without impacting functionality. All active code paths use the unified AI framework (`run_ai_task` → `AIEngine` → `BaseAIFunction`), making these legacy functions obsolete.
**Recommendation:** Proceed with removal in phases as outlined above, with thorough testing after each phase.

802
docs/ai-docs/AI_MASTER_ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,802 @@
# AI Master Architecture Document
## Clustering, Idea Generation, and Content Generation
**Version:** 1.0
**Date:** 2025-01-XX
**Scope:** Complete architecture for 3 verified AI functions (clustering, idea generation, content generation)
---
## Table of Contents
1. [Common Architecture](#1-common-architecture)
2. [Auto Cluster Keywords](#2-auto-cluster-keywords)
3. [Generate Ideas](#3-generate-ideas)
4. [Generate Content](#4-generate-content)
---
## 1. Common Architecture
### 1.1 Core Framework Files
#### Entry Point
- **File:** `backend/igny8_core/ai/tasks.py`
- **Function:** `run_ai_task`
- **Purpose:** Unified Celery task entrypoint for all AI functions
- **Parameters:** `function_name` (str), `payload` (dict), `account_id` (int)
- **Flow:** Loads function from registry → Creates AIEngine → Executes function
#### Engine Orchestrator
- **File:** `backend/igny8_core/ai/engine.py`
- **Class:** `AIEngine`
- **Purpose:** Central orchestrator managing lifecycle, progress, logging, cost tracking
- **Methods:**
- `execute` - Main execution pipeline (6 phases: INIT, PREP, AI_CALL, PARSE, SAVE, DONE)
- `_handle_error` - Centralized error handling
- `_log_to_database` - Logs to AITaskLog model
- Helper methods: `_get_input_description`, `_build_validation_message`, `_get_prep_message`, `_get_ai_call_message`, `_get_parse_message`, `_get_parse_message_with_count`, `_get_save_message`, `_calculate_credits_for_clustering`
#### Base Function Class
- **File:** `backend/igny8_core/ai/base.py`
- **Class:** `BaseAIFunction`
- **Purpose:** Abstract base class defining interface for all AI functions
- **Abstract Methods:**
- `get_name` - Returns function name (e.g., 'auto_cluster')
- `prepare` - Loads and prepares data
- `build_prompt` - Builds AI prompt
- `parse_response` - Parses AI response
- `save_output` - Saves results to database
- **Optional Methods:**
- `get_metadata` - Returns display name, description, phases
- `get_max_items` - Returns max items limit (or None)
- `validate` - Validates input payload (default: checks for 'ids')
- `get_model` - Returns model override (default: None, uses account default)
#### Function Registry
- **File:** `backend/igny8_core/ai/registry.py`
- **Functions:**
- `register_function` - Registers function class
- `register_lazy_function` - Registers lazy loader
- `get_function` - Gets function class by name (lazy loads if needed)
- `get_function_instance` - Gets function instance by name
- `list_functions` - Lists all registered functions
- **Lazy Loaders:**
- `_load_auto_cluster` - Loads AutoClusterFunction
- `_load_generate_ideas` - Loads GenerateIdeasFunction
- `_load_generate_content` - Loads GenerateContentFunction
#### AI Core Handler
- **File:** `backend/igny8_core/ai/ai_core.py`
- **Class:** `AICore`
- **Purpose:** Centralized AI request handler for all text generation
- **Methods:**
- `run_ai_request` - Makes API call to OpenAI/Runware
- `extract_json` - Extracts JSON from response (handles markdown code blocks)
#### Prompt Registry
- **File:** `backend/igny8_core/ai/prompts.py`
- **Class:** `PromptRegistry`
- **Purpose:** Centralized prompt management with hierarchical resolution
- **Method:** `get_prompt` - Gets prompt with resolution order:
1. Task-level prompt_override (if exists)
2. DB prompt for (account, function)
3. Default fallback from DEFAULT_PROMPTS registry
- **Prompt Types:**
- `clustering` - For auto_cluster function
- `ideas` - For generate_ideas function
- `content_generation` - For generate_content function
- **Context Placeholders:**
- `[IGNY8_KEYWORDS]` - Replaced with keyword list
- `[IGNY8_CLUSTERS]` - Replaced with cluster list
- `[IGNY8_CLUSTER_KEYWORDS]` - Replaced with cluster keywords
- `[IGNY8_IDEA]` - Replaced with idea data
- `[IGNY8_CLUSTER]` - Replaced with cluster data
- `[IGNY8_KEYWORDS]` - Replaced with keywords (for content)
#### Model Settings
- **File:** `backend/igny8_core/ai/settings.py`
- **Constants:**
- `MODEL_CONFIG` - Model configurations per function (model, max_tokens, temperature, response_format)
- `FUNCTION_ALIASES` - Legacy function name mappings
- **Functions:**
- `get_model_config` - Gets model config for function (reads from IntegrationSettings if account provided)
- `get_model` - Gets model name for function
- `get_max_tokens` - Gets max tokens for function
- `get_temperature` - Gets temperature for function
#### Validators
- **File:** `backend/igny8_core/ai/validators.py`
- **Functions:**
- `validate_ids` - Validates 'ids' array in payload
- `validate_keywords_exist` - Validates keywords exist in database
- `validate_cluster_exists` - Validates cluster exists
- `validate_tasks_exist` - Validates tasks exist
- `validate_cluster_limits` - Validates plan limits (currently disabled - always returns valid)
- `validate_api_key` - Validates API key is configured
- `validate_model` - Validates model is in supported list
- `validate_image_size` - Validates image size for model
#### Progress Tracking
- **File:** `backend/igny8_core/ai/tracker.py`
- **Classes:**
- `StepTracker` - Tracks request/response steps
- `ProgressTracker` - Tracks Celery progress updates
- `CostTracker` - Tracks API costs and tokens
- `ConsoleStepTracker` - Console-based step logging
#### Database Logging
- **File:** `backend/igny8_core/ai/models.py`
- **Model:** `AITaskLog`
- **Fields:** `task_id`, `function_name`, `account`, `phase`, `message`, `status`, `duration`, `cost`, `tokens`, `request_steps`, `response_steps`, `error`, `payload`, `result`
### 1.2 Execution Flow (All Functions)
```
1. API Endpoint (views.py)
↓
2. run_ai_task (tasks.py)
- Gets account from account_id
- Gets function instance from registry
- Creates AIEngine
↓
3. AIEngine.execute (engine.py)
Phase 1: INIT (0-10%)
- Calls function.validate()
- Updates progress tracker
↓
Phase 2: PREP (10-25%)
- Calls function.prepare()
- Calls function.build_prompt()
- Updates progress tracker
↓
Phase 3: AI_CALL (25-70%)
- Gets model config from settings
- Calls AICore.run_ai_request()
- Tracks cost and tokens
- Updates progress tracker
↓
Phase 4: PARSE (70-85%)
- Calls function.parse_response()
- Updates progress tracker
↓
Phase 5: SAVE (85-98%)
- Calls function.save_output()
- Logs credit usage
- Updates progress tracker
↓
Phase 6: DONE (98-100%)
- Logs to AITaskLog
- Returns result
```
---
## 2. Auto Cluster Keywords
### 2.1 Function Implementation
- **File:** `backend/igny8_core/ai/functions/auto_cluster.py`
- **Class:** `AutoClusterFunction`
- **Inherits:** `BaseAIFunction`
### 2.2 API Endpoint
- **File:** `backend/igny8_core/modules/planner/views.py`
- **ViewSet:** `KeywordViewSet`
- **Action:** `auto_cluster`
- **Method:** POST
- **URL Path:** `/v1/planner/keywords/auto_cluster/`
- **Payload:**
- `ids` (list[int]) - Keyword IDs to cluster
- `sector_id` (int, optional) - Sector ID for filtering
- **Response:**
- `success` (bool)
- `task_id` (str) - Celery task ID if async
- `clusters_created` (int) - Number of clusters created
- `keywords_updated` (int) - Number of keywords updated
- `message` (str)
### 2.3 Function Methods
#### `get_name()`
- **Returns:** `'auto_cluster'`
#### `get_metadata()`
- **Returns:** Dict with `display_name`, `description`, `phases` (INIT, PREP, AI_CALL, PARSE, SAVE, DONE)
#### `get_max_items()`
- **Returns:** `None` (no limit)
#### `validate(payload, account)`
- **Validates:**
- Calls `validate_ids` to check for 'ids' array
- Calls `validate_keywords_exist` to verify keywords exist
- **Returns:** Dict with `valid` (bool) and optional `error` (str)
#### `prepare(payload, account)`
- **Loads:**
- Keywords from database (filters by `ids`, `account`, optional `sector_id`)
- Uses `select_related` for: `account`, `site`, `site__account`, `sector`, `sector__site`
- **Returns:** Dict with:
- `keywords` (list[Keyword objects])
- `keyword_data` (list[dict]) - Formatted data with: `id`, `keyword`, `volume`, `difficulty`, `intent`
- `sector_id` (int, optional)
#### `build_prompt(data, account)`
- **Gets Prompt:**
- Calls `PromptRegistry.get_prompt(function_name='auto_cluster', account, context)`
- Context includes: `KEYWORDS` (formatted keyword list), optional `SECTOR` (sector name)
- **Formatting:**
- Formats keywords as: `"- {keyword} (Volume: {volume}, Difficulty: {difficulty}, Intent: {intent})"`
- Replaces `[IGNY8_KEYWORDS]` placeholder
- Adds JSON mode instruction if not present
- **Returns:** Prompt string
#### `parse_response(response, step_tracker)`
- **Parsing:**
- Tries direct JSON parse first
- Falls back to `AICore.extract_json()` if needed (handles markdown code blocks)
- **Extraction:**
- Extracts `clusters` array from JSON
- Handles both dict with 'clusters' key and direct array
- **Returns:** List[Dict] with cluster data:
- `name` (str) - Cluster name
- `description` (str) - Cluster description
- `keywords` (list[str]) - List of keyword strings
#### `save_output(parsed, original_data, account, progress_tracker, step_tracker)`
- **Input:**
- `parsed` - List of cluster dicts from parse_response
- `original_data` - Dict from prepare() with `keywords` and `sector_id`
- **Process:**
- Gets account, site, sector from first keyword
- For each cluster in parsed:
- Gets or creates `Clusters` record:
- Fields: `name`, `description`, `account`, `site`, `sector`, `status='active'`
- Uses `get_or_create` with name + account + site + sector
- Matches keywords (case-insensitive):
- Normalizes cluster keywords and available keywords to lowercase
- Updates matched `Keywords` records:
- Sets `cluster` foreign key
- Sets `status='mapped'`
- Recalculates cluster metrics:
- `keywords_count` - Count of keywords in cluster
- `volume` - Sum of keyword volumes (uses `volume_override` if available, else `seed_keyword__volume`)
- **Returns:** Dict with:
- `count` (int) - Clusters created
- `clusters_created` (int) - Clusters created
- `keywords_updated` (int) - Keywords updated
### 2.4 Database Models
#### Keywords Model
- **File:** `backend/igny8_core/modules/planner/models.py`
- **Model:** `Keywords`
- **Fields Used:**
- `id` - Keyword ID
- `seed_keyword` (ForeignKey) - Reference to SeedKeyword
- `keyword` (property) - Gets keyword text from seed_keyword
- `volume` (property) - Gets volume from volume_override or seed_keyword
- `difficulty` (property) - Gets difficulty from difficulty_override or seed_keyword
- `intent` (property) - Gets intent from seed_keyword
- `cluster` (ForeignKey) - Assigned cluster
- `status` - Status ('active', 'pending', 'mapped', 'archived')
- `account`, `site`, `sector` - From SiteSectorBaseModel
#### Clusters Model
- **File:** `backend/igny8_core/modules/planner/models.py`
- **Model:** `Clusters`
- **Fields Used:**
- `name` - Cluster name (unique)
- `description` - Cluster description
- `keywords_count` - Count of keywords (recalculated)
- `volume` - Sum of keyword volumes (recalculated)
- `status` - Status ('active')
- `account`, `site`, `sector` - From SiteSectorBaseModel
### 2.5 AI Response Format
**Expected JSON:**
```json
{
"clusters": [
{
"name": "Cluster Name",
"description": "Cluster description",
"keywords": ["keyword1", "keyword2", "keyword3"]
}
]
}
```
### 2.6 Progress Messages
- **INIT:** "Validating {keyword1}, {keyword2}, {keyword3} and {X} more keywords" (shows first 3, then count)
- **PREP:** "Loading {count} keyword(s)"
- **AI_CALL:** "Generating clusters with Igny8 Semantic SEO Model"
- **PARSE:** "{count} cluster(s) created"
- **SAVE:** "Saving {count} cluster(s)"
---
## 3. Generate Ideas
### 3.1 Function Implementation
- **File:** `backend/igny8_core/ai/functions/generate_ideas.py`
- **Class:** `GenerateIdeasFunction`
- **Inherits:** `BaseAIFunction`
### 3.2 API Endpoint
- **File:** `backend/igny8_core/modules/planner/views.py`
- **ViewSet:** `ClusterViewSet`
- **Action:** `auto_generate_ideas`
- **Method:** POST
- **URL Path:** `/v1/planner/clusters/auto_generate_ideas/`
- **Payload:**
- `ids` (list[int]) - Cluster IDs (max 10)
- **Response:**
- `success` (bool)
- `task_id` (str) - Celery task ID if async
- `ideas_created` (int) - Number of ideas created
- `message` (str)
### 3.3 Function Methods
#### `get_name()`
- **Returns:** `'generate_ideas'`
#### `get_metadata()`
- **Returns:** Dict with `display_name`, `description`, `phases` (INIT, PREP, AI_CALL, PARSE, SAVE, DONE)
#### `get_max_items()`
- **Returns:** `10` (max clusters per generation)
#### `validate(payload, account)`
- **Validates:**
- Calls `super().validate()` to check for 'ids' array and max_items limit
- Calls `validate_cluster_exists` for first cluster ID
- Calls `validate_cluster_limits` for plan limits (currently disabled)
- **Returns:** Dict with `valid` (bool) and optional `error` (str)
#### `prepare(payload, account)`
- **Loads:**
- Clusters from database (filters by `ids`, `account`)
- Uses `select_related` for: `sector`, `account`, `site`, `sector__site`
- Uses `prefetch_related` for: `keywords`
- **Gets Keywords:**
- For each cluster, loads `Keywords` with `select_related('seed_keyword')`
- Extracts keyword text from `seed_keyword.keyword`
- **Returns:** Dict with:
- `clusters` (list[Cluster objects])
- `cluster_data` (list[dict]) - Formatted data with: `id`, `name`, `description`, `keywords` (list[str])
- `account` (Account object)
#### `build_prompt(data, account)`
- **Gets Prompt:**
- Calls `PromptRegistry.get_prompt(function_name='generate_ideas', account, context)`
- Context includes:
- `CLUSTERS` - Formatted cluster list: `"Cluster ID: {id} | Name: {name} | Description: {description}"`
- `CLUSTER_KEYWORDS` - Formatted cluster keywords: `"Cluster ID: {id} | Name: {name} | Keywords: {keyword1}, {keyword2}"`
- **Replaces Placeholders:**
- `[IGNY8_CLUSTERS]` → clusters_text
- `[IGNY8_CLUSTER_KEYWORDS]` → cluster_keywords_text
- **Returns:** Prompt string
#### `parse_response(response, step_tracker)`
- **Parsing:**
- Calls `AICore.extract_json()` to extract JSON from response
- Validates 'ideas' key exists in JSON
- **Returns:** List[Dict] with idea data:
- `title` (str) - Idea title
- `description` (str or dict) - Idea description (can be JSON string)
- `content_type` (str) - Content type ('blog_post', 'article', etc.)
- `content_structure` (str) - Content structure ('cluster_hub', 'supporting_page', etc.)
- `cluster_id` (int, optional) - Cluster ID reference
- `cluster_name` (str, optional) - Cluster name reference
- `estimated_word_count` (int) - Estimated word count
- `covered_keywords` or `target_keywords` (str) - Target keywords
#### `save_output(parsed, original_data, account, progress_tracker, step_tracker)`
- **Input:**
- `parsed` - List of idea dicts from parse_response
- `original_data` - Dict from prepare() with `clusters` and `cluster_data`
- **Process:**
- For each idea in parsed:
- Matches cluster:
- First tries by `cluster_id` from AI response
- Falls back to `cluster_name` matching
- Last resort: position-based matching (first idea → first cluster)
- Gets site from cluster (or cluster.sector.site)
- Handles description:
- If dict, converts to JSON string
- If not string, converts to string
- Creates `ContentIdeas` record:
- Fields:
- `idea_title` - From `title`
- `description` - Processed description
- `content_type` - From `content_type` (default: 'blog_post')
- `content_structure` - From `content_structure` (default: 'supporting_page')
- `target_keywords` - From `covered_keywords` or `target_keywords`
- `keyword_cluster` - Matched cluster
- `estimated_word_count` - From `estimated_word_count` (default: 1500)
- `status` - 'new'
- `account`, `site`, `sector` - From cluster
- **Returns:** Dict with:
- `count` (int) - Ideas created
- `ideas_created` (int) - Ideas created
### 3.4 Database Models
#### Clusters Model
- **File:** `backend/igny8_core/modules/planner/models.py`
- **Model:** `Clusters`
- **Fields Used:**
- `id` - Cluster ID
- `name` - Cluster name
- `description` - Cluster description
- `keywords` (related_name) - Related Keywords
- `account`, `site`, `sector` - From SiteSectorBaseModel
#### ContentIdeas Model
- **File:** `backend/igny8_core/modules/planner/models.py`
- **Model:** `ContentIdeas`
- **Fields Used:**
- `idea_title` - Idea title
- `description` - Idea description (can be JSON string)
- `content_type` - Content type ('blog_post', 'article', 'guide', 'tutorial')
- `content_structure` - Content structure ('cluster_hub', 'landing_page', 'pillar_page', 'supporting_page')
- `target_keywords` - Target keywords string
- `keyword_cluster` (ForeignKey) - Related cluster
- `estimated_word_count` - Estimated word count
- `status` - Status ('new', 'scheduled', 'published')
- `account`, `site`, `sector` - From SiteSectorBaseModel
### 3.5 AI Response Format
**Expected JSON:**
```json
{
"ideas": [
{
"title": "Idea Title",
"description": "Idea description or JSON structure",
"content_type": "blog_post",
"content_structure": "supporting_page",
"cluster_id": 1,
"cluster_name": "Cluster Name",
"estimated_word_count": 1500,
"covered_keywords": "keyword1, keyword2"
}
]
}
```
### 3.6 Progress Messages
- **INIT:** "Verifying cluster integrity"
- **PREP:** "Loading cluster keywords"
- **AI_CALL:** "Generating ideas with Igny8 Semantic AI"
- **PARSE:** "{count} high-opportunity idea(s) generated"
- **SAVE:** "Content Outline for Ideas generated"
---
## 4. Generate Content
### 4.1 Function Implementation
- **File:** `backend/igny8_core/ai/functions/generate_content.py`
- **Class:** `GenerateContentFunction`
- **Inherits:** `BaseAIFunction`
### 4.2 API Endpoint
- **File:** `backend/igny8_core/modules/writer/views.py`
- **ViewSet:** `TasksViewSet`
- **Action:** `auto_generate_content`
- **Method:** POST
- **URL Path:** `/v1/writer/tasks/auto_generate_content/`
- **Payload:**
- `ids` (list[int]) - Task IDs (max 10)
- **Response:**
- `success` (bool)
- `task_id` (str) - Celery task ID if async
- `tasks_updated` (int) - Number of tasks updated
- `message` (str)
### 4.3 Function Methods
#### `get_name()`
- **Returns:** `'generate_content'`
#### `get_metadata()`
- **Returns:** Dict with `display_name`, `description`, `phases` (INIT, PREP, AI_CALL, PARSE, SAVE, DONE)
#### `get_max_items()`
- **Returns:** `50` (max tasks per batch)
#### `validate(payload, account)`
- **Validates:**
- Calls `super().validate()` to check for 'ids' array and max_items limit
- Calls `validate_tasks_exist` to verify tasks exist
- **Returns:** Dict with `valid` (bool) and optional `error` (str)
#### `prepare(payload, account)`
- **Loads:**
- Tasks from database (filters by `ids`, `account`)
- Uses `select_related` for: `account`, `site`, `sector`, `cluster`, `idea`
- **Returns:** List[Task objects]
#### `build_prompt(data, account)`
- **Input:** Can be single Task or list[Task] (handles first task if list)
- **Builds Idea Data:**
- `title` - From task.title
- `description` - From task.description
- `outline` - From task.idea.description (handles JSON structure):
- If JSON, formats as: `"## {H2 heading}\n### {H3 subheading}\nContent Type: {type}\nDetails: {details}"`
- If plain text, uses as-is
- `structure` - From task.idea.content_structure or task.content_structure
- `type` - From task.idea.content_type or task.content_type
- `estimated_word_count` - From task.idea.estimated_word_count
- **Builds Cluster Data:**
- `cluster_name` - From task.cluster.name
- `description` - From task.cluster.description
- `status` - From task.cluster.status
- **Builds Keywords Data:**
- From task.keywords (legacy) or task.idea.target_keywords
- **Gets Prompt:**
- Calls `PromptRegistry.get_prompt(function_name='generate_content', account, task, context)`
- Context includes:
- `IDEA` - Formatted idea data string
- `CLUSTER` - Formatted cluster data string
- `KEYWORDS` - Keywords string
- **Returns:** Prompt string
#### `parse_response(response, step_tracker)`
- **Parsing:**
- First tries JSON parse:
- If successful and dict, returns dict
- Falls back to plain text:
- Calls `normalize_content()` from `content_normalizer` to convert to HTML
- Returns dict with `content` field
- **Returns:** Dict with:
- **If JSON:**
- `content` (str) - HTML content
- `title` (str, optional) - Content title
- `meta_title` (str, optional) - Meta title
- `meta_description` (str, optional) - Meta description
- `word_count` (int, optional) - Word count
- `primary_keyword` (str, optional) - Primary keyword
- `secondary_keywords` (list, optional) - Secondary keywords
- `tags` (list, optional) - Tags
- `categories` (list, optional) - Categories
- **If Plain Text:**
- `content` (str) - Normalized HTML content
#### `save_output(parsed, original_data, account, progress_tracker, step_tracker)`
- **Input:**
- `parsed` - Dict from parse_response
- `original_data` - Task object or list[Task] (handles first task if list)
- **Process:**
- Extracts content fields from parsed dict:
- `content_html` - From `content` field
- `title` - From `title` or task.title
- `meta_title` - From `meta_title` or task.meta_title or task.title
- `meta_description` - From `meta_description` or task.meta_description or task.description
- `word_count` - From `word_count` or calculated from content
- `primary_keyword` - From `primary_keyword`
- `secondary_keywords` - From `secondary_keywords` (converts to list if needed)
- `tags` - From `tags` (converts to list if needed)
- `categories` - From `categories` (converts to list if needed)
- Calculates word count if not provided:
- Strips HTML tags and counts words
- Gets or creates `Content` record:
- Uses `get_or_create` with `task` (OneToOne relationship)
- Defaults: `html_content`, `word_count`, `status='draft'`, `account`, `site`, `sector`
- Updates `Content` fields:
- `html_content` - Content HTML
- `word_count` - Word count
- `title` - Content title
- `meta_title` - Meta title
- `meta_description` - Meta description
- `primary_keyword` - Primary keyword
- `secondary_keywords` - Secondary keywords (JSONField)
- `tags` - Tags (JSONField)
- `categories` - Categories (JSONField)
- `status` - Always 'draft' for newly generated content
- `metadata` - Extra fields from parsed dict (excludes standard fields)
- `account`, `site`, `sector`, `task` - Aligned from task
- Updates `Tasks` record:
- Sets `status='completed'`
- Updates `updated_at`
- **Returns:** Dict with:
- `count` (int) - Tasks updated (always 1 per task)
- `tasks_updated` (int) - Tasks updated
- `word_count` (int) - Word count
### 4.4 Database Models
#### Tasks Model
- **File:** `backend/igny8_core/modules/writer/models.py`
- **Model:** `Tasks`
- **Fields Used:**
- `id` - Task ID
- `title` - Task title
- `description` - Task description
- `keywords` - Keywords string (legacy)
- `cluster` (ForeignKey) - Related cluster
- `idea` (ForeignKey) - Related ContentIdeas
- `content_structure` - Content structure
- `content_type` - Content type
- `status` - Status ('queued', 'completed')
- `meta_title` - Meta title
- `meta_description` - Meta description
- `account`, `site`, `sector` - From SiteSectorBaseModel
#### Content Model
- **File:** `backend/igny8_core/modules/writer/models.py`
- **Model:** `Content`
- **Fields Used:**
- `task` (OneToOneField) - Related task
- `html_content` - HTML content
- `word_count` - Word count
- `title` - Content title
- `meta_title` - Meta title
- `meta_description` - Meta description
- `primary_keyword` - Primary keyword
- `secondary_keywords` (JSONField) - Secondary keywords list
- `tags` (JSONField) - Tags list
- `categories` (JSONField) - Categories list
- `status` - Status ('draft', 'review', 'published')
- `metadata` (JSONField) - Additional metadata
- `account`, `site`, `sector` - From SiteSectorBaseModel (auto-set from task)
### 4.5 AI Response Format
**Expected JSON:**
```json
{
"content": "<html>Content HTML</html>",
"title": "Content Title",
"meta_title": "Meta Title",
"meta_description": "Meta description",
"word_count": 1500,
"primary_keyword": "primary keyword",
"secondary_keywords": ["keyword1", "keyword2"],
"tags": ["tag1", "tag2"],
"categories": ["category1"]
}
```
**Or Plain Text:**
```
Plain text content that will be normalized to HTML
```
### 4.6 Progress Messages
- **INIT:** "Validating task"
- **PREP:** "Preparing content idea"
- **AI_CALL:** "Writing article with Igny8 Semantic AI"
- **PARSE:** "{count} article(s) created"
- **SAVE:** "Saving article"
---
## 5. Change Guide
### 5.1 Where to Change Validation Logic
- **File:** `backend/igny8_core/ai/validators.py`
- **Functions:** `validate_ids`, `validate_keywords_exist`, `validate_cluster_exists`, `validate_tasks_exist`
- **Or:** Override `validate()` method in function class
### 5.2 Where to Change Data Loading
- **File:** Function-specific file (e.g., `auto_cluster.py`)
- **Method:** `prepare()`
- **Change:** Modify queryset filters, select_related, prefetch_related
### 5.3 Where to Change Prompts
- **File:** `backend/igny8_core/ai/prompts.py`
- **Method:** `PromptRegistry.get_prompt()`
- **Change:** Modify `DEFAULT_PROMPTS` dict or update database prompts
### 5.4 Where to Change Model Configuration
- **File:** `backend/igny8_core/ai/settings.py`
- **Constant:** `MODEL_CONFIG`
- **Change:** Update model, max_tokens, temperature, response_format per function
### 5.5 Where to Change Response Parsing
- **File:** Function-specific file (e.g., `generate_content.py`)
- **Method:** `parse_response()`
- **Change:** Modify JSON extraction or plain text handling
### 5.6 Where to Change Database Saving
- **File:** Function-specific file (e.g., `auto_cluster.py`)
- **Method:** `save_output()`
- **Change:** Modify model creation/update logic, field mappings
### 5.7 Where to Change Progress Messages
- **File:** `backend/igny8_core/ai/engine.py`
- **Methods:** `_get_prep_message()`, `_get_ai_call_message()`, `_get_parse_message()`, `_get_save_message()`
- **Or:** Override in function class `get_metadata()` phases
### 5.8 Where to Change Error Handling
- **File:** `backend/igny8_core/ai/engine.py`
- **Method:** `_handle_error()`
- **Change:** Modify error logging, error response format
---
## 6. Dependencies
### 6.1 Function Dependencies
- All functions depend on: `BaseAIFunction`, `AICore`, `PromptRegistry`, `get_model_config`
- Clustering depends on: `Keywords`, `Clusters` models
- Ideas depends on: `Clusters`, `ContentIdeas`, `Keywords` models
- Content depends on: `Tasks`, `Content`, `ContentIdeas`, `Clusters` models
### 6.2 External Dependencies
- **Celery:** For async task execution (`run_ai_task`)
- **OpenAI API:** For AI text generation (via `AICore.run_ai_request`)
- **Django ORM:** For database operations
- **IntegrationSettings:** For account-specific model configuration
---
## 7. Key Relationships
### 7.1 Clustering Flow
```
Keywords → Clusters (many-to-one)
- Keywords.cluster (ForeignKey)
- Clusters.keywords (related_name)
```
### 7.2 Ideas Flow
```
Clusters → ContentIdeas (one-to-many)
- ContentIdeas.keyword_cluster (ForeignKey)
- Clusters.ideas (related_name, if exists)
```
### 7.3 Content Flow
```
Tasks → Content (one-to-one)
- Content.task (OneToOneField)
- Tasks.content_record (related_name)
Tasks → ContentIdeas (many-to-one)
- Tasks.idea (ForeignKey)
- ContentIdeas.tasks (related_name)
Tasks → Clusters (many-to-one)
- Tasks.cluster (ForeignKey)
- Clusters.tasks (related_name)
```
---
## 8. Notes
- All functions use the same execution pipeline through `AIEngine.execute()`
- Progress tracking is handled automatically by `AIEngine`
- Cost tracking is handled automatically by `CostTracker`
- Database logging is handled automatically by `AITaskLog`
- Model configuration can be overridden per account via `IntegrationSettings`
- Prompts can be overridden per account via database prompts
- All functions support both async (Celery) and sync execution
- Error handling is centralized in `AIEngine._handle_error()`