Add SEO fields to Tasks model, improve content generation response handling, and enhance progress bar animation
- Added primary_keyword, secondary_keywords, tags, and categories fields to Tasks model - Updated generate_content function to handle full JSON response with all SEO fields - Improved progress bar animation: smooth 1% increments every 300ms - Enhanced step detection for content generation vs clustering vs ideas - Fixed progress modal to show correct messages for each function type - Added comprehensive logging to Keywords and Tasks pages for AI functions - Fixed error handling to show meaningful error messages instead of generic failures
This commit is contained in:
Gitea Deploy
271
docs/AI-FUNCTION-FILES.md
Normal file
271
docs/AI-FUNCTION-FILES.md
Normal file
@@ -0,0 +1,271 @@
|
||||
# AI Function Related Files
|
||||
|
||||
This document lists all files containing code related to:
|
||||
- Auto Cluster Keywords
|
||||
- Auto Generate Ideas
|
||||
- Auto Generate Content
|
||||
- Auto Generate Images
|
||||
|
||||
---
|
||||
|
||||
## Backend Files
|
||||
|
||||
### Auto Cluster Keywords
|
||||
|
||||
#### Core Implementation
|
||||
- `backend/igny8_core/ai/functions/auto_cluster.py` - **Main AI function implementation** (BaseAIFunction)
|
||||
- `backend/igny8_core/ai/base.py` - Base AI function class
|
||||
- `backend/igny8_core/ai/engine.py` - AI engine orchestrator
|
||||
- `backend/igny8_core/ai/processor.py` - AI processor wrapper
|
||||
- `backend/igny8_core/ai/tasks.py` - Unified Celery task entrypoint
|
||||
- `backend/igny8_core/ai/registry.py` - Function registry
|
||||
- `backend/igny8_core/ai/tracker.py` - Progress and cost tracking
|
||||
|
||||
#### API Endpoints & Views
|
||||
- `backend/igny8_core/modules/planner/views.py` - **KeywordViewSet.auto_cluster()** action
|
||||
- `backend/igny8_core/modules/planner/urls.py` - URL routing
|
||||
|
||||
#### Celery Tasks (Legacy/Alternative)
|
||||
- `backend/igny8_core/modules/planner/tasks.py` - **auto_cluster_keywords_task()** (legacy implementation)
|
||||
|
||||
#### AI Processor
|
||||
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.cluster_keywords()** method
|
||||
|
||||
#### Models
|
||||
- `backend/igny8_core/modules/planner/models.py` - Keywords, Clusters models
|
||||
|
||||
#### Serializers
|
||||
- `backend/igny8_core/modules/planner/serializers.py` - Keyword, Cluster serializers
|
||||
- `backend/igny8_core/modules/planner/cluster_serializers.py` - Cluster-specific serializers
|
||||
|
||||
#### System Integration
|
||||
- `backend/igny8_core/modules/system/schemas.py` - Schema definitions
|
||||
- `backend/igny8_core/modules/system/utils.py` - Prompt loading utilities
|
||||
|
||||
---
|
||||
|
||||
### Auto Generate Ideas
|
||||
|
||||
#### Core Implementation
|
||||
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.generate_ideas()** method
|
||||
|
||||
#### API Endpoints & Views
|
||||
- `backend/igny8_core/modules/planner/views.py` - **ClusterViewSet.auto_generate_ideas()** action
|
||||
- `backend/igny8_core/modules/planner/urls.py` - URL routing
|
||||
|
||||
#### Celery Tasks
|
||||
- `backend/igny8_core/modules/planner/tasks.py` - **auto_generate_ideas_task()** and **generate_single_idea_core()**
|
||||
|
||||
#### Models
|
||||
- `backend/igny8_core/modules/planner/models.py` - Clusters, ContentIdeas models
|
||||
|
||||
#### Serializers
|
||||
- `backend/igny8_core/modules/planner/serializers.py` - Cluster, ContentIdeas serializers
|
||||
|
||||
#### System Integration
|
||||
- `backend/igny8_core/modules/system/utils.py` - Prompt loading utilities
|
||||
|
||||
---
|
||||
|
||||
### Auto Generate Content
|
||||
|
||||
#### Core Implementation
|
||||
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.generate_content()** method
|
||||
|
||||
#### API Endpoints & Views
|
||||
- `backend/igny8_core/modules/writer/views.py` - **TasksViewSet.auto_generate_content()** action
|
||||
- `backend/igny8_core/modules/writer/urls.py` - URL routing
|
||||
|
||||
#### Celery Tasks
|
||||
- `backend/igny8_core/modules/writer/tasks.py` - **auto_generate_content_task()**
|
||||
|
||||
#### Models
|
||||
- `backend/igny8_core/modules/writer/models.py` - Tasks, Content models
|
||||
|
||||
#### Serializers
|
||||
- `backend/igny8_core/modules/writer/serializers.py` - Task, Content serializers
|
||||
|
||||
#### System Integration
|
||||
- `backend/igny8_core/modules/system/schemas.py` - Schema definitions
|
||||
- `backend/igny8_core/modules/system/utils.py` - Prompt loading utilities
|
||||
|
||||
---
|
||||
|
||||
### Auto Generate Images
|
||||
|
||||
#### Core Implementation
|
||||
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.extract_image_prompts()** and **AIProcessor.generate_image()** methods
|
||||
|
||||
#### API Endpoints & Views
|
||||
- `backend/igny8_core/modules/writer/views.py` - **TasksViewSet.auto_generate_images()** action
|
||||
- `backend/igny8_core/modules/writer/urls.py` - URL routing
|
||||
|
||||
#### Celery Tasks
|
||||
- `backend/igny8_core/modules/writer/tasks.py` - **auto_generate_images_task()**
|
||||
|
||||
#### Models
|
||||
- `backend/igny8_core/modules/writer/models.py` - Tasks, Images models
|
||||
|
||||
#### Serializers
|
||||
- `backend/igny8_core/modules/writer/serializers.py` - Task, Images serializers
|
||||
|
||||
#### System Integration
|
||||
- `backend/igny8_core/modules/system/schemas.py` - Schema definitions
|
||||
- `backend/igny8_core/modules/system/integration_views.py` - Integration settings for image generation
|
||||
|
||||
---
|
||||
|
||||
## Frontend Files
|
||||
|
||||
### Auto Cluster Keywords
|
||||
|
||||
#### Pages
|
||||
- `frontend/src/pages/Planner/Keywords.tsx` - **Main page component** with auto cluster functionality
|
||||
|
||||
#### API Services
|
||||
- `frontend/src/services/api.ts` - **autoClusterKeywords()** function
|
||||
|
||||
#### Configuration
|
||||
- `frontend/src/config/pages/keywords.config.tsx` - Page configuration
|
||||
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
|
||||
|
||||
#### State Management
|
||||
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
|
||||
|
||||
#### Hooks
|
||||
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
|
||||
|
||||
---
|
||||
|
||||
### Auto Generate Ideas
|
||||
|
||||
#### Pages
|
||||
- `frontend/src/pages/Planner/Clusters.tsx` - **Main page component** with auto generate ideas functionality
|
||||
|
||||
#### API Services
|
||||
- `frontend/src/services/api.ts` - **autoGenerateIdeas()** function
|
||||
|
||||
#### Configuration
|
||||
- `frontend/src/config/pages/clusters.config.tsx` - Page configuration
|
||||
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
|
||||
|
||||
#### State Management
|
||||
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
|
||||
|
||||
#### Hooks
|
||||
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
|
||||
|
||||
---
|
||||
|
||||
### Auto Generate Content
|
||||
|
||||
#### Pages
|
||||
- `frontend/src/pages/Writer/Tasks.tsx` - **Main page component** with auto generate content functionality
|
||||
|
||||
#### API Services
|
||||
- `frontend/src/services/api.ts` - **autoGenerateContent()** function
|
||||
|
||||
#### Configuration
|
||||
- `frontend/src/config/pages/tasks.config.tsx` - Page configuration
|
||||
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
|
||||
|
||||
#### State Management
|
||||
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
|
||||
|
||||
#### Hooks
|
||||
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
|
||||
|
||||
---
|
||||
|
||||
### Auto Generate Images
|
||||
|
||||
#### Pages
|
||||
- `frontend/src/pages/Writer/Tasks.tsx` - **Main page component** with auto generate images functionality
|
||||
|
||||
#### API Services
|
||||
- `frontend/src/services/api.ts` - **autoGenerateImages()** function
|
||||
|
||||
#### Configuration
|
||||
- `frontend/src/config/pages/tasks.config.tsx` - Page configuration
|
||||
- `frontend/src/config/pages/images.config.tsx` - Image-related configuration
|
||||
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
|
||||
|
||||
#### State Management
|
||||
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
|
||||
|
||||
#### Hooks
|
||||
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
|
||||
|
||||
---
|
||||
|
||||
## Shared/Common Files
|
||||
|
||||
### AI Framework (Backend)
|
||||
- `backend/igny8_core/ai/__init__.py` - AI module initialization and auto-registration
|
||||
- `backend/igny8_core/ai/models.py` - AITaskLog model for unified logging
|
||||
- `backend/igny8_core/ai/types.py` - Shared dataclasses and types
|
||||
- `backend/igny8_core/ai/admin.py` - Admin interface for AITaskLog
|
||||
|
||||
### Progress Tracking (Backend)
|
||||
- `backend/igny8_core/modules/system/integration_views.py` - **task_progress()** endpoint
|
||||
|
||||
### Progress Tracking (Frontend)
|
||||
- `frontend/src/components/common/ProgressModal.tsx` - Progress modal component
|
||||
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook
|
||||
|
||||
### Common Components (Frontend)
|
||||
- `frontend/src/components/common/FormModal.tsx` - Form modal used in AI function pages
|
||||
|
||||
---
|
||||
|
||||
## Summary by Function
|
||||
|
||||
### Auto Cluster Keywords
|
||||
**Backend**: 15 files
|
||||
**Frontend**: 5 files
|
||||
**Total**: 20 files
|
||||
|
||||
### Auto Generate Ideas
|
||||
**Backend**: 7 files
|
||||
**Frontend**: 5 files
|
||||
**Total**: 12 files
|
||||
|
||||
### Auto Generate Content
|
||||
**Backend**: 7 files
|
||||
**Frontend**: 5 files
|
||||
**Total**: 12 files
|
||||
|
||||
### Auto Generate Images
|
||||
**Backend**: 7 files
|
||||
**Frontend**: 5 files
|
||||
**Total**: 12 files
|
||||
|
||||
### Shared/Common
|
||||
**Backend**: 4 files
|
||||
**Frontend**: 3 files
|
||||
**Total**: 7 files
|
||||
|
||||
---
|
||||
|
||||
## Key Files (Most Important)
|
||||
|
||||
### Backend Core Files
|
||||
1. `backend/igny8_core/ai/functions/auto_cluster.py` - Auto cluster AI function
|
||||
2. `backend/igny8_core/utils/ai_processor.py` - Unified AI processor (all functions)
|
||||
3. `backend/igny8_core/modules/planner/views.py` - Auto cluster & ideas API endpoints
|
||||
4. `backend/igny8_core/modules/writer/views.py` - Content & images API endpoints
|
||||
5. `backend/igny8_core/modules/planner/tasks.py` - Planner Celery tasks
|
||||
6. `backend/igny8_core/modules/writer/tasks.py` - Writer Celery tasks
|
||||
7. `backend/igny8_core/ai/tasks.py` - Unified AI task entrypoint
|
||||
|
||||
### Frontend Core Files
|
||||
1. `frontend/src/pages/Planner/Keywords.tsx` - Auto cluster UI
|
||||
2. `frontend/src/pages/Planner/Clusters.tsx` - Auto generate ideas UI
|
||||
3. `frontend/src/pages/Writer/Tasks.tsx` - Content & images generation UI
|
||||
4. `frontend/src/services/api.ts` - All API functions
|
||||
5. `frontend/src/hooks/useProgressModal.ts` - Progress tracking hook
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-01-XX
|
||||
|
||||
191
docs/AI-NEW-DESIGN/STAGE1-REORGANIZATION-COMPLETE.md
Normal file
191
docs/AI-NEW-DESIGN/STAGE1-REORGANIZATION-COMPLETE.md
Normal file
@@ -0,0 +1,191 @@
|
||||
# Stage 1 - AI Folder Structure & Functional Split - COMPLETE ✅
|
||||
|
||||
## Summary
|
||||
|
||||
Successfully reorganized the AI backend into a clean, modular structure where every AI function lives inside its own file within `/ai/functions/`.
|
||||
|
||||
## ✅ Completed Deliverables
|
||||
|
||||
### 1. Folder Structure Created
|
||||
|
||||
```
|
||||
backend/igny8_core/ai/
|
||||
├── functions/
|
||||
│ ├── __init__.py ✅
|
||||
│ ├── auto_cluster.py ✅
|
||||
│ ├── generate_ideas.py ✅
|
||||
│ ├── generate_content.py ✅
|
||||
│ └── generate_images.py ✅
|
||||
├── ai_core.py ✅ (Shared operations)
|
||||
├── validators.py ✅ (Consolidated validation)
|
||||
├── constants.py ✅ (Model pricing, valid models)
|
||||
├── engine.py ✅ (Updated to use AICore)
|
||||
├── tracker.py ✅ (Existing)
|
||||
├── base.py ✅ (Existing)
|
||||
├── processor.py ✅ (Existing wrapper)
|
||||
├── registry.py ✅ (Updated with new functions)
|
||||
└── __init__.py ✅ (Updated exports)
|
||||
```
|
||||
|
||||
### 2. Shared Modules Created
|
||||
|
||||
#### `ai_core.py`
|
||||
- **Purpose**: Shared operations for all AI functions
|
||||
- **Features**:
|
||||
- API call construction (`call_openai`)
|
||||
- Model selection (`get_model`, `get_api_key`)
|
||||
- Response parsing (`extract_json`)
|
||||
- Image generation (`generate_image`)
|
||||
- Cost calculation (`calculate_cost`)
|
||||
- **Status**: ✅ Complete
|
||||
|
||||
#### `validators.py`
|
||||
- **Purpose**: Consolidated validation logic
|
||||
- **Functions**:
|
||||
- `validate_ids()` - Base ID validation
|
||||
- `validate_keywords_exist()` - Keyword existence check
|
||||
- `validate_cluster_limits()` - Plan limit checks
|
||||
- `validate_cluster_exists()` - Cluster existence
|
||||
- `validate_tasks_exist()` - Task existence
|
||||
- `validate_api_key()` - API key validation
|
||||
- `validate_model()` - Model validation
|
||||
- `validate_image_size()` - Image size validation
|
||||
- **Status**: ✅ Complete
|
||||
|
||||
#### `constants.py`
|
||||
- **Purpose**: AI-related constants
|
||||
- **Constants**:
|
||||
- `MODEL_RATES` - Text model pricing
|
||||
- `IMAGE_MODEL_RATES` - Image model pricing
|
||||
- `VALID_OPENAI_IMAGE_MODELS` - Valid image models
|
||||
- `VALID_SIZES_BY_MODEL` - Valid sizes per model
|
||||
- `DEFAULT_AI_MODEL` - Default model name
|
||||
- `JSON_MODE_MODELS` - Models supporting JSON mode
|
||||
- **Status**: ✅ Complete
|
||||
|
||||
### 3. Function Files Created
|
||||
|
||||
#### `functions/auto_cluster.py`
|
||||
- **Status**: ✅ Updated to use new validators and AICore
|
||||
- **Changes**:
|
||||
- Uses `validate_ids()`, `validate_keywords_exist()`, `validate_cluster_limits()` from validators
|
||||
- Uses `AICore.extract_json()` for JSON parsing
|
||||
- Maintains backward compatibility
|
||||
|
||||
#### `functions/generate_ideas.py`
|
||||
- **Status**: ✅ Created
|
||||
- **Features**:
|
||||
- `GenerateIdeasFunction` class (BaseAIFunction)
|
||||
- `generate_ideas_core()` legacy function for backward compatibility
|
||||
- Uses AICore for API calls
|
||||
- Uses validators for validation
|
||||
|
||||
#### `functions/generate_content.py`
|
||||
- **Status**: ✅ Created
|
||||
- **Features**:
|
||||
- `GenerateContentFunction` class (BaseAIFunction)
|
||||
- `generate_content_core()` legacy function for backward compatibility
|
||||
- Uses AICore for API calls
|
||||
- Uses validators for validation
|
||||
|
||||
#### `functions/generate_images.py`
|
||||
- **Status**: ✅ Created
|
||||
- **Features**:
|
||||
- `GenerateImagesFunction` class (BaseAIFunction)
|
||||
- `generate_images_core()` legacy function for backward compatibility
|
||||
- Uses AICore for image generation
|
||||
- Uses validators for validation
|
||||
|
||||
### 4. Import Paths Updated
|
||||
|
||||
#### Updated Files:
|
||||
- ✅ `modules/planner/views.py` - Uses `generate_ideas_core` from new location
|
||||
- ✅ `modules/planner/tasks.py` - Imports `generate_ideas_core` from new location
|
||||
- ✅ `modules/writer/tasks.py` - Imports `generate_content_core` and `generate_images_core` from new locations
|
||||
- ✅ `ai/engine.py` - Uses `AICore` instead of `AIProcessor`
|
||||
- ✅ `ai/functions/auto_cluster.py` - Uses new validators and AICore
|
||||
- ✅ `ai/registry.py` - Registered all new functions
|
||||
- ✅ `ai/__init__.py` - Exports all new modules
|
||||
|
||||
### 5. Dependencies Verified
|
||||
|
||||
#### No Circular Dependencies ✅
|
||||
- Functions depend on: `ai_core`, `validators`, `constants`, `base`
|
||||
- `ai_core` depends on: `utils.ai_processor` (legacy, will be refactored later)
|
||||
- `validators` depends on: `constants`, models
|
||||
- `engine` depends on: `ai_core`, `base`, `tracker`
|
||||
- All imports are clean and modular
|
||||
|
||||
#### Modular Structure ✅
|
||||
- Each function file is self-contained
|
||||
- Shared logic in `ai_core.py`
|
||||
- Validation logic in `validators.py`
|
||||
- Constants in `constants.py`
|
||||
- No scattered or duplicated logic
|
||||
|
||||
## 📋 File Structure Details
|
||||
|
||||
### Core AI Modules
|
||||
|
||||
| File | Purpose | Dependencies |
|
||||
|------|---------|--------------|
|
||||
| `ai_core.py` | Shared AI operations | `utils.ai_processor` (legacy) |
|
||||
| `validators.py` | All validation logic | `constants`, models |
|
||||
| `constants.py` | AI constants | None |
|
||||
| `engine.py` | Execution orchestrator | `ai_core`, `base`, `tracker` |
|
||||
| `base.py` | Base function class | None |
|
||||
| `tracker.py` | Progress/step tracking | None |
|
||||
| `registry.py` | Function registry | `base`, function modules |
|
||||
|
||||
### Function Files
|
||||
|
||||
| File | Function Class | Legacy Function | Status |
|
||||
|------|----------------|-----------------|--------|
|
||||
| `auto_cluster.py` | `AutoClusterFunction` | N/A (uses engine) | ✅ Updated |
|
||||
| `generate_ideas.py` | `GenerateIdeasFunction` | `generate_ideas_core()` | ✅ Created |
|
||||
| `generate_content.py` | `GenerateContentFunction` | `generate_content_core()` | ✅ Created |
|
||||
| `generate_images.py` | `GenerateImagesFunction` | `generate_images_core()` | ✅ Created |
|
||||
|
||||
## 🔄 Import Path Changes
|
||||
|
||||
### Old Imports (Still work, but deprecated)
|
||||
```python
|
||||
from igny8_core.utils.ai_processor import AIProcessor
|
||||
from igny8_core.modules.planner.tasks import _generate_single_idea_core
|
||||
```
|
||||
|
||||
### New Imports (Recommended)
|
||||
```python
|
||||
from igny8_core.ai.functions.generate_ideas import generate_ideas_core
|
||||
from igny8_core.ai.functions.generate_content import generate_content_core
|
||||
from igny8_core.ai.functions.generate_images import generate_images_core
|
||||
from igny8_core.ai.ai_core import AICore
|
||||
from igny8_core.ai.validators import validate_ids, validate_cluster_limits
|
||||
from igny8_core.ai.constants import MODEL_RATES, DEFAULT_AI_MODEL
|
||||
```
|
||||
|
||||
## ✅ Verification Checklist
|
||||
|
||||
- [x] All function files created in `ai/functions/`
|
||||
- [x] Shared modules (`ai_core`, `validators`, `constants`) created
|
||||
- [x] No circular dependencies
|
||||
- [x] All imports updated in views and tasks
|
||||
- [x] Functions registered in registry
|
||||
- [x] `__init__.py` files updated
|
||||
- [x] Backward compatibility maintained (legacy functions still work)
|
||||
- [x] No linting errors
|
||||
- [x] Structure matches required layout
|
||||
|
||||
## 🎯 Next Steps (Future Stages)
|
||||
|
||||
- **Stage 2**: Inject tracker into all functions
|
||||
- **Stage 3**: Simplify logging
|
||||
- **Stage 4**: Clean up legacy code
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
- Legacy `AIProcessor` from `utils.ai_processor` is still used by `ai_core.py` as a wrapper
|
||||
- This will be refactored in later stages
|
||||
- All existing API endpoints continue to work
|
||||
- No functional changes - only structural reorganization
|
||||
|
||||
220
docs/AI-NEW-DESIGN/STAGE2-EXECUTION-LOGGING-COMPLETE.md
Normal file
220
docs/AI-NEW-DESIGN/STAGE2-EXECUTION-LOGGING-COMPLETE.md
Normal file
@@ -0,0 +1,220 @@
|
||||
# Stage 2 - AI Execution & Logging Layer - COMPLETE ✅
|
||||
|
||||
## Summary
|
||||
|
||||
Successfully created a centralized, consistent, and traceable execution layer for all AI requests with unified request handler and clean console-based logging.
|
||||
|
||||
## ✅ Completed Deliverables
|
||||
|
||||
### 1. Centralized Execution in `ai_core.py`
|
||||
|
||||
#### `run_ai_request()` Method
|
||||
- **Purpose**: Single entry point for all AI text generation requests
|
||||
- **Features**:
|
||||
- Step-by-step console logging with `print()` statements
|
||||
- Standardized request payload construction
|
||||
- Error handling with detailed logging
|
||||
- Token counting and cost calculation
|
||||
- Rate limit detection and logging
|
||||
- Timeout handling
|
||||
- JSON mode auto-enablement for supported models
|
||||
|
||||
#### Console Logging Format
|
||||
```
|
||||
[AI][function_name] Step 1: Preparing request...
|
||||
[AI][function_name] Step 2: Using model: gpt-4o
|
||||
[AI][function_name] Step 3: Auto-enabled JSON mode for gpt-4o
|
||||
[AI][function_name] Step 4: Prompt length: 1234 characters
|
||||
[AI][function_name] Step 5: Request payload prepared (model=gpt-4o, max_tokens=4000, temp=0.7)
|
||||
[AI][function_name] Step 6: Sending request to OpenAI API...
|
||||
[AI][function_name] Step 7: Received response in 2.34s (status=200)
|
||||
[AI][function_name] Step 8: Received 150 tokens (input: 50, output: 100)
|
||||
[AI][function_name] Step 9: Content length: 450 characters
|
||||
[AI][function_name] Step 10: Cost calculated: $0.000123
|
||||
[AI][function_name][Success] Request completed successfully
|
||||
```
|
||||
|
||||
#### Error Logging Format
|
||||
```
|
||||
[AI][function_name][Error] OpenAI Rate Limit - waiting 60s
|
||||
[AI][function_name][Error] HTTP 429 error: Rate limit exceeded (Rate limit - retry after 60s)
|
||||
[AI][function_name][Error] Request timeout (60s exceeded)
|
||||
[AI][function_name][Error] Failed to parse JSON response: ...
|
||||
```
|
||||
|
||||
### 2. Image Generation with Logging
|
||||
|
||||
#### `generate_image()` Method
|
||||
- **Purpose**: Centralized image generation with console logging
|
||||
- **Features**:
|
||||
- Supports OpenAI DALL-E and Runware
|
||||
- Model and size validation
|
||||
- Step-by-step console logging
|
||||
- Error handling with detailed messages
|
||||
- Cost calculation
|
||||
|
||||
#### Console Logging Format
|
||||
```
|
||||
[AI][generate_images] Step 1: Preparing image generation request...
|
||||
[AI][generate_images] Provider: OpenAI
|
||||
[AI][generate_images] Step 2: Using model: dall-e-3, size: 1024x1024
|
||||
[AI][generate_images] Step 3: Sending request to OpenAI Images API...
|
||||
[AI][generate_images] Step 4: Received response in 5.67s (status=200)
|
||||
[AI][generate_images] Step 5: Image generated successfully
|
||||
[AI][generate_images] Step 6: Cost: $0.0400
|
||||
[AI][generate_images][Success] Image generation completed
|
||||
```
|
||||
|
||||
### 3. Updated All Function Files
|
||||
|
||||
#### `functions/auto_cluster.py`
|
||||
- ✅ Uses `AICore.extract_json()` for JSON parsing
|
||||
- ✅ Engine calls `run_ai_request()` (via engine.py)
|
||||
|
||||
#### `functions/generate_ideas.py`
|
||||
- ✅ Updated `generate_ideas_core()` to use `run_ai_request()`
|
||||
- ✅ Console logging enabled with function name
|
||||
|
||||
#### `functions/generate_content.py`
|
||||
- ✅ Updated `generate_content_core()` to use `run_ai_request()`
|
||||
- ✅ Console logging enabled with function name
|
||||
|
||||
#### `functions/generate_images.py`
|
||||
- ✅ Updated to use `run_ai_request()` for prompt extraction
|
||||
- ✅ Updated to use `generate_image()` with logging
|
||||
- ✅ Console logging enabled
|
||||
|
||||
### 4. Updated Engine
|
||||
|
||||
#### `engine.py`
|
||||
- ✅ Updated to use `run_ai_request()` instead of `call_openai()`
|
||||
- ✅ Passes function name for logging context
|
||||
- ✅ Maintains backward compatibility
|
||||
|
||||
### 5. Deprecated Old Code
|
||||
|
||||
#### `processor.py`
|
||||
- ✅ Marked as DEPRECATED
|
||||
- ✅ Redirects all calls to `AICore`
|
||||
- ✅ Kept for backward compatibility only
|
||||
- ✅ All methods now use `AICore` internally
|
||||
|
||||
### 6. Edge Case Handling
|
||||
|
||||
#### Implemented in `run_ai_request()`:
|
||||
- ✅ **API Key Validation**: Logs error if not configured
|
||||
- ✅ **Prompt Length**: Logs character count
|
||||
- ✅ **Rate Limits**: Detects and logs retry-after time
|
||||
- ✅ **Timeouts**: Handles 60s timeout with clear error
|
||||
- ✅ **JSON Parsing Errors**: Logs decode errors with context
|
||||
- ✅ **Empty Responses**: Validates content exists
|
||||
- ✅ **Token Overflow**: Max tokens enforced
|
||||
- ✅ **Model Validation**: Auto-selects JSON mode for supported models
|
||||
|
||||
### 7. Standardized Request Schema
|
||||
|
||||
#### OpenAI Request Payload
|
||||
```python
|
||||
{
|
||||
"model": "gpt-4o",
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 4000,
|
||||
"response_format": {"type": "json_object"} # Auto-enabled for supported models
|
||||
}
|
||||
```
|
||||
|
||||
#### All Functions Use Same Logic:
|
||||
- Model selection (account default or override)
|
||||
- JSON mode auto-enablement
|
||||
- Token limits
|
||||
- Temperature settings
|
||||
- Error handling
|
||||
|
||||
### 8. Test Script Created
|
||||
|
||||
#### `ai/tests/test_run.py`
|
||||
- ✅ Test script for all AI functions
|
||||
- ✅ Tests `run_ai_request()` directly
|
||||
- ✅ Tests JSON extraction
|
||||
- ✅ Placeholder tests for all functions
|
||||
- ✅ Can be run standalone to verify logging
|
||||
|
||||
## 📋 File Changes Summary
|
||||
|
||||
| File | Changes | Status |
|
||||
|------|---------|--------|
|
||||
| `ai_core.py` | Complete rewrite with `run_ai_request()` and console logging | ✅ Complete |
|
||||
| `engine.py` | Updated to use `run_ai_request()` | ✅ Complete |
|
||||
| `processor.py` | Marked deprecated, redirects to AICore | ✅ Complete |
|
||||
| `functions/auto_cluster.py` | Uses AICore methods | ✅ Complete |
|
||||
| `functions/generate_ideas.py` | Uses `run_ai_request()` | ✅ Complete |
|
||||
| `functions/generate_content.py` | Uses `run_ai_request()` | ✅ Complete |
|
||||
| `functions/generate_images.py` | Uses `run_ai_request()` and `generate_image()` | ✅ Complete |
|
||||
| `tests/test_run.py` | Test script created | ✅ Complete |
|
||||
|
||||
## 🔄 Migration Path
|
||||
|
||||
### Old Code (Deprecated)
|
||||
```python
|
||||
from igny8_core.utils.ai_processor import AIProcessor
|
||||
processor = AIProcessor(account=account)
|
||||
result = processor._call_openai(prompt, model=model)
|
||||
```
|
||||
|
||||
### New Code (Recommended)
|
||||
```python
|
||||
from igny8_core.ai.ai_core import AICore
|
||||
ai_core = AICore(account=account)
|
||||
result = ai_core.run_ai_request(
|
||||
prompt=prompt,
|
||||
model=model,
|
||||
function_name='my_function'
|
||||
)
|
||||
```
|
||||
|
||||
## ✅ Verification Checklist
|
||||
|
||||
- [x] `run_ai_request()` created with console logging
|
||||
- [x] All function files updated to use `run_ai_request()`
|
||||
- [x] Engine updated to use `run_ai_request()`
|
||||
- [x] Old processor code deprecated
|
||||
- [x] Edge cases handled with logging
|
||||
- [x] Request schema standardized
|
||||
- [x] Test script created
|
||||
- [x] No linting errors
|
||||
- [x] Backward compatibility maintained
|
||||
|
||||
## 🎯 Benefits Achieved
|
||||
|
||||
1. **Centralized Execution**: All AI requests go through one method
|
||||
2. **Consistent Logging**: Every request logs steps to console
|
||||
3. **Better Debugging**: Clear step-by-step visibility
|
||||
4. **Error Handling**: Comprehensive error detection and logging
|
||||
5. **Reduced Duplication**: No scattered AI call logic
|
||||
6. **Easy Testing**: Single point to test/mock
|
||||
7. **Future Ready**: Easy to add retry logic, backoff, etc.
|
||||
|
||||
## 📝 Console Output Example
|
||||
|
||||
When running any AI function, you'll see:
|
||||
```
|
||||
[AI][generate_ideas] Step 1: Preparing request...
|
||||
[AI][generate_ideas] Step 2: Using model: gpt-4o
|
||||
[AI][generate_ideas] Step 3: Auto-enabled JSON mode for gpt-4o
|
||||
[AI][generate_ideas] Step 4: Prompt length: 2345 characters
|
||||
[AI][generate_ideas] Step 5: Request payload prepared (model=gpt-4o, max_tokens=4000, temp=0.7)
|
||||
[AI][generate_ideas] Step 6: Sending request to OpenAI API...
|
||||
[AI][generate_ideas] Step 7: Received response in 3.45s (status=200)
|
||||
[AI][generate_ideas] Step 8: Received 250 tokens (input: 100, output: 150)
|
||||
[AI][generate_ideas] Step 9: Content length: 600 characters
|
||||
[AI][generate_ideas] Step 10: Cost calculated: $0.000250
|
||||
[AI][generate_ideas][Success] Request completed successfully
|
||||
```
|
||||
|
||||
## 🚀 Next Steps (Future Stages)
|
||||
|
||||
- **Stage 3**: Simplify logging (optional - console logging already implemented)
|
||||
- **Stage 4**: Clean up legacy code (remove old processor completely)
|
||||
- **Future**: Add retry logic, exponential backoff, request queuing
|
||||
|
||||
171
docs/AI-NEW-DESIGN/STAGE3-LOGGING-COMPLETE.md
Normal file
171
docs/AI-NEW-DESIGN/STAGE3-LOGGING-COMPLETE.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# Stage 3 - Clean Logging, Unified Debug Flow & Step Traceability - COMPLETE ✅
|
||||
|
||||
## Summary
|
||||
|
||||
Successfully replaced all fragmented or frontend-based debugging systems with a consistent, lightweight backend-only logging flow. All AI activity is now tracked via structured console messages with no UI panels, no Zustand state, and no silent failures.
|
||||
|
||||
## ✅ Completed Deliverables
|
||||
|
||||
### 1. ConsoleStepTracker Created
|
||||
|
||||
#### `tracker.py` - ConsoleStepTracker Class
|
||||
- **Purpose**: Lightweight console-based step tracker for AI functions
|
||||
- **Features**:
|
||||
- Logs each step to console with timestamps and clear labels
|
||||
- Only logs if `DEBUG_MODE` is True
|
||||
- Standardized phase methods: `init()`, `prep()`, `ai_call()`, `parse()`, `save()`, `done()`
|
||||
- Error logging: `error()`, `timeout()`, `rate_limit()`, `malformed_json()`
|
||||
- Retry logging: `retry()`
|
||||
- Duration tracking
|
||||
|
||||
#### Log Format
|
||||
```
|
||||
[HH:MM:SS] [function_name] [PHASE] message
|
||||
[HH:MM:SS] [function_name] [PHASE] ✅ success message
|
||||
[HH:MM:SS] [function_name] [PHASE] [ERROR] error message
|
||||
[function_name] === AI Task Complete ===
|
||||
```
|
||||
|
||||
### 2. DEBUG_MODE Constant Added
|
||||
|
||||
#### `constants.py`
|
||||
- Added `DEBUG_MODE = True` constant
|
||||
- Controls all console logging
|
||||
- Can be set to `False` in production to disable verbose logging
|
||||
- All print statements check `DEBUG_MODE` before logging
|
||||
|
||||
### 3. Integrated Tracker into AI Functions
|
||||
|
||||
#### `generate_ideas.py`
|
||||
- ✅ Added `ConsoleStepTracker` initialization
|
||||
- ✅ Logs: INIT → PREP → AI_CALL → PARSE → SAVE → DONE
|
||||
- ✅ Error handling with tracker.error()
|
||||
- ✅ Passes tracker to `run_ai_request()`
|
||||
|
||||
#### `ai_core.py`
|
||||
- ✅ Updated `run_ai_request()` to accept optional tracker parameter
|
||||
- ✅ All logging now uses tracker methods
|
||||
- ✅ Replaced all `print()` statements with tracker calls
|
||||
- ✅ Standardized error logging format
|
||||
|
||||
### 4. Frontend Debug Systems Deprecated
|
||||
|
||||
#### `TablePageTemplate.tsx`
|
||||
- ✅ Commented out `AIRequestLogsSection` component
|
||||
- ✅ Commented out import of `useAIRequestLogsStore`
|
||||
- ✅ Added deprecation comments
|
||||
|
||||
#### Frontend Store (Kept for now, but unused)
|
||||
- `aiRequestLogsStore.ts` - Still exists but no longer used
|
||||
- All calls to `addLog`, `updateLog`, `addRequestStep`, `addResponseStep` are deprecated
|
||||
|
||||
### 5. Error Standardization
|
||||
|
||||
#### Standardized Error Format
|
||||
```
|
||||
[ERROR] {function_name}: {error_type} – {message}
|
||||
```
|
||||
|
||||
#### Error Types
|
||||
- `ConfigurationError` - API key not configured
|
||||
- `ValidationError` - Input validation failed
|
||||
- `HTTPError` - HTTP request failed
|
||||
- `Timeout` - Request timeout
|
||||
- `RateLimit` - Rate limit hit
|
||||
- `MalformedJSON` - JSON parsing failed
|
||||
- `EmptyResponse` - No content in response
|
||||
- `ParseError` - Response parsing failed
|
||||
- `Exception` - Unexpected exception
|
||||
|
||||
### 6. Example Console Output
|
||||
|
||||
#### Successful Execution
|
||||
```
|
||||
[14:23:45] [generate_ideas] [INIT] Task started
|
||||
[14:23:45] [generate_ideas] [PREP] Loading account and cluster data...
|
||||
[14:23:45] [generate_ideas] [PREP] Validating input...
|
||||
[14:23:45] [generate_ideas] [PREP] Loading cluster with keywords...
|
||||
[14:23:45] [generate_ideas] [PREP] Building prompt...
|
||||
[14:23:45] [generate_ideas] [AI_CALL] Preparing request...
|
||||
[14:23:45] [generate_ideas] [AI_CALL] Using model: gpt-4o
|
||||
[14:23:45] [generate_ideas] [AI_CALL] Auto-enabled JSON mode for gpt-4o
|
||||
[14:23:45] [generate_ideas] [AI_CALL] Prompt length: 1234 characters
|
||||
[14:23:45] [generate_ideas] [AI_CALL] Request payload prepared (model=gpt-4o, max_tokens=4000, temp=0.7)
|
||||
[14:23:45] [generate_ideas] [AI_CALL] Sending request to OpenAI API...
|
||||
[14:23:48] [generate_ideas] [AI_CALL] Received response in 2.34s (status=200)
|
||||
[14:23:48] [generate_ideas] [PARSE] Received 250 tokens (input: 100, output: 150)
|
||||
[14:23:48] [generate_ideas] [PARSE] Content length: 600 characters
|
||||
[14:23:48] [generate_ideas] [PARSE] Cost calculated: $0.000250
|
||||
[14:23:48] [generate_ideas] [DONE] ✅ Request completed successfully (Duration: 3.12s)
|
||||
[14:23:48] [generate_ideas] [PARSE] Parsing AI response...
|
||||
[14:23:48] [generate_ideas] [PARSE] Parsed 1 idea(s)
|
||||
[14:23:48] [generate_ideas] [SAVE] Saving idea to database...
|
||||
[14:23:48] [generate_ideas] [SAVE] Saved 1 idea(s)
|
||||
[14:23:48] [generate_ideas] [DONE] ✅ Idea 'My Great Idea' created successfully (Duration: 3.15s)
|
||||
[generate_ideas] === AI Task Complete ===
|
||||
```
|
||||
|
||||
#### Error Execution
|
||||
```
|
||||
[14:25:10] [generate_ideas] [INIT] Task started
|
||||
[14:25:10] [generate_ideas] [PREP] Loading account and cluster data...
|
||||
[14:25:10] [generate_ideas] [PREP] Validating input...
|
||||
[14:25:10] [generate_ideas] [PREP] [ERROR] ValidationError – No cluster found
|
||||
```
|
||||
|
||||
## 📋 File Changes Summary
|
||||
|
||||
| File | Changes | Status |
|
||||
|------|---------|--------|
|
||||
| `tracker.py` | Added `ConsoleStepTracker` class | ✅ Complete |
|
||||
| `constants.py` | Added `DEBUG_MODE` constant | ✅ Complete |
|
||||
| `ai_core.py` | Updated to use tracker, removed print() statements | ✅ Complete |
|
||||
| `generate_ideas.py` | Integrated ConsoleStepTracker | ✅ Complete |
|
||||
| `TablePageTemplate.tsx` | Commented out frontend debug UI | ✅ Complete |
|
||||
|
||||
## 🔄 Remaining Work
|
||||
|
||||
### Functions Still Need Tracker Integration
|
||||
- [ ] `auto_cluster.py` - Add tracker to core function
|
||||
- [ ] `generate_content.py` - Add tracker to core function
|
||||
- [ ] `generate_images.py` - Add tracker to core function
|
||||
|
||||
### Image Generation Logging
|
||||
- [ ] Update `_generate_image_openai()` to use tracker
|
||||
- [ ] Update `_generate_image_runware()` to use tracker
|
||||
- [ ] Replace all print() statements with tracker calls
|
||||
|
||||
### Frontend Cleanup
|
||||
- [ ] Remove or fully comment out `AIRequestLogsSection` function body
|
||||
- [ ] Remove unused imports from `api.ts` and `useProgressModal.ts`
|
||||
- [ ] Optionally delete `aiRequestLogsStore.ts` (or keep for reference)
|
||||
|
||||
## ✅ Verification Checklist
|
||||
|
||||
- [x] ConsoleStepTracker created with all methods
|
||||
- [x] DEBUG_MODE constant added
|
||||
- [x] `run_ai_request()` updated to use tracker
|
||||
- [x] `generate_ideas.py` integrated with tracker
|
||||
- [x] Frontend debug UI commented out
|
||||
- [x] Error logging standardized
|
||||
- [ ] All function files integrated (partial)
|
||||
- [ ] Image generation logging updated (pending)
|
||||
- [ ] All print() statements replaced (partial)
|
||||
|
||||
## 🎯 Benefits Achieved
|
||||
|
||||
1. **Unified Logging**: All AI functions use same logging format
|
||||
2. **Backend-Only**: No frontend state management needed
|
||||
3. **Production Ready**: Can disable logs via DEBUG_MODE
|
||||
4. **Clear Traceability**: Every step visible in console
|
||||
5. **Error Visibility**: All errors clearly labeled and logged
|
||||
6. **No Silent Failures**: Every failure prints its cause
|
||||
|
||||
## 📝 Next Steps
|
||||
|
||||
1. Complete tracker integration in remaining functions
|
||||
2. Update image generation methods
|
||||
3. Remove remaining print() statements
|
||||
4. Test end-to-end with all four AI flows
|
||||
5. Optionally clean up frontend debug code completely
|
||||
|
||||
220
docs/AI-NEW-DESIGN/STAGE4-PROMPT-REGISTRY-COMPLETE.md
Normal file
220
docs/AI-NEW-DESIGN/STAGE4-PROMPT-REGISTRY-COMPLETE.md
Normal file
@@ -0,0 +1,220 @@
|
||||
# Stage 4 - Prompt Registry, Model Unification, and Final Function Hooks - COMPLETE ✅
|
||||
|
||||
## Summary
|
||||
|
||||
Successfully created a centralized prompt registry system, unified model configurations, and standardized all AI function execution with clean, minimal function files.
|
||||
|
||||
## ✅ Completed Deliverables
|
||||
|
||||
### 1. Prompt Registry System Created
|
||||
|
||||
#### `ai/prompts.py` - PromptRegistry Class
|
||||
- **Purpose**: Centralized prompt management with hierarchical resolution
|
||||
- **Features**:
|
||||
- Hierarchical prompt resolution:
|
||||
1. Task-level `prompt_override` (if exists)
|
||||
2. DB prompt for (account, function)
|
||||
3. Default fallback from registry
|
||||
- Supports both `.format()` style and `[IGNY8_*]` placeholder replacement
|
||||
- Function-to-prompt-type mapping
|
||||
- Convenience methods: `get_image_prompt_template()`, `get_negative_prompt()`
|
||||
|
||||
#### Prompt Resolution Priority
|
||||
```python
|
||||
# Priority 1: Task override
|
||||
if task.prompt_override:
|
||||
use task.prompt_override
|
||||
|
||||
# Priority 2: DB prompt
|
||||
elif DB prompt for (account, function) exists:
|
||||
use DB prompt
|
||||
|
||||
# Priority 3: Default fallback
|
||||
else:
|
||||
use default from registry
|
||||
```
|
||||
|
||||
### 2. Model Configuration Centralized
|
||||
|
||||
#### `ai/settings.py` - MODEL_CONFIG
|
||||
- **Purpose**: Centralized model configurations for all AI functions
|
||||
- **Configurations**:
|
||||
```python
|
||||
MODEL_CONFIG = {
|
||||
"auto_cluster": {
|
||||
"model": "gpt-4o-mini",
|
||||
"max_tokens": 3000,
|
||||
"temperature": 0.7,
|
||||
"response_format": {"type": "json_object"},
|
||||
},
|
||||
"generate_ideas": {
|
||||
"model": "gpt-4.1",
|
||||
"max_tokens": 4000,
|
||||
"temperature": 0.7,
|
||||
"response_format": {"type": "json_object"},
|
||||
},
|
||||
"generate_content": {
|
||||
"model": "gpt-4.1",
|
||||
"max_tokens": 8000,
|
||||
"temperature": 0.7,
|
||||
"response_format": None, # Text output
|
||||
},
|
||||
"generate_images": {
|
||||
"model": "dall-e-3",
|
||||
"size": "1024x1024",
|
||||
"provider": "openai",
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
#### Helper Functions
|
||||
- `get_model_config(function_name)` - Get full config
|
||||
- `get_model(function_name)` - Get model name
|
||||
- `get_max_tokens(function_name)` - Get max tokens
|
||||
- `get_temperature(function_name)` - Get temperature
|
||||
|
||||
### 3. Updated All AI Functions
|
||||
|
||||
#### `functions/auto_cluster.py`
|
||||
- ✅ Uses `PromptRegistry.get_prompt()`
|
||||
- ✅ Uses `get_model_config()` for model settings
|
||||
- ✅ Removed direct `get_prompt_value()` calls
|
||||
|
||||
#### `functions/generate_ideas.py`
|
||||
- ✅ Uses `PromptRegistry.get_prompt()` with context
|
||||
- ✅ Uses `get_model_config()` for model settings
|
||||
- ✅ Clean prompt building with context variables
|
||||
|
||||
#### `functions/generate_content.py`
|
||||
- ✅ Uses `PromptRegistry.get_prompt()` with task support
|
||||
- ✅ Uses `get_model_config()` for model settings
|
||||
- ✅ Supports task-level prompt overrides
|
||||
|
||||
#### `functions/generate_images.py`
|
||||
- ✅ Uses `PromptRegistry.get_prompt()` for extraction
|
||||
- ✅ Uses `PromptRegistry.get_image_prompt_template()`
|
||||
- ✅ Uses `PromptRegistry.get_negative_prompt()`
|
||||
- ✅ Uses `get_model_config()` for model settings
|
||||
|
||||
### 4. Updated Engine
|
||||
|
||||
#### `engine.py`
|
||||
- ✅ Uses `get_model_config()` instead of `fn.get_model()`
|
||||
- ✅ Passes model config to `run_ai_request()`
|
||||
- ✅ Unified model selection across all functions
|
||||
|
||||
### 5. Standardized Response Format
|
||||
|
||||
All functions now return consistent format:
|
||||
```python
|
||||
{
|
||||
"success": True/False,
|
||||
"output": "HTML or image_url or data",
|
||||
"raw": raw_response_json, # Optional
|
||||
"meta": {
|
||||
"word_count": 1536, # For content
|
||||
"keywords": [...], # For clusters
|
||||
"model_used": "gpt-4.1",
|
||||
"tokens": 250,
|
||||
"cost": 0.000123
|
||||
},
|
||||
"error": None or error_message
|
||||
}
|
||||
```
|
||||
|
||||
## 📋 File Changes Summary
|
||||
|
||||
| File | Changes | Status |
|
||||
|------|---------|--------|
|
||||
| `prompts.py` | Created PromptRegistry class | ✅ Complete |
|
||||
| `settings.py` | Created MODEL_CONFIG and helpers | ✅ Complete |
|
||||
| `functions/auto_cluster.py` | Updated to use registry and settings | ✅ Complete |
|
||||
| `functions/generate_ideas.py` | Updated to use registry and settings | ✅ Complete |
|
||||
| `functions/generate_content.py` | Updated to use registry and settings | ✅ Complete |
|
||||
| `functions/generate_images.py` | Updated to use registry and settings | ✅ Complete |
|
||||
| `engine.py` | Updated to use model config | ✅ Complete |
|
||||
| `__init__.py` | Exported new modules | ✅ Complete |
|
||||
|
||||
## 🔄 Migration Path
|
||||
|
||||
### Old Code (Deprecated)
|
||||
```python
|
||||
from igny8_core.modules.system.utils import get_prompt_value, get_default_prompt
|
||||
prompt_template = get_prompt_value(account, 'clustering')
|
||||
prompt = prompt_template.replace('[IGNY8_KEYWORDS]', keywords_text)
|
||||
```
|
||||
|
||||
### New Code (Recommended)
|
||||
```python
|
||||
from igny8_core.ai.prompts import PromptRegistry
|
||||
from igny8_core.ai.settings import get_model_config
|
||||
|
||||
# Get prompt from registry
|
||||
prompt = PromptRegistry.get_prompt(
|
||||
function_name='auto_cluster',
|
||||
account=account,
|
||||
context={'KEYWORDS': keywords_text}
|
||||
)
|
||||
|
||||
# Get model config
|
||||
model_config = get_model_config('auto_cluster')
|
||||
```
|
||||
|
||||
## ✅ Verification Checklist
|
||||
|
||||
- [x] PromptRegistry created with hierarchical resolution
|
||||
- [x] MODEL_CONFIG created with all function configs
|
||||
- [x] All functions updated to use registry
|
||||
- [x] All functions updated to use model config
|
||||
- [x] Engine updated to use model config
|
||||
- [x] Response format standardized
|
||||
- [x] No direct prompt utility calls in functions
|
||||
- [x] Task-level overrides supported
|
||||
- [x] DB prompts supported
|
||||
- [x] Default fallbacks working
|
||||
|
||||
## 🎯 Benefits Achieved
|
||||
|
||||
1. **Centralized Prompts**: All prompts in one registry
|
||||
2. **Hierarchical Resolution**: Task → DB → Default
|
||||
3. **Model Unification**: All models configured in one place
|
||||
4. **Easy Customization**: Tenant admins can override prompts
|
||||
5. **Consistent Execution**: All functions use same pattern
|
||||
6. **Traceability**: Prompt source clearly identifiable
|
||||
7. **Minimal Functions**: Functions are clean and focused
|
||||
|
||||
## 📝 Prompt Source Traceability
|
||||
|
||||
Each prompt execution logs its source:
|
||||
- `[PROMPT] Using task-level prompt override for generate_content`
|
||||
- `[PROMPT] Using DB prompt for generate_ideas (account 123)`
|
||||
- `[PROMPT] Using default prompt for auto_cluster`
|
||||
|
||||
## 🚀 Final Structure
|
||||
|
||||
```
|
||||
/ai/
|
||||
├── functions/
|
||||
│ ├── auto_cluster.py ← Uses registry + settings
|
||||
│ ├── generate_ideas.py ← Uses registry + settings
|
||||
│ ├── generate_content.py ← Uses registry + settings
|
||||
│ └── generate_images.py ← Uses registry + settings
|
||||
├── prompts.py ← Prompt Registry ✅
|
||||
├── settings.py ← Model Configs ✅
|
||||
├── ai_core.py ← Unified execution ✅
|
||||
├── engine.py ← Uses settings ✅
|
||||
└── tracker.py ← Console logging ✅
|
||||
```
|
||||
|
||||
## ✅ Expected Outcomes Achieved
|
||||
|
||||
- ✅ All AI executions use common format
|
||||
- ✅ Prompt customization is dynamic and override-able
|
||||
- ✅ No duplication across AI functions
|
||||
- ✅ Every AI task has:
|
||||
- ✅ Clean inputs
|
||||
- ✅ Unified execution
|
||||
- ✅ Standard outputs
|
||||
- ✅ Clear error tracking
|
||||
- ✅ Prompt traceability
|
||||
|
||||
@@ -1,666 +0,0 @@
|
||||
# IGNY8 AI Functions Documentation
|
||||
|
||||
**Version:** 1.0
|
||||
**Last Updated:** 2025-01-XX
|
||||
**Purpose:** Complete documentation of all AI functions in the IGNY8 system, their workflows, dependencies, and integration points.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [AI Functions Overview](#ai-functions-overview)
|
||||
2. [AIProcessor Class](#aiprocessor-class)
|
||||
3. [Planner Module AI Functions](#planner-module-ai-functions)
|
||||
4. [Writer Module AI Functions](#writer-module-ai-functions)
|
||||
5. [System Module AI Functions](#system-module-ai-functions)
|
||||
6. [Progress Tracking](#progress-tracking)
|
||||
7. [Error Handling](#error-handling)
|
||||
8. [Dependencies](#dependencies)
|
||||
|
||||
---
|
||||
|
||||
## AI Functions Overview
|
||||
|
||||
The IGNY8 system includes **8 primary AI functions** across 3 modules:
|
||||
|
||||
1. **Planner Module (2 functions):**
|
||||
- Auto Cluster Keywords (`auto_cluster`)
|
||||
- Auto Generate Ideas (`auto_generate_ideas`)
|
||||
|
||||
2. **Writer Module (3 functions):**
|
||||
- Auto Generate Content (`auto_generate_content`)
|
||||
- Auto Generate Images (`auto_generate_images`)
|
||||
- Extract Image Prompts (`extract_image_prompts`)
|
||||
|
||||
3. **System Module (3 functions):**
|
||||
- Test OpenAI Integration (`test_openai`)
|
||||
- Test Runware Integration (`test_runware`)
|
||||
- Generate Image (Test) (`generate_image`)
|
||||
|
||||
All AI functions use the unified `AIProcessor` class as the core interface for AI operations.
|
||||
|
||||
---
|
||||
|
||||
## AIProcessor Class
|
||||
|
||||
**Location**: `backend/igny8_core/utils/ai_processor.py`
|
||||
|
||||
**Purpose**: Unified AI interface for all AI operations.
|
||||
|
||||
### Initialization
|
||||
|
||||
**Method**: `AIProcessor.__init__()`
|
||||
|
||||
**Parameters**:
|
||||
- `account` (optional): Account object for loading account-specific settings
|
||||
|
||||
**Dependencies**:
|
||||
- `IntegrationSettings` model (for API keys and model configuration)
|
||||
- Django settings (fallback for API keys)
|
||||
|
||||
**Initializes**:
|
||||
- `self.openai_api_key`: Loaded from IntegrationSettings or Django settings
|
||||
- `self.runware_api_key`: Loaded from IntegrationSettings or Django settings
|
||||
- `self.default_model`: Loaded from IntegrationSettings or Django settings
|
||||
- `self.model_rates`: Model pricing rates (per 1M tokens)
|
||||
- `self.image_model_rates`: Image model pricing rates (per image)
|
||||
|
||||
### Core Methods
|
||||
|
||||
#### cluster_keywords()
|
||||
**Purpose**: Cluster keywords using AI-based semantic similarity.
|
||||
|
||||
**Parameters**:
|
||||
- `keywords`: List of keyword dictionaries
|
||||
- `sector_name`: Optional sector name for context
|
||||
- `account`: Optional account object
|
||||
- `response_steps`: Optional list for step tracking
|
||||
- `progress_callback`: Optional callback function
|
||||
|
||||
**Dependencies**:
|
||||
- `AIPrompt` model (for clustering prompt template)
|
||||
- `get_default_prompt()` utility function
|
||||
- `_call_openai()` method
|
||||
- `_extract_json_from_response()` method
|
||||
|
||||
**Returns**: Dictionary with `clusters` array and `response_steps`
|
||||
|
||||
#### generate_ideas()
|
||||
**Purpose**: Generate content ideas for clusters using AI.
|
||||
|
||||
**Parameters**:
|
||||
- `clusters`: List of cluster dictionaries
|
||||
- `account`: Optional account object
|
||||
|
||||
**Dependencies**:
|
||||
- `AIPrompt` model (for ideas prompt template)
|
||||
- `get_default_prompt()` utility function
|
||||
- `_call_openai()` method
|
||||
- `_extract_json_from_response()` method
|
||||
|
||||
**Returns**: Dictionary with `ideas` array
|
||||
|
||||
#### generate_content()
|
||||
**Purpose**: Generate text content using OpenAI GPT models.
|
||||
|
||||
**Parameters**:
|
||||
- `prompt`: The prompt text
|
||||
- `model`: Optional model name (defaults to self.default_model)
|
||||
- `max_tokens`: Maximum tokens to generate
|
||||
- `temperature`: Sampling temperature
|
||||
|
||||
**Dependencies**:
|
||||
- `_call_openai()` method
|
||||
|
||||
**Returns**: Dictionary with `content`, `tokens_used`, `model`, `cost`, `error`
|
||||
|
||||
#### extract_image_prompts()
|
||||
**Purpose**: Extract image prompts from content using AI.
|
||||
|
||||
**Parameters**:
|
||||
- `content`: Article content text
|
||||
- `title`: Article title
|
||||
- `max_images`: Maximum number of images to extract prompts for
|
||||
- `account`: Optional account object
|
||||
|
||||
**Dependencies**:
|
||||
- `AIPrompt` model (for image prompt extraction template)
|
||||
- `get_default_prompt()` utility function
|
||||
- `_call_openai()` method
|
||||
- `_extract_json_from_response()` method
|
||||
|
||||
**Returns**: Dictionary with `featured_prompt` and `in_article_prompts` array
|
||||
|
||||
#### generate_image()
|
||||
**Purpose**: Generate images using OpenAI DALL-E or Runware.
|
||||
|
||||
**Parameters**:
|
||||
- `prompt`: Image generation prompt
|
||||
- `model`: Image model name (dall-e-3, dall-e-2, or runware model)
|
||||
- `size`: Image size (e.g., "1024x1024")
|
||||
- `n`: Number of images to generate
|
||||
- `account`: Optional account object
|
||||
|
||||
**Dependencies**:
|
||||
- `IntegrationSettings` model (for API keys)
|
||||
- OpenAI API or Runware API
|
||||
|
||||
**Returns**: Dictionary with `image_url`, `revised_prompt`, `cost`, `provider`, `model`
|
||||
|
||||
### Internal Methods
|
||||
|
||||
#### _call_openai()
|
||||
**Purpose**: Make HTTP request to OpenAI API.
|
||||
|
||||
**Parameters**:
|
||||
- `prompt`: The prompt text
|
||||
- `model`: Model name
|
||||
- `max_tokens`: Maximum tokens
|
||||
- `temperature`: Sampling temperature
|
||||
- `response_format`: Optional JSON response format
|
||||
- `response_steps`: Optional list for step tracking
|
||||
|
||||
**Dependencies**:
|
||||
- `requests` library
|
||||
- OpenAI API endpoint
|
||||
|
||||
**Returns**: Dictionary with `content`, `input_tokens`, `output_tokens`, `total_tokens`, `model`, `cost`, `error`, `api_id`
|
||||
|
||||
#### _extract_json_from_response()
|
||||
**Purpose**: Extract JSON from OpenAI response (handles various formats).
|
||||
|
||||
**Parameters**:
|
||||
- `response_text`: Raw response text from OpenAI
|
||||
|
||||
**Dependencies**:
|
||||
- `json` library
|
||||
- `re` library (for regex patterns)
|
||||
|
||||
**Returns**: Parsed JSON dictionary or None
|
||||
|
||||
#### _get_api_key()
|
||||
**Purpose**: Get API key from IntegrationSettings or Django settings.
|
||||
|
||||
**Parameters**:
|
||||
- `integration_type`: Type of integration ('openai' or 'runware')
|
||||
- `account`: Optional account object
|
||||
|
||||
**Dependencies**:
|
||||
- `IntegrationSettings` model
|
||||
- Django settings
|
||||
|
||||
**Returns**: API key string or None
|
||||
|
||||
#### _get_model()
|
||||
**Purpose**: Get default model from IntegrationSettings or Django settings.
|
||||
|
||||
**Parameters**:
|
||||
- `integration_type`: Type of integration ('openai')
|
||||
- `account`: Optional account object
|
||||
|
||||
**Dependencies**:
|
||||
- `IntegrationSettings` model
|
||||
- Django settings
|
||||
|
||||
**Returns**: Model name string or default
|
||||
|
||||
#### get_prompt()
|
||||
**Purpose**: Get prompt template from AIPrompt model or default.
|
||||
|
||||
**Parameters**:
|
||||
- `prompt_type`: Type of prompt ('clustering', 'ideas', 'content_generation', etc.)
|
||||
- `account`: Optional account object
|
||||
|
||||
**Dependencies**:
|
||||
- `AIPrompt` model
|
||||
- `get_default_prompt()` utility function
|
||||
|
||||
**Returns**: Prompt template string
|
||||
|
||||
---
|
||||
|
||||
## Planner Module AI Functions
|
||||
|
||||
### 1. Auto Cluster Keywords
|
||||
|
||||
**Function Name**: `auto_cluster`
|
||||
|
||||
**API Endpoint**: `POST /v1/planner/keywords/auto_cluster/`
|
||||
|
||||
**ViewSet Action**: `KeywordViewSet.auto_cluster()`
|
||||
|
||||
**Celery Task**: `auto_cluster_keywords_task()`
|
||||
|
||||
**Core Function**: `_auto_cluster_keywords_core()`
|
||||
|
||||
**AIProcessor Method**: `AIProcessor.cluster_keywords()`
|
||||
|
||||
**Purpose**: Automatically group related keywords into semantic clusters using AI.
|
||||
|
||||
#### Complete Workflow (16 Steps)
|
||||
|
||||
**Request Phase (Steps 1-7)**:
|
||||
1. **API Endpoint Validation**: Validate keyword IDs array
|
||||
2. **Account/Site Extraction**: Extract account and site from request
|
||||
3. **Task Queuing**: Queue Celery task
|
||||
4. **Keyword Loading & Validation**: Load keywords from database
|
||||
5. **Relationship Validation**: Validate account, site, sector relationships
|
||||
6. **AIProcessor Creation**: Create AIProcessor instance
|
||||
7. **AI Call Preparation**: Format keywords for AI processing
|
||||
|
||||
**AI Processing Phase (Steps 8-12)**:
|
||||
8. **HTTP Response Validation**: Validate OpenAI API response
|
||||
9. **Content Extraction**: Extract content from response
|
||||
10. **Cost Calculation**: Calculate API cost
|
||||
11. **JSON Extraction & Parsing**: Extract and parse JSON from response
|
||||
12. **Cluster Data Validation**: Validate cluster data structure
|
||||
|
||||
**Database Phase (Steps 13-16)**:
|
||||
13. **Database Transaction Start**: Begin atomic transaction
|
||||
14. **Cluster Creation/Update**: Create or update clusters
|
||||
15. **Keyword Matching & Assignment**: Match keywords to clusters
|
||||
16. **Metrics Recalculation & Commit**: Recalculate cluster metrics and commit
|
||||
|
||||
#### Dependencies
|
||||
|
||||
**Models**:
|
||||
- `Keywords` model
|
||||
- `Clusters` model
|
||||
- `Account` model
|
||||
- `Site` model
|
||||
- `Sector` model
|
||||
- `IntegrationSettings` model (for OpenAI API key)
|
||||
- `AIPrompt` model (for clustering prompt template)
|
||||
- `Plan` model (for plan limits)
|
||||
|
||||
**Functions**:
|
||||
- `KeywordViewSet.auto_cluster()` (API endpoint)
|
||||
- `auto_cluster_keywords_task()` (Celery task)
|
||||
- `_auto_cluster_keywords_core()` (core function)
|
||||
- `AIProcessor.cluster_keywords()` (AI processing)
|
||||
|
||||
**Files**:
|
||||
- `backend/igny8_core/modules/planner/views.py`
|
||||
- `backend/igny8_core/modules/planner/tasks.py`
|
||||
- `backend/igny8_core/utils/ai_processor.py`
|
||||
|
||||
### 2. Auto Generate Ideas
|
||||
|
||||
**Function Name**: `auto_generate_ideas`
|
||||
|
||||
**API Endpoint**: `POST /v1/planner/clusters/auto_generate_ideas/`
|
||||
|
||||
**ViewSet Action**: `ClusterViewSet.auto_generate_ideas()`
|
||||
|
||||
**Celery Task**: `auto_generate_ideas_task()`
|
||||
|
||||
**Core Function**: `_generate_single_idea_core()`
|
||||
|
||||
**AIProcessor Method**: `AIProcessor.generate_ideas()`
|
||||
|
||||
**Purpose**: Generate content ideas for keyword clusters using AI.
|
||||
|
||||
#### Workflow
|
||||
|
||||
1. **API Endpoint**: Validate cluster IDs, extract account
|
||||
2. **Task Queuing**: Queue Celery task
|
||||
3. **Cluster Loading**: Load clusters from database
|
||||
4. **AI Processing**: Call `AIProcessor.generate_ideas()`
|
||||
5. **Idea Creation**: Create ContentIdeas records
|
||||
|
||||
#### Dependencies
|
||||
|
||||
**Models**:
|
||||
- `Clusters` model
|
||||
- `ContentIdeas` model
|
||||
- `Keywords` model
|
||||
- `Account` model
|
||||
- `Site` model
|
||||
- `Sector` model
|
||||
- `IntegrationSettings` model (for OpenAI API key)
|
||||
- `AIPrompt` model (for ideas prompt template)
|
||||
- `Plan` model (for plan limits)
|
||||
|
||||
**Functions**:
|
||||
- `ClusterViewSet.auto_generate_ideas()` (API endpoint)
|
||||
- `auto_generate_ideas_task()` (Celery task)
|
||||
- `_generate_single_idea_core()` (core function)
|
||||
- `AIProcessor.generate_ideas()` (AI processing)
|
||||
|
||||
**Files**:
|
||||
- `backend/igny8_core/modules/planner/views.py`
|
||||
- `backend/igny8_core/modules/planner/tasks.py`
|
||||
- `backend/igny8_core/utils/ai_processor.py`
|
||||
|
||||
---
|
||||
|
||||
## Writer Module AI Functions
|
||||
|
||||
### 3. Auto Generate Content
|
||||
|
||||
**Function Name**: `auto_generate_content`
|
||||
|
||||
**API Endpoint**: `POST /v1/writer/tasks/auto_generate_content/`
|
||||
|
||||
**ViewSet Action**: `TasksViewSet.auto_generate_content()`
|
||||
|
||||
**Celery Task**: `auto_generate_content_task()`
|
||||
|
||||
**AIProcessor Method**: `AIProcessor.generate_content()`
|
||||
|
||||
**Purpose**: Generate complete blog post/article content using AI.
|
||||
|
||||
#### Workflow
|
||||
|
||||
1. **API Endpoint**: Validate task IDs, extract account
|
||||
2. **Task Queuing**: Queue Celery task
|
||||
3. **Task Loading**: Load tasks with relationships
|
||||
4. **Content Generation**: Call `AIProcessor.generate_content()` for each task
|
||||
5. **Content Saving**: Save generated content to Content model
|
||||
|
||||
#### Dependencies
|
||||
|
||||
**Models**:
|
||||
- `Tasks` model
|
||||
- `Content` model
|
||||
- `ContentIdeas` model
|
||||
- `Clusters` model
|
||||
- `Keywords` model
|
||||
- `Account` model
|
||||
- `Site` model
|
||||
- `Sector` model
|
||||
- `IntegrationSettings` model (for OpenAI API key)
|
||||
- `AIPrompt` model (for content generation prompt template)
|
||||
- `AuthorProfile` model (for writing style)
|
||||
- `Plan` model (for plan limits)
|
||||
|
||||
**Functions**:
|
||||
- `TasksViewSet.auto_generate_content()` (API endpoint)
|
||||
- `auto_generate_content_task()` (Celery task)
|
||||
- `AIProcessor.generate_content()` (AI processing)
|
||||
|
||||
**Files**:
|
||||
- `backend/igny8_core/modules/writer/views.py`
|
||||
- `backend/igny8_core/modules/writer/tasks.py`
|
||||
- `backend/igny8_core/utils/ai_processor.py`
|
||||
|
||||
### 4. Auto Generate Images
|
||||
|
||||
**Function Name**: `auto_generate_images`
|
||||
|
||||
**API Endpoint**: `POST /v1/writer/tasks/auto_generate_images/`
|
||||
|
||||
**ViewSet Action**: `TasksViewSet.auto_generate_images()`
|
||||
|
||||
**Celery Task**: `auto_generate_images_task()`
|
||||
|
||||
**AIProcessor Methods**: `AIProcessor.extract_image_prompts()`, `AIProcessor.generate_image()`
|
||||
|
||||
**Purpose**: Generate images for content tasks using AI (OpenAI DALL-E or Runware).
|
||||
|
||||
#### Workflow
|
||||
|
||||
1. **API Endpoint**: Validate task IDs, extract account
|
||||
2. **Task Queuing**: Queue Celery task
|
||||
3. **Task Loading**: Load tasks with content
|
||||
4. **Image Prompt Extraction**: Call `AIProcessor.extract_image_prompts()`
|
||||
5. **Image Generation**: Call `AIProcessor.generate_image()` for each prompt
|
||||
6. **Image Saving**: Save generated images to Images model
|
||||
|
||||
#### Dependencies
|
||||
|
||||
**Models**:
|
||||
- `Tasks` model
|
||||
- `Images` model
|
||||
- `Content` model
|
||||
- `Account` model
|
||||
- `IntegrationSettings` model (for OpenAI/Runware API keys)
|
||||
- `AIPrompt` model (for image prompt templates)
|
||||
- `Plan` model (for plan limits and image model choices)
|
||||
|
||||
**Functions**:
|
||||
- `TasksViewSet.auto_generate_images()` (API endpoint)
|
||||
- `auto_generate_images_task()` (Celery task)
|
||||
- `AIProcessor.extract_image_prompts()` (prompt extraction)
|
||||
- `AIProcessor.generate_image()` (image generation)
|
||||
|
||||
**Files**:
|
||||
- `backend/igny8_core/modules/writer/views.py`
|
||||
- `backend/igny8_core/modules/writer/tasks.py`
|
||||
- `backend/igny8_core/utils/ai_processor.py`
|
||||
|
||||
### 5. Extract Image Prompts
|
||||
|
||||
**Function Name**: `extract_image_prompts`
|
||||
|
||||
**AIProcessor Method**: `AIProcessor.extract_image_prompts()`
|
||||
|
||||
**Purpose**: Extract image prompts from generated content for image generation.
|
||||
|
||||
**Note**: This is typically called internally by `auto_generate_images`, but can be called directly.
|
||||
|
||||
#### Dependencies
|
||||
|
||||
**Models**:
|
||||
- `Content` model
|
||||
- `AIPrompt` model (for image prompt extraction template)
|
||||
- `IntegrationSettings` model (for OpenAI API key)
|
||||
|
||||
**Functions**:
|
||||
- `AIProcessor.extract_image_prompts()` (AI processing)
|
||||
|
||||
**Files**:
|
||||
- `backend/igny8_core/utils/ai_processor.py`
|
||||
|
||||
---
|
||||
|
||||
## System Module AI Functions
|
||||
|
||||
### 6. Test OpenAI Integration
|
||||
|
||||
**Function Name**: `test_openai`
|
||||
|
||||
**API Endpoint**: `POST /v1/system/settings/integrations/{id}/test_openai/`
|
||||
|
||||
**ViewSet Action**: `IntegrationSettingsViewSet.test_openai()`
|
||||
|
||||
**Purpose**: Test OpenAI API connection and validate API key.
|
||||
|
||||
#### Workflow
|
||||
|
||||
1. **Get API Key**: From request or saved IntegrationSettings
|
||||
2. **Test Connection**: Call OpenAI API (`/v1/models` or `/v1/chat/completions`)
|
||||
3. **Return Result**: Success or error message
|
||||
|
||||
#### Dependencies
|
||||
|
||||
**Models**:
|
||||
- `IntegrationSettings` model
|
||||
- `Account` model
|
||||
|
||||
**Functions**:
|
||||
- `IntegrationSettingsViewSet.test_openai()` (API endpoint)
|
||||
- `AIProcessor` class (for API key loading)
|
||||
|
||||
**Files**:
|
||||
- `backend/igny8_core/modules/system/integration_views.py`
|
||||
- `backend/igny8_core/utils/ai_processor.py`
|
||||
|
||||
### 7. Test Runware Integration
|
||||
|
||||
**Function Name**: `test_runware`
|
||||
|
||||
**API Endpoint**: `POST /v1/system/settings/integrations/{id}/test_runware/`
|
||||
|
||||
**ViewSet Action**: `IntegrationSettingsViewSet.test_runware()`
|
||||
|
||||
**Purpose**: Test Runware API connection and validate API key.
|
||||
|
||||
#### Workflow
|
||||
|
||||
1. **Get API Key**: From request or saved IntegrationSettings
|
||||
2. **Test Connection**: Call Runware API (`/v1`)
|
||||
3. **Return Result**: Success or error message
|
||||
|
||||
#### Dependencies
|
||||
|
||||
**Models**:
|
||||
- `IntegrationSettings` model
|
||||
- `Account` model
|
||||
|
||||
**Functions**:
|
||||
- `IntegrationSettingsViewSet.test_runware()` (API endpoint)
|
||||
- `AIProcessor` class (for API key loading)
|
||||
|
||||
**Files**:
|
||||
- `backend/igny8_core/modules/system/integration_views.py`
|
||||
- `backend/igny8_core/utils/ai_processor.py`
|
||||
|
||||
### 8. Generate Image (Test)
|
||||
|
||||
**Function Name**: `generate_image`
|
||||
|
||||
**API Endpoint**: `POST /v1/system/settings/integrations/{id}/generate_image/`
|
||||
|
||||
**ViewSet Action**: `IntegrationSettingsViewSet.generate_image()`
|
||||
|
||||
**Purpose**: Test image generation with OpenAI DALL-E or Runware.
|
||||
|
||||
#### Workflow
|
||||
|
||||
1. **Get Configuration**: From request or saved IntegrationSettings
|
||||
2. **Generate Image**: Call `AIProcessor.generate_image()`
|
||||
3. **Return Result**: Image URL or error message
|
||||
|
||||
#### Dependencies
|
||||
|
||||
**Models**:
|
||||
- `IntegrationSettings` model
|
||||
- `Account` model
|
||||
|
||||
**Functions**:
|
||||
- `IntegrationSettingsViewSet.generate_image()` (API endpoint)
|
||||
- `AIProcessor.generate_image()` (image generation)
|
||||
|
||||
**Files**:
|
||||
- `backend/igny8_core/modules/system/integration_views.py`
|
||||
- `backend/igny8_core/utils/ai_processor.py`
|
||||
|
||||
---
|
||||
|
||||
## Progress Tracking
|
||||
|
||||
### Task Progress Endpoint
|
||||
|
||||
**Endpoint**: `GET /v1/system/settings/task_progress/{task_id}/`
|
||||
|
||||
**ViewSet Action**: `IntegrationSettingsViewSet.task_progress()`
|
||||
|
||||
**Purpose**: Get Celery task status and progress.
|
||||
|
||||
**Response Format**:
|
||||
```json
|
||||
{
|
||||
"state": "PROGRESS",
|
||||
"meta": {
|
||||
"current": 2,
|
||||
"total": 10,
|
||||
"percentage": 20,
|
||||
"message": "Processing...",
|
||||
"request_steps": [...],
|
||||
"response_steps": [...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**States**:
|
||||
- `PENDING`: Task is queued but not started
|
||||
- `PROGRESS`: Task is in progress
|
||||
- `SUCCESS`: Task completed successfully
|
||||
- `FAILURE`: Task failed with error
|
||||
|
||||
### Frontend Integration
|
||||
|
||||
**Hook**: `useProgressModal`
|
||||
|
||||
**Functionality**:
|
||||
- Polls task progress endpoint every 2 seconds
|
||||
- Displays progress percentage and message
|
||||
- Shows step-by-step logs (request_steps and response_steps)
|
||||
- Auto-closes on completion
|
||||
|
||||
**Store**: `aiRequestLogsStore`
|
||||
|
||||
**Functionality**:
|
||||
- Stores AI request/response logs
|
||||
- Tracks step-by-step execution
|
||||
- Displays logs in UI
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Types
|
||||
|
||||
1. **API Errors**: OpenAI/Runware API errors
|
||||
2. **Validation Errors**: Input validation failures
|
||||
3. **Database Errors**: Database operation failures
|
||||
4. **Connection Errors**: Celery/Redis connection issues
|
||||
|
||||
### Error Propagation
|
||||
|
||||
1. **API Endpoint**: Catches errors, returns error response with request_steps
|
||||
2. **Celery Task**: Catches errors, updates task state to FAILURE with error details
|
||||
3. **Core Function**: Catches errors, logs to request_steps/response_steps
|
||||
4. **AIProcessor**: Catches errors, returns error in result dictionary
|
||||
|
||||
### Error Logging
|
||||
|
||||
- All errors logged to `request_steps` or `response_steps`
|
||||
- Error details include: error type, message, function name, step number
|
||||
- Frontend displays errors in AI Request/Response Logs section
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Database Models
|
||||
|
||||
- `Account`: Account isolation
|
||||
- `Site`, `Sector`: Site/sector hierarchy
|
||||
- `Keywords`, `Clusters`, `ContentIdeas`: Planner data
|
||||
- `Tasks`, `Content`, `Images`: Writer data
|
||||
- `IntegrationSettings`: API keys and configuration
|
||||
- `AIPrompt`: Prompt templates
|
||||
- `Plan`: Plan limits
|
||||
|
||||
### External Services
|
||||
|
||||
- **OpenAI API**: GPT models for text generation
|
||||
- **OpenAI DALL-E**: Image generation
|
||||
- **Runware API**: Alternative image generation
|
||||
|
||||
### Utilities
|
||||
|
||||
- `get_default_prompt()`: Default prompt templates
|
||||
- `AIProcessor`: Unified AI interface
|
||||
- Celery: Asynchronous task processing
|
||||
- Redis: Celery broker
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
The IGNY8 AI functions system provides:
|
||||
|
||||
1. **Unified Interface**: Single AIProcessor class for all AI operations
|
||||
2. **Account Isolation**: All AI functions respect account boundaries
|
||||
3. **Progress Tracking**: Real-time progress updates with step-by-step logs
|
||||
4. **Error Handling**: Comprehensive error logging and propagation
|
||||
5. **Flexible Configuration**: Account-specific API keys and prompts
|
||||
6. **Asynchronous Processing**: Celery tasks for long-running operations
|
||||
7. **Multiple Providers**: Support for OpenAI and Runware
|
||||
8. **Complete Workflows**: End-to-end workflows from API to database
|
||||
|
||||
All AI functions integrate seamlessly with the account/site/user/plan architecture and provide detailed logging for debugging and monitoring.
|
||||
|
||||
1153
docs/Architecture/MASTER-ARCHITECTURE.md
Normal file
1153
docs/Architecture/MASTER-ARCHITECTURE.md
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user