salman/igny8
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

new docs

Browse Source
This commit is contained in:
alorig
2025-11-09 19:07:06 +05:00
parent 6b738d1ab7
commit 8cd036d8ce
13 changed files with 929 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

300
docs/ActiveDocs/AI-ELEMENTS-EXTRACTED.md Normal file
View File

@@ -0,0 +1,300 @@
# IGNY8 AI Elements Reference Table
Generated by extract_ai_elements.py analysis
---
## 🧠 AI Core Functions
| Function Name | Category | Type | File | Line | Uses AIProcessor | Celery | Progress | Steps | Prompt Source | Model Source |
|---------------|----------|------|------|------|------------------|--------|----------|-------|---------------|--------------|
| `_auto_cluster_keywords_core` | cluster | core_function | `backend/igny8_core/modules/planner/tasks.py` | 26 | ✅ | ❌ | ✅ | ✅ | Database (get_prompt_value) | AIProcessor.default_model |
| `_generate_single_idea_core` | ideas | core_function | `backend/igny8_core/modules/planner/tasks.py` | 1047 | ✅ | ❌ | ✅ | ✅ | Database (get_prompt_value) | AIProcessor.default_model |
| `auto_generate_content_task` | content | celery_task | `backend/igny8_core/modules/writer/tasks.py` | 27 | ✅ | ✅ | ✅ | ❌ | Database (get_prompt_value) | AIProcessor.default_model |
| `auto_generate_images_task` | image | celery_task | `backend/igny8_core/modules/writer/tasks.py` | 741 | ✅ | ✅ | ✅ | ❌ | Database (get_prompt_value) | AIProcessor.default_model |
| `AutoClusterFunction` | cluster | class | `backend/igny8_core/ai/functions/auto_cluster.py` | 14 | ✅ | ❌ | ✅ | ✅ | Database (get_prompt_value) | Function.get_model() |
| `cluster_keywords` | cluster | method | `backend/igny8_core/utils/ai_processor.py` | 1080 | ✅ | ❌ | ✅ | ✅ | Inline/Hardcoded | AIProcessor.default_model |
| `generate_ideas` | ideas | method | `backend/igny8_core/utils/ai_processor.py` | 1280 | ✅ | ❌ | ✅ | ✅ | Inline/Hardcoded | AIProcessor.default_model |
| `generate_content` | content | method | `backend/igny8_core/utils/ai_processor.py` | 446 | ✅ | ❌ | ❌ | ❌ | Inline/Hardcoded | AIProcessor.default_model |
| `generate_image` | image | method | `backend/igny8_core/utils/ai_processor.py` | 656 | ✅ | ❌ | ❌ | ❌ | Inline/Hardcoded | Parameter or default |
| `run_ai_task` | unified | celery_task | `backend/igny8_core/ai/tasks.py` | 13 | ❌ | ✅ | ✅ | ✅ | Via function | Via function |
| `execute` | unified | method | `backend/igny8_core/ai/engine.py` | 26 | ✅ | ❌ | ✅ | ✅ | Via function | Via function |
---
## 🧱 Prompt Sources
| Prompt Type | Source | File | Retrieval Method |
|-------------|--------|------|------------------|
| `clustering` | Hardcoded in get_default_prompt() | `backend/igny8_core/modules/system/utils.py` | `get_prompt_value()` -> AIPrompt model or default |
| `ideas` | Hardcoded in get_default_prompt() | `backend/igny8_core/modules/system/utils.py` | `get_prompt_value()` -> AIPrompt model or default |
| `content_generation` | Hardcoded in get_default_prompt() | `backend/igny8_core/modules/system/utils.py` | `get_prompt_value()` -> AIPrompt model or default |
| `image_prompt_extraction` | Hardcoded in get_default_prompt() | `backend/igny8_core/modules/system/utils.py` | `get_prompt_value()` -> AIPrompt model or default |
| `image_prompt_template` | Hardcoded in get_default_prompt() | `backend/igny8_core/modules/system/utils.py` | `get_prompt_value()` -> AIPrompt model or default |
| `negative_prompt` | Hardcoded in get_default_prompt() | `backend/igny8_core/modules/system/utils.py` | `get_prompt_value()` -> AIPrompt model or default |
**Prompt Storage:**
- **Database Model**: `AIPrompt` in `backend/igny8_core/modules/system/models.py`
- **Table**: `igny8_ai_prompts`
- **Fields**: `prompt_type`, `prompt_value`, `default_prompt`, `account` (FK)
- **Unique Constraint**: `(account, prompt_type)`
**Prompt Retrieval Flow:**
1. `get_prompt_value(account, prompt_type)` in `modules/system/utils.py:108`
2. Tries: `AIPrompt.objects.get(account=account, prompt_type=prompt_type, is_active=True)`
3. Falls back to: `get_default_prompt(prompt_type)` if not found
---
## 🧾 Model Configuration
| Model Name | Source | File | Selection Method |
|------------|--------|------|------------------|
| `gpt-4.1` | MODEL_RATES constant | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor._get_model()` -> IntegrationSettings or Django settings |
| `gpt-4o-mini` | MODEL_RATES constant | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor._get_model()` -> IntegrationSettings or Django settings |
| `gpt-4o` | MODEL_RATES constant | `backend/igny8_core/utils/ai_processor.py` | `AIProcessor._get_model()` -> IntegrationSettings or Django settings |
| `dall-e-3` | IMAGE_MODEL_RATES constant | `backend/igny8_core/utils/ai_processor.py` | Parameter or default in `generate_image()` |
| `dall-e-2` | IMAGE_MODEL_RATES constant | `backend/igny8_core/utils/ai_processor.py` | Parameter or default in `generate_image()` |
**Model Selection Flow:**
1. `AIProcessor.__init__(account)` in `utils/ai_processor.py:54`
2. Calls `_get_model('openai', account)` in `utils/ai_processor.py:98`
3. Tries: `IntegrationSettings.objects.filter(integration_type='openai', account=account, is_active=True).first().config.get('model')`
4. Validates model is in `MODEL_RATES` dict
5. Falls back to: `settings.DEFAULT_AI_MODEL` (default: 'gpt-4.1')
**Model Storage:**
- **Database Model**: `IntegrationSettings` in `backend/igny8_core/modules/system/models.py`
- **Table**: `igny8_integration_settings`
- **Fields**: `integration_type`, `config` (JSONField), `account` (FK)
- **Config Structure**: `{"apiKey": "...", "model": "gpt-4.1", "enabled": true}`
---
## ⚠️ Validation & Limits
| Function | Validation Checks | Limit Checks |
|----------|-------------------|--------------|
| `_auto_cluster_keywords_core` | Has validate() call, Keywords exist check | Credit check, Plan limits (daily_cluster_limit, max_clusters) |
| `AutoClusterFunction.validate()` | Base validation (ids array, max_items), Keywords exist | Plan limits (daily_cluster_limit, max_clusters) |
| `auto_generate_content_task` | Task existence, Account validation | Credit check (via CreditService) |
| `auto_generate_images_task` | Task existence, Account validation | Credit check (via CreditService) |
| `generate_image` | Model validation (VALID_OPENAI_IMAGE_MODELS), Size validation (VALID_SIZES_BY_MODEL) | None |
| `AIProcessor._get_model()` | Model in MODEL_RATES validation | None |
**Validation Details:**
1. **Plan Limits** (in `AutoClusterFunction.validate()`):
- `plan.daily_cluster_limit` - Max clusters per day
- `plan.max_clusters` - Total max clusters
- Checked in `backend/igny8_core/ai/functions/auto_cluster.py:59-79`
2. **Credit Checks**:
- `CreditService.check_credits(account, required_credits)` in `modules/billing/services.py:16`
- Used before AI operations
3. **Model Validation**:
- OpenAI images: Only `dall-e-3` and `dall-e-2` valid (line 704-708 in `ai_processor.py`)
- Size validation per model (line 719-724 in `ai_processor.py`)
4. **Input Validation**:
- Base validation in `BaseAIFunction.validate()` checks for 'ids' array and max_items limit
- `AutoClusterFunction.get_max_items()` returns 20 (max keywords per cluster)
---
## 🔁 Retry & Error Handling
| Component | Retry Logic | Error Handling | Fallback |
|-----------|-------------|----------------|----------|
| `run_ai_task` | `max_retries=3` (Celery decorator) | Exception caught, task state updated to FAILURE | None |
| `auto_generate_content_task` | `max_retries=3` (Celery decorator) | Try/except blocks, error logging | None |
| `_call_openai` | None (single attempt) | HTTP error handling, JSON parse errors, timeout (60s) | Returns error dict |
| `_get_api_key` | None | Exception caught, logs warning | Falls back to Django settings (`OPENAI_API_KEY`, `RUNWARE_API_KEY`) |
| `_get_model` | None | Exception caught, logs warning | Falls back to Django settings (`DEFAULT_AI_MODEL`) |
---
## 🪵 AI Debug Steps
| Function | Request Steps | Response Steps | Step Tracking Method |
|----------|---------------|----------------|---------------------|
| `_auto_cluster_keywords_core` | ✅ (manual list) | ✅ (manual list) | Manual `request_steps.append()` and `response_steps.append()` |
| `AutoClusterFunction` | ✅ (via StepTracker) | ✅ (via StepTracker) | `StepTracker.add_request_step()` and `add_response_step()` |
| `run_ai_task` | ✅ (via engine) | ✅ (via engine) | Extracted from `engine.execute()` result |
| `AIEngine.execute` | ✅ (via StepTracker) | ✅ (via StepTracker) | `StepTracker` instance tracks all steps |
| `auto_generate_content_task` | ❌ | ❌ | No step tracking (legacy) |
| `auto_generate_images_task` | ❌ | ❌ | No step tracking (legacy) |
**Step Tracking Implementation:**
1. **New Framework** (AIEngine):
- Uses `StepTracker` class in `backend/igny8_core/ai/tracker.py`
- Steps added at each phase: INIT, PREP, AI_CALL, PARSE, SAVE, DONE
- Steps stored in `request_steps` and `response_steps` lists
- Returned in result dict and logged to `AITaskLog` model
2. **Legacy Functions**:
- Manual step tracking with lists
- Steps added to `meta` dict for Celery task progress
- Extracted in `integration_views.py:task_progress()` for frontend
3. **Step Structure**:
```python
{
'stepNumber': int,
'stepName': str, # INIT, PREP, AI_CALL, PARSE, SAVE, DONE
'functionName': str,
'status': 'success' | 'error' | 'pending',
'message': str,
'error': str (optional),
'duration': int (milliseconds, optional)
}
```
---
## 📦 Request/Response Structuring
| Function | Request Format | Response Format | JSON Mode | Parsing Method |
|----------|----------------|-----------------|-----------|----------------|
| `_call_openai` | OpenAI API format: `{'model': str, 'messages': [...], 'temperature': float, 'max_tokens': int, 'response_format': dict}` | `{'content': str, 'input_tokens': int, 'output_tokens': int, 'total_tokens': int, 'model': str, 'cost': float, 'error': str, 'api_id': str}` | ✅ (if `response_format={'type': 'json_object'}`) | `_extract_json_from_response()` |
| `cluster_keywords` | Prompt string with keywords | JSON with 'clusters' array | ✅ (auto-enabled for json_models) | `_extract_json_from_response()` then extract 'clusters' |
| `generate_ideas` | Prompt string with clusters | JSON with 'ideas' array | ✅ (auto-enabled for json_models) | `_extract_json_from_response()` then extract 'ideas' |
| `generate_image` (OpenAI) | `{'prompt': str, 'model': str, 'n': int, 'size': str}` | `{'url': str, 'revised_prompt': str, 'cost': float}` | N/A | Direct JSON response |
| `generate_image` (Runware) | Array format with `imageInference` tasks | `{'url': str, 'cost': float}` | N/A | Extract from nested response structure |
**JSON Mode Auto-Enable:**
- Models: `['gpt-4o', 'gpt-4o-mini', 'gpt-4-turbo-preview']`
- Auto-enabled in `AIProcessor.call()` if `response_format` not specified
- Location: `backend/igny8_core/ai/processor.py:40-42`
**JSON Extraction:**
- Primary: Direct `json.loads()` on response
- Fallback: `_extract_json_from_response()` handles:
- Markdown code blocks (```json ... ```)
- Multiline JSON
- Partial JSON extraction
- Location: `backend/igny8_core/utils/ai_processor.py:334-440`
---
## 📍 Paths & Constants
| Constant | Value | File | Usage |
|----------|-------|------|-------|
| `OPENAI_API_KEY` | Django setting | `backend/igny8_core/utils/ai_processor.py:93` | Fallback API key |
| `RUNWARE_API_KEY` | Django setting | `backend/igny8_core/utils/ai_processor.py:95` | Fallback API key |
| `DEFAULT_AI_MODEL` | Django setting (default: 'gpt-4.1') | `backend/igny8_core/utils/ai_processor.py:121` | Fallback model |
| OpenAI API URL | `'https://api.openai.com/v1/chat/completions'` | `backend/igny8_core/utils/ai_processor.py:163` | Text generation endpoint |
| OpenAI Images URL | `'https://api.openai.com/v1/images/generations'` | `backend/igny8_core/utils/ai_processor.py:735` | Image generation endpoint |
| Runware API URL | `'https://api.runware.ai/v1'` | `backend/igny8_core/utils/ai_processor.py:844` | Runware image generation |
| `MODEL_RATES` | Dict with pricing per 1M tokens | `backend/igny8_core/utils/ai_processor.py:19` | Cost calculation |
| `IMAGE_MODEL_RATES` | Dict with pricing per image | `backend/igny8_core/utils/ai_processor.py:26` | Image cost calculation |
| `VALID_OPENAI_IMAGE_MODELS` | `{'dall-e-3', 'dall-e-2'}` | `backend/igny8_core/utils/ai_processor.py:34` | Model validation |
| `VALID_SIZES_BY_MODEL` | Dict mapping models to valid sizes | `backend/igny8_core/utils/ai_processor.py:41` | Size validation |
---
## 💰 Cost Tracking
| Component | Cost Calculation | Token Tracking | Storage |
|-----------|------------------|----------------|---------|
| `_call_openai` | Calculated from `MODEL_RATES` based on input/output tokens | ✅ (input_tokens, output_tokens, total_tokens) | Returned in result dict |
| `generate_image` (OpenAI) | `IMAGE_MODEL_RATES[model] * n` | N/A | Returned in result dict |
| `generate_image` (Runware) | `0.036 * n` (hardcoded) | N/A | Returned in result dict |
| `CostTracker` | Aggregates costs from multiple operations | ✅ (total_tokens) | In-memory during execution |
| `AITaskLog` | Stored in `cost` field (DecimalField) | ✅ (stored in `tokens` field) | Database table `igny8_ai_task_logs` |
| `CreditUsageLog` | Stored in `cost_usd` field | ✅ (tokens_input, tokens_output) | Database table (billing module) |
**Cost Calculation Formula:**
```python
# Text generation
input_cost = (input_tokens / 1_000_000) * MODEL_RATES[model]['input']
output_cost = (output_tokens / 1_000_000) * MODEL_RATES[model]['output']
total_cost = input_cost + output_cost
# Image generation
cost = IMAGE_MODEL_RATES[model] * n_images
```
---
## 📊 Progress Tracking
| Function | Progress Method | Phase Tracking | Percentage Mapping |
|----------|-----------------|----------------|-------------------|
| `_auto_cluster_keywords_core` | `progress_callback()` function | Manual phase strings | Manual percentage |
| `auto_generate_content_task` | `self.update_state()` (Celery) | Manual phase strings | Manual percentage |
| `AIEngine.execute` | `ProgressTracker.update()` | Automatic (INIT, PREP, AI_CALL, PARSE, SAVE, DONE) | Automatic: INIT (0-10%), PREP (10-25%), AI_CALL (25-70%), PARSE (70-85%), SAVE (85-98%), DONE (98-100%) |
| `run_ai_task` | Via `AIEngine` | Via `AIEngine` | Via `AIEngine` |
**Progress Tracker:**
- Class: `ProgressTracker` in `backend/igny8_core/ai/tracker.py:77`
- Updates Celery task state via `task.update_state()`
- Tracks: phase, percentage, message, current, total, meta
---
## 🗄️ Database Logging
| Component | Log Table | Fields Logged | When Logged |
|-----------|-----------|---------------|-------------|
| `AIEngine.execute` | `AITaskLog` | task_id, function_name, phase, message, status, duration, cost, tokens, request_steps, response_steps, error, payload, result | After execution (success or error) |
| Credit usage | `CreditUsageLog` | account, operation_type, credits_used, cost_usd, model_used, tokens_input, tokens_output | After successful save operation |
**AITaskLog Model:**
- Table: `igny8_ai_task_logs`
- Location: `backend/igny8_core/ai/models.py:8`
- Fields: All execution details including steps, costs, tokens, errors
---
## 🔄 Celery Integration
| Task | Entrypoint | Task ID | State Updates | Error Handling |
|------|------------|---------|---------------|----------------|
| `run_ai_task` | `backend/igny8_core/ai/tasks.py:13` | `self.request.id` | Via `ProgressTracker` | Updates state to FAILURE, raises exception |
| `auto_generate_content_task` | `backend/igny8_core/modules/writer/tasks.py:27` | `self.request.id` | Manual `self.update_state()` | Try/except, logs error |
| `auto_generate_images_task` | `backend/igny8_core/modules/writer/tasks.py:741` | `self.request.id` | Manual `self.update_state()` | Try/except, logs error |
**Task Progress Endpoint:**
- Route: `/api/v1/system/settings/task_progress/{task_id}/`
- Handler: `IntegrationSettingsViewSet.task_progress()` in `modules/system/integration_views.py:936`
- Extracts: `request_steps` and `response_steps` from task meta
- Returns: Progress data to frontend for debug panel
---
## Summary
**Key Findings:**
1. **Two AI Systems Coexist:**
- **Legacy**: Direct functions in `modules/planner/tasks.py` and `modules/writer/tasks.py`
- **New Framework**: `AIEngine` + `BaseAIFunction` classes in `ai/` directory
2. **Unified Entrypoint:**
- `run_ai_task()` in `ai/tasks.py` is the unified Celery entrypoint
- Uses `AIEngine` to execute any registered AI function
3. **Prompt Management:**
- All prompts stored in `AIPrompt` model (database)
- Fallback to hardcoded defaults in `get_default_prompt()`
- Retrieved via `get_prompt_value(account, prompt_type)`
4. **Model Selection:**
- Per-account via `IntegrationSettings.config['model']`
- Falls back to Django `DEFAULT_AI_MODEL` setting
- Validated against `MODEL_RATES` dict
5. **Step Tracking:**
- New framework uses `StepTracker` class
- Legacy functions use manual lists
- Both stored in Celery task meta and `AITaskLog` model
6. **Cost Tracking:**
- Calculated from `MODEL_RATES` and `IMAGE_MODEL_RATES`
- Logged to `AITaskLog` and `CreditUsageLog`
- Tracked via `CostTracker` during execution

271
docs/ActiveDocs/AI-FUNCTION-FILES.md
View File

@@ -1,271 +0,0 @@
# AI Function Related Files
This document lists all files containing code related to:
- Auto Cluster Keywords
- Auto Generate Ideas
- Auto Generate Content
- Auto Generate Images
---
## Backend Files
### Auto Cluster Keywords
#### Core Implementation
- `backend/igny8_core/ai/functions/auto_cluster.py` - **Main AI function implementation** (BaseAIFunction)
- `backend/igny8_core/ai/base.py` - Base AI function class
- `backend/igny8_core/ai/engine.py` - AI engine orchestrator
- `backend/igny8_core/ai/processor.py` - AI processor wrapper
- `backend/igny8_core/ai/tasks.py` - Unified Celery task entrypoint
- `backend/igny8_core/ai/registry.py` - Function registry
- `backend/igny8_core/ai/tracker.py` - Progress and cost tracking
#### API Endpoints & Views
- `backend/igny8_core/modules/planner/views.py` - **KeywordViewSet.auto_cluster()** action
- `backend/igny8_core/modules/planner/urls.py` - URL routing
#### Celery Tasks (Legacy/Alternative)
- `backend/igny8_core/modules/planner/tasks.py` - **auto_cluster_keywords_task()** (legacy implementation)
#### AI Processor
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.cluster_keywords()** method
#### Models
- `backend/igny8_core/modules/planner/models.py` - Keywords, Clusters models
#### Serializers
- `backend/igny8_core/modules/planner/serializers.py` - Keyword, Cluster serializers
- `backend/igny8_core/modules/planner/cluster_serializers.py` - Cluster-specific serializers
#### System Integration
- `backend/igny8_core/modules/system/schemas.py` - Schema definitions
- `backend/igny8_core/modules/system/utils.py` - Prompt loading utilities
---
### Auto Generate Ideas
#### Core Implementation
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.generate_ideas()** method
#### API Endpoints & Views
- `backend/igny8_core/modules/planner/views.py` - **ClusterViewSet.auto_generate_ideas()** action
- `backend/igny8_core/modules/planner/urls.py` - URL routing
#### Celery Tasks
- `backend/igny8_core/modules/planner/tasks.py` - **auto_generate_ideas_task()** and **generate_single_idea_core()**
#### Models
- `backend/igny8_core/modules/planner/models.py` - Clusters, ContentIdeas models
#### Serializers
- `backend/igny8_core/modules/planner/serializers.py` - Cluster, ContentIdeas serializers
#### System Integration
- `backend/igny8_core/modules/system/utils.py` - Prompt loading utilities
---
### Auto Generate Content
#### Core Implementation
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.generate_content()** method
#### API Endpoints & Views
- `backend/igny8_core/modules/writer/views.py` - **TasksViewSet.auto_generate_content()** action
- `backend/igny8_core/modules/writer/urls.py` - URL routing
#### Celery Tasks
- `backend/igny8_core/modules/writer/tasks.py` - **auto_generate_content_task()**
#### Models
- `backend/igny8_core/modules/writer/models.py` - Tasks, Content models
#### Serializers
- `backend/igny8_core/modules/writer/serializers.py` - Task, Content serializers
#### System Integration
- `backend/igny8_core/modules/system/schemas.py` - Schema definitions
- `backend/igny8_core/modules/system/utils.py` - Prompt loading utilities
---
### Auto Generate Images
#### Core Implementation
- `backend/igny8_core/utils/ai_processor.py` - **AIProcessor.extract_image_prompts()** and **AIProcessor.generate_image()** methods
#### API Endpoints & Views
- `backend/igny8_core/modules/writer/views.py` - **TasksViewSet.auto_generate_images()** action
- `backend/igny8_core/modules/writer/urls.py` - URL routing
#### Celery Tasks
- `backend/igny8_core/modules/writer/tasks.py` - **auto_generate_images_task()**
#### Models
- `backend/igny8_core/modules/writer/models.py` - Tasks, Images models
#### Serializers
- `backend/igny8_core/modules/writer/serializers.py` - Task, Images serializers
#### System Integration
- `backend/igny8_core/modules/system/schemas.py` - Schema definitions
- `backend/igny8_core/modules/system/integration_views.py` - Integration settings for image generation
---
## Frontend Files
### Auto Cluster Keywords
#### Pages
- `frontend/src/pages/Planner/Keywords.tsx` - **Main page component** with auto cluster functionality
#### API Services
- `frontend/src/services/api.ts` - **autoClusterKeywords()** function
#### Configuration
- `frontend/src/config/pages/keywords.config.tsx` - Page configuration
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
#### State Management
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
#### Hooks
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
---
### Auto Generate Ideas
#### Pages
- `frontend/src/pages/Planner/Clusters.tsx` - **Main page component** with auto generate ideas functionality
#### API Services
- `frontend/src/services/api.ts` - **autoGenerateIdeas()** function
#### Configuration
- `frontend/src/config/pages/clusters.config.tsx` - Page configuration
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
#### State Management
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
#### Hooks
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
---
### Auto Generate Content
#### Pages
- `frontend/src/pages/Writer/Tasks.tsx` - **Main page component** with auto generate content functionality
#### API Services
- `frontend/src/services/api.ts` - **autoGenerateContent()** function
#### Configuration
- `frontend/src/config/pages/tasks.config.tsx` - Page configuration
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
#### State Management
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
#### Hooks
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
---
### Auto Generate Images
#### Pages
- `frontend/src/pages/Writer/Tasks.tsx` - **Main page component** with auto generate images functionality
#### API Services
- `frontend/src/services/api.ts` - **autoGenerateImages()** function
#### Configuration
- `frontend/src/config/pages/tasks.config.tsx` - Page configuration
- `frontend/src/config/pages/images.config.tsx` - Image-related configuration
- `frontend/src/config/pages/table-actions.config.tsx` - Action button configurations
#### State Management
- `frontend/src/store/aiRequestLogsStore.ts` - AI request/response logs store
#### Hooks
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook for tracking task progress
---
## Shared/Common Files
### AI Framework (Backend)
- `backend/igny8_core/ai/__init__.py` - AI module initialization and auto-registration
- `backend/igny8_core/ai/models.py` - AITaskLog model for unified logging
- `backend/igny8_core/ai/types.py` - Shared dataclasses and types
- `backend/igny8_core/ai/admin.py` - Admin interface for AITaskLog
### Progress Tracking (Backend)
- `backend/igny8_core/modules/system/integration_views.py` - **task_progress()** endpoint
### Progress Tracking (Frontend)
- `frontend/src/components/common/ProgressModal.tsx` - Progress modal component
- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook
### Common Components (Frontend)
- `frontend/src/components/common/FormModal.tsx` - Form modal used in AI function pages
---
## Summary by Function
### Auto Cluster Keywords
**Backend**: 15 files
**Frontend**: 5 files
**Total**: 20 files
### Auto Generate Ideas
**Backend**: 7 files
**Frontend**: 5 files
**Total**: 12 files
### Auto Generate Content
**Backend**: 7 files
**Frontend**: 5 files
**Total**: 12 files
### Auto Generate Images
**Backend**: 7 files
**Frontend**: 5 files
**Total**: 12 files
### Shared/Common
**Backend**: 4 files
**Frontend**: 3 files
**Total**: 7 files
---
## Key Files (Most Important)
### Backend Core Files
1. `backend/igny8_core/ai/functions/auto_cluster.py` - Auto cluster AI function
2. `backend/igny8_core/utils/ai_processor.py` - Unified AI processor (all functions)
3. `backend/igny8_core/modules/planner/views.py` - Auto cluster & ideas API endpoints
4. `backend/igny8_core/modules/writer/views.py` - Content & images API endpoints
5. `backend/igny8_core/modules/planner/tasks.py` - Planner Celery tasks
6. `backend/igny8_core/modules/writer/tasks.py` - Writer Celery tasks
7. `backend/igny8_core/ai/tasks.py` - Unified AI task entrypoint
### Frontend Core Files
1. `frontend/src/pages/Planner/Keywords.tsx` - Auto cluster UI
2. `frontend/src/pages/Planner/Clusters.tsx` - Auto generate ideas UI
3. `frontend/src/pages/Writer/Tasks.tsx` - Content & images generation UI
4. `frontend/src/services/api.ts` - All API functions
5. `frontend/src/hooks/useProgressModal.ts` - Progress tracking hook
---
**Last Updated**: 2025-01-XX

121
docs/ActiveDocs/COLOR-TOKENS-COMPARISON.md Normal file
View File

@@ -0,0 +1,121 @@
# Color Tokens Comparison
Side-by-side comparison of provided tokens vs. global color palette
| Color Token | Provided Value | Our Global Palette | Match Status | Notes |
|-------------|----------------|-------------------|--------------|-------|
| **RED** |
| `--color-red-50` | `oklch(.971 .013 17.38)` | `oklch(0.971 0.013 17.38)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:10` |
| `--color-red-100` | `oklch(.936 .032 17.717)` | `oklch(0.936 0.032 17.717)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:11` |
| `--color-red-500` | `oklch(.637 .237 25.331)` | `oklch(0.637 0.237 25.331)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:15` |
| `--color-red-600` | `oklch(.577 .245 27.325)` | `oklch(0.577 0.245 27.325)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:16` |
| `--color-red-700` | `oklch(.505 .213 27.518)` | `oklch(0.505 0.213 27.518)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:17` |
| **ORANGE** |
| `--color-orange-50` | `#fff6ed` | `oklch(0.98 0.016 73.684)` ❌ | **DIFFERENT** | Provided: hex, Ours: oklch (in `tailwindcss/theme.css:22`) |
| `--color-orange-100` | `#ffead5` | `oklch(0.954 0.038 75.164)` ❌ | **DIFFERENT** | Provided: hex, Ours: oklch (in `tailwindcss/theme.css:23`) |
| `--color-orange-400` | `#fd853a` | `oklch(0.75 0.183 55.934)` ❌ | **DIFFERENT** | Provided: hex, Ours: oklch (in `tailwindcss/theme.css:26`) |
| `--color-orange-500` | `#fb6514` | `oklch(0.705 0.213 47.604)` ❌ | **DIFFERENT** | Provided: hex, Ours: oklch (in `tailwindcss/theme.css:27`) |
| `--color-orange-600` | `#ec4a0a` | `oklch(0.646 0.222 41.116)` ❌ | **DIFFERENT** | Provided: hex, Ours: oklch (in `tailwindcss/theme.css:28`) |
| `--color-orange-700` | `#c4320a` | `oklch(0.553 0.195 38.402)` ❌ | **DIFFERENT** | Provided: hex, Ours: oklch (in `tailwindcss/theme.css:29`) |
| **YELLOW** |
| `--color-yellow-100` | `oklch(.973 .071 103.193)` | `oklch(0.973 0.071 103.193)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:47` |
| `--color-yellow-600` | `oklch(.681 .162 75.834)` | `oklch(0.681 0.162 75.834)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:52` |
| **GREEN** |
| `--color-green-50` | `oklch(.982 .018 155.826)` | `oklch(0.982 0.018 155.826)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:70` |
| `--color-green-100` | `oklch(.962 .044 156.743)` | `oklch(0.962 0.044 156.743)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:71` |
| `--color-green-500` | `oklch(.723 .219 149.579)` | `oklch(0.723 0.219 149.579)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:75` |
| `--color-green-600` | `oklch(.627 .194 149.214)` | `oklch(0.627 0.194 149.214)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:76` |
| `--color-green-700` | `oklch(.527 .154 150.069)` | `oklch(0.527 0.154 150.069)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:77` |
| **CYAN** |
| `--color-cyan-100` | `oklch(.956 .045 203.388)` | `oklch(0.956 0.045 203.388)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:107` |
| `--color-cyan-600` | `oklch(.609 .126 221.723)` | `oklch(0.609 0.126 221.723)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:112` |
| **BLUE** |
| `--color-blue-50` | `oklch(.97 .014 254.604)` | `oklch(0.97 0.014 254.604)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:130` |
| `--color-blue-100` | `oklch(.932 .032 255.585)` | `oklch(0.932 0.032 255.585)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:131` |
| `--color-blue-500` | `oklch(.623 .214 259.815)` | `oklch(0.623 0.214 259.815)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:135` |
| `--color-blue-700` | `oklch(.488 .243 264.376)` | `oklch(0.488 0.243 264.376)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:137` |
| **PURPLE** |
| `--color-purple-50` | `oklch(.977 .014 308.299)` | `oklch(0.977 0.014 308.299)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:166` |
| `--color-purple-100` | `oklch(.946 .033 307.174)` | `oklch(0.946 0.033 307.174)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:167` |
| `--color-purple-400` | `oklch(.714 .203 305.504)` | `oklch(0.714 0.203 305.504)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:170` |
| `--color-purple-500` | `oklch(.627 .265 303.9)` | `oklch(0.627 0.265 303.9)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:171` |
| `--color-purple-600` | `oklch(.558 .288 302.321)` | `oklch(0.558 0.288 302.321)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:172` |
| `--color-purple-700` | `oklch(.496 .265 301.924)` | `oklch(0.496 0.265 301.924)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:173` |
| **PINK** |
| `--color-pink-100` | `oklch(.948 .028 342.258)` | `oklch(0.948 0.028 342.258)` ✅ | **EXACT MATCH** | In `tailwindcss/theme.css:191` |
| `--color-pink-600` | `oklch(.592 .249 .584)` | `oklch(0.592 0.249 0.584)` ⚠️ | **FORMAT DIFFERENCE** | Provided: `.584` (missing hue), Ours: full oklch (in `tailwindcss/theme.css:196`) |
| **GRAY** |
| `--color-gray-50` | `#f9fafb` | `#f9fafb` ✅ | **EXACT MATCH** | In `index.css:87` |
| `--color-gray-100` | `#f2f4f7` | `#f2f4f7` ✅ | **EXACT MATCH** | In `index.css:88` |
| `--color-gray-200` | `#e4e7ec` | `#e4e7ec` ✅ | **EXACT MATCH** | In `index.css:89` |
| `--color-gray-300` | `#d0d5dd` | `#d0d5dd` ✅ | **EXACT MATCH** | In `index.css:90` |
| `--color-gray-400` | `#98a2b3` | `#98a2b3` ✅ | **EXACT MATCH** | In `index.css:91` |
| `--color-gray-500` | `#667085` | `#667085` ✅ | **EXACT MATCH** | In `index.css:92` |
| `--color-gray-600` | `#475467` | `#475467` ✅ | **EXACT MATCH** | In `index.css:93` |
| `--color-gray-700` | `#344054` | `#344054` ✅ | **EXACT MATCH** | In `index.css:94` |
| `--color-gray-800` | `#1d2939` | `#1d2939` ✅ | **EXACT MATCH** | In `index.css:95` |
| `--color-gray-900` | `#101828` | `#101828` ✅ | **EXACT MATCH** | In `index.css:96` |
| `--color-gray-950` | `#0c111d` | `#0c111d` ✅ | **EXACT MATCH** | In `index.css:97` |
| **BLACK & WHITE** |
| `--color-black` | `#101828` | `#101828` ✅ | **EXACT MATCH** | In `index.css:58` |
| `--color-white` | `#fff` | `#ffffff` ✅ | **EQUIVALENT** | In `index.css:57` (same color, different format) |
---
## Summary
### ✅ **Matches (28 tokens)**
- **Red**: All 5 tokens match exactly (oklch format)
- **Yellow**: Both tokens match exactly (oklch format)
- **Green**: All 5 tokens match exactly (oklch format)
- **Cyan**: Both tokens match exactly (oklch format)
- **Blue**: All 4 tokens match exactly (oklch format)
- **Purple**: All 6 tokens match exactly (oklch format)
- **Gray**: All 11 tokens match exactly (hex format)
- **Black/White**: Both match (hex format)
### ❌ **Differences (6 tokens)**
- **Orange**: All 6 tokens differ
- Provided: Hex format (`#fff6ed`, `#ffead5`, etc.)
- Ours: oklch format in Tailwind theme
- **Note**: These are likely the same colors, just different formats. The hex values you provided match our custom orange palette in `index.css:100-111` (which uses hex), but Tailwind's theme uses oklch.
### ⚠️ **Format Issue (1 token)**
- **Pink-600**: `oklch(.592 .249 .584)` - Missing hue value (should be 3 values: lightness, chroma, hue)
---
## Location of Colors in Codebase
### Tailwind Default Colors (oklch format)
- **File**: `frontend/node_modules/tailwindcss/theme.css`
- Contains: red, orange, yellow, green, cyan, blue, purple, pink (all in oklch)
### Custom IGNY8 Colors (hex format)
- **File**: `frontend/src/index.css` (lines 55-154)
- Contains:
- Gray scale (hex)
- Orange scale (hex) - **Matches your provided orange values!**
- Brand colors (hex)
- Success/Error/Warning colors (hex)
### IGNY8 Brand Colors
- **File**: `frontend/src/styles/igny8-colors.css`
- Contains: Custom brand colors with `--igny8-` prefix
---
## Recommendation
**Your provided orange tokens match our custom orange palette in `index.css`!**
The orange colors you provided are already in our codebase:
- `--color-orange-50: #fff6ed` ✅ (line 101)
- `--color-orange-100: #ffead5` ✅ (line 102)
- `--color-orange-400: #fd853a` ✅ (line 105)
- `--color-orange-500: #fb6514` ✅ (line 106)
- `--color-orange-600: #ec4a0a` ✅ (line 107)
- `--color-orange-700: #c4320a` ✅ (line 108)
All other colors (red, yellow, green, cyan, blue, purple, gray, black, white) also match either in Tailwind's theme or our custom palette.

149
docs/ActiveDocs/verify-venv-dependencies.sh
View File

@@ -1,149 +0,0 @@
#!/bin/bash
# Verification Script: Check if venv folders are in use
# Run this script to verify nothing active depends on .venv or venv folders
echo "=========================================="
echo "IGNY8 Venv Dependency Verification Script"
echo "=========================================="
echo ""
# Colors for output
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
VENV_PATH="/data/app/igny8/backend/venv"
DOTVENV_PATH="/data/app/igny8/backend/.venv"
echo "Step 1: Check if any processes are using venv folders"
echo "------------------------------------------------------"
echo "Checking for processes with venv in command line..."
ps aux | grep -E "(venv|\.venv)" | grep -v grep || echo -e "${GREEN}✓ No processes found using venv${NC}"
echo ""
echo "Step 2: Check if Python processes are running from venv"
echo "------------------------------------------------------"
echo "Checking Python processes..."
ps aux | grep python | grep -v grep | while read line; do
if echo "$line" | grep -qE "(venv|\.venv)"; then
echo -e "${RED}✗ Found Python process using venv:${NC}"
echo " $line"
fi
done
echo -e "${GREEN}✓ No Python processes using venv found${NC}"
echo ""
echo "Step 3: Check if Docker containers reference venv"
echo "------------------------------------------------------"
echo "Checking Docker container mounts..."
docker inspect $(docker ps -q) --format '{{.Name}}: {{range .Mounts}}{{.Source}} {{end}}' 2>/dev/null | grep -E "(venv|\.venv)" || echo -e "${GREEN}✓ No Docker containers mount venv folders${NC}"
echo ""
echo "Step 4: Check if any files are open/locked in venv folders"
echo "------------------------------------------------------"
echo "Checking for open file handles..."
if command -v lsof >/dev/null 2>&1; then
lsof +D "$VENV_PATH" 2>/dev/null | head -20 || echo -e "${GREEN}✓ No open files in venv/${NC}"
lsof +D "$DOTVENV_PATH" 2>/dev/null | head -20 || echo -e "${GREEN}✓ No open files in .venv/${NC}"
else
echo -e "${YELLOW}⚠ lsof not available, skipping file handle check${NC}"
fi
echo ""
echo "Step 5: Check if any scripts reference venv paths"
echo "------------------------------------------------------"
echo "Checking for venv references in shell scripts..."
find /data/app/igny8 -name "*.sh" -type f 2>/dev/null | xargs grep -lE "(venv|\.venv)" 2>/dev/null | while read file; do
echo -e "${YELLOW}⚠ Found reference in: $file${NC}"
grep -nE "(venv|\.venv)" "$file" | head -3
done || echo -e "${GREEN}✓ No shell scripts reference venv${NC}"
echo ""
echo "Step 6: Check Docker container Python paths"
echo "------------------------------------------------------"
echo "Checking Python executable paths in containers..."
for container in igny8_backend igny8_celery_worker igny8_celery_beat; do
if docker ps --format '{{.Names}}' | grep -q "^${container}$"; then
echo "Container: $container"
docker exec "$container" which python3 2>/dev/null || echo " (container not accessible)"
docker exec "$container" python3 -c "import sys; print('Python path:', sys.executable)" 2>/dev/null | grep -v "Python path:" || echo " (container not accessible)"
docker exec "$container" sh -c 'echo $VIRTUAL_ENV' 2>/dev/null | grep -v "^$" && echo -e "${RED}✗ VIRTUAL_ENV is set${NC}" || echo -e "${GREEN} ✓ No VIRTUAL_ENV set${NC}"
fi
done
echo ""
echo "Step 7: Check if venv folders are in Dockerfile"
echo "------------------------------------------------------"
echo "Checking Dockerfile for venv references..."
if [ -f "/data/app/igny8/backend/Dockerfile" ]; then
grep -nE "(venv|\.venv)" /data/app/igny8/backend/Dockerfile 2>/dev/null && echo -e "${YELLOW}⚠ Found venv reference in Dockerfile${NC}" || echo -e "${GREEN}✓ No venv references in Dockerfile${NC}"
else
echo -e "${YELLOW}⚠ Dockerfile not found${NC}"
fi
echo ""
echo "Step 8: Check docker-compose files for venv references"
echo "------------------------------------------------------"
echo "Checking docker-compose files..."
find /data/app -name "docker-compose*.yml" -type f 2>/dev/null | while read file; do
if grep -qE "(venv|\.venv)" "$file" 2>/dev/null; then
echo -e "${YELLOW}⚠ Found reference in: $file${NC}"
grep -nE "(venv|\.venv)" "$file"
fi
done || echo -e "${GREEN}✓ No venv references in docker-compose files${NC}"
echo ""
echo "Step 9: Verify containers are running correctly"
echo "------------------------------------------------------"
echo "Checking container health..."
docker ps --format "table {{.Names}}\t{{.Status}}" | grep -E "(igny8_backend|igny8_frontend|igny8_celery)" | while read line; do
if echo "$line" | grep -q "healthy\|Up"; then
echo -e "${GREEN}✓ $line${NC}"
else
echo -e "${RED}✗ $line${NC}"
fi
done
echo ""
echo "Step 10: Test backend API endpoint (verify it works without venv)"
echo "------------------------------------------------------"
echo "Testing backend health endpoint..."
response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8011/api/v1/system/status/ 2>/dev/null)
if [ "$response" = "200" ]; then
echo -e "${GREEN}✓ Backend API is responding (HTTP $response)${NC}"
else
echo -e "${RED}✗ Backend API not responding (HTTP $response)${NC}"
fi
echo ""
echo "Step 11: Check if venv folders exist and their sizes"
echo "------------------------------------------------------"
if [ -d "$VENV_PATH" ]; then
size=$(du -sh "$VENV_PATH" 2>/dev/null | cut -f1)
echo -e "${YELLOW}⚠ venv/ exists: $size${NC}"
else
echo -e "${GREEN}✓ venv/ does not exist${NC}"
fi
if [ -d "$DOTVENV_PATH" ]; then
size=$(du -sh "$DOTVENV_PATH" 2>/dev/null | cut -f1)
echo -e "${YELLOW}⚠ .venv/ exists: $size${NC}"
else
echo -e "${GREEN}✓ .venv/ does not exist${NC}"
fi
echo ""
echo "=========================================="
echo "Verification Complete"
echo "=========================================="
echo ""
echo "Summary:"
echo "- If all checks show ✓, venv folders are safe to delete"
echo "- If any check shows ✗, investigate before deleting"
echo "- If any check shows ⚠, review but likely safe"
echo ""
echo "To delete venv folders (if safe):"
echo " rm -rf $VENV_PATH"
echo " rm -rf $DOTVENV_PATH"