Stage 1 & 2 refactor of AI engine

2025-11-09 19:22:15 +05:00
parent 8cd036d8ce
commit 375473308d
18 changed files with 2396 additions and 76 deletions
--- a/docs/ActiveDocs/STAGE1-REORGANIZATION-COMPLETE.md
+++ b/docs/ActiveDocs/STAGE1-REORGANIZATION-COMPLETE.md
@@ -0,0 +1,191 @@
+# Stage 1 - AI Folder Structure & Functional Split - COMPLETE ✅
+
+## Summary
+
+Successfully reorganized the AI backend into a clean, modular structure where every AI function lives inside its own file within `/ai/functions/`.
+
+## ✅ Completed Deliverables
+
+### 1. Folder Structure Created
+
+```
+backend/igny8_core/ai/
+├── functions/
+│   ├── __init__.py ✅
+│   ├── auto_cluster.py ✅
+│   ├── generate_ideas.py ✅
+│   ├── generate_content.py ✅
+│   └── generate_images.py ✅
+├── ai_core.py ✅ (Shared operations)
+├── validators.py ✅ (Consolidated validation)
+├── constants.py ✅ (Model pricing, valid models)
+├── engine.py ✅ (Updated to use AICore)
+├── tracker.py ✅ (Existing)
+├── base.py ✅ (Existing)
+├── processor.py ✅ (Existing wrapper)
+├── registry.py ✅ (Updated with new functions)
+└── __init__.py ✅ (Updated exports)
+```
+
+### 2. Shared Modules Created
+
+#### `ai_core.py`
+- **Purpose**: Shared operations for all AI functions
+- **Features**:
+  - API call construction (`call_openai`)
+  - Model selection (`get_model`, `get_api_key`)
+  - Response parsing (`extract_json`)
+  - Image generation (`generate_image`)
+  - Cost calculation (`calculate_cost`)
+- **Status**: ✅ Complete
+
+#### `validators.py`
+- **Purpose**: Consolidated validation logic
+- **Functions**:
+  - `validate_ids()` - Base ID validation
+  - `validate_keywords_exist()` - Keyword existence check
+  - `validate_cluster_limits()` - Plan limit checks
+  - `validate_cluster_exists()` - Cluster existence
+  - `validate_tasks_exist()` - Task existence
+  - `validate_api_key()` - API key validation
+  - `validate_model()` - Model validation
+  - `validate_image_size()` - Image size validation
+- **Status**: ✅ Complete
+
+#### `constants.py`
+- **Purpose**: AI-related constants
+- **Constants**:
+  - `MODEL_RATES` - Text model pricing
+  - `IMAGE_MODEL_RATES` - Image model pricing
+  - `VALID_OPENAI_IMAGE_MODELS` - Valid image models
+  - `VALID_SIZES_BY_MODEL` - Valid sizes per model
+  - `DEFAULT_AI_MODEL` - Default model name
+  - `JSON_MODE_MODELS` - Models supporting JSON mode
+- **Status**: ✅ Complete
+
+### 3. Function Files Created
+
+#### `functions/auto_cluster.py`
+- **Status**: ✅ Updated to use new validators and AICore
+- **Changes**:
+  - Uses `validate_ids()`, `validate_keywords_exist()`, `validate_cluster_limits()` from validators
+  - Uses `AICore.extract_json()` for JSON parsing
+  - Maintains backward compatibility
+
+#### `functions/generate_ideas.py`
+- **Status**: ✅ Created
+- **Features**:
+  - `GenerateIdeasFunction` class (BaseAIFunction)
+  - `generate_ideas_core()` legacy function for backward compatibility
+  - Uses AICore for API calls
+  - Uses validators for validation
+
+#### `functions/generate_content.py`
+- **Status**: ✅ Created
+- **Features**:
+  - `GenerateContentFunction` class (BaseAIFunction)
+  - `generate_content_core()` legacy function for backward compatibility
+  - Uses AICore for API calls
+  - Uses validators for validation
+
+#### `functions/generate_images.py`
+- **Status**: ✅ Created
+- **Features**:
+  - `GenerateImagesFunction` class (BaseAIFunction)
+  - `generate_images_core()` legacy function for backward compatibility
+  - Uses AICore for image generation
+  - Uses validators for validation
+
+### 4. Import Paths Updated
+
+#### Updated Files:
+- ✅ `modules/planner/views.py` - Uses `generate_ideas_core` from new location
+- ✅ `modules/planner/tasks.py` - Imports `generate_ideas_core` from new location
+- ✅ `modules/writer/tasks.py` - Imports `generate_content_core` and `generate_images_core` from new locations
+- ✅ `ai/engine.py` - Uses `AICore` instead of `AIProcessor`
+- ✅ `ai/functions/auto_cluster.py` - Uses new validators and AICore
+- ✅ `ai/registry.py` - Registered all new functions
+- ✅ `ai/__init__.py` - Exports all new modules
+
+### 5. Dependencies Verified
+
+#### No Circular Dependencies ✅
+- Functions depend on: `ai_core`, `validators`, `constants`, `base`
+- `ai_core` depends on: `utils.ai_processor` (legacy, will be refactored later)
+- `validators` depends on: `constants`, models
+- `engine` depends on: `ai_core`, `base`, `tracker`
+- All imports are clean and modular
+
+#### Modular Structure ✅
+- Each function file is self-contained
+- Shared logic in `ai_core.py`
+- Validation logic in `validators.py`
+- Constants in `constants.py`
+- No scattered or duplicated logic
+
+## 📋 File Structure Details
+
+### Core AI Modules
+
+| File | Purpose | Dependencies |
+|------|---------|--------------|
+| `ai_core.py` | Shared AI operations | `utils.ai_processor` (legacy) |
+| `validators.py` | All validation logic | `constants`, models |
+| `constants.py` | AI constants | None |
+| `engine.py` | Execution orchestrator | `ai_core`, `base`, `tracker` |
+| `base.py` | Base function class | None |
+| `tracker.py` | Progress/step tracking | None |
+| `registry.py` | Function registry | `base`, function modules |
+
+### Function Files
+
+| File | Function Class | Legacy Function | Status |
+|------|----------------|-----------------|--------|
+| `auto_cluster.py` | `AutoClusterFunction` | N/A (uses engine) | ✅ Updated |
+| `generate_ideas.py` | `GenerateIdeasFunction` | `generate_ideas_core()` | ✅ Created |
+| `generate_content.py` | `GenerateContentFunction` | `generate_content_core()` | ✅ Created |
+| `generate_images.py` | `GenerateImagesFunction` | `generate_images_core()` | ✅ Created |
+
+## 🔄 Import Path Changes
+
+### Old Imports (Still work, but deprecated)
+```python
+from igny8_core.utils.ai_processor import AIProcessor
+from igny8_core.modules.planner.tasks import _generate_single_idea_core
+```
+
+### New Imports (Recommended)
+```python
+from igny8_core.ai.functions.generate_ideas import generate_ideas_core
+from igny8_core.ai.functions.generate_content import generate_content_core
+from igny8_core.ai.functions.generate_images import generate_images_core
+from igny8_core.ai.ai_core import AICore
+from igny8_core.ai.validators import validate_ids, validate_cluster_limits
+from igny8_core.ai.constants import MODEL_RATES, DEFAULT_AI_MODEL
+```
+
+## ✅ Verification Checklist
+
+- [x] All function files created in `ai/functions/`
+- [x] Shared modules (`ai_core`, `validators`, `constants`) created
+- [x] No circular dependencies
+- [x] All imports updated in views and tasks
+- [x] Functions registered in registry
+- [x] `__init__.py` files updated
+- [x] Backward compatibility maintained (legacy functions still work)
+- [x] No linting errors
+- [x] Structure matches required layout
+
+## 🎯 Next Steps (Future Stages)
+
+- **Stage 2**: Inject tracker into all functions
+- **Stage 3**: Simplify logging
+- **Stage 4**: Clean up legacy code
+
+## 📝 Notes
+
+- Legacy `AIProcessor` from `utils.ai_processor` is still used by `ai_core.py` as a wrapper
+- This will be refactored in later stages
+- All existing API endpoints continue to work
+- No functional changes - only structural reorganization
+