igny8/docs/AI-NEW-DESIGN/STAGE1-REORGANIZATION-COMPLETE.md

Stage 1 - AI Folder Structure & Functional Split - COMPLETE ✅

Summary

Successfully reorganized the AI backend into a clean, modular structure where every AI function lives inside its own file within /ai/functions/.

✅ Completed Deliverables

1. Folder Structure Created

backend/igny8_core/ai/
├── functions/
│   ├── __init__.py ✅
│   ├── auto_cluster.py ✅
│   ├── generate_ideas.py ✅
│   ├── generate_content.py ✅
│   └── generate_images.py ✅
├── ai_core.py ✅ (Shared operations)
├── validators.py ✅ (Consolidated validation)
├── constants.py ✅ (Model pricing, valid models)
├── engine.py ✅ (Updated to use AICore)
├── tracker.py ✅ (Existing)
├── base.py ✅ (Existing)
├── processor.py ✅ (Existing wrapper)
├── registry.py ✅ (Updated with new functions)
└── __init__.py ✅ (Updated exports)

2. Shared Modules Created

ai_core.py

• Purpose: Shared operations for all AI functions
• Features:
  • API call construction (call_openai)
  • Model selection (get_model, get_api_key)
  • Response parsing (extract_json)
  • Image generation (generate_image)
  • Cost calculation (calculate_cost)
• Status: ✅ Complete

validators.py

• Purpose: Consolidated validation logic
• Functions:
  • validate_ids() - Base ID validation
  • validate_keywords_exist() - Keyword existence check
  • validate_cluster_limits() - Plan limit checks
  • validate_cluster_exists() - Cluster existence
  • validate_tasks_exist() - Task existence
  • validate_api_key() - API key validation
  • validate_model() - Model validation
  • validate_image_size() - Image size validation
• Status: ✅ Complete

constants.py

• Purpose: AI-related constants
• Constants:
  • MODEL_RATES - Text model pricing
  • IMAGE_MODEL_RATES - Image model pricing
  • VALID_OPENAI_IMAGE_MODELS - Valid image models
  • VALID_SIZES_BY_MODEL - Valid sizes per model
  • DEFAULT_AI_MODEL - Default model name
  • JSON_MODE_MODELS - Models supporting JSON mode
• Status: ✅ Complete

3. Function Files Created

functions/auto_cluster.py

• Status: ✅ Updated to use new validators and AICore
• Changes:
  • Uses validate_ids(), validate_keywords_exist(), validate_cluster_limits() from validators
  • Uses AICore.extract_json() for JSON parsing
  • Maintains backward compatibility

functions/generate_ideas.py

• Status: ✅ Created
• Features:
  • GenerateIdeasFunction class (BaseAIFunction)
  • generate_ideas_core() legacy function for backward compatibility
  • Uses AICore for API calls
  • Uses validators for validation

functions/generate_content.py

• Status: ✅ Created
• Features:
  • GenerateContentFunction class (BaseAIFunction)
  • generate_content_core() legacy function for backward compatibility
  • Uses AICore for API calls
  • Uses validators for validation

functions/generate_images.py

• Status: ✅ Created
• Features:
  • GenerateImagesFunction class (BaseAIFunction)
  • generate_images_core() legacy function for backward compatibility
  • Uses AICore for image generation
  • Uses validators for validation

4. Import Paths Updated

Updated Files:

• ✅ modules/planner/views.py - Uses generate_ideas_core from new location
• ✅ modules/planner/tasks.py - Imports generate_ideas_core from new location
• ✅ modules/writer/tasks.py - Imports generate_content_core and generate_images_core from new locations
• ✅ ai/engine.py - Uses AICore instead of AIProcessor
• ✅ ai/functions/auto_cluster.py - Uses new validators and AICore
• ✅ ai/registry.py - Registered all new functions
• ✅ ai/__init__.py - Exports all new modules

5. Dependencies Verified

No Circular Dependencies ✅

• Functions depend on: ai_core, validators, constants, base
• ai_core depends on: utils.ai_processor (legacy, will be refactored later)
• validators depends on: constants, models
• engine depends on: ai_core, base, tracker
• All imports are clean and modular

Modular Structure ✅

• Each function file is self-contained
• Shared logic in ai_core.py
• Validation logic in validators.py
• Constants in constants.py
• No scattered or duplicated logic

📋 File Structure Details

Core AI Modules

File	Purpose	Dependencies
ai_core.py	Shared AI operations	utils.ai_processor (legacy)
validators.py	All validation logic	constants, models
constants.py	AI constants	None
engine.py	Execution orchestrator	ai_core, base, tracker
base.py	Base function class	None
tracker.py	Progress/step tracking	None
registry.py	Function registry	base, function modules

Function Files

File	Function Class	Legacy Function	Status
auto_cluster.py	AutoClusterFunction	N/A (uses engine)	✅ Updated
generate_ideas.py	GenerateIdeasFunction	generate_ideas_core()	✅ Created
generate_content.py	GenerateContentFunction	generate_content_core()	✅ Created
generate_images.py	GenerateImagesFunction	generate_images_core()	✅ Created

🔄 Import Path Changes

Old Imports (Still work, but deprecated)

from igny8_core.utils.ai_processor import AIProcessor
from igny8_core.modules.planner.tasks import _generate_single_idea_core

New Imports (Recommended)

from igny8_core.ai.functions.generate_ideas import generate_ideas_core
from igny8_core.ai.functions.generate_content import generate_content_core
from igny8_core.ai.functions.generate_images import generate_images_core
from igny8_core.ai.ai_core import AICore
from igny8_core.ai.validators import validate_ids, validate_cluster_limits
from igny8_core.ai.constants import MODEL_RATES, DEFAULT_AI_MODEL

✅ Verification Checklist

• All function files created in ai/functions/
• Shared modules (ai_core, validators, constants) created
• No circular dependencies
• All imports updated in views and tasks
• Functions registered in registry
• __init__.py files updated
• Backward compatibility maintained (legacy functions still work)
• No linting errors
• Structure matches required layout

🎯 Next Steps (Future Stages)

• Stage 2: Inject tracker into all functions
• Stage 3: Simplify logging
• Stage 4: Clean up legacy code

📝 Notes

• Legacy AIProcessor from utils.ai_processor is still used by ai_core.py as a wrapper
• This will be refactored in later stages
• All existing API endpoints continue to work
• No functional changes - only structural reorganization