22 KiB
22 KiB
Complete End-to-End Analysis & Restructuring Plan
Current State Summary (from actual database queries)
AIModelConfig (6 records - all needed):
| model_name | type | provider | active | default | cost | tokens_per_credit |
|---|---|---|---|---|---|---|
| gpt-4o-mini | text | openai | ✅ | ❌ | $0.15/$0.60 per 1M | 10,000 |
| gpt-4o | text | openai | ❌ | ❌ | $2.50/$10.00 per 1M | 1,000 |
| gpt-5.1 | text | openai | ✅ | ✅ | $1.25/$10.00 per 1M | 1,000 |
| runware:97@1 | image | runware | ✅ | ❌ | $0.012/image | 1 credit/image |
| dall-e-3 | image | openai | ✅ | ✅ | $0.04/image | 5 credits/image |
| google:4@2 | image | runware | ✅ | ❌ | $0.14/image | 15 credits/image |
Current End-to-End Flow (Traced from Code)
┌─────────────────────────────────────────────────────────────────────────┐
│ CURRENT ARCHITECTURE │
└─────────────────────────────────────────────────────────────────────────┘
1. CONFIGURATION LAYER
┌────────────────────────────────────────┐
│ GlobalIntegrationSettings (singleton) │ ← API keys stored here
│ - openai_api_key │
│ - runware_api_key │
│ - anthropic_api_key (unused) │
│ - bria_api_key (unused) │
│ - openai_model: gpt-4o-mini ❌ │ ← Should be gpt-5.1
│ - runware_model: bria:10@1 ❌ │ ← Model doesn't exist!
│ - HARDCODED CHOICES duplicating DB │
└────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ AIModelConfig (database) │ ← Source of truth for models
│ - model_name, provider, costs │
│ - is_active, is_default │
└────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ constants.py │ ← DUPLICATE/LEGACY
│ - MODEL_RATES (hardcoded) │
│ - IMAGE_MODEL_RATES (hardcoded) │
└────────────────────────────────────────┘
2. SETTINGS RESOLUTION LAYER
┌────────────────────────────────────────┐
│ settings.py │
│ get_model_config(function, account) │
│ - Gets model from GlobalIntegration │
│ - Gets max_tokens from AIModelConfig │
│ - Allows IntegrationSettings override │
└────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ model_registry.py │
│ ModelRegistry.get_model(model_id) │
│ - Try DB (AIModelConfig) first │
│ - Fallback to constants.py ❌ │ ← Should only use DB
└────────────────────────────────────────┘
3. AI EXECUTION LAYER
┌────────────────────────────────────────┐
│ engine.py (AIEngine) │
│ - Orchestrates all AI functions │
│ - Progress tracking, cost tracking │
└────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ ai_core.py (AICore) │
│ - _load_account_settings() │ ← Gets API keys from Global
│ - run_ai_request() for text │
│ - generate_image() for images │
│ - Uses IMAGE_MODEL_RATES fallback ❌ │
└────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ ai/functions/ │
│ - generate_images.py │
│ - generate_content.py │
│ - auto_cluster.py │
│ - Each function uses AICore │
└────────────────────────────────────────┘
4. CREDIT CALCULATION LAYER
┌────────────────────────────────────────┐
│ CreditCostConfig (database) │
│ - operation_type │
│ - tokens_per_credit │
│ - min_credits │
│ - price_per_credit_usd │
│ - For images: 50 tokens/credit, min 5 │
└────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ CreditService │
│ - calculate_credits_from_tokens() │
│ - deduct_credits_for_operation() │
│ - Text: tokens → credits AFTER call │
│ - Images: ??? (not token-based) │
└────────────────────────────────────────┘
5. FRONTEND (Sites/Settings.tsx)
┌────────────────────────────────────────┐
│ HARDCODED model choices ❌ │
│ - QUALITY_TO_CONFIG │
│ - RUNWARE_MODEL_CHOICES │
│ - DALLE_MODEL_CHOICES │
│ - MODEL_LANDSCAPE_SIZES │
└────────────────────────────────────────┘
Problems Identified
| # | Problem | Location | Impact |
|---|---|---|---|
| 1 | GlobalIntegrationSettings has hardcoded model choices | global_settings_models.py | Duplicates AIModelConfig |
| 2 | runware_model = "bria:10@1" but model doesn't exist | GlobalIntegrationSettings | Broken fallback |
| 3 | API keys mixed with model config | GlobalIntegrationSettings | No separation of concerns |
| 4 | IMAGE_MODEL_RATES still used as fallback | ai_core.py, model_registry.py | Inconsistent pricing |
| 5 | Frontend hardcodes model choices | Settings.tsx | Not dynamic |
| 6 | Image credit calculation unclear | CreditService | Not based on cost_per_image |
| 7 | constants.py duplicates DB data | constants.py | Maintenance burden |
Target Architecture
┌─────────────────────────────────────────────────────────────────────────┐
│ TARGET ARCHITECTURE │
└─────────────────────────────────────────────────────────────────────────┘
1. NEW: IntegrationProvider Model (stores ALL 3rd party API keys)
┌────────────────────────────────────────┐
│ IntegrationProvider │ ← Future-proof: ALL integrations
│ - provider_id: str (primary key) │
│ Examples: openai, runware, google, │
│ resend, stripe, etc. │
│ - display_name: str │
│ - provider_type: ai | email | payment │
│ - api_key: encrypted str │
│ - api_endpoint: URL (optional) │
│ - is_active: bool │
│ - config: JSON (rate limits, etc.) │
└────────────────────────────────────────┘
│
▼
2. CLEANED: AIModelConfig (references IntegrationProvider)
┌────────────────────────────────────────┐
│ AIModelConfig │
│ - model_name: str │
│ - display_name: str │
│ - model_type: text | image │
│ - provider: str → IntegrationProvider │
│ - cost fields (unchanged) │
│ + credits_per_image: int (NEW) │ ← For image models
│ + tokens_per_credit: int (NEW) │ ← For text models
│ + quality_tier: basic|quality|premium │ ← For UI display
│ - is_default: bool │ ← Loads automatically
└────────────────────────────────────────┘
│
▼
3. SIMPLIFIED: GlobalIntegrationSettings
┌────────────────────────────────────────┐
│ GlobalIntegrationSettings │
│ NO hardcoded model names │
│ NO API keys (in IntegrationProvider) │
│ Loads defaults from AIModelConfig │
│ where is_default=True │
│ - image_style: str │
│ - max_in_article_images: int │
│ - image_quality: str │
└────────────────────────────────────────┘
│
▼
4. UNIFIED: Model Resolution
┌────────────────────────────────────────┐
│ ModelRegistry │
│ - get_default_model(type) → from DB │
│ - get_model(model_id) → AIModelConfig │
│ - get_provider(id) → IntegrationProv │
│ - get_api_key(provider) → key │
│ - NO fallback to constants │
│ - NO hardcoded defaults │
└────────────────────────────────────────┘
│
▼
5. DYNAMIC: Frontend API
┌────────────────────────────────────────┐
│ GET /api/v1/system/ai-models/ │
│ Returns models from DB with defaults │
│ marked, no hardcoding needed │
└────────────────────────────────────────┘
Implementation Plan (Complete)
Phase 1: Database Schema Changes
1.1 Create IntegrationProvider Model (Future-proof for ALL integrations)
File: models.py
class IntegrationProvider(models.Model):
"""
Centralized 3rd party integration provider configuration.
Single location for ALL external service API keys and configs.
"""
PROVIDER_TYPE_CHOICES = [
('ai', 'AI Provider'),
('email', 'Email Service'),
('payment', 'Payment Gateway'),
('storage', 'Storage Service'),
('other', 'Other'),
]
provider_id = models.CharField(max_length=50, unique=True, primary_key=True)
# Examples: openai, runware, google, resend, stripe, aws_s3, etc.
display_name = models.CharField(max_length=100)
provider_type = models.CharField(max_length=20, choices=PROVIDER_TYPE_CHOICES, default='ai')
api_key = models.CharField(max_length=500, blank=True) # Should be encrypted
api_secret = models.CharField(max_length=500, blank=True) # For services needing secret
api_endpoint = models.URLField(blank=True) # Custom endpoint if needed
webhook_secret = models.CharField(max_length=500, blank=True) # For Stripe etc.
is_active = models.BooleanField(default=True)
config = models.JSONField(default=dict, blank=True) # Rate limits, regions, etc.
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
class Meta:
db_table = 'igny8_integration_providers'
verbose_name = 'Integration Provider'
verbose_name_plural = 'Integration Providers'
1.2 Add fields to AIModelConfig
# Add to AIModelConfig for IMAGE models
credits_per_image = models.IntegerField(
null=True, blank=True,
help_text="Fixed credits per image generated. For image models only."
)
# Add to AIModelConfig for TEXT models
tokens_per_credit = models.IntegerField(
null=True, blank=True,
help_text="Number of tokens that equal 1 credit. For text models only."
)
# Add quality tier for UI display (image models)
quality_tier = models.CharField(
max_length=20,
choices=[('basic', 'Basic'), ('quality', 'Quality'), ('premium', 'Premium')],
null=True, blank=True,
help_text="Quality tier for frontend UI display"
)
1.3 Migration to populate data
Create IntegrationProvider records:
| provider_id | display_name | provider_type | Notes |
|-------------|-------------------|---------------|--------------------------|
| openai | OpenAI | ai | GPT models, DALL-E |
| runware | Runware | ai | Image generation |
| google | Google Cloud | ai | Future: Gemini, etc. |
| resend | Resend | email | Transactional email |
| stripe | Stripe | payment | Payment processing |
Update AIModelConfig with credit/token values:
| model_name | type | tokens_per_credit | credits_per_image | quality_tier |
|---------------|-------|-------------------|-------------------|--------------|
| gpt-4o-mini | text | 10000 | - | - |
| gpt-4o | text | 1000 | - | - |
| gpt-5.1 | text | 1000 | - | - |
| runware:97@1 | image | - | 1 | basic |
| dall-e-3 | image | - | 5 | quality |
| google:4@2 | image | - | 15 | premium |
Phase 2: Backend Code Changes
2.1 Remove hardcoded constants
File: constants.py
- Remove: MODEL_RATES, IMAGE_MODEL_RATES
- Keep: JSON_MODE_MODELS (or move to AIModelConfig.supports_json_mode check)
2.2 Update ModelRegistry
File: model_registry.py
- Remove fallback to constants.py
- Add:
get_default_model(model_type) → AIModelConfig where is_default=True - Add:
get_provider(provider_id) → IntegrationProvider - Add:
get_api_key(provider_id) → str - NO hardcoded model names - always query DB
2.3 Update AICore
File: ai_core.py
- Change
_load_account_settings()to use IntegrationProvider for API keys - Remove IMAGE_MODEL_RATES import and usage
- Use
ModelRegistry.get_default_model('text')instead of hardcoded model - Use
ModelRegistry.calculate_cost()exclusively
2.4 Update CreditService
File: credit_service.py
For IMAGE models:
def calculate_credits_for_image(model_name: str, num_images: int) -> int:
"""Calculate credits for image generation from AIModelConfig"""
model = AIModelConfig.objects.get(model_name=model_name, is_active=True)
return model.credits_per_image * num_images
For TEXT models:
def calculate_credits_from_tokens(model_name: str, total_tokens: int) -> int:
"""Calculate credits from token usage based on model's tokens_per_credit"""
model = AIModelConfig.objects.get(model_name=model_name, is_active=True)
tokens_per_credit = model.tokens_per_credit or 1000 # fallback
return math.ceil(total_tokens / tokens_per_credit)
2.5 Simplify GlobalIntegrationSettings
File: global_settings_models.py
- Remove: All API key fields (moved to IntegrationProvider)
- Remove: All hardcoded CHOICES
- Remove: Model name fields (defaults loaded from AIModelConfig.is_default)
- Keep: image_style, max_in_article_images, image_quality
- Add: Helper methods to get defaults from AIModelConfig
Phase 3: API Endpoints
3.1 New endpoint: GET /api/v1/system/ai-models/
Returns all active models from database with defaults marked (no hardcoding):
{
"text_models": [
{
"model_name": "gpt-5.1",
"display_name": "GPT-5.1 Premium",
"is_default": true,
"tokens_per_credit": 1000,
"max_output_tokens": 8192
},
{
"model_name": "gpt-4o-mini",
"display_name": "GPT-4o Mini",
"is_default": false,
"tokens_per_credit": 10000,
"max_output_tokens": 16000
}
],
"image_models": [
{
"model_name": "runware:97@1",
"display_name": "Basic",
"quality_tier": "basic",
"is_default": false,
"credits_per_image": 1,
"valid_sizes": ["1024x1024", "1280x768"]
},
{
"model_name": "dall-e-3",
"display_name": "Quality",
"quality_tier": "quality",
"is_default": true,
"credits_per_image": 5,
"valid_sizes": ["1024x1024", "1792x1024"]
},
{
"model_name": "google:4@2",
"display_name": "Premium",
"quality_tier": "premium",
"is_default": false,
"credits_per_image": 15,
"valid_sizes": ["1024x1024", "1376x768"]
}
],
"image_settings": {
"style": "photorealistic",
"max_in_article_images": 4,
"quality": "hd"
}
}
Phase 4: Frontend Changes
4.1 Settings.tsx
- Remove: QUALITY_TO_CONFIG, RUNWARE_MODEL_CHOICES, DALLE_MODEL_CHOICES hardcodes
- Remove: MODEL_LANDSCAPE_SIZES hardcodes
- Add: Fetch models from
/api/v1/system/ai-models/ - Load valid_sizes from API response per model
- Display to user (no provider/model names visible):
- "Basic (1 credit/image)"
- "Quality (5 credits/image)"
- "Premium (15 credits/image)"
- Default selection: model where
is_default=truefrom API
Phase 5: Cleanup
5.1 Files to clean/remove
- Remove unused fields from GlobalIntegrationSettings: anthropic_, bria_, all API key fields, hardcoded model fields
- Remove deprecated methods from AICore
- Update all imports removing constants.py usage
- Remove CreditCostConfig dependency for image operations (use AIModelConfig.credits_per_image directly)
Credit Calculation Summary
Text Models (token-based):
| Model | tokens_per_credit | Example: 5000 tokens |
|---|---|---|
| gpt-5.1 | 1,000 | 5 credits |
| gpt-4o | 1,000 | 5 credits |
| gpt-4o-mini | 10,000 | 1 credit |
Image Models (per-image):
| Model | credits_per_image | quality_tier | Display |
|---|---|---|---|
| runware:97@1 | 1 | basic | "Basic (1 credit/image)" |
| dall-e-3 | 5 | quality | "Quality (5 credits/image)" |
| google:4@2 | 15 | premium | "Premium (15 credits/image)" |
Migration Order
- Create IntegrationProvider model + migration
- Add credits_per_image, tokens_per_credit, quality_tier to AIModelConfig + migration
- Data migration: populate IntegrationProvider, update AIModelConfig with credit values
- Update ModelRegistry (remove constants fallback, add get_default_model)
- Update AICore (use IntegrationProvider for keys)
- Update CreditService (model-based credit calculation)
- Create API endpoint /api/v1/system/ai-models/
- Update frontend (load from API, no hardcodes)
- Cleanup GlobalIntegrationSettings (remove API keys, hardcoded choices)
- Remove constants.py hardcoded rates
Files Changed Summary
| File | Action |
|---|---|
| models.py | Add IntegrationProvider, update AIModelConfig with credit fields |
| model_registry.py | Remove constants fallback, add get_default_model(), get_provider() |
| ai_core.py | Use IntegrationProvider for keys, ModelRegistry for defaults |
| constants.py | Remove MODEL_RATES, IMAGE_MODEL_RATES |
| credit_service.py | Model-based credit calculation for both text and images |
| global_settings_models.py | Remove API keys, hardcoded choices, model fields |
| backend/igny8_core/api/views/system.py | Add ai-models endpoint |
| Settings.tsx | Load models from API, remove all hardcodes |
Key Principles
- No hardcoded model names - GlobalIntegrationSettings loads defaults from AIModelConfig where is_default=True
- Single source of truth - AIModelConfig is THE source for all model info including credit costs
- Future-proof - IntegrationProvider handles ALL 3rd party integrations (AI, email, payment, etc.)
- Dynamic frontend - All model choices loaded from API, not hardcoded
- Configurable credits - Change credits_per_image or tokens_per_credit in admin, no code changes needed