940 lines
35 KiB
Markdown
940 lines
35 KiB
Markdown
# Safe Migration & Testing Plan
|
|
|
|
## Overview
|
|
|
|
This document ensures the refactor is done safely with zero breakage. All existing functionality must work identically before and after migration.
|
|
|
|
---
|
|
|
|
## Current Working System Components
|
|
|
|
### 1. AI Functions (Core - Used by ALL paths)
|
|
|
|
All AI operations go through these functions in `backend/igny8_core/ai/functions/`:
|
|
|
|
| Function | File | Operation Type | Credits | Called Via |
|
|
|----------|------|----------------|---------|------------|
|
|
| Auto Cluster | `auto_cluster.py` | `clustering` | Token-based | AIEngine |
|
|
| Generate Ideas | `generate_ideas.py` | `idea_generation` | Token-based | AIEngine |
|
|
| Generate Content | `generate_content.py` | `content_generation` | Token-based | AIEngine |
|
|
| Generate Image Prompts | `generate_image_prompts.py` | `image_prompt_extraction` | Token-based | AIEngine |
|
|
| Generate Images | `generate_images.py` | `image_generation` | Per-image | AICore direct |
|
|
| Optimize Content | `optimize_content.py` | `content_optimization` | Token-based | OptimizerService |
|
|
|
|
**Operation Type Mapping** (in `engine.py`):
|
|
```python
|
|
mapping = {
|
|
'auto_cluster': 'clustering',
|
|
'generate_ideas': 'idea_generation',
|
|
'generate_content': 'content_generation',
|
|
'generate_image_prompts': 'image_prompt_extraction',
|
|
'generate_images': 'image_generation',
|
|
'generate_site_structure': 'site_structure_generation',
|
|
}
|
|
```
|
|
|
|
### 1.1 Non-AIEngine Credit Deduction Points
|
|
|
|
| Service | Operation Type | Direct Credit Deduction |
|
|
|---------|---------------|------------------------|
|
|
| LinkerService | `internal_linking` | ✅ Uses CreditService directly |
|
|
| OptimizerService | `content_optimization` | ✅ Uses CreditService directly |
|
|
|
|
### 2. Execution Paths (All use same AI functions)
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ ALL EXECUTION PATHS │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
PATH 1: Manual Module Page Buttons (via run_ai_task Celery task)
|
|
├── Planner Page → "Cluster Keywords" button → api/planner.py → run_ai_task.delay('auto_cluster')
|
|
├── Planner Page → "Generate Ideas" button → api/planner.py → run_ai_task.delay('generate_ideas')
|
|
├── Writer Page → "Generate Content" button → api/writer.py → run_ai_task.delay('generate_content')
|
|
├── Writer Page → "Extract Prompts" button → api/writer.py → run_ai_task.delay('generate_image_prompts')
|
|
├── Writer Page → "Generate Images" button → api/writer.py → process_image_generation_queue.delay()
|
|
└── All go through: Celery task → AIEngine → AI Function → AICore → API → CreditService
|
|
|
|
PATH 2: Automation Page Manual Run
|
|
├── Automation Page → "Run Now" button
|
|
├── automation/tasks.py → run_automation_task
|
|
├── AutomationService.run_automation()
|
|
│ Stage 1: _run_clustering() → AIEngine.execute('auto_cluster')
|
|
│ Stage 2: _run_idea_generation() → AIEngine.execute('generate_ideas')
|
|
│ Stage 3: _run_task_creation() → Database only (no AI)
|
|
│ Stage 4: _run_content_generation() → AIEngine.execute('generate_content')
|
|
│ Stage 5: _run_image_prompt_extraction() → AIEngine.execute('generate_image_prompts')
|
|
│ Stage 6: _run_image_generation() → process_image_generation_queue.delay()
|
|
│ Stage 7: _run_publishing() → WordPress API (no AI)
|
|
└── Same credit deduction path (stages 1,2,4,5 via AIEngine, stage 6 via AICore)
|
|
|
|
PATH 3: Scheduled Automation Run
|
|
├── Celery Beat scheduler → check_scheduled_automations task (hourly)
|
|
├── automation_tasks.py → run_automation_task.delay()
|
|
├── AutomationService.run_automation()
|
|
├── Calls same AI functions through AIEngine
|
|
└── Same credit deduction path
|
|
|
|
PATH 4: Direct Service Operations (Bypass AIEngine)
|
|
├── Linker Module → LinkerService → CreditService.deduct_credits_for_operation('internal_linking')
|
|
├── Optimizer Module → OptimizerService → CreditService.deduct_credits_for_operation('content_optimization')
|
|
└── These use direct credit deduction, not via AIEngine
|
|
```
|
|
|
|
### 3. Credit Flow (Current)
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ CURRENT CREDIT FLOW │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
For TEXT operations (clustering, ideas, content) - VIA AIEngine:
|
|
1. API endpoint receives request
|
|
2. Celery task run_ai_task triggered
|
|
3. AIEngine.execute(function_name, payload)
|
|
a. Step 2.5: CreditService.check_credits() - PRE-CHECK
|
|
b. Step 3: AICore.run_ai_request() - makes OpenAI API call
|
|
c. Response includes token usage (input_tokens, output_tokens)
|
|
d. Step 5.5: CreditService.deduct_credits_for_operation() - DEDUCTION
|
|
4. CreditService.calculate_credits_from_tokens() uses CreditCostConfig
|
|
5. Creates CreditTransaction (balance history)
|
|
6. Creates CreditUsageLog (detailed tracking)
|
|
7. Updates Account.credits
|
|
|
|
For IMAGE operations - VIA process_image_generation_queue:
|
|
1. API endpoint receives request
|
|
2. Celery task process_image_generation_queue triggered
|
|
3. For each image: AICore.generate_image()
|
|
a. Makes Runware/OpenAI API call
|
|
b. Calculates cost via ModelRegistry.calculate_cost()
|
|
c. Returns image URL
|
|
4. ⚠️ Credit deduction handled AFTER all images generated
|
|
5. Uses CreditCostConfig.image_generation entry (min_credits, tokens_per_credit)
|
|
6. Creates CreditUsageLog
|
|
|
|
For DIRECT SERVICE OPERATIONS (Linker, Optimizer):
|
|
1. Service method called (e.g., LinkerService.generate_links())
|
|
2. AI call made directly to AICore or external service
|
|
3. CreditService.deduct_credits_for_operation() called directly
|
|
4. Uses operation-specific CreditCostConfig entry
|
|
```
|
|
|
|
### 3.1 Celery Task Definitions
|
|
|
|
| Task File | Task Name | Entry Point For |
|
|
|-----------|-----------|-----------------|
|
|
| `ai/tasks.py` | `run_ai_task` | All manual AI buttons (universal task) |
|
|
| `ai/tasks.py` | `process_image_generation_queue` | Image generation queue |
|
|
| `automation/tasks.py` | `check_scheduled_automations` | Hourly scheduler check |
|
|
| `automation/tasks.py` | `run_automation_task` | Full automation pipeline |
|
|
| `automation/tasks.py` | `resume_paused_automation` | Resume after pause |
|
|
|
|
### 4. Key Files Currently Working
|
|
|
|
| File | Purpose | Must Keep Working |
|
|
|------|---------|-------------------|
|
|
| `ai/engine.py` | Orchestrates AI functions | ✅ |
|
|
| `ai/ai_core.py` | API calls, key loading | ✅ |
|
|
| `ai/model_registry.py` | Model config lookup | ✅ |
|
|
| `ai/settings.py` | get_model_config() | ✅ |
|
|
| `ai/tasks.py` | Celery tasks | ✅ |
|
|
| `ai/functions/*.py` | All AI functions | ✅ |
|
|
| `billing/services/credit_service.py` | Credit calculation | ✅ |
|
|
| `billing/models.py` | CreditUsageLog, CreditCostConfig | ✅ |
|
|
| `automation/services/automation_service.py` | Automation runner | ✅ |
|
|
| `modules/system/global_settings_models.py` | API keys, defaults | ✅ |
|
|
|
|
---
|
|
|
|
## Potential Issues to Verify Before Migration
|
|
|
|
### ⚠️ Issue 1: Image Credit Deduction May Be Incomplete
|
|
|
|
The `process_image_generation_queue` task calculates cost via `ModelRegistry.calculate_cost()` but credit deduction needs verification:
|
|
- Does it call `CreditService.deduct_credits_for_operation()`?
|
|
- Or does it just log cost without deducting credits?
|
|
|
|
**Verification Query:**
|
|
```bash
|
|
docker exec igny8_backend python manage.py shell -c "
|
|
from igny8_core.business.billing.models import CreditUsageLog
|
|
logs = CreditUsageLog.objects.filter(operation_type='image_generation').order_by('-created_at')[:5]
|
|
for log in logs:
|
|
print(f'{log.operation_type}: {log.credits_used} credits, cost=${log.cost_usd}, model={log.model_used}')
|
|
if not logs:
|
|
print('⚠️ NO image_generation credit logs found - may not be tracking credits!')
|
|
"
|
|
```
|
|
|
|
### ⚠️ Issue 2: GlobalIntegrationSettings.runware_model Points to Non-Existent Model
|
|
|
|
Current value: `bria:10@1` - but this model doesn't exist in AIModelConfig.
|
|
This may cause fallback issues.
|
|
|
|
### ⚠️ Issue 3: Multiple Credit Calculation Paths
|
|
|
|
Credits are calculated in different ways depending on the path:
|
|
- Via AIEngine → uses `CreditCostConfig` token-based calculation
|
|
- Via direct service → uses `CreditCostConfig` with operation-specific rates
|
|
- Via image generation → may use different calculation
|
|
|
|
Need to unify after migration.
|
|
|
|
---
|
|
|
|
## Pre-Migration Baseline Capture
|
|
|
|
### Step 1: Database State Snapshot
|
|
|
|
Run this BEFORE any changes to establish baseline:
|
|
|
|
```bash
|
|
docker exec igny8_backend python manage.py shell << 'EOF'
|
|
import json
|
|
from datetime import datetime
|
|
|
|
print("=" * 80)
|
|
print(f"BASELINE SNAPSHOT - {datetime.now().isoformat()}")
|
|
print("=" * 80)
|
|
|
|
# 1. AIModelConfig
|
|
print("\n=== AIModelConfig ===")
|
|
from igny8_core.business.billing.models import AIModelConfig
|
|
for m in AIModelConfig.objects.all().order_by('model_type', 'sort_order'):
|
|
print(json.dumps({
|
|
'model_name': m.model_name,
|
|
'model_type': m.model_type,
|
|
'provider': m.provider,
|
|
'is_active': m.is_active,
|
|
'is_default': m.is_default,
|
|
'cost_per_image': str(m.cost_per_image) if m.cost_per_image else None,
|
|
'input_cost_per_1m': str(m.input_cost_per_1m) if m.input_cost_per_1m else None,
|
|
'output_cost_per_1m': str(m.output_cost_per_1m) if m.output_cost_per_1m else None,
|
|
}))
|
|
|
|
# 2. CreditCostConfig
|
|
print("\n=== CreditCostConfig ===")
|
|
from igny8_core.business.billing.models import CreditCostConfig
|
|
for c in CreditCostConfig.objects.filter(is_active=True):
|
|
print(json.dumps({
|
|
'operation_type': c.operation_type,
|
|
'tokens_per_credit': c.tokens_per_credit,
|
|
'min_credits': c.min_credits,
|
|
'price_per_credit_usd': str(c.price_per_credit_usd),
|
|
}))
|
|
|
|
# 3. GlobalIntegrationSettings
|
|
print("\n=== GlobalIntegrationSettings ===")
|
|
from igny8_core.modules.system.global_settings_models import GlobalIntegrationSettings
|
|
g = GlobalIntegrationSettings.get_instance()
|
|
print(json.dumps({
|
|
'openai_model': g.openai_model,
|
|
'openai_temperature': g.openai_temperature,
|
|
'openai_max_tokens': g.openai_max_tokens,
|
|
'default_text_provider': g.default_text_provider,
|
|
'dalle_model': g.dalle_model,
|
|
'runware_model': g.runware_model,
|
|
'default_image_service': g.default_image_service,
|
|
'image_style': g.image_style,
|
|
'max_in_article_images': g.max_in_article_images,
|
|
'has_openai_key': bool(g.openai_api_key),
|
|
'has_runware_key': bool(g.runware_api_key),
|
|
}))
|
|
|
|
# 4. Recent Credit Usage (last 20)
|
|
print("\n=== Recent CreditUsageLog (last 20) ===")
|
|
from igny8_core.business.billing.models import CreditUsageLog
|
|
for log in CreditUsageLog.objects.all().order_by('-created_at')[:20]:
|
|
print(json.dumps({
|
|
'operation_type': log.operation_type,
|
|
'credits_used': log.credits_used,
|
|
'model_used': log.model_used,
|
|
'tokens_input': log.tokens_input,
|
|
'tokens_output': log.tokens_output,
|
|
'cost_usd': str(log.cost_usd) if log.cost_usd else None,
|
|
}))
|
|
|
|
# 5. Account Credits
|
|
print("\n=== Account Credits ===")
|
|
from igny8_core.auth.models import Account
|
|
for acc in Account.objects.all()[:5]:
|
|
print(json.dumps({
|
|
'account_id': acc.id,
|
|
'name': acc.name,
|
|
'credits': acc.credits,
|
|
}))
|
|
|
|
print("\n" + "=" * 80)
|
|
print("BASELINE CAPTURE COMPLETE")
|
|
print("=" * 80)
|
|
EOF
|
|
```
|
|
|
|
Save this output to: `4th-jan-refactor/baseline-snapshot.json`
|
|
|
|
---
|
|
|
|
## Integration Tests (Must Pass Before AND After)
|
|
|
|
### Test File: `backend/igny8_core/tests/test_ai_system_integration.py`
|
|
|
|
```python
|
|
"""
|
|
AI System Integration Tests
|
|
===========================
|
|
These tests verify the entire AI pipeline works end-to-end.
|
|
Run BEFORE migration to establish baseline.
|
|
Run AFTER each phase to verify nothing broke.
|
|
|
|
Usage:
|
|
docker exec igny8_backend python manage.py test igny8_core.tests.test_ai_system_integration
|
|
"""
|
|
from decimal import Decimal
|
|
from django.test import TestCase, TransactionTestCase
|
|
from django.db import transaction
|
|
from unittest.mock import patch, MagicMock
|
|
import json
|
|
|
|
|
|
class AIModelConfigTests(TestCase):
|
|
"""Test AIModelConfig loading and access"""
|
|
|
|
def test_text_models_exist(self):
|
|
"""All required text models exist and are configured"""
|
|
from igny8_core.business.billing.models import AIModelConfig
|
|
|
|
required_text_models = ['gpt-5.1', 'gpt-4o-mini']
|
|
for model_name in required_text_models:
|
|
model = AIModelConfig.objects.filter(model_name=model_name).first()
|
|
self.assertIsNotNone(model, f"Text model {model_name} not found")
|
|
self.assertEqual(model.model_type, 'text')
|
|
|
|
def test_image_models_exist(self):
|
|
"""All required image models exist and are configured"""
|
|
from igny8_core.business.billing.models import AIModelConfig
|
|
|
|
required_image_models = ['runware:97@1', 'dall-e-3', 'google:4@2']
|
|
for model_name in required_image_models:
|
|
model = AIModelConfig.objects.filter(model_name=model_name).first()
|
|
self.assertIsNotNone(model, f"Image model {model_name} not found")
|
|
self.assertEqual(model.model_type, 'image')
|
|
self.assertIsNotNone(model.cost_per_image, f"{model_name} missing cost_per_image")
|
|
|
|
def test_default_text_model_exists(self):
|
|
"""Exactly one default text model is set"""
|
|
from igny8_core.business.billing.models import AIModelConfig
|
|
|
|
defaults = AIModelConfig.objects.filter(model_type='text', is_default=True, is_active=True)
|
|
self.assertEqual(defaults.count(), 1, "Should have exactly 1 default text model")
|
|
self.assertEqual(defaults.first().model_name, 'gpt-5.1')
|
|
|
|
def test_default_image_model_exists(self):
|
|
"""Exactly one default image model is set"""
|
|
from igny8_core.business.billing.models import AIModelConfig
|
|
|
|
defaults = AIModelConfig.objects.filter(model_type='image', is_default=True, is_active=True)
|
|
self.assertEqual(defaults.count(), 1, "Should have exactly 1 default image model")
|
|
|
|
|
|
class ModelRegistryTests(TestCase):
|
|
"""Test ModelRegistry functionality"""
|
|
|
|
def test_get_model_from_db(self):
|
|
"""ModelRegistry.get_model() returns model from database"""
|
|
from igny8_core.ai.model_registry import ModelRegistry
|
|
|
|
model = ModelRegistry.get_model('gpt-5.1')
|
|
self.assertIsNotNone(model)
|
|
# Should be AIModelConfig instance or dict with model_name
|
|
if hasattr(model, 'model_name'):
|
|
self.assertEqual(model.model_name, 'gpt-5.1')
|
|
else:
|
|
self.assertEqual(model.get('model_name'), 'gpt-5.1')
|
|
|
|
def test_get_image_model(self):
|
|
"""ModelRegistry returns image models correctly"""
|
|
from igny8_core.ai.model_registry import ModelRegistry
|
|
|
|
for model_name in ['runware:97@1', 'dall-e-3', 'google:4@2']:
|
|
model = ModelRegistry.get_model(model_name)
|
|
self.assertIsNotNone(model, f"Model {model_name} not found")
|
|
|
|
def test_calculate_cost_text(self):
|
|
"""Cost calculation for text models works"""
|
|
from igny8_core.ai.model_registry import ModelRegistry
|
|
|
|
cost = ModelRegistry.calculate_cost('gpt-5.1', input_tokens=1000, output_tokens=500)
|
|
self.assertIsInstance(cost, (int, float, Decimal))
|
|
self.assertGreater(cost, 0)
|
|
|
|
def test_calculate_cost_image(self):
|
|
"""Cost calculation for image models works"""
|
|
from igny8_core.ai.model_registry import ModelRegistry
|
|
|
|
cost = ModelRegistry.calculate_cost('dall-e-3', num_images=1)
|
|
self.assertIsInstance(cost, (int, float, Decimal))
|
|
self.assertGreater(cost, 0)
|
|
|
|
|
|
class APIKeyLoadingTests(TestCase):
|
|
"""Test API key loading from GlobalIntegrationSettings"""
|
|
|
|
def test_openai_key_loads(self):
|
|
"""OpenAI API key loads from GlobalIntegrationSettings"""
|
|
from igny8_core.ai.ai_core import AICore
|
|
|
|
ai_core = AICore()
|
|
key = ai_core.get_api_key('openai')
|
|
self.assertIsNotNone(key, "OpenAI API key not configured")
|
|
self.assertTrue(len(key) > 10, "OpenAI API key too short")
|
|
|
|
def test_runware_key_loads(self):
|
|
"""Runware API key loads from GlobalIntegrationSettings"""
|
|
from igny8_core.ai.ai_core import AICore
|
|
|
|
ai_core = AICore()
|
|
key = ai_core.get_api_key('runware')
|
|
self.assertIsNotNone(key, "Runware API key not configured")
|
|
|
|
|
|
class CreditServiceTests(TestCase):
|
|
"""Test credit calculation and deduction"""
|
|
|
|
def test_calculate_credits_text_operation(self):
|
|
"""Credit calculation for text operations works"""
|
|
from igny8_core.business.billing.services.credit_service import CreditService
|
|
|
|
# Test content generation
|
|
credits = CreditService.calculate_credits_from_tokens(
|
|
'content_generation',
|
|
tokens_input=1000,
|
|
tokens_output=2000
|
|
)
|
|
self.assertIsInstance(credits, int)
|
|
self.assertGreater(credits, 0)
|
|
|
|
def test_calculate_credits_image_operation(self):
|
|
"""Credit calculation for image operations works"""
|
|
from igny8_core.business.billing.services.credit_service import CreditService
|
|
|
|
# Image generation uses min_credits from CreditCostConfig
|
|
credits = CreditService.calculate_credits_from_tokens(
|
|
'image_generation',
|
|
tokens_input=0,
|
|
tokens_output=0
|
|
)
|
|
self.assertIsInstance(credits, int)
|
|
self.assertGreaterEqual(credits, 1)
|
|
|
|
def test_credit_cost_config_exists(self):
|
|
"""All required CreditCostConfig entries exist"""
|
|
from igny8_core.business.billing.models import CreditCostConfig
|
|
|
|
required_ops = ['clustering', 'idea_generation', 'content_generation',
|
|
'image_generation', 'image_prompt_extraction']
|
|
for op in required_ops:
|
|
config = CreditCostConfig.objects.filter(operation_type=op, is_active=True).first()
|
|
self.assertIsNotNone(config, f"CreditCostConfig for {op} not found")
|
|
|
|
|
|
class AISettingsTests(TestCase):
|
|
"""Test get_model_config() function"""
|
|
|
|
def test_get_model_config_returns_config(self):
|
|
"""get_model_config() returns valid configuration"""
|
|
from igny8_core.ai.settings import get_model_config
|
|
from igny8_core.auth.models import Account
|
|
|
|
account = Account.objects.first()
|
|
if account:
|
|
config = get_model_config('content_generation', account)
|
|
|
|
self.assertIn('model', config)
|
|
self.assertIn('max_tokens', config)
|
|
self.assertIn('temperature', config)
|
|
self.assertIsNotNone(config['model'])
|
|
|
|
|
|
class AIFunctionValidationTests(TestCase):
|
|
"""Test AI function classes load correctly"""
|
|
|
|
def test_auto_cluster_function_loads(self):
|
|
"""AutoClusterFunction loads and validates"""
|
|
from igny8_core.ai.functions.auto_cluster import AutoClusterFunction
|
|
|
|
fn = AutoClusterFunction()
|
|
self.assertEqual(fn.get_name(), 'auto_cluster')
|
|
self.assertIsNotNone(fn.get_metadata())
|
|
|
|
def test_generate_ideas_function_loads(self):
|
|
"""GenerateIdeasFunction loads and validates"""
|
|
from igny8_core.ai.functions.generate_ideas import GenerateIdeasFunction
|
|
|
|
fn = GenerateIdeasFunction()
|
|
self.assertEqual(fn.get_name(), 'generate_ideas')
|
|
|
|
def test_generate_content_function_loads(self):
|
|
"""GenerateContentFunction loads and validates"""
|
|
from igny8_core.ai.functions.generate_content import GenerateContentFunction
|
|
|
|
fn = GenerateContentFunction()
|
|
self.assertEqual(fn.get_name(), 'generate_content')
|
|
|
|
def test_generate_images_function_loads(self):
|
|
"""GenerateImagesFunction loads and validates"""
|
|
from igny8_core.ai.functions.generate_images import GenerateImagesFunction
|
|
|
|
fn = GenerateImagesFunction()
|
|
self.assertEqual(fn.get_name(), 'generate_images')
|
|
|
|
|
|
class GlobalIntegrationSettingsTests(TestCase):
|
|
"""Test GlobalIntegrationSettings singleton"""
|
|
|
|
def test_singleton_loads(self):
|
|
"""GlobalIntegrationSettings singleton loads"""
|
|
from igny8_core.modules.system.global_settings_models import GlobalIntegrationSettings
|
|
|
|
settings = GlobalIntegrationSettings.get_instance()
|
|
self.assertIsNotNone(settings)
|
|
self.assertEqual(settings.pk, 1)
|
|
|
|
def test_image_settings_exist(self):
|
|
"""Image settings are configured"""
|
|
from igny8_core.modules.system.global_settings_models import GlobalIntegrationSettings
|
|
|
|
settings = GlobalIntegrationSettings.get_instance()
|
|
self.assertIsNotNone(settings.image_style)
|
|
self.assertIsNotNone(settings.max_in_article_images)
|
|
self.assertGreater(settings.max_in_article_images, 0)
|
|
|
|
|
|
class AutomationServiceTests(TestCase):
|
|
"""Test AutomationService uses same AI functions"""
|
|
|
|
def test_automation_service_imports(self):
|
|
"""AutomationService can import AI functions"""
|
|
from igny8_core.business.automation.services.automation_service import AutomationService
|
|
|
|
# Just verify the import works and class exists
|
|
self.assertIsNotNone(AutomationService)
|
|
```
|
|
|
|
### Running Tests
|
|
|
|
```bash
|
|
# Run all integration tests
|
|
docker exec igny8_backend python manage.py test igny8_core.tests.test_ai_system_integration -v 2
|
|
|
|
# Run specific test class
|
|
docker exec igny8_backend python manage.py test igny8_core.tests.test_ai_system_integration.AIModelConfigTests -v 2
|
|
|
|
# Run with coverage
|
|
docker exec igny8_backend coverage run manage.py test igny8_core.tests.test_ai_system_integration
|
|
docker exec igny8_backend coverage report
|
|
```
|
|
|
|
---
|
|
|
|
## Phased Migration with Verification
|
|
|
|
### Phase 1: Add New Models (NO BREAKING CHANGES)
|
|
|
|
**Changes:**
|
|
- Add `IntegrationProvider` model
|
|
- Add new fields to `AIModelConfig`: `credits_per_image`, `tokens_per_credit`, `quality_tier`
|
|
- Create migrations
|
|
- Populate new data
|
|
|
|
**Verification:**
|
|
```bash
|
|
# 1. Run migrations
|
|
docker exec igny8_backend python manage.py makemigrations
|
|
docker exec igny8_backend python manage.py migrate
|
|
|
|
# 2. Run ALL tests - must pass
|
|
docker exec igny8_backend python manage.py test igny8_core.tests.test_ai_system_integration -v 2
|
|
|
|
# 3. Manual verification - existing features still work
|
|
# - Go to Writer page, generate content (check credits deducted)
|
|
# - Go to Writer page, generate images (check credits deducted)
|
|
# - Run automation manually (check it completes)
|
|
```
|
|
|
|
**Rollback:** Delete new tables/fields (no existing data affected)
|
|
|
|
---
|
|
|
|
### Phase 2: Add Parallel Code Paths (OLD KEEPS WORKING)
|
|
|
|
**Changes:**
|
|
- Add new methods to `ModelRegistry` (keep old ones)
|
|
- Add new methods to `CreditService` (keep old ones)
|
|
- Add comparison logging
|
|
|
|
**Pattern:**
|
|
```python
|
|
# model_registry.py
|
|
@classmethod
|
|
def get_default_model_NEW(cls, model_type: str):
|
|
"""NEW: Get default model from AIModelConfig.is_default"""
|
|
from igny8_core.business.billing.models import AIModelConfig
|
|
return AIModelConfig.objects.filter(
|
|
model_type=model_type,
|
|
is_default=True,
|
|
is_active=True
|
|
).first()
|
|
|
|
@classmethod
|
|
def get_model(cls, model_id: str):
|
|
"""EXISTING: Keep working exactly as before"""
|
|
# ... existing code unchanged ...
|
|
```
|
|
|
|
```python
|
|
# credit_service.py
|
|
@staticmethod
|
|
def calculate_credits_for_image_NEW(model_name: str, num_images: int) -> int:
|
|
"""NEW: Calculate from AIModelConfig.credits_per_image"""
|
|
from igny8_core.business.billing.models import AIModelConfig
|
|
try:
|
|
model = AIModelConfig.objects.get(model_name=model_name, is_active=True)
|
|
if model.credits_per_image:
|
|
return model.credits_per_image * num_images
|
|
except AIModelConfig.DoesNotExist:
|
|
pass
|
|
return None # Signal to use old method
|
|
|
|
@staticmethod
|
|
def calculate_credits_from_tokens(operation_type, tokens_input, tokens_output):
|
|
"""EXISTING: Keep working, add comparison logging"""
|
|
# Calculate old way
|
|
old_result = cls._calculate_old(operation_type, tokens_input, tokens_output)
|
|
|
|
# Calculate new way (for comparison only)
|
|
new_result = cls._calculate_new(operation_type, tokens_input, tokens_output)
|
|
|
|
# Log if different
|
|
if old_result != new_result:
|
|
logger.warning(f"[MIGRATION] Credit calc mismatch: {operation_type} old={old_result} new={new_result}")
|
|
|
|
# Return OLD (safe) for now
|
|
return old_result
|
|
```
|
|
|
|
**Verification:**
|
|
```bash
|
|
# 1. Run tests
|
|
docker exec igny8_backend python manage.py test igny8_core.tests.test_ai_system_integration -v 2
|
|
|
|
# 2. Check logs for any mismatch warnings
|
|
docker logs igny8_backend 2>&1 | grep "MIGRATION"
|
|
|
|
# 3. Manual test all paths
|
|
```
|
|
|
|
**Rollback:** Remove new methods only
|
|
|
|
---
|
|
|
|
### Phase 3: Add IntegrationProvider for API Keys (OLD FALLBACK)
|
|
|
|
**Changes:**
|
|
- `AICore._load_account_settings()` tries `IntegrationProvider` first, falls back to `GlobalIntegrationSettings`
|
|
|
|
**Pattern:**
|
|
```python
|
|
# ai_core.py
|
|
def _load_account_settings(self):
|
|
"""Load API keys - try new IntegrationProvider, fallback to old"""
|
|
|
|
# Try NEW way first
|
|
try:
|
|
from igny8_core.business.billing.models import IntegrationProvider
|
|
openai_provider = IntegrationProvider.objects.filter(
|
|
provider_id='openai', is_active=True
|
|
).first()
|
|
if openai_provider and openai_provider.api_key:
|
|
self._openai_api_key = openai_provider.api_key
|
|
logger.info("[MIGRATION] Loaded OpenAI key from IntegrationProvider")
|
|
else:
|
|
raise Exception("Fallback to old method")
|
|
except Exception:
|
|
# FALLBACK to old GlobalIntegrationSettings
|
|
from igny8_core.modules.system.global_settings_models import GlobalIntegrationSettings
|
|
global_settings = GlobalIntegrationSettings.get_instance()
|
|
self._openai_api_key = global_settings.openai_api_key
|
|
logger.info("[MIGRATION] Loaded OpenAI key from GlobalIntegrationSettings (fallback)")
|
|
|
|
# Same pattern for runware...
|
|
```
|
|
|
|
**Verification:**
|
|
```bash
|
|
# 1. Run tests
|
|
docker exec igny8_backend python manage.py test igny8_core.tests.test_ai_system_integration -v 2
|
|
|
|
# 2. Verify API calls work
|
|
# - Generate content (uses OpenAI)
|
|
# - Generate images with Runware model
|
|
# - Generate images with DALL-E model
|
|
|
|
# 3. Check logs
|
|
docker logs igny8_backend 2>&1 | grep "MIGRATION"
|
|
```
|
|
|
|
**Rollback:** Revert `_load_account_settings()` to old version
|
|
|
|
---
|
|
|
|
### Phase 4: Switch to New Credit Calculation
|
|
|
|
**Changes:**
|
|
- Use `AIModelConfig.credits_per_image` for image operations
|
|
- Use `AIModelConfig.tokens_per_credit` for text operations
|
|
- Keep `CreditCostConfig` as fallback
|
|
|
|
**Verification:**
|
|
```bash
|
|
# 1. Before switching, compare calculations for 1 week
|
|
# Log both old and new results, verify they match
|
|
|
|
# 2. Run tests
|
|
docker exec igny8_backend python manage.py test igny8_core.tests.test_ai_system_integration -v 2
|
|
|
|
# 3. Verify credit logs show correct values
|
|
docker exec igny8_backend python manage.py shell -c "
|
|
from igny8_core.business.billing.models import CreditUsageLog
|
|
for log in CreditUsageLog.objects.order_by('-created_at')[:10]:
|
|
print(f'{log.operation_type}: {log.credits_used} credits, model={log.model_used}')
|
|
"
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 5: Create API Endpoint
|
|
|
|
**Changes:**
|
|
- Add `/api/v1/system/ai-models/` endpoint
|
|
- Returns models from database
|
|
|
|
**Verification:**
|
|
```bash
|
|
# 1. Test endpoint
|
|
curl -H "Authorization: Bearer $TOKEN" http://localhost:8000/api/v1/system/ai-models/
|
|
|
|
# 2. Verify response structure
|
|
# Should have text_models, image_models, image_settings
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 6: Update Frontend
|
|
|
|
**Changes:**
|
|
- Remove hardcoded model choices from `Settings.tsx`
|
|
- Load from API
|
|
|
|
**Verification:**
|
|
```bash
|
|
# 1. Frontend tests
|
|
cd frontend && npm run test
|
|
|
|
# 2. Manual verification
|
|
# - Open Sites > Settings > Image Settings
|
|
# - Verify dropdown shows models from API
|
|
# - Change model, save, verify it persists
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 7: Cleanup (After 1 Week Stable)
|
|
|
|
**Remove from GlobalIntegrationSettings:**
|
|
```python
|
|
# API key fields (moved to IntegrationProvider)
|
|
- openai_api_key
|
|
- anthropic_api_key # unused
|
|
- bria_api_key # unused
|
|
- runware_api_key
|
|
|
|
# Model selection fields (now from AIModelConfig.is_default)
|
|
- openai_model
|
|
- dalle_model
|
|
- runware_model
|
|
- default_text_provider
|
|
- default_image_service
|
|
|
|
# Hardcoded CHOICES
|
|
- OPENAI_MODEL_CHOICES
|
|
- DALLE_MODEL_CHOICES
|
|
- RUNWARE_MODEL_CHOICES
|
|
- TEXT_PROVIDER_CHOICES
|
|
- IMAGE_SERVICE_CHOICES
|
|
```
|
|
|
|
**Remove from constants.py:**
|
|
```python
|
|
- MODEL_RATES = {...}
|
|
- IMAGE_MODEL_RATES = {...}
|
|
```
|
|
|
|
**Remove from model_registry.py:**
|
|
```python
|
|
- Fallback to constants.py (all lookups must use DB)
|
|
- Hardcoded default model names
|
|
```
|
|
|
|
**Remove from ai_core.py:**
|
|
```python
|
|
- IMAGE_MODEL_RATES import and usage
|
|
- Direct GlobalIntegrationSettings key access (use IntegrationProvider)
|
|
```
|
|
|
|
**Remove CreditCostConfig entries (optional - use AIModelConfig):**
|
|
```python
|
|
- image_generation entry (use AIModelConfig.credits_per_image instead)
|
|
```
|
|
|
|
**Remove from frontend Settings.tsx:**
|
|
```python
|
|
- QUALITY_TO_CONFIG hardcoded mapping
|
|
- RUNWARE_MODEL_CHOICES hardcoded array
|
|
- DALLE_MODEL_CHOICES hardcoded array
|
|
- MODEL_LANDSCAPE_SIZES hardcoded mapping
|
|
- Any hardcoded model names
|
|
```
|
|
|
|
**Verification:**
|
|
```bash
|
|
# 1. Run full test suite
|
|
docker exec igny8_backend python manage.py test
|
|
|
|
# 2. Run integration tests
|
|
docker exec igny8_backend python manage.py test igny8_core.tests.test_ai_system_integration -v 2
|
|
|
|
# 3. Full manual test of all paths
|
|
```
|
|
|
|
---
|
|
|
|
## Complete Test Checklist
|
|
|
|
### Manual Test Checklist (Run After Each Phase)
|
|
|
|
#### Path 1: Manual Module Page Actions
|
|
|
|
| Test | Page | Button/Action | Expected Result | Verify Credits | Pass? |
|
|
|------|------|---------------|-----------------|----------------|-------|
|
|
| Cluster Keywords | Planner | "Cluster" button | Creates clusters | Check CreditUsageLog for `clustering` | ☐ |
|
|
| Generate Ideas | Planner | "Generate Ideas" button | Creates ideas | Check CreditUsageLog for `idea_generation` | ☐ |
|
|
| Generate Content | Writer | "Generate" button | Creates content | Check CreditUsageLog for `content_generation` | ☐ |
|
|
| Generate Product Content | Writer | "Generate Product" button | Creates product content | Check CreditUsageLog for `content_generation` | ☐ |
|
|
| Generate Service Content | Writer | "Generate Service" button | Creates service content | Check CreditUsageLog for `content_generation` | ☐ |
|
|
| Extract Image Prompts | Writer | "Extract Prompts" button | Creates prompts | Check CreditUsageLog for `image_prompt_extraction` | ☐ |
|
|
| Generate Images (Basic) | Writer | Image gen w/ Runware basic | Creates images | Check CreditUsageLog: 1 credit/image | ☐ |
|
|
| Generate Images (Quality) | Writer | Image gen w/ DALL-E | Creates images | Check CreditUsageLog: 5 credits/image | ☐ |
|
|
| Generate Images (Premium) | Writer | Image gen w/ Google | Creates images | Check CreditUsageLog: 15 credits/image | ☐ |
|
|
| Internal Linking | Linker | "Generate Links" button | Creates links | Check CreditUsageLog for `internal_linking` | ☐ |
|
|
| Optimize Content | Optimizer | "Optimize" button | Optimizes content | Check CreditUsageLog for `content_optimization` | ☐ |
|
|
|
|
#### Path 2: Automation Manual Run
|
|
|
|
| Test | Page | Action | Expected Result | Verify Credits | Pass? |
|
|
|------|------|--------|-----------------|----------------|-------|
|
|
| Run Full Automation | Automation | "Run Now" button | All stages complete | Credits for each stage | ☐ |
|
|
| Run Stage 1 Only | Automation | Run clustering stage | Clusters created | Check `clustering` credits | ☐ |
|
|
| Run Stage 2 Only | Automation | Run idea stage | Ideas created | Check `idea_generation` credits | ☐ |
|
|
| Run Stage 4 Only | Automation | Run content stage | Content created | Check `content_generation` credits | ☐ |
|
|
| Run Stage 5 Only | Automation | Run prompts stage | Prompts created | Check `image_prompt_extraction` credits | ☐ |
|
|
| Run Stage 6 Only | Automation | Run image stage | Images created | Check `image_generation` credits | ☐ |
|
|
| Pause/Resume | Automation | Pause then resume | Continues correctly | No double-charging | ☐ |
|
|
|
|
#### Path 3: Scheduled Automation
|
|
|
|
| Test | Setup | Expected | Verify | Pass? |
|
|
|------|-------|----------|--------|-------|
|
|
| Schedule Triggers | Set schedule for automation | Runs on schedule | Check logs at scheduled time | ☐ |
|
|
| Credits Deducted | After scheduled run | Credits reduced | Check account balance | ☐ |
|
|
| Multiple Automations | Multiple scheduled | All run | Each deducts credits | ☐ |
|
|
|
|
#### Credit Verification Queries
|
|
|
|
```bash
|
|
# Check recent credit usage logs
|
|
docker exec igny8_backend python manage.py shell -c "
|
|
from igny8_core.business.billing.models import CreditUsageLog
|
|
for log in CreditUsageLog.objects.order_by('-created_at')[:20]:
|
|
print(f'{log.created_at}: {log.operation_type} | {log.credits_used} credits | model={log.model_used} | tokens={log.tokens_input}+{log.tokens_output}')
|
|
"
|
|
|
|
# Check account balance
|
|
docker exec igny8_backend python manage.py shell -c "
|
|
from igny8_core.auth.models import Account
|
|
for acc in Account.objects.all()[:5]:
|
|
print(f'{acc.name}: {acc.credits} credits')
|
|
"
|
|
|
|
# Check credit transactions
|
|
docker exec igny8_backend python manage.py shell -c "
|
|
from igny8_core.business.billing.models import CreditTransaction
|
|
for txn in CreditTransaction.objects.order_by('-created_at')[:10]:
|
|
print(f'{txn.created_at}: {txn.transaction_type} | {txn.credits_amount} | balance={txn.balance_after}')
|
|
"
|
|
```
|
|
|
|
#### Frontend Verification
|
|
|
|
| Test | Page | Action | Expected | Pass? |
|
|
|------|------|--------|----------|-------|
|
|
| Settings Load | Sites > Settings | Open page | Image settings dropdown populated | ☐ |
|
|
| Model Selection | Sites > Settings | Change image model | Shows Basic/Quality/Premium | ☐ |
|
|
| Model Persists | Sites > Settings | Save and reload | Selected model persists | ☐ |
|
|
| Credits Display | Any page | Header/sidebar | Shows correct credit balance | ☐ |
|
|
| Usage Analytics | Analytics | View usage | Shows correct breakdown | ☐ |
|
|
|
|
---
|
|
|
|
## Rollback Procedures
|
|
|
|
### If Phase 1 Fails
|
|
```bash
|
|
# Remove new migrations
|
|
docker exec igny8_backend python manage.py migrate billing <previous_migration>
|
|
docker exec igny8_backend python manage.py migrate system <previous_migration>
|
|
```
|
|
|
|
### If Phase 2-6 Fails
|
|
```bash
|
|
# Revert code changes via git
|
|
git checkout -- backend/igny8_core/ai/
|
|
git checkout -- backend/igny8_core/business/billing/
|
|
|
|
# Restart containers
|
|
docker-compose restart backend
|
|
```
|
|
|
|
### If Phase 7 Fails (Cleanup)
|
|
```bash
|
|
# This is why we wait 1 week before cleanup
|
|
# Restore from backup or revert git
|
|
```
|
|
|
|
---
|
|
|
|
## Success Criteria
|
|
|
|
Migration is COMPLETE when:
|
|
|
|
1. ✅ All integration tests pass
|
|
2. ✅ All manual tests pass
|
|
3. ✅ No errors in logs for 1 week
|
|
4. ✅ Credit calculations match expected values
|
|
5. ✅ All execution paths work (manual, automation, scheduled)
|
|
6. ✅ Frontend loads models from API
|
|
7. ✅ No legacy code remains:
|
|
- No `MODEL_RATES` / `IMAGE_MODEL_RATES` in constants.py
|
|
- No API keys in `GlobalIntegrationSettings`
|
|
- No hardcoded models in frontend
|
|
- No fallback to constants in `ModelRegistry`
|