350 lines
11 KiB
Markdown
350 lines
11 KiB
Markdown
# Phase 2 Implementation vs Previous Commits Analysis
|
|
|
|
**Date:** December 23, 2025
|
|
**Commits Analyzed:**
|
|
- e041cb8e: "ai & tokens" (Dec 19, 2025)
|
|
- c17b22e9: "credits adn tokens final correct setup" (Dec 20, 2025)
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
**Current Implementation Status:** ✅ Phase 1 Complete, 🔄 Phase 2 In Progress
|
|
|
|
### What Was in Previous Commits (Now Reverted)
|
|
|
|
The commits 8-9 (e041cb8e + c17b22e9) implemented a comprehensive token-based system that was later reverted. Here's what they built:
|
|
|
|
---
|
|
|
|
## Feature Comparison
|
|
|
|
### 1. Analytics & Reports (✅ IMPLEMENTED IN COMMITS)
|
|
|
|
#### ✅ Token Usage Report (`token_usage_report`)
|
|
**Location:** `backend/igny8_core/admin/reports.py`
|
|
**Status:** Was fully implemented, currently NOT in our codebase
|
|
|
|
**Features:**
|
|
- Total tokens (input + output) with breakdown
|
|
- Token usage by model (top 10)
|
|
- Token usage by operation/function (top 10)
|
|
- Token usage by account (top 15 consumers)
|
|
- Daily token trends (time series chart)
|
|
- Hourly usage patterns (peak times)
|
|
- Cost per 1K tokens calculation
|
|
- Success rate and token efficiency metrics
|
|
|
|
**URL:** `/admin/reports/token-usage/`
|
|
|
|
**Current Status:** ❌ NOT IMPLEMENTED
|
|
**Action Needed:** ✅ COPY THIS IMPLEMENTATION
|
|
|
|
---
|
|
|
|
#### ✅ AI Cost Analysis Report (`ai_cost_analysis`)
|
|
**Location:** `backend/igny8_core/admin/reports.py`
|
|
**Status:** Was fully implemented, currently NOT in our codebase
|
|
|
|
**Features:**
|
|
- Total AI API costs with breakdown
|
|
- Cost by model (with cost per 1K tokens)
|
|
- Cost by account (top spenders)
|
|
- Cost by operation/function
|
|
- Daily cost trends (time series)
|
|
- Projected monthly cost (30-day forecast)
|
|
- Cost anomalies (calls >3x average cost)
|
|
- Model comparison matrix
|
|
- Hourly cost distribution
|
|
- Cost efficiency score
|
|
|
|
**URL:** `/admin/reports/ai-cost-analysis/`
|
|
|
|
**Current Status:** ❌ NOT IMPLEMENTED
|
|
**Action NEEDED:** ✅ COPY THIS IMPLEMENTATION
|
|
|
|
---
|
|
|
|
### 2. Admin Templates (✅ IMPLEMENTED IN COMMITS)
|
|
|
|
#### ✅ Token Usage Template
|
|
**Location:** `backend/igny8_core/templates/admin/reports/token_usage.html`
|
|
**Features:**
|
|
- Chart.js visualizations (line charts, bar charts, pie charts)
|
|
- Token breakdown by model, function, account
|
|
- Daily trends with date range filter
|
|
- Hourly heatmap for peak usage times
|
|
- Export data functionality
|
|
|
|
**Current Status:** ❌ NOT IMPLEMENTED
|
|
|
|
---
|
|
|
|
#### ✅ AI Cost Analysis Template
|
|
**Location:** `backend/igny8_core/templates/admin/reports/ai_cost_analysis.html`
|
|
**Features:**
|
|
- Cost visualizations (line charts, bar charts)
|
|
- Model cost comparison table
|
|
- Anomaly detection table (expensive calls)
|
|
- Projected monthly costs
|
|
- Cost efficiency metrics
|
|
- Export to CSV functionality
|
|
|
|
**Current Status:** ❌ NOT IMPLEMENTED
|
|
|
|
---
|
|
|
|
### 3. Database Models & Migrations (⚠️ PARTIALLY DIFFERENT)
|
|
|
|
#### ❌ BillingConfiguration Model (REMOVED IN OUR APPROACH)
|
|
**Previous Implementation:**
|
|
```python
|
|
class BillingConfiguration(models.Model):
|
|
default_tokens_per_credit = models.IntegerField(default=100)
|
|
default_credit_price_usd = models.DecimalField(default=0.01)
|
|
rounding_mode = models.CharField(choices=[('up', 'Up'), ('down', 'Down'), ('nearest', 'Nearest')])
|
|
token_reporting_enabled = models.BooleanField(default=True)
|
|
```
|
|
|
|
**Our Implementation:** ✅ REPLACED WITH `AIModelConfig`
|
|
- Instead of global `tokens_per_credit`, we use per-model ratios
|
|
- More flexible (GPT-4 = 50 tokens/credit, GPT-3.5 = 200 tokens/credit)
|
|
|
|
---
|
|
|
|
#### ⚠️ CreditCostConfig Updates (SIMILAR BUT DIFFERENT)
|
|
**Previous Implementation:**
|
|
```python
|
|
class CreditCostConfig:
|
|
tokens_per_credit = IntegerField # Per-operation ratio
|
|
min_credits = IntegerField
|
|
price_per_credit_usd = DecimalField
|
|
```
|
|
|
|
**Our Implementation:** ✅ BETTER APPROACH
|
|
```python
|
|
class CreditCostConfig:
|
|
unit = CharField(choices=[..., 'per_100_tokens', 'per_1000_tokens']) # Token units added
|
|
default_model = FK(AIModelConfig) # Links to centralized model config
|
|
```
|
|
|
|
**Difference:**
|
|
- Previous: Each operation had its own `tokens_per_credit`
|
|
- Current: Operations reference a shared `AIModelConfig` with unified pricing
|
|
|
|
---
|
|
|
|
#### ✅ CreditUsageLog Fields (MOSTLY SAME)
|
|
**Previous Implementation:**
|
|
```python
|
|
class CreditUsageLog:
|
|
tokens_input = IntegerField
|
|
tokens_output = IntegerField
|
|
cost_usd = DecimalField
|
|
model_used = CharField(max_length=100)
|
|
```
|
|
|
|
**Our Implementation:** ✅ ENHANCED
|
|
```python
|
|
class CreditUsageLog:
|
|
tokens_input = IntegerField
|
|
tokens_output = IntegerField
|
|
cost_usd_input = DecimalField # NEW: Separate input cost
|
|
cost_usd_output = DecimalField # NEW: Separate output cost
|
|
cost_usd_total = DecimalField # NEW: Total cost
|
|
model_config = FK(AIModelConfig) # NEW: FK instead of string
|
|
model_name = CharField # Kept for backward compatibility
|
|
```
|
|
|
|
**Status:** ✅ OUR APPROACH IS BETTER (granular cost tracking)
|
|
|
|
---
|
|
|
|
### 4. Credit Calculation Logic (✅ IMPLEMENTED BY US)
|
|
|
|
#### ✅ Token-Based Credit Calculation
|
|
**Previous Implementation:**
|
|
```python
|
|
def calculate_credits(tokens_input, tokens_output, operation_type):
|
|
config = CreditCostConfig.objects.get(operation_type=operation_type)
|
|
total_tokens = tokens_input + tokens_output
|
|
credits = total_tokens / config.tokens_per_credit
|
|
return max(credits, config.min_credits)
|
|
```
|
|
|
|
**Our Implementation:** ✅ SIMILAR + MODEL-AWARE
|
|
```python
|
|
def calculate_credits_from_tokens(operation_type, tokens_input, tokens_output, model_config):
|
|
config = CreditCostConfig.objects.get(operation_type=operation_type)
|
|
total_tokens = tokens_input + tokens_output
|
|
tokens_per_credit = model_config.tokens_per_credit # Model-specific ratio
|
|
credits = total_tokens / tokens_per_credit
|
|
return max(credits, config.credits_cost)
|
|
```
|
|
|
|
**Status:** ✅ IMPLEMENTED (our approach is more flexible)
|
|
|
|
---
|
|
|
|
#### ✅ Model Selection Logic
|
|
**Previous Implementation:** ❌ NOT PRESENT (used global default)
|
|
|
|
**Our Implementation:** ✅ IMPLEMENTED
|
|
```python
|
|
def get_model_for_operation(account, operation_type, task_override=None):
|
|
# Priority: Task > Account Default > Operation Default > System Default
|
|
if task_override:
|
|
return task_override
|
|
if account.integration.default_text_model:
|
|
return account.integration.default_text_model
|
|
if operation_config.default_model:
|
|
return operation_config.default_model
|
|
return AIModelConfig.objects.get(is_default=True)
|
|
```
|
|
|
|
**Status:** ✅ NEW FEATURE (not in previous commits)
|
|
|
|
---
|
|
|
|
### 5. AIEngine Updates (⚠️ PARTIALLY IMPLEMENTED)
|
|
|
|
#### ✅ Token Extraction (BOTH HAVE IT)
|
|
**Previous Implementation:**
|
|
```python
|
|
tokens_input = raw_response.get('input_tokens', 0)
|
|
tokens_output = raw_response.get('output_tokens', 0)
|
|
```
|
|
|
|
**Our Implementation:** ❌ NOT YET IN AIEngine
|
|
**Action Needed:** Need to update `backend/igny8_core/ai/engine.py` to extract tokens
|
|
|
|
---
|
|
|
|
### 6. Management Commands (✅ IMPLEMENTED IN COMMITS)
|
|
|
|
#### ✅ Backfill Tokens Command
|
|
**Location:** `backend/igny8_core/management/commands/backfill_tokens.py`
|
|
**Purpose:** Migrate old CreditUsageLog records to have token data
|
|
|
|
**Current Status:** ❌ NOT IMPLEMENTED (but may not be needed if we don't have legacy data)
|
|
|
|
---
|
|
|
|
### 7. Integration with Services (⚠️ PARTIALLY IMPLEMENTED)
|
|
|
|
#### Previous Implementation:
|
|
- Updated `linker_service.py` to pass tokens
|
|
- Updated `optimizer_service.py` to pass tokens
|
|
- Other services not modified
|
|
|
|
#### Our Implementation:
|
|
- ❌ NOT YET UPDATED (Phase 2.3 - pending)
|
|
- Need to update: clustering, ideas, content, image, optimizer, linker services
|
|
|
|
---
|
|
|
|
## What's Missing from Our Current Implementation
|
|
|
|
### ❌ Critical Missing Features (Should Copy from Commits)
|
|
|
|
1. **Token Usage Report** (`token_usage_report` view + template)
|
|
- Full analytics with charts
|
|
- By model, function, account, time
|
|
- Export functionality
|
|
|
|
2. **AI Cost Analysis Report** (`ai_cost_analysis` view + template)
|
|
- Cost tracking and forecasting
|
|
- Anomaly detection
|
|
- Model cost comparison
|
|
|
|
3. **Admin URL Routes** (register the reports)
|
|
- Need to add to `backend/igny8_core/admin/site.py`
|
|
|
|
4. **AIEngine Token Extraction**
|
|
- Extract `input_tokens`, `output_tokens` from AI responses
|
|
- Pass to CreditService
|
|
|
|
5. **Service Updates** (Phase 2.3)
|
|
- Update all AI services to use token-based calculation
|
|
- Pass `model_config`, `tokens_input`, `tokens_output`
|
|
|
|
---
|
|
|
|
## What We Did Better
|
|
|
|
### ✅ Improvements Over Previous Commits
|
|
|
|
1. **AIModelConfig Model**
|
|
- Centralized pricing (one source of truth)
|
|
- Support multiple providers (OpenAI, Anthropic, Runware)
|
|
- Per-model token ratios (GPT-4 ≠ GPT-3.5)
|
|
- Easier to add new models
|
|
|
|
2. **Granular Cost Tracking**
|
|
- Separate `cost_usd_input` and `cost_usd_output`
|
|
- Can track input vs output costs accurately
|
|
- Better for margin analysis
|
|
|
|
3. **Model Selection Priority**
|
|
- Task override > Account default > Operation default > System default
|
|
- More flexible than global default
|
|
|
|
4. **IntegrationSettings Enhancement**
|
|
- Account-level model selection
|
|
- `default_text_model` and `default_image_model`
|
|
- Per-account cost optimization
|
|
|
|
5. **Cleaner Migration Path**
|
|
- Previous: Changed field types (risky)
|
|
- Current: Added new fields, kept old for compatibility
|
|
|
|
---
|
|
|
|
## Action Items
|
|
|
|
### ✅ Phase 1: COMPLETE
|
|
- [x] AIModelConfig model
|
|
- [x] Migrations applied
|
|
- [x] Seed data (7 models)
|
|
- [x] Admin interface
|
|
|
|
### 🔄 Phase 2: IN PROGRESS
|
|
- [x] CreditService.calculate_credits_from_tokens()
|
|
- [x] CreditService.get_model_for_operation()
|
|
- [x] Updated deduct_credits() with model_config FK
|
|
- [ ] Update AIEngine to extract tokens
|
|
- [ ] Update AI services (clustering, ideas, content, image, optimizer, linker)
|
|
|
|
### ❌ Phase 3: ANALYTICS (COPY FROM COMMITS)
|
|
- [ ] Copy `token_usage_report()` from commit c17b22e9
|
|
- [ ] Copy `ai_cost_analysis()` from commit e041cb8e
|
|
- [ ] Copy `token_usage.html` template
|
|
- [ ] Copy `ai_cost_analysis.html` template
|
|
- [ ] Register URLs in admin site
|
|
|
|
### ❌ Phase 4: TESTING & DOCUMENTATION
|
|
- [ ] Test token-based calculation end-to-end
|
|
- [ ] Verify reports work with new data
|
|
- [ ] Update user documentation
|
|
|
|
---
|
|
|
|
## Conclusion
|
|
|
|
**Previous Commits:** Comprehensive token system with excellent analytics, but over-engineered with per-operation configs
|
|
|
|
**Current Implementation:** Cleaner architecture with centralized AIModelConfig, better model selection, but missing the analytics dashboards
|
|
|
|
**Best Path Forward:**
|
|
1. ✅ Keep our Phase 1 foundation (AIModelConfig approach is superior)
|
|
2. ✅ Complete Phase 2 (CreditService mostly done, need AIEngine + services)
|
|
3. 📋 Copy Phase 3 analytics from commits (token_usage_report + ai_cost_analysis)
|
|
4. 🧪 Test everything end-to-end
|
|
|
|
**Timeline:**
|
|
- Phase 2 completion: 1-2 hours (AIEngine + service updates)
|
|
- Phase 3 analytics: 30-60 minutes (copy + adapt templates)
|
|
- Phase 4 testing: 30 minutes
|
|
|
|
**Estimated Total:** 2-3 hours to full implementation
|