salman/igny8
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
38bc015d96818fd9fbc032af6dca4cf206485bcb
igny8/REMOTE-COMMITS-INTEGRATION-PLAN.md

808 lines
30 KiB
Markdown
Raw Blame History

# Remote Commits Integration Plan
**Created:** December 23, 2025
**Current Branch:** main (commit d768ed71 - New Model & tokens/credits updates)
**Remote Branch:** origin/main (9 commits ahead)
**Purpose:** Integrate all remote features while maintaining new AIModelConfig token-based system
---
## Executive Summary
### Current Situation
- **Local:** Implemented Phase 1 & 2.1 of AIModelConfig refactor (token-based billing with centralized model pricing)
- **Remote:** 9 commits with features we need to integrate:
1. Token analytics reports (e041cb8e, c17b22e9) - ALREADY ANALYZED
2. Global settings system (3283a83b, 9e8ff4fb, 7a1e952a, 5c9ef81a, 646095da)
3. Admin bulk actions (ab0d6469)
4. Frontend cleanup (eb6cba79)
### Integration Strategy
1. **Direct Apply:** Non-conflicting changes (frontend cleanup, bulk actions)
2. **Adapt & Merge:** Global settings to work with AIModelConfig
3. **Skip:** Old token system (BillingConfiguration) - replaced by AIModelConfig
4. **Enhance:** Analytics reports adapted for new schema
---
## Commit-by-Commit Analysis
### Commit 1: e041cb8e - "ai & tokens" (Dec 19, 2025)
**Status:** ✅ Already Analyzed in PHASE2-COMMIT-COMPARISON.md
**Files Changed:**
- backend/igny8_core/admin/reports.py (+322 lines)
- backend/igny8_core/admin/site.py (+2 lines)
- backend/igny8_core/templates/admin/reports/ai_cost_analysis.html (+218 lines)
- backend/igny8_core/templates/admin/reports/token_usage.html (+414 lines)
**Features:**
- Token Usage Report view with Chart.js visualizations
- AI Cost Analysis Report with forecasting and anomaly detection
- Admin URL routes registered
**Integration Decision:** ADAPT FOR AIMODELCONFIG
- Reports use `model_used` CharField - need to adapt for `model_config` FK
- Cost tracking in `cost_usd` - need to adapt for `cost_usd_input/output/total`
- Token tracking already compatible
---
### Commit 2: c17b22e9 - "credits adn tokens final correct setup" (Dec 20, 2025)
**Status:** ✅ Already Analyzed in PHASE2-COMMIT-COMPARISON.md
**Files Changed:**
- CREDITS-TOKENS-GUIDE.md (new)
- backend/igny8_core/admin/reports.py (updated token_usage_report)
- backend/igny8_core/ai/engine.py (token extraction)
- backend/igny8_core/business/billing/models.py (BillingConfiguration)
- backend/igny8_core/business/billing/services/credit_service.py (token calculation)
- backend/igny8_core/business/linking/services/linker_service.py (token usage)
- backend/igny8_core/business/optimization/services/optimizer_service.py (token usage)
- backend/igny8_core/management/commands/backfill_tokens.py (new)
- backend/igny8_core/modules/billing/admin.py (token config admin)
- Migrations: 0018_remove_creditcostconfig_credits_cost_and_more.py, 0019_populate_token_based_config.py
- Templates: Updated analytics templates
**Features:**
- BillingConfiguration model with `default_tokens_per_credit = 100`
- Per-operation token ratios in CreditCostConfig
- AIEngine token extraction from AI responses
- Service updates to pass tokens
- Backfill command to populate historical data
**Integration Decision:** PARTIALLY SKIP, ADAPT KEY PARTS
- ❌ Skip BillingConfiguration model (replaced by AIModelConfig)
- ❌ Skip per-operation tokens_per_credit (using AIModelConfig.tokens_per_credit instead)
- ✅ Copy AIEngine token extraction logic
- ✅ Copy service update patterns (linker, optimizer)
- ✅ Adapt backfill command for new schema
---
### Commit 3: ab0d6469 - "bulk actions & some next audits docs" (Dec 20, 2025)
**Files Changed:**
- AWS_ADMIN_ACCOUNT_AUDIT_REPORT.md (+601 lines)
- DATA_SEGREGATION_SYSTEM_VS_USER.md (+356 lines)
- SESSION_SUMMARY_DJANGO_ADMIN_ENHANCEMENT.md (+226 lines)
- backend/igny8_core/ai/admin.py (+38 lines)
- backend/igny8_core/auth/admin.py (+468 lines)
- backend/igny8_core/business/automation/admin.py (+137 lines)
- backend/igny8_core/business/billing/admin.py (+69 lines)
- backend/igny8_core/business/integration/admin.py (+60 lines)
- backend/igny8_core/business/optimization/admin.py (+36 lines)
- backend/igny8_core/business/publishing/admin.py (+63 lines)
- backend/igny8_core/modules/billing/admin.py (+171 lines)
- backend/igny8_core/modules/planner/admin.py (+189 lines)
- backend/igny8_core/modules/system/admin.py (+160 lines)
- backend/igny8_core/modules/writer/admin.py (+469 lines)
- content-generation-prompt.md (deleted)
- idea-generation-prompt.md (deleted)
**Features Added:**
- Bulk actions across all admin models:
* Activate/Deactivate items
* Export to CSV/JSON
* Batch update status
* Clone/duplicate items
* Delete with confirmation
- Enhanced admin list displays with filters
- Improved search functionality
- Audit documentation for AWS admin account
**Integration Decision:** ✅ DIRECT APPLY (Non-conflicting)
- Admin bulk actions don't conflict with AIModelConfig
- Can be applied directly after resolving any merge conflicts
- Documentation files can be added as-is
---
### Commit 4: eb6cba79 - "cleanup - froentend pages removed" (Dec 20, 2025)
**Files Changed (43 deletions):**
Frontend pages deleted:
- frontend/src/pages/Admin/* (AdminBilling, AdminCreditCosts, AdminAPIMonitor, AdminAccountLimits, AdminActivityLogs, AdminAnalytics, AdminBillingHistory, AdminCostBreakdown, AdminCreditUsage, AdminDashboard, AdminGlobalSettings, AdminModelCosts)
- frontend/src/pages/Settings/UiElements/* (20+ UI component showcase pages)
- frontend/src/pages/Settings/ApiMonitor.tsx
- frontend/src/pages/Settings/DebugStatus.tsx
- frontend/src/pages/Settings/MasterStatus.tsx
- frontend/src/components/sidebar/ApiStatusIndicator.tsx
- frontend/src/components/auth/AdminGuard.tsx
Documentation files added:
- COMPREHENSIVE_REFACTORING_PLAN.md (+1615 lines)
- DJANGO_ADMIN_ACTIONS_COMPLETED.md (+453 lines)
- DJANGO_ADMIN_ACTIONS_QUICK_REFERENCE.md (+511 lines)
- DJANGO_ADMIN_ACTIONS_TODO.md (+317 lines)
- FRONTEND_ADMIN_PAGES_COMPREHENSIVE_AUDIT.md (+311 lines)
- FRONTEND_ADMIN_REFACTORING_COMPLETE.md (+467 lines)
- SYSTEM_ARCHITECTURE_ANALYSIS_SUPERUSER_STRATEGY.md (+696 lines)
**Rationale:**
- Move admin functionality to Django Admin interface
- Remove duplicate frontend admin pages
- Eliminate unmaintained UI showcase pages
- Simplify frontend architecture
**Integration Decision:** ✅ DIRECT APPLY (Non-conflicting)
- File deletions don't conflict with AIModelConfig
- Documentation provides context for architectural decisions
- Can be cherry-picked directly
---
### Commit 5: 3283a83b - "feat(migrations): Rename indexes and update global integration settings..." (Dec 20, 2025)
**Files Changed (51 files):**
**New Models:**
- backend/igny8_core/modules/system/global_settings_models.py (+270 lines)
* GlobalIntegrationSettings (singleton, pk=1)
* Fields: openai_api_key, openai_model, openai_temperature, openai_max_tokens
* Fields: dalle_api_key, dalle_model, dalle_size, dalle_quality, dalle_style
* Fields: runware_api_key, runware_model
* Fields: default_image_service, image_quality, image_style
* Purpose: Platform-wide API keys and default settings
**Updated Models:**
- backend/igny8_core/modules/system/models.py (IntegrationSettings refactored)
* Removed API key storage (moved to GlobalIntegrationSettings)
* Changed to store only model/parameter overrides in JSON `config` field
* Free plan: Cannot override, uses global defaults
* Paid plans: Can override model, temperature, tokens, image settings
**Migrations:**
- backend/igny8_core/modules/system/migrations/0002_add_global_settings_models.py (+186 lines)
- backend/igny8_core/modules/system/migrations/0004_fix_global_settings_remove_override.py (+108 lines)
**Admin Enhancements:**
- backend/igny8_core/admin/monitoring.py (+406 lines)
* API monitoring dashboard
* Debug console
* System health checks
- backend/igny8_core/templates/admin/monitoring/*.html (3 new templates)
**AI System Changes:**
- backend/igny8_core/management/commands/populate_global_prompts.py (+238 lines)
- backend/igny8_core/ai/prompts.py (refactored, -640 lines)
- backend/igny8_core/ai/ai_core.py (+25 lines)
- backend/igny8_core/ai/settings.py (+113 lines)
**Removed Files (IMPORTANT):**
- backend/igny8_core/api/base.py, permissions.py, throttles.py (deleted -118 lines)
- backend/igny8_core/auth/middleware.py, utils.py (deleted -32 lines)
- Reason: Consolidated into core modules
**Frontend Changes:**
- frontend/src/App.tsx (-11 lines): Removed AwsAdminGuard imports
- frontend/src/components/auth/AwsAdminGuard.tsx (deleted -31 lines)
- Various frontend components cleaned up
**Documentation:**
- 02_COMPREHENSIVE_REFACTORING_PLAN.md (renamed from COMPREHENSIVE_REFACTORING_PLAN.md)
- 03_COMPLETE-IMPLEMENTATION-GUIDE.md (+1100 lines)
- 04_GLOBAL-SETTINGS-ACCESS-GUIDE.md (+322 lines)
- 05_GLOBAL-SETTINGS-CORRECT-IMPLEMENTATION.md (+320 lines)
- docs/AI_CLEANUP_SUMMARY.md, docs/AI_SYSTEM_AUDIT.md (moved to docs/)
**Integration Decision:** ⚠️ ADAPT FOR AIMODELCONFIG
**CRITICAL:** This commit introduces GlobalIntegrationSettings which CONFLICTS with our AIModelConfig approach
**Problems:**
1. GlobalIntegrationSettings stores model names as CharField with hardcoded choices
2. Our AIModelConfig uses database-driven model config with pricing
3. Duplicate model selection logic
**Solution:**
- Keep GlobalIntegrationSettings for API keys ONLY
- Remove model selection fields from GlobalIntegrationSettings
- Use AIModelConfig for all model selection and pricing
- Adapt IntegrationSettings.config to reference AIModelConfig FKs
**What to Keep:**
- ✅ GlobalIntegrationSettings for API keys (openai_api_key, dalle_api_key, runware_api_key)
- ✅ Admin monitoring templates (system health, debug console)
- ✅ populate_global_prompts command
- ✅ Frontend cleanup changes
- ❌ Model selection fields from GlobalIntegrationSettings (use AIModelConfig instead)
---
### Commit 6: 9e8ff4fb - "globals" (Dec 20, 2025)
**Files Changed (18 files):**
**Key Changes:**
- INTEGRATION-SETTINGS-WORKFLOW.md (+223 lines) - Workflow documentation
- backend/igny8_core/modules/system/global_settings_models.py (enhanced +129 lines)
* Added more model choices (GPT-5.1, GPT-5.2)
* Added image service selection
* Enhanced configuration options
- backend/igny8_core/modules/system/integration_views.py (refactored -349 lines)
* Simplified integration settings API
* Removed complex override logic
* Uses GlobalIntegrationSettings as source of truth
- backend/igny8_core/modules/system/admin.py (+20 lines)
* Enhanced GlobalIntegrationSettings admin
- backend/igny8_core/ai/prompts.py (massive refactor -640 lines)
* Simplified prompt management
* Removed hardcoded prompts
**Migrations:**
- 0004_fix_global_settings_remove_override.py (revised)
- 0005_add_model_choices.py (+33 lines)
- 0006_fix_image_settings.py (+44 lines)
- 0007_add_image_defaults.py (+28 lines)
- 0008_add_default_image_service.py (+18 lines)
- 0009_fix_variables_optional.py (+18 lines)
**Image Generation:**
- backend/igny8_core/business/automation/migrations/0005_add_default_image_service.py (+18 lines)
**Frontend:**
- frontend/src/components/common/ValidationCard.tsx (+23 lines)
- frontend/src/layout/AppSidebar.tsx (+29 lines)
- frontend/src/pages/Settings/Integration.tsx (+13 lines)
**Integration Decision:** ⚠️ ADAPT FOR AIMODELCONFIG
- Same issue as commit 3283a83b - model choices hardcoded
- Workflow documentation useful but needs updating for AIModelConfig
- Migrations will conflict - need to merge with our migrations
**Adaptation Strategy:**
- Extract API key management from GlobalIntegrationSettings
- Replace hardcoded model choices with AIModelConfig references
- Update workflow documentation for AIModelConfig system
---
### Commit 7: 7a1e952a - "feat: Add Global Module Settings and Caption to Images" (Dec 20, 2025)
**Files Changed (16 files):**
**New Model:**
- backend/igny8_core/modules/system/global_settings_models.py (+71 lines)
* GlobalModuleSettings model
* Fields: is_clustering_enabled, is_ideas_enabled, is_content_enabled, is_optimization_enabled, is_linking_enabled, is_images_enabled, is_publishing_enabled
* Purpose: Platform-wide enable/disable for modules
* Singleton pattern (pk=1)
**Image Model Enhancement:**
- backend/igny8_core/business/content/models.py (+1 line)
* Added `caption` TextField to Images model
- backend/igny8_core/modules/writer/migrations/0013_add_caption_to_images.py (+18 lines)
**AI Functions:**
- backend/igny8_core/ai/functions/generate_image_prompts.py (+38 lines)
* Updated to handle caption in image prompt generation
**System Module:**
- backend/igny8_core/modules/system/migrations/0010_globalmodulesettings_and_more.py (+36 lines)
- backend/igny8_core/modules/system/admin.py (+53 lines) - GlobalModuleSettings admin
- backend/igny8_core/modules/system/serializers.py (+13 lines)
- backend/igny8_core/modules/system/settings_views.py (+64 lines)
- backend/igny8_core/modules/system/utils.py (refactor -337 lines)
- backend/igny8_core/modules/system/views.py (+53 lines)
**Frontend:**
- frontend/src/pages/Thinker/Prompts.tsx (+36 lines) - Caption field in UI
- frontend/src/services/api.ts (+1 line)
- frontend/src/store/settingsStore.ts (+15 lines)
- frontend/src/templates/ContentViewTemplate.tsx (+14 lines) - Display captions
**Integration Decision:** ✅ DIRECT APPLY (Non-conflicting)
- GlobalModuleSettings is independent feature
- Image caption field is enhancement
- No conflicts with AIModelConfig
- Can be applied after migration number adjustments
---
### Commit 8: 5c9ef81a - "moduels setigns rmeove from frotneend" (Dec 20, 2025)
**Files Changed (10 files):**
**Backend Cleanup:**
- backend/igny8_core/modules/system/settings_admin.py (+21 lines)
- backend/igny8_core/modules/system/settings_serializers.py (+13 lines)
- backend/igny8_core/modules/system/settings_views.py (-152 lines cleanup)
**Frontend Cleanup:**
- frontend/src/App.tsx (-114 lines)
* Removed module settings routes
- frontend/src/components/common/ModuleGuard.tsx (-35 lines cleanup)
- frontend/src/config/modules.config.ts (-28 lines cleanup)
- frontend/src/layout/AppSidebar.tsx (-120 lines cleanup)
- frontend/src/pages/Settings/Modules.tsx (deleted -91 lines)
* Removed frontend module settings page
- frontend/src/services/api.ts (-39 lines)
- frontend/src/store/settingsStore.ts (-76 lines cleanup)
**Rationale:**
- Module settings moved to Django Admin only
- Removed duplicate frontend UI
- Simplified architecture
**Integration Decision:** ✅ DIRECT APPLY (Non-conflicting)
- File deletions don't conflict
- Cleanup is beneficial
- Can be cherry-picked
---
### Commit 9: 646095da - "moduel setgins fixed" (Dec 20, 2025)
**Files Changed (7 files):**
**Backend:**
- backend/igny8_core/modules/system/settings_views.py (+9 lines) - Bug fixes
- backend/igny8_core/modules/system/urls.py (+4 lines) - Route fixes
**Frontend:**
- frontend/src/App.tsx (+9 lines) - Module settings route restoration
- frontend/src/layout/AppSidebar.tsx (+105 lines) - Restored module toggle UI
- frontend/src/services/api.ts (+19 lines) - Module API endpoints
- frontend/src/store/moduleStore.ts (+59 lines new file) - Module state management
- frontend/test-module-settings.html (+69 lines new file) - Test page
**Rationale:**
- Reverted partial removal from commit 5c9ef81a
- Module settings needed in frontend for user convenience
- Added dedicated moduleStore for state management
**Integration Decision:** ✅ DIRECT APPLY (Non-conflicting)
- Latest version of module settings
- Can be applied after commit 8
---
## Dependency Graph
```
Timeline (oldest to newest):
e041cb8e → c17b22e9 → ab0d6469 → eb6cba79 → 3283a83b → 9e8ff4fb → 7a1e952a → 5c9ef81a → 646095da
│ │ │ │ │ │ │ │ │
└──────────┴───────────┴───────────┴───────────┴───────────┴───────────┴───────────┴───────────┘
Token System Admin Cleanup Global Settings System Module Settings
Dependencies:
1. e041cb8e + c17b22e9: Token analytics (Phase 3 of our plan)
2. 3283a83b + 9e8ff4fb: Global settings foundation → needs AIModelConfig adaptation
3. 7a1e952a: GlobalModuleSettings (depends on global settings models)
4. 5c9ef81a + 646095da: Module settings UI (latest = 646095da)
5. ab0d6469: Admin bulk actions (independent)
6. eb6cba79: Frontend cleanup (independent)
```
---
## Integration Phases
### Phase A: Non-Conflicting Features (Safe to Apply)
**Estimated Time:** 2 hours
1. **Frontend Cleanup (eb6cba79)**
- Delete 43 unused admin/UI pages
- Add architecture documentation
- No code conflicts
2. **Admin Bulk Actions (ab0d6469)**
- Apply bulk action enhancements to all admin models
- Add audit documentation
- Compatible with our admin changes
3. **Module Settings Final (5c9ef81a + 646095da)**
- Apply latest module settings UI
- Add moduleStore for state management
- Independent feature
4. **Image Captions (from 7a1e952a)**
- Add caption field to Images model
- Update image generation functions
- Update frontend components
- No conflicts
---
### Phase B: Global Settings Adaptation (Complex)
**Estimated Time:** 6-8 hours
**Current State:**
- Remote: GlobalIntegrationSettings with hardcoded model choices
- Local: AIModelConfig with database-driven model configs
**Target Architecture:**
```
GlobalIntegrationSettings (API Keys ONLY)
├── openai_api_key
├── anthropic_api_key
├── runware_api_key
└── (remove all model selection fields)
AIModelConfig (Our System)
├── model_name
├── provider
├── cost_per_1k_input_tokens
├── cost_per_1k_output_tokens
├── tokens_per_credit
└── is_default
IntegrationSettings (Account Overrides)
├── default_text_model → FK to AIModelConfig
├── default_image_model → FK to AIModelConfig
└── config: {temperature, max_tokens, image_size, etc.}
```
**Steps:**
1. **Create Hybrid GlobalIntegrationSettings Model**
```python
class GlobalIntegrationSettings(models.Model):
# API Keys (from remote commits)
openai_api_key = CharField(...)
anthropic_api_key = CharField(...)
runware_api_key = CharField(...)
# Default Models (link to AIModelConfig)
default_text_model = ForeignKey('AIModelConfig', related_name='global_text_default')
default_image_model = ForeignKey('AIModelConfig', related_name='global_image_default')
# Global Parameters (can be overridden per account)
default_temperature = FloatField(default=0.7)
default_max_tokens = IntegerField(default=8192)
default_image_size = CharField(default='1024x1024')
default_image_quality = CharField(default='standard')
default_image_style = CharField(default='realistic')
```
2. **Adapt IntegrationSettings (Already Done)**
- We already have default_text_model and default_image_model FKs
- Keep config JSON for parameter overrides
- No changes needed
3. **Create Migration Strategy**
```
Migration 0020_add_global_integration_settings:
- Create GlobalIntegrationSettings table
- Populate with default AIModelConfig references
- Copy API keys from first IntegrationSettings (if exists)
Migration 0021_migrate_integration_settings:
- Update IntegrationSettings to use new structure
- Backfill default_text_model, default_image_model
```
4. **Update API Key Access Pattern**
```python
# OLD (from remote commits):
integration_settings = account.integration_settings.first()
api_key = integration_settings.openai_api_key
# NEW (hybrid approach):
global_settings = GlobalIntegrationSettings.objects.get(pk=1)
api_key = global_settings.openai_api_key
```
5. **Update Model Selection Logic (Already Done)**
- Our CreditService.get_model_for_operation() already implements priority
- Just need to add GlobalIntegrationSettings fallback
6. **Admin Interface**
- Adapt admin templates from 3283a83b monitoring.py
- Keep API monitoring, debug console, system health
- Update to show AIModelConfig instead of hardcoded choices
---
### Phase C: GlobalModuleSettings Integration
**Estimated Time:** 2 hours
1. **Copy GlobalModuleSettings Model (from 7a1e952a)**
```python
class GlobalModuleSettings(models.Model):
is_clustering_enabled = BooleanField(default=True)
is_ideas_enabled = BooleanField(default=True)
is_content_enabled = BooleanField(default=True)
is_optimization_enabled = BooleanField(default=True)
is_linking_enabled = BooleanField(default=True)
is_images_enabled = BooleanField(default=True)
is_publishing_enabled = BooleanField(default=True)
```
2. **Create Migration**
- Add to modules/system/migrations/
3. **Update Admin**
- Add GlobalModuleSettings admin interface
- Singleton pattern enforcement
4. **Update Frontend**
- Copy module settings UI components
- Update AppSidebar to respect module toggles
---
### Phase D: Token Analytics (Already Planned in Phase 3)
**Estimated Time:** 4 hours
1. **Adapt Reports from e041cb8e + c17b22e9**
- Copy token_usage_report() view
- Copy ai_cost_analysis() view
- Update to use model_config FK instead of model_used CharField
- Update to use cost_usd_input/output/total fields
2. **Copy Templates**
- token_usage.html with Chart.js
- ai_cost_analysis.html with visualizations
3. **Register URLs**
- Add to admin/site.py
---
### Phase E: AIEngine & Services (Already Planned in Phase 2.2-2.3)
**Estimated Time:** 3 hours
1. **Copy Token Extraction from c17b22e9:engine.py**
2. **Update Services from c17b22e9 Pattern**
- linker_service.py
- optimizer_service.py
- clustering_service.py
- ideas_service.py
- content_service.py
- image_service.py
---
## Migration File Consolidation
### Remote Migration Numbers (Conflicts)
```
Commit 3283a83b:
- 0002_add_global_settings_models.py
- 0004_fix_global_settings_remove_override.py
Commit 9e8ff4fb:
- 0004_fix_global_settings_remove_override.py (revised)
- 0005_add_model_choices.py
- 0006_fix_image_settings.py
- 0007_add_image_defaults.py
- 0008_add_default_image_service.py
- 0009_fix_variables_optional.py
Commit 7a1e952a:
- 0010_globalmodulesettings_and_more.py
- 0013_add_caption_to_images.py (writer module)
Our Local:
- 0019_add_ai_model_config.py (billing)
- 0002_add_model_fk_to_integrations.py (system)
```
### Renumbering Strategy
```
modules/system/migrations/:
- 0001_initial.py (existing)
- 0002_add_model_fk_to_integrations.py (OUR - keep)
- 0003_add_global_integration_settings.py (NEW - API keys only)
- 0004_add_global_module_settings.py (from remote 7a1e952a)
- 0005_add_caption_to_images.py (from remote 7a1e952a writer module)
modules/billing/migrations/:
- 0018_... (existing)
- 0019_add_ai_model_config.py (OUR - keep)
- 0020_update_credit_usage_log_costs.py (NEW - add cost_usd_input/output/total)
```
---
## Testing Strategy
### Unit Tests
1. GlobalIntegrationSettings API key access
2. AIModelConfig model selection priority
3. Credit calculation with new cost fields
4. Token extraction from AI responses
### Integration Tests
1. Full AI generation flow with token tracking
2. Model selection cascade (Task → Account → Operation → System → Fallback)
3. Credit deduction with granular costs
4. Analytics report data accuracy
### Frontend Tests
1. Module settings toggle
2. Image caption display
3. Integration settings UI with new structure
---
## Rollback Plan
### Git Strategy
```bash
# Create backup branch before integration
git branch backup-before-remote-integration
# Create feature branches for each phase
git checkout -b feature/phase-a-non-conflicting
git checkout -b feature/phase-b-global-settings
git checkout -b feature/phase-c-module-settings
git checkout -b feature/phase-d-analytics
git checkout -b feature/phase-e-services
# Merge phases incrementally
# Test after each phase
# Rollback if issues:
git checkout main
git reset --hard backup-before-remote-integration
```
### Database Rollback
- Keep migration rollback scripts for each phase
- Test migrations on staging database first
- Create database backup before applying
---
## File Checklist
### Files to Copy Directly (No Changes)
- [ ] AWS_ADMIN_ACCOUNT_AUDIT_REPORT.md
- [ ] DATA_SEGREGATION_SYSTEM_VS_USER.md
- [ ] SESSION_SUMMARY_DJANGO_ADMIN_ENHANCEMENT.md
- [ ] COMPREHENSIVE_REFACTORING_PLAN.md → 02_COMPREHENSIVE_REFACTORING_PLAN.md
- [ ] DJANGO_ADMIN_ACTIONS_COMPLETED.md
- [ ] DJANGO_ADMIN_ACTIONS_QUICK_REFERENCE.md
- [ ] DJANGO_ADMIN_ACTIONS_TODO.md
- [ ] FRONTEND_ADMIN_PAGES_COMPREHENSIVE_AUDIT.md
- [ ] FRONTEND_ADMIN_REFACTORING_COMPLETE.md
- [ ] SYSTEM_ARCHITECTURE_ANALYSIS_SUPERUSER_STRATEGY.md
- [ ] INTEGRATION-SETTINGS-WORKFLOW.md
- [ ] frontend/test-module-settings.html
### Files to Delete (Frontend Cleanup)
- [ ] frontend/src/pages/Admin/* (12 files)
- [ ] frontend/src/pages/Settings/UiElements/* (25 files)
- [ ] frontend/src/pages/Settings/ApiMonitor.tsx
- [ ] frontend/src/pages/Settings/DebugStatus.tsx
- [ ] frontend/src/pages/Settings/MasterStatus.tsx
- [ ] frontend/src/components/sidebar/ApiStatusIndicator.tsx
- [ ] frontend/src/components/auth/AdminGuard.tsx
### Files to Adapt (Merge Required)
- [ ] backend/igny8_core/modules/system/models.py (GlobalIntegrationSettings API keys only)
- [ ] backend/igny8_core/modules/system/global_settings_models.py (new, adapted)
- [ ] backend/igny8_core/admin/reports.py (token analytics adapted)
- [ ] backend/igny8_core/templates/admin/reports/token_usage.html (adapted)
- [ ] backend/igny8_core/templates/admin/reports/ai_cost_analysis.html (adapted)
- [ ] backend/igny8_core/ai/engine.py (token extraction)
- [ ] backend/igny8_core/business/linking/services/linker_service.py (token usage)
- [ ] backend/igny8_core/business/optimization/services/optimizer_service.py (token usage)
### Files to Create (New)
- [ ] backend/igny8_core/modules/system/migrations/0003_add_global_integration_settings.py
- [ ] backend/igny8_core/modules/system/migrations/0004_add_global_module_settings.py
- [ ] backend/igny8_core/modules/billing/migrations/0020_update_credit_usage_log_costs.py
- [ ] backend/igny8_core/management/commands/backfill_cost_fields.py
- [ ] frontend/src/store/moduleStore.ts
### Bulk Admin Actions to Add
- [ ] backend/igny8_core/ai/admin.py (bulk actions)
- [ ] backend/igny8_core/auth/admin.py (bulk actions)
- [ ] backend/igny8_core/business/automation/admin.py (bulk actions)
- [ ] backend/igny8_core/business/billing/admin.py (bulk actions)
- [ ] backend/igny8_core/business/integration/admin.py (bulk actions)
- [ ] backend/igny8_core/business/optimization/admin.py (bulk actions)
- [ ] backend/igny8_core/business/publishing/admin.py (bulk actions)
- [ ] backend/igny8_core/modules/billing/admin.py (bulk actions)
- [ ] backend/igny8_core/modules/planner/admin.py (bulk actions)
- [ ] backend/igny8_core/modules/system/admin.py (bulk actions)
- [ ] backend/igny8_core/modules/writer/admin.py (bulk actions)
---
## Estimated Timeline
| Phase | Tasks | Time | Priority |
|-------|-------|------|----------|
| Phase A | Non-conflicting (cleanup, bulk actions, modules, captions) | 2 hours | HIGH |
| Phase B | Global settings adaptation | 6-8 hours | CRITICAL |
| Phase C | GlobalModuleSettings | 2 hours | MEDIUM |
| Phase D | Token analytics | 4 hours | HIGH |
| Phase E | AIEngine & services | 3 hours | CRITICAL |
| **Total** | | **17-19 hours** | |
---
## Success Criteria
### Phase A Complete
- ✅ 43 frontend files deleted successfully
- ✅ Bulk actions working in all admin models
- ✅ Module settings UI functional
- ✅ Image captions field added and working
### Phase B Complete
- ✅ GlobalIntegrationSettings storing API keys
- ✅ AIModelConfig used for all model selection
- ✅ IntegrationSettings.config working with parameter overrides
- ✅ Admin monitoring templates functional
- ✅ No hardcoded model choices anywhere
### Phase C Complete
- ✅ GlobalModuleSettings admin accessible
- ✅ Module toggles working in frontend
- ✅ Disabled modules inaccessible to users
### Phase D Complete
- ✅ Token usage report showing accurate data
- ✅ AI cost analysis with forecasting
- ✅ Charts rendering correctly
- ✅ All reports use model_config FK
### Phase E Complete
- ✅ AIEngine extracting tokens from all providers
- ✅ All 6 services passing tokens to CreditService
- ✅ Token data flowing through entire system
- ✅ Credit calculation accurate for all models
---
## Next Steps
1. **Immediate:** Start Phase A (non-conflicting features)
```bash
git checkout -b feature/phase-a-integration
git cherry-pick eb6cba79 # Frontend cleanup
git cherry-pick ab0d6469 # Bulk actions
git cherry-pick 646095da # Module settings
# Test, then merge to main
```
2. **Critical:** Design GlobalIntegrationSettings migration (Phase B)
- Create hybrid model combining API keys + AIModelConfig references
- Write migration to create table and populate defaults
- Update all API key access code
3. **Follow-up:** Execute phases C, D, E in sequence
- Each phase builds on previous
- Test thoroughly after each phase
4. **Documentation:** Update all docs for final architecture
- Revise AI-MODEL-COST-REFACTOR-PLAN.md with global settings
- Update CREDITS-TOKENS-GUIDE.md
- Create final architecture diagram
---
## Conclusion
This plan integrates 9 remote commits (~4,500 lines added, ~9,000 lines removed) while preserving our superior AIModelConfig architecture. The phased approach minimizes risk and allows incremental testing. Estimated completion: 17-19 hours of focused development.
**Key Innovation:** Hybrid GlobalIntegrationSettings that stores API keys centrally while delegating model selection to our AIModelConfig system - best of both approaches.
Reference in New Issue View Git Blame Copy Permalink