Files
igny8/IGNY8_AI_UNIFICATION_MIGRATION_PLAN.md
2025-11-10 23:36:14 +05:00

650 lines
22 KiB
Markdown

# IGNY8 AI System Unification — Complete Migration Plan
**Date:** 2024-12-19
**Goal:** Unify all AI functions into single structure, remove all redundancy, implement checklist-style progress UI
**Estimated Time:** 5 stages, ~2-3 days total
---
## Overview
This migration plan unifies the IGNY8 AI system by:
1. Standardizing backend progress messages with input data
2. Implementing checklist-style progress UI with 3 states (pending/in-progress/completed)
3. Migrating all views to use unified `run_ai_task` entrypoint
4. Removing duplicate code and deprecated files
5. Final cleanup and verification
---
## Stage 1: Backend — Standardize Progress Messages with Input Data
**Goal:** Update `AIEngine` to send user-friendly messages with actual input data
**Files to Modify:** `backend/igny8_core/ai/engine.py`
**Estimated Time:** 1-2 hours
### Step 1.1: Add Helper Methods to AIEngine
**File:** `backend/igny8_core/ai/engine.py`
Add these helper methods to the `AIEngine` class (after `__init__` method):
```python
def _get_input_description(self, function_name: str, payload: dict, count: int) -> str:
"""Get user-friendly input description"""
if function_name == 'auto_cluster':
return f"{count} keyword{'s' if count != 1 else ''}"
elif function_name == 'generate_ideas':
return f"{count} cluster{'s' if count != 1 else ''}"
elif function_name == 'generate_content':
return f"{count} task{'s' if count != 1 else ''}"
elif function_name == 'generate_images':
return f"{count} task{'s' if count != 1 else ''}"
return f"{count} item{'s' if count != 1 else ''}"
def _get_prep_message(self, function_name: str, count: int, data: Any) -> str:
"""Get user-friendly prep message"""
if function_name == 'auto_cluster':
return f"Loading {count} keyword{'s' if count != 1 else ''}"
elif function_name == 'generate_ideas':
return f"Loading {count} cluster{'s' if count != 1 else ''}"
elif function_name == 'generate_content':
return f"Preparing {count} content idea{'s' if count != 1 else ''}"
elif function_name == 'generate_images':
return f"Extracting image prompts from {count} task{'s' if count != 1 else ''}"
return f"Preparing {count} item{'s' if count != 1 else ''}"
def _get_ai_call_message(self, function_name: str, count: int) -> str:
"""Get user-friendly AI call message"""
if function_name == 'auto_cluster':
return f"Grouping {count} keyword{'s' if count != 1 else ''} into clusters"
elif function_name == 'generate_ideas':
return f"Generating content ideas for {count} cluster{'s' if count != 1 else ''}"
elif function_name == 'generate_content':
return f"Writing article{'s' if count != 1 else ''} with AI"
elif function_name == 'generate_images':
return f"Creating image{'s' if count != 1 else ''} with AI"
return f"Processing with AI"
def _get_parse_message(self, function_name: str) -> str:
"""Get user-friendly parse message"""
if function_name == 'auto_cluster':
return "Organizing clusters"
elif function_name == 'generate_ideas':
return "Structuring outlines"
elif function_name == 'generate_content':
return "Formatting content"
elif function_name == 'generate_images':
return "Processing images"
return "Processing results"
def _get_save_message(self, function_name: str, count: int) -> str:
"""Get user-friendly save message"""
if function_name == 'auto_cluster':
return f"Saving {count} cluster{'s' if count != 1 else ''}"
elif function_name == 'generate_ideas':
return f"Saving {count} idea{'s' if count != 1 else ''}"
elif function_name == 'generate_content':
return f"Saving {count} article{'s' if count != 1 else ''}"
elif function_name == 'generate_images':
return f"Saving {count} image{'s' if count != 1 else ''}"
return f"Saving {count} item{'s' if count != 1 else ''}"
```
### Step 1.2: Update execute() Method to Use Helper Methods
**File:** `backend/igny8_core/ai/engine.py`
In the `execute()` method, replace step tracking messages:
**Replace lines 48-57 (INIT phase):**
```python
# OLD:
self.console_tracker.prep("Validating input payload")
validated = fn.validate(payload, self.account)
if not validated['valid']:
self.console_tracker.error('ValidationError', validated['error'])
return self._handle_error(validated['error'], fn)
self.console_tracker.prep("Validation complete")
self.step_tracker.add_request_step("INIT", "success", "Validation complete")
self.tracker.update("INIT", 10, "Validation complete", meta=self.step_tracker.get_meta())
# NEW:
# Extract input data for user-friendly messages
ids = payload.get('ids', [])
input_count = len(ids) if ids else 0
input_description = self._get_input_description(function_name, payload, input_count)
self.console_tracker.prep(f"Validating {input_description}")
validated = fn.validate(payload, self.account)
if not validated['valid']:
self.console_tracker.error('ValidationError', validated['error'])
return self._handle_error(validated['error'], fn)
validation_message = f"Validating {input_description}"
self.console_tracker.prep("Validation complete")
self.step_tracker.add_request_step("INIT", "success", validation_message)
self.tracker.update("INIT", 10, validation_message, meta=self.step_tracker.get_meta())
```
**Replace lines 59-79 (PREP phase):**
```python
# OLD:
self.console_tracker.prep("Loading data from database")
data = fn.prepare(payload, self.account)
# ... existing data_count logic ...
self.console_tracker.prep(f"Building prompt from {data_count} items")
prompt = fn.build_prompt(data, self.account)
self.console_tracker.prep(f"Prompt built: {len(prompt)} characters")
self.step_tracker.add_request_step("PREP", "success", f"Loaded {data_count} items, built prompt ({len(prompt)} chars)")
self.tracker.update("PREP", 25, f"Data prepared: {data_count} items", meta=self.step_tracker.get_meta())
# NEW:
prep_message = self._get_prep_message(function_name, input_count, payload)
self.console_tracker.prep(prep_message)
data = fn.prepare(payload, self.account)
# ... existing data_count logic ...
prompt = fn.build_prompt(data, self.account)
self.console_tracker.prep(f"Prompt built: {len(prompt)} characters")
self.step_tracker.add_request_step("PREP", "success", prep_message)
self.tracker.update("PREP", 25, prep_message, meta=self.step_tracker.get_meta())
```
**Replace lines 136-141 (AI_CALL phase):**
```python
# OLD:
self.step_tracker.add_response_step(
"AI_CALL",
"success",
f"Calling {model or 'default'} model..."
)
self.tracker.update("AI_CALL", 30, f"Sending to {model or 'default'}...", meta=self.step_tracker.get_meta())
# NEW:
ai_call_message = self._get_ai_call_message(function_name, data_count)
self.step_tracker.add_response_step("AI_CALL", "success", ai_call_message)
self.tracker.update("AI_CALL", 50, ai_call_message, meta=self.step_tracker.get_meta())
```
**Find PARSE phase (around line 200-210) and replace:**
```python
# OLD:
self.step_tracker.add_response_step("PARSE", "success", "Parsing response...")
self.tracker.update("PARSE", 70, "Parsing response...", meta=self.step_tracker.get_meta())
# NEW:
parse_message = self._get_parse_message(function_name)
self.step_tracker.add_response_step("PARSE", "success", parse_message)
self.tracker.update("PARSE", 70, parse_message, meta=self.step_tracker.get_meta())
```
**Find SAVE phase (around line 250-260) and replace:**
```python
# OLD:
self.step_tracker.add_response_step("SAVE", "success", "Saving results...")
self.tracker.update("SAVE", 85, "Saving results...", meta=self.step_tracker.get_meta())
# NEW:
save_message = self._get_save_message(function_name, data_count)
self.step_tracker.add_response_step("SAVE", "success", save_message)
self.tracker.update("SAVE", 85, save_message, meta=self.step_tracker.get_meta())
```
### Step 1.3: Remove Technical Debug Messages
**File:** `backend/igny8_core/ai/engine.py`
Remove or comment out lines 115-124 (model configuration tracking in step tracker):
```python
# REMOVE these lines (keep console logging, but not step tracker):
# self.step_tracker.add_request_step(
# "PREP",
# "success",
# f"AI model in settings: {model_from_integration or 'Not set'}"
# )
# self.step_tracker.add_request_step(
# "PREP",
# "success",
# f"AI model selected for request: {model or 'default'}"
# )
```
### Verification Checklist for Stage 1:
- [ ] Helper methods added to `AIEngine` class
- [ ] All phase messages updated to use helper methods
- [ ] Messages include actual input counts (e.g., "Validating 5 keywords")
- [ ] No technical terms like "database" or "parsing" in user-facing messages
- [ ] Test: Run `auto_cluster` and verify messages in step logs
---
## Stage 2: Frontend — Implement Checklist-Style Progress Modal
**Goal:** Replace progress bar with checklist UI showing 3 states (pending/in-progress/completed)
**Files to Modify:** `frontend/src/components/common/ProgressModal.tsx`
**Files to Create:** None
**Estimated Time:** 2-3 hours
### Step 2.1: Replace ProgressModal Component
**File:** `frontend/src/components/common/ProgressModal.tsx`
Replace the entire file with the new checklist-style implementation (see previous response for full code).
Key changes:
- Remove progress bar component
- Add checklist-style step display
- Add 3-state logic (pending/in-progress/completed)
- Add success alert in same modal when completed
- Use step logs to determine current phase
### Step 2.2: Update useProgressModal Hook (Simplify)
**File:** `frontend/src/hooks/useProgressModal.ts`
Remove all `aiRequestLogsStore` references:
- Remove lines 501-642 (all store-related code)
- Keep only polling logic and state management
- Simplify step mapping logic
### Verification Checklist for Stage 2:
- [ ] ProgressModal shows checklist instead of progress bar
- [ ] Steps show as pending (gray/disabled) initially
- [ ] Steps show as in-progress (blue/spinner) when active
- [ ] Steps show as completed (green/checkmark) when done
- [ ] Success alert appears in same modal when completed
- [ ] No errors in browser console
- [ ] Test: Run `auto_cluster` and verify checklist UI
---
## Stage 3: Migrate Views to Unified Entrypoint
**Goal:** Update all views to use `run_ai_task` instead of legacy task functions
**Files to Modify:**
- `backend/igny8_core/modules/writer/views.py`
- `backend/igny8_core/modules/planner/views.py` (verify already migrated)
**Estimated Time:** 2-3 hours
### Step 3.1: Migrate auto_generate_content View
**File:** `backend/igny8_core/modules/writer/views.py`
**Replace lines 180-228** (the entire try/except block for Celery task):
```python
# OLD:
from .tasks import auto_generate_content_task
if hasattr(auto_generate_content_task, 'delay'):
task = auto_generate_content_task.delay(ids, account_id=account_id)
# ... rest of old code
# NEW:
from igny8_core.ai.tasks import run_ai_task
from kombu.exceptions import OperationalError as KombuOperationalError
try:
if hasattr(run_ai_task, 'delay'):
task = run_ai_task.delay(
function_name='generate_content',
payload={'ids': ids},
account_id=account_id
)
logger.info(f"Task queued: {task.id}")
return Response({
'success': True,
'task_id': str(task.id),
'message': 'Content generation started'
}, status=status.HTTP_200_OK)
else:
# Celery not available - execute synchronously
logger.info("auto_generate_content: Executing synchronously (Celery not available)")
result = run_ai_task(
function_name='generate_content',
payload={'ids': ids},
account_id=account_id
)
if result.get('success'):
return Response({
'success': True,
'tasks_updated': result.get('count', 0),
'message': 'Content generated successfully'
}, status=status.HTTP_200_OK)
else:
return Response({
'error': result.get('error', 'Content generation failed'),
'type': 'TaskExecutionError'
}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
except KombuOperationalError as e:
logger.error(f"Celery connection error: {str(e)}")
return Response({
'error': 'Task queue unavailable. Please try again.',
'type': 'QueueError'
}, status=status.HTTP_503_SERVICE_UNAVAILABLE)
except Exception as e:
logger.error(f"Error queuing content generation task: {str(e)}", exc_info=True)
return Response({
'error': f'Failed to start content generation: {str(e)}',
'type': 'TaskError'
}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
```
### Step 3.2: Migrate auto_generate_images View
**File:** `backend/igny8_core/modules/writer/views.py`
**Replace lines 358-377** (the entire try/except block):
```python
# OLD:
from .tasks import auto_generate_images_task
if hasattr(auto_generate_images_task, 'delay'):
task = auto_generate_images_task.delay(task_ids, account_id=account_id)
# ... rest of old code
# NEW:
from igny8_core.ai.tasks import run_ai_task
from kombu.exceptions import OperationalError as KombuOperationalError
try:
if hasattr(run_ai_task, 'delay'):
task = run_ai_task.delay(
function_name='generate_images',
payload={'ids': task_ids},
account_id=account_id
)
return Response({
'success': True,
'task_id': str(task.id),
'message': 'Image generation started'
}, status=status.HTTP_200_OK)
else:
# Celery not available - execute synchronously
result = run_ai_task(
function_name='generate_images',
payload={'ids': task_ids},
account_id=account_id
)
if result.get('success'):
return Response({
'success': True,
'images_created': result.get('count', 0),
'message': result.get('message', 'Image generation completed')
}, status=status.HTTP_200_OK)
else:
return Response({
'error': result.get('error', 'Image generation failed'),
'type': 'TaskExecutionError'
}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
except KombuOperationalError as e:
logger.error(f"Celery connection error: {str(e)}")
return Response({
'error': 'Task queue unavailable. Please try again.',
'type': 'QueueError'
}, status=status.HTTP_503_SERVICE_UNAVAILABLE)
except Exception as e:
logger.error(f"Error queuing image generation task: {str(e)}", exc_info=True)
return Response({
'error': f'Failed to start image generation: {str(e)}',
'type': 'TaskError'
}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
```
### Step 3.3: Verify Planner Views Already Migrated
**File:** `backend/igny8_core/modules/planner/views.py`
Verify that `auto_cluster` and `auto_generate_ideas` already use `run_ai_task`:
- [ ] `auto_cluster` uses `run_ai_task` (line 480)
- [ ] `auto_generate_ideas` uses `run_ai_task` (line 756)
### Verification Checklist for Stage 3:
- [ ] `auto_generate_content` uses `run_ai_task`
- [ ] `auto_generate_images` uses `run_ai_task`
- [ ] All views return consistent response format
- [ ] Error handling is consistent
- [ ] Test: Generate content and verify it works
- [ ] Test: Generate images and verify it works
---
## Stage 4: Remove Duplicate Code and Deprecated Files
**Goal:** Clean up duplicate constants, remove deprecated files, simplify code
**Files to Delete:** 6 files
**Files to Modify:** 5 files
**Estimated Time:** 2-3 hours
### Step 4.1: Remove Duplicate Constants
**File:** `backend/igny8_core/utils/ai_processor.py`
**Replace lines 18-44** (duplicate constants):
```python
# OLD:
MODEL_RATES = { ... }
IMAGE_MODEL_RATES = { ... }
VALID_OPENAI_IMAGE_MODELS = { ... }
VALID_SIZES_BY_MODEL = { ... }
# NEW:
from igny8_core.ai.constants import (
MODEL_RATES,
IMAGE_MODEL_RATES,
VALID_OPENAI_IMAGE_MODELS,
VALID_SIZES_BY_MODEL,
DEFAULT_AI_MODEL,
JSON_MODE_MODELS,
)
```
### Step 4.2: Remove response_steps Parameter
**File:** `backend/igny8_core/utils/ai_processor.py`
Remove `response_steps` parameter from all methods:
- Find all method signatures with `response_steps=None` (lines 1064, 1135, etc.)
- Remove the parameter
- Remove all `response_steps.append()` calls (lines 1135-1299)
- Keep the file (still used by legacy code temporarily)
### Step 4.3: Simplify task_progress Endpoint
**File:** `backend/igny8_core/modules/system/integration_views.py`
**Simplify `task_progress()` method** (lines 734-1163):
Replace complex extraction logic with simple meta retrieval:
```python
# In task_progress method, replace lines 784-1100 with:
meta = {}
request_steps = []
response_steps = []
try:
if hasattr(task, 'info') and task.info:
if isinstance(task.info, dict):
meta = task.info.get('meta', {})
if isinstance(meta, dict):
request_steps = meta.get('request_steps', [])
response_steps = meta.get('response_steps', [])
except Exception as e:
logger.debug(f"Error extracting meta: {str(e)}")
# Use request_steps and response_steps in response
```
### Step 4.4: Remove Deprecated Store References
**File:** `frontend/src/services/api.ts`
Remove all `aiRequestLogsStore` imports and references:
- Remove line 4: `import { useAIRequestLogsStore } from '../store/aiRequestLogsStore';`
- Remove lines 579, 601, 671-672, 730-731, 812-813, 1185-1186, 1265-1266 (all store references)
**File:** `frontend/src/hooks/useProgressModal.ts`
Remove all `aiRequestLogsStore` references (lines 501-642)
**File:** `frontend/src/templates/TablePageTemplate.tsx`
Remove commented import (line 44)
### Step 4.5: Delete Deprecated Files
**Delete these files:**
1. `backend/igny8_core/modules/planner/tasks.py`
- Already deprecated, no longer used
2. `backend/igny8_core/modules/writer/tasks.py`
- No longer used after Stage 3 migration
3. `backend/igny8_core/ai/processor.py`
- Deprecated wrapper, redirects to AICore
4. `frontend/src/store/aiRequestLogsStore.ts`
- Deprecated debug store
5. `frontend/src/components/debug/ResourceDebugOverlay.tsx`
- Optional: Delete if not needed
6. `frontend/src/components/debug/ResourceDebugToggle.tsx`
- Optional: Delete if not needed (or keep if still used)
### Verification Checklist for Stage 4:
- [ ] Duplicate constants removed from `ai_processor.py`
- [ ] `response_steps` parameter removed from all methods
- [ ] `task_progress` endpoint simplified
- [ ] All deprecated store references removed
- [ ] Deprecated files deleted
- [ ] No import errors after deletions
- [ ] Test: Verify all AI functions still work
---
## Stage 5: Final Cleanup and Verification
**Goal:** Final testing, documentation, and cleanup
**Estimated Time:** 1-2 hours
### Step 5.1: Remove Debug Overlay from Layout (if deleted)
**File:** `frontend/src/layout/AppLayout.tsx`
If you deleted debug components, remove:
- Line 12: `import ResourceDebugOverlay from "../components/debug/ResourceDebugOverlay";`
- Lines 166-180: Debug toggle listener
- Lines 197-198: `<ResourceDebugOverlay enabled={debugEnabled} />`
### Step 5.2: Update Function Metadata
**File:** `backend/igny8_core/ai/functions/auto_cluster.py`
Verify `get_metadata()` returns correct phase messages (should already be correct)
**File:** `backend/igny8_core/ai/functions/generate_ideas.py`
Verify `get_metadata()` returns correct phase messages
**File:** `backend/igny8_core/ai/functions/generate_content.py`
Verify `get_metadata()` returns correct phase messages
**File:** `backend/igny8_core/ai/functions/generate_images.py`
Verify `get_metadata()` returns correct phase messages
### Step 5.3: Comprehensive Testing
Test each AI function end-to-end:
**Test 1: Keyword Clustering**
- [ ] Select 5-10 keywords
- [ ] Click "Auto Cluster"
- [ ] Verify checklist shows: "Validating 5 keywords" → "Loading 5 keywords" → etc.
- [ ] Verify success message: "Clustering complete — keywords grouped into meaningful clusters."
- [ ] Verify clusters created in database
**Test 2: Idea Generation**
- [ ] Select 1-2 clusters
- [ ] Click "Generate Ideas"
- [ ] Verify checklist shows correct steps
- [ ] Verify success message: "Content ideas and outlines created successfully."
- [ ] Verify ideas created in database
**Test 3: Content Generation**
- [ ] Select 1-2 tasks
- [ ] Click "Generate Content"
- [ ] Verify checklist shows correct steps
- [ ] Verify success message: "Article drafted successfully."
- [ ] Verify content saved to tasks
**Test 4: Image Generation**
- [ ] Select 1-2 tasks with content
- [ ] Click "Generate Images"
- [ ] Verify checklist shows correct steps
- [ ] Verify success message: "Images created and saved successfully."
- [ ] Verify images created in database
### Step 5.4: Code Review Checklist
- [ ] All AI functions use `run_ai_task` entrypoint
- [ ] All progress messages include input data
- [ ] No duplicate constants
- [ ] No deprecated code references
- [ ] Frontend shows checklist UI correctly
- [ ] Success messages appear in modal
- [ ] No console errors
- [ ] No TypeScript errors
- [ ] No Python linting errors
### Step 5.5: Documentation Update
Update any relevant documentation:
- [ ] Update API documentation if needed
- [ ] Update developer guide if needed
- [ ] Mark this migration as complete
---
## Rollback Plan
If issues occur during migration:
1. **Stage 1-2 Issues:** Revert `engine.py` and `ProgressModal.tsx` changes
2. **Stage 3 Issues:** Revert view changes, keep using legacy tasks temporarily
3. **Stage 4 Issues:** Restore deleted files from git history
4. **Stage 5 Issues:** Fix specific issues without rolling back
---
## Success Criteria
Migration is complete when:
- ✅ All AI functions use unified `run_ai_task` entrypoint
- ✅ All progress messages are user-friendly with input data
- ✅ Frontend shows checklist-style progress UI
- ✅ Success messages appear in modal
- ✅ No duplicate code remains
- ✅ All deprecated files deleted
- ✅ All tests pass
- ✅ No console/terminal errors
---
## Notes
- **Backward Compatibility:** Legacy code in `utils/ai_processor.py` is kept temporarily for any remaining references
- **Debug Components:** ResourceDebugOverlay can be kept if still needed for other debugging
- **Testing:** Test each stage before moving to next stage
- **Git:** Commit after each stage for easy rollback
---
**End of Migration Plan**