785 lines
27 KiB
Plaintext
785 lines
27 KiB
Plaintext
# IGNY8 Automation System - Detailed Task List for AI Agent
|
|
|
|
## CRITICAL ANALYSIS
|
|
|
|
Based on the documentation and current implementation status, I've identified significant issues with the automation system and legacy SiteBuilder references that need systematic resolution.
|
|
|
|
---
|
|
## SECTION 1: udapte correct status, adn assoitae keywrods ot lcuster properly
|
|
Auto cluster AI function is currently setting status of clusters to active, and keywords are not mapped to clusters when run with automation. Update the original auto cluster AI function to use status new instead of active, and identify whether the keyword-to-cluster mapping issue is in the AI function or in the automation.
|
|
|
|
Actualy the orringal ai fucntion has both of this issue, once fixed tehn ai fucntion wil lwork fine and autoamtion also, will run better,
|
|
|
|
|
|
|
|
## SECTION 2: LEGACY SITEBUILDER/BLUEPRINT REMOVAL
|
|
|
|
### Task 2.1: Database Models Cleanup
|
|
**Files to Remove Completely:**
|
|
1. `backend/igny8_core/business/site_building/models.py` - Already stubbed, remove entirely
|
|
2. Migration already exists: `0002_remove_blueprint_models.py` - Verify it ran successfully
|
|
|
|
**Database Verification:**
|
|
1. Connect to production database
|
|
2. Run SQL: `SELECT tablename FROM pg_tables WHERE tablename LIKE '%blueprint%' OR tablename LIKE '%site_building%';`
|
|
3. Expected result: No tables (already dropped)
|
|
4. If tables exist, manually run DROP TABLE commands from migration
|
|
|
|
**Foreign Key Cleanup:**
|
|
1. Check `igny8_deployment_records` table - verify `site_blueprint_id` column removed
|
|
2. Check `igny8_publishing_records` table - verify `site_blueprint_id` column removed
|
|
3. Confirm indexes dropped: `igny8_publishing_recor_site_blueprint_id_des_b7c4e5f8_idx`
|
|
|
|
---
|
|
|
|
### Task 2.2: Backend Code References Removal
|
|
|
|
**Phase 2.2.1: Remove Stub Models**
|
|
- **File:** `backend/igny8_core/business/site_building/models.py`
|
|
- **Action:** Delete entire file
|
|
- **Reason:** Contains only stub classes (`SiteBlueprint`, `PageBlueprint`, `SiteBlueprintCluster`, `SiteBlueprintTaxonomy`) with no functionality
|
|
|
|
**Phase 2.2.2: Remove Entire site_building App**
|
|
- **Directory:** `backend/igny8_core/business/site_building/`
|
|
- **Action:** Delete entire directory
|
|
- **Reason:** All functionality deprecated, no active code
|
|
|
|
**Files to Delete:**
|
|
1. `services/structure_generation_service.py` - Calls deprecated AI function
|
|
2. `services/page_generation_service.py` - References PageBlueprint
|
|
3. `services/taxonomy_service.py` - Uses SiteBlueprintTaxonomy
|
|
4. `services/file_management_service.py` - SiteBuilder file management
|
|
5. `tests/` - All test files reference removed models
|
|
6. `admin.py` - Already commented out
|
|
7. `migrations/` - Keep for database history, but app removal makes them inert
|
|
|
|
**Phase 2.2.3: Remove site_builder Module**
|
|
- **Directory:** `backend/igny8_core/modules/site_builder.backup/`
|
|
- **Action:** Delete entire directory (already marked `.backup`)
|
|
- **Contains:** Deprecated API endpoints, serializers, views for blueprint management
|
|
|
|
---
|
|
|
|
### Task 2.3: Settings Configuration Cleanup
|
|
|
|
**File:** `backend/igny8_core/settings.py`
|
|
|
|
**Changes:**
|
|
1. Line 56: Already commented out - Remove comment entirely
|
|
2. Line 61: Already commented out - Remove comment entirely
|
|
3. Verify `INSTALLED_APPS` list is clean
|
|
|
|
**Verification:**
|
|
- Run `python manage.py check` - Should pass
|
|
- Run `python manage.py migrate --plan` - Should show no pending site_building migrations
|
|
|
|
---
|
|
|
|
### Task 2.4: URL Routing Cleanup
|
|
|
|
**File:** `backend/igny8_core/urls.py`
|
|
|
|
**Changes:**
|
|
1. Line 42: Comment already exists - Remove comment entirely
|
|
2. Verify no routing to `site-builder/` endpoints exists
|
|
|
|
**Verification:**
|
|
- Run Django server
|
|
- Attempt to access `/api/v1/site-builder/blueprints/` - Should return 404
|
|
- Check API root `/api/v1/` - Should not list site-builder endpoints
|
|
|
|
---
|
|
|
|
### Task 2.5: AI Function Removal
|
|
|
|
**File:** `backend/igny8_core/ai/functions/generate_page_content.py`
|
|
|
|
**Problem:** This AI function depends on `PageBlueprint` model which no longer exists.
|
|
|
|
**Action Required:**
|
|
1. **DELETE FILE:** `generate_page_content.py` (21 references to PageBlueprint)
|
|
2. **UPDATE:** `backend/igny8_core/ai/registry.py` - Remove lazy loader registration
|
|
3. **UPDATE:** `backend/igny8_core/ai/engine.py` - Remove from operation type mappings (line 599)
|
|
|
|
**Verification:**
|
|
- Search codebase for `generate_page_content` function calls
|
|
- Ensure no active code relies on this function
|
|
- Confirm AI function registry no longer lists it
|
|
|
|
---
|
|
|
|
### Task 2.6: Backend Import Statement Cleanup
|
|
|
|
**Files with Import Statements to Update:**
|
|
|
|
1. **backend/igny8_core/business/integration/services/content_sync_service.py**
|
|
- Lines 378, 488: `from igny8_core.business.site_building.models import SiteBlueprint`
|
|
- **Action:** Remove import, remove dependent code blocks (lines 382-388, 491-496)
|
|
- **Alternative:** Service should use `ContentTaxonomy` directly (post-migration model)
|
|
|
|
2. **backend/igny8_core/business/integration/services/sync_health_service.py**
|
|
- Line 335: `from igny8_core.business.site_building.models import SiteBlueprint, SiteBlueprintTaxonomy`
|
|
- **Action:** Remove import, refactor taxonomy checks to use `ContentTaxonomy`
|
|
|
|
3. **backend/igny8_core/business/publishing/services/adapters/sites_renderer_adapter.py**
|
|
- Line 15: `from igny8_core.business.site_building.models import SiteBlueprint`
|
|
- **Action:** Entire adapter is deprecated - DELETE FILE
|
|
- **Reason:** Designed to deploy SiteBlueprint instances, no longer applicable
|
|
|
|
4. **backend/igny8_core/business/publishing/services/deployment_readiness_service.py**
|
|
- Line 10: `from igny8_core.business.site_building.models import SiteBlueprint`
|
|
- **Action:** DELETE FILE or refactor to remove blueprint checks
|
|
- **Reason:** Service checks blueprint readiness for deployment
|
|
|
|
5. **backend/igny8_core/business/publishing/services/deployment_service.py**
|
|
- Line 10: `from igny8_core.business.site_building.models import SiteBlueprint`
|
|
- **Action:** Remove blueprint-specific deployment methods
|
|
|
|
---
|
|
|
|
### Task 2.7: Frontend Files Removal
|
|
|
|
**Phase 2.7.1: Remove Type Definitions**
|
|
- **File:** `frontend/src/types/siteBuilder.ts`
|
|
- **Action:** Delete file entirely
|
|
- **References:** Used in store and components
|
|
|
|
**Phase 2.7.2: Remove API Service**
|
|
- **File:** `frontend/src/services/siteBuilder.api.ts`
|
|
- **Action:** Delete file
|
|
- **Contains:** API methods for blueprint CRUD operations
|
|
|
|
**Phase 2.7.3: Remove Pages**
|
|
- **Directory:** `frontend/src/pages/Sites/`
|
|
- **Files to Review:**
|
|
- `Editor.tsx` - Uses PageBlueprint, SiteBlueprint types (lines 15-36)
|
|
- `PageManager.tsx` - Fetches blueprints (lines 126-137)
|
|
- `DeploymentPanel.tsx` - Blueprint deployment UI (46 references)
|
|
|
|
**Action for Pages:**
|
|
1. If pages ONLY deal with blueprints - DELETE
|
|
2. If pages have mixed functionality - REFACTOR to remove blueprint code
|
|
3. Likely DELETE: `Editor.tsx`, `DeploymentPanel.tsx`
|
|
4. Likely REFACTOR: `Dashboard.tsx` (remove blueprint widget)
|
|
|
|
**Phase 2.7.4: Remove Store**
|
|
- **File:** `frontend/src/store/siteDefinitionStore.ts`
|
|
- **Action:** Review dependencies, likely DELETE
|
|
- **Alternative:** If used for non-blueprint purposes, refactor to remove PageBlueprint types
|
|
|
|
**Phase 2.7.5: Remove Components**
|
|
- **File:** `frontend/src/components/sites/SiteProgressWidget.tsx`
|
|
- **Action:** DELETE if blueprint-specific
|
|
- **Uses:** `blueprintId` prop, calls `fetchSiteProgress(blueprintId)`
|
|
|
|
---
|
|
|
|
### Task 2.8: Frontend Import and Reference Cleanup
|
|
|
|
**Files Requiring Updates:**
|
|
|
|
1. **frontend/src/services/api.ts**
|
|
- Lines 2302-2532: Multiple blueprint-related functions
|
|
- **Action:** Remove these function exports:
|
|
- `fetchDeploymentReadiness`
|
|
- `createSiteBlueprint`, `updateSiteBlueprint`
|
|
- `attachClustersToBlueprint`, `detachClustersFromBlueprint`
|
|
- `fetchBlueprintsTaxonomies`, `createBlueprintTaxonomy`
|
|
- `importBlueprintsTaxonomies`
|
|
- `updatePageBlueprint`, `regeneratePageBlueprint`
|
|
|
|
2. **frontend/src/pages/Planner/Dashboard.tsx**
|
|
- Lines 30-31: Commented imports
|
|
- **Action:** Remove commented lines entirely
|
|
|
|
3. **frontend/src/config/pages/tasks.config.tsx**
|
|
- Lines 110-111: Logic for `[Site Builder]` task title formatting
|
|
- **Action:** Remove special handling, update title display logic
|
|
|
|
---
|
|
|
|
### Task 2.9: Sites Renderer Cleanup
|
|
|
|
**File:** `sites/src/loaders/loadSiteDefinition.ts`
|
|
|
|
**Current Behavior (Lines 103-159):**
|
|
- API load fails → Falls back to blueprint endpoint
|
|
- Transforms blueprint to site definition format
|
|
|
|
**Required Changes:**
|
|
1. Remove fallback to blueprint endpoint (lines 116-127)
|
|
2. Remove `transformBlueprintToSiteDefinition` function (lines 137-159)
|
|
3. If API fails, return proper error instead of fallback
|
|
4. Update error messages to remove blueprint references
|
|
|
|
---
|
|
|
|
### Task 2.10: Documentation Cleanup
|
|
|
|
**Files to Remove:**
|
|
1. `docs/igny8-pp/TAXONOMY/QUICK-REFERENCE-TAXONOMY.md` - References SiteBuilder removal
|
|
2. Update `docs/tech-stack/00-SYSTEM-ARCHITECTURE-MASTER-REFERENCE.md`:
|
|
- Remove "Site Blueprints" from feature list (line 45)
|
|
- Remove `site_builder/` from architecture diagrams (lines 179-180)
|
|
- Remove SiteBuilder from system overview (line 1517)
|
|
|
|
**Files to Update:**
|
|
1. `docs/igny8-pp/01-IGNY8-REST-API-COMPLETE-REFERENCE.md`:
|
|
- Remove entire section: "Site Blueprints" (lines 1201-1230)
|
|
- Remove "Page Blueprints" section (lines 1230-1247)
|
|
- Update deployment endpoints to remove blueprint references
|
|
|
|
2. `docs/igny8-pp/02-PLANNER-WRITER-WORKFLOW-TECHNICAL-GUIDE.md`:
|
|
- Remove SiteBlueprintTaxonomy references (lines 114, 151)
|
|
|
|
---
|
|
|
|
### Task 2.11: Test Files Cleanup
|
|
|
|
**Backend Tests:**
|
|
1. DELETE: `backend/igny8_core/ai/tests/test_generate_site_structure_function.py`
|
|
2. DELETE: `backend/igny8_core/business/site_building/tests/` (entire directory)
|
|
3. DELETE: `backend/igny8_core/business/publishing/tests/test_deployment_service.py`
|
|
4. DELETE: `backend/igny8_core/business/publishing/tests/test_publisher_service.py`
|
|
5. DELETE: `backend/igny8_core/business/publishing/tests/test_adapters.py`
|
|
|
|
**Frontend Tests:**
|
|
1. DELETE: `frontend/src/__tests__/sites/BulkGeneration.test.tsx`
|
|
2. UPDATE: `frontend/src/__tests__/sites/PromptManagement.test.tsx`:
|
|
- Remove site_structure_generation prompt type checks (lines 27-28)
|
|
3. UPDATE: `frontend/src/__tests__/sites/SiteManagement.test.tsx`:
|
|
- Remove `[Site Builder]` task filtering logic (lines 50-51)
|
|
|
|
---
|
|
|
|
### Task 2.12: Database Migration Verification
|
|
|
|
**Critical Checks:**
|
|
1. Verify `0002_remove_blueprint_models.py` migration applied in all environments
|
|
2. Check for orphaned data:
|
|
- Query for any `Tasks` with `taxonomy_id` pointing to deleted SiteBlueprintTaxonomy
|
|
- Query for any `ContentIdeas` with old taxonomy foreign keys
|
|
3. If orphaned data found, create data migration to:
|
|
- Set taxonomy foreign keys to NULL
|
|
- Or migrate to ContentTaxonomy if mapping exists
|
|
|
|
**SQL Verification Queries:**
|
|
```sql
|
|
-- Check for blueprint tables (should return empty)
|
|
SELECT tablename FROM pg_tables
|
|
WHERE tablename LIKE '%blueprint%' OR tablename LIKE '%site_building%';
|
|
|
|
-- Check for foreign key constraints (should return empty)
|
|
SELECT constraint_name FROM information_schema.table_constraints
|
|
WHERE constraint_name LIKE '%blueprint%';
|
|
|
|
-- Check for orphaned taxonomy references
|
|
SELECT COUNT(*) FROM igny8_tasks WHERE taxonomy_id IS NOT NULL;
|
|
SELECT COUNT(*) FROM igny8_content_ideas WHERE taxonomy_id IS NOT NULL;
|
|
```
|
|
|
|
---
|
|
|
|
## SECTION 3: AUTOMATION PAGE UI IMPROVEMENTS
|
|
|
|
### Task 3.1: Stage Card Visual Redesign
|
|
|
|
**Current Issues:**
|
|
- Icons too large, taking excessive space
|
|
- Stage names not clearly associated with stage numbers
|
|
- Inconsistent visual hierarchy
|
|
|
|
**Required Changes:**
|
|
1. **Reduce Icon Size:**
|
|
- Current: Large colored square icons
|
|
- New: Smaller icons (32x32px instead of current size)
|
|
- Position: Top-left of card, not centered
|
|
|
|
2. **Restructure Stage Header:**
|
|
- Move stage name directly below "Stage N" text
|
|
- Format: "Stage 1" (bold) / "Keywords → Clusters" (regular weight, smaller font)
|
|
- Remove redundant text repetition
|
|
|
|
3. **Status Badge Positioning:**
|
|
- Move from separate row to same line as stage number
|
|
- Right-align badge next to stage number
|
|
|
|
**Layout Example (No Code):**
|
|
```
|
|
┌─────────────────────────────┐
|
|
│ [Icon] Stage 1 [Ready] │
|
|
│ Keywords → Clusters │
|
|
│ │
|
|
│ Total Queue: 7 │
|
|
│ Processed: 0 │
|
|
│ Remaining: 7 │
|
|
└─────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3.2: Add Progress Bars to Stage Cards
|
|
|
|
**Implementation Requirements:**
|
|
1. **Individual Stage Progress Bar:**
|
|
- Display below queue metrics
|
|
- Calculate: `(Processed / Total Queue) * 100`
|
|
- Visual: Colored bar matching stage color
|
|
- Show percentage label
|
|
|
|
2. **Overall Pipeline Progress Bar:**
|
|
- Large bar above all stage cards
|
|
- Calculate: `(Sum of Processed Items Across All Stages) / (Sum of Total Queue Across All Stages) * 100`
|
|
- Display current stage indicator: "Stage 4/7"
|
|
- Show estimated completion time
|
|
|
|
3. **Progress Bar States:**
|
|
- Empty (0%): Gray/outline only
|
|
- In Progress (1-99%): Animated gradient
|
|
- Complete (100%): Solid color, checkmark icon
|
|
|
|
---
|
|
|
|
### Task 3.3: Add Total Metrics Cards Above Pipeline
|
|
|
|
**New Component: MetricsSummary Cards**
|
|
|
|
**Cards to Display (Row above pipeline overview):**
|
|
1. **Keywords Card:**
|
|
- Total: Count from database
|
|
- Processed: Keywords with `status='mapped'`
|
|
- Pending: Keywords with `status='new'`
|
|
|
|
2. **Clusters Card:**
|
|
- Total: All clusters for site
|
|
- Processed: Clusters with ideas generated
|
|
- Pending: Clusters without ideas
|
|
|
|
3. **Ideas Card:**
|
|
- Total: All ideas for site
|
|
- Processed: Ideas converted to tasks (`status='in_progress'`)
|
|
- Pending: Ideas with `status='new'`
|
|
|
|
4. **Content Card:**
|
|
- Total: All content for site
|
|
- Processed: Content with `status='draft'` + all images generated
|
|
- Pending: Content without images or in generation
|
|
|
|
5. **Images Card:**
|
|
- Total: All image records for site content
|
|
- Processed: Images with `status='generated'`
|
|
- Pending: Images with `status='pending'`
|
|
|
|
**Card Layout:**
|
|
- Width: Equal distribution across row
|
|
- Display: Icon, Title, Total/Processed/Pending counts
|
|
- Color: Match stage colors for visual consistency
|
|
|
|
---
|
|
|
|
### Task 3.4: Pipeline Status Card Redesign
|
|
|
|
**Current:** Wide row with text "Pipeline Status - Ready to run | 22 items pending"
|
|
|
|
**New Design Requirements:**
|
|
1. **Convert to Centered Card:**
|
|
- Position: Above stage cards, below metrics summary
|
|
- Width: Narrower than full width, centered
|
|
- Style: Elevated/shadowed for emphasis
|
|
|
|
2. **Content Structure:**
|
|
- Large status indicator (icon + text)
|
|
- Prominent pending items count
|
|
- Quick action buttons (Run Now, Pause, Configure)
|
|
|
|
3. **Status Visual States:**
|
|
- Ready: Green pulse animation
|
|
- Running: Blue animated progress
|
|
- Paused: Yellow warning icon
|
|
- Failed: Red alert icon
|
|
|
|
---
|
|
|
|
### Task 3.5: Remove/Compact Header Elements
|
|
|
|
**Changes to Automation Page Header:**
|
|
|
|
1. **Remove "Pipeline Overview" Section:**
|
|
- Delete heading: "📊 Pipeline Overview"
|
|
- Delete subtitle: "Complete view of automation pipeline status and pending items"
|
|
- Reason: Redundant with visible pipeline cards
|
|
|
|
2. **Compact Schedule Panel:**
|
|
- Current: Large panel with heading, status row, action buttons
|
|
- New: Single compact row
|
|
- Layout: `[Status Badge] | [Schedule Text] | [Last Run] | [Estimated Credits] | [Configure Button] | [Run Now Button]`
|
|
- Remove empty space and excessive padding
|
|
|
|
---
|
|
|
|
### Task 3.6: AI Request Delays Implementation
|
|
|
|
**Problem:** Rapid sequential AI requests may hit rate limits or overload AI service.
|
|
|
|
**Required Changes:**
|
|
|
|
1. **Within-Stage Delay (between batches):**
|
|
- Location: `AutomationService` class methods
|
|
- Add delay after each batch completion before processing next batch
|
|
- Configurable: 3-5 seconds (default 3 seconds)
|
|
- Implementation point: After each AI function call completes in stage loop
|
|
|
|
2. **Between-Stage Delay:**
|
|
- Add delay after stage completion before triggering next stage
|
|
- Configurable: 5-10 seconds (default 5 seconds)
|
|
- Implementation point: After `_execute_stage()` returns before incrementing `current_stage`
|
|
|
|
3. **Configuration:**
|
|
- Add to `AutomationConfig` model: `within_stage_delay` (integer, seconds)
|
|
- Add to `AutomationConfig` model: `between_stage_delay` (integer, seconds)
|
|
- Expose in Configure modal for user adjustment
|
|
|
|
4. **Logging:**
|
|
- Log delay start: "Waiting 3 seconds before next batch..."
|
|
- Log delay end: "Delay complete, resuming processing"
|
|
|
|
---
|
|
|
|
## SECTION 4: AUTOMATION STAGE PROCESSING FIXES
|
|
|
|
### Task 4.1: Verify Stage Sequential Processing Logic
|
|
|
|
**Problem:** Pipeline not following strict sequential stage completion before moving to next stage.
|
|
|
|
**Analysis Required:**
|
|
1. Review `AutomationService.start_automation()` method
|
|
2. Verify stage loop waits for 100% completion before `current_stage += 1`
|
|
3. Check for any parallel execution logic that bypasses sequential flow
|
|
|
|
**Verification Steps:**
|
|
1. Each stage method (`run_stage_1()` to `run_stage_7()`) must return ONLY after ALL batches processed
|
|
2. Stage N+1 should NOT start if Stage N has `pending > 0`
|
|
3. Add explicit completion check before stage transition
|
|
|
|
**Required Fixes:**
|
|
- Add validation: Before starting Stage N, verify Stage N-1 has 0 pending items
|
|
- If pending items found, log warning and halt automation
|
|
- Return error status with message: "Stage N-1 incomplete, cannot proceed to Stage N"
|
|
|
|
---
|
|
|
|
### Task 4.2: Fix Batch Size Configuration Reading
|
|
|
|
**Problem:** Manual "Run Now" only processes 5 keywords instead of respecting configured batch size (20).
|
|
|
|
**Root Cause Analysis:**
|
|
1. Check if `run_stage_1()` reads from `AutomationConfig.stage_1_batch_size`
|
|
2. Verify query limit: `Keywords.objects.filter(...)[:batch_size]` uses correct variable
|
|
3. Confirm configuration loaded at automation start: `config = AutomationConfig.objects.get(site=self.site)`
|
|
|
|
**Expected Behavior:**
|
|
- If queue has 7 keywords and batch_size = 20: Process all 7 (not limit to 5)
|
|
- If queue has 47 keywords and batch_size = 20: Process 20, then next batch of 20, then final 7
|
|
- Batch size should be dynamic based on queue size: `min(queue_count, batch_size)`
|
|
|
|
**Fix Implementation:**
|
|
1. Ensure configuration loaded once at automation start
|
|
2. Pass batch_size to each stage method
|
|
3. Update query to use: `[:min(pending_count, batch_size)]`
|
|
4. Log batch selection: "Processing batch 1/3: 20 keywords"
|
|
|
|
---
|
|
|
|
### Task 4.3: Fix Stage 4 Processing Not Completing Full Queue
|
|
|
|
**Problem:** Stage 4 (Tasks → Content) not processing all tasks before moving to Stage 5.
|
|
|
|
**Investigation Steps:**
|
|
1. Check `run_stage_4()` implementation in `AutomationService`
|
|
2. Verify loop structure: Does it process tasks one-by-one until queue empty?
|
|
3. Look for premature loop exit conditions
|
|
|
|
**Expected Logic:**
|
|
```
|
|
While tasks with status='pending' exist:
|
|
1. Get next task
|
|
2. Call generate_content AI function
|
|
3. Wait for completion
|
|
4. Verify Content created
|
|
5. Check if more pending tasks exist
|
|
6. If yes, continue loop
|
|
7. If no, return stage complete
|
|
```
|
|
|
|
**Common Issues to Check:**
|
|
- Loop exits after first task instead of continuing
|
|
- No loop at all - only processes one batch
|
|
- Error handling breaks loop prematurely
|
|
|
|
---
|
|
|
|
### Task 4.4: Fix Stage 5 Not Triggering (Image Prompts Generation)
|
|
|
|
**Problem:** Automation exits after Stage 4 without generating image prompts.
|
|
|
|
**Analysis Required:**
|
|
1. Verify Stage 4 completion status set correctly
|
|
2. Check if Stage 5 start condition is met
|
|
3. Review database query in `run_stage_5()`:
|
|
- Query: Content with `status='draft'` AND `images_count=0`
|
|
- Verify Content records created in Stage 4 have correct status
|
|
|
|
**Potential Issues:**
|
|
1. Content created with status other than 'draft'
|
|
2. Images count annotation incorrect (should use `annotate(images_count=Count('images'))`)
|
|
3. Stage handover logic doesn't trigger Stage 5
|
|
|
|
**Fix Steps:**
|
|
1. Verify Content model save in Stage 4 sets `status='draft'`
|
|
2. Ensure Stage 5 query matches Content records from Stage 4
|
|
3. Add logging: "Stage 5: Found X content pieces without images"
|
|
4. If X > 0, process; if X = 0, skip stage gracefully
|
|
|
|
---
|
|
|
|
### Task 4.5: Add Stage Handover Validation
|
|
|
|
**New Logic Required Between Each Stage:**
|
|
|
|
1. **Pre-Stage Validation:**
|
|
- Before starting Stage N, check Stage N-1 completion:
|
|
- Query pending items for Stage N-1
|
|
- If pending > 0: Log error, halt automation
|
|
- If pending = 0: Log success, proceed
|
|
|
|
2. **Post-Stage Validation:**
|
|
- After Stage N completes, verify:
|
|
- All input items processed
|
|
- Expected output items created
|
|
- No errors in stage result
|
|
- Log validation result before moving to Stage N+1
|
|
|
|
3. **Validation Logging:**
|
|
- Stage 1 → Stage 2: "Verified: 0 keywords pending, 8 clusters created"
|
|
- Stage 2 → Stage 3: "Verified: 0 clusters pending, 56 ideas created"
|
|
- Stage 3 → Stage 4: "Verified: 0 ideas pending, 56 tasks created"
|
|
- Stage 4 → Stage 5: "Verified: 0 tasks pending, 56 content pieces created"
|
|
- Stage 5 → Stage 6: "Verified: 0 content without images, 224 prompts created"
|
|
- Stage 6 → Stage 7: "Verified: 0 pending images, 224 images generated"
|
|
|
|
---
|
|
|
|
### Task 4.6: Implement Dynamic Batch Size Logic
|
|
|
|
**Problem:** Fixed batch sizes don't adapt to actual queue sizes.
|
|
|
|
**Required Smart Batch Logic:**
|
|
|
|
1. **For Stages 1, 2, 3, 5:**
|
|
- If `queue_count <= batch_size`: Process ALL items in one batch
|
|
- If `queue_count > batch_size`: Split into batches
|
|
|
|
2. **For Stage 4 (Tasks → Content):**
|
|
- Always process one task at a time (sequential)
|
|
- Reason: Content generation is expensive, better control
|
|
- Batch size config for Stage 4 can be deprecated
|
|
|
|
3. **For Stage 6 (Images):**
|
|
- Process one image at a time (current behavior)
|
|
- Reason: Image generation has rate limits
|
|
|
|
**Configuration Update:**
|
|
- Stage 1-3, 5: Batch size applies
|
|
- Stage 4, 6: Batch size ignored (always 1)
|
|
- Update Configure modal to clarify batch size usage per stage
|
|
|
|
---
|
|
|
|
## SECTION 5: STAGE CARD LAYOUT RESTRUCTURE
|
|
|
|
### Task 5.1: Add Missing Stage 5 Card (Content → Image Prompts)
|
|
|
|
**Problem:** Current UI combines Stages 3 & 4 into one card, Stage 5 missing.
|
|
|
|
**Required Change:**
|
|
- Create separate card for Stage 5
|
|
- Display: "Content → Image Prompts"
|
|
- Queue metrics: Content without images (not total content)
|
|
- Show progress bar for prompt extraction
|
|
|
|
---
|
|
|
|
### Task 5.2: Separate Stages 3 & 4 into Individual Cards
|
|
|
|
**Current:** One card shows "Ideas → Tasks → Content" with nested metrics.
|
|
|
|
**New Structure:**
|
|
1. **Stage 3 Card:** "Ideas → Tasks"
|
|
- Total Queue: Ideas with `status='new'`
|
|
- Processed: Ideas converted to tasks
|
|
- Progress: Task creation count
|
|
|
|
2. **Stage 4 Card:** "Tasks → Content"
|
|
- Total Queue: Tasks with `status='pending'`
|
|
- Processed: Tasks with `status='completed'`
|
|
- Progress: Content generation count
|
|
|
|
---
|
|
|
|
### Task 5.3: Restructure Stage Card Rows
|
|
|
|
**New Layout Requirements:**
|
|
|
|
**Row 1 (Stages 1-4):**
|
|
- Stage 1: Keywords → Clusters
|
|
- Stage 2: Clusters → Ideas
|
|
- Stage 3: Ideas → Tasks
|
|
- Stage 4: Tasks → Content
|
|
|
|
**Row 2 (Stages 5-8):**
|
|
- Stage 5: Content → Image Prompts
|
|
- Stage 6: Image Prompts → Images
|
|
- Stage 7: Review Gate (with action buttons)
|
|
- Stage 8: Status Summary (new informational card)
|
|
|
|
**Responsive Behavior:**
|
|
- Desktop: 4 cards per row
|
|
- Tablet: 2 cards per row
|
|
- Mobile: 1 card per row (vertical stack)
|
|
|
|
---
|
|
|
|
### Task 5.4: Design Stage 7 Card (Review Gate)
|
|
|
|
**Unique Requirements:**
|
|
1. **Visual Distinction:**
|
|
- Different color scheme (amber/orange warning color)
|
|
- Icon: Stop sign or review icon
|
|
- Border: Dashed or highlighted
|
|
|
|
2. **Content:**
|
|
- Title: "Manual Review Gate"
|
|
- Status: "Automation Stops Here"
|
|
- Count: Number of content pieces ready for review
|
|
- Two buttons:
|
|
- "Go to Review Page" (navigates to Writer Content page filtered by status='review')
|
|
- "Publish Without Review" (disabled initially, placeholder for future feature)
|
|
|
|
3. **Button Behavior:**
|
|
- Review button: Active when count > 0
|
|
- Publish button: Disabled with tooltip "Coming soon"
|
|
|
|
---
|
|
|
|
### Task 5.5: Design Stage 8 Card (Status Summary)
|
|
|
|
**New Informational Card:**
|
|
|
|
**Purpose:** Display current automation run status without queue processing.
|
|
|
|
**Content:**
|
|
1. **Title:** "Current Status"
|
|
2. **Large Status Icon:** Based on run status
|
|
- Running: Animated spinner
|
|
- Completed: Checkmark
|
|
- Failed: X icon
|
|
- Paused: Pause icon
|
|
|
|
3. **Metrics Display:**
|
|
- Run ID
|
|
- Started at timestamp
|
|
- Current stage indicator
|
|
- Total credits used
|
|
- Completion percentage
|
|
|
|
4. **Visual Style:**
|
|
- No queue metrics
|
|
- No action buttons
|
|
- Read-only information display
|
|
- Distinct styling (different background color, no hover effects)
|
|
|
|
---
|
|
|
|
### Task 5.6: Adjust Card Width for New Layout
|
|
|
|
**Current:** Stage cards likely using equal width across full viewport.
|
|
|
|
**New Requirements:**
|
|
- Row 1 (4 cards): Each card 23% width (with 2% gap)
|
|
- Row 2 (4 cards): Same width distribution
|
|
- Stage 8 card: Can be wider or styled differently as summary card
|
|
|
|
**Implementation Considerations:**
|
|
- Use CSS Grid or Flexbox for responsive layout
|
|
- Ensure consistent spacing between cards
|
|
- Maintain card aspect ratio for visual balance
|
|
|
|
---
|
|
|
|
## SECTION 6: ADDITIONAL ENHANCEMENTS
|
|
|
|
### Task 6.1: Add Credit Usage Tracking per Stage
|
|
|
|
**Value Addition:** Real-time visibility into credit consumption.
|
|
|
|
**Implementation:**
|
|
1. Track credits used at end of each stage in `stage_N_result` JSON field
|
|
2. Display in stage card: "Credits Used: X"
|
|
3. Add running total in overall pipeline progress bar
|
|
4. Compare estimated vs actual credits used
|
|
5. Alert if actual exceeds estimated by >20%
|
|
|
|
---
|
|
|
|
### Task 6.2: Add Estimated Completion Time per Stage
|
|
|
|
**Value Addition:** Predictable automation runtime for planning.
|
|
|
|
**Implementation:**
|
|
1. Calculate average time per item based on historical runs
|
|
2. Estimate: `Remaining Items * Average Time per Item`
|
|
3. Display in stage card: "ETA: 45 minutes"
|
|
4. Update dynamically as items process
|
|
5. Store metrics in database for accuracy improvement over time
|
|
|
|
---
|
|
|
|
|
|
### Task 6.3: Add Error Rate Monitoring
|
|
|
|
**Value Addition:** Proactive issue detection.
|
|
|
|
**Implementation:**
|
|
1. Track error count per stage
|
|
2. Display: "Errors: X (Y%)"
|
|
3. Highlight stages with >5% error rate
|
|
4. Add "View Errors" button to navigate to error log
|
|
5. Set up alerts for error rate spikes
|
|
|
|
---
|
|
|
|
### Task 6.4: Add Stage Completion Percentage
|
|
|
|
**Value Addition:** Clear progress visualization.
|
|
|
|
**Implementation:**
|
|
1. Calculate: `(Completed Items / Total Items) * 100`
|
|
2. Display as progress bar in stage card
|
|
3. Color code:
|
|
- Green: >75%
|
|
- Yellow: 25-75%
|
|
- Red: <25%
|
|
4. Animate progress bar during active stages
|
|
5. Show exact percentage in text format
|
|
|
|
---
|
|
|
|
### Task 6.5: Add Stage Start/End Timestamps
|
|
|
|
**Value Addition:** Audit trail for automation runs.
|
|
|
|
**Implementation:**
|
|
1. Store start/end timestamps in `stage_N_result`
|
|
2. Display in stage card: "Started: 10:30 AM | Ended: 11:15 AM"
|
|
3
|