384 lines
13 KiB
Markdown
384 lines
13 KiB
Markdown
# AI Automation Pipeline - Implementation Complete
|
|
|
|
## Overview
|
|
|
|
The IGNY8 AI Automation Pipeline is a fully automated content creation system that orchestrates existing AI functions into a 7-stage pipeline, transforming keywords into published content without manual intervention.
|
|
|
|
## Architecture
|
|
|
|
### Backend Components
|
|
|
|
#### 1. Models (`/backend/igny8_core/business/automation/models.py`)
|
|
|
|
**AutomationConfig**
|
|
- Per-site configuration for automation
|
|
- Fields: `is_enabled`, `frequency` (daily/weekly/monthly), `scheduled_time`, batch sizes for all 7 stages
|
|
- OneToOne relationship with Site model
|
|
|
|
**AutomationRun**
|
|
- Tracks execution of automation runs
|
|
- Fields: `run_id`, `status`, `current_stage`, `stage_1_result` through `stage_7_result` (JSON), `total_credits_used`
|
|
- Status choices: running, paused, completed, failed
|
|
|
|
#### 2. Services
|
|
|
|
**AutomationLogger** (`services/automation_logger.py`)
|
|
- File-based logging system
|
|
- Log structure: `logs/automation/{account_id}/{site_id}/{run_id}/`
|
|
- Files: `automation_run.log`, `stage_1.log` through `stage_7.log`
|
|
- Methods: `start_run()`, `log_stage_start()`, `log_stage_progress()`, `log_stage_complete()`, `log_stage_error()`
|
|
|
|
**AutomationService** (`services/automation_service.py`)
|
|
- Core orchestrator for automation pipeline
|
|
- Methods:
|
|
- `start_automation()` - Initialize new run with credit check
|
|
- `run_stage_1()` through `run_stage_7()` - Execute each pipeline stage
|
|
- `pause_automation()`, `resume_automation()` - Control run execution
|
|
- `estimate_credits()` - Pre-run credit estimation
|
|
- `from_run_id()` - Create service from existing run
|
|
|
|
#### 3. API Endpoints (`views.py`)
|
|
|
|
All endpoints at `/api/v1/automation/`:
|
|
|
|
- `GET /config/?site_id=123` - Get automation configuration
|
|
- `PUT /update_config/?site_id=123` - Update configuration
|
|
- `POST /run_now/?site_id=123` - Trigger immediate run
|
|
- `GET /current_run/?site_id=123` - Get active run status
|
|
- `POST /pause/?run_id=abc` - Pause running automation
|
|
- `POST /resume/?run_id=abc` - Resume paused automation
|
|
- `GET /history/?site_id=123` - Get past runs (last 20)
|
|
- `GET /logs/?run_id=abc&lines=100` - Get run logs
|
|
- `GET /estimate/?site_id=123` - Estimate credits needed
|
|
|
|
#### 4. Celery Tasks (`tasks.py`)
|
|
|
|
**check_scheduled_automations**
|
|
- Runs hourly via Celery Beat
|
|
- Checks AutomationConfig records for scheduled runs
|
|
- Triggers automation based on frequency and scheduled_time
|
|
|
|
**run_automation_task**
|
|
- Main background task that executes all 7 stages sequentially
|
|
- Called by `run_now` API endpoint or scheduled trigger
|
|
- Handles errors and updates AutomationRun status
|
|
|
|
**resume_automation_task**
|
|
- Resumes paused automation from `current_stage`
|
|
- Called by `resume` API endpoint
|
|
|
|
#### 5. Database Migration
|
|
|
|
Located at `/backend/igny8_core/business/automation/migrations/0001_initial.py`
|
|
|
|
Run with: `python manage.py migrate`
|
|
|
|
### Frontend Components
|
|
|
|
#### 1. Service (`/frontend/src/services/automationService.ts`)
|
|
|
|
TypeScript API client with methods matching backend endpoints:
|
|
- `getConfig()`, `updateConfig()`, `runNow()`, `getCurrentRun()`
|
|
- `pause()`, `resume()`, `getHistory()`, `getLogs()`, `estimate()`
|
|
|
|
#### 2. Pages
|
|
|
|
**AutomationPage** (`pages/Automation/AutomationPage.tsx`)
|
|
- Main dashboard at `/automation`
|
|
- Displays current run status, stage progress, activity log, history
|
|
- Real-time polling (5s interval when run is active)
|
|
- Controls: Run Now, Pause, Resume, Configure
|
|
|
|
#### 3. Components
|
|
|
|
**StageCard** (`components/Automation/StageCard.tsx`)
|
|
- Visual representation of each stage (1-7)
|
|
- Shows status: pending (⏳), active (🔄), complete (✅)
|
|
- Displays stage results (items processed, credits used, etc.)
|
|
|
|
**ActivityLog** (`components/Automation/ActivityLog.tsx`)
|
|
- Real-time log viewer with terminal-style display
|
|
- Auto-refreshes every 3 seconds
|
|
- Configurable line count (50, 100, 200, 500)
|
|
|
|
**ConfigModal** (`components/Automation/ConfigModal.tsx`)
|
|
- Modal for editing automation settings
|
|
- Fields: Enable/disable, frequency, scheduled time, batch sizes
|
|
- Form validation and save
|
|
|
|
**RunHistory** (`components/Automation/RunHistory.tsx`)
|
|
- Table of past automation runs
|
|
- Columns: run_id, status, trigger, started, completed, credits, stage
|
|
- Status badges with color coding
|
|
|
|
## 7-Stage Pipeline
|
|
|
|
### Stage 1: Keywords → Clusters (AI)
|
|
- **Query**: `Keywords` with `status='new'`, `cluster__isnull=True`, `disabled=False`
|
|
- **Batch Size**: Default 20 keywords
|
|
- **AI Function**: `AutoCluster().execute()`
|
|
- **Output**: Creates `Clusters` records
|
|
- **Credits**: ~1 per 5 keywords
|
|
|
|
### Stage 2: Clusters → Ideas (AI)
|
|
- **Query**: `Clusters` with `status='new'`, exclude those with existing ideas
|
|
- **Batch Size**: Default 1 cluster
|
|
- **AI Function**: `GenerateIdeas().execute()`
|
|
- **Output**: Creates `ContentIdeas` records
|
|
- **Credits**: ~2 per cluster
|
|
|
|
### Stage 3: Ideas → Tasks (Local Queue)
|
|
- **Query**: `ContentIdeas` with `status='new'`
|
|
- **Batch Size**: Default 20 ideas
|
|
- **Operation**: Local database creation (no AI)
|
|
- **Output**: Creates `Tasks` records with status='queued'
|
|
- **Credits**: 0 (local operation)
|
|
|
|
### Stage 4: Tasks → Content (AI)
|
|
- **Query**: `Tasks` with `status='queued'`, `content__isnull=True`
|
|
- **Batch Size**: Default 1 task
|
|
- **AI Function**: `GenerateContent().execute()`
|
|
- **Output**: Creates `Content` records with status='draft'
|
|
- **Credits**: ~5 per content (2500 words avg)
|
|
|
|
### Stage 5: Content → Image Prompts (AI)
|
|
- **Query**: `Content` with `status='draft'`, `images_count=0` (annotated)
|
|
- **Batch Size**: Default 1 content
|
|
- **AI Function**: `GenerateImagePromptsFunction().execute()`
|
|
- **Output**: Creates `Images` records with status='pending' (contains prompts)
|
|
- **Credits**: ~2 per content (4 prompts avg)
|
|
|
|
### Stage 6: Image Prompts → Generated Images (AI)
|
|
- **Query**: `Images` with `status='pending'`
|
|
- **Batch Size**: Default 1 image
|
|
- **AI Function**: `GenerateImages().execute()`
|
|
- **Output**: Updates `Images` to status='generated' with `image_url`
|
|
- **Side Effect**: Automatically sets `Content.status='review'` when all images complete (via `ai/tasks.py:723`)
|
|
- **Credits**: ~2 per image
|
|
|
|
### Stage 7: Manual Review Gate
|
|
- **Query**: `Content` with `status='review'`
|
|
- **Operation**: Count only, no processing
|
|
- **Output**: Returns list of content IDs ready for review
|
|
- **Credits**: 0
|
|
|
|
## Key Design Principles
|
|
|
|
### 1. NO Duplication of AI Function Logic
|
|
|
|
The automation system ONLY handles:
|
|
- Batch selection and sequencing
|
|
- Stage orchestration
|
|
- Credit estimation and checking
|
|
- Progress tracking and logging
|
|
- Scheduling and triggers
|
|
|
|
It does NOT handle:
|
|
- Credit deduction (done by `AIEngine.execute()` at line 395)
|
|
- Status updates (done within AI functions)
|
|
- Progress tracking (StepTracker emits events automatically)
|
|
|
|
### 2. Correct Image Model Understanding
|
|
|
|
- **NO separate ImagePrompts model** - this was a misunderstanding
|
|
- `Images` model serves dual purpose:
|
|
- `status='pending'` = has prompt, needs image URL
|
|
- `status='generated'` = has image_url
|
|
- Stage 5 creates Images records with prompts
|
|
- Stage 6 updates same records with URLs
|
|
|
|
### 3. Automatic Content Status Changes
|
|
|
|
- `Content.status` changes from 'draft' to 'review' automatically
|
|
- Happens in `ai/tasks.py:723` when all images complete
|
|
- Automation does NOT manually update this status
|
|
|
|
### 4. Distributed Locking
|
|
|
|
- Uses Django cache with `automation_lock_{site.id}` key
|
|
- 6-hour timeout to prevent deadlocks
|
|
- Released on completion, pause, or failure
|
|
|
|
## Configuration
|
|
|
|
### Schedule Configuration UI
|
|
|
|
Located at `/automation` page → [Configure] button
|
|
|
|
**Options:**
|
|
- **Enable/Disable**: Toggle automation on/off
|
|
- **Frequency**: Daily, Weekly (Mondays), Monthly (1st)
|
|
- **Scheduled Time**: Time of day to run (24-hour format)
|
|
- **Batch Sizes**: Per-stage item counts
|
|
|
|
**Defaults:**
|
|
- Stage 1: 20 keywords
|
|
- Stage 2: 1 cluster
|
|
- Stage 3: 20 ideas
|
|
- Stage 4: 1 task
|
|
- Stage 5: 1 content
|
|
- Stage 6: 1 image
|
|
|
|
### Credit Estimation
|
|
|
|
Before starting, system estimates:
|
|
- Stage 1: keywords_count / 5
|
|
- Stage 2: clusters_count * 2
|
|
- Stage 4: tasks_count * 5
|
|
- Stage 5: content_count * 2
|
|
- Stage 6: content_count * 8 (4 images * 2 credits avg)
|
|
|
|
Requires 20% buffer: `account.credits_balance >= estimated * 1.2`
|
|
|
|
## Deployment Checklist
|
|
|
|
### Backend
|
|
|
|
1. ✅ Models created in `business/automation/models.py`
|
|
2. ✅ Services created (`AutomationLogger`, `AutomationService`)
|
|
3. ✅ Views created (`AutomationViewSet`)
|
|
4. ✅ URLs registered in `igny8_core/urls.py`
|
|
5. ✅ Celery tasks created (`check_scheduled_automations`, `run_automation_task`, `resume_automation_task`)
|
|
6. ✅ Celery beat schedule updated in `celery.py`
|
|
7. ⏳ Migration created (needs to run: `python manage.py migrate`)
|
|
|
|
### Frontend
|
|
|
|
8. ✅ API service created (`services/automationService.ts`)
|
|
9. ✅ Main page created (`pages/Automation/AutomationPage.tsx`)
|
|
10. ✅ Components created (`StageCard`, `ActivityLog`, `ConfigModal`, `RunHistory`)
|
|
11. ⏳ Route registration (add to router: `/automation` → `AutomationPage`)
|
|
|
|
### Infrastructure
|
|
|
|
12. ⏳ Celery worker running (for background tasks)
|
|
13. ⏳ Celery beat running (for scheduled checks)
|
|
14. ⏳ Redis/cache backend configured (for distributed locks)
|
|
15. ⏳ Log directory writable: `/data/app/igny8/backend/logs/automation/`
|
|
|
|
## Usage
|
|
|
|
### Manual Trigger
|
|
|
|
1. Navigate to `/automation` page
|
|
2. Verify credit balance is sufficient (shows in header)
|
|
3. Click [Run Now] button
|
|
4. Monitor progress in real-time:
|
|
- Stage cards show current progress
|
|
- Activity log shows detailed logs
|
|
- Credits used updates live
|
|
|
|
### Scheduled Automation
|
|
|
|
1. Navigate to `/automation` page
|
|
2. Click [Configure] button
|
|
3. Enable automation
|
|
4. Set frequency and time
|
|
5. Configure batch sizes
|
|
6. Save configuration
|
|
7. Automation will run automatically at scheduled time
|
|
|
|
### Pause/Resume
|
|
|
|
- During active run, click [Pause] to halt execution
|
|
- Click [Resume] to continue from current stage
|
|
- Useful for credit management or issue investigation
|
|
|
|
### Viewing History
|
|
|
|
- Run History table shows last 20 runs
|
|
- Filter by status, date, trigger type
|
|
- Click run_id to view detailed logs
|
|
|
|
## Monitoring
|
|
|
|
### Log Files
|
|
|
|
Located at: `logs/automation/{account_id}/{site_id}/{run_id}/`
|
|
|
|
- `automation_run.log` - Main activity log
|
|
- `stage_1.log` through `stage_7.log` - Stage-specific logs
|
|
|
|
### Database Records
|
|
|
|
**AutomationRun** table tracks:
|
|
- Current status and stage
|
|
- Stage results (JSON)
|
|
- Credits used
|
|
- Error messages
|
|
- Timestamps
|
|
|
|
**AutomationConfig** table tracks:
|
|
- Last run timestamp
|
|
- Next scheduled run
|
|
- Configuration changes
|
|
|
|
## Troubleshooting
|
|
|
|
### Run stuck in "running" status
|
|
|
|
1. Check Celery worker logs: `docker logs <celery_container>`
|
|
2. Check for cache lock: `redis-cli GET automation_lock_<site_id>`
|
|
3. Manually release lock if needed: `redis-cli DEL automation_lock_<site_id>`
|
|
4. Update run status: `AutomationRun.objects.filter(run_id='...').update(status='failed')`
|
|
|
|
### Insufficient credits
|
|
|
|
1. Check estimate: GET `/api/v1/automation/estimate/?site_id=123`
|
|
2. Add credits via billing page
|
|
3. Retry run
|
|
|
|
### Stage failures
|
|
|
|
1. View logs: GET `/api/v1/automation/logs/?run_id=...`
|
|
2. Check `error_message` field in AutomationRun
|
|
3. Verify AI function is working: test individually via existing UI
|
|
4. Check credit balance mid-run
|
|
|
|
## Future Enhancements
|
|
|
|
1. Email notifications on completion/failure
|
|
2. Slack/webhook integrations
|
|
3. Per-stage retry logic
|
|
4. Partial run resumption after failure
|
|
5. Advanced scheduling (specific days, multiple times)
|
|
6. Content preview before Stage 7
|
|
7. Auto-publish to WordPress option
|
|
8. Credit usage analytics and forecasting
|
|
|
|
## File Locations Summary
|
|
|
|
```
|
|
backend/igny8_core/business/automation/
|
|
├── __init__.py
|
|
├── models.py # AutomationConfig, AutomationRun
|
|
├── views.py # AutomationViewSet (API endpoints)
|
|
├── tasks.py # Celery tasks
|
|
├── urls.py # URL routing
|
|
├── migrations/
|
|
│ ├── __init__.py
|
|
│ └── 0001_initial.py # Database schema
|
|
└── services/
|
|
├── __init__.py
|
|
├── automation_logger.py # File logging service
|
|
└── automation_service.py # Core orchestrator
|
|
|
|
frontend/src/
|
|
├── services/
|
|
│ └── automationService.ts # API client
|
|
├── pages/Automation/
|
|
│ └── AutomationPage.tsx # Main dashboard
|
|
└── components/Automation/
|
|
├── StageCard.tsx # Stage status display
|
|
├── ActivityLog.tsx # Log viewer
|
|
├── ConfigModal.tsx # Settings modal
|
|
└── RunHistory.tsx # Past runs table
|
|
```
|
|
|
|
## Credits
|
|
|
|
Implemented according to `automation-plan.md` with corrections for:
|
|
- Image model structure (no separate ImagePrompts)
|
|
- AI function internal logic (no duplication)
|
|
- Content status changes (automatic in background)
|