453 lines
15 KiB
Markdown
453 lines
15 KiB
Markdown
# Automation Module
|
|
|
|
**Last Verified:** January 18, 2026
|
|
**Version:** 1.8.1
|
|
**Status:** ✅ Active
|
|
**Backend Path:** `backend/igny8_core/business/automation/`
|
|
**Frontend Path:** `frontend/src/pages/Automation/`
|
|
|
|
---
|
|
|
|
## Quick Reference
|
|
|
|
| What | File | Key Items |
|
|
|------|------|-----------|
|
|
| Models | `business/automation/models.py` | `AutomationConfig`, `AutomationRun`, `DefaultAutomationConfig` |
|
|
| Service | `business/automation/services/automation_service.py` | `AutomationService` |
|
|
| Logger | `business/automation/services/automation_logger.py` | `AutomationLogger` |
|
|
| Celery Tasks | `business/automation/tasks.py` | `run_automation_task`, `check_scheduled_automations`, `check_test_triggers` |
|
|
| Publishing Tasks | `igny8_core/tasks/publishing_scheduler.py` | Scheduled publishing |
|
|
| **Unified Settings** | `api/unified_settings.py` | **v1.8.0** Consolidated settings API |
|
|
| **Default Settings API** | `api/unified_settings.py` | **v1.8.1** `DefaultSettingsAPIView` for reset |
|
|
| Frontend Overview | `pages/Automation/AutomationOverview.tsx` | **v1.8.0** Run history dashboard |
|
|
| Frontend Run Detail | `pages/Automation/AutomationRunDetail.tsx` | **v1.8.0** Detailed run view |
|
|
| Frontend Manual Run | `pages/Automation/AutomationPage.tsx` | Manual run UI |
|
|
| **Site Settings** | `pages/Sites/AIAutomationSettings.tsx` | **v1.8.0** Unified settings UI |
|
|
| Progress Bar | `components/Automation/GlobalProgressBar.tsx` | Full pipeline progress |
|
|
| Processing Card | `components/Automation/CurrentProcessingCard.tsx` | Real-time progress |
|
|
| **Scheduling Doc** | `docs/40-WORKFLOWS/AUTOMATION-AND-PUBLISHING-SCHEDULING.md` | **v1.8.1** Complete scheduling guide |
|
|
|
|
---
|
|
|
|
## Purpose
|
|
|
|
The Automation module runs the complete 7-stage content pipeline automatically:
|
|
|
|
```
|
|
Keywords → Clusters → Ideas → Tasks → Content → Image Prompts → Images → Published
|
|
```
|
|
|
|
**Settings Location (v1.8.0+):** Site Settings → Automation tab
|
|
|
|
---
|
|
|
|
## Scheduling (v1.8.1)
|
|
|
|
| Task | Schedule | Purpose |
|
|
|------|----------|---------|
|
|
| `check_scheduled_automations` | Every hour at `:05` | Check if automations should run |
|
|
| `check_test_triggers` | Every minute | Check for admin test triggers |
|
|
|
|
**How it works:**
|
|
- Users select hour (12-hour AM/PM format), stored as `HH:00` (24-hour)
|
|
- Celery compares `scheduled_hour == current_hour`
|
|
- 23-hour block prevents re-runs within same day
|
|
|
|
**Test Mode (Admin):**
|
|
- `test_mode_enabled` + `test_trigger_at` fields on `AutomationConfig`
|
|
- Allows immediate triggering without waiting for schedule
|
|
- Bypasses 23-hour blocking
|
|
|
|
---
|
|
|
|
## 7-Stage Pipeline
|
|
|
|
| Stage | Name | AI Function | Credit Cost | Can Skip |
|
|
|-------|------|-------------|-------------|----------|
|
|
| 1 | Keywords → Clusters | `AutoClusterFunction` | Per batch | ✅ |
|
|
| 2 | Clusters → Ideas | `GenerateIdeasFunction` | Per idea | ✅ |
|
|
| 3 | Ideas → Tasks | None (local) | None | ✅ |
|
|
| 4 | Tasks → Content | `GenerateContentFunction` | Per 100 words | ✅ |
|
|
| 5 | Content → Image Prompts | `GenerateImagePromptsFunction` | Per prompt | ✅ |
|
|
| 6 | Image Prompts → Images | `process_image_generation_queue` | Per image | ✅ |
|
|
| 7 | Review → Published | Publishing Scheduler | None | ✅ |
|
|
|
|
**Note:** Stage 7 uses the Publishing Scheduler with `PublishingSettings` for auto-approval and scheduling.
|
|
|
|
---
|
|
|
|
## Data Models
|
|
|
|
### AutomationConfig
|
|
|
|
| Field | Type | Purpose |
|
|
|-------|------|---------|
|
|
| account | FK | Owner account |
|
|
| site | FK | Target site |
|
|
| enabled | Boolean | Enable/disable automation |
|
|
| frequency | CharField | hourly/daily/weekly |
|
|
| scheduled_time | TimeField | Time to run |
|
|
| stage_1_batch_size | Integer | Keywords per batch |
|
|
| stage_2_batch_size | Integer | Clusters per batch |
|
|
| stage_3_batch_size | Integer | Ideas per batch |
|
|
| stage_4_batch_size | Integer | Tasks per batch |
|
|
| stage_5_batch_size | Integer | Content per batch |
|
|
| stage_6_batch_size | Integer | Images per batch |
|
|
| **stage_1_enabled** | Boolean | **v1.8.0** Enable stage 1 |
|
|
| **stage_2_enabled** | Boolean | **v1.8.0** Enable stage 2 |
|
|
| **stage_3_enabled** | Boolean | **v1.8.0** Enable stage 3 |
|
|
| **stage_4_enabled** | Boolean | **v1.8.0** Enable stage 4 |
|
|
| **stage_5_enabled** | Boolean | **v1.8.0** Enable stage 5 |
|
|
| **stage_6_enabled** | Boolean | **v1.8.0** Enable stage 6 |
|
|
| **stage_7_enabled** | Boolean | **v1.8.0** Enable stage 7 |
|
|
| **stage_X_use_testing** | Boolean | **v1.8.1** Use testing model (AI stages 1,2,4,5,6) |
|
|
| **stage_X_budget_pct** | Integer | **v1.8.1** Credit budget % (AI stages) |
|
|
| **max_keywords_per_run** | Integer | **v1.8.0** Per-run limit stage 1 |
|
|
| **max_clusters_per_run** | Integer | **v1.8.0** Per-run limit stage 2 |
|
|
| **max_ideas_per_run** | Integer | **v1.8.0** Per-run limit stage 3 |
|
|
| **max_tasks_per_run** | Integer | **v1.8.0** Per-run limit stage 4 |
|
|
| **max_content_per_run** | Integer | **v1.8.0** Per-run limit stage 5 |
|
|
| **max_images_per_run** | Integer | **v1.8.0** Per-run limit stage 6 |
|
|
| within_stage_delay | Integer | Seconds between batches |
|
|
| between_stage_delay | Integer | Seconds between stages |
|
|
| last_run_at | DateTime | Last execution |
|
|
| next_run_at | DateTime | Next scheduled run |
|
|
| **test_mode_enabled** | Boolean | **v1.8.1** Enable test mode for admin |
|
|
| **test_trigger_at** | DateTime | **v1.8.1** When to trigger test run |
|
|
|
|
### DefaultAutomationConfig (v1.8.1)
|
|
|
|
Singleton model for centralized default settings. See [AUTOMATION-AND-PUBLISHING-SCHEDULING.md](../40-WORKFLOWS/AUTOMATION-AND-PUBLISHING-SCHEDULING.md) for full schema.
|
|
|
|
| Field | Type | Purpose |
|
|
|-------|------|---------|
|
|
| is_enabled | Boolean | Default: Enable scheduled automation |
|
|
| frequency | CharField | Default frequency (daily/weekly/monthly) |
|
|
| next_scheduled_hour | Integer | Next hour to assign (auto-increments) |
|
|
| stage_X_enabled | Boolean | Default: Enable each stage |
|
|
| stage_X_batch_size | Integer | Default: Batch size per stage |
|
|
| stage_X_use_testing | Boolean | Default: Use testing model |
|
|
| stage_X_budget_pct | Integer | Default: Credit budget % |
|
|
| max_X_per_run | Integer | Default: Per-run limits |
|
|
| auto_approval_enabled | Boolean | Default: Auto-approve content |
|
|
| auto_publish_enabled | Boolean | Default: Auto-publish content |
|
|
| publish_days | JSONField | Default: Days to publish |
|
|
| publish_time_slots | JSONField | Default: Time slots to publish |
|
|
|
|
### AutomationRun
|
|
|
|
| Field | Type | Purpose |
|
|
|-------|------|---------|
|
|
| config | FK | Parent config |
|
|
| trigger_type | CharField | manual/scheduled |
|
|
| status | CharField | running/paused/cancelled/completed/failed |
|
|
| current_stage | Integer | Current stage (1-7) |
|
|
| started_at | DateTime | Start time |
|
|
| paused_at | DateTime | Pause time (nullable) |
|
|
| resumed_at | DateTime | Resume time (nullable) |
|
|
| cancelled_at | DateTime | Cancel time (nullable) |
|
|
| completed_at | DateTime | Completion time (nullable) |
|
|
| total_credits_used | Decimal | Total credits consumed |
|
|
| **initial_snapshot** | JSON | **v1.3.0** Queue sizes at run start |
|
|
| stage_1_result | JSON | Stage 1 results |
|
|
| stage_2_result | JSON | Stage 2 results |
|
|
| stage_3_result | JSON | Stage 3 results |
|
|
| stage_4_result | JSON | Stage 4 results |
|
|
| stage_5_result | JSON | Stage 5 results |
|
|
| stage_6_result | JSON | Stage 6 results |
|
|
| stage_7_result | JSON | Stage 7 results |
|
|
| error_message | TextField | Error details (nullable) |
|
|
|
|
---
|
|
|
|
## API Endpoints
|
|
|
|
| Method | Path | Handler | Purpose |
|
|
|--------|------|---------|---------|
|
|
| GET | `/api/v1/automation/config/` | Get/create config | Get automation config |
|
|
| PUT | `/api/v1/automation/update_config/` | Update config | Update settings |
|
|
| POST | `/api/v1/automation/run_now/` | Start manual run | Start automation |
|
|
| GET | `/api/v1/automation/current_run/` | Get current run | Run status/progress |
|
|
| GET | `/api/v1/automation/pipeline_overview/` | Get pipeline | Stage status counts |
|
|
| GET | `/api/v1/automation/current_processing/` | Get processing | Live processing status |
|
|
| **GET** | `/api/v1/automation/run_progress/` | **v1.3.0** | Unified progress data |
|
|
| POST | `/api/v1/automation/pause/` | Pause run | Pause after current item |
|
|
| POST | `/api/v1/automation/resume/` | Resume run | Resume from saved stage |
|
|
| POST | `/api/v1/automation/cancel/` | Cancel run | Cancel after current item |
|
|
| GET | `/api/v1/automation/history/` | Get history | Last 20 runs |
|
|
| GET | `/api/v1/automation/logs/` | Get logs | Activity log for run |
|
|
| GET | `/api/v1/automation/estimate/` | Get estimate | Credit estimate |
|
|
| **GET** | `/api/v1/integration/settings/defaults/` | **v1.8.1** | Get default settings for reset |
|
|
|
|
**Query Parameters:** All require `?site_id=`, run-specific require `?run_id=`
|
|
|
|
### Default Settings Endpoint (v1.8.1)
|
|
|
|
Returns centralized defaults from `DefaultAutomationConfig`:
|
|
```json
|
|
{
|
|
"automation": { "enabled": false, "frequency": "daily", "time": "02:00" },
|
|
"stages": [
|
|
{ "number": 1, "enabled": true, "batch_size": 50, "per_run_limit": 0, "use_testing": false, "budget_pct": 15 },
|
|
...
|
|
],
|
|
"delays": { "within_stage": 3, "between_stage": 5 },
|
|
"publishing": {
|
|
"auto_approval_enabled": false,
|
|
"auto_publish_enabled": false,
|
|
"daily_publish_limit": 3,
|
|
"publish_days": ["mon", "tue", "wed", "thu", "fri"],
|
|
"time_slots": ["09:00", "14:00", "18:00"]
|
|
}
|
|
}
|
|
```
|
|
|
|
### run_progress Endpoint (v1.3.0)
|
|
|
|
Returns unified progress data for frontend:
|
|
```json
|
|
{
|
|
"run": { "run_id": "...", "status": "running", "current_stage": 3 },
|
|
"global_progress": { "total_items": 100, "completed_items": 45, "percentage": 45 },
|
|
"stages": [
|
|
{ "number": 1, "status": "completed", "input_count": 50, "processed_count": 50 },
|
|
...
|
|
],
|
|
"metrics": { "credits_used": 120, "duration_seconds": 3600 },
|
|
"initial_snapshot": { "stage_1_initial": 50, ... }
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Execution Flow
|
|
|
|
### Manual Run
|
|
|
|
1. User clicks "Run Now" on frontend
|
|
2. Frontend calls `POST /automation/run_now/?site_id=X`
|
|
3. Backend acquires cache lock `automation_lock_{site_id}`
|
|
4. **v1.3.0:** Captures initial snapshot with `_capture_initial_snapshot()`
|
|
5. Estimates credits required (1.2x buffer)
|
|
6. Validates balance >= estimate
|
|
7. Creates `AutomationRun` record
|
|
8. Enqueues `run_automation_task` Celery task
|
|
8. Returns run ID immediately
|
|
|
|
### Stage Execution
|
|
|
|
For each stage (1-7):
|
|
|
|
1. Check `_check_should_stop()` (paused/cancelled?)
|
|
2. Load items for processing
|
|
3. Process in batches (respecting batch_size)
|
|
4. For AI stages: Call AIEngine function
|
|
5. Wait `within_stage_delay` between batches
|
|
6. Save stage result JSON
|
|
7. Wait `between_stage_delay` before next stage
|
|
|
|
### Stage Result Fields
|
|
|
|
**Stage 1 (Clustering):**
|
|
```json
|
|
{
|
|
"keywords_processed": 150,
|
|
"clusters_created": 12,
|
|
"batches_run": 3,
|
|
"credits_used": 45,
|
|
"time_elapsed": 120,
|
|
"skipped": false,
|
|
"partial": false
|
|
}
|
|
```
|
|
|
|
**Stage 2 (Ideas):**
|
|
```json
|
|
{
|
|
"clusters_processed": 12,
|
|
"ideas_created": 36,
|
|
"batches_run": 2,
|
|
"credits_used": 72
|
|
}
|
|
```
|
|
|
|
**Stage 3 (Tasks):**
|
|
```json
|
|
{
|
|
"ideas_processed": 36,
|
|
"tasks_created": 36,
|
|
"batches_run": 4
|
|
}
|
|
```
|
|
|
|
**Stage 4 (Content):**
|
|
```json
|
|
{
|
|
"tasks_processed": 36,
|
|
"content_created": 36,
|
|
"total_words": 54000,
|
|
"batches_run": 6,
|
|
"credits_used": 540
|
|
}
|
|
```
|
|
|
|
**Stage 5 (Image Prompts):**
|
|
```json
|
|
{
|
|
"content_processed": 36,
|
|
"prompts_created": 180,
|
|
"batches_run": 4,
|
|
"credits_used": 36
|
|
}
|
|
```
|
|
|
|
**Stage 6 (Images):**
|
|
```json
|
|
{
|
|
"images_processed": 180,
|
|
"images_generated": 180,
|
|
"batches_run": 18
|
|
}
|
|
```
|
|
|
|
**Stage 7 (Review):**
|
|
```json
|
|
{
|
|
"ready_for_review": 36
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Settings Configuration (v1.8.0)
|
|
|
|
**Location:** Site Settings → Automation tab
|
|
**API:** `GET/PATCH /api/v1/integration/sites/{site_id}/unified-settings/`
|
|
|
|
> ⚠️ **v1.8.0 Change:** Settings are now consolidated in Site Settings. The previous standalone `/automation/settings` page has been removed.
|
|
|
|
### Settings UI Sections
|
|
|
|
1. **Schedule & Frequency Card**
|
|
- Enable/disable toggle
|
|
- Frequency (hourly/daily/weekly)
|
|
- Days of week selection
|
|
- Time slot selection
|
|
- Next run display
|
|
|
|
2. **Capacity Card**
|
|
- Total items per run (calculated)
|
|
- Stage-by-stage breakdown
|
|
|
|
3. **AI Configuration Card**
|
|
- Testing model selection (is_testing=true)
|
|
- Live model selection (is_testing=false)
|
|
- Image model selection
|
|
|
|
4. **Stage Configuration (Matrix)**
|
|
- Per-stage Enable/Disable toggle
|
|
- Per-stage batch size
|
|
- Per-stage max items per run (**new in v1.8.0**)
|
|
|
|
5. **Help Cards**
|
|
- Pipeline flow visualization
|
|
- Stage descriptions
|
|
|
|
---
|
|
|
|
## Scheduling
|
|
|
|
**Celery Beat Task:** `check_scheduled_automations`
|
|
**Frequency:** Hourly
|
|
|
|
**Logic:**
|
|
1. Find configs where `enabled=True`
|
|
2. Check if `next_run_at <= now`
|
|
3. Check if no active run exists
|
|
4. Start `run_automation_task` for eligible configs
|
|
5. Update `next_run_at` based on frequency
|
|
|
|
---
|
|
|
|
## Lock Mechanism
|
|
|
|
**Purpose:** Prevent concurrent runs for same site
|
|
|
|
**Key:** `automation_lock_{site_id}`
|
|
**Storage:** Redis cache
|
|
**Acquired:** On run start
|
|
**Released:** On completion/failure/cancel
|
|
|
|
---
|
|
|
|
## Credit Validation
|
|
|
|
Before starting:
|
|
1. Calculate estimated credits for all stages
|
|
2. Apply 1.2x safety buffer
|
|
3. Compare with account balance
|
|
4. Reject if balance < estimate
|
|
|
|
During execution:
|
|
- Each AI stage checks credits before processing
|
|
- Deductions happen after successful AI calls
|
|
- `total_credits_used` accumulates across stages
|
|
|
|
---
|
|
|
|
## Frontend Integration
|
|
|
|
### AutomationOverview (v1.8.0)
|
|
|
|
**Path:** `/automation/overview`
|
|
**Purpose:** Run history dashboard with:
|
|
- All automation runs with status
|
|
- Filter by status, date range
|
|
- Click to view detailed run information
|
|
|
|
### AutomationRunDetail (v1.8.0)
|
|
|
|
**Path:** `/automation/runs/:runId`
|
|
**Purpose:** Detailed view of individual run with:
|
|
- Stage-by-stage progress
|
|
- Items processed per stage
|
|
- Errors and logs
|
|
|
|
### AutomationPage
|
|
|
|
**Path:** `/automation`
|
|
**Purpose:** Manual run control with:
|
|
- **Pipeline Cards:** Stage-by-stage status with pending counts
|
|
- **Processing Card:** Live processing status during run
|
|
- **Control Buttons:** Run Now, Pause, Resume, Cancel
|
|
- **Activity Log:** Real-time log streaming
|
|
|
|
### Polling
|
|
|
|
- Every ~5s while run is running/paused
|
|
- Fetches: `current_run`, `pipeline_overview`, `current_processing`
|
|
- Lighter polling when idle
|
|
|
|
---
|
|
|
|
## Common Issues
|
|
|
|
| Issue | Cause | Fix |
|
|
|-------|-------|-----|
|
|
| "Already running" error | Lock exists from previous run | Wait or check if stuck |
|
|
| Insufficient credits | Balance < 1.2x estimate | Add credits |
|
|
| Stage skipped | No items to process | Check previous stages |
|
|
| Run stuck | Worker crashed | Clear lock, restart |
|
|
| Images not generating | Stage 5 didn't create prompts | Check stage 5 result |
|
|
|
|
---
|
|
|
|
## Changelog
|
|
|
|
| Version | Changes |
|
|
|---------|---------|
|
|
| v1.8.0 | Unified settings in Site Settings, per-run limits, skip stage functionality |
|
|
| v1.7.0 | AutomationOverview dashboard, AutomationRunDetail page |
|
|
| v1.3.2 | Stage 7 publishing scheduler integration |
|
|
| v1.3.0 | Progress tracking with initial snapshots |
|