salman/igny8
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
igny8/docs/10-MODULES/AUTOMATION.md
IGNY8 VPS (Salman) 6c69ca3feb Version 1.8.1
2026-01-18 15:18:25 +00:00

15 KiB
Raw Permalink Blame History

Automation Module

Last Verified: January 18, 2026
Version: 1.8.1
Status: ✅ Active
Backend Path: backend/igny8_core/business/automation/
Frontend Path: frontend/src/pages/Automation/

Quick Reference

What File Key Items
Models business/automation/models.py AutomationConfig, AutomationRun, DefaultAutomationConfig
Service business/automation/services/automation_service.py AutomationService
Logger business/automation/services/automation_logger.py AutomationLogger
Celery Tasks business/automation/tasks.py run_automation_task, check_scheduled_automations, check_test_triggers
Publishing Tasks igny8_core/tasks/publishing_scheduler.py Scheduled publishing
Unified Settings api/unified_settings.py v1.8.0 Consolidated settings API
Default Settings API api/unified_settings.py v1.8.1 DefaultSettingsAPIView for reset
Frontend Overview pages/Automation/AutomationOverview.tsx v1.8.0 Run history dashboard
Frontend Run Detail pages/Automation/AutomationRunDetail.tsx v1.8.0 Detailed run view
Frontend Manual Run pages/Automation/AutomationPage.tsx Manual run UI
Site Settings pages/Sites/AIAutomationSettings.tsx v1.8.0 Unified settings UI
Progress Bar components/Automation/GlobalProgressBar.tsx Full pipeline progress
Processing Card components/Automation/CurrentProcessingCard.tsx Real-time progress
Scheduling Doc docs/40-WORKFLOWS/AUTOMATION-AND-PUBLISHING-SCHEDULING.md v1.8.1 Complete scheduling guide

Purpose

The Automation module runs the complete 7-stage content pipeline automatically:

Keywords → Clusters → Ideas → Tasks → Content → Image Prompts → Images → Published

Settings Location (v1.8.0+): Site Settings → Automation tab

Scheduling (v1.8.1)

Task Schedule Purpose
check_scheduled_automations Every hour at :05 Check if automations should run
check_test_triggers Every minute Check for admin test triggers

How it works:

  • Users select hour (12-hour AM/PM format), stored as HH:00 (24-hour)
  • Celery compares scheduled_hour == current_hour
  • 23-hour block prevents re-runs within same day

Test Mode (Admin):

  • test_mode_enabled + test_trigger_at fields on AutomationConfig
  • Allows immediate triggering without waiting for schedule
  • Bypasses 23-hour blocking

7-Stage Pipeline

Stage Name AI Function Credit Cost Can Skip
1 Keywords → Clusters AutoClusterFunction Per batch ✅
2 Clusters → Ideas GenerateIdeasFunction Per idea ✅
3 Ideas → Tasks None (local) None ✅
4 Tasks → Content GenerateContentFunction Per 100 words ✅
5 Content → Image Prompts GenerateImagePromptsFunction Per prompt ✅
6 Image Prompts → Images process_image_generation_queue Per image ✅
7 Review → Published Publishing Scheduler None ✅

Note: Stage 7 uses the Publishing Scheduler with PublishingSettings for auto-approval and scheduling.

Data Models

AutomationConfig

Field Type Purpose
account FK Owner account
site FK Target site
enabled Boolean Enable/disable automation
frequency CharField hourly/daily/weekly
scheduled_time TimeField Time to run
stage_1_batch_size Integer Keywords per batch
stage_2_batch_size Integer Clusters per batch
stage_3_batch_size Integer Ideas per batch
stage_4_batch_size Integer Tasks per batch
stage_5_batch_size Integer Content per batch
stage_6_batch_size Integer Images per batch
stage_1_enabled Boolean v1.8.0 Enable stage 1
stage_2_enabled Boolean v1.8.0 Enable stage 2
stage_3_enabled Boolean v1.8.0 Enable stage 3
stage_4_enabled Boolean v1.8.0 Enable stage 4
stage_5_enabled Boolean v1.8.0 Enable stage 5
stage_6_enabled Boolean v1.8.0 Enable stage 6
stage_7_enabled Boolean v1.8.0 Enable stage 7
stage_X_use_testing Boolean v1.8.1 Use testing model (AI stages 1,2,4,5,6)
stage_X_budget_pct Integer v1.8.1 Credit budget % (AI stages)
max_keywords_per_run Integer v1.8.0 Per-run limit stage 1
max_clusters_per_run Integer v1.8.0 Per-run limit stage 2
max_ideas_per_run Integer v1.8.0 Per-run limit stage 3
max_tasks_per_run Integer v1.8.0 Per-run limit stage 4
max_content_per_run Integer v1.8.0 Per-run limit stage 5
max_images_per_run Integer v1.8.0 Per-run limit stage 6
within_stage_delay Integer Seconds between batches
between_stage_delay Integer Seconds between stages
last_run_at DateTime Last execution
next_run_at DateTime Next scheduled run
test_mode_enabled Boolean v1.8.1 Enable test mode for admin
test_trigger_at DateTime v1.8.1 When to trigger test run

DefaultAutomationConfig (v1.8.1)

Singleton model for centralized default settings. See AUTOMATION-AND-PUBLISHING-SCHEDULING.md for full schema.

Field Type Purpose
is_enabled Boolean Default: Enable scheduled automation
frequency CharField Default frequency (daily/weekly/monthly)
next_scheduled_hour Integer Next hour to assign (auto-increments)
stage_X_enabled Boolean Default: Enable each stage
stage_X_batch_size Integer Default: Batch size per stage
stage_X_use_testing Boolean Default: Use testing model
stage_X_budget_pct Integer Default: Credit budget %
max_X_per_run Integer Default: Per-run limits
auto_approval_enabled Boolean Default: Auto-approve content
auto_publish_enabled Boolean Default: Auto-publish content
publish_days JSONField Default: Days to publish
publish_time_slots JSONField Default: Time slots to publish

AutomationRun

Field Type Purpose
config FK Parent config
trigger_type CharField manual/scheduled
status CharField running/paused/cancelled/completed/failed
current_stage Integer Current stage (1-7)
started_at DateTime Start time
paused_at DateTime Pause time (nullable)
resumed_at DateTime Resume time (nullable)
cancelled_at DateTime Cancel time (nullable)
completed_at DateTime Completion time (nullable)
total_credits_used Decimal Total credits consumed
initial_snapshot JSON v1.3.0 Queue sizes at run start
stage_1_result JSON Stage 1 results
stage_2_result JSON Stage 2 results
stage_3_result JSON Stage 3 results
stage_4_result JSON Stage 4 results
stage_5_result JSON Stage 5 results
stage_6_result JSON Stage 6 results
stage_7_result JSON Stage 7 results
error_message TextField Error details (nullable)

API Endpoints

Method Path Handler Purpose
GET /api/v1/automation/config/ Get/create config Get automation config
PUT /api/v1/automation/update_config/ Update config Update settings
POST /api/v1/automation/run_now/ Start manual run Start automation
GET /api/v1/automation/current_run/ Get current run Run status/progress
GET /api/v1/automation/pipeline_overview/ Get pipeline Stage status counts
GET /api/v1/automation/current_processing/ Get processing Live processing status
GET /api/v1/automation/run_progress/ v1.3.0 Unified progress data
POST /api/v1/automation/pause/ Pause run Pause after current item
POST /api/v1/automation/resume/ Resume run Resume from saved stage
POST /api/v1/automation/cancel/ Cancel run Cancel after current item
GET /api/v1/automation/history/ Get history Last 20 runs
GET /api/v1/automation/logs/ Get logs Activity log for run
GET /api/v1/automation/estimate/ Get estimate Credit estimate
GET /api/v1/integration/settings/defaults/ v1.8.1 Get default settings for reset

Query Parameters: All require ?site_id=, run-specific require ?run_id=

Default Settings Endpoint (v1.8.1)

Returns centralized defaults from DefaultAutomationConfig:

{
  "automation": { "enabled": false, "frequency": "daily", "time": "02:00" },
  "stages": [
    { "number": 1, "enabled": true, "batch_size": 50, "per_run_limit": 0, "use_testing": false, "budget_pct": 15 },
    ...
  ],
  "delays": { "within_stage": 3, "between_stage": 5 },
  "publishing": {
    "auto_approval_enabled": false,
    "auto_publish_enabled": false,
    "daily_publish_limit": 3,
    "publish_days": ["mon", "tue", "wed", "thu", "fri"],
    "time_slots": ["09:00", "14:00", "18:00"]
  }
}

run_progress Endpoint (v1.3.0)

Returns unified progress data for frontend:

{
  "run": { "run_id": "...", "status": "running", "current_stage": 3 },
  "global_progress": { "total_items": 100, "completed_items": 45, "percentage": 45 },
  "stages": [
    { "number": 1, "status": "completed", "input_count": 50, "processed_count": 50 },
    ...
  ],
  "metrics": { "credits_used": 120, "duration_seconds": 3600 },
  "initial_snapshot": { "stage_1_initial": 50, ... }
}

Execution Flow

Manual Run

  1. User clicks "Run Now" on frontend
  2. Frontend calls POST /automation/run_now/?site_id=X
  3. Backend acquires cache lock automation_lock_{site_id}
  4. v1.3.0: Captures initial snapshot with _capture_initial_snapshot()
  5. Estimates credits required (1.2x buffer)
  6. Validates balance >= estimate
  7. Creates AutomationRun record
  8. Enqueues run_automation_task Celery task
  9. Returns run ID immediately

Stage Execution

For each stage (1-7):

  1. Check _check_should_stop() (paused/cancelled?)
  2. Load items for processing
  3. Process in batches (respecting batch_size)
  4. For AI stages: Call AIEngine function
  5. Wait within_stage_delay between batches
  6. Save stage result JSON
  7. Wait between_stage_delay before next stage

Stage Result Fields

Stage 1 (Clustering):

{
  "keywords_processed": 150,
  "clusters_created": 12,
  "batches_run": 3,
  "credits_used": 45,
  "time_elapsed": 120,
  "skipped": false,
  "partial": false
}

Stage 2 (Ideas):

{
  "clusters_processed": 12,
  "ideas_created": 36,
  "batches_run": 2,
  "credits_used": 72
}

Stage 3 (Tasks):

{
  "ideas_processed": 36,
  "tasks_created": 36,
  "batches_run": 4
}

Stage 4 (Content):

{
  "tasks_processed": 36,
  "content_created": 36,
  "total_words": 54000,
  "batches_run": 6,
  "credits_used": 540
}

Stage 5 (Image Prompts):

{
  "content_processed": 36,
  "prompts_created": 180,
  "batches_run": 4,
  "credits_used": 36
}

Stage 6 (Images):

{
  "images_processed": 180,
  "images_generated": 180,
  "batches_run": 18
}

Stage 7 (Review):

{
  "ready_for_review": 36
}

Settings Configuration (v1.8.0)

Location: Site Settings → Automation tab
API: GET/PATCH /api/v1/integration/sites/{site_id}/unified-settings/

⚠️ v1.8.0 Change: Settings are now consolidated in Site Settings. The previous standalone /automation/settings page has been removed.

Settings UI Sections

  1. Schedule & Frequency Card

    • Enable/disable toggle
    • Frequency (hourly/daily/weekly)
    • Days of week selection
    • Time slot selection
    • Next run display

  2. Capacity Card

    • Total items per run (calculated)
    • Stage-by-stage breakdown

  3. AI Configuration Card

    • Testing model selection (is_testing=true)
    • Live model selection (is_testing=false)
    • Image model selection

  4. Stage Configuration (Matrix)

    • Per-stage Enable/Disable toggle
    • Per-stage batch size
    • Per-stage max items per run (new in v1.8.0)

  5. Help Cards

    • Pipeline flow visualization
    • Stage descriptions

Scheduling

Celery Beat Task: check_scheduled_automations
Frequency: Hourly

Logic:

  1. Find configs where enabled=True
  2. Check if next_run_at <= now
  3. Check if no active run exists
  4. Start run_automation_task for eligible configs
  5. Update next_run_at based on frequency

Lock Mechanism

Purpose: Prevent concurrent runs for same site

Key: automation_lock_{site_id}
Storage: Redis cache
Acquired: On run start
Released: On completion/failure/cancel

Credit Validation

Before starting:

  1. Calculate estimated credits for all stages
  2. Apply 1.2x safety buffer
  3. Compare with account balance
  4. Reject if balance < estimate

During execution:

  • Each AI stage checks credits before processing
  • Deductions happen after successful AI calls
  • total_credits_used accumulates across stages

Frontend Integration

AutomationOverview (v1.8.0)

Path: /automation/overview
Purpose: Run history dashboard with:

  • All automation runs with status
  • Filter by status, date range
  • Click to view detailed run information

AutomationRunDetail (v1.8.0)

Path: /automation/runs/:runId
Purpose: Detailed view of individual run with:

  • Stage-by-stage progress
  • Items processed per stage
  • Errors and logs

AutomationPage

Path: /automation
Purpose: Manual run control with:

  • Pipeline Cards: Stage-by-stage status with pending counts
  • Processing Card: Live processing status during run
  • Control Buttons: Run Now, Pause, Resume, Cancel
  • Activity Log: Real-time log streaming

Polling

  • Every ~5s while run is running/paused
  • Fetches: current_run, pipeline_overview, current_processing
  • Lighter polling when idle

Common Issues

Issue Cause Fix
"Already running" error Lock exists from previous run Wait or check if stuck
Insufficient credits Balance < 1.2x estimate Add credits
Stage skipped No items to process Check previous stages
Run stuck Worker crashed Clear lock, restart
Images not generating Stage 5 didn't create prompts Check stage 5 result

Changelog

Version Changes
v1.8.0 Unified settings in Site Settings, per-run limits, skip stage functionality
v1.7.0 AutomationOverview dashboard, AutomationRunDetail page
v1.3.2 Stage 7 publishing scheduler integration
v1.3.0 Progress tracking with initial snapshots
Reference in New Issue View Git Blame Copy Permalink