igny8/master-docs/10-backend/automation/AUTOMATION-MODULE-REFERENCE.md

Automation Module Reference

Purpose

Document how the automation module orchestrates multi-stage AI pipelines, exposes API endpoints, enforces tenancy/credits, and manages runs, configs, and logging.

Code Locations (exact paths)

• Models: backend/igny8_core/business/automation/models.py
• Services: backend/igny8_core/business/automation/services/automation_service.py, automation_logger.py
• Tasks (Celery): backend/igny8_core/business/automation/tasks.py
• API views and routing: backend/igny8_core/business/automation/views.py, urls.py
• Supporting AI functions: backend/igny8_core/ai/functions/auto_cluster.py, generate_ideas.py, generate_content.py, generate_image_prompts.py, image queue in backend/igny8_core/ai/tasks.py
• Tenancy/auth context: backend/igny8_core/auth/middleware.py, backend/igny8_core/api/base.py

High-Level Responsibilities

• Maintain per-site automation configs (batch sizes, delays, schedule, enable flag) and track run state with detailed per-stage results.
• Provide APIs to configure, trigger, pause/resume/cancel, inspect, and log automation runs.
• Execute seven sequential stages that transform planner/writer data via AI and local operations, with credit checks and pause/cancel handling.
• Enforce tenant/site scoping on all automation resources and API operations.

Detailed Behavior

• AutomationConfig stores enablement, frequency, scheduled time, batch sizes for stages 1–6, and within/between-stage delays. Config is created lazily per site.
• AutomationRun captures run metadata: trigger type (manual/scheduled), status (running/paused/cancelled/completed/failed), current stage, pause/cancel timestamps, per-stage JSON results, total credits used, and error message.
• AutomationService orchestrates the pipeline:
  • Locks per site via cache (automation_lock_{site.id}) to prevent concurrent runs.
  • Estimates credits before start and requires a 20% buffer over the estimate against Account.credits.
  • Creates AutomationRun with generated run_id and logs start via AutomationLogger.
  • Executes stages in order; each stage logs start/progress/complete, applies within/between-stage delays from config, and writes stage result JSON (counts, credits, timestamps, partial flags).
  • Pause/cancel checks occur inside loops; state is persisted so resumed runs continue from the recorded stage.
  • Stage credit usage is derived from AI task logs difference before/after the stage.
• API layer (AutomationViewSet):
  • config/update_config read/write AutomationConfig for a given site_id (scoped to the user’s account).
  • run_now triggers AutomationService.start_automation and enqueues Celery run_automation_task.
  • current_run, history, logs, current_processing, estimate, pipeline_overview expose run status, history, logs, credit estimates, and per-stage pending counts.
  • pause, resume, cancel endpoints update run status and enqueue resume tasks when needed.
• Celery tasks:
  • check_scheduled_automations scans enabled configs hourly and triggers runs when frequency/time matches and no recent run exists.
  • run_automation_task performs full pipeline execution.
  • resume_automation_task/continue_automation_task continue a paused run from its recorded stage.

Data Structures / Models Involved (no code)

• AutomationConfig, AutomationRun (automation state).
• Planner models: Keywords, Clusters, ContentIdeas.
• Writer models: Tasks, Content, Images.
• AI task log (AITaskLog) for credit usage measurement.
• Tenancy entities: Account, Site (scoping every query).

Execution Flow

• API call → DRF auth → tenant/site resolved → viewset method → AutomationService operations → Celery task (for long-running execution).
• Pipeline stages run in-process inside Celery workers, reading planner/writer data, invoking AI functions, updating models, logging progress, and writing stage results to AutomationRun.
• Completion (or failure) updates run status and releases the site lock.

Cross-Module Interactions

• Planner/writer models supply inputs and receive outputs (clusters, ideas, tasks, content, images).
• AI engine executes clustering, idea generation, content generation, and image prompt generation; image rendering uses the AI image queue.
• Billing credits are checked against Account.credits; credit usage is inferred from AI task logs (deduction logic handled in billing services when those AI calls occur).
• Integration/publishing modules consume content/images produced downstream (outside automation).

State Transitions

• Run status moves through running → (paused/cancelled/failed/completed); current_stage increments after each stage finishes; partial flags and timestamps mark mid-stage exits.
• Config changes take effect on the next run; pause/resume toggles update run timestamps.

Error Handling

• Start blocks if a run is already active for the site or cache lock is held.
• Stage loops log and continue on per-batch/item errors; pause/cancel results are persisted mid-stage.
• Failures in Celery run mark AutomationRun as failed, store error message, timestamp completion, and release the lock.
• API endpoints return 400 for missing params or invalid state transitions, 404 for unknown runs, 500 on unexpected errors.

Tenancy Rules

• All automation queries filter by site tied to the authenticated user’s account; config/run creation sets account and site explicitly.
• API endpoints fetch Site with account=request.user.account; automation locks are per site.
• No cross-tenant access; privileged role bypass is handled by DRF auth/permissions upstream.

Billing Rules

• Start requires Account.credits ≥ 1.2× estimated credits; otherwise a 400 is returned.
• Credits actually deducted by AI tasks are reflected via AI task logs and billing services (outside this module); automation aggregates usage per stage in AutomationRun.

Background Tasks / Schedulers

• Hourly check_scheduled_automations respects config frequency/time and last run; skips if a run is already active.
• Pipeline execution and resume steps run inside Celery tasks; within-stage sleeps apply delays from config.

Key Design Considerations

• Single-run-per-site enforced via cache lock to prevent overlapping credit use or data contention.
• Pause/resume/cancel is cooperative, checked inside stage loops, with partial results persisted.
• Stage-by-stage logging and result JSON make pipeline progress observable and resumable.
• Configurable batch sizes and delays balance throughput and API/credit usage.

How Developers Should Work With This Module

• Use AutomationService.start_automation for new runs; never bypass the cache lock or credit check.
• When extending stages, preserve pause/cancel checks, result recording, and credit delta calculation.
• Add new API actions through AutomationViewSet if they manipulate automation state; keep site/account scoping.
• For new schedulers, reuse the lock pattern and AutomationConfig fields, and update next_run_at appropriately.