# Automation Module Reference

## Purpose
Document how the automation module orchestrates multi-stage AI pipelines, exposes API endpoints, enforces tenancy/credits, and manages runs, configs, and logging.

## Code Locations (exact paths)
- Models: `backend/igny8_core/business/automation/models.py`
- Services: `backend/igny8_core/business/automation/services/automation_service.py`, `automation_logger.py`
- Tasks (Celery): `backend/igny8_core/business/automation/tasks.py`
- API views and routing: `backend/igny8_core/business/automation/views.py`, `urls.py`
- Supporting AI functions: `backend/igny8_core/ai/functions/auto_cluster.py`, `generate_ideas.py`, `generate_content.py`, `generate_image_prompts.py`, image queue in `backend/igny8_core/ai/tasks.py`
- Tenancy/auth context: `backend/igny8_core/auth/middleware.py`, `backend/igny8_core/api/base.py`

## High-Level Responsibilities
- Maintain per-site automation configs (batch sizes, delays, schedule, enable flag) and track run state with detailed per-stage results.
- Provide APIs to configure, trigger, pause/resume/cancel, inspect, and log automation runs.
- Execute seven sequential stages that transform planner/writer data via AI and local operations, with credit checks and pause/cancel handling.
- Enforce tenant/site scoping on all automation resources and API operations.

## Detailed Behavior
- `AutomationConfig` stores enablement, frequency, scheduled time, batch sizes for stages 1–6, and within/between-stage delays. Config is created lazily per site.
- `AutomationRun` captures run metadata: trigger type (manual/scheduled), status (`running/paused/cancelled/completed/failed`), current stage, pause/cancel timestamps, per-stage JSON results, total credits used, and error message.
- `AutomationService` orchestrates the pipeline:
  - Locks per site via cache (`automation_lock_{site.id}`) to prevent concurrent runs.
  - Estimates credits before start and requires a 20% buffer over the estimate against `Account.credits`.
  - Creates `AutomationRun` with generated `run_id` and logs start via `AutomationLogger`.
  - Executes stages in order; each stage logs start/progress/complete, applies within/between-stage delays from config, and writes stage result JSON (counts, credits, timestamps, partial flags).
  - Pause/cancel checks occur inside loops; state is persisted so resumed runs continue from the recorded stage.
  - Stage credit usage is derived from AI task logs difference before/after the stage.
- API layer (`AutomationViewSet`):
  - `config`/`update_config` read/write `AutomationConfig` for a given `site_id` (scoped to the user’s account).
  - `run_now` triggers `AutomationService.start_automation` and enqueues Celery `run_automation_task`.
  - `current_run`, `history`, `logs`, `current_processing`, `estimate`, `pipeline_overview` expose run status, history, logs, credit estimates, and per-stage pending counts.
  - `pause`, `resume`, `cancel` endpoints update run status and enqueue resume tasks when needed.
- Celery tasks:
  - `check_scheduled_automations` scans enabled configs hourly and triggers runs when frequency/time matches and no recent run exists.
  - `run_automation_task` performs full pipeline execution.
  - `resume_automation_task`/`continue_automation_task` continue a paused run from its recorded stage.

## Data Structures / Models Involved (no code)
- `AutomationConfig`, `AutomationRun` (automation state).
- Planner models: `Keywords`, `Clusters`, `ContentIdeas`.
- Writer models: `Tasks`, `Content`, `Images`.
- AI task log (`AITaskLog`) for credit usage measurement.
- Tenancy entities: `Account`, `Site` (scoping every query).

## Execution Flow
- API call → DRF auth → tenant/site resolved → viewset method → `AutomationService` operations → Celery task (for long-running execution).
- Pipeline stages run in-process inside Celery workers, reading planner/writer data, invoking AI functions, updating models, logging progress, and writing stage results to `AutomationRun`.
- Completion (or failure) updates run status and releases the site lock.

## Cross-Module Interactions
- Planner/writer models supply inputs and receive outputs (clusters, ideas, tasks, content, images).
- AI engine executes clustering, idea generation, content generation, and image prompt generation; image rendering uses the AI image queue.
- Billing credits are checked against `Account.credits`; credit usage is inferred from AI task logs (deduction logic handled in billing services when those AI calls occur).
- Integration/publishing modules consume content/images produced downstream (outside automation).

## State Transitions
- Run status moves through `running` → (`paused`/`cancelled`/`failed`/`completed`); `current_stage` increments after each stage finishes; partial flags and timestamps mark mid-stage exits.
- Config changes take effect on the next run; pause/resume toggles update run timestamps.

## Error Handling
- Start blocks if a run is already active for the site or cache lock is held.
- Stage loops log and continue on per-batch/item errors; pause/cancel results are persisted mid-stage.
- Failures in Celery run mark `AutomationRun` as failed, store error message, timestamp completion, and release the lock.
- API endpoints return 400 for missing params or invalid state transitions, 404 for unknown runs, 500 on unexpected errors.

## Tenancy Rules
- All automation queries filter by `site` tied to the authenticated user’s `account`; config/run creation sets `account` and `site` explicitly.
- API endpoints fetch `Site` with `account=request.user.account`; automation locks are per site.
- No cross-tenant access; privileged role bypass is handled by DRF auth/permissions upstream.

## Billing Rules
- Start requires `Account.credits` ≥ 1.2× estimated credits; otherwise a 400 is returned.
- Credits actually deducted by AI tasks are reflected via AI task logs and billing services (outside this module); automation aggregates usage per stage in `AutomationRun`.

## Background Tasks / Schedulers
- Hourly `check_scheduled_automations` respects config frequency/time and last run; skips if a run is already active.
- Pipeline execution and resume steps run inside Celery tasks; within-stage sleeps apply delays from config.

## Key Design Considerations
- Single-run-per-site enforced via cache lock to prevent overlapping credit use or data contention.
- Pause/resume/cancel is cooperative, checked inside stage loops, with partial results persisted.
- Stage-by-stage logging and result JSON make pipeline progress observable and resumable.
- Configurable batch sizes and delays balance throughput and API/credit usage.

## How Developers Should Work With This Module
- Use `AutomationService.start_automation` for new runs; never bypass the cache lock or credit check.
- When extending stages, preserve pause/cancel checks, result recording, and credit delta calculation.
- Add new API actions through `AutomationViewSet` if they manipulate automation state; keep site/account scoping.
- For new schedulers, reuse the lock pattern and `AutomationConfig` fields, and update `next_run_at` appropriately.