# Automation Pipeline Stages

## Purpose
Detail the seven pipeline stages executed by `AutomationService`, including inputs, queries, validations, delays, credit handling, and state recording.

## Code Locations (exact paths)
- Orchestration: `backend/igny8_core/business/automation/services/automation_service.py`
- Models: `backend/igny8_core/business/automation/models.py`
- AI functions: `backend/igny8_core/ai/functions/auto_cluster.py`, `generate_ideas.py`, `generate_content.py`, `generate_image_prompts.py`
- Image queue: `backend/igny8_core/ai/tasks.py` (`process_image_generation_queue`)
- Stage entrypoints: `backend/igny8_core/business/automation/tasks.py` (Celery `run_automation_task`, `resume_automation_task`)

## High-Level Responsibilities
- Execute a fixed seven-stage sequence that moves data from planner keywords through content with images and into manual review readiness.
- Enforce batch sizes/delays from `AutomationConfig`, support pause/cancel, and write per-stage results into `AutomationRun`.
- Track credit deltas per stage using AI task log counts.

## Detailed Behavior
Across all stages:
- Each stage logs start/progress/complete via `AutomationLogger`, respects `within_stage_delay` between batches/items, and `between_stage_delay` between stages.
- Pause/cancel is checked inside loops; on pause/cancel, the stage records partial counts, credits, elapsed time, and reason, then exits.
- Credits used per stage are computed from `AITaskLog` count delta relative to stage start.

### Stage 1: Keywords → Clusters (AI)
- Input query: `Keywords` where `site=current`, `status='new'`, `cluster__isnull=True`, `disabled=False`.
- Validation: `validate_minimum_keywords` requires at least 5 keywords; if not valid, stage is skipped with result noting skip reason and `current_stage` advances to 2.
- Processing: Batch size = `stage_1_batch_size` (capped to total). For each batch, calls `AIEngine.execute(AutoClusterFunction, payload={'ids': batch})`; waits on task ID; logs per-batch progress. Errors are logged and skipped; pipeline continues.
- Result: counts keywords processed, clusters created since run start, batches, credits used, time elapsed; sets `current_stage=2`.

### Stage 2: Clusters → Ideas (AI)
- Pre-check: warns if any `Keywords` still pending from Stage 1.
- Input query: `Clusters` where `site=current`, `status='new'`, `disabled=False`.
- Processing: Iterates clusters one-by-one; for each, calls `AIEngine.execute(GenerateIdeasFunction, payload={'cluster_id': cluster.id})`; waits on task ID; logs progress. Errors are logged and skipped.
- Result: counts clusters processed, ideas created since run start, credits used, time elapsed; sets `current_stage=3`.

### Stage 3: Ideas → Tasks (Local)
- Pre-check: warns if clusters remain without ideas.
- Input query: `ContentIdeas` where `site=current`, `status='new'`.
- Processing: Batched by `stage_3_batch_size`. For each idea, builds keyword string (M2M keywords or `target_keywords`) and creates `Tasks` with queued status, copying account/site/sector, cluster, content type/structure, and description. Idea status set to `queued`.
- Result: ideas processed, tasks created, batches, time elapsed (credits 0 because local); sets `current_stage=4`.

### Stage 4: Tasks → Content (AI)
- Pre-check: warns if `ContentIdeas` remain `new`.
- Input query: `Tasks` where `site=current`, `status='queued'`.
- Processing: Batched by `stage_4_batch_size`. Uses `GenerateContentFunction` via `AIEngine` per batch (payload contains task IDs). Waits on task IDs, logs progress, continues on errors. Tracks total words by summing generated content word_count.
- Result: tasks processed, content created count, total_words, credits used, time elapsed; sets `current_stage=5`.

### Stage 5: Content → Image Prompts (AI)
- Input query: `Content` where `site=current`, `status='draft'`, with zero images (annotated count=0).
- Processing: Batched by `stage_5_batch_size`. For each batch, calls `GenerateImagePromptsFunction` via `AIEngine` (payload content IDs). Waits on task IDs, logs progress; continues on errors.
- Result: content processed, prompts created (from AI task logs), credits used, time elapsed; sets `current_stage=6`.

### Stage 6: Image Prompts → Images (AI image queue)
- Input query: `Images` where `site=current`, `status='pending'`.
- Processing: Iterates pending images; for each, enqueues `process_image_generation_queue.delay(image_ids=[id], account_id, content_id)` when Celery is available, or calls directly in sync fallback. Waits on task IDs with continue-on-error to avoid blocking the stage. Logs progress per image; applies within-stage delay between images.
- Result: images processed, images generated (status `generated` since run start), content moved to `review`, credits used, time elapsed; sets `current_stage=7`.

### Stage 7: Manual Review Gate (Count-only)
- Input query: `Content` where `site=current`, `status='review'`.
- Processing: Counts review-ready content, logs IDs (truncated), marks run `status='completed'`, sets `completed_at`, and releases the site lock.
- Result: ready_for_review count and content IDs stored in `stage_7_result`.

## Execution Flow
- Celery task `run_automation_task` instantiates `AutomationService.from_run_id` and calls stages 1→7 sequentially.
- Stage transitions update `AutomationRun.current_stage`; between-stage delays applied via `between_stage_delay`.
- Resume path (`resume_automation_task`) starts from the recorded `current_stage` and continues through remaining stages.

## Cross-Module Interactions
- Planner: Stage 1/2 use `Keywords`/`Clusters`; Stage 3 converts `ContentIdeas` into `Tasks`.
- Writer: Stages 4–6 create `Content` and `Images` and move content toward review.
- AI engine and functions are invoked in Stages 1, 2, 4, 5; Stage 6 uses the AI image queue.
- Billing: Credits are consumed by AI calls; automation records deltas per stage from AI task logs.

## State Transitions
- `AutomationRun.status` moves to `completed` at Stage 7; can be set to `failed` on exceptions or `cancelled` via API; `paused` can be set mid-run and resumed.
- `current_stage` increments after each successful stage; partial stage results include a `partial` flag and stop reason.
- Domain models change status along the pipeline (`Keywords` → clusters, `Clusters` → ideas, `ContentIdeas` → queued/tasks, `Tasks` → completed/content, `Content` → draft/review, `Images` → generated).

## Error Handling
- Each stage logs errors and continues to next batch/item; pause/cancel checks short-circuit with partial results saved.
- Task wait helper tolerates Celery backend errors; can continue on error when flagged.
- Stage start may be skipped with explicit skip reason (e.g., insufficient keywords).

## Tenancy Rules
- All queries filter by `site` (and implicit account via tenancy bases); account/site set on created `Tasks` and inherited on `Images` and other records through model save hooks.
- Locks and runs are per site; API scoping requires the authenticated user’s account to own the site.

## Billing Rules
- Start requires sufficient credits (1.2× estimate). Credits used are inferred from AI task log counts per stage; actual deductions occur in AI/billing services invoked by the AI functions.

## Background Tasks / Schedulers
- Entire stage chain runs inside Celery workers; within-stage sleeps respect config delays; between-stage sleeps applied after each stage.

## Key Design Considerations
- Idempotent, resume-capable progression with partial state persisted in `AutomationRun`.
- Configurable batch sizes/delays mitigate rate limits and manage credit burn.
- Continue-on-error semantics prevent single failures from stopping the pipeline while still recording issues.

## How Developers Should Work With This Module
- When modifying stages, keep pause/cancel checks, stage result recording, and credit delta calculation.
- Add new AI stages by wiring through `AIEngine.execute` and the task wait helper; ensure queries are site-scoped and statuses updated.
- For new items types, add pending queries and status transitions consistent with existing patterns.
