92 lines
7.0 KiB
Markdown
92 lines
7.0 KiB
Markdown
# Automation Module Reference
|
||
|
||
## Purpose
|
||
Document how the automation module orchestrates multi-stage AI pipelines, exposes API endpoints, enforces tenancy/credits, and manages runs, configs, and logging.
|
||
|
||
## Code Locations (exact paths)
|
||
- Models: `backend/igny8_core/business/automation/models.py`
|
||
- Services: `backend/igny8_core/business/automation/services/automation_service.py`, `automation_logger.py`
|
||
- Tasks (Celery): `backend/igny8_core/business/automation/tasks.py`
|
||
- API views and routing: `backend/igny8_core/business/automation/views.py`, `urls.py`
|
||
- Supporting AI functions: `backend/igny8_core/ai/functions/auto_cluster.py`, `generate_ideas.py`, `generate_content.py`, `generate_image_prompts.py`, image queue in `backend/igny8_core/ai/tasks.py`
|
||
- Tenancy/auth context: `backend/igny8_core/auth/middleware.py`, `backend/igny8_core/api/base.py`
|
||
|
||
## High-Level Responsibilities
|
||
- Maintain per-site automation configs (batch sizes, delays, schedule, enable flag) and track run state with detailed per-stage results.
|
||
- Provide APIs to configure, trigger, pause/resume/cancel, inspect, and log automation runs.
|
||
- Execute seven sequential stages that transform planner/writer data via AI and local operations, with credit checks and pause/cancel handling.
|
||
- Enforce tenant/site scoping on all automation resources and API operations.
|
||
|
||
## Detailed Behavior
|
||
- `AutomationConfig` stores enablement, frequency, scheduled time, batch sizes for stages 1–6, and within/between-stage delays. Config is created lazily per site.
|
||
- `AutomationRun` captures run metadata: trigger type (manual/scheduled), status (`running/paused/cancelled/completed/failed`), current stage, pause/cancel timestamps, per-stage JSON results, total credits used, and error message.
|
||
- `AutomationService` orchestrates the pipeline:
|
||
- Locks per site via cache (`automation_lock_{site.id}`) to prevent concurrent runs.
|
||
- Estimates credits before start and requires a 20% buffer over the estimate against `Account.credits`.
|
||
- Creates `AutomationRun` with generated `run_id` and logs start via `AutomationLogger`.
|
||
- Executes stages in order; each stage logs start/progress/complete, applies within/between-stage delays from config, and writes stage result JSON (counts, credits, timestamps, partial flags).
|
||
- Pause/cancel checks occur inside loops; state is persisted so resumed runs continue from the recorded stage.
|
||
- Stage credit usage is derived from AI task logs difference before/after the stage.
|
||
- API layer (`AutomationViewSet`):
|
||
- `config`/`update_config` read/write `AutomationConfig` for a given `site_id` (scoped to the user’s account).
|
||
- `run_now` triggers `AutomationService.start_automation` and enqueues Celery `run_automation_task`.
|
||
- `current_run`, `history`, `logs`, `current_processing`, `estimate`, `pipeline_overview` expose run status, history, logs, credit estimates, and per-stage pending counts.
|
||
- `pause`, `resume`, `cancel` endpoints update run status and enqueue resume tasks when needed.
|
||
- Celery tasks:
|
||
- `check_scheduled_automations` scans enabled configs hourly and triggers runs when frequency/time matches and no recent run exists.
|
||
- `run_automation_task` performs full pipeline execution.
|
||
- `resume_automation_task`/`continue_automation_task` continue a paused run from its recorded stage.
|
||
|
||
## Data Structures / Models Involved (no code)
|
||
- `AutomationConfig`, `AutomationRun` (automation state).
|
||
- Planner models: `Keywords`, `Clusters`, `ContentIdeas`.
|
||
- Writer models: `Tasks`, `Content`, `Images`.
|
||
- AI task log (`AITaskLog`) for credit usage measurement.
|
||
- Tenancy entities: `Account`, `Site` (scoping every query).
|
||
|
||
## Execution Flow
|
||
- API call → DRF auth → tenant/site resolved → viewset method → `AutomationService` operations → Celery task (for long-running execution).
|
||
- Pipeline stages run in-process inside Celery workers, reading planner/writer data, invoking AI functions, updating models, logging progress, and writing stage results to `AutomationRun`.
|
||
- Completion (or failure) updates run status and releases the site lock.
|
||
|
||
## Cross-Module Interactions
|
||
- Planner/writer models supply inputs and receive outputs (clusters, ideas, tasks, content, images).
|
||
- AI engine executes clustering, idea generation, content generation, and image prompt generation; image rendering uses the AI image queue.
|
||
- Billing credits are checked against `Account.credits`; credit usage is inferred from AI task logs (deduction logic handled in billing services when those AI calls occur).
|
||
- Integration/publishing modules consume content/images produced downstream (outside automation).
|
||
|
||
## State Transitions
|
||
- Run status moves through `running` → (`paused`/`cancelled`/`failed`/`completed`); `current_stage` increments after each stage finishes; partial flags and timestamps mark mid-stage exits.
|
||
- Config changes take effect on the next run; pause/resume toggles update run timestamps.
|
||
|
||
## Error Handling
|
||
- Start blocks if a run is already active for the site or cache lock is held.
|
||
- Stage loops log and continue on per-batch/item errors; pause/cancel results are persisted mid-stage.
|
||
- Failures in Celery run mark `AutomationRun` as failed, store error message, timestamp completion, and release the lock.
|
||
- API endpoints return 400 for missing params or invalid state transitions, 404 for unknown runs, 500 on unexpected errors.
|
||
|
||
## Tenancy Rules
|
||
- All automation queries filter by `site` tied to the authenticated user’s `account`; config/run creation sets `account` and `site` explicitly.
|
||
- API endpoints fetch `Site` with `account=request.user.account`; automation locks are per site.
|
||
- No cross-tenant access; privileged role bypass is handled by DRF auth/permissions upstream.
|
||
|
||
## Billing Rules
|
||
- Start requires `Account.credits` ≥ 1.2× estimated credits; otherwise a 400 is returned.
|
||
- Credits actually deducted by AI tasks are reflected via AI task logs and billing services (outside this module); automation aggregates usage per stage in `AutomationRun`.
|
||
|
||
## Background Tasks / Schedulers
|
||
- Hourly `check_scheduled_automations` respects config frequency/time and last run; skips if a run is already active.
|
||
- Pipeline execution and resume steps run inside Celery tasks; within-stage sleeps apply delays from config.
|
||
|
||
## Key Design Considerations
|
||
- Single-run-per-site enforced via cache lock to prevent overlapping credit use or data contention.
|
||
- Pause/resume/cancel is cooperative, checked inside stage loops, with partial results persisted.
|
||
- Stage-by-stage logging and result JSON make pipeline progress observable and resumable.
|
||
- Configurable batch sizes and delays balance throughput and API/credit usage.
|
||
|
||
## How Developers Should Work With This Module
|
||
- Use `AutomationService.start_automation` for new runs; never bypass the cache lock or credit check.
|
||
- When extending stages, preserve pause/cancel checks, result recording, and credit delta calculation.
|
||
- Add new API actions through `AutomationViewSet` if they manipulate automation state; keep site/account scoping.
|
||
- For new schedulers, reuse the lock pattern and `AutomationConfig` fields, and update `next_run_at` appropriately.
|