76 lines
4.4 KiB
Markdown
76 lines
4.4 KiB
Markdown
# Automation Scheduler
|
||
|
||
## Purpose
|
||
Describe how scheduled runs are detected, triggered, and resumed using Celery tasks and automation configs.
|
||
|
||
## Code Locations (exact paths)
|
||
- Celery tasks: `backend/igny8_core/business/automation/tasks.py`
|
||
- Models: `backend/igny8_core/business/automation/models.py`
|
||
- Service invoked: `backend/igny8_core/business/automation/services/automation_service.py`
|
||
|
||
## High-Level Responsibilities
|
||
- Periodically scan enabled automation configs to start scheduled runs.
|
||
- Prevent overlapping runs per site via cache locks and active run checks.
|
||
- Resume paused runs from their recorded stage.
|
||
|
||
## Detailed Behavior
|
||
- `check_scheduled_automations` (Celery, hourly):
|
||
- Iterates `AutomationConfig` with `is_enabled=True`.
|
||
- Frequency rules:
|
||
- `daily`: run when current hour matches `scheduled_time.hour`.
|
||
- `weekly`: run Mondays at the scheduled hour.
|
||
- `monthly`: run on the 1st of the month at the scheduled hour.
|
||
- Skips if `last_run_at` is within ~23 hours or if an `AutomationRun` with `status='running'` exists for the site.
|
||
- On trigger: instantiates `AutomationService(account, site)`, calls `start_automation(trigger_type='scheduled')`, updates `last_run_at` and `next_run_at` (via `_calculate_next_run`), saves config, and enqueues `run_automation_task.delay(run_id)`.
|
||
- Exceptions are logged per site; lock release is handled by the service on failure paths.
|
||
- `run_automation_task`:
|
||
- Loads service via `from_run_id`, runs stages 1–7 sequentially.
|
||
- On exception: marks run failed, records error/completed_at, and deletes site lock.
|
||
- `resume_automation_task` / alias `continue_automation_task`:
|
||
- Loads service via `from_run_id`, uses `current_stage` to continue remaining stages.
|
||
- On exception: marks run failed, records error/completed_at.
|
||
- `_calculate_next_run`:
|
||
- Computes next run datetime based on frequency and `scheduled_time`, resetting seconds/microseconds; handles month rollover for monthly frequency.
|
||
|
||
## Data Structures / Models Involved (no code)
|
||
- `AutomationConfig`: contains schedule fields (`frequency`, `scheduled_time`, `last_run_at`, `next_run_at`, `is_enabled`).
|
||
- `AutomationRun`: records run status/stage used during resume/failure handling.
|
||
|
||
## Execution Flow
|
||
1) Celery beat (or cron) invokes `check_scheduled_automations` hourly.
|
||
2) Eligible configs spawn new runs via `AutomationService.start_automation` (includes lock + credit check).
|
||
3) `run_automation_task` executes the pipeline asynchronously.
|
||
4) Paused runs can be resumed by enqueueing `resume_automation_task`/`continue_automation_task`, which restart at `current_stage`.
|
||
5) Failures set run status to `failed` and release locks.
|
||
|
||
## Cross-Module Interactions
|
||
- Uses planner/writer data inside the pipeline (see pipeline doc); billing/credits enforced at start.
|
||
- Locking is done via Django cache, independent of other modules but prevents concurrent Celery runs per site.
|
||
|
||
## State Transitions
|
||
- Config timestamps (`last_run_at`, `next_run_at`) update on scheduled launch.
|
||
- Run status changes to `failed` on task exceptions; to `completed` at stage 7; to `paused/cancelled` via API.
|
||
|
||
## Error Handling
|
||
- Scheduled start is skipped with log messages if recently run or already running.
|
||
- Exceptions during run execution mark the run failed, record error message, set `completed_at`, and release the cache lock.
|
||
|
||
## Tenancy Rules
|
||
- Configs and runs are site- and account-scoped; scheduler uses stored account/site from the config; no cross-tenant scheduling.
|
||
|
||
## Billing Rules
|
||
- Start uses `AutomationService.start_automation`, which enforces credit sufficiency before scheduling the Celery execution.
|
||
|
||
## Background Tasks / Schedulers
|
||
- Hourly `check_scheduled_automations` plus the long-running `run_automation_task` and resume tasks run in Celery workers.
|
||
|
||
## Key Design Considerations
|
||
- Hourly scan with coarse matching keeps implementation simple while honoring per-site schedules.
|
||
- Cache lock and active-run checks prevent double-starts from overlapping schedules or manual triggers.
|
||
- Resume task reuses the same stage methods to keep behavior consistent between fresh and resumed runs.
|
||
|
||
## How Developers Should Work With This Module
|
||
- When adding new frequencies, extend `check_scheduled_automations` and `_calculate_next_run` consistently.
|
||
- Ensure Celery beat (or an equivalent scheduler) runs `check_scheduled_automations` hourly in production.
|
||
- Preserve lock acquisition and failure handling when modifying task flows to avoid orphaned locks.
|