63 lines
4.1 KiB
Markdown
63 lines
4.1 KiB
Markdown
# Automation Logging
|
||
|
||
## Purpose
|
||
Explain how automation runs are logged, where artifacts are written, and how logs are retrieved via API.
|
||
|
||
## Code Locations (exact paths)
|
||
- Logger implementation: `backend/igny8_core/business/automation/services/automation_logger.py`
|
||
- Service usage: `backend/igny8_core/business/automation/services/automation_service.py`
|
||
- API exposure: `backend/igny8_core/business/automation/views.py` (`logs` action)
|
||
- Run metadata: `backend/igny8_core/business/automation/models.py`
|
||
|
||
## High-Level Responsibilities
|
||
- Create per-run directories and log files for automation runs (primary and optional shared mirror).
|
||
- Record stage-level start/progress/complete events and structured traces.
|
||
- Provide tail retrieval of logs through an API endpoint.
|
||
|
||
## Detailed Behavior
|
||
- `AutomationLogger`:
|
||
- On `start_run`: generates `run_id` (`run_YYYYMMDD_HHMMSS_<trigger>`), creates run directory under `base_log_dir` (default `/data/app/logs/automation`), and optionally mirrors to `shared_log_dir` when configured. Initializes `automation_run.log` with run metadata and writes a JSONL trace entry (`run_trace.jsonl`).
|
||
- `log_stage_start`: appends to main log and writes a stage-specific log file `stage_<n>.log`; mirrors to shared dir if configured; writes a trace event (`stage_start` with pending count).
|
||
- `log_stage_progress`: appends progress lines to main and stage logs (and mirrored copies), and writes a `stage_progress` trace event.
|
||
- `log_stage_complete`: writes completion info with processed count, time elapsed, credits used to main/stage logs; emits trace (`stage_complete`).
|
||
- `log_stage_error`: writes error lines to logs and emits an error trace.
|
||
- `append_trace`: helper used by the service for per-item events; appends JSONL entries.
|
||
- `get_activity_log`: reads and returns the tail of `automation_run.log` (used by API).
|
||
- `AutomationService` uses the logger at every stage boundary and significant progress point; also logs warnings/errors and mirrors events via the logger’s trace helpers.
|
||
- Stage result data is stored in `AutomationRun.stage_n_result` JSON fields alongside logs, providing structured state in the DB.
|
||
|
||
## Data Structures / Models Involved (no code)
|
||
- `AutomationRun`: stores per-stage result JSON and run status used to correlate with log files.
|
||
|
||
## Execution Flow
|
||
- Run start → `start_run` creates directories/files → each stage emits start/progress/complete/error logs and traces → API `logs` endpoint reads `automation_run.log` tail for a given `run_id`.
|
||
|
||
## Cross-Module Interactions
|
||
- Logging uses the automation run/account/site context; no external modules write to these files. AI stages provide progress data; billing is inferred from AI task logs but not logged here.
|
||
|
||
## State Transitions
|
||
- Log files accumulate throughout a run; traces document run lifecycle events. Completion or failure is also reflected in `AutomationRun` status/results.
|
||
|
||
## Error Handling
|
||
- Logger operations are best-effort; exceptions during file/trace writes are swallowed to avoid breaking the run.
|
||
- API `logs` returns 404 if the run does not exist; otherwise returns the requested number of lines.
|
||
|
||
## Tenancy Rules
|
||
- Run directories are keyed by account/site/run; API access is constrained by site ownership through the viewset query for `AutomationRun`.
|
||
|
||
## Billing Rules
|
||
- None directly; credit use is recorded in stage result JSON and inferred from AI task logs, not in the file logs.
|
||
|
||
## Background Tasks / Schedulers
|
||
- Logging occurs inside Celery-executed stages; no separate scheduled logging tasks.
|
||
|
||
## Key Design Considerations
|
||
- File-based logs per run make multi-stage processes auditable and debuggable without DB bloat.
|
||
- Optional shared log dir supports centralized aggregation when multiple containers run workers.
|
||
- JSONL traces allow machine parsing for UI status feeds while text logs support human debugging.
|
||
|
||
## How Developers Should Work With This Module
|
||
- When adding stage steps, emit progress via `AutomationLogger` to keep observability consistent.
|
||
- Keep log writes best-effort; do not raise from logger helpers.
|
||
- Ensure any new API log endpoints read from the same per-run directory structure and remain site/account scoped.
|