Automation Logging

Purpose

Explain how automation runs are logged, where artifacts are written, and how logs are retrieved via API.

Logger implementation: backend/igny8_core/business/automation/services/automation_logger.py
Service usage: backend/igny8_core/business/automation/services/automation_service.py
API exposure: backend/igny8_core/business/automation/views.py (logs action)
Run metadata: backend/igny8_core/business/automation/models.py

Create per-run directories and log files for automation runs (primary and optional shared mirror).
Record stage-level start/progress/complete events and structured traces.
Provide tail retrieval of logs through an API endpoint.

AutomationLogger:
- On start_run: generates run_id (run_YYYYMMDD_HHMMSS_<trigger>), creates run directory under base_log_dir (default /data/app/logs/automation), and optionally mirrors to shared_log_dir when configured. Initializes automation_run.log with run metadata and writes a JSONL trace entry (run_trace.jsonl).
- log_stage_start: appends to main log and writes a stage-specific log file stage_<n>.log; mirrors to shared dir if configured; writes a trace event (stage_start with pending count).
- log_stage_progress: appends progress lines to main and stage logs (and mirrored copies), and writes a stage_progress trace event.
- log_stage_complete: writes completion info with processed count, time elapsed, credits used to main/stage logs; emits trace (stage_complete).
- log_stage_error: writes error lines to logs and emits an error trace.
- append_trace: helper used by the service for per-item events; appends JSONL entries.
- get_activity_log: reads and returns the tail of automation_run.log (used by API).
AutomationService uses the logger at every stage boundary and significant progress point; also logs warnings/errors and mirrors events via the logger’s trace helpers.
Stage result data is stored in AutomationRun.stage_n_result JSON fields alongside logs, providing structured state in the DB.

AutomationRun: stores per-stage result JSON and run status used to correlate with log files.

Run start → start_run creates directories/files → each stage emits start/progress/complete/error logs and traces → API logs endpoint reads automation_run.log tail for a given run_id.

Logging uses the automation run/account/site context; no external modules write to these files. AI stages provide progress data; billing is inferred from AI task logs but not logged here.

Log files accumulate throughout a run; traces document run lifecycle events. Completion or failure is also reflected in AutomationRun status/results.

Logger operations are best-effort; exceptions during file/trace writes are swallowed to avoid breaking the run.
API logs returns 404 if the run does not exist; otherwise returns the requested number of lines.

Run directories are keyed by account/site/run; API access is constrained by site ownership through the viewset query for AutomationRun.

None directly; credit use is recorded in stage result JSON and inferred from AI task logs, not in the file logs.

Logging occurs inside Celery-executed stages; no separate scheduled logging tasks.

File-based logs per run make multi-stage processes auditable and debuggable without DB bloat.
Optional shared log dir supports centralized aggregation when multiple containers run workers.
JSONL traces allow machine parsing for UI status feeds while text logs support human debugging.

When adding stage steps, emit progress via AutomationLogger to keep observability consistent.
Keep log writes best-effort; do not raise from logger helpers.
Ensure any new API log endpoints read from the same per-run directory structure and remain site/account scoped.