igny8/docs/00-SYSTEM/DATA-FLOWS.md

Data Flow Diagrams (Narrative)

Purpose

Describe end-to-end data movement through the system based on current routing, middleware, and model conventions. No diagrams are embedded; flows are explained textually.

Code Locations (exact paths)

• Request routing: backend/igny8_core/urls.py
• Middleware: backend/igny8_core/middleware/request_id.py, backend/igny8_core/auth/middleware.py, backend/igny8_core/middleware/resource_tracker.py
• DRF base viewsets: backend/igny8_core/api/base.py
• Authentication classes: backend/igny8_core/api/authentication.py
• Tenancy models: backend/igny8_core/auth/models.py
• Celery configuration: backend/igny8_core/settings.py

High-Level Responsibilities

• Trace how HTTP requests are processed, tenant-scoped, authorized, and persisted.
• Show where async processing departs to Celery and where responses are shaped.

Detailed Behavior

• Incoming request → Django middleware stack:
  • Security/WhiteNoise/CORS/session/common/CSRF/Django auth.
  • RequestIDMiddleware assigns request.request_id and returns it in X-Request-ID.
  • AccountContextMiddleware resolves user/account (session or JWT) and enforces active plan; sets request.account.
  • ResourceTrackingMiddleware optionally tracks resource usage for admin/developer users when the X-Debug-Resource-Tracking header is true; adds X-Resource-Tracking-ID.
• URL dispatch via urls.py routes to module routers (auth, account, planner, writer, system, billing, automation, linker, optimizer, publisher, integration) under /api/v1/*.
• DRF viewset pipeline:
  • Authentication classes (API key → JWT → session → basic) establish request.user (and optionally request.account/request.site).
  • Base viewsets (AccountModelViewSet, SiteSectorModelViewSet) filter querysets by account/site/sector and attach account on create.
  • Serializers handle validation; responses are wrapped by unified helpers to standardize success/error payloads and pagination.
• Persistence:
  • Tenant-scoped models inherit AccountBaseModel or SiteSectorBaseModel; save hooks enforce account/site/sector alignment.
  • Soft deletion is used where models implement soft_delete, respecting retention windows from account settings.
• Async/Background:
  • Celery uses Redis broker/backend; tasks inherit JSON payloads and time limits from settings.py.
  • Automation, AI, publishing, and billing tasks enqueue via Celery; results return through database/state updates, not synchronous responses.
• Response:
  • Unified response wrappers ensure success, data/error, and request ID are present; paginated responses include count/next/previous/results.
  • Throttling headers apply per-scope (as configured in REST_FRAMEWORK throttles).

Data Structures / Models Involved (no code)

• Tenancy bases: AccountBaseModel, SiteSectorBaseModel.
• Core entities: Account, Plan, Site, Sector, User, SiteUserAccess.
• Module-specific models follow the same tenancy bases (documented in module-specific files).

Execution Flow

1. HTTP request hits middleware; IDs and tenant context are set.
2. DRF authentication authenticates and sets user/account/site.
3. Viewset filters data by tenant/site/sector and runs serializer validation.
4. DB operations persist data with enforced tenant alignment.
5. Optional Celery tasks are queued for long-running work.
6. Response returns unified JSON with request IDs and optional throttling/pagination headers.

Cross-Module Interactions

• Auth context set in middleware is consumed by all module viewsets for scoping.
• API key auth provides site context for integration/publisher flows.
• Celery configuration is shared by automation/AI/publishing/billing task modules.

State Transitions (if applicable)

• Entity lifecycle changes (create/update/delete/soft-delete) flow through base viewsets and tenancy bases, ensuring account/site/sector consistency.
• Request lifecycle includes request ID creation, optional resource tracking, and unified response wrapping.

Error Handling

• Middleware can short-circuit with JSON errors for missing account/plan.
• Viewset overrides wrap validation and server errors into unified responses; missing objects return 404 payloads.
• Throttling (scope-based) returns standard DRF throttle responses with headers.

Tenancy Rules

• All tenant-bound data flows require request.account; filtering and save hooks prevent cross-tenant access.
• Admin/developer/system-account users may bypass tenant filtering; system accounts are guarded against deletion.
• Site/sector alignment is validated on save for models inheriting SiteSectorBaseModel.

Billing Rules (if applicable)

• Plan activation is validated in middleware. Credit debits and billing workflows occur in billing modules (covered elsewhere) after tenant resolution.

Background Tasks / Schedulers (if applicable)

• Celery broker/backend configuration in settings.py governs async flow; tasks should include account/site identifiers to maintain scoping.

Key Design Considerations

• Request ID and resource tracking enable traceability and performance debugging.
• Middleware ordering ensures tenant context precedes view logic.
• Unified response format keeps clients consistent across modules.

How Developers Should Work With This Module

• Preserve middleware order; new middleware must not break request ID or tenant context.
• Ensure new viewsets inherit the base classes to pick up scoping and unified responses.
• When adding async tasks, include tenant/site identifiers and respect Celery limits from settings.