docs re-org

docs/00-SYSTEM/DATA-FLOWS.md

@@ -0,0 +1,85 @@
+# Data Flow Diagrams (Narrative)
+
+## Purpose
+Describe end-to-end data movement through the system based on current routing, middleware, and model conventions. No diagrams are embedded; flows are explained textually.
+
+## Code Locations (exact paths)
+- Request routing: `backend/igny8_core/urls.py`
+- Middleware: `backend/igny8_core/middleware/request_id.py`, `backend/igny8_core/auth/middleware.py`, `backend/igny8_core/middleware/resource_tracker.py`
+- DRF base viewsets: `backend/igny8_core/api/base.py`
+- Authentication classes: `backend/igny8_core/api/authentication.py`
+- Tenancy models: `backend/igny8_core/auth/models.py`
+- Celery configuration: `backend/igny8_core/settings.py`
+
+## High-Level Responsibilities
+- Trace how HTTP requests are processed, tenant-scoped, authorized, and persisted.
+- Show where async processing departs to Celery and where responses are shaped.
+
+## Detailed Behavior
+- Incoming request → Django middleware stack:
+  - Security/WhiteNoise/CORS/session/common/CSRF/Django auth.
+  - `RequestIDMiddleware` assigns `request.request_id` and returns it in `X-Request-ID`.
+  - `AccountContextMiddleware` resolves user/account (session or JWT) and enforces active plan; sets `request.account`.
+  - `ResourceTrackingMiddleware` optionally tracks resource usage for admin/developer users when the `X-Debug-Resource-Tracking` header is true; adds `X-Resource-Tracking-ID`.
+- URL dispatch via `urls.py` routes to module routers (auth, account, planner, writer, system, billing, automation, linker, optimizer, publisher, integration) under `/api/v1/*`.
+- DRF viewset pipeline:
+  - Authentication classes (API key → JWT → session → basic) establish `request.user` (and optionally `request.account`/`request.site`).
+  - Base viewsets (`AccountModelViewSet`, `SiteSectorModelViewSet`) filter querysets by `account`/`site`/`sector` and attach `account` on create.
+  - Serializers handle validation; responses are wrapped by unified helpers to standardize success/error payloads and pagination.
+- Persistence:
+  - Tenant-scoped models inherit `AccountBaseModel` or `SiteSectorBaseModel`; save hooks enforce account/site/sector alignment.
+  - Soft deletion is used where models implement `soft_delete`, respecting retention windows from account settings.
+- Async/Background:
+  - Celery uses Redis broker/backend; tasks inherit JSON payloads and time limits from `settings.py`.
+  - Automation, AI, publishing, and billing tasks enqueue via Celery; results return through database/state updates, not synchronous responses.
+- Response:
+  - Unified response wrappers ensure `success`, `data`/`error`, and request ID are present; paginated responses include `count/next/previous/results`.
+  - Throttling headers apply per-scope (as configured in `REST_FRAMEWORK` throttles).
+
+## Data Structures / Models Involved (no code)
+- Tenancy bases: `AccountBaseModel`, `SiteSectorBaseModel`.
+- Core entities: `Account`, `Plan`, `Site`, `Sector`, `User`, `SiteUserAccess`.
+- Module-specific models follow the same tenancy bases (documented in module-specific files).
+
+## Execution Flow
+1) HTTP request hits middleware; IDs and tenant context are set.  
+2) DRF authentication authenticates and sets user/account/site.  
+3) Viewset filters data by tenant/site/sector and runs serializer validation.  
+4) DB operations persist data with enforced tenant alignment.  
+5) Optional Celery tasks are queued for long-running work.  
+6) Response returns unified JSON with request IDs and optional throttling/pagination headers.
+
+## Cross-Module Interactions
+- Auth context set in middleware is consumed by all module viewsets for scoping.
+- API key auth provides site context for integration/publisher flows.
+- Celery configuration is shared by automation/AI/publishing/billing task modules.
+
+## State Transitions (if applicable)
+- Entity lifecycle changes (create/update/delete/soft-delete) flow through base viewsets and tenancy bases, ensuring account/site/sector consistency.
+- Request lifecycle includes request ID creation, optional resource tracking, and unified response wrapping.
+
+## Error Handling
+- Middleware can short-circuit with JSON errors for missing account/plan.
+- Viewset overrides wrap validation and server errors into unified responses; missing objects return 404 payloads.
+- Throttling (scope-based) returns standard DRF throttle responses with headers.
+
+## Tenancy Rules
+- All tenant-bound data flows require `request.account`; filtering and save hooks prevent cross-tenant access.
+- Admin/developer/system-account users may bypass tenant filtering; system accounts are guarded against deletion.
+- Site/sector alignment is validated on save for models inheriting `SiteSectorBaseModel`.
+
+## Billing Rules (if applicable)
+- Plan activation is validated in middleware. Credit debits and billing workflows occur in billing modules (covered elsewhere) after tenant resolution.
+
+## Background Tasks / Schedulers (if applicable)
+- Celery broker/backend configuration in `settings.py` governs async flow; tasks should include account/site identifiers to maintain scoping.
+
+## Key Design Considerations
+- Request ID and resource tracking enable traceability and performance debugging.
+- Middleware ordering ensures tenant context precedes view logic.
+- Unified response format keeps clients consistent across modules.
+
+## How Developers Should Work With This Module
+- Preserve middleware order; new middleware must not break request ID or tenant context.
+- Ensure new viewsets inherit the base classes to pick up scoping and unified responses.
+- When adding async tasks, include tenant/site identifiers and respect Celery limits from settings.