From 1dd2d53a8e23ac7543e0570c17fc45174c97101b Mon Sep 17 00:00:00 2001 From: "IGNY8 VPS (Salman)" Date: Sun, 7 Dec 2025 12:03:45 +0000 Subject: [PATCH] doce revapm phase 1 --- .../00-SYSTEM-ARCHITECTURE-OVERVIEW.md | 79 ++++++++++++++++ master-docs/00-system/01-TECH-STACK.md | 61 +++++++++++++ .../00-system/02-MULTITENANCY-MODEL.md | 79 ++++++++++++++++ .../03-IDENTITY-AND-AUTHENTICATION.md | 73 +++++++++++++++ .../00-system/04-DATA-FLOW-DIAGRAMS.md | 85 +++++++++++++++++ .../05-PERMISSIONS-AND-ACCESS-CONTROL.md | 74 +++++++++++++++ .../00-system/06-ENVIRONMENT-VARIABLES.md | 91 +++++++++++++++++++ 7 files changed, 542 insertions(+) diff --git a/master-docs/00-system/00-SYSTEM-ARCHITECTURE-OVERVIEW.md b/master-docs/00-system/00-SYSTEM-ARCHITECTURE-OVERVIEW.md index e69de29b..f34dc9c8 100644 --- a/master-docs/00-system/00-SYSTEM-ARCHITECTURE-OVERVIEW.md +++ b/master-docs/00-system/00-SYSTEM-ARCHITECTURE-OVERVIEW.md @@ -0,0 +1,79 @@ +# System Architecture Overview + +## Purpose +Describe how IGNY8 is structured across backend, frontend, and integrations, grounded in the current codebase. Covers core services, middleware, and platform composition. + +## Code Locations (exact paths) +- Backend project root: `backend/igny8_core/` +- Settings and service wiring: `backend/igny8_core/settings.py` +- URL routing: `backend/igny8_core/urls.py` +- Middleware: `backend/igny8_core/middleware/request_id.py`, `backend/igny8_core/middleware/resource_tracker.py`, `backend/igny8_core/auth/middleware.py` +- Auth models and tenancy bases: `backend/igny8_core/auth/models.py` +- DRF base behaviors: `backend/igny8_core/api/base.py` +- Custom auth classes: `backend/igny8_core/api/authentication.py` +- Frontend SPA: `frontend/` (Vite + React; dependencies in `frontend/package.json`) + +## High-Level Responsibilities +- Django/DRF backend providing multi-tenant APIs for planner, writer, system, billing, automation, linker, optimizer, publisher, and integration modules. +- Middleware adds per-request IDs, tenant context, and optional resource tracking for admin diagnostics. +- REST API routing under `/api/v1/*` with unified response/error handling and scoped throttling. +- Celery-backed async work (configured in settings) for AI, automation, and publishing. +- React/Vite frontend consuming the API; authentication via JWT or session; API key support for WordPress bridge. + +## Detailed Behavior +- Backend apps registered in `INSTALLED_APPS` include auth, AI framework, planner, writer, system, billing, automation, optimization, publishing, integration, linker, optimizer, and publisher modules; these are loaded via Django app configs in `settings.py`. +- Middleware order enforces security, CORS, session, CSRF, Django auth, then custom layers: request ID, account context, resource tracking, messages, and clickjacking protection. +- URL map (`urls.py`) exposes admin, CSV admin utilities for industries/seed keywords, and module routers for auth, account, planner, writer, system, billing (user + admin), automation, linker, optimizer, publisher, and integration. OpenAPI docs available at `/api/docs` and `/api/redoc`. +- REST framework defaults use custom pagination, filtering, search, ordering, and a custom exception handler (enabled unless `IGNY8_USE_UNIFIED_EXCEPTION_HANDLER` is false). Authentication stack orders API key, JWT, CSRF-exempt session, then basic auth. Throttle scopes are predefined per domain (AI, content, auth, planner, writer, system, billing, linker, optimizer, integration). +- CORS allows the IGNY8 domains plus local development hosts; credentials and specific debug headers are permitted, and request/resource tracking IDs are exposed in response headers. +- Celery is configured to use Redis by default for broker and result backend, with JSON serialization, task time limits, and single-prefetch workers. +- Logging writes to console and rotating files for publish/sync, WordPress API calls, and webhooks, with request IDs available via middleware. + +## Data Structures / Models Involved (no code, just explanation) +- Multi-tenancy base classes (`AccountBaseModel`, `SiteSectorBaseModel`) add `account`, `site`, and `sector` scoping plus validation; defined in `auth/models.py`. +- Core tenancy entities: `Account`, `Plan`, `Subscription`, `Site`, `Sector`, `Industry`, `IndustrySector`, `SeedKeyword`, `SiteUserAccess`, `User`, and `PasswordResetToken` in `auth/models.py`. +- These bases are consumed by downstream modules (planner, writer, billing, automation, etc.) to enforce tenant, site, and sector ownership. + +## Execution Flow +- Incoming HTTP requests enter Django middleware: security → WhiteNoise → CORS → session → common/CSRF → Django auth → `RequestIDMiddleware` (assigns `X-Request-ID`) → `AccountContextMiddleware` (sets `request.account` via session or JWT) → `ResourceTrackingMiddleware` (when enabled for admins) → messages → clickjacking. +- DRF viewsets inherit from base classes in `api/base.py` to auto-filter querysets by account (and site/sector where applicable) and emit unified responses. +- URL dispatch routes to module-specific routers under `/api/v1/*`. CSV admin helpers are mounted before admin to avoid routing conflicts. +- Background tasks are dispatched via Celery using Redis endpoints from `settings.py`. + +## Cross-Module Interactions +- Auth middleware sets `request.account` consumed by `AccountModelViewSet` and `SiteSectorModelViewSet` to filter data. +- API key auth (WordPress bridge) sets both `request.account` and `request.site` for integration endpoints. +- Resource tracking middleware exposes metrics via cache keyed by the request-specific tracking ID added to responses. +- OpenAPI generation (drf-spectacular) documents all modules and respects unified response schemas. + +## State Transitions (if applicable) +- Request lifecycle: ID assignment → tenant resolution → optional resource profiling → viewset execution → unified response with optional pagination and request/resource IDs in headers. +- Tenancy lifecycle: Account/Plan/Subscription fields in models track status and retention; soft delete support on models that implement it. + +## Error Handling +- Global DRF exception handler (custom when enabled) wraps errors into unified JSON with `success=false`. +- Account middleware denies access when account or plan is missing/inactive, returning structured JSON and logging out session users. +- Viewset overrides in `api/base.py` return unified error payloads for validation and 404/500 cases. + +## Tenancy Rules +- Account is injected via middleware (session or JWT). Base viewsets filter querysets by `account`, unless the user is admin/developer/system account (override path). +- `SiteSectorModelViewSet` adds site/sector filtering when models carry those fields; validation ensures site/sector belong to the same account. +- Role helpers (`User.is_admin_or_developer`, `User.is_system_account_user`) allow bypass for privileged users; otherwise, data is restricted to the resolved account (and site/sector for derived models). + +## Billing Rules (if applicable) +- Plan and account billing fields live in `auth/models.py` (`Plan.included_credits`, `Account.credits`, Stripe IDs). Status enforcement occurs in `AccountContextMiddleware` by requiring an active plan; billing modules implement credit logic (covered in backend/billing docs). + +## Background Tasks / Schedulers (if applicable) +- Celery configuration in `settings.py` sets Redis broker/backend, JSON serialization, time limits, and worker tuning. Task modules (AI, automation, publishing) use this setup; scheduling uses Celery Beat state. + +## Key Design Considerations +- Middleware-first tenant resolution ensures consistent scoping before view logic. +- Admin/developer/system-account overrides allow cross-tenant operations for ops while protecting system accounts from deletion. +- Unified API responses and throttling scopes enforce consistent client behavior and rate safety. +- Redis-backed Celery keeps async workloads out of request path with strict time limits. + +## How Developers Should Work With This Module +- Add new apps to `INSTALLED_APPS` and mount URLs under `/api/v1/*` in `urls.py`. +- Inherit from `AccountModelViewSet` or `SiteSectorModelViewSet` to automatically enforce tenant/site/sector scoping and unified responses. +- Use existing middleware; do not reorder request ID or account context layers, as downstream views rely on them. +- Configure environment via `settings.py` variables (DB, JWT, Celery, CORS, Stripe/PayPal) rather than hardcoding values. diff --git a/master-docs/00-system/01-TECH-STACK.md b/master-docs/00-system/01-TECH-STACK.md index e69de29b..85a88970 100644 --- a/master-docs/00-system/01-TECH-STACK.md +++ b/master-docs/00-system/01-TECH-STACK.md @@ -0,0 +1,61 @@ +# Tech Stack + +## Purpose +Document the concrete technologies and dependencies in use across backend and frontend as defined in the repository. + +## Code Locations (exact paths) +- Backend dependency manifest: `backend/requirements.txt` +- Backend settings (framework integration): `backend/igny8_core/settings.py` +- Frontend dependency manifest: `frontend/package.json` +- Frontend build tooling: `frontend/vite.config.ts`, `frontend/tsconfig*.json` + +## High-Level Responsibilities +- Backend: Django 5.x with DRF for APIs, Celery for async tasks, Redis for broker/result, PostgreSQL or SQLite for data, drf-spectacular for OpenAPI, Stripe/PayPal configs for billing, WhiteNoise for static serving. +- Frontend: React 19 with Vite, TypeScript, TailwindCSS, Zustand state, React Router 7, ApexCharts and FullCalendar for UI widgets. + +## Detailed Behavior +- Backend settings wire DRF pagination, filters, authentication (API key, JWT, session, basic), throttling, schema generation, CORS, Celery, logging, and Stripe/PayPal credentials. Static assets are served via WhiteNoise; admin uses Django contrib. +- `requirements.txt` enumerates runtime libs: Django, gunicorn, psycopg2-binary (PostgreSQL), redis, WhiteNoise, DRF, django-filter, django-cors-headers, PyJWT, requests, Celery, BeautifulSoup4, psutil, docker (for ops scripts), drf-spectacular, and stripe. +- Frontend `package.json` pins React 19, React Router 7, Zustand 5, Vite 6, TailwindCSS 4, ApexCharts 4, FullCalendar 6, react-dnd, dropzone, lucide/react-heroicons, and testing/tooling (Vitest, Testing Library, ESLint). +- Build scripts use Vite for dev and production builds, with separate marketing mode; tests via Vitest; lint via ESLint; type-check via `tsc -b`. + +## Data Structures / Models Involved (no code) +- Not applicable; this file tracks technology components rather than domain models. + +## Execution Flow +- Backend runs under Django/DRF with middleware and installed apps per `settings.py`. ASGI/WSGI entrypoints in `igny8_core/asgi.py` and `igny8_core/wsgi.py` (default WSGI via gunicorn). +- Celery worker/beat use Redis URLs from `settings.py` and respect JSON serialization/time limits. +- Frontend builds with Vite, consuming environment variables defined via Vite conventions. + +## Cross-Module Interactions +- DRF auth classes depend on PyJWT and custom utilities for token handling. +- Celery tasks in AI/automation/publishing rely on Redis connectivity and the configured serializer/time limits. +- Stripe/PayPal keys in settings are consumed by billing modules. +- Frontend API calls rely on the DRF endpoints exposed in `igny8_core/urls.py`. + +## State Transitions (if applicable) +- Dependency-driven: none beyond the build/runtime phases (install → build → serve). + +## Error Handling +- Backend error handling configured via custom DRF exception handler (enabled by default) and logging setup in `settings.py`. +- Frontend build/test errors are surfaced through Vite/TypeScript/Vitest/ESLint tooling. + +## Tenancy Rules +- Implemented at runtime by backend middleware and viewsets; the tech stack provides JWT, session, and API key support to carry tenant context. + +## Billing Rules (if applicable) +- Stripe and PayPal keys in `settings.py` enable billing integrations; credit logic is implemented in billing modules (documented elsewhere). + +## Background Tasks / Schedulers (if applicable) +- Celery configured in `settings.py` with Redis broker/backend, JSON serialization, and task limits; beat scheduling persists in `celerybeat-schedule`. + +## Key Design Considerations +- Keep dependency manifests authoritative (`requirements.txt`, `package.json`). +- Redis is the default async backbone; Postgres is the default DB with SQLite fallback for local/dev. +- Vite + React 19 selected for fast dev/build; TailwindCSS 4 used for styling; Zustand for state. + +## How Developers Should Work With This Module +- Add backend dependencies to `requirements.txt` and pin versions appropriately; update settings if new middleware/auth is added. +- Add frontend dependencies to `package.json`; run `npm install` and ensure Vite/TSC builds remain clean. +- Respect configured auth stack (API key → JWT → session) when adding API clients. +- Keep CORS and env vars aligned with the domains/ports in use for local and production. diff --git a/master-docs/00-system/02-MULTITENANCY-MODEL.md b/master-docs/00-system/02-MULTITENANCY-MODEL.md index e69de29b..1f295ade 100644 --- a/master-docs/00-system/02-MULTITENANCY-MODEL.md +++ b/master-docs/00-system/02-MULTITENANCY-MODEL.md @@ -0,0 +1,79 @@ +# Multitenancy Model + +## Purpose +Explain how tenant, site, and sector isolation is enforced across models, middleware, and viewsets, based on the current implementation. + +## Code Locations (exact paths) +- Tenant base models: `backend/igny8_core/auth/models.py` (`AccountBaseModel`, `SiteSectorBaseModel`) +- Core entities: `backend/igny8_core/auth/models.py` (`Account`, `Plan`, `Site`, `Sector`, `Industry`, `IndustrySector`, `SeedKeyword`, `SiteUserAccess`, `User`) +- Middleware for context: `backend/igny8_core/auth/middleware.py` +- DRF base viewsets: `backend/igny8_core/api/base.py` +- Auth utilities and JWT: `backend/igny8_core/api/authentication.py`, `backend/igny8_core/auth/utils.py` +- URL routing (module mounting): `backend/igny8_core/urls.py` + +## High-Level Responsibilities +- Enforce per-account isolation for all models carrying an `account` FK. +- Enforce per-site and per-sector scoping for content models via `SiteSectorBaseModel`. +- Inject tenant context on every request (session or JWT/API key), then apply scoping in base viewsets. +- Allow controlled overrides for admins, developers, and system accounts. + +## Detailed Behavior +- `AccountBaseModel` adds an `account` FK plus timestamps and indexes; all tenant-scoped models inherit this to guarantee account linkage. +- `SiteSectorBaseModel` extends `AccountBaseModel` with `site` and `sector` FKs, indexes on `(account, site, sector)`, and a save hook that sets `account` from `site` and validates that `sector` belongs to the same `site`; raises validation errors on mismatch. +- `AccountContextMiddleware` sets `request.account` by refreshing the authenticated session user (with account and plan) or by decoding JWT tokens; it rejects requests when account is missing or plan is inactive, returning structured JSON and logging out session users. It skips admin and auth endpoints to avoid interference. +- JWT authentication (`api/authentication.py`) decodes tokens and sets `request.account` from the token payload; API key authentication sets both `request.account` and `request.site` for WordPress bridge calls. +- `AccountModelViewSet` auto-filters querysets by `account` when models expose that field. It bypasses filtering for admins/developers/system-account users; otherwise, it uses `request.account` or falls back to the authenticated user’s account. Creates set `account` on save when present. +- `SiteSectorModelViewSet` extends the above to filter by site/sector if those fields exist, using request query parameters and tenancy context. +- `User` role helpers (`is_admin_or_developer`, `is_system_account_user`) and account checks gate override behavior. +- `SiteUserAccess` provides explicit per-site access for non-admin roles; `User.get_accessible_sites` respects system account, developer, owner/admin, and granted access rules. + +## Data Structures / Models Involved (no code) +- `Account`: tenant container with plan, credits, billing fields, status, and retention settings. +- `Plan`: defines limits (max users/sites/industries/author profiles), credit inclusion, Stripe IDs. +- `Site`: belongs to an account, optionally an industry; includes status, hosting type, legacy WP fields, and SEO metadata. +- `Sector`: belongs to a site and industry sector template; enforces account alignment and plan-based max-sector validation. +- `Industry`, `IndustrySector`, `SeedKeyword`: global reference data not bound to a single account. +- `SiteUserAccess`: explicit many-to-many grants between users and sites. +- `User`: links to account with role-based access flags. + +## Execution Flow +- Request enters middleware; `AccountContextMiddleware` determines `request.account` (session or JWT/API key), validating plan status. +- Viewsets inheriting from `AccountModelViewSet`/`SiteSectorModelViewSet` filter querysets by `account` (and site/sector when present) before pagination/serialization. +- Object creation sets `account` automatically when the serializer’s model has that field; site/sector-based models validate alignment on save. +- Admin/developer/system-account users skip account filtering; other users remain constrained. + +## Cross-Module Interactions +- All module viewsets depend on the base viewsets for scoping. +- Automation, planner, writer, billing, linker, optimizer, publisher, and integration models inherit from the tenancy bases to enforce account/site/sector ownership. +- API key flows for WordPress set `request.site`, enabling integration-specific logic to run in a site-aware context. + +## State Transitions (if applicable) +- Account status changes (active, suspended, trial, cancelled) and plan activation directly affect access through middleware plan validation. +- Sector creation enforces plan-based limits for active sectors per site. + +## Error Handling +- Middleware returns JSON errors for missing account or inactive plan, with HTTP 403 or 402 semantics and logs out session users. +- Base viewsets wrap CRUD operations in unified responses; validation failures or missing objects are returned in structured error payloads. +- Save-time validation on `SiteSectorBaseModel` and `Sector` raises validation errors when site/sector alignment or plan limits are violated. + +## Tenancy Rules +- Every tenant-scoped model carries `account`; site/sector-aware models carry `site` and `sector` and must align to the same account. +- Middleware populates `request.account`; base viewsets enforce filtering unless the user is an admin/developer/system-account member. +- System accounts (`aws-admin`, `default-account`, `default`) and privileged roles can bypass scoping; protected from deletion via guard clauses. + +## Billing Rules (if applicable) +- Middleware requires an active plan before allowing requests (except auth/admin paths). Credits, charges, and plan enforcement are handled in billing modules (documented elsewhere). + +## Background Tasks / Schedulers (if applicable) +- Celery tasks inherit tenant context via payloads supplied by calling viewsets/services; the tenancy bases ensure stored records retain `account`/`site`/`sector`. + +## Key Design Considerations +- Tenancy is enforced as early as middleware to avoid leakage in view logic. +- Base viewsets centralize scoping and unified responses to reduce duplication across modules. +- Role-based overrides exist for ops and system accounts; safeguards prevent system account deletion. + +## How Developers Should Work With This Module +- Inherit from `AccountBaseModel` or `SiteSectorBaseModel` for any new tenant/site/sector data models. +- Inherit viewsets from `AccountModelViewSet` or `SiteSectorModelViewSet` to get automatic scoping and unified responses. +- Do not bypass `AccountContextMiddleware`; ensure new endpoints live under `/api/v1/*` and rely on the auth stack (API key → JWT → session). +- Validate that new background tasks carry account/site/sector identifiers so downstream saves remain scoped. diff --git a/master-docs/00-system/03-IDENTITY-AND-AUTHENTICATION.md b/master-docs/00-system/03-IDENTITY-AND-AUTHENTICATION.md index e69de29b..d5a6eef8 100644 --- a/master-docs/00-system/03-IDENTITY-AND-AUTHENTICATION.md +++ b/master-docs/00-system/03-IDENTITY-AND-AUTHENTICATION.md @@ -0,0 +1,73 @@ +# Identity and Authentication + +## Purpose +Document how user identity, JWT handling, API keys, and session flows work, including middleware and validation rules. + +## Code Locations (exact paths) +- JWT utilities: `backend/igny8_core/auth/utils.py` +- Account context middleware: `backend/igny8_core/auth/middleware.py` +- DRF authentication classes: `backend/igny8_core/api/authentication.py` +- DRF settings for auth/throttle: `backend/igny8_core/settings.py` +- User model and roles: `backend/igny8_core/auth/models.py` +- Auth URLs and views: `backend/igny8_core/auth/urls.py`, `backend/igny8_core/auth/views.py` + +## High-Level Responsibilities +- Support multiple auth mechanisms: API key (WordPress bridge), JWT bearer tokens, session auth without CSRF for APIs, and basic auth fallback. +- Populate tenant/site context alongside user identity so downstream viewsets enforce isolation. +- Enforce active account/plan presence before serving protected endpoints (except admin/auth routes). + +## Detailed Behavior +- Authentication order (DRF `DEFAULT_AUTHENTICATION_CLASSES` in `settings.py`): API key → JWT → CSRF-exempt session → basic auth. The first class that authenticates sets `request.user`; `request.account` may also be set by API key or JWT. +- API Key flow (`APIKeyAuthentication`): expects `Authorization: Bearer ` that is not JWT-like; finds an active `Site` with `wp_api_key`, loads its `account`, and selects an active user (owner preferred, else any active developer/owner/admin). Sets `request.account` and `request.site`. Rejects short/invalid keys; returns an auth failure if no active user exists. +- JWT flow (`JWTAuthentication`): expects `Authorization: Bearer `; decodes via `auth.utils.decode_token`; only accepts tokens with `type == access`. Retrieves `User` by `user_id`; optional `account_id` is resolved to `Account` and set on `request.account`. Invalid/expired tokens fall through to other auth classes. +- Session flow (`CSRFExemptSessionAuthentication`): uses Django session cookies without CSRF enforcement for API calls. +- Basic auth: last resort; does not set tenant context. +- Token utilities (`auth/utils.py`) generate and decode access/refresh tokens using expiries from settings (`JWT_ACCESS_TOKEN_EXPIRY`, `JWT_REFRESH_TOKEN_EXPIRY`), embedding `user_id`, `account_id`, `email`, issued/expiry timestamps, and token `type`. +- Middleware (`AccountContextMiddleware`) runs on every request except admin/auth paths: refreshes session users from DB to pick up current account/plan, validates presence of an active plan, sets `request.account`, and logs out session users when invalid. For JWT-bearing requests it decodes the token directly and sets `request.account`. If account/plan is missing or inactive, it returns JSON with `success=false` and appropriate HTTP status. + +## Data Structures / Models Involved (no code) +- `User` with `role` and `account` FKs in `auth/models.py`. +- `Account` with plan and billing fields; plan status is used for access gating. +- `Site` with `wp_api_key` for API key auth; `SiteUserAccess` for per-site grants. +- `PasswordResetToken` model for password reset flows. +- JWT payload fields: `user_id`, `account_id`, `email`, `exp`, `iat`, `type`. + +## Execution Flow +- Middleware step: `AccountContextMiddleware` determines `request.account` (session or JWT) and validates plan status; skips admin/auth routes. +- DRF auth step: API key/JWT/session/basic authenticators run in order, potentially setting `request.account` (API key/JWT) and `request.site` (API key). +- Viewsets then apply role/permission checks and tenant/site/sector filtering via base classes in `api/base.py`. + +## Cross-Module Interactions +- All module viewsets rely on `request.user` and `request.account` set by the auth stack. Site-aware modules can read `request.site` when API key auth is used. +- Role helpers (`is_admin_or_developer`, `is_system_account_user`) influence filtering bypass in base viewsets. + +## State Transitions (if applicable) +- JWT lifetimes: access tokens default to 15 minutes; refresh tokens to 30 days (configurable in settings). +- Session users are refreshed on each request to pick up plan/account changes. +- Password reset tokens track expiry and usage via `expires_at` and `used` flags. + +## Error Handling +- Middleware returns JSON errors for missing account or inactive plan and logs out session users in those cases. +- Invalid/expired JWTs cause the JWT authenticator to return `None`, allowing other auth methods; decoding errors raise `InvalidTokenError` in utilities. +- API key auth raises an auth failure when no active user is available for the resolved account. + +## Tenancy Rules +- `request.account` is set early; base viewsets enforce account filtering unless user has admin/developer/system-account privileges. +- API key auth also sets `request.site` for integration contexts; site/sector filtering occurs in `SiteSectorModelViewSet`. + +## Billing Rules (if applicable) +- Active plan is required for access (middleware enforces). Credit debits/charges are handled in billing modules, not in the auth layer. + +## Background Tasks / Schedulers (if applicable) +- Token generation/validation is synchronous. Background tasks should receive explicit user/account identifiers in their payloads when invoked. + +## Key Design Considerations +- Authentication stack is ordered to give integration API keys precedence, then JWT for app clients, then session for browser-based flows. +- Tenant context must be established before view logic; do not move or remove `AccountContextMiddleware`. +- Expiry durations and JWT secrets are centrally configured in `settings.py`. + +## How Developers Should Work With This Module +- Use token helpers from `auth/utils.py` when issuing tokens; do not handcraft JWTs. +- Mount new auth-sensitive endpoints under existing routers and rely on DRF auth classes instead of custom header parsing. +- Ensure new features that require site context can work with API key auth by checking `request.site`. +- Keep plan enforcement in place; bypass only for admin/system routes when justified. diff --git a/master-docs/00-system/04-DATA-FLOW-DIAGRAMS.md b/master-docs/00-system/04-DATA-FLOW-DIAGRAMS.md index e69de29b..b5846f67 100644 --- a/master-docs/00-system/04-DATA-FLOW-DIAGRAMS.md +++ b/master-docs/00-system/04-DATA-FLOW-DIAGRAMS.md @@ -0,0 +1,85 @@ +# Data Flow Diagrams (Narrative) + +## Purpose +Describe end-to-end data movement through the system based on current routing, middleware, and model conventions. No diagrams are embedded; flows are explained textually. + +## Code Locations (exact paths) +- Request routing: `backend/igny8_core/urls.py` +- Middleware: `backend/igny8_core/middleware/request_id.py`, `backend/igny8_core/auth/middleware.py`, `backend/igny8_core/middleware/resource_tracker.py` +- DRF base viewsets: `backend/igny8_core/api/base.py` +- Authentication classes: `backend/igny8_core/api/authentication.py` +- Tenancy models: `backend/igny8_core/auth/models.py` +- Celery configuration: `backend/igny8_core/settings.py` + +## High-Level Responsibilities +- Trace how HTTP requests are processed, tenant-scoped, authorized, and persisted. +- Show where async processing departs to Celery and where responses are shaped. + +## Detailed Behavior +- Incoming request → Django middleware stack: + - Security/WhiteNoise/CORS/session/common/CSRF/Django auth. + - `RequestIDMiddleware` assigns `request.request_id` and returns it in `X-Request-ID`. + - `AccountContextMiddleware` resolves user/account (session or JWT) and enforces active plan; sets `request.account`. + - `ResourceTrackingMiddleware` optionally tracks resource usage for admin/developer users when the `X-Debug-Resource-Tracking` header is true; adds `X-Resource-Tracking-ID`. +- URL dispatch via `urls.py` routes to module routers (auth, account, planner, writer, system, billing, automation, linker, optimizer, publisher, integration) under `/api/v1/*`. +- DRF viewset pipeline: + - Authentication classes (API key → JWT → session → basic) establish `request.user` (and optionally `request.account`/`request.site`). + - Base viewsets (`AccountModelViewSet`, `SiteSectorModelViewSet`) filter querysets by `account`/`site`/`sector` and attach `account` on create. + - Serializers handle validation; responses are wrapped by unified helpers to standardize success/error payloads and pagination. +- Persistence: + - Tenant-scoped models inherit `AccountBaseModel` or `SiteSectorBaseModel`; save hooks enforce account/site/sector alignment. + - Soft deletion is used where models implement `soft_delete`, respecting retention windows from account settings. +- Async/Background: + - Celery uses Redis broker/backend; tasks inherit JSON payloads and time limits from `settings.py`. + - Automation, AI, publishing, and billing tasks enqueue via Celery; results return through database/state updates, not synchronous responses. +- Response: + - Unified response wrappers ensure `success`, `data`/`error`, and request ID are present; paginated responses include `count/next/previous/results`. + - Throttling headers apply per-scope (as configured in `REST_FRAMEWORK` throttles). + +## Data Structures / Models Involved (no code) +- Tenancy bases: `AccountBaseModel`, `SiteSectorBaseModel`. +- Core entities: `Account`, `Plan`, `Site`, `Sector`, `User`, `SiteUserAccess`. +- Module-specific models follow the same tenancy bases (documented in module-specific files). + +## Execution Flow +1) HTTP request hits middleware; IDs and tenant context are set. +2) DRF authentication authenticates and sets user/account/site. +3) Viewset filters data by tenant/site/sector and runs serializer validation. +4) DB operations persist data with enforced tenant alignment. +5) Optional Celery tasks are queued for long-running work. +6) Response returns unified JSON with request IDs and optional throttling/pagination headers. + +## Cross-Module Interactions +- Auth context set in middleware is consumed by all module viewsets for scoping. +- API key auth provides site context for integration/publisher flows. +- Celery configuration is shared by automation/AI/publishing/billing task modules. + +## State Transitions (if applicable) +- Entity lifecycle changes (create/update/delete/soft-delete) flow through base viewsets and tenancy bases, ensuring account/site/sector consistency. +- Request lifecycle includes request ID creation, optional resource tracking, and unified response wrapping. + +## Error Handling +- Middleware can short-circuit with JSON errors for missing account/plan. +- Viewset overrides wrap validation and server errors into unified responses; missing objects return 404 payloads. +- Throttling (scope-based) returns standard DRF throttle responses with headers. + +## Tenancy Rules +- All tenant-bound data flows require `request.account`; filtering and save hooks prevent cross-tenant access. +- Admin/developer/system-account users may bypass tenant filtering; system accounts are guarded against deletion. +- Site/sector alignment is validated on save for models inheriting `SiteSectorBaseModel`. + +## Billing Rules (if applicable) +- Plan activation is validated in middleware. Credit debits and billing workflows occur in billing modules (covered elsewhere) after tenant resolution. + +## Background Tasks / Schedulers (if applicable) +- Celery broker/backend configuration in `settings.py` governs async flow; tasks should include account/site identifiers to maintain scoping. + +## Key Design Considerations +- Request ID and resource tracking enable traceability and performance debugging. +- Middleware ordering ensures tenant context precedes view logic. +- Unified response format keeps clients consistent across modules. + +## How Developers Should Work With This Module +- Preserve middleware order; new middleware must not break request ID or tenant context. +- Ensure new viewsets inherit the base classes to pick up scoping and unified responses. +- When adding async tasks, include tenant/site identifiers and respect Celery limits from settings. diff --git a/master-docs/00-system/05-PERMISSIONS-AND-ACCESS-CONTROL.md b/master-docs/00-system/05-PERMISSIONS-AND-ACCESS-CONTROL.md index e69de29b..d42cb615 100644 --- a/master-docs/00-system/05-PERMISSIONS-AND-ACCESS-CONTROL.md +++ b/master-docs/00-system/05-PERMISSIONS-AND-ACCESS-CONTROL.md @@ -0,0 +1,74 @@ +# Permissions and Access Control + +## Purpose +Explain role-based permissions, account ownership rules, and how viewsets enforce access across the stack. + +## Code Locations (exact paths) +- Role-based permissions: `backend/igny8_core/auth/permissions.py` +- User roles and helpers: `backend/igny8_core/auth/models.py` +- Account context middleware: `backend/igny8_core/auth/middleware.py` +- Base viewsets with override/bypass logic: `backend/igny8_core/api/base.py` +- DRF settings: `backend/igny8_core/settings.py` + +## High-Level Responsibilities +- Enforce role-based access (developer, owner, admin, editor, viewer, system_bot). +- Enforce tenant/site/sector scoping with selective overrides for privileged roles. +- Block access when accounts lack active plans. + +## Detailed Behavior +- Roles are stored on `User.role`; helper methods (`is_admin_or_developer`, `is_developer`, `is_system_account_user`, `has_role`) drive bypass logic. +- Permission classes: + - `IsOwnerOrAdmin`: allows authenticated users with roles owner/admin/developer or superuser. + - `IsEditorOrAbove`: allows editor/admin/owner/developer or superuser. + - `IsViewerOrAbove`: allows any authenticated user. + - `AccountPermission`: ensures the user is authenticated, not a system bot unless intended, and matches object account when present; system bots bypass account checks. +- Base viewsets: + - `AccountModelViewSet` filters by `account` unless `is_admin_or_developer` or `is_system_account_user` is true; create sets `account` automatically; delete protects system account slugs. + - `SiteSectorModelViewSet` extends filtering to `site`/`sector` when models have those fields. +- Middleware: + - `AccountContextMiddleware` denies requests when user lacks an account or active plan (403/402), logging out session users and returning JSON errors. +- System accounts (`aws-admin`, `default-account`, `default`) and system bots are privileged; system accounts are protected from deletion. + +## Data Structures / Models Involved (no code) +- `User` with `role` and account FK. +- `Account` with `status` and `plan` used for gating. +- `SiteUserAccess` for per-site grants evaluated by `User.get_accessible_sites`. + +## Execution Flow +- Middleware validates account/plan and sets `request.account`. +- DRF authentication sets `request.user`; permission classes on views (and base viewsets) check role/account match. +- Querysets are filtered by tenant/site/sector; privileged roles may bypass filtering. +- Soft-delete or delete operations respect system account protection. + +## Cross-Module Interactions +- All module viewsets inherit base classes, so account/site/sector scoping and role-based bypass apply uniformly. +- Integration endpoints using API key auth rely on `request.site`/`request.account` set earlier. + +## State Transitions (if applicable) +- Account status and plan activation affect access; changing these alters middleware decisions immediately due to per-request user refresh. + +## Error Handling +- Middleware returns structured JSON errors for missing account or inactive plan. +- Base viewsets wrap permission and validation failures into unified error responses. +- Permission classes return standard DRF permission denials for unauthenticated/unauthorized users. + +## Tenancy Rules +- Default: filter data to `request.account`; site/sector models further constrain by site/sector. +- Overrides: developers/admins/system-account users can see across accounts; system bots can access any account/object; system accounts cannot be deleted. + +## Billing Rules (if applicable) +- Active plan required; enforced in middleware before permission checks. + +## Background Tasks / Schedulers (if applicable) +- Background tasks should be invoked with account/site identifiers to maintain the same isolation rules; role checks occur at request time, not within tasks. + +## Key Design Considerations +- Centralize role checks to avoid per-view duplication. +- Maintain middleware-first gating to prevent expensive processing before authorization. +- Preserve protection of system accounts and allow ops-level overrides without compromising tenant isolation. + +## How Developers Should Work With This Module +- Apply permission classes (`IsOwnerOrAdmin`, `IsEditorOrAbove`, etc.) on viewsets/endpoints as needed. +- Always inherit from `AccountModelViewSet`/`SiteSectorModelViewSet` for scoped data access. +- Avoid bypassing middleware plan checks; if admin-only, use explicit admin routes instead. +- When adding new privileged behaviors, reuse `User` role helpers and system-account detection. diff --git a/master-docs/00-system/06-ENVIRONMENT-VARIABLES.md b/master-docs/00-system/06-ENVIRONMENT-VARIABLES.md index e69de29b..9fedf748 100644 --- a/master-docs/00-system/06-ENVIRONMENT-VARIABLES.md +++ b/master-docs/00-system/06-ENVIRONMENT-VARIABLES.md @@ -0,0 +1,91 @@ +# Environment Variables + +## Purpose +List environment variables that influence runtime behavior, taken directly from `settings.py` and related auth/Celery configuration. + +## Code Locations (exact paths) +- Primary definitions and defaults: `backend/igny8_core/settings.py` +- JWT helpers: `backend/igny8_core/auth/utils.py` + +## High-Level Responsibilities +- Configure secrets, debug flags, DB connections, CORS, auth, Celery, and billing providers without code changes. + +## Detailed Behavior +- Core flags: + - `SECRET_KEY` (required for production) + - `DEBUG` (enables debug when set to true) + - `IGNY8_DEBUG_THROTTLE` (bypass rate limiting when true; defaults to `DEBUG`) + - `IGNY8_USE_UNIFIED_EXCEPTION_HANDLER` (controls custom DRF exception handler; enabled by default) + - `USE_SITE_BUILDER_REFACTOR` (feature flag for site builder) + - `USE_SECURE_COOKIES` (sets session/CSRF secure cookies) + - `USE_SECURE_PROXY_HEADER` (enables `SECURE_PROXY_SSL_HEADER`) + - `USE_X_FORWARDED_HOST` is disabled by default; no env provided. +- Database selection (PostgreSQL default with SQLite fallbacks): + - `DATABASE_URL` (parsed for engine/user/pass/host/port/name; supports postgres or sqlite) + - `DB_ENGINE` (forces sqlite when set to sqlite/sqlite3) + - `DJANGO_FORCE_POSTGRES` (force Postgres even in debug) + - `DB_NAME`, `DB_USER`, `DB_PASSWORD`, `DB_HOST`, `DB_PORT` (used when `DATABASE_URL` not provided) + - `USE_SQLITE` and `SQLITE_NAME` (explicit sqlite override) +- CORS/hosts: + - CORS origins are hardcoded; no env toggle beyond `DEBUG`. Trusted CSRF origins are static in settings. +- Auth/JWT: + - `JWT_SECRET_KEY` (defaults to `SECRET_KEY`) + - `JWT_ALGORITHM` (defaults to HS256) + - `JWT_ACCESS_TOKEN_EXPIRY` (timedelta in settings; default 15 minutes) + - `JWT_REFRESH_TOKEN_EXPIRY` (default 30 days) +- Celery/Redis: + - `CELERY_BROKER_URL` (defaults to `redis://{REDIS_HOST}:{REDIS_PORT}/0`) + - `CELERY_RESULT_BACKEND` (defaults to same as broker) + - `REDIS_HOST`, `REDIS_PORT` (defaults redis:6379) + - `REDIS_SENTINEL_ENABLED` (when true, adds sentinel backend options) + - `REDIS_SSL_ENABLED` (enables Redis backend SSL) +- Feature/config paths: + - `PUBLISH_SYNC_LOG_DIR` is derived; no env override. +- Payments: + - `STRIPE_PUBLIC_KEY`, `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET` + - `PAYPAL_CLIENT_ID`, `PAYPAL_CLIENT_SECRET`, `PAYPAL_API_BASE` (defaults to sandbox base) +- Rate limiting and throttles: + - Scopes are defined in settings; toggling is via `IGNY8_DEBUG_THROTTLE`. +- Security/Cookies: + - `USE_SECURE_COOKIES`, `USE_SECURE_PROXY_HEADER` control secure cookie and proxy behavior. + +## Data Structures / Models Involved (no code) +- None; variables configure runtime services used by auth, database, Celery, and billing. + +## Execution Flow +- Django reads env vars at import of `settings.py`; defaults apply when unset. +- Token helpers read JWT secrets/algorithms/expiries for generation and validation. +- Celery settings are consumed by workers/beat at startup. + +## Cross-Module Interactions +- Auth stack uses JWT settings and secrets. +- Account middleware uses plan validation; DB settings drive tenancy storage. +- Celery settings affect AI/automation/publishing/billing tasks. +- Stripe/PayPal keys are consumed by billing modules. + +## State Transitions (if applicable) +- Changes to env vars take effect on process restart; token expiry/secret changes invalidate existing tokens accordingly. + +## Error Handling +- Missing/invalid DB env falls back to defaults (SQLite in debug unless forced). +- Missing JWT secret falls back to `SECRET_KEY`; an absent secret raises errors during decode if unset. + +## Tenancy Rules +- Env vars do not alter tenancy logic; they configure infrastructure supporting it. + +## Billing Rules (if applicable) +- Stripe/PayPal keys must be present for payment webhooks/operations; plan enforcement itself is not env-driven. + +## Background Tasks / Schedulers (if applicable) +- Celery broker/backend URLs and Redis options come from env; task timeouts and serializer are fixed in settings. + +## Key Design Considerations +- Favor explicit env configuration for production (secrets, DB, Redis, Stripe/PayPal). +- Keep debug-only flags (`DEBUG`, `IGNY8_DEBUG_THROTTLE`) off in production. +- Use `DATABASE_URL` for portability; fallback logic supports sqlite for local dev. + +## How Developers Should Work With This Module +- Set required secrets (`SECRET_KEY`, `JWT_SECRET_KEY`, Stripe/PayPal keys) before deploying. +- Choose DB via `DATABASE_URL` or explicit `DB_*` vars; avoid sqlite in production unless intentional. +- Configure Redis URLs for Celery in non-dev environments. +- Restart services after changing env vars to apply new configuration.