diff --git a/master-docs/10-backend/00-BACKEND-ARCHITECTURE.md b/master-docs/10-backend/00-BACKEND-ARCHITECTURE.md index e69de29b..cbf77fd8 100644 --- a/master-docs/10-backend/00-BACKEND-ARCHITECTURE.md +++ b/master-docs/10-backend/00-BACKEND-ARCHITECTURE.md @@ -0,0 +1,82 @@ +# Backend Architecture + +## Purpose +Explain how the backend is structured, wired, and executed across Django/DRF, middleware, routing, async processing, and logging. + +## Code Locations (exact paths) +- Settings and app wiring: `backend/igny8_core/settings.py` +- URL routing: `backend/igny8_core/urls.py` +- Middleware: `backend/igny8_core/middleware/request_id.py`, `backend/igny8_core/auth/middleware.py`, `backend/igny8_core/middleware/resource_tracker.py` +- Auth stack: `backend/igny8_core/api/authentication.py`, `backend/igny8_core/auth/utils.py` +- Base viewsets and unified responses: `backend/igny8_core/api/base.py` +- Domain models: `backend/igny8_core/auth/models.py` plus `backend/igny8_core/business/*/models.py` +- Async/Celery config: `backend/igny8_core/settings.py` +- Logging: `backend/igny8_core/settings.py` (publish/webhook logs), automation logging (`backend/igny8_core/business/automation/services/automation_logger.py`) + +## High-Level Responsibilities +- Django/DRF API surface under `/api/v1/*` for planner, writer, system, billing, automation, linker, optimizer, publisher, and integration modules. +- Middleware establishes request IDs, tenant context, and optional resource tracking before views run. +- DRF configuration standardizes pagination, filtering, authentication, throttling, and exception handling. +- Celery + Redis provide async execution for AI, automation, publishing, and other long-running tasks. +- Logging and CORS/security settings are centralized in settings. + +## Detailed Behavior +- Apps registered in `INSTALLED_APPS` include auth, AI framework, planner, writer, system, billing, automation, optimization, publishing, integration, linker, optimizer, publisher. Custom admin config is loaded first. +- Middleware order: security → WhiteNoise → CORS → session/common/CSRF → Django auth → `RequestIDMiddleware` → `AccountContextMiddleware` (tenant + plan enforcement) → `ResourceTrackingMiddleware` (opt-in for admin/developer with header) → messages → clickjacking. +- Routing (`urls.py`): admin plus CSV helpers for industries/seed keywords, then `/api/v1/` routers for auth, account, planner, writer, system, billing (user/admin), automation, linker, optimizer, publisher, integration. OpenAPI served at `/api/docs` and `/api/redoc`. +- DRF defaults: unified exception handler (unless env disables), custom pagination, filtering/search/ordering, auth stack (API key → JWT → CSRF-exempt session → basic), scoped throttling per operation class, drf-spectacular schema generation with tag ordering. +- CORS: IGNY8 domains and local dev ports allowed; credentials enabled; request/resource tracking headers exposed. +- Celery: Redis broker/result; JSON serialization; task/soft time limits; single-prefetch; sentinel/SSL toggles via env. +- Logging: console plus rotating files for publish/sync, WordPress API, webhooks; request IDs from middleware surface in responses; automation has dedicated file-based logger per run. + +## Data Structures / Models Involved (no code) +- Tenancy/identity: `Account`, `Plan`, `Subscription`, `Site`, `Sector`, `User`, `SiteUserAccess`, `PasswordResetToken` (in auth models). +- Domain models across planner, writer, billing, automation, publishing, integration, optimization (see domain models doc). +- Middleware uses request-scoped `request.account`, `request.site`, and `request.request_id`. + +## Execution Flow +1) Request enters middleware stack; IDs and tenant context are set; plan is verified. +2) DRF auth classes authenticate user/api-key/JWT/session. +3) Base viewsets filter by tenant/site/sector and handle unified responses. +4) Serializers validate; database writes enforce tenant alignment via model bases. +5) Optional Celery tasks are enqueued for long-running work; responses remain synchronous JSON with pagination/throttle headers when applicable. +6) Logging writes to configured handlers; request/resource IDs are added to responses. + +## Cross-Module Interactions +- Tenant context from middleware is consumed by all module viewsets. +- AI and automation invoke Celery tasks and AI functions; billing services deduct credits used by automation/AI flows. +- Integration/publishing modules rely on API key auth to set `request.site` for WordPress and other platforms. + +## State Transitions (if applicable) +- Account/plan validity is checked per request; system accounts are protected from deletion. +- Soft-delete is available on many models via `SoftDeletableModel`. +- Request lifecycle includes request ID creation, optional resource tracking, and unified response formatting. + +## Error Handling +- Custom DRF exception handler wraps errors with `success=false`. +- `AccountContextMiddleware` blocks requests lacking account or active plan (403/402 JSON). +- Base viewsets wrap validation/404/500 errors into unified payloads. + +## Tenancy Rules +- Request-level `account` (and `site`/`sector` where applicable) is injected by middleware/auth; base viewsets enforce filtering. +- Admin/developer/system-account users bypass tenant filtering; system accounts are guarded from deletion. + +## Billing Rules (if applicable) +- Billing env keys configured in settings; plan enforcement occurs in middleware; credit debits handled by billing services during operations. + +## Background Tasks / Schedulers (if applicable) +- Celery worker/beat use Redis URLs from settings; task limits and JSON serialization enforced. +- Automation, AI, publishing, and optimization tasks run async; automation logger writes per-run files. + +## Key Design Considerations +- Middleware-first tenant enforcement ensures isolation. +- Unified API standards (responses, throttles, schema) keep clients consistent. +- Celery offloads expensive AI/automation/publishing work with bounded execution. +- Logging and request IDs aid observability; resource tracking is opt-in for admins. + +## How Developers Should Work With This Module +- Register new apps in `INSTALLED_APPS` and route them under `/api/v1/*`. +- Use existing middleware order; add new middleware only if it does not break account/request ID handling. +- Inherit from base viewsets for scoping and response consistency. +- Use Celery for long-running tasks; respect Redis/task time-limit settings. +- Keep env vars (DB, JWT, Redis, Stripe/PayPal) set per `settings.py`; avoid hardcoded secrets. diff --git a/master-docs/10-backend/01-DOMAIN-MODELS.md b/master-docs/10-backend/01-DOMAIN-MODELS.md index e69de29b..00416c47 100644 --- a/master-docs/10-backend/01-DOMAIN-MODELS.md +++ b/master-docs/10-backend/01-DOMAIN-MODELS.md @@ -0,0 +1,110 @@ +# Domain Models + +## Purpose +Describe the key backend models, their responsibilities, constraints, and tenancy rules, grounded in current implementations. + +## Code Locations (exact paths) +- Tenancy bases and identity: `backend/igny8_core/auth/models.py` +- Planner models: `backend/igny8_core/business/planning/models.py` +- Writer/content models: `backend/igny8_core/business/content/models.py` +- Automation models: `backend/igny8_core/business/automation/models.py` +- Billing models: `backend/igny8_core/business/billing/models.py` +- Integration models: `backend/igny8_core/business/integration/models.py` +- Publishing models: `backend/igny8_core/business/publishing/models.py` +- Optimization models: `backend/igny8_core/business/optimization/models.py` + +## High-Level Responsibilities +- Provide tenant-scoped storage for planner (keywords/clusters/ideas), writer (tasks/content/taxonomies/images/attributes), automation runs/config, billing (credits, invoices, payments), integration metadata, publishing records, and optimization tasks. +- Enforce account/site/sector alignment via base classes and save-time validation. +- Track external platform links (WordPress/Shopify/custom), credit usage, and publishing/optimization state. + +## Detailed Behavior +### Tenancy Bases (auth/models.py) +- `AccountBaseModel`: adds `account`, timestamps, and indexes; all tenant models inherit. +- `SiteSectorBaseModel`: extends with `site` and `sector`; save enforces site → account alignment and sector belonging to the same site; raises validation errors if mismatched. + +### Planner (business/planning/models.py) +- `Clusters`: tenant/site/sector-scoped keyword group; tracks counts, volume, mapped pages, status (`new/mapped`), disable flag; unique per site/sector by name; soft-deletable. +- `Keywords`: tenant/site/sector-scoped keyword tied to a global `SeedKeyword`; optional overrides for volume/difficulty/attributes; optional cluster link (same sector enforced); validation ensures seed keyword industry/sector matches site/sector; status (`new/mapped`), disable flag; soft-deletable. +- `ContentIdeas`: ideas tied to clusters and optional keywords; tracks status (`new/queued/completed`), content type/structure, estimated word count; soft-deletable. + +### Writer / Content (business/content/models.py) +- `Tasks`: queue items for content generation; tied to cluster (required) and optional idea/taxonomy; content type/structure, keywords text, target word count, status (`queued/completed`); soft-deletable. +- `Content`: generated or imported content; stores HTML, word count, SEO fields, cluster link, content type/structure, taxonomy M2M, external IDs/URLs/metadata, sync status, source (`igny8/wordpress`), and status (`draft/review/published`); soft-deletable. +- `ContentTaxonomy`: simplified taxonomy (category/tag) with external taxonomy/ID, sync status, description, count, metadata; unique per site by slug/type and by external ID/taxonomy. +- `Images`: images linked to content or task; auto-populates account/site/sector from the linked object; tracks type, URL/path, prompt, status, position; soft-deletable. +- `ContentClusterMap`: maps content/tasks to clusters with role (`hub/supporting/attribute`) and source (`blueprint/manual/import`); auto-populates tenant context from linked content/task; unique per content+cluster+role. +- `ContentAttribute` (alias `ContentAttributeMap`): tenant/site/sector-scoped attributes for content/task/cluster; typed (`product_spec/service_modifier/semantic_facet`), with optional external IDs, sources, and metadata; auto-populates tenant context from linked content/task. + +### Automation (business/automation/models.py) +- `AutomationConfig`: per-site config with enable flag, frequency (`daily/weekly/monthly`), scheduled time, batch sizes per stage, within/between-stage delays, and next/last run timestamps. +- `AutomationRun`: tracks each run with trigger (`manual/scheduled`), status (`running/paused/cancelled/completed/failed`), current stage, pause/cancel timestamps, start/end, total credits used, per-stage JSON results, and optional error message. + +### Billing (business/billing/models.py) +- `CreditTransaction`: ledger of credit changes (purchase/subscription/refund/deduction/adjustment) with balance-after and metadata. +- `CreditUsageLog`: detailed AI usage log with operation type (clustering/idea/content/image/reparse/legacy names), credits used, optional cost/model/tokens, related object references, and metadata. +- `CreditCostConfig`: admin-configurable credit costs per operation with unit (per request/words/items/images), display metadata, active flag, audit fields, and previous cost tracking. +- `Invoice`: tenant invoice with amounts, status (`draft/pending/paid/void/uncollectible`), dates, subscription link, line items JSON, payment metadata, Stripe IDs, notes; helper properties mirror legacy fields. +- `Payment`: payment records per invoice with status lifecycle (pending/processing/succeeded/completed/failed/refunded/cancelled), method (Stripe/PayPal/bank/local wallet/manual), provider references, manual notes/approval fields, failure reason, timestamps, metadata. +- `CreditPackage`: one-time credit bundles with price, discount, Stripe/PayPal IDs, active/featured flags, description/features, sort order. +- `PaymentMethodConfig`: per-country payment-method availability and display/instruction fields; includes bank/local wallet metadata; unique per country+method. +- `AccountPaymentMethod`: account-level payment metadata (non-sensitive) with type, display name, default/enabled/verified flags, country code, instructions, metadata; unique per account+display name. + +### Integration (business/integration/models.py) +- `SiteIntegration`: tenant/site-specific integration config with platform (`wordpress/shopify/custom`), platform type (`cms/ecommerce/custom_api`), config JSON, credentials JSON, active/sync flags, sync status, last sync/error, timestamps; unique per site+platform. +- `SyncEvent`: event log per integration/site with event/action types, success flag, optional content/external IDs, details JSON, error, duration, and timestamps; indexed for debugging feeds. + +### Publishing (business/publishing/models.py) +- `PublishingRecord`: tracks content publishing to destinations (wordpress/sites/shopify) with destination IDs/URLs, status (`pending/publishing/published/failed`), timestamps, errors, metadata; site/sector scoped via base. +- `DeploymentRecord`: tracks site deployments (sites renderer) with version/deployed_version, status (`pending/deploying/deployed/failed/rolled_back`), deployment URL, error, metadata, timestamps; site/sector scoped. + +### Optimization (business/optimization/models.py) +- `OptimizationTask`: content optimization runs with before/after scores and HTML, status (`pending/running/completed/failed`), credits used, metadata; auto-sets account from content; tenant scoped. + +## Execution Flow +- Tenant context is inherited from base models; many save methods propagate account/site/sector from related entities (e.g., Images, ContentClusterMap, ContentAttribute). +- Planner → Writer linkage: Keywords and Clusters feed ContentIdeas; Tasks reference clusters/ideas; Content references clusters and taxonomies; Images/Attributes link to Tasks/Content. +- Automation runs reference planner/writer models and record per-stage outputs; configs control batching/delays. +- Billing logs and cost configs govern credit debits triggered by services (see services doc). +- Integration/publishing models bind site integrations and publishing deployments to site-scoped content. +- Optimization tasks attach to content and capture before/after artifacts. + +## Cross-Module Interactions +- Planner and writer share clusters/ideas/tasks/content relationships. +- Billing models are invoked by services during AI/automation/image/content operations. +- Integration events reference content IDs and external IDs for sync traces. +- Publishing records reference writer content; deployment records reference sites. +- Optimization tasks reference writer content and can influence publishing readiness downstream. + +## State Transitions (if applicable) +- Soft-delete is available on planner keywords/clusters/ideas and writer tasks/content/images via `SoftDeletableModel`. +- Status fields track lifecycle: planner (`new/mapped/queued/completed`), writer tasks (`queued/completed`), content (`draft/review/published`), automation (`running/paused/cancelled/completed/failed`), publishing/deployment statuses, payment/invoice statuses, optimization statuses. + +## Error Handling +- Save-time validation in `SiteSectorBaseModel` and `Keywords` ensures tenant/site/sector alignment and industry/sector matching. +- Unique constraints prevent duplicate clusters/keywords per site/sector and overlapping taxonomies/external IDs. +- Automation runs store error messages and partial stage results; publishing/deployment records store error text. + +## Tenancy Rules +- All models shown are tenant scoped via `AccountBaseModel` or `SiteSectorBaseModel`; save hooks propagate context from related objects where needed. +- Privileged roles can bypass filtering at the viewset layer, but persisted records retain account/site/sector ownership. + +## Billing Rules (if applicable) +- Credits reside on `Account`; transactions/usage logs record debits/credits; cost configs define per-operation pricing. +- Invoices/payments/credit packages configure monetary flows; payment methods can be toggled per country or per account. + +## Background Tasks / Schedulers (if applicable) +- Automation configs drive scheduled runs; automation runs record stage outputs and timing. +- Publishing/optimization tasks may be executed async via services/Celery (see services doc). + +## Key Design Considerations +- Tenant isolation is encoded at the model layer via base classes and validation, ensuring downstream services inherit scoping. +- Cross-module links (clusters ↔ tasks ↔ content ↔ publishing/optimization) keep content lifecycle traceable. +- Billing and integration models include metadata fields to avoid schema churn while capturing provider-specific details. + +## How Developers Should Work With This Module +- Inherit new tenant models from `AccountBaseModel` or `SiteSectorBaseModel` to enforce scoping automatically. +- Validate cross-entity alignment (site/sector/industry) when relating planner and writer records. +- Use existing status fields/choices when extending lifecycles; preserve unique constraints when adding fields. +- When integrating new providers, extend or add models parallel to `SiteIntegration`/`SyncEvent` and keep platform-specific data in JSON fields. + diff --git a/master-docs/10-backend/02-SERVICES-AND-MODULES.md b/master-docs/10-backend/02-SERVICES-AND-MODULES.md index e69de29b..7f06ded5 100644 --- a/master-docs/10-backend/02-SERVICES-AND-MODULES.md +++ b/master-docs/10-backend/02-SERVICES-AND-MODULES.md @@ -0,0 +1,96 @@ +# Services and Modules + +## Purpose +Describe the backend service layer and module wiring that orchestrate domain models, AI/automation, billing, integration, and publishing. + +## Code Locations (exact paths) +- Automation services: `backend/igny8_core/business/automation/services/automation_service.py`, `automation_logger.py` +- Billing services: `backend/igny8_core/business/billing/services/credit_service.py`, `invoice_service.py`, `payment_service.py` +- Integration service: `backend/igny8_core/business/integration/services/integration_service.py` +- Additional service directories (see module-specific docs for details): + - Planner: `backend/igny8_core/business/planning/services/*` + - Content/Writer: `backend/igny8_core/business/content/services/*` + - Automation tasks: `backend/igny8_core/business/automation/tasks.py` + - Integration sync: `backend/igny8_core/business/integration/services/*` + - Publishing: `backend/igny8_core/business/publishing/services/*` + - Optimization: `backend/igny8_core/business/optimization/services/*` + - Linking: `backend/igny8_core/business/linking/services/*` + +## High-Level Responsibilities +- Orchestrate multi-stage automation that chains planner and writer operations using AI and credits. +- Manage credit pricing, balance checks, deductions, and ledger logging. +- Generate invoices and handle billing documents. +- Create, update, test, and list site integrations for external platforms. +- Provide domain-specific service hooks for planner, writer, publishing, linking, and optimization flows (behavior documented in module-specific files). + +## Detailed Behavior +### Automation +- `AutomationService` enforces single concurrent run per site via cache lock, estimates required credits, checks account balance (with buffer), and creates an `AutomationRun`. It sequences stages (keywords→clusters, clusters→ideas, ideas→tasks/content, image prompt generation, image generation queue) using AI functions (`AutoClusterFunction`, `GenerateIdeasFunction`, `GenerateContentFunction`, `GenerateImagePromptsFunction`) through `AIEngine`, and the Celery image queue (`process_image_generation_queue`). It supports pause/cancel checks mid-stage, records partial progress, and advances `current_stage` with per-stage result JSON and credit tallies. Batch sizes and delays respect `AutomationConfig`. +- `AutomationLogger` creates per-run directories (and optional shared mirrors), generates run IDs, writes main/stage logs, mirrors to shared folders when configured, and emits structured JSONL trace events for run start, progress, completion, and errors. + +### Billing +- `CreditService` computes credit cost by first consulting `CreditCostConfig` (unit-aware for words/items/images) and falling back to constants. It checks balances, deducts credits atomically, updates account balance, writes `CreditTransaction`, and logs usage in `CreditUsageLog` with optional model/token metadata. It can also add credits and provides legacy check helpers. +- `InvoiceService` generates invoice numbers per account/month, creates invoices for subscriptions or credit packages (adding line items and computing totals), supports custom invoices, and marks invoices paid or void. PDF generation is currently a placeholder. +- `PaymentService` (see file) processes payments against invoices and updates statuses; details are documented in the module file. + +### Integration +- `IntegrationService` creates, updates, deletes, fetches, and lists `SiteIntegration` records, setting account/site automatically. It can test connections per platform (WordPress, Shopify) and delegates to platform-specific test helpers inside the service; unimplemented platforms raise `NotImplementedError`. Credentials are set via `set_credentials`, which currently stores JSON as-is. +- Additional sync services (`content_sync_service.py`, `sync_service.py`, `sync_metadata_service.py`, `sync_health_service.py`) coordinate publish/sync flows and health checks; see module docs for specifics. + +### Other Service Areas (structure) +- Planner services (`clustering_service.py`, `ideas_service.py`) handle clustering/idea logic. +- Content services (`content_generation_service.py`, `content_pipeline_service.py`, `metadata_mapping_service.py`, `validation_service.py`) manage content generation, pipelines, metadata mapping, and validation. +- Publishing services (`publisher_service.py`, `deployment_service.py`, `adapters/`) manage publishing/deployment flows and destination adapters. +- Optimization services (`analyzer.py`, `optimizer_service.py`) analyze and optimize content. +- Linking services (`candidate_engine.py`, `injection_engine.py`, `linker_service.py`) prepare and apply link suggestions. +- Automation tasks (`business/automation/tasks.py`) provide Celery entrypoints for automation runs. + +## Execution Flow +- Automation: `AutomationService.start_automation` acquires lock → credit estimate/check → create `AutomationRun` → stage methods query planner/writer models, call AI functions via `AIEngine`, respect batch sizes/delays, and update run state/logs → credits are tallied from `AITaskLog` differences. +- Billing: operations call `CreditService.check_*`/`deduct_*` before AI or content operations; invoices are created through `InvoiceService` and payments processed via `PaymentService`. +- Integration: API endpoints invoke `IntegrationService` to persist integrations, retrieve lists, and run connection tests; sync services handle subsequent data movement. +- Other domains: planner/content/publishing/linking/optimization services orchestrate their models and, where applicable, AI or external adapters; see domain docs for invocation points. + +## Cross-Module Interactions +- Automation stages consume planner (Keywords/Clusters/ContentIdeas) and writer (Tasks/Content/Images) data and rely on credit usage logs from AI tasks. +- Billing services are used by automation/AI flows to enforce credit availability and record deductions. +- Integration services connect site data and publishing/sync flows to external platforms; publishing services depend on integration metadata when targeting destinations. +- Planner/content services feed data used by publishing and optimization tasks. + +## State Transitions (if applicable) +- Automation runs move through stages, can pause/cancel, and record partial progress with stage result JSON. +- Credit balances mutate through add/deduct operations; transactions/usage logs capture each change. +- Invoices progress through draft/pending/paid/void and payments through their status lifecycle. +- Integrations toggle active/sync flags and update sync status/errors. + +## Error Handling +- Automation: checks for concurrent runs; validates minimum keywords; pauses/cancels mid-stage; writes stage error messages; releases locks on failures. +- Billing: raises when credit cost unknown or balance insufficient; wraps changes in atomic transactions. +- Integration: platform test errors are logged and returned with `success=false`; unsupported platforms raise `NotImplementedError`. +- Invoice service prevents voiding paid invoices and returns placeholder PDF until implemented. + +## Tenancy Rules +- Services operate on tenant-scoped models; constructors typically receive account/site or derive them from models. Integration creation sets account from site; credit operations mutate `Account.credits`. +- Privileged role bypass applies at the viewset layer; persisted records maintain account/site ownership. + +## Billing Rules (if applicable) +- Costs resolved via `CreditCostConfig` (preferred) or constants; units can be per request, words (100/200), item, or image. +- Deduct operations both adjust `Account.credits` and log to `CreditTransaction` and `CreditUsageLog`. +- Invoice creation links to subscriptions or credit packages and uses account billing email/plan pricing. + +## Background Tasks / Schedulers (if applicable) +- Automation uses Celery for image generation and may be triggered by scheduled runs (frequency/time in `AutomationConfig`). +- Other long-running tasks (publishing, optimization, sync) are handled via their Celery tasks/adapters in respective service modules. + +## Key Design Considerations +- Automation enforces exclusivity per site and accounts for credit sufficiency before starting. +- Logging (AutomationLogger) produces per-run artifacts and structured traces for observability. +- Credit handling is centralized to keep ledger and usage logs consistent and atomic. +- Integration services abstract platform handling and allow per-platform test logic. + +## How Developers Should Work With This Module +- Reuse `AutomationService` for running/continuing automation; respect locks and stage APIs. +- Use `CreditService` before AI/content-heavy operations to check/deduct/add credits and to log usage. +- Create invoices via `InvoiceService` helpers rather than constructing manually; update invoice/payment status through service methods. +- Manage integrations through `IntegrationService` (create/update/test/list) and extend platform-specific tests as needed. +- For domain-specific flows (planner/content/publishing/linking/optimization), place orchestration in the existing service modules and keep tenant context explicit. diff --git a/master-docs/10-backend/automation/AUTOMATION-LOGGING.md b/master-docs/10-backend/automation/AUTOMATION-LOGGING.md index e69de29b..08f3d42e 100644 --- a/master-docs/10-backend/automation/AUTOMATION-LOGGING.md +++ b/master-docs/10-backend/automation/AUTOMATION-LOGGING.md @@ -0,0 +1,62 @@ +# Automation Logging + +## Purpose +Explain how automation runs are logged, where artifacts are written, and how logs are retrieved via API. + +## Code Locations (exact paths) +- Logger implementation: `backend/igny8_core/business/automation/services/automation_logger.py` +- Service usage: `backend/igny8_core/business/automation/services/automation_service.py` +- API exposure: `backend/igny8_core/business/automation/views.py` (`logs` action) +- Run metadata: `backend/igny8_core/business/automation/models.py` + +## High-Level Responsibilities +- Create per-run directories and log files for automation runs (primary and optional shared mirror). +- Record stage-level start/progress/complete events and structured traces. +- Provide tail retrieval of logs through an API endpoint. + +## Detailed Behavior +- `AutomationLogger`: + - On `start_run`: generates `run_id` (`run_YYYYMMDD_HHMMSS_`), creates run directory under `base_log_dir` (default `/data/app/logs/automation`), and optionally mirrors to `shared_log_dir` when configured. Initializes `automation_run.log` with run metadata and writes a JSONL trace entry (`run_trace.jsonl`). + - `log_stage_start`: appends to main log and writes a stage-specific log file `stage_.log`; mirrors to shared dir if configured; writes a trace event (`stage_start` with pending count). + - `log_stage_progress`: appends progress lines to main and stage logs (and mirrored copies), and writes a `stage_progress` trace event. + - `log_stage_complete`: writes completion info with processed count, time elapsed, credits used to main/stage logs; emits trace (`stage_complete`). + - `log_stage_error`: writes error lines to logs and emits an error trace. + - `append_trace`: helper used by the service for per-item events; appends JSONL entries. + - `get_activity_log`: reads and returns the tail of `automation_run.log` (used by API). +- `AutomationService` uses the logger at every stage boundary and significant progress point; also logs warnings/errors and mirrors events via the logger’s trace helpers. +- Stage result data is stored in `AutomationRun.stage_n_result` JSON fields alongside logs, providing structured state in the DB. + +## Data Structures / Models Involved (no code) +- `AutomationRun`: stores per-stage result JSON and run status used to correlate with log files. + +## Execution Flow +- Run start → `start_run` creates directories/files → each stage emits start/progress/complete/error logs and traces → API `logs` endpoint reads `automation_run.log` tail for a given `run_id`. + +## Cross-Module Interactions +- Logging uses the automation run/account/site context; no external modules write to these files. AI stages provide progress data; billing is inferred from AI task logs but not logged here. + +## State Transitions +- Log files accumulate throughout a run; traces document run lifecycle events. Completion or failure is also reflected in `AutomationRun` status/results. + +## Error Handling +- Logger operations are best-effort; exceptions during file/trace writes are swallowed to avoid breaking the run. +- API `logs` returns 404 if the run does not exist; otherwise returns the requested number of lines. + +## Tenancy Rules +- Run directories are keyed by account/site/run; API access is constrained by site ownership through the viewset query for `AutomationRun`. + +## Billing Rules +- None directly; credit use is recorded in stage result JSON and inferred from AI task logs, not in the file logs. + +## Background Tasks / Schedulers +- Logging occurs inside Celery-executed stages; no separate scheduled logging tasks. + +## Key Design Considerations +- File-based logs per run make multi-stage processes auditable and debuggable without DB bloat. +- Optional shared log dir supports centralized aggregation when multiple containers run workers. +- JSONL traces allow machine parsing for UI status feeds while text logs support human debugging. + +## How Developers Should Work With This Module +- When adding stage steps, emit progress via `AutomationLogger` to keep observability consistent. +- Keep log writes best-effort; do not raise from logger helpers. +- Ensure any new API log endpoints read from the same per-run directory structure and remain site/account scoped. diff --git a/master-docs/10-backend/automation/AUTOMATION-MODULE-REFERENCE.md b/master-docs/10-backend/automation/AUTOMATION-MODULE-REFERENCE.md index e69de29b..60e1cc5b 100644 --- a/master-docs/10-backend/automation/AUTOMATION-MODULE-REFERENCE.md +++ b/master-docs/10-backend/automation/AUTOMATION-MODULE-REFERENCE.md @@ -0,0 +1,91 @@ +# Automation Module Reference + +## Purpose +Document how the automation module orchestrates multi-stage AI pipelines, exposes API endpoints, enforces tenancy/credits, and manages runs, configs, and logging. + +## Code Locations (exact paths) +- Models: `backend/igny8_core/business/automation/models.py` +- Services: `backend/igny8_core/business/automation/services/automation_service.py`, `automation_logger.py` +- Tasks (Celery): `backend/igny8_core/business/automation/tasks.py` +- API views and routing: `backend/igny8_core/business/automation/views.py`, `urls.py` +- Supporting AI functions: `backend/igny8_core/ai/functions/auto_cluster.py`, `generate_ideas.py`, `generate_content.py`, `generate_image_prompts.py`, image queue in `backend/igny8_core/ai/tasks.py` +- Tenancy/auth context: `backend/igny8_core/auth/middleware.py`, `backend/igny8_core/api/base.py` + +## High-Level Responsibilities +- Maintain per-site automation configs (batch sizes, delays, schedule, enable flag) and track run state with detailed per-stage results. +- Provide APIs to configure, trigger, pause/resume/cancel, inspect, and log automation runs. +- Execute seven sequential stages that transform planner/writer data via AI and local operations, with credit checks and pause/cancel handling. +- Enforce tenant/site scoping on all automation resources and API operations. + +## Detailed Behavior +- `AutomationConfig` stores enablement, frequency, scheduled time, batch sizes for stages 1–6, and within/between-stage delays. Config is created lazily per site. +- `AutomationRun` captures run metadata: trigger type (manual/scheduled), status (`running/paused/cancelled/completed/failed`), current stage, pause/cancel timestamps, per-stage JSON results, total credits used, and error message. +- `AutomationService` orchestrates the pipeline: + - Locks per site via cache (`automation_lock_{site.id}`) to prevent concurrent runs. + - Estimates credits before start and requires a 20% buffer over the estimate against `Account.credits`. + - Creates `AutomationRun` with generated `run_id` and logs start via `AutomationLogger`. + - Executes stages in order; each stage logs start/progress/complete, applies within/between-stage delays from config, and writes stage result JSON (counts, credits, timestamps, partial flags). + - Pause/cancel checks occur inside loops; state is persisted so resumed runs continue from the recorded stage. + - Stage credit usage is derived from AI task logs difference before/after the stage. +- API layer (`AutomationViewSet`): + - `config`/`update_config` read/write `AutomationConfig` for a given `site_id` (scoped to the user’s account). + - `run_now` triggers `AutomationService.start_automation` and enqueues Celery `run_automation_task`. + - `current_run`, `history`, `logs`, `current_processing`, `estimate`, `pipeline_overview` expose run status, history, logs, credit estimates, and per-stage pending counts. + - `pause`, `resume`, `cancel` endpoints update run status and enqueue resume tasks when needed. +- Celery tasks: + - `check_scheduled_automations` scans enabled configs hourly and triggers runs when frequency/time matches and no recent run exists. + - `run_automation_task` performs full pipeline execution. + - `resume_automation_task`/`continue_automation_task` continue a paused run from its recorded stage. + +## Data Structures / Models Involved (no code) +- `AutomationConfig`, `AutomationRun` (automation state). +- Planner models: `Keywords`, `Clusters`, `ContentIdeas`. +- Writer models: `Tasks`, `Content`, `Images`. +- AI task log (`AITaskLog`) for credit usage measurement. +- Tenancy entities: `Account`, `Site` (scoping every query). + +## Execution Flow +- API call → DRF auth → tenant/site resolved → viewset method → `AutomationService` operations → Celery task (for long-running execution). +- Pipeline stages run in-process inside Celery workers, reading planner/writer data, invoking AI functions, updating models, logging progress, and writing stage results to `AutomationRun`. +- Completion (or failure) updates run status and releases the site lock. + +## Cross-Module Interactions +- Planner/writer models supply inputs and receive outputs (clusters, ideas, tasks, content, images). +- AI engine executes clustering, idea generation, content generation, and image prompt generation; image rendering uses the AI image queue. +- Billing credits are checked against `Account.credits`; credit usage is inferred from AI task logs (deduction logic handled in billing services when those AI calls occur). +- Integration/publishing modules consume content/images produced downstream (outside automation). + +## State Transitions +- Run status moves through `running` → (`paused`/`cancelled`/`failed`/`completed`); `current_stage` increments after each stage finishes; partial flags and timestamps mark mid-stage exits. +- Config changes take effect on the next run; pause/resume toggles update run timestamps. + +## Error Handling +- Start blocks if a run is already active for the site or cache lock is held. +- Stage loops log and continue on per-batch/item errors; pause/cancel results are persisted mid-stage. +- Failures in Celery run mark `AutomationRun` as failed, store error message, timestamp completion, and release the lock. +- API endpoints return 400 for missing params or invalid state transitions, 404 for unknown runs, 500 on unexpected errors. + +## Tenancy Rules +- All automation queries filter by `site` tied to the authenticated user’s `account`; config/run creation sets `account` and `site` explicitly. +- API endpoints fetch `Site` with `account=request.user.account`; automation locks are per site. +- No cross-tenant access; privileged role bypass is handled by DRF auth/permissions upstream. + +## Billing Rules +- Start requires `Account.credits` ≥ 1.2× estimated credits; otherwise a 400 is returned. +- Credits actually deducted by AI tasks are reflected via AI task logs and billing services (outside this module); automation aggregates usage per stage in `AutomationRun`. + +## Background Tasks / Schedulers +- Hourly `check_scheduled_automations` respects config frequency/time and last run; skips if a run is already active. +- Pipeline execution and resume steps run inside Celery tasks; within-stage sleeps apply delays from config. + +## Key Design Considerations +- Single-run-per-site enforced via cache lock to prevent overlapping credit use or data contention. +- Pause/resume/cancel is cooperative, checked inside stage loops, with partial results persisted. +- Stage-by-stage logging and result JSON make pipeline progress observable and resumable. +- Configurable batch sizes and delays balance throughput and API/credit usage. + +## How Developers Should Work With This Module +- Use `AutomationService.start_automation` for new runs; never bypass the cache lock or credit check. +- When extending stages, preserve pause/cancel checks, result recording, and credit delta calculation. +- Add new API actions through `AutomationViewSet` if they manipulate automation state; keep site/account scoping. +- For new schedulers, reuse the lock pattern and `AutomationConfig` fields, and update `next_run_at` appropriately. diff --git a/master-docs/10-backend/automation/AUTOMATION-PIPELINE-STAGES.md b/master-docs/10-backend/automation/AUTOMATION-PIPELINE-STAGES.md index e69de29b..df2bbeca 100644 --- a/master-docs/10-backend/automation/AUTOMATION-PIPELINE-STAGES.md +++ b/master-docs/10-backend/automation/AUTOMATION-PIPELINE-STAGES.md @@ -0,0 +1,102 @@ +# Automation Pipeline Stages + +## Purpose +Detail the seven pipeline stages executed by `AutomationService`, including inputs, queries, validations, delays, credit handling, and state recording. + +## Code Locations (exact paths) +- Orchestration: `backend/igny8_core/business/automation/services/automation_service.py` +- Models: `backend/igny8_core/business/automation/models.py` +- AI functions: `backend/igny8_core/ai/functions/auto_cluster.py`, `generate_ideas.py`, `generate_content.py`, `generate_image_prompts.py` +- Image queue: `backend/igny8_core/ai/tasks.py` (`process_image_generation_queue`) +- Stage entrypoints: `backend/igny8_core/business/automation/tasks.py` (Celery `run_automation_task`, `resume_automation_task`) + +## High-Level Responsibilities +- Execute a fixed seven-stage sequence that moves data from planner keywords through content with images and into manual review readiness. +- Enforce batch sizes/delays from `AutomationConfig`, support pause/cancel, and write per-stage results into `AutomationRun`. +- Track credit deltas per stage using AI task log counts. + +## Detailed Behavior +Across all stages: +- Each stage logs start/progress/complete via `AutomationLogger`, respects `within_stage_delay` between batches/items, and `between_stage_delay` between stages. +- Pause/cancel is checked inside loops; on pause/cancel, the stage records partial counts, credits, elapsed time, and reason, then exits. +- Credits used per stage are computed from `AITaskLog` count delta relative to stage start. + +### Stage 1: Keywords → Clusters (AI) +- Input query: `Keywords` where `site=current`, `status='new'`, `cluster__isnull=True`, `disabled=False`. +- Validation: `validate_minimum_keywords` requires at least 5 keywords; if not valid, stage is skipped with result noting skip reason and `current_stage` advances to 2. +- Processing: Batch size = `stage_1_batch_size` (capped to total). For each batch, calls `AIEngine.execute(AutoClusterFunction, payload={'ids': batch})`; waits on task ID; logs per-batch progress. Errors are logged and skipped; pipeline continues. +- Result: counts keywords processed, clusters created since run start, batches, credits used, time elapsed; sets `current_stage=2`. + +### Stage 2: Clusters → Ideas (AI) +- Pre-check: warns if any `Keywords` still pending from Stage 1. +- Input query: `Clusters` where `site=current`, `status='new'`, `disabled=False`. +- Processing: Iterates clusters one-by-one; for each, calls `AIEngine.execute(GenerateIdeasFunction, payload={'cluster_id': cluster.id})`; waits on task ID; logs progress. Errors are logged and skipped. +- Result: counts clusters processed, ideas created since run start, credits used, time elapsed; sets `current_stage=3`. + +### Stage 3: Ideas → Tasks (Local) +- Pre-check: warns if clusters remain without ideas. +- Input query: `ContentIdeas` where `site=current`, `status='new'`. +- Processing: Batched by `stage_3_batch_size`. For each idea, builds keyword string (M2M keywords or `target_keywords`) and creates `Tasks` with queued status, copying account/site/sector, cluster, content type/structure, and description. Idea status set to `queued`. +- Result: ideas processed, tasks created, batches, time elapsed (credits 0 because local); sets `current_stage=4`. + +### Stage 4: Tasks → Content (AI) +- Pre-check: warns if `ContentIdeas` remain `new`. +- Input query: `Tasks` where `site=current`, `status='queued'`. +- Processing: Batched by `stage_4_batch_size`. Uses `GenerateContentFunction` via `AIEngine` per batch (payload contains task IDs). Waits on task IDs, logs progress, continues on errors. Tracks total words by summing generated content word_count. +- Result: tasks processed, content created count, total_words, credits used, time elapsed; sets `current_stage=5`. + +### Stage 5: Content → Image Prompts (AI) +- Input query: `Content` where `site=current`, `status='draft'`, with zero images (annotated count=0). +- Processing: Batched by `stage_5_batch_size`. For each batch, calls `GenerateImagePromptsFunction` via `AIEngine` (payload content IDs). Waits on task IDs, logs progress; continues on errors. +- Result: content processed, prompts created (from AI task logs), credits used, time elapsed; sets `current_stage=6`. + +### Stage 6: Image Prompts → Images (AI image queue) +- Input query: `Images` where `site=current`, `status='pending'`. +- Processing: Iterates pending images; for each, enqueues `process_image_generation_queue.delay(image_ids=[id], account_id, content_id)` when Celery is available, or calls directly in sync fallback. Waits on task IDs with continue-on-error to avoid blocking the stage. Logs progress per image; applies within-stage delay between images. +- Result: images processed, images generated (status `generated` since run start), content moved to `review`, credits used, time elapsed; sets `current_stage=7`. + +### Stage 7: Manual Review Gate (Count-only) +- Input query: `Content` where `site=current`, `status='review'`. +- Processing: Counts review-ready content, logs IDs (truncated), marks run `status='completed'`, sets `completed_at`, and releases the site lock. +- Result: ready_for_review count and content IDs stored in `stage_7_result`. + +## Execution Flow +- Celery task `run_automation_task` instantiates `AutomationService.from_run_id` and calls stages 1→7 sequentially. +- Stage transitions update `AutomationRun.current_stage`; between-stage delays applied via `between_stage_delay`. +- Resume path (`resume_automation_task`) starts from the recorded `current_stage` and continues through remaining stages. + +## Cross-Module Interactions +- Planner: Stage 1/2 use `Keywords`/`Clusters`; Stage 3 converts `ContentIdeas` into `Tasks`. +- Writer: Stages 4–6 create `Content` and `Images` and move content toward review. +- AI engine and functions are invoked in Stages 1, 2, 4, 5; Stage 6 uses the AI image queue. +- Billing: Credits are consumed by AI calls; automation records deltas per stage from AI task logs. + +## State Transitions +- `AutomationRun.status` moves to `completed` at Stage 7; can be set to `failed` on exceptions or `cancelled` via API; `paused` can be set mid-run and resumed. +- `current_stage` increments after each successful stage; partial stage results include a `partial` flag and stop reason. +- Domain models change status along the pipeline (`Keywords` → clusters, `Clusters` → ideas, `ContentIdeas` → queued/tasks, `Tasks` → completed/content, `Content` → draft/review, `Images` → generated). + +## Error Handling +- Each stage logs errors and continues to next batch/item; pause/cancel checks short-circuit with partial results saved. +- Task wait helper tolerates Celery backend errors; can continue on error when flagged. +- Stage start may be skipped with explicit skip reason (e.g., insufficient keywords). + +## Tenancy Rules +- All queries filter by `site` (and implicit account via tenancy bases); account/site set on created `Tasks` and inherited on `Images` and other records through model save hooks. +- Locks and runs are per site; API scoping requires the authenticated user’s account to own the site. + +## Billing Rules +- Start requires sufficient credits (1.2× estimate). Credits used are inferred from AI task log counts per stage; actual deductions occur in AI/billing services invoked by the AI functions. + +## Background Tasks / Schedulers +- Entire stage chain runs inside Celery workers; within-stage sleeps respect config delays; between-stage sleeps applied after each stage. + +## Key Design Considerations +- Idempotent, resume-capable progression with partial state persisted in `AutomationRun`. +- Configurable batch sizes/delays mitigate rate limits and manage credit burn. +- Continue-on-error semantics prevent single failures from stopping the pipeline while still recording issues. + +## How Developers Should Work With This Module +- When modifying stages, keep pause/cancel checks, stage result recording, and credit delta calculation. +- Add new AI stages by wiring through `AIEngine.execute` and the task wait helper; ensure queries are site-scoped and statuses updated. +- For new items types, add pending queries and status transitions consistent with existing patterns. diff --git a/master-docs/10-backend/automation/AUTOMATION-SCHEDULER.md b/master-docs/10-backend/automation/AUTOMATION-SCHEDULER.md index e69de29b..1e327a15 100644 --- a/master-docs/10-backend/automation/AUTOMATION-SCHEDULER.md +++ b/master-docs/10-backend/automation/AUTOMATION-SCHEDULER.md @@ -0,0 +1,75 @@ +# Automation Scheduler + +## Purpose +Describe how scheduled runs are detected, triggered, and resumed using Celery tasks and automation configs. + +## Code Locations (exact paths) +- Celery tasks: `backend/igny8_core/business/automation/tasks.py` +- Models: `backend/igny8_core/business/automation/models.py` +- Service invoked: `backend/igny8_core/business/automation/services/automation_service.py` + +## High-Level Responsibilities +- Periodically scan enabled automation configs to start scheduled runs. +- Prevent overlapping runs per site via cache locks and active run checks. +- Resume paused runs from their recorded stage. + +## Detailed Behavior +- `check_scheduled_automations` (Celery, hourly): + - Iterates `AutomationConfig` with `is_enabled=True`. + - Frequency rules: + - `daily`: run when current hour matches `scheduled_time.hour`. + - `weekly`: run Mondays at the scheduled hour. + - `monthly`: run on the 1st of the month at the scheduled hour. + - Skips if `last_run_at` is within ~23 hours or if an `AutomationRun` with `status='running'` exists for the site. + - On trigger: instantiates `AutomationService(account, site)`, calls `start_automation(trigger_type='scheduled')`, updates `last_run_at` and `next_run_at` (via `_calculate_next_run`), saves config, and enqueues `run_automation_task.delay(run_id)`. + - Exceptions are logged per site; lock release is handled by the service on failure paths. +- `run_automation_task`: + - Loads service via `from_run_id`, runs stages 1–7 sequentially. + - On exception: marks run failed, records error/completed_at, and deletes site lock. +- `resume_automation_task` / alias `continue_automation_task`: + - Loads service via `from_run_id`, uses `current_stage` to continue remaining stages. + - On exception: marks run failed, records error/completed_at. +- `_calculate_next_run`: + - Computes next run datetime based on frequency and `scheduled_time`, resetting seconds/microseconds; handles month rollover for monthly frequency. + +## Data Structures / Models Involved (no code) +- `AutomationConfig`: contains schedule fields (`frequency`, `scheduled_time`, `last_run_at`, `next_run_at`, `is_enabled`). +- `AutomationRun`: records run status/stage used during resume/failure handling. + +## Execution Flow +1) Celery beat (or cron) invokes `check_scheduled_automations` hourly. +2) Eligible configs spawn new runs via `AutomationService.start_automation` (includes lock + credit check). +3) `run_automation_task` executes the pipeline asynchronously. +4) Paused runs can be resumed by enqueueing `resume_automation_task`/`continue_automation_task`, which restart at `current_stage`. +5) Failures set run status to `failed` and release locks. + +## Cross-Module Interactions +- Uses planner/writer data inside the pipeline (see pipeline doc); billing/credits enforced at start. +- Locking is done via Django cache, independent of other modules but prevents concurrent Celery runs per site. + +## State Transitions +- Config timestamps (`last_run_at`, `next_run_at`) update on scheduled launch. +- Run status changes to `failed` on task exceptions; to `completed` at stage 7; to `paused/cancelled` via API. + +## Error Handling +- Scheduled start is skipped with log messages if recently run or already running. +- Exceptions during run execution mark the run failed, record error message, set `completed_at`, and release the cache lock. + +## Tenancy Rules +- Configs and runs are site- and account-scoped; scheduler uses stored account/site from the config; no cross-tenant scheduling. + +## Billing Rules +- Start uses `AutomationService.start_automation`, which enforces credit sufficiency before scheduling the Celery execution. + +## Background Tasks / Schedulers +- Hourly `check_scheduled_automations` plus the long-running `run_automation_task` and resume tasks run in Celery workers. + +## Key Design Considerations +- Hourly scan with coarse matching keeps implementation simple while honoring per-site schedules. +- Cache lock and active-run checks prevent double-starts from overlapping schedules or manual triggers. +- Resume task reuses the same stage methods to keep behavior consistent between fresh and resumed runs. + +## How Developers Should Work With This Module +- When adding new frequencies, extend `check_scheduled_automations` and `_calculate_next_run` consistently. +- Ensure Celery beat (or an equivalent scheduler) runs `check_scheduled_automations` hourly in production. +- Preserve lock acquisition and failure handling when modifying task flows to avoid orphaned locks. diff --git a/master-docs/10-backend/billing/BILLING-MODULE-REFERENCE.md b/master-docs/10-backend/billing/BILLING-MODULE-REFERENCE.md index e69de29b..b0f891d2 100644 --- a/master-docs/10-backend/billing/BILLING-MODULE-REFERENCE.md +++ b/master-docs/10-backend/billing/BILLING-MODULE-REFERENCE.md @@ -0,0 +1,98 @@ +# Billing Module Reference + +## Purpose +Explain how billing is implemented: invoices, payments, credit packages, credit history/usage, payment methods, and limits, as exposed by billing services and API viewsets. + +## Code Locations (exact paths) +- Models: `backend/igny8_core/business/billing/models.py` +- Services: `backend/igny8_core/business/billing/services/credit_service.py`, `invoice_service.py`, `payment_service.py` +- API (business billing): `backend/igny8_core/business/billing/views.py`, `backend/igny8_core/business/billing/urls.py` +- API (core billing endpoints): `backend/igny8_core/modules/billing/views.py`, `backend/igny8_core/modules/billing/urls.py` +- Plan metadata (included credits/limits): `backend/igny8_core/auth/models.py` (`Plan`, `Account`) + +## High-Level Responsibilities +- Maintain credit ledger, usage logs, configurable per-operation costs, invoices, payments, credit packages, country-level payment method configs, and account-scoped payment methods. +- Expose endpoints for invoices/payments/credit packages/transactions, credit balance/usage/limits, and admin billing operations. +- Deduct or add credits atomically while recording both transactions and usage logs. +- Provide manual and (placeholder) Stripe/PayPal payment flows. + +## Detailed Behavior +- Invoices (`InvoiceViewSet`): + - `list`/`retrieve`: fetch account-scoped invoices; include totals, line items, billing period, dates. + - `download_pdf`: returns PDF bytes from `InvoiceService.generate_pdf` (currently placeholder text payload). +- Payments (`PaymentViewSet`): + - `list`: account-scoped payments with amounts, methods, status, invoice linkage, timestamps. + - `available_methods`: delegates to `PaymentService.get_available_payment_methods` (country/config-driven; returns methods plus metadata). + - `create_manual_payment`: creates a manual payment for an invoice (bank/local wallet); requires invoice_id, method, reference; rejects already-paid invoices; returns pending-approval status. +- Account payment methods (`AccountPaymentMethodViewSet`): + - CRUD for account-level payment metadata (non-sensitive). `perform_create`/`perform_update` enforce one default by resetting others when `is_default` true or none exists. + - `set_default` toggles default within the account. +- Credit packages (`CreditPackageViewSet`): + - `list`: active packages with credits/price/discount/featured/sort order. + - `purchase`: creates invoice via `InvoiceService.create_credit_package_invoice`; returns next action depending on requested payment method (`stripe`/`paypal` placeholders, manual fallback returns invoice info). +- Credit transactions (`CreditTransactionViewSet` in business billing views; also registered under `credits/transactions` in URLs): lists credit ledger entries per account. +- Core billing endpoints (`modules/billing/views.py`): + - `CreditBalanceViewSet.list`: returns current credits, plan monthly credits, credits used this month. + - `CreditUsageViewSet`: paginated credit usage logs with filters (operation_type, date range); `summary` aggregates by operation/model and totals (credits, USD); `limits` returns plan/account limits and usage (users/sites/etc.) based on `Plan` fields. +- Routing (`business/billing/urls.py`): + - Routers for invoices, payments, credit packages, transactions, payment methods, and canonical credit endpoints (balance/usage/transactions). Legacy available-methods endpoint exposed at `/payment-methods/available/`. + +## Data Structures / Models Involved (no code) +- `CreditTransaction`: ledger with type, amount (positive/negative), balance_after, metadata. +- `CreditUsageLog`: per-AI-operation usage with operation_type, credits_used, cost_usd, model_used, tokens, related object references, metadata. +- `CreditCostConfig`: per-operation cost config with unit, display metadata, active flag, audit of previous cost. +- `Invoice`: invoice_number, status, amounts (subtotal/tax/total), currency, billing_period, line_items JSON, subscription link, payment metadata, timestamps. +- `Payment`: status lifecycle, method (stripe/paypal/bank/local wallet/manual), provider references, manual notes/approval, failure reason, metadata. +- `CreditPackage`, `PaymentMethodConfig` (country/method availability + instructions/bank/local wallet data), `AccountPaymentMethod` (account-scoped payment metadata). +- Plan/account credits and limits: `Plan.included_credits`, `Plan.max_users/sites/industries/author_profiles`, `Account.credits`. + +## Execution Flow +- Credits: + - Operations call `CreditService.get_credit_cost` → `check_credits`/`deduct_credits` or `deduct_credits_for_operation` → updates `Account.credits`, writes `CreditTransaction` and `CreditUsageLog`. + - Costs resolved from `CreditCostConfig` when active; otherwise constants. Units support per request, per 100/200 words, per item/image/idea. +- Invoicing/Payments: + - Credit package purchase → invoice creation → next-action response (manual vs pending Stripe/PayPal integration). + - Manual payment submission → `PaymentService.create_manual_payment` persists pending approval. +- Credit balance/usage: + - Balance endpoint computes plan credits and month-to-date usage (from `CreditUsageLog`). + - Usage endpoint filters logs; summary aggregates by operation/model and totals; limits endpoint computes plan-based limits and usage (users/sites/etc.) per account. + +## Cross-Module Interactions +- AI/automation/planner/writer flows trigger credit checks/deductions through `CreditService`; usage logs can be filtered by operation_type (clustering, idea_generation, content_generation, image_generation, optimization, reparse). +- Plan limits surfaced in billing limits endpoint affect account/user/site management elsewhere. +- Invoice/payment metadata may be used by frontend billing UI; manual payments require admin follow-up. + +## State Transitions +- Credits mutate `Account.credits`; ledger captures each change; usage logs accumulate per operation. +- Invoices move through statuses (`draft/pending/paid/void/uncollectible`); payments through (`pending/pending_approval/processing/succeeded/completed/failed/refunded/cancelled`). +- Credit packages and payment methods have active/default flags; cost configs can be toggled active/inactive. + +## Error Handling +- Insufficient credits raise `InsufficientCreditsError`; API returns 402 when caught (e.g., content generation, credit deduct flows). +- Manual payment rejects missing fields or already-paid invoices. +- Usage/balance endpoints return empty data when account/plan missing instead of hard errors. +- Validation errors on site/sector/account scoping propagate from base viewsets and model constraints. + +## Tenancy Rules +- Account scoping enforced via `AccountModelViewSet` or explicit account filtering in viewsets; all billing data is per-account. +- Payment methods, invoices, payments, transactions, usage logs, and balances are filtered to `request.user.account` (or `request.account` from middleware). +- Plan data comes from the account’s associated plan; no cross-tenant visibility. + +## Billing Rules +- Credit costs configurable via `CreditCostConfig`; fallback constants apply when no config. +- Credit deductions and ledger entries are atomic; usage logs record the same operation with optional model/token metadata. +- Balance endpoint includes plan monthly credits; limits endpoint reports plan/user/site limits but does not enforce them here. + +## Background Tasks / Schedulers +- None specific to billing in this module; Stripe/PayPal integrations are placeholders. Any future webhooks should persist to these models and adjust credits/invoices/payments accordingly. + +## Key Design Considerations +- Credit handling is centralized in `CreditService` to keep ledger and usage logs consistent. +- Usage/balance endpoints are read-mostly, with graceful handling when plan/account data is missing. +- Manual payments support bank/local wallet flows without storing sensitive data; account payment methods store display metadata only. + +## How Developers Should Work With This Module +- Use `CreditService` for all credit checks/deductions/additions; do not mutate `Account.credits` directly. +- When adding new AI operations, add `operation_type` to `CreditUsageLog.OPERATION_TYPE_CHOICES` and `CreditCostConfig` seed/constants, then use `deduct_credits_for_operation`. +- Extend payment flows by implementing Stripe/PayPal handlers in `PaymentService` and wiring webhooks to update `Payment`/`Invoice`. +- Use existing serializers/viewsets when exposing billing data; keep queries scoped to `request.account`. diff --git a/master-docs/10-backend/billing/CREDIT-SYSTEM.md b/master-docs/10-backend/billing/CREDIT-SYSTEM.md index e69de29b..47f60528 100644 --- a/master-docs/10-backend/billing/CREDIT-SYSTEM.md +++ b/master-docs/10-backend/billing/CREDIT-SYSTEM.md @@ -0,0 +1,80 @@ +# Credit System + +## Purpose +Detail how credits are priced, checked, deducted, logged, and reported across the platform. + +## Code Locations (exact paths) +- Credit logic: `backend/igny8_core/business/billing/services/credit_service.py` +- Credit costs config: `backend/igny8_core/business/billing/models.py` (`CreditCostConfig`) +- Ledger and usage logs: `backend/igny8_core/business/billing/models.py` (`CreditTransaction`, `CreditUsageLog`) +- API endpoints: `backend/igny8_core/modules/billing/views.py` (`CreditBalanceViewSet`, `CreditUsageViewSet`) +- Constants and exceptions: `backend/igny8_core/business/billing/constants.py`, `exceptions.py` +- Plan included credits: `backend/igny8_core/auth/models.py` (`Plan.included_credits`, `Account.credits`) + +## High-Level Responsibilities +- Compute operation costs (configurable per operation and unit) and enforce sufficient balance before AI/content actions. +- Deduct or add credits atomically while writing both transaction and usage log records. +- Provide balance, usage history, summaries, and limits endpoints for clients. +- Allow admin-configurable cost overrides via `CreditCostConfig`. + +## Detailed Behavior +- Cost resolution (`CreditService.get_credit_cost`): + - First checks `CreditCostConfig` for the operation (active records). Supports units: `per_request`, `per_100_words`, `per_200_words`, `per_item`, `per_image`. Applies amounts when provided (e.g., words, items, images). + - Falls back to hardcoded `CREDIT_COSTS` constants; raises `CreditCalculationError` for unknown operations. + - Legacy variable costs: content_generation per 100 words, optimization per 200 words, image_generation per image, idea_generation per idea. +- Balance enforcement: + - `check_credits`/`check_credits_legacy` raise `InsufficientCreditsError` when balance insufficient. + - `deduct_credits_for_operation` computes cost, checks balance, builds a default description per operation when not supplied, then calls `deduct_credits`. +- Deduction (`deduct_credits`): + - Atomic: decrements `Account.credits`, writes `CreditTransaction` with negative amount and balance_after, writes `CreditUsageLog` capturing operation_type, cost_usd/model/tokens/related object references, metadata. +- Add credits (`add_credits`): + - Atomic: increments `Account.credits`, writes `CreditTransaction` with positive amount and balance_after. +- Logging: + - `CreditTransaction` is the authoritative ledger; `CreditUsageLog` tracks per-operation usage for analytics and summaries. + - `CreditCostConfig` tracks `previous_cost` on save when cost changes. +- Reporting endpoints (modules/billing): + - `CreditBalanceViewSet.list`: returns current credits, plan monthly credits, credits used this month (from `CreditUsageLog`), and remaining credits. + - `CreditUsageViewSet.list`: paginated usage logs filtered by operation_type and date range, account-scoped. + - `CreditUsageViewSet.summary`: aggregates credits and USD cost by operation and model over a date range (defaults to current month). + - `CreditUsageViewSet.limits`: returns plan limits (users/sites/industries/author profiles) and current usage counts, plus credits info; returns empty limits if account/plan missing. + +## Data Structures / Models Involved (no code) +- `CreditCostConfig`: operation_type, credits_cost, unit, display_name, description, is_active, previous_cost, updated_by, timestamps. +- `CreditTransaction`: transaction_type (purchase/subscription/refund/deduction/adjustment), amount (+/-), balance_after, description, metadata, reference_id, created_at. +- `CreditUsageLog`: operation_type (clustering/idea_generation/content_generation/image_generation/reparse/legacy names), credits_used, cost_usd, model_used, tokens_in/out, related object type/id, metadata, created_at. +- `Account.credits`: current balance; `Plan.included_credits`: monthly included credits. + +## Execution Flow +- Caller requests an operation → `get_credit_cost` computes cost → `check_credits` ensures balance → `deduct_credits` mutates balance and writes ledger + usage → upstream operation proceeds (AI/content). +- Balance/usage endpoints read from `Account`, `CreditUsageLog`, `Plan` to present current balances, month-to-date usage, summaries, and limits. + +## Cross-Module Interactions +- AI/automation/planner/writer services call `CreditService` before expensive operations; automation aggregates credits used per stage from AI task logs, while billing logs capture actual deductions at AI call sites. +- Plan limits (users/sites/industries/authors) referenced in billing limits endpoint inform account management elsewhere. + +## State Transitions +- Balance changes recorded per transaction; usage accumulates per operation. Cost configs can be toggled active/inactive and updated with audit of previous cost. + +## Error Handling +- Unknown operation types raise `CreditCalculationError`; insufficient balance raises `InsufficientCreditsError` (mapped to HTTP 402 in callers). +- Balance/usage endpoints fall back to zeros/empty when account/plan missing rather than erroring. + +## Tenancy Rules +- All credit operations and queries are account-scoped; viewsets filter by `request.account`/`request.user.account`. No cross-tenant access. + +## Billing Rules +- Costs derived from `CreditCostConfig` > constants; units must match supplied amount. Ledger and usage logs are mandatory for every deduction/addition. + +## Background Tasks / Schedulers +- None dedicated; credit usage often originates from AI Celery tasks but logging happens at the service call site. + +## Key Design Considerations +- Centralized credit math and atomic DB updates prevent drift between balance, ledger, and usage logs. +- Configurable costs allow runtime tuning without code changes. +- Reporting endpoints are read-only and tolerant of missing plan data to keep dashboards resilient. + +## How Developers Should Work With This Module +- Always route credit checks/deductions/additions through `CreditService`. +- When adding new AI operations, register operation_type in `CreditUsageLog`/constants and optionally seed `CreditCostConfig`. +- Include amount (words/items/images) when calling `deduct_credits_for_operation` so unit-based pricing applies. +- Use balance/usage endpoints for UI/analytics; avoid custom queries that bypass account scoping. diff --git a/master-docs/10-backend/billing/PRICING-AND-PLANS.md b/master-docs/10-backend/billing/PRICING-AND-PLANS.md index e69de29b..e29011d2 100644 --- a/master-docs/10-backend/billing/PRICING-AND-PLANS.md +++ b/master-docs/10-backend/billing/PRICING-AND-PLANS.md @@ -0,0 +1,68 @@ +# Pricing and Plans + +## Purpose +Describe how plans and pricing are represented in the backend and how plan data is used for credits and limits. + +## Code Locations (exact paths) +- Plan and account fields: `backend/igny8_core/auth/models.py` (`Plan`, `Account`) +- Billing limits endpoint: `backend/igny8_core/modules/billing/views.py` (`CreditUsageViewSet.limits`) +- Credit balance endpoint: `backend/igny8_core/modules/billing/views.py` (`CreditBalanceViewSet`) + +## High-Level Responsibilities +- Define subscription plans with included credits, billing cycle, pricing, feature flags, account limits, and Stripe metadata. +- Attach plans to accounts and expose plan-based limits and credits to billing endpoints. +- Keep credit computations aligned with `included_credits` via `Plan.get_effective_credits_per_month`. + +## Detailed Behavior +- `Plan` fields: + - Pricing: `price`, `billing_cycle` (`monthly/annual`), `extra_credit_price`, `allow_credit_topup`, optional auto-topup thresholds/amounts. + - Credits: `included_credits` (preferred) with legacy `credits_per_month` fallback via `get_effective_credits_per_month`. + - Limits: `max_users`, `max_sites`, `max_industries`, `max_author_profiles`. + - Stripe: `stripe_product_id`, `stripe_price_id`. + - Features: JSON `features` list; `is_active` flag. + - Validation ensures `max_sites >= 1`, `included_credits >= 0`. +- `Account` fields: + - `plan` FK, `credits` balance, status (active/suspended/trial/cancelled), billing contact/address fields, tax_id, retention days, Stripe customer id. + - Soft-delete guard prevents deleting system accounts. +- Limits computation (billing `limits` endpoint): + - Calculates current usage counts (users, sites, industries, author profiles) and returns alongside plan limits. + - Returns empty limits when account or plan is missing to keep UI resilient. +- Credit balance: + - Balance endpoint includes `plan_credits_per_month` using `Plan.get_effective_credits_per_month`. + +## Data Structures / Models Involved (no code) +- `Plan`, `Account` (plan attachment, credits, limits, billing info). + +## Execution Flow +- Account creation/association sets `plan`; credit balance endpoints read plan credits and account credits. +- Limits endpoint reads plan limits and computes current usage counts per account. +- Credit usage/deductions run against `Account.credits`; top-ups/invoices handled through billing services. + +## Cross-Module Interactions +- Billing endpoints surface plan credits and limits; credit cost/deductions happen in billing services. +- Automation start checks `Account.credits` but not plan limits; plan status enforced in `AccountContextMiddleware` (requires active plan). + +## State Transitions +- Plan activation toggles `is_active`; account plan changes immediately affect limits/balance reporting. +- Account status/plan changes enforced at middleware for access gating. + +## Error Handling +- Plan validation errors for invalid limits/credits on save; billing endpoints return empty limits when plan/account missing instead of failing. + +## Tenancy Rules +- Plans are global; accounts reference a single plan. All limit calculations are per account. + +## Billing Rules +- Credits included per plan reported via balance endpoint; extra credit price/top-up fields are present but actual purchase flows are through credit packages/invoices (not auto-charging in code shown). + +## Background Tasks / Schedulers +- None specific to plans; any subscription renewal logic would generate invoices/payments via billing services (not implemented here). + +## Key Design Considerations +- `included_credits` supersedes legacy `credits_per_month`; keep `get_effective_credits_per_month` for backward compatibility. +- Limits are informational in the billing module; enforcement must occur in the respective domains if required. + +## How Developers Should Work With This Module +- When adding new plan features or limits, extend `Plan` fields and ensure billing limits endpoint surfaces them. +- Use `get_effective_credits_per_month` wherever plan credits are displayed or calculated. +- Keep plan validation consistent (non-negative credits, minimum site/user counts). diff --git a/master-docs/10-backend/integrations/THIRD-PARTY-SERVICES.md b/master-docs/10-backend/integrations/THIRD-PARTY-SERVICES.md index e69de29b..994bab75 100644 --- a/master-docs/10-backend/integrations/THIRD-PARTY-SERVICES.md +++ b/master-docs/10-backend/integrations/THIRD-PARTY-SERVICES.md @@ -0,0 +1,85 @@ +# Third-Party Services (Integrations) + +## Purpose +Document how site-level integrations are modeled and exposed via API, including WordPress connection testing, sync, and metadata updates. + +## Code Locations (exact paths) +- Models: `backend/igny8_core/business/integration/models.py` (`SiteIntegration`, `SyncEvent`) +- Services: `backend/igny8_core/business/integration/services/integration_service.py`, `sync_service.py`, `sync_metadata_service.py`, `sync_health_service.py`, `content_sync_service.py` +- API views and routing: `backend/igny8_core/modules/integration/views.py`, `backend/igny8_core/modules/integration/urls.py` +- Webhooks: `backend/igny8_core/modules/integration/webhooks.py` + +## High-Level Responsibilities +- Store per-site integration configs and credentials for platforms (WordPress, Shopify, custom). +- Provide CRUD, test-connection, sync, sync-status, and structure-update endpoints for integrations. +- Track sync events for debugging/monitoring. +- Enforce tenant/site scoping and role-based permissions on integration APIs. + +## Detailed Behavior +- Model behavior: + - `SiteIntegration`: site+account scoped; fields for platform, platform_type, config_json (content types, sync settings), credentials_json (API keys, etc.), active/sync flags, sync status, last_sync_at, sync_error; unique per site+platform. + - `SyncEvent`: per-integration/site event log with event_type/action, success flag, optional content_id/external_id, details JSON, error_message, duration, timestamps. + - `SiteIntegration.get_credentials` / `set_credentials` currently store JSON as-is (encryption to be added). +- Service behavior (`IntegrationService`): + - `create_integration` sets account from site and saves config/credentials with platform/platform_type. + - `update_integration` updates config/credentials/is_active; `delete_integration` removes the record. + - `get_integration`/`list_integrations`/`get_integrations_for_site` fetch active integrations by site/platform. + - `test_connection` delegates to platform-specific testers (WordPress/Shopify implemented; others raise `NotImplementedError`). +- API (`IntegrationViewSet`, router `/api/v1/integration/integrations/`): + - Inherits `SiteSectorModelViewSet` but overrides `get_queryset` to filter by site_id (no sector field). + - Permissions: `IsAuthenticatedAndActive` + `IsEditorOrAbove`; throttle scope `integration`. + - Serializer exposes derived `api_key` for WordPress; validates that WordPress integrations include an API key. + - Actions: + - `test_connection` (detail): calls `IntegrationService.test_connection` for the existing integration. + - `test_connection_collection` (AllowAny, no throttle): accepts site_id/api_key/site_url; validates site ownership or matching `Site.wp_api_key`; creates WordPress integration if missing, tests via `_test_wordpress_connection`, deletes if test fails, returns integration_id on success. + - `sync`: calls `SyncService.sync` (direction metadata/both/to_external/from_external); metadata-only uses `SyncMetadataService.sync_wordpress_structure`. + - `sync_status`: returns sync status via `SyncService.get_sync_status`. + - `update_site_structure`: updates `config_json.content_types` from WordPress push (post_types, taxonomies, plugin flags, timestamp). + - `get_content_types`: returns content type config. + - `debug_status`: uses `SyncHealthService` for debug data. + - `sync_wordpress_content`: calls `ContentSyncService.sync_from_wordpress` for a specific content_id. +- Webhooks: + - Registered under `/api/v1/integration/webhooks/wordpress/status/` and `/metadata/` for incoming WordPress callbacks (handling in `webhooks.py`). + +## Data Structures / Models Involved (no code) +- `SiteIntegration` config/credentials/sync flags/status/error. +- `SyncEvent` event logs. +- Site/account context from `auth.models.Site` and `Account`. + +## Execution Flow +- CRUD/test/sync requests → DRF auth → site-filtered queryset → service calls (integration/sync/health/content-sync) → model updates or sync actions. +- Collection-level test may create integration, run test, and delete on failure. +- Webhooks ingest external events; service handlers update integrations/sync events (implementation-specific). + +## Cross-Module Interactions +- Integrations reference sites and are consumed by publishing/sync flows when pushing/pulling content. +- WordPress API key path relies on `Site.wp_api_key` and integration credentials during connection tests. +- Sync services interact with planner/writer content when syncing content or structure. + +## State Transitions +- `SiteIntegration.sync_status/last_sync_at` update during sync; `is_active`/`sync_enabled` toggled via API. +- `SyncEvent` records per-action success/failure with timestamps and optional IDs. + +## Error Handling +- Unsupported platforms raise `NotImplementedError`; connection tests return structured success/message; newly created integrations are deleted if test fails in collection endpoint. +- API returns 403/404/400 for invalid site/auth/API key; sync errors surface in responses and may populate `sync_error`. + +## Tenancy Rules +- All integrations are site- and account-scoped; viewsets filter by site_id and require authenticated user with editor-or-above, or matching API key for collection-level WordPress test. + +## Billing Rules +- None; integration operations do not alter credits. + +## Background Tasks / Schedulers +- None dedicated; syncs run inline in service calls unless delegated inside sync services. + +## Key Design Considerations +- Unique integration per site+platform avoids duplication. +- Collection-level test supports plugin onboarding without prior authenticated session while still requiring site ownership or API key. +- Credentials are currently stored plain JSON—add encryption before production use. + +## How Developers Should Work With This Module +- Use `IntegrationService` for CRUD/test; avoid direct credential mutation outside service helpers. +- Extend platform support by adding platform-specific test logic and serializer validation. +- When adding new sync flows, use `SyncService`/`SyncMetadataService`/`ContentSyncService` and keep site/account scoping intact. + diff --git a/master-docs/10-backend/planner/PLANNER-MODELS-AND-FLOW.md b/master-docs/10-backend/planner/PLANNER-MODELS-AND-FLOW.md index e69de29b..cb396375 100644 --- a/master-docs/10-backend/planner/PLANNER-MODELS-AND-FLOW.md +++ b/master-docs/10-backend/planner/PLANNER-MODELS-AND-FLOW.md @@ -0,0 +1,69 @@ +# Planner Models and Flow + +## Purpose +Provide a detailed view of planner models, their constraints, and how planner CRUD/bulk operations move data toward the writer pipeline. + +## Code Locations (exact paths) +- Models: `backend/igny8_core/business/planning/models.py` +- Serializers: `backend/igny8_core/modules/planner/serializers.py` +- Views: `backend/igny8_core/modules/planner/views.py` +- Services: `backend/igny8_core/business/planning/services/clustering_service.py`, `ideas_service.py` + +## High-Level Responsibilities +- Maintain tenant/site/sector-scoped planner entities (Keywords, Clusters, ContentIdeas) backed by global SeedKeywords. +- Enforce industry/sector alignment and uniqueness within site/sector. +- Support search, filtering, ordering, and bulk operations for planner data. +- Feed automation and writer tasks with curated ideas and clusters. + +## Detailed Behavior +- Models: + - `Keywords`: links to `SeedKeyword`; overrides for volume/difficulty; optional cluster; status `new/mapped`; disabled flag; validation ensures seed keyword industry matches site industry and seed sector matches site sector’s industry sector; unique per site/sector and seed keyword; soft-delete enabled. + - `Clusters`: topic groups with counts, volume, mapped_pages, status `new/mapped`; unique per site/sector by name; soft-delete enabled. + - `ContentIdeas`: status `new/queued/completed`, content_type/structure choices, estimated word count, optional keyword objects M2M, keyword_cluster FK; soft-delete enabled. +- Viewsets: + - `KeywordViewSet`: search by seed_keyword.keyword; filter by status, cluster, seed intent; ordering by created_at, seed volume/difficulty; custom range filters for difficulty/volume; requires site_id/sector_id on create; bulk_delete; two `bulk_update` actions (status); `bulk_add_from_seed` to create Keywords from SeedKeywords with site/sector validation. + - `ClusterViewSet`: site/sector filtered; CRUD; status updates; inherits filtering from base with developer/admin/system override. + - `ContentIdeasViewSet`: CRUD; filters by status/cluster; requires site_id/sector_id on create. +- Serializers: + - `KeywordSerializer` demands `seed_keyword_id` on create; derives read-only keyword/volume/difficulty/intent from seed; supports volume/difficulty overrides; exposes cluster/sector names; write-only site_id/sector_id. + - `ClusterSerializer` exposes sector name; write-only site_id/sector_id; counts/volume/mapped_pages read-only. + - `ContentIdeasSerializer` exposes cluster/sector names; write-only site_id/sector_id. +- Services: + - `ClusteringService` and `IdeasService` (not expanded here) are used by automation/AI flows for clustering and idea generation. + +## Execution Flow +- Planner CRUD/bulk requests → DRF auth → `SiteSectorModelViewSet` filtering → serializer validation (seed/site/sector) → model save → unified response. +- Bulk operations act on the filtered queryset (account/site/sector scoped). +- Automation stages 1–3 consume planner data and advance statuses. + +## Cross-Module Interactions +- Writer tasks and content reference clusters/ideas; automation transforms planner data into tasks/content/images. +- Billing credit checks occur in automation/AI calls that process planner entities. + +## State Transitions +- Keywords: `new` → `mapped`; Clusters: `new` → `mapped`; ContentIdeas: `new` → `queued` → `completed`. +- Disabled flag excludes records from automation queries. + +## Error Handling +- Validation errors for missing site/sector or seed keyword; industry/sector mismatch raises validation errors in `Keywords`. +- Bulk operations return 400 when IDs/status missing; generic errors return unified 500 responses. + +## Tenancy Rules +- All planner models inherit tenant/site/sector fields; base viewsets filter accordingly. Admin/developer/system accounts can bypass filtering; persisted data retains tenant/site/sector ownership. +- Create paths require explicit site/sector; account set from site/user/middleware. + +## Billing Rules +- Planner endpoints do not directly deduct credits; AI-based clustering/idea generation uses billing when invoked via services/automation. + +## Background Tasks / Schedulers +- None in planner endpoints; automation runs stages asynchronously using planner data. + +## Key Design Considerations +- Strict site/sector + seed alignment prevents cross-site contamination. +- Read-only seed-derived fields avoid drift while allowing per-site overrides. +- Bulk endpoints and range filters support large keyword sets efficiently. + +## How Developers Should Work With This Module +- Use existing serializers/viewsets to add fields or validations; keep site/sector required on create. +- When adding AI-powered planner actions, ensure credit checks go through billing services and queries respect tenant/site/sector scoping. +- Clean up duplicate `bulk_update` definitions if extending bulk behaviors to avoid route collisions. diff --git a/master-docs/10-backend/planner/PLANNER-OVERVIEW.md b/master-docs/10-backend/planner/PLANNER-OVERVIEW.md index e69de29b..46992ddf 100644 --- a/master-docs/10-backend/planner/PLANNER-OVERVIEW.md +++ b/master-docs/10-backend/planner/PLANNER-OVERVIEW.md @@ -0,0 +1,80 @@ +# Planner Overview + +## Purpose +Explain how the planner module manages keywords, clusters, and content ideas, including tenancy, validation, and API behaviors. + +## Code Locations (exact paths) +- Models: `backend/igny8_core/business/planning/models.py` +- Serializers: `backend/igny8_core/modules/planner/serializers.py` +- Views: `backend/igny8_core/modules/planner/views.py` +- URLs: `backend/igny8_core/modules/planner/urls.py` +- Services: `backend/igny8_core/business/planning/services/clustering_service.py`, `ideas_service.py` + +## High-Level Responsibilities +- Store site/sector-scoped keywords, clusters, and content ideas linked to global seed keywords. +- Provide CRUD and bulk operations for keywords/clusters/ideas with search, filter, and ordering. +- Validate industry/sector alignment between seed keywords and site/sector. +- Feed downstream writer/automation stages with structured ideas and clusters. + +## Detailed Behavior +- Models (site/sector scoped via `SiteSectorBaseModel` and soft-delete): + - `Keywords`: references global `SeedKeyword`; optional volume/difficulty overrides; optional cluster; status `new/mapped`; disabled flag; validation ensures seed keyword industry/sector match the site/sector. + - `Clusters`: topic groups with counts, volume, mapped_pages, status `new/mapped`; unique per site/sector by name. + - `ContentIdeas`: ideas tied to clusters and optional keyword objects; status `new/queued/completed`; content_type/structure and estimated_word_count; disabled flag. +- Serializers: + - `KeywordSerializer` enforces `seed_keyword_id` on create; exposes seed keyword-derived fields (keyword/volume/difficulty/intent) read-only; optional overrides; includes site_id/sector_id write-only; provides cluster/sector names via getters; optional `attribute_values` when `USE_SITE_BUILDER_REFACTOR` flag is enabled. + - `ClusterSerializer` exposes site/sector IDs, sector name, read-only counts/volume/mapped_pages. + - `ContentIdeasSerializer` exposes cluster/sector names, content type/structure, keyword/target fields, site/sector write-only. +- Views (`KeywordViewSet`, `ClusterViewSet`, `ContentIdeasViewSet`): + - Inherit `SiteSectorModelViewSet` for tenant/site/sector filtering and unified responses. + - Permissions: `IsAuthenticatedAndActive` + viewer-or-above (search/list CRUD). + - Pagination: `CustomPageNumberPagination`; throttle scope `planner`. + - Filters/Search/Ordering: keywords searchable by seed_keyword.keyword; filter by status, cluster_id, seed intent; ordering by created_at, volume, difficulty; custom range filters for difficulty/volume; clusters filter by status; ideas filter by status/cluster. + - Creation requires explicit `site_id` and `sector_id`; viewsets validate site/sector existence and alignment; set `account/site/sector` explicitly on save. + - Bulk endpoints: + - Keywords: `bulk_delete`, two definitions of `bulk_update` (status update) present; `bulk_add_from_seed` to create Keywords from SeedKeywords with site/sector validation. + - Clusters: bulk update endpoints in view (not shown above) adjust status. + - Ideas: creation and status updates follow standard CRUD; not shown as bulk endpoints in code snippet. + - Error handling: viewsets wrap errors into unified responses; return 400/404/500 as appropriate. +- Services: + - `ClusteringService` and `IdeasService` (not detailed here) provide additional business logic for clustering/idea generation when invoked by automation or endpoints. + +## Data Structures / Models Involved (no code) +- `Keywords`, `Clusters`, `ContentIdeas`; global `SeedKeyword`. +- Tenancy entities: `Account`, `Site`, `Sector`. + +## Execution Flow +- Requests → DRF auth → `SiteSectorModelViewSet` filters by account/site/sector → serializers validate seed/site/sector alignment → models persisted → responses returned via unified helpers; bulk actions operate on filtered querysets. + +## Cross-Module Interactions +- Automation stages 1–3 consume planner data (keywords → clusters → ideas → tasks). +- Writer tasks reference clusters/ideas produced here. +- Billing credits may be checked upstream for AI clustering/idea generation (via services invoked in automation or planner actions). + +## State Transitions +- Keywords: `new` → `mapped`; Clusters: `new` → `mapped`; ContentIdeas: `new` → `queued` → `completed`. +- Disabled flag can exclude records from processes. + +## Error Handling +- Validation enforces seed/site/sector industry alignment; missing site/sector on create raises validation errors. +- Bulk operations return 400 on missing IDs/status; general errors return unified 500 responses. + +## Tenancy Rules +- All planner models inherit `SiteSectorBaseModel`; queries are filtered by account/site/sector; admin/developer/system roles can bypass filtering via base viewset logic. +- Create requires explicit site and sector; viewsets fetch and set account from site/user/middleware. + +## Billing Rules +- None directly in the planner endpoints; credit costs for clustering/idea generation are enforced where those services are called (automation or planner service usage). + +## Background Tasks / Schedulers +- Planner endpoints run synchronously; automation may batch planner data into AI tasks via Celery (documented in automation). + +## Key Design Considerations +- Strict site/sector validation prevents cross-tenant or cross-site leaks. +- Seed keyword linkage centralizes keyword metadata; overrides allow per-site tuning. +- Pagination/filtering/search are optimized for large keyword sets. + +## How Developers Should Work With This Module +- Use provided viewsets and serializers; ensure site_id/sector_id are supplied on create. +- When adding fields, extend serializers with read-only/write-only behavior consistent with current patterns. +- For new bulk operations, follow existing patterns: validate IDs, operate on filtered queryset, return unified responses. diff --git a/master-docs/10-backend/writer/WRITER-OVERVIEW.md b/master-docs/10-backend/writer/WRITER-OVERVIEW.md index e69de29b..bdecb9cd 100644 --- a/master-docs/10-backend/writer/WRITER-OVERVIEW.md +++ b/master-docs/10-backend/writer/WRITER-OVERVIEW.md @@ -0,0 +1,82 @@ +# Writer Overview + +## Purpose +Describe how the writer module manages tasks, content, images, and taxonomies, including validations, tenancy, and API behaviors. + +## Code Locations (exact paths) +- Models: `backend/igny8_core/business/content/models.py` +- Serializers: `backend/igny8_core/modules/writer/serializers.py` +- Views: `backend/igny8_core/modules/writer/views.py` +- URLs: `backend/igny8_core/modules/writer/urls.py` +- Services: `backend/igny8_core/business/content/services/content_generation_service.py`, `validation_service.py`, `metadata_mapping_service.py` +- Automation linkages: `backend/igny8_core/business/automation/services/automation_service.py` (stages 3–6) + +## High-Level Responsibilities +- Manage tenant/site/sector-scoped tasks, generated/imported content, images, and taxonomies. +- Provide CRUD and bulk operations for tasks and images; content generation triggers; taxonomy CRUD. +- Enforce site/sector alignment on creation and propagate account/site/sector to related records. +- Feed automation and publishing flows with content and images, and integrate with billing credits for AI operations. + +## Detailed Behavior +- Models (all inherit tenant/site/sector bases; many soft-deletable): + - `Tasks`: queued work items tied to clusters/ideas/taxonomy; content type/structure; keywords text; word_count target; status `queued/completed`. + - `Content`: generated/imported content with HTML, SEO fields, cluster link, content type/structure, taxonomy M2M, external IDs/URLs/metadata, sync status, source (`igny8/wordpress`), status (`draft/review/published`), word_count, timestamps. + - `Images`: linked to content or task; auto-sets account/site/sector from provider; tracks type, URL/path, prompt, status, position. + - `ContentTaxonomy`: site/sector taxonomy terms (category/tag) with external taxonomy/id, sync status, description, count, metadata; unique per site by slug+type and by external id+taxonomy. + - Supporting: `ContentClusterMap`, `ContentAttribute` (documented in domain models). +- Serializers: + - `TasksSerializer`: requires cluster/content_type/content_structure on create; exposes cluster/sector names; write-only site_id/sector_id; defaults status to queued if absent. + - `ImagesSerializer`: exposes task/content titles; read-only derived fields; creation requires site/sector (validated, with fallback to request user active site default sector); sets account/site/sector via base perform_create. + - `ContentSerializer`: exposes cluster/sector names; taxonomy terms grouped (tags/categories); image status flags; write-only site_id/sector_id; validates presence of required fields and status transitions. + - Group serializers for images support grouped responses by content. +- Views: + - `TasksViewSet`: filters/search/order; bulk_delete, bulk_update status; `auto_generate_content` triggers `ContentGenerationService.generate_content` for up to 10 tasks, enforcing account presence and credit checks (returns 402 on `InsufficientCreditsError`); inherits tenant/site/sector filtering. + - `ImagesViewSet`: CRUD with ordering/filtering; perform_create enforces site/sector and populates context; `serve_image_file` serves local files with basic checks. + - `ContentViewSet`: CRUD for content; ordering/filtering; handles taxonomy assignments; exposes status/SEO fields; uses base scoping. + - `ContentTaxonomyViewSet`: CRUD for taxonomy terms (category/tag) scoped to site/sector; inherits base scoping. +- Services: + - `ContentGenerationService`: invoked by `auto_generate_content`; orchestrates AI content generation for tasks, returns async task_id when queued or synchronous result; raises `InsufficientCreditsError` on low balance. + - `ContentValidationService`, `MetadataMappingService`: additional processing/validation/mapping (used in writer flows and publishing). +- Automation linkage: + - Stage 3 creates tasks from ideas; Stage 4 generates content; Stage 5 generates image prompts; Stage 6 generates images and moves content to review. + +## Data Structures / Models Involved (no code) +- `Tasks`, `Content`, `Images`, `ContentTaxonomy`, plus `ContentClusterMap`/`ContentAttribute`. +- Tenancy entities: `Account`, `Site`, `Sector`. + +## Execution Flow +- Requests → DRF auth → `SiteSectorModelViewSet` filtering → serializer validation (site/sector required on create) → model save → unified response. +- `auto_generate_content` validates task IDs/account, calls service → service may enqueue Celery AI tasks or run synchronously → response with task_id or result. +- Images creation auto-inherits account/site/sector from linked task/content; local file serving bypasses account filter for read-only access. + +## Cross-Module Interactions +- Planner clusters/ideas feed tasks; tasks/content/images feed publishing and automation stages. +- Billing credits enforced via `ContentGenerationService` when AI is used; automation aggregates credits used per stage from AI task logs. +- Integration/publishing sync uses content/taxonomy/image data when pushing to external platforms. + +## State Transitions +- Tasks: `queued` → `completed`; Content: `draft` → `review` → `published`; Images: `pending` → `generated`; Taxonomies created/updated/deleted via CRUD. + +## Error Handling +- Viewsets return unified errors; bulk operations validate IDs/status; `auto_generate_content` returns 400/404/402/500 depending on validation, missing tasks, insufficient credits, or unexpected errors. +- Images file serving returns 404 on missing file/image. + +## Tenancy Rules +- Base viewset filters enforce account/site/sector scoping; admins/developers/system can bypass filtering; persisted records retain tenant/site/sector ownership. +- Creation requires site/sector; account set from request/site/user; Images perform_create validates presence of site/sector. + +## Billing Rules +- AI content generation is guarded by credit checks; InsufficientCredits returns HTTP 402; credit logging occurs in billing services at AI call sites. + +## Background Tasks / Schedulers +- AI generation may run via Celery when `ContentGenerationService` returns a task_id; automation stages run inside Celery workers for bulk flows. + +## Key Design Considerations +- Strict site/sector requirements prevent cross-site contamination. +- Bulk limits (max 10 tasks in auto_generate_content) to control credit burn and processing load. +- Image serving permits public access via endpoint while still performing existence checks. + +## How Developers Should Work With This Module +- Use existing serializers/viewsets; keep site/sector required on create. +- When extending AI generation, ensure billing checks (`CreditService`) and account/site scoping are preserved. +- For new bulk actions, follow current patterns: validate input, operate on filtered queryset, return unified responses. diff --git a/master-docs/20-api/API-OVERVIEW.md b/master-docs/20-api/API-OVERVIEW.md index e69de29b..058ea00b 100644 --- a/master-docs/20-api/API-OVERVIEW.md +++ b/master-docs/20-api/API-OVERVIEW.md @@ -0,0 +1,79 @@ +# API Overview + +## Purpose +Describe the overall API surface, routing, authentication, pagination, throttling, and response format implemented in the backend. + +## Code Locations (exact paths) +- Root routing: `backend/igny8_core/urls.py` +- Module routers: + - Auth: `backend/igny8_core/auth/urls.py` + - Account management: `backend/igny8_core/api/urls.py` + - Planner: `backend/igny8_core/modules/planner/urls.py` + - Writer: `backend/igny8_core/modules/writer/urls.py` + - System: `backend/igny8_core/modules/system/urls.py` + - Billing (full): `backend/igny8_core/business/billing/urls.py` + - Billing admin: `backend/igny8_core/modules/billing/admin_urls.py` + - Automation: `backend/igny8_core/business/automation/urls.py` + - Linker: `backend/igny8_core/modules/linker/urls.py` + - Optimizer: `backend/igny8_core/modules/optimizer/urls.py` + - Publisher: `backend/igny8_core/modules/publisher/urls.py` + - Integration: `backend/igny8_core/modules/integration/urls.py` +- DRF settings: `backend/igny8_core/settings.py` (REST_FRAMEWORK config) +- Base viewsets/responses: `backend/igny8_core/api/base.py`, `backend/igny8_core/api/response.py` + +## High-Level Responsibilities +- Expose REST endpoints under `/api/v1/*` for auth, account/team, planner, writer, system, billing, automation, linker, optimizer, publisher, and integration. +- Enforce unified response format, tenant scoping, authentication (API key/JWT/session/basic), and scoped throttling. +- Provide OpenAPI schema and interactive docs at `/api/schema`, `/api/docs`, `/api/redoc`. + +## Detailed Behavior +- Routing (root): `/api/v1/auth/`, `/api/v1/account/`, `/api/v1/planner/`, `/api/v1/writer/`, `/api/v1/system/`, `/api/v1/billing/`, `/api/v1/admin/` (billing admin), `/api/v1/automation/`, `/api/v1/linker/`, `/api/v1/optimizer/`, `/api/v1/publisher/`, `/api/v1/integration/`. Admin + CSV helpers mounted under `/admin/`. +- Auth stack (settings REST_FRAMEWORK): + - `APIKeyAuthentication` (WordPress bridge, sets request.account/site when Bearer token is not JWT-like). + - `JWTAuthentication` (Bearer JWT access tokens). + - `CSRFExemptSessionAuthentication` (session without CSRF for API). + - Basic auth fallback. +- Permissions default AllowAny in settings; individual viewsets specify stricter permissions (e.g., `IsAuthenticatedAndActive`, role checks). +- Tenancy: `AccountContextMiddleware` sets `request.account`; base viewsets (`AccountModelViewSet`, `SiteSectorModelViewSet`) auto-filter querysets by account/site/sector and set account on create. +- Pagination: `CustomPageNumberPagination` (page/ page_size; defaults in settings). +- Filtering/search/ordering: enabled globally via DRF filter backends; viewsets declare fields and search options. +- Throttling: `DebugScopedRateThrottle` with scope keys defined in settings for auth, planner, writer, billing, automation, etc.; bypassed when `IGNY8_DEBUG_THROTTLE` true. +- Responses: success/error helpers wrap payloads with `success`, `data`/`error`, optional `errors`, and `request_id`. +- OpenAPI: drf-spectacular configured with tags, schema routes; interactive docs at `/api/docs` (Swagger UI) and `/api/redoc`. + +## Data Structures / Models Involved (no code) +- See module-specific docs; API layer uses serializers tied to planner/writer/billing/auth/etc. models. + +## Execution Flow +- Request → middleware (request ID, account context, resource tracking) → DRF auth (API key/JWT/session) → viewset/action → serializer validation → model ops → unified response. Pagination/filtering/search applied where configured. + +## Cross-Module Interactions +- Auth endpoints issue JWTs consumed by all other modules. +- Tenancy middleware/context is shared across all module viewsets. +- Billing credits enforced in content/automation flows via service calls, surfaced through billing endpoints. + +## State Transitions +- Handled in module viewsets (e.g., task status changes, run statuses); API layer mediates CRUD and custom actions. + +## Error Handling +- Custom exception handler (enabled by default) wraps errors in unified format. +- Viewsets return structured 400/401/403/404/500 responses; throttling returns standard DRF throttle responses. + +## Tenancy Rules +- Account/site/sector scoping enforced by middleware + base viewsets; admin/developer/system roles may bypass filters at the viewset layer per role helpers. + +## Billing Rules +- Credit checks occur in service calls (content generation, automation); billing endpoints report balances/usage/limits. Some endpoints return 402 on insufficient credits. + +## Background Tasks / Schedulers +- Automation and some AI flows run via Celery; API endpoints trigger tasks (e.g., automation run_now, writer auto_generate_content). + +## Key Design Considerations +- Consistent routing under `/api/v1/` with module-specific routers. +- Unified responses and throttling scopes to keep client behavior consistent. +- OpenAPI docs generated from viewset/serializer annotations for discoverability. + +## How Developers Should Work With This Module +- Mount new modules under `/api/v1//` in `igny8_core/urls.py`. +- Inherit from base viewsets to get scoping/unified responses; declare permissions, filters, and throttles per viewset. +- Annotate endpoints with drf-spectacular for accurate schema; keep success/error responses consistent via helpers. diff --git a/master-docs/20-api/ENDPOINTS/accounts.md b/master-docs/20-api/ENDPOINTS/accounts.md index e69de29b..9b7dc293 100644 --- a/master-docs/20-api/ENDPOINTS/accounts.md +++ b/master-docs/20-api/ENDPOINTS/accounts.md @@ -0,0 +1,56 @@ +# Account Endpoints + +## Purpose +Document account management endpoints for settings, team management, and usage analytics. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/api/urls.py` +- Views: `backend/igny8_core/api/account_views.py` +- Settings/tenancy middleware: `backend/igny8_core/auth/middleware.py` + +## High-Level Responsibilities +- Provide account settings retrieval/update, team member management, and usage analytics summaries for the authenticated account. + +## Detailed Behavior +- Base path: `/api/v1/account/` +- Settings: + - `GET /api/v1/account/settings/` → returns account settings (ViewSet `retrieve`). + - `PATCH /api/v1/account/settings/` → partial update of account settings. +- Team: + - `GET /api/v1/account/team/` → list team members. + - `POST /api/v1/account/team/` → create/add a team member. + - `DELETE /api/v1/account/team//` → remove a team member. +- Usage analytics: + - `GET /api/v1/account/usage/analytics/` → returns usage analytics overview (implementation in `UsageAnalyticsViewSet.overview`). +- Permissions/tenancy: views expect authenticated users; account context from middleware; data scoped to `request.account`. + +## Data Structures / Models Involved (no code) +- Account, User, and related analytics data as computed in account views. + +## Execution Flow +- Requests hit `account_views` ViewSets; account context derived from session/JWT via middleware; view methods return unified responses. + +## Cross-Module Interactions +- Usage analytics may summarize module usage (planner/writer/etc.) depending on `UsageAnalyticsViewSet` implementation. + +## State Transitions +- Account settings updates; team membership changes. + +## Error Handling +- Standard unified responses for validation/auth errors; 404 when deleting non-existent team members. + +## Tenancy Rules +- All endpoints scoped to authenticated user’s account; no cross-tenant access. + +## Billing Rules +- None directly; account limits and credits are exposed via billing endpoints. + +## Background Tasks / Schedulers +- None. + +## Key Design Considerations +- Non-router endpoints used for simplicity; still leverage DRF ViewSet actions. + +## How Developers Should Work With This Module +- Keep settings/team/usage endpoints under `/api/v1/account/`; ensure account context is required for all operations. + diff --git a/master-docs/20-api/ENDPOINTS/authentication.md b/master-docs/20-api/ENDPOINTS/authentication.md index e69de29b..19792a25 100644 --- a/master-docs/20-api/ENDPOINTS/authentication.md +++ b/master-docs/20-api/ENDPOINTS/authentication.md @@ -0,0 +1,62 @@ +# Authentication Endpoints + +## Purpose +Detail authentication-related endpoints for registration, login, password change, token refresh, and auth-adjacent resources (users/accounts/plans/sites/sectors/industries/seed keywords). + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/auth/urls.py` +- Views: `backend/igny8_core/auth/views.py` +- Serializers: `backend/igny8_core/auth/serializers.py` +- Auth utils: `backend/igny8_core/auth/utils.py` + +## High-Level Responsibilities +- Issue JWT access/refresh tokens and session auth on login. +- Register users, change passwords, refresh tokens. +- Provide CRUD for users, accounts, subscriptions, site access, plans, sites, sectors, industries, seed keywords via routers. + +## Detailed Behavior +- APIViews: + - `POST /api/v1/auth/register/` → `RegisterView`: validates via `RegisterSerializer`, creates user, returns user data. + - `POST /api/v1/auth/login/` → `LoginView`: validates credentials, logs in user (session), generates access/refresh JWTs with expiries, returns user data and tokens; 401 on invalid credentials. + - `POST /api/v1/auth/change-password/` → `ChangePasswordView`: requires auth, validates old/new passwords, updates password; 400 on invalid current password or validation errors. + - `POST /api/v1/auth/refresh/` → `RefreshTokenView`: accepts refresh token, validates, issues new access token with expiry; 401/400 on invalid/expired tokens. +- Routers under `/api/v1/auth/`: + - `groups/`, `users/`, `accounts/`, `subscriptions/`, `site-access/`, `plans/`, `sites/`, `sectors/`, `industries/`, `seed-keywords/` mapped to corresponding viewsets in `auth/views.py`. These use base viewsets with tenant filtering and role checks (see module docs). +- CSV admin helpers (admin UI, not public API): CSV import/export for industry/industrysector/seedkeyword under `/admin/igny8_core_auth/...`. + +## Data Structures / Models Involved (no code) +- `User`, `Account`, `Plan`, `Site`, `Sector`, `Industry`, `SeedKeyword`, `Subscription`, `SiteUserAccess`, `Group`. + +## Execution Flow +- APIView endpoints use serializers to validate, then issue tokens via auth utils (JWT encode with user_id/account_id, type, exp). +- Router endpoints inherit base viewsets; AccountContextMiddleware sets `request.account`; JWT/Session auth applied per settings; permissions enforced per viewset. + +## Cross-Module Interactions +- Tokens issued here are required by planner/writer/billing/automation/etc. +- Account/plan/site data informs tenancy and plan enforcement in middleware and billing limits. + +## State Transitions +- User creation, password changes, session login, token issuance/refresh. +- CRUD on accounts/sites/sectors/industries/seed keywords via routers. + +## Error Handling +- Unified error responses via helpers; login returns 401 on invalid credentials; refresh returns 401/400 on invalid/expired tokens; validation errors return 400 with field errors. + +## Tenancy Rules +- Login loads user with account+plan; middleware later enforces active plan. Router viewsets inherit account/site/sector filtering where applicable. + +## Billing Rules +- None directly; plan data carried on Account; credits handled elsewhere. + +## Background Tasks / Schedulers +- None. + +## Key Design Considerations +- JWT payload includes user_id, account_id, type, exp/iat; uses `JWT_SECRET_KEY`/`JWT_ALGORITHM` from settings. +- Session login occurs alongside JWT issuance to support browser-based flows. + +## How Developers Should Work With This Module +- Use provided APIViews for auth; do not handcraft JWTs—use `generate_access_token`/`generate_refresh_token`. +- When extending auth resources (e.g., new user fields), update serializers and viewsets; keep router paths stable. +- Maintain unified responses and permission/tenancy checks in viewsets. + diff --git a/master-docs/20-api/ENDPOINTS/automation.md b/master-docs/20-api/ENDPOINTS/automation.md index e69de29b..6e0d24ae 100644 --- a/master-docs/20-api/ENDPOINTS/automation.md +++ b/master-docs/20-api/ENDPOINTS/automation.md @@ -0,0 +1,65 @@ +# Automation Endpoints + +## Purpose +Describe automation API endpoints that configure, trigger, monitor, and control automation runs. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/business/automation/urls.py` +- Views: `backend/igny8_core/business/automation/views.py` +- Services/Tasks: `backend/igny8_core/business/automation/services/automation_service.py`, `automation_logger.py`, `backend/igny8_core/business/automation/tasks.py` + +## High-Level Responsibilities +- Expose configuration CRUD (get/update) and operational controls (run, pause, resume, cancel). +- Provide run history, status, logs, pipeline overview, credit estimate, and current processing state. +- Enforce site/account scoping and authentication. + +## Detailed Behavior +- Base path: `/api/v1/automation/` +- Key endpoints (actions on `AutomationViewSet`): + - `GET config?site_id=` → get/create `AutomationConfig` for site (includes batch sizes, delays, schedule, flags, last/next run). + - `PUT update_config?site_id=` → update config fields (enable, frequency, scheduled_time, batch sizes, delays). + - `POST run_now?site_id=` → start a run (checks locks/credits) and enqueue Celery `run_automation_task`. + - `GET current_run?site_id=` → current running/paused run details with per-stage results and totals. + - `GET history?site_id=` → last 20 runs with status and timestamps. + - `GET logs?run_id=&lines=` → tail of automation log file for run. + - `GET estimate?site_id=` → estimated credits required plus current balance and sufficiency flag (1.2× buffer). + - `GET pipeline_overview?site_id=` → counts/pending by stage across planner/writer models. + - `GET current_processing?site_id=&run_id=` → live processing state for active run. + - `POST pause?run_id=` / `POST resume?run_id=` / `POST cancel?run_id=` → control run state (pause/resume/cancel). +- Permissions/tenancy: `IsAuthenticated`; site fetched with `account=request.user.account`; automation lock is per site; `request.account` set by middleware. + +## Data Structures / Models Involved (no code) +- `AutomationConfig`, `AutomationRun`, planner models (Keywords/Clusters/ContentIdeas), writer models (Tasks/Content/Images). + +## Execution Flow +- View actions call `AutomationService` to start/control runs; long-running work executed by Celery tasks (`run_automation_task`, `resume_automation_task`). +- Config updates persist to DB; run state and logs updated as stages progress. + +## Cross-Module Interactions +- Uses planner/writer data for counts and processing; AI functions run inside service; billing credits checked via account balance before start and inferred from AI task logs during execution. + +## State Transitions +- Run status: running ↔ paused → completed/failed/cancelled; `current_stage` advances per stage; per-stage results stored in `AutomationRun`. +- Config `last_run_at`/`next_run_at` updated on scheduled runs. + +## Error Handling +- Missing site_id/run_id → 400; run not found → 404; insufficient credits or concurrent run → 400; server errors → 500. +- Logs endpoint 404s when run missing. + +## Tenancy Rules +- All operations scoped to the authenticated user’s account/site; no cross-tenant access; locks are per site. + +## Billing Rules +- Start requires credits ≥ estimated * 1.2; credits consumed by AI tasks are recorded via AI task logs and aggregated per stage. + +## Background Tasks / Schedulers +- Celery tasks for scheduled runs (`check_scheduled_automations`) and pipeline execution/resume. + +## Key Design Considerations +- Cooperative pause/resume/cancel with persisted partial results. +- Log access via filesystem tail; structured per-stage results stored in DB for quick status reads. + +## How Developers Should Work With This Module +- Trigger runs via `run_now`; avoid bypassing locks/credit checks. +- Extend actions by reusing `AutomationService` and Celery tasks; keep site scoping and error handling consistent. + diff --git a/master-docs/20-api/ENDPOINTS/billing.md b/master-docs/20-api/ENDPOINTS/billing.md index e69de29b..5bdcaba4 100644 --- a/master-docs/20-api/ENDPOINTS/billing.md +++ b/master-docs/20-api/ENDPOINTS/billing.md @@ -0,0 +1,77 @@ +# Billing Endpoints + +## Purpose +Detail billing API endpoints for invoices, payments, credit packages, transactions, payment methods, and credit balance/usage. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/business/billing/urls.py`, `backend/igny8_core/modules/billing/urls.py` +- Views: `backend/igny8_core/business/billing/views.py`, `backend/igny8_core/modules/billing/views.py` +- Services: `backend/igny8_core/business/billing/services/credit_service.py`, `invoice_service.py`, `payment_service.py` + +## High-Level Responsibilities +- Expose billing resources (invoices, payments, credit packages, payment methods, credit transactions). +- Provide credit balance/usage/limits endpoints. +- Support manual payment submission and available payment methods lookup. + +## Detailed Behavior +- Base paths: + - `/api/v1/billing/` (business billing views via router) + - `/api/v1/admin/` (admin billing URLs) + - `/api/v1/billing/credits/...` (balance/usage/transactions aliases) +- Invoice endpoints (`InvoiceViewSet`): + - `GET /invoices/` list account invoices (status filter optional). + - `GET /invoices/{id}/` invoice detail. + - `GET /invoices/{id}/download_pdf/` returns PDF bytes (placeholder). +- Payment endpoints (`PaymentViewSet`): + - `GET /payments/` list payments (status filter optional). + - `GET /payment-methods/available/` returns available methods (country/config driven). + - `POST /payments/manual/` submit manual payment for an invoice (requires invoice_id, payment_method, transaction_reference; rejects already paid). +- Account payment methods (`AccountPaymentMethodViewSet`): + - CRUD at `/payment-methods/`; `POST /payment-methods/{id}/set_default/` toggles default for the account. +- Credit packages (`CreditPackageViewSet`): + - `GET /credit-packages/` list active packages. + - `POST /credit-packages/{id}/purchase/` creates invoice and returns next_action (Stripe/PayPal placeholders; manual fallback returns invoice info). +- Credit transactions (`CreditTransactionViewSet`): + - `GET /transactions/` ledger of credit changes; also registered under `/credits/transactions/`. +- Credit balance/usage/limits (`modules/billing/views.py`): + - `GET /credits/balance/` returns credits, plan monthly credits, month-to-date credits used. + - `GET /credits/usage/` returns paginated usage logs with filters; `GET /credits/usage/summary/` aggregates by operation/model and totals; `GET /credits/usage/limits/` returns plan/account limits and usage counts. +- Permissions/tenancy: authenticated users; account-scoped querysets; viewsets rely on `request.account` or `request.user.account`. + +## Data Structures / Models Involved (no code) +- `Invoice`, `Payment`, `CreditPackage`, `CreditTransaction`, `AccountPaymentMethod`, `PaymentMethodConfig`, `CreditUsageLog`, `CreditCostConfig`, `Account`, `Plan`. + +## Execution Flow +- Router dispatch → viewset → service calls (invoice/payment/credit) → DB updates → unified responses. Balance/usage endpoints compute aggregates on demand. + +## Cross-Module Interactions +- Credit costs/checks used by writer/automation AI flows; balances/usage reflect those deductions. +- Plan limits surfaced in limits endpoint relate to account management in auth/account domains. + +## State Transitions +- Invoices: draft/pending/paid/void/uncollectible; Payments: pending/pending_approval/processing/succeeded/completed/failed/refunded/cancelled; Credits balance changes via transactions. + +## Error Handling +- Manual payment: 400 on missing fields or already-paid invoice. +- Credit operations: 402 returned by upstream callers on `InsufficientCreditsError`. +- Balance/usage: returns zeros/empty when account/plan missing. + +## Tenancy Rules +- All billing data filtered by account; no cross-tenant access. Default viewsets inherit account scoping from base classes. + +## Billing Rules +- Credit costs come from `CreditCostConfig` or constants; deductions recorded in ledger + usage logs. +- Available methods and packages are filtered by active flags/config. + +## Background Tasks / Schedulers +- None specific; Stripe/PayPal integrations are placeholders; any future webhooks should update invoices/payments/credits accordingly. + +## Key Design Considerations +- Ledger + usage logs kept in sync via `CreditService`. +- Manual payment flow avoids storing sensitive data; account payment methods store metadata only. + +## How Developers Should Work With This Module +- Use `CreditService` for credit math; do not mutate balances directly. +- Implement Stripe/PayPal flows in `PaymentService` if adding gateways; wire webhooks to update models. +- Keep account scoping on all queries; reuse existing viewsets for new billing features. + diff --git a/master-docs/20-api/ENDPOINTS/integration.md b/master-docs/20-api/ENDPOINTS/integration.md new file mode 100644 index 00000000..f3fdc9e2 --- /dev/null +++ b/master-docs/20-api/ENDPOINTS/integration.md @@ -0,0 +1,66 @@ +# Integration Endpoints + +## Purpose +Document integration API endpoints for managing site integrations, testing connections, syncing, and handling WordPress webhooks. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/modules/integration/urls.py` +- Views: `backend/igny8_core/modules/integration/views.py` +- Webhooks: `backend/igny8_core/modules/integration/webhooks.py` +- Services: `backend/igny8_core/business/integration/services/*` + +## High-Level Responsibilities +- CRUD for `SiteIntegration` records. +- Test connections (detail and collection-level), trigger syncs, fetch sync status, update site structure, and sync WordPress content. +- Expose WordPress webhook endpoints for status/metadata updates. + +## Detailed Behavior +- Base path: `/api/v1/integration/` +- Viewset: `IntegrationViewSet` (`/integrations/`, inherits `SiteSectorModelViewSet` but overrides queryset to filter by site_id since integrations have site only). + - Permissions: `IsAuthenticatedAndActive`, `IsEditorOrAbove`; throttle scope `integration`. + - Serializer exposes derived `api_key` (from credentials) and validates WordPress integrations require API key. + - Actions: + - `test_connection` (detail): calls `IntegrationService.test_connection` on existing integration; returns success/error. + - `test_connection_collection` (POST `integrations/test-connection/`, AllowAny): accepts site_id/api_key/site_url; validates site ownership or matching `Site.wp_api_key`; creates integration if missing; tests WordPress connection; deletes integration on failure; returns integration_id on success. + - `sync`: triggers `SyncService.sync` (direction metadata/both/to_external/from_external); metadata-only path uses `SyncMetadataService` internally. + - `sync_status`: returns sync status from `SyncService`. + - `update_site_structure`: updates `config_json.content_types` (post_types/taxonomies, plugin flags, timestamp) from WordPress push payload. + - `get_content_types`: returns content type config from integration config. + - `debug_status`: returns debug data via `SyncHealthService`. + - `sync_wordpress_content`: calls `ContentSyncService.sync_from_wordpress` for a specific content_id. +- Webhooks: + - `/webhooks/wordpress/status/` and `/webhooks/wordpress/metadata/` endpoints for WordPress callbacks (handled in `webhooks.py`). + +## Data Structures / Models Involved (no code) +- `SiteIntegration`, `SyncEvent`, `Site`, `Account`. + +## Execution Flow +- CRUD/test/sync requests → DRF auth → site-filtered queryset → service calls → model updates/sync actions. Collection-level test may create/delete integration based on result. Webhooks call handlers for incoming WordPress events. + +## Cross-Module Interactions +- Integrations used by publishing/sync flows to external platforms; WordPress API key path relies on `Site.wp_api_key` and integration credentials. +- Sync services interact with writer/planner content for metadata/content sync. + +## State Transitions +- Integration `sync_status/last_sync_at/is_active/sync_enabled` updated by sync/test actions; `SyncEvent` records per-action success/failure with timestamps and IDs. + +## Error Handling +- Invalid site/auth/API key return 403/404/400; unsupported platforms raise `NotImplementedError`; failed collection-level tests delete newly created integrations. + +## Tenancy Rules +- All endpoints scoped to the site/account; queryset filters by site_id and permissions enforce authenticated editor-or-above unless using collection-level API key path. + +## Billing Rules +- None; integration endpoints do not change credits. + +## Background Tasks / Schedulers +- None specific; sync logic executes in service calls (may delegate internally). + +## Key Design Considerations +- Single integration per site+platform; collection-level test supports plugin onboarding with API key. +- Credentials stored as JSON (encryption pending). + +## How Developers Should Work With This Module +- Use `IntegrationService` for CRUD/test; extend platform support via new test logic and serializer validation. +- Keep site scoping and API key validation intact for collection-level tests. + diff --git a/master-docs/20-api/ENDPOINTS/linker.md b/master-docs/20-api/ENDPOINTS/linker.md new file mode 100644 index 00000000..fd3d531c --- /dev/null +++ b/master-docs/20-api/ENDPOINTS/linker.md @@ -0,0 +1,48 @@ +# Linker Endpoints + +## Purpose +Document linker API endpoints for internal linking operations. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/modules/linker/urls.py` +- Views: `backend/igny8_core/modules/linker/views.py` +- Services: `backend/igny8_core/business/linking/services/*` + +## High-Level Responsibilities +- Provide linker-related endpoints (as defined in `LinkerViewSet`) for managing internal linking logic. + +## Detailed Behavior +- Base path: `/api/v1/linker/` +- Router registers root `LinkerViewSet` (see module code for actions/fields). +- Inherits base viewset scoping, unified responses, permissions, throttling defined in module views. + +## Data Structures / Models Involved (no code) +- Linking-related models/services (see business/linking). + +## Execution Flow +- Requests → DRF auth → LinkerViewSet → service logic → unified response. + +## Cross-Module Interactions +- May consume writer content to generate linking recommendations; interacts with linker services. + +## State Transitions +- As defined by LinkerViewSet actions (not detailed in code snippet). + +## Error Handling +- Standard unified responses and throttling. + +## Tenancy Rules +- Inherits account/site scoping from base classes. + +## Billing Rules +- None specified. + +## Background Tasks / Schedulers +- None specified in endpoints. + +## Key Design Considerations +- Endpoint actions rely on business/linking services; keep scoping intact. + +## How Developers Should Work With This Module +- Extend LinkerViewSet/actions as needed; ensure base scoping and unified responses remain consistent. + diff --git a/master-docs/20-api/ENDPOINTS/optimizer.md b/master-docs/20-api/ENDPOINTS/optimizer.md new file mode 100644 index 00000000..5a6561ec --- /dev/null +++ b/master-docs/20-api/ENDPOINTS/optimizer.md @@ -0,0 +1,48 @@ +# Optimizer Endpoints + +## Purpose +Document optimizer API endpoints for content optimization tasks. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/modules/optimizer/urls.py` +- Views: `backend/igny8_core/modules/optimizer/views.py` +- Models/Services: `backend/igny8_core/business/optimization/models.py`, `services/*` + +## High-Level Responsibilities +- Expose optimizer-related endpoints (as defined in `OptimizerViewSet`) to manage optimization tasks and results. + +## Detailed Behavior +- Base path: `/api/v1/optimizer/` +- Router registers root `OptimizerViewSet` (see module code for actions/fields). +- Inherits base viewset scoping, unified responses, permissions, throttling defined in module views. + +## Data Structures / Models Involved (no code) +- `OptimizationTask` and related optimization services. + +## Execution Flow +- Requests → DRF auth → OptimizerViewSet → service logic → unified response. + +## Cross-Module Interactions +- Operates on writer `Content`; may influence publishing readiness and optimization scores. + +## State Transitions +- As defined by OptimizerViewSet actions (e.g., pending/running/completed/failed). + +## Error Handling +- Standard unified responses and throttling. + +## Tenancy Rules +- Inherits account/site scoping from base classes. + +## Billing Rules +- Optimization credits are tracked in optimization tasks; deductions occur in billing services when AI optimization runs (see optimization module docs). + +## Background Tasks / Schedulers +- Optimization runs may execute via services/Celery; endpoints trigger/manage these flows. + +## Key Design Considerations +- Keep scoping intact; reuse services for optimization execution. + +## How Developers Should Work With This Module +- Extend OptimizerViewSet with new actions as needed; ensure billing/credit checks are applied when adding AI-driven optimization. + diff --git a/master-docs/20-api/ENDPOINTS/payments.md b/master-docs/20-api/ENDPOINTS/payments.md index e69de29b..90e9032f 100644 --- a/master-docs/20-api/ENDPOINTS/payments.md +++ b/master-docs/20-api/ENDPOINTS/payments.md @@ -0,0 +1,53 @@ +# Payments Endpoints + +## Purpose +Document payment-related endpoints exposed under billing APIs. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/business/billing/urls.py` +- Views: `backend/igny8_core/business/billing/views.py` (`PaymentViewSet`) + +## High-Level Responsibilities +- List payments for the current account and submit manual payments tied to invoices. +- Expose available payment methods per account/country configuration. + +## Detailed Behavior +- Base path: `/api/v1/billing/` +- `PaymentViewSet`: + - `GET /payments/` → list payments (account-scoped; optional status filter). Returns amount, currency, payment_method, status, timestamps, invoice linkage, transaction_reference, failure_reason. + - `GET /payment-methods/available/` → returns available methods (driven by `PaymentService.get_available_payment_methods`; includes methods array plus metadata). + - `POST /payments/manual/` → submit manual payment for an invoice: + - Body: `invoice_id`, `payment_method` (`bank_transfer` or `local_wallet`), `transaction_reference` (or `reference`), optional `notes`. + - Validates invoice belongs to account and is not already paid; creates payment via `PaymentService.create_manual_payment`; returns pending-approval status and payment id. +- Permissions: authenticated users; account scoping via `request.user.account`. + +## Data Structures / Models Involved (no code) +- `Payment`, `Invoice`, `Account`, `PaymentMethodConfig`, `AccountPaymentMethod`. + +## Execution Flow +- Router dispatch → PaymentViewSet → service calls → model updates → unified responses. Manual payment creates a pending payment record for later approval/processing. + +## Cross-Module Interactions +- Payments link to invoices; account scoping enforced. Available methods depend on payment method configs; manual payments may require admin follow-up. + +## State Transitions +- Payments move through statuses (`pending`, `pending_approval`, `processing`, `succeeded/completed`, `failed`, `refunded`, `cancelled`). + +## Error Handling +- 400 on missing required fields or already-paid invoice; 404 if invoice not found for account. + +## Tenancy Rules +- All queries scoped to `request.user.account`; no cross-tenant access. + +## Billing Rules +- Payment records tie to invoices; credit balance changes happen when payments are processed (handled in payment service/admin flows). + +## Background Tasks / Schedulers +- None specific in endpoints; payment processing/approval handled by services/admin workflows. + +## Key Design Considerations +- Manual payment endpoint avoids storing sensitive data; relies on transaction_reference and admin approval. + +## How Developers Should Work With This Module +- Use manual payment endpoint for bank/local wallet flows; implement gateway flows (Stripe/PayPal) in `PaymentService` and add endpoints if needed; keep account scoping and invoice validation. + diff --git a/master-docs/20-api/ENDPOINTS/planner.md b/master-docs/20-api/ENDPOINTS/planner.md index e69de29b..25061312 100644 --- a/master-docs/20-api/ENDPOINTS/planner.md +++ b/master-docs/20-api/ENDPOINTS/planner.md @@ -0,0 +1,62 @@ +# Planner Endpoints + +## Purpose +Document planner API endpoints for keywords, clusters, and content ideas, including filters, bulk actions, and validation rules. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/modules/planner/urls.py` +- Views: `backend/igny8_core/modules/planner/views.py` +- Serializers: `backend/igny8_core/modules/planner/serializers.py` + +## High-Level Responsibilities +- CRUD for planner entities scoped by account/site/sector with search, filtering, ordering, and bulk operations. +- Enforce site/sector requirements and seed keyword alignment on creation. + +## Detailed Behavior +- Base path: `/api/v1/planner/` +- Viewsets (all inherit `SiteSectorModelViewSet`, scoped by account/site/sector, throttled with scope `planner`, paginated): + - `KeywordViewSet` (`/keywords/`): + - Search: `seed_keyword__keyword`. + - Filters: status, cluster_id, seed_keyword intent/id; custom difficulty_min/max and volume_min/max (uses overrides or seed values). + - Ordering: created_at, seed volume/difficulty. + - Bulk: `bulk_delete`, `bulk_update` (status), `bulk_add_from_seed` (create Keywords from SeedKeywords after validating site/sector). + - Create requires site_id and sector_id; validates site/sector existence and alignment. + - `ClusterViewSet` (`/clusters/`): + - Filters: status; standard CRUD; site/sector required on create. + - `ContentIdeasViewSet` (`/ideas/`): + - Filters: status, cluster; standard CRUD; site/sector required on create. +- Serializers enforce required fields on create (`KeywordSerializer` requires seed_keyword_id; site/sector write-only), expose read-only seed-derived fields and cluster/sector names. +- Error handling: unified responses; validation errors for missing site/sector/seed; bulk actions return 400 when IDs/status missing. + +## Data Structures / Models Involved (no code) +- `Keywords`, `Clusters`, `ContentIdeas`, `SeedKeyword`, plus `Account`, `Site`, `Sector`. + +## Execution Flow +- Requests → DRF auth → base viewset filters (account/site/sector) → serializer validation → model save → unified response; bulk actions operate on filtered queryset. + +## Cross-Module Interactions +- Feeds automation Stage 1–3 and writer tasks/content generation. +- Billing credits may be checked upstream when AI clustering/idea generation is invoked. + +## State Transitions +- Keywords: `new` → `mapped`; Clusters: `new` → `mapped`; ContentIdeas: `new` → `queued` → `completed`. + +## Error Handling +- Missing required IDs or validation failures return 400; unified 500 on unhandled errors. + +## Tenancy Rules +- All endpoints scoped to account/site/sector via base viewset; admin/developer/system roles can bypass filtering, but data retains tenant/site/sector fields. + +## Billing Rules +- No direct deductions in these endpoints; AI clustering/idea generation costs enforced where services are called. + +## Background Tasks / Schedulers +- None in planner endpoints; automation handles async flows. + +## Key Design Considerations +- Strict site/sector requirements prevent cross-site contamination. +- Seed linkage keeps keyword metadata consistent; overrides allow site-specific tuning. + +## How Developers Should Work With This Module +- Supply site_id/sector_id on create; use bulk actions for batch changes; extend filters/search/order as needed while respecting base scoping. + diff --git a/master-docs/20-api/ENDPOINTS/publisher.md b/master-docs/20-api/ENDPOINTS/publisher.md new file mode 100644 index 00000000..3140df28 --- /dev/null +++ b/master-docs/20-api/ENDPOINTS/publisher.md @@ -0,0 +1,53 @@ +# Publisher Endpoints + +## Purpose +Document publisher API endpoints for publishing records and deployments. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/modules/publisher/urls.py` +- Viewsets: `backend/igny8_core/modules/publisher/views.py` +- Models: `backend/igny8_core/business/publishing/models.py` + +## High-Level Responsibilities +- Expose CRUD for publishing records (content publishing to destinations) and deployment records (site deployments). +- Scope resources by account/site/sector and provide unified responses. + +## Detailed Behavior +- Base path: `/api/v1/publisher/` +- Routers register: + - `publishing-records/` → `PublishingRecordViewSet` + - `deployments/` → `DeploymentRecordViewSet` + - Root publisher viewset (`PublisherViewSet`) for additional publisher actions (see module code). +- Viewsets inherit account/site/sector filtering via base classes; permissions/throttles defined in module views. +- Publishing records track destination/status/URLs/errors; deployments track versions/status/URL/metadata. + +## Data Structures / Models Involved (no code) +- `PublishingRecord`, `DeploymentRecord`, `Account`, `Site`, `Sector`. + +## Execution Flow +- Requests → DRF auth → base scoping → serializer validation → model CRUD → unified responses. + +## Cross-Module Interactions +- Publishing records reference writer content; deployments reference sites; integration data may influence destinations (see integration/publishing services). + +## State Transitions +- Publishing status: pending/publishing/published/failed; Deployment status: pending/deploying/deployed/failed/rolled_back. + +## Error Handling +- Unified responses; validation errors for missing/invalid fields; standard 4xx/5xx on failures. + +## Tenancy Rules +- All endpoints scoped to account/site/sector via base viewsets and permissions; privileged roles may bypass filtering but data retains tenant fields. + +## Billing Rules +- None directly in publisher endpoints; any credit usage would occur in upstream AI flows. + +## Background Tasks / Schedulers +- None specific in endpoints; deployments/publishing actions may be driven by services or Celery tasks elsewhere. + +## Key Design Considerations +- Separation of publishing records and deployment records keeps content publishing and site deployment distinct. + +## How Developers Should Work With This Module +- Use existing viewsets; maintain site/sector scoping and unified response patterns when extending publisher functionality. + diff --git a/master-docs/20-api/ENDPOINTS/sites.md b/master-docs/20-api/ENDPOINTS/sites.md index e69de29b..b80d20e7 100644 --- a/master-docs/20-api/ENDPOINTS/sites.md +++ b/master-docs/20-api/ENDPOINTS/sites.md @@ -0,0 +1,71 @@ +# Site Endpoints + +## Purpose +Document site management endpoints under the auth module, including creation, listing, sector selection, and activation. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/auth/urls.py` +- Views: `backend/igny8_core/auth/views.py` (`SiteViewSet`) +- Serializers: `backend/igny8_core/auth/serializers.py` (`SiteSerializer`, `SectorSerializer`) + +## High-Level Responsibilities +- CRUD for sites scoped to an account, with role-based access. +- Allow public read by slug for active sites (renderer support). +- Manage sector selection and activation per site. + +## Detailed Behavior +- Base path: `/api/v1/auth/sites/` +- Permissions: + - `list` with slug filter: `AllowAny` (public read by slug, active sites only). + - `create`: authenticated users (viewer+); sets site account from request/user. + - Other actions: `IsEditorOrAbove` with tenant access. +- Queryset logic: + - Unauthenticated with `slug` query param: returns active site by slug. + - Authenticated admin/developer: all sites. + - Authenticated owner/admin: sites for user’s account. + - Other users: sites where they have `SiteUserAccess`. +- Actions: + - `GET /sites/` list (with filters above). + - `POST /sites/` create site; account set from request/user. + - `GET /sites/{id}/sectors/` list active sectors for the site. + - `POST /sites/{id}/set_active/` marks site active (no deactivation of others). + - `POST /sites/{id}/select_sectors/` sets industry and sectors: + - Requires `industry_slug`; optional `sector_slugs` list. + - Validates industry exists/active; enforces plan sector limit (`get_max_sectors_limit`). + - Deactivates sectors not in provided list; ensures sectors belong to industry; creates missing site sectors as needed; returns site + sectors. +- Create/update: + - `perform_create` sets account; no single-active constraint. + - `perform_update` retains account; no single-active constraint. + +## Data Structures / Models Involved (no code) +- `Site`, `Sector`, `Industry`, `IndustrySector`, `Plan` (for sector limits), `SiteUserAccess`, `Account`. + +## Execution Flow +- Requests → DRF auth (JWT/session) → permissions → queryset scoping → serializer validation → model ops → unified response. Public slug reads bypass account filtering. + +## Cross-Module Interactions +- Sites link to planner/writer records via site/sector; integration uses `Site.wp_api_key`; plan sector limits referenced here. + +## State Transitions +- Site activation via `set_active`; sector activation/deactivation via `select_sectors`; industry assignment updated on selection. + +## Error Handling +- 400 for missing industry slug or exceeding sector limits; 404 for missing site/industry; 500 if site lacks account when selecting sectors. + +## Tenancy Rules +- Account-scoped except public slug read; role-based overrides for admins/developers; SiteUserAccess governs non-admin access. + +## Billing Rules +- Sector limit enforced from plan (`max_industries` via `get_max_sectors_limit`). + +## Background Tasks / Schedulers +- None in site endpoints. + +## Key Design Considerations +- Public slug access supports site renderer without auth. +- Sector selection enforces industry/plan constraints and deactivates unselected sectors. + +## How Developers Should Work With This Module +- Use `select_sectors` to set industry/sector with plan-aware limits. +- When extending site data, keep public slug path safe and scoped to active sites; maintain role-based access controls. + diff --git a/master-docs/20-api/ENDPOINTS/system.md b/master-docs/20-api/ENDPOINTS/system.md new file mode 100644 index 00000000..b7f8ec2e --- /dev/null +++ b/master-docs/20-api/ENDPOINTS/system.md @@ -0,0 +1,61 @@ +# System Endpoints + +## Purpose +Document system module endpoints for AI prompts, author profiles, strategies, and system/settings resources. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/modules/system/urls.py` +- Views: `backend/igny8_core/modules/system/views.py` +- Models/serializers: `backend/igny8_core/modules/system/models.py`, `serializers.py` + +## High-Level Responsibilities +- Manage account-scoped AI prompts, author profiles, and strategies. +- Expose system/user/module/AI settings endpoints. +- Provide prompt retrieval by type and save/reset helpers with role checks. + +## Detailed Behavior +- Base path: `/api/v1/system/` +- Routers register: + - `prompts/` → `AIPromptViewSet` (CRUD, `by_type`, `save_prompt`, `reset_prompt`) + - `author-profiles/` → `AuthorProfileViewSet` (CRUD) + - `strategies/` → `StrategyViewSet` (CRUD) + - `settings/system`, `settings/account`, `settings/user`, `settings/modules`, `settings/ai` → respective settings viewsets +- AIPromptViewSet: + - Permissions: `IsAuthenticatedAndActive`, `HasTenantAccess`; throttle scope `system`; paginated. + - `by_type` returns prompt by prompt_type, or default when missing. + - `save_prompt`/`reset_prompt` require editor-or-above; set/read account from request/user or first account fallback (dev). + - Queryset ordered by prompt_type. +- Settings viewsets expose retrieve/update for their scope (system/account/user/modules/ai). Permissions and scoping follow AccountModelViewSet. +- AuthorProfile/Strategy viewsets provide standard CRUD under account scoping. + +## Data Structures / Models Involved (no code) +- `AIPrompt` (prompt_type, prompt_value, default_prompt, is_active), `AuthorProfile`, `Strategy`, settings models. + +## Execution Flow +- Requests → DRF auth → AccountModelViewSet filtering → serializer validation → model CRUD → unified responses. Custom actions handle prompt save/reset/type lookups. + +## Cross-Module Interactions +- Prompts are consumed by AI functions; settings may influence module behavior (e.g., AI config). + +## State Transitions +- Prompt updates, resets; settings updates; author profile/strategy CRUD. + +## Error Handling +- Unified errors; 403 when lacking editor role for prompt save/reset; 400 on missing prompt_type/value. + +## Tenancy Rules +- All resources are account scoped via AccountModelViewSet and permissions. + +## Billing Rules +- None; prompts/settings do not affect credits directly. + +## Background Tasks / Schedulers +- None. + +## Key Design Considerations +- Prompt defaults are returned when custom prompt missing to keep AI flows resilient. +- Editor-or-above requirement protects prompt modifications. + +## How Developers Should Work With This Module +- Use provided viewsets/actions; ensure account context is set. Extend settings endpoints by adding serializers/models and wiring to the router. + diff --git a/master-docs/20-api/ENDPOINTS/writer.md b/master-docs/20-api/ENDPOINTS/writer.md index e69de29b..f0315cf1 100644 --- a/master-docs/20-api/ENDPOINTS/writer.md +++ b/master-docs/20-api/ENDPOINTS/writer.md @@ -0,0 +1,68 @@ +# Writer Endpoints + +## Purpose +Document writer API endpoints for tasks, content, images, and taxonomies, including bulk actions and AI content generation triggers. + +## Code Locations (exact paths) +- Routing: `backend/igny8_core/modules/writer/urls.py` +- Views: `backend/igny8_core/modules/writer/views.py` +- Serializers: `backend/igny8_core/modules/writer/serializers.py` +- Services: `backend/igny8_core/business/content/services/content_generation_service.py` + +## High-Level Responsibilities +- CRUD for writer entities scoped by account/site/sector. +- Provide bulk operations and AI content generation for tasks. +- Manage images and taxonomy terms linked to content/tasks. + +## Detailed Behavior +- Base path: `/api/v1/writer/` +- Viewsets (inherit `SiteSectorModelViewSet`, throttled `writer`, paginated): + - `TasksViewSet` (`/tasks/`): + - Filters: status, cluster_id, content_type, content_structure; search title/keywords; ordering by title/created_at/status. + - Bulk: `bulk_delete`, `bulk_update` (status). + - `auto_generate_content`: accepts task IDs (max 10), requires account, enforces credit checks via `ContentGenerationService`, returns async task_id or synchronous result; 402 on insufficient credits. + - Create requires site_id/sector_id; validates site/sector alignment; sets account/site/sector explicitly. + - `ImagesViewSet` (`/images/`): + - Filters: task_id, content_id, image_type, status; ordering by created_at/position/id. + - perform_create enforces site/sector presence (from request or defaults), sets account/site/sector; serves local files via `file` action with existence checks. + - `ContentViewSet` (`/content/`): + - Filters/order/search defined in viewset (titles, status, etc.); manages taxonomy relationships; exposes status, SEO fields, cluster link, external IDs/URLs. + - `ContentTaxonomyViewSet` (`/taxonomies/`): + - CRUD for taxonomy terms (category/tag) scoped to site/sector. +- Serializers enforce required fields on create (e.g., cluster/content_type/content_structure for tasks; site/sector for content) and expose read-only derived fields (cluster/sector names, image/taxonomy info). +- Error handling: unified responses; validation errors return 400; missing resources return 404; `auto_generate_content` can return 402/500 on credits/other errors. + +## Data Structures / Models Involved (no code) +- `Tasks`, `Content`, `Images`, `ContentTaxonomy`, and related domain models; `Account`, `Site`, `Sector`. + +## Execution Flow +- Requests → DRF auth → base viewset filtering (account/site/sector) → serializer validation → model save → unified response; bulk/AI actions invoke services. + +## Cross-Module Interactions +- Planner clusters/ideas feed tasks; automation stages 3–6 operate on writer data; publishing/integration uses content/taxonomies/images. +- Billing credits enforced during AI content generation via `ContentGenerationService`. + +## State Transitions +- Tasks: `queued` → `completed`; Content: `draft` → `review` → `published`; Images: `pending` → `generated`. + +## Error Handling +- Standard unified responses; credit errors mapped to 402 in `auto_generate_content`; file serving returns 404 for missing images. + +## Tenancy Rules +- All endpoints scoped to account/site/sector; base viewset and permissions enforce access; admin/developer/system may bypass filtering but records retain tenant/site/sector fields. + +## Billing Rules +- AI generation actions check/deduct credits via billing services; other CRUD does not alter credits. + +## Background Tasks / Schedulers +- `auto_generate_content` may enqueue Celery AI tasks when returning task_id; automation uses writer endpoints indirectly via services. + +## Key Design Considerations +- Strict site/sector validation on create; max 10 tasks per auto_generate_content to manage load/credits. +- Image file serving is permissive but guarded by existence checks. + +## How Developers Should Work With This Module +- Supply site_id/sector_id on create; use bulk actions for batch changes. +- When extending AI flows, keep credit checks and account/site scoping intact. +- Reuse existing viewsets/serializers for new fields; ensure filters/throttles stay aligned with module scope. + diff --git a/master-docs/20-api/REST-API-REFERENCE.md b/master-docs/20-api/REST-API-REFERENCE.md index e69de29b..b3af1e92 100644 --- a/master-docs/20-api/REST-API-REFERENCE.md +++ b/master-docs/20-api/REST-API-REFERENCE.md @@ -0,0 +1,73 @@ +# REST API Reference + +## Purpose +Summarize the major API groups, their base paths, and key actions as implemented in routing and viewsets. + +## Code Locations (exact paths) +- Root routing: `backend/igny8_core/urls.py` +- Auth endpoints: `backend/igny8_core/auth/urls.py` (APIView + routers) +- Account management: `backend/igny8_core/api/urls.py` +- Planner: `backend/igny8_core/modules/planner/urls.py` +- Writer: `backend/igny8_core/modules/writer/urls.py` +- System: `backend/igny8_core/modules/system/urls.py` +- Billing: `backend/igny8_core/business/billing/urls.py`, `backend/igny8_core/modules/billing/urls.py` +- Automation: `backend/igny8_core/business/automation/urls.py` +- Integration: `backend/igny8_core/modules/integration/urls.py` +- Linker/Optimizer/Publisher: respective `modules/*/urls.py` + +## High-Level Responsibilities +- Provide RESTful CRUD endpoints for core resources (auth, planner, writer, billing, automation, integration, etc.) plus custom actions for domain workflows (automation run, billing payments, integration sync, content generation). + +## Detailed Behavior (by group) +- Authentication (`/api/v1/auth/`): + - Routers: groups, users, accounts, subscriptions, site-access, plans, sites, sectors, industries, seed-keywords. + - APIViews: register, login, change password, refresh token; responses issue JWTs and session login. + - CSV admin helpers exposed separately under `/admin/igny8_core_auth/...` for industries/seed keywords. +- Account management (`/api/v1/account/`): + - Settings (`settings/` get/patch), team (`team/` list/create, `team//` delete), usage analytics (`usage/analytics/`). +- Planner (`/api/v1/planner/`): + - `keywords/`, `clusters/`, `ideas/` (CRUD, filtering/search/ordering, bulk delete/update, bulk add from seed). +- Writer (`/api/v1/writer/`): + - `tasks/`, `images/`, `content/`, `taxonomies/` (CRUD, filtering/search/ordering). + - Custom actions: `tasks/auto_generate_content` (AI generation), `tasks/bulk_*`, `images/file`, grouped image utilities; content/taxonomy helpers in viewset. +- System (`/api/v1/system/`): system settings, prompts, author profiles, etc. (see module docs). +- Billing (`/api/v1/billing/`): + - Invoices (`invoices/`, detail, `download_pdf`), payments (`payments/`, `available_methods`, manual payment), credit packages (`credit-packages/`, purchase), transactions (`transactions/`), payment methods (`payment-methods/`, `set_default`), credits (`credits/balance`, `credits/usage`, `credits/transactions`). + - Admin billing under `/api/v1/admin/` (see module billing admin URLs). +- Automation (`/api/v1/automation/`): + - Config (`config`, `update_config`), run (`run_now`), current run, history, logs, estimate, pipeline_overview, current_processing, pause/resume/cancel. +- Integration (`/api/v1/integration/`): + - `integrations/` CRUD, `test_connection`, collection-level `test-connection`, `sync`, `sync_status`, `update-structure`, `get_content_types`, `debug_status`, `sync_wordpress_content`. + - Webhooks: `/webhooks/wordpress/status/`, `/webhooks/wordpress/metadata/`. +- Linker/Optimizer/Publisher: endpoints under `/api/v1/linker/`, `/api/v1/optimizer/`, `/api/v1/publisher/` (viewsets for respective domain resources; see module docs). + +## Data Structures / Models Involved (no code) +- Auth/account models, planner models (Keywords/Clusters/ContentIdeas), writer models (Tasks/Content/Images/Taxonomy), billing models (invoices/payments/credits), automation models (runs/configs), integration models (SiteIntegration/SyncEvent), and other module models as routed above. + +## Execution Flow +- Requests hit module routers; viewsets/actions enforce permissions, throttles, and tenancy; serializers map models; responses use unified format. + +## Cross-Module Interactions +- Auth JWTs used by all modules. +- Billing credits checked in writer/automation flows; billing endpoints expose balances/usage. +- Integration and publishing rely on writer content; automation depends on planner/writer data. + +## State Transitions +- Managed by respective viewsets (e.g., task status, automation status, invoice/payment status). + +## Error Handling +- Unified error responses; 4xx/5xx codes per viewset logic; throttling per scope. + +## Tenancy Rules +- Base viewsets enforce account/site/sector filtering; API key auth supports WordPress paths; admin/developer/system roles may bypass filters as coded in base viewsets/auth middleware. + +## Billing Rules +- Credits enforced in writer/automation service calls; billing endpoints expose ledger/usage/balance; insufficient credits return 402 where applicable. + +## Background Tasks / Schedulers +- Automation actions enqueue Celery runs; writer content generation may enqueue AI tasks; other modules mostly synchronous. + +## Key Design Considerations +- Consistent `/api/v1/` namespace with per-module routers. +- OpenAPI schema available at `/api/schema`, Swagger at `/api/docs`, Redoc at `/api/redoc`. +- Use module docs for endpoint-level details; this file outlines the map of API groups and actions. diff --git a/master-docs/30-frontend/FRONTEND-ARCHITECTURE.md b/master-docs/30-frontend/FRONTEND-ARCHITECTURE.md index e69de29b..722120bb 100644 --- a/master-docs/30-frontend/FRONTEND-ARCHITECTURE.md +++ b/master-docs/30-frontend/FRONTEND-ARCHITECTURE.md @@ -0,0 +1,84 @@ +# Frontend Architecture + +## Purpose +Explain the frontend structure, routing, state management, providers, and module organization in the React/Vite app. + +## Code Locations (exact paths) +- Entry: `frontend/src/main.tsx` +- Root app/routing: `frontend/src/App.tsx` +- State (Zustand stores): `frontend/src/store/*` (authStore, siteStore, billingStore, settingsStore, onboardingStore, columnVisibilityStore, pageSizeStore) +- API layer: `frontend/src/api/*` and `frontend/src/services/api.ts` +- Layout and components: `frontend/src/layout/*`, `frontend/src/components/*` +- Pages: `frontend/src/pages/*` (modules and settings) +- Config: `frontend/src/config/*` +- Styles: `frontend/src/index.css`, `frontend/src/styles/igny8-colors.css` + +## High-Level Responsibilities +- Provide SPA routing with protected routes and module guards. +- Maintain global state for auth, site context, billing, settings, UI preferences via Zustand. +- Lazy-load module pages to optimize initial load. +- Wrap the app with providers for theming, toasts, header metrics, error boundary, and router. + +## Detailed Behavior +- Entry (`main.tsx`): + - Imports global styles and vendor CSS (Swiper, Flatpickr). + - Wraps the app with `ErrorBoundary`, `ThemeProvider`, `HeaderMetricsProvider`, `ToastProvider`, `BrowserRouter`, then renders `` via React 18 `createRoot`. +- Routing (`App.tsx`): + - Uses `BrowserRouter` (in main) with ``/`` inside App. + - Public routes: `/signin`, `/signup`. + - Protected routes wrapped by `ProtectedRoute` → `AppLayout`; `ModuleGuard` used per-module. + - Default dashboard `/` loads `Dashboard/Home`. + - Planner routes redirect `/planner` → `/planner/keywords`; writer routes redirect `/writer` → `/writer/tasks`. + - Extensive lazy-loaded pages for Planner, Writer, Automation, Linker, Optimizer, Thinker (system), Billing, Admin, Settings, Sites, Help, Reference, Components, UI elements, etc. + - `GlobalErrorDisplay` and `LoadingStateMonitor` mounted globally; `ScrollToTop` for navigation changes. + - Auth refresh: on mount, if authenticated with token, calls `refreshUser`; on failure with missing credentials message, ignores; otherwise logs out. +- State (Zustand examples): + - `authStore`: persists user/token/refreshToken; `login`/`register` hit `/api/v1/auth/...`; enforces account and active plan presence; `refreshUser` fetches user from `/api/v1/auth/users/me/`; `refreshToken` hits `/api/v1/auth/refresh/`; logout clears all auth fields. + - `siteStore`: stores current site/sector and list; used for context across modules. + - `billingStore`, `settingsStore`, `onboardingStore`, `columnVisibilityStore`, `pageSizeStore`: manage billing data, settings flags, onboarding steps, table visibility, pagination sizes. +- API layer: + - `services/api.ts` provides `fetchAPI` helper for authenticated calls (uses token from auth store, handles JSON, throws on non-OK). + - `api/*` modules define typed client calls per domain (auth, planner, writer, etc.). +- Layout/components: + - `AppLayout` wraps sidebar/header/content; `ModuleGuard` enforces module access flags; `ProtectedRoute` enforces auth. + - UI utilities: toasts, error boundary, global loading monitor, scroll-to-top, etc. + +## Data Structures / Models Involved (no code) +- Frontend state shapes defined in Zustand stores (auth user, site context, billing info, settings). + +## Execution Flow +- App bootstrap → providers → App routes → ProtectedRoute checks auth → ModuleGuard checks module access → lazy-loaded pages fetch data via `api/*` using tokens from authStore → state updates via stores. + +## Cross-Module Interactions +- Auth store tokens used by fetchAPI for all modules. + - Site store context influences planner/writer/automation calls. + - Billing store interacts with billing endpoints; settings store influences UI/features; onboarding/table stores affect page rendering. + +## State Transitions +- Auth: login/register → set user/token; refresh → update tokens/user; logout clears. +- Site context changes update current site/sector for downstream calls. +- Billing/settings/onboarding/table state mutate via respective store actions. + +## Error Handling +- `GlobalErrorDisplay` shows errors; `LoadingStateMonitor` tracks loading; `ProtectedRoute` logs out on failed refresh. +- `fetchAPI` throws on non-OK; authStore catches and surfaces messages; ModuleGuard/ProtectedRoute handle unauthorized access. + +## Tenancy Rules +- Enforced via tokens and site context; ModuleGuard and ProtectedRoute rely on backend responses (account/plan) and module flags. + +## Billing Rules +- Auth store enforces active plan on login; billing store/pages call billing endpoints for balances/transactions/purchases. + +## Background Tasks / Schedulers +- None on frontend; relies on backend/Celery for async work; uses Suspense for lazy loading. + +## Key Design Considerations +- Lazy-loaded routes minimize initial bundle. +- Zustand persistence keeps auth/session across reloads. +- Providers wrap app for theming, toasts, error boundary to improve UX stability. + +## How Developers Should Work With This Module +- Add routes inside `App.tsx` under ProtectedRoute for authenticated pages; use ModuleGuard for module-gated pages. +- Create new stores in `src/store` for shared state; use `fetchAPI` for authenticated requests. +- Keep lazy loading for heavy pages and maintain Suspense fallbacks where needed. + diff --git a/master-docs/30-frontend/GLOBAL-UI-COMPONENTS.md b/master-docs/30-frontend/GLOBAL-UI-COMPONENTS.md index e69de29b..533667f7 100644 --- a/master-docs/30-frontend/GLOBAL-UI-COMPONENTS.md +++ b/master-docs/30-frontend/GLOBAL-UI-COMPONENTS.md @@ -0,0 +1,76 @@ +# Global UI Components and Providers + +## Purpose +Describe global layout, guards, and utility components/providers used throughout the frontend. + +## Code Locations (exact paths) +- App composition and routing: `frontend/src/App.tsx` +- Entry/providers: `frontend/src/main.tsx` +- Layout: `frontend/src/layout/AppLayout.tsx`, `frontend/src/layout/AppSidebar.tsx`, `frontend/src/layout/AppHeader.tsx` (and related layout files) +- Guards: `frontend/src/components/auth/ProtectedRoute.tsx`, `frontend/src/components/common/ModuleGuard.tsx` +- Global utilities: `frontend/src/components/common/ScrollToTop.tsx`, `frontend/src/components/common/GlobalErrorDisplay.tsx`, `frontend/src/components/common/LoadingStateMonitor.tsx`, `frontend/src/components/common/ErrorBoundary.tsx` +- Providers: `frontend/src/context/ThemeContext.tsx`, `frontend/src/context/HeaderMetricsContext.tsx`, `frontend/src/components/ui/toast/ToastContainer.tsx` + +## High-Level Responsibilities +- Wrap the app with error handling, theming, metrics, toasts, and routing. +- Enforce authentication and module access at the route level. +- Provide global UI behaviors (scroll reset, error banner, loading monitor). +- Supply consistent layout (sidebar/header/content) for protected areas. + +## Detailed Behavior +- Providers (`main.tsx`): + - `ErrorBoundary` wraps the entire app. + - `ThemeProvider` supplies theme context. + - `HeaderMetricsProvider` supplies header metrics context. + - `ToastProvider` exposes toast notifications. + - `BrowserRouter` provides routing; renders ``. +- Routing shell (`App.tsx`): + - `GlobalErrorDisplay` renders global errors; `LoadingStateMonitor` tracks loading states. + - `ScrollToTop` resets scroll on route changes. + - Public routes: `/signin`, `/signup`. + - Protected routes: wrapped in `ProtectedRoute` → `AppLayout`; `ModuleGuard` used per-module. + - Lazy-loaded module pages inside routes; ModuleGuard enforces module access flags. +- Guards: + - `ProtectedRoute`: checks `useAuthStore` for authentication; redirects unauthenticated users to sign-in; logs out on failed refresh in App effect. + - `ModuleGuard`: gates module pages based on module enable settings/permissions. +- Layout: + - `AppLayout` composes sidebar/header/content; `AppSidebar` uses auth store for user/nav; header components provide top-level actions and metrics. +- Utilities: + - `ScrollToTop` listens to route changes and scrolls to top. + - `GlobalErrorDisplay` shows global error banners. + - `LoadingStateMonitor` tracks loading indicators globally. + - `ToastProvider` supplies toast UI primitives for notifications. + +## Data Structures / Models Involved (no code) +- Context values from theme, header metrics, toast providers; auth state from `authStore`; module enable settings from `settingsStore`. + +## Execution Flow +- App bootstrap wraps providers → routes render → ProtectedRoute checks auth → ModuleGuard checks module access → layout renders with sidebar/header → pages load lazily and fetch data. + +## Cross-Module Interactions +- Auth/module settings drive guards; toasts/errors/loading monitors are available to all pages; layout navigation links modules. + +## Error Handling +- `ErrorBoundary` catches render errors. + - `GlobalErrorDisplay` surfaces application-level errors. + - `ProtectedRoute` logs out on failed refresh in App effect; unauthorized users are redirected. + +## Tenancy Rules +- Enforced via backend auth and module enable settings; guards rely on auth/module settings to gate access. + +## Billing Rules +- None in UI components; billing info shown in pages that consume billing store/endpoints. + +## Background Tasks / Schedulers +- None; components react to store state and router changes. + +## Key Design Considerations +- Provider stack covers theme, metrics, toasts, error boundary, routing. +- Guards ensure unauthorized access is blocked at the route level. +- Global utilities avoid repeated boilerplate for scroll/error/loading behaviors. + +## How Developers Should Work With This Module +- Add new protected pages under `ProtectedRoute` and wrap with `ModuleGuard` when module-scoped. +- Use existing toast/error/loading utilities instead of duplicating. +- Keep provider order consistent (ErrorBoundary → Theme/Header/Toast → Router → App). + diff --git a/master-docs/30-frontend/STATE-MANAGEMENT.md b/master-docs/30-frontend/STATE-MANAGEMENT.md index e69de29b..bb465b83 100644 --- a/master-docs/30-frontend/STATE-MANAGEMENT.md +++ b/master-docs/30-frontend/STATE-MANAGEMENT.md @@ -0,0 +1,90 @@ +# State Management + +## Purpose +Describe how the frontend manages global state with Zustand stores, including authentication, site context, billing, and settings. + +## Code Locations (exact paths) +- Auth store: `frontend/src/store/authStore.ts` +- Site store: `frontend/src/store/siteStore.ts` +- Billing store: `frontend/src/store/billingStore.ts` +- Settings store: `frontend/src/store/settingsStore.ts` +- Other UI/state stores: `frontend/src/store/onboardingStore.ts`, `columnVisibilityStore.ts`, `pageSizeStore.ts`, `settingsStore.ts`, etc. + +## High-Level Responsibilities +- Persist authentication state (user, tokens), refresh sessions, and enforce account/plan presence. +- Track the active site (and sector via sector store) across modules, with persistence and change events. +- Fetch and cache billing balances/usage/limits. +- Load and update account/module settings with coalesced fetches and persistence. + +## Detailed Behavior +- `authStore`: + - Persists `user`, `token`, `refreshToken`, `isAuthenticated`, `loading`. + - `login(email,password)`: POST `/api/v1/auth/login/`; stores tokens/user; errors mapped to PLAN_REQUIRED/ACCOUNT_REQUIRED/AUTH_FAILED; resets loading on error. + - `register(data)`: POST `/api/v1/auth/register/`; stores tokens/user; resets loading on error. + - `refreshToken()`: POST `/api/v1/auth/refresh/`; updates access token; attempts `refreshUser`; logs out on failure. + - `refreshUser()`: GET `/api/v1/auth/users/me/`; requires account+plan; updates user or logs out on auth failure; clears loading guard. + - `logout()`: clears all auth fields. +- `siteStore`: + - Persists `activeSite`; stores `loading`/`error`. + - `setActiveSite(site)`: sets/persists site, dispatches `siteChanged` event, triggers sector store load for the site. + - `loadActiveSite()`: fetches sites via `fetchSites`; picks prior site if still active, else first active; persists selection. + - `refreshActiveSite()`: refreshes current site from server; reloads if inactive/missing. +- `billingStore`: + - Holds `balance`, `usageSummary`, `usageLimits`, `loading`/`error`, `lastUpdated`. + - `loadBalance()`: calls `getCreditBalance`; keeps prior balance during retry; sets `lastUpdated`. + - `loadUsageSummary(startDate?, endDate?)`: calls `getCreditUsageSummary`; updates summary. + - `loadUsageLimits()`: calls `getCreditUsageLimits`; tolerates 404 by returning null; records errors otherwise. + - `reset()`: clears billing state. +- `settingsStore`: + - Persists `accountSettings`, `moduleSettings`, `moduleEnableSettings`, `loading`/`error`. + - Account settings: load all, load one, create/update with create/update endpoints; handles typed errors (not found vs validation). + - Module settings: load/update per module; stores per-module map. + - Module enable settings: caches for 60s, coalesces concurrent fetches; `isModuleEnabled` helper. + - `reset()` clears settings state. +- Other stores: + - `onboardingStore`, `columnVisibilityStore`, `pageSizeStore` manage UI/onboarding/table preferences (see respective files for exact fields/actions). + +## Data Structures / Models Involved (no code) +- Store state interfaces defined in each file (auth user shape, site, billing balance/usage, settings maps). + +## Execution Flow +- Components/hooks read/write state via `useXStore` hooks. +- Auth flows call store actions, which perform fetches and update state; ProtectedRoute/ModuleGuard rely on auth and settings/module enable flags. +- Site changes dispatch `siteChanged` for downstream listeners and trigger sector loading. +- Billing/settings loads fetch data on demand and cache/persist where configured. + +## Cross-Module Interactions +- Auth tokens consumed by `fetchAPI` for all API calls. +- Active site influences planner/writer/automation calls via query params/context. +- Module enable settings influence `ModuleGuard`. +- Billing data feeds billing pages and credit displays. + +## State Transitions +- Auth: unauthenticated → login/register → authenticated; refresh updates tokens/user; logout clears. +- Site: load/refresh/set updates `activeSite` and notifies listeners. +- Billing/settings: load/update actions mutate cached maps and loading/error flags. + +## Error Handling +- Auth store resets loading on all error paths; throws descriptive errors. +- Billing store tolerates missing limits (404) and surfaces messages; balance retry preserves prior state. +- Settings store differentiates not-found vs validation errors; stores messages in `error`. + +## Tenancy Rules +- Auth store requires account+plan; site store filters on accessible sites; billing/settings calls use authenticated endpoints tied to the user’s account. + +## Billing Rules +- Enforced via backend; billing store reads balances/usage/limits and does not mutate credits. + +## Background Tasks / Schedulers +- None; all store actions are user-driven fetches with immediate state updates. + +## Key Design Considerations +- Zustand `persist` used for auth/site/settings to survive reloads. +- Guarded loading flags and error handling prevent stuck states. +- Site changes broadcast events for dependent components/stores. + +## How Developers Should Work With This Module +- Use existing store actions for auth/site/billing/settings instead of duplicating fetch logic. +- When adding new shared state, create a dedicated store in `src/store` and persist only needed slices. +- Keep error/reset semantics consistent (always clear loading on failure). + diff --git a/master-docs/30-frontend/accounts/ACCOUNT-PAGE.md b/master-docs/30-frontend/accounts/ACCOUNT-PAGE.md index e69de29b..e80a3a02 100644 --- a/master-docs/30-frontend/accounts/ACCOUNT-PAGE.md +++ b/master-docs/30-frontend/accounts/ACCOUNT-PAGE.md @@ -0,0 +1,56 @@ + # Account Page (Account Settings) + + ## Purpose + Provide a full account-level settings form for name, slug (read-only), billing contact email, billing address, and tax ID. Acts as the frontend surface for `getAccountSettings`/`updateAccountSettings` and reflects saved tenant metadata. + + ## Code Locations (exact paths) + - Page component: `frontend/src/pages/account/AccountSettingsPage.tsx` + - API client: `frontend/src/services/billing.api` (`getAccountSettings`, `updateAccountSettings`, `AccountSettings` type) + - UI primitives: `frontend/src/components/ui/card` + + ## High-Level Responsibilities + - Fetch current account settings on load and bind them into a local form. + - Allow updating account display name, billing contact email, billing address fields, and tax/VAT ID. + - Keep account slug immutable/read-only while exposing it for reference. + - Surface success/error feedback inline. + + ## Detailed Behavior + - On mount: calls `getAccountSettings()`; populates `settings` and `formData` with server values. + - Form fields: `name`, `billing_email`, `billing_address_line1/2`, `billing_city`, `billing_state`, `billing_postal_code`, `billing_country`, `tax_id`; `slug` shown read-only from `settings.slug`. + - Submit: `handleSubmit` -> `updateAccountSettings(formData)`; on success shows success banner then reloads settings; on failure shows error banner. + - Loading states: full-page spinner before data load; button shows saving state. + - Errors surfaced from caught exceptions (`err.message`) with console logging. + + ## Data Structures / Models Involved (no code) + - Account settings DTO returned by `getAccountSettings` with fields listed above plus `slug`. + + ## Execution Flow + - `useEffect` → `loadSettings` → `getAccountSettings` → set local state. + - User edits controlled inputs → `handleSubmit` → `updateAccountSettings` → reload via `loadSettings`. + - All network ops are direct REST calls via `billing.api`. + + ## Cross-Module Interactions + - Shares account identity with billing pages (same `billing.api` source). Updated billing email/address is used by invoicing flows. + + ## State Transitions (if applicable) + - View state: `loading` → loaded; `saving` toggles during submit. Success/error banners reflect last operation outcome. + + ## Error Handling + - Catches fetch/update errors; shows inline red banner; logs to console. No retries. + + ## Tenancy Rules + - Relies on backend to scope `getAccountSettings`/`updateAccountSettings` to the authenticated user’s account; no client-side tenancy logic beyond using the current auth token. + + ## Billing Rules (if applicable) + - Billing contact and address collected here feed invoice generation; no pricing logic on this page. + + ## Background Tasks / Schedulers (if applicable) + - None. + + ## Key Design Considerations + - Slug is immutable to prevent accidental tenant identifier changes. + - Form is fully controlled; reload after save ensures UI mirrors server truth. + + ## How Developers Should Work With This Module + - Add new account-level fields by extending `AccountSettings` in `billing.api`, wiring controlled inputs, and persisting via `updateAccountSettings`. + - Keep slug read-only; any slug mutation must be handled server-side with explicit UX. diff --git a/master-docs/30-frontend/accounts/BILLING-PAGE.md b/master-docs/30-frontend/accounts/BILLING-PAGE.md index e69de29b..560dfaa8 100644 --- a/master-docs/30-frontend/accounts/BILLING-PAGE.md +++ b/master-docs/30-frontend/accounts/BILLING-PAGE.md @@ -0,0 +1,62 @@ + # Billing Page (Plans & Billing Dashboard) + + ## Purpose + Present a consolidated billing dashboard with tabs for credit balance, plans, invoices, payments, and available payment methods. Acts as the UI client for billing endpoints (balance, invoices, payments, credit packages, payment methods, plans, subscriptions). + + ## Code Locations (exact paths) + - Page: `frontend/src/pages/account/AccountBillingPage.tsx` + - Billing API client: `frontend/src/services/billing.api` (`getInvoices`, `getPayments`, `getCreditBalance`, `getCreditPackages`, `getAvailablePaymentMethods`, `downloadInvoicePDF`, `getPlans`, `getSubscriptions`) + - UI: `frontend/src/components/ui/card`, `frontend/src/components/billing/BillingRecentTransactions`, `frontend/src/components/ui/pricing-table/PricingTable` + + ## High-Level Responsibilities + - Fetch and render account-scoped billing data (credits, invoices, payments, packages, payment methods, plans, subscriptions). + - Provide tabbed navigation across overview, plans, invoices, payments, methods. + - Allow invoice PDF download. + + ## Detailed Behavior + - On mount: `loadData()` runs seven parallel calls for balance, invoices, payments, packages, payment methods, plans, subscriptions. Results populate local state; errors render a banner. + - Overview tab: shows four metric cards (balance, monthly allocation, used this month, plan status) from `creditBalance`. + - Plans tab: displays `PricingTable` with static plan catalog (Starter/Growth/Scale) plus dynamic plans from API; shows current subscription chips if present. + - Invoices tab: lists invoices with amount/currency/status/date; supports PDF download via `downloadInvoicePDF(invoiceId)` using blob download flow. + - Payments tab: lists payments with status badges derived from status-to-style map (paid/succeeded/completed/pending/pending_approval/processing/failed/refunded/cancelled/void/uncollectible). + - Methods tab: lists available payment methods from `getAvailablePaymentMethods`. + - “Purchase Credits” button links to `/account/purchase-credits`. + - Loading: full-page spinner until `loadData` completes. + + ## Data Structures / Models Involved (no code) + - `Invoice`, `Payment`, `CreditBalance`, `CreditPackage`, `PaymentMethod`, `Plan`, `Subscription` DTOs from `billing.api`. + + ## Execution Flow + - `useEffect` → `loadData` (Promise.all) → set state. + - Tab click sets `activeTab`; conditional renders per tab. + - Invoice download: fetch blob → create object URL → anchor click → revoke URL. + + ## Cross-Module Interactions + - Reads auth user from `useAuthStore` for contextual display; all billing data is backend-scoped to account. + - Plans tab uses static catalog for marketing copy but still shows live plan/subscription data from API. + + ## State Transitions + - `activeTab` switches UI sections; `loading` gates initial fetch; error banner shows on fetch failure. + - Status badges reflect backend payment/invoice statuses only; no client mutation flows here. + + ## Error Handling + - Catches fetch errors → sets `error` message → red banner. + - Invoice download failure shows `alert`. + + ## Tenancy Rules + - Relies on backend scoping through authenticated requests; no client-side account selection. + + ## Billing Rules (if applicable) + - Read-only view of credits, invoices, payments, and plans; purchases/plan changes are triggered elsewhere (Purchase Credits page or separate plan change flows). + + ## Background Tasks / Schedulers (if applicable) + - None client-side; periodic refresh is not scheduled (one-time load per mount). + + ## Key Design Considerations + - Minimizes throttling risk by batching fetches but does not paginate invoices/payments on the client; relies on API pagination responses. + - Uses static plan catalog for UX but shows actual subscriptions from API to avoid mismatch with assigned plan. + + ## How Developers Should Work With This Module + - Add or reorder tabs by extending the `TabType` union and tab config array. + - When adding new billing surfaces (e.g., disputes/refunds), mirror the pattern: fetch via `billing.api`, guard with `loading/error`, render tab content. + - Keep invoice download flow consistent (blob + object URL) for cross-browser support. diff --git a/master-docs/30-frontend/accounts/PAYMENT-METHODS-PAGE.md b/master-docs/30-frontend/accounts/PAYMENT-METHODS-PAGE.md index e69de29b..4d819f97 100644 --- a/master-docs/30-frontend/accounts/PAYMENT-METHODS-PAGE.md +++ b/master-docs/30-frontend/accounts/PAYMENT-METHODS-PAGE.md @@ -0,0 +1,59 @@ + # Payment Methods Page + + ## Purpose + Display available payment methods for the tenant and support selection for purchases/subscriptions. Methods originate from backend billing configs and are reused across plan changes and credit purchases. + + ## Code Locations (exact paths) + - Consolidated flows: `frontend/src/pages/account/PlansAndBillingPage.tsx` (tab `payment-methods`) + - Purchase credits: `frontend/src/pages/account/PurchaseCreditsPage.tsx` (method selection + manual payment proof) + - API: `frontend/src/services/billing.api` (`getAvailablePaymentMethods`, `createPaymentMethod`, `deletePaymentMethod`, `setDefaultPaymentMethod`) + + ## High-Level Responsibilities + - Fetch account-scoped payment methods (bank transfer, local wallet, manual/other). + - Allow choosing a default method for purchases/subscriptions. + - Provide instructions/details (bank or wallet) for manual/offline flows. + + ## Detailed Behavior + - PlansAndBillingPage: + - On load, calls `getAvailablePaymentMethods()` with other billing calls; filters out disabled methods (`is_enabled === false`). + - Chooses preferred method for selection: bank_transfer, then manual, else default flag, else first item. + - Renders payment-methods tab listing methods, default flag, instructions; supports create/delete/default actions via `createPaymentMethod`, `deletePaymentMethod`, `setDefaultPaymentMethod`. + - Selection state (`selectedPaymentMethod`) is reused when purchasing packages or creating subscriptions. + - PurchaseCreditsPage: + - Fetches methods and auto-selects bank_transfer, then manual, else first. + - For bank/local/manual methods, shows instructions and bank/wallet details and collects `transaction_reference` + `notes` when the payment is manual/offline. + - Stripe/PayPal placeholders show “integration pending”. + + ## Data Structures / Models Involved (no code) + - `PaymentMethod` DTO: `id`, `type` (`bank_transfer`, `local_wallet`, `manual`, `stripe`, `paypal`), `display_name`, `instructions`, optional `bank_details`/`wallet_details`, `is_default`, `is_enabled`. + + ## Execution Flow + - Load methods → set local `paymentMethods` → derive `selectedPaymentMethod`. + - User selects method → stored in component state; used by purchase/subscription handlers. + - Optional CRUD (create/delete/default) triggered in `PlansAndBillingPage` tab using billing API helpers. + + ## Cross-Module Interactions + - Used by credit purchase (`purchaseCreditPackage`) and subscription updates (`createSubscription`, `cancelSubscription`) within `PlansAndBillingPage`. + + ## State Transitions + - Local selection state only; backend default flag can be updated via `setDefaultPaymentMethod`. + + ## Error Handling + - Errors surface via toast/banner in `PlansAndBillingPage`; inline error message in `PurchaseCreditsPage` for missing selections or failed calls. + + ## Tenancy Rules + - All calls are account-scoped by backend auth; no client multi-tenant switching. + + ## Billing Rules (if applicable) + - Selected method is passed to credit purchase and subscription creation; manual methods require proof submission (transaction reference) and remain pending approval server-side. + + ## Background Tasks / Schedulers (if applicable) + - None on client. + + ## Key Design Considerations + - Prefers offline-safe methods (bank/manual) to avoid blocking when card gateways are unavailable. + - Keeps selection sticky within the session to reduce user friction across tabs. + + ## How Developers Should Work With This Module + - To add a new gateway type, extend `PaymentMethod` type, update selection logic, and render gateway-specific instructions or redirection flows. + - Keep manual/offline instructions editable via backend configs; surface them read-only in the UI. diff --git a/master-docs/30-frontend/accounts/SUBSCRIPTION-PAGE.md b/master-docs/30-frontend/accounts/SUBSCRIPTION-PAGE.md index e69de29b..c5ff3cd9 100644 --- a/master-docs/30-frontend/accounts/SUBSCRIPTION-PAGE.md +++ b/master-docs/30-frontend/accounts/SUBSCRIPTION-PAGE.md @@ -0,0 +1,66 @@ + # Subscription Page (Plans & Upgrades) + + ## Purpose + Let tenants view current subscription, browse available plans, start or cancel subscriptions, and align payment method selection with plan changes. Uses the same billing data as the Billing dashboard but focuses on plan lifecycle. + + ## Code Locations (exact paths) + - Page: `frontend/src/pages/account/PlansAndBillingPage.tsx` (tabs `plan` and `upgrade`) + - API: `frontend/src/services/billing.api` (`getPlans`, `getSubscriptions`, `createSubscription`, `cancelSubscription`) + - Payment selection (shared): same page via `getAvailablePaymentMethods` + + ## High-Level Responsibilities + - Fetch active plans and the tenant’s current subscription. + - Enable plan change (create/update subscription) gated by payment method selection. + - Allow cancellation of the active subscription. + + ## Detailed Behavior + - Initial load (`loadData`): + - Fetches credit balance, packages, invoices, payments, payment methods, plans, subscriptions (with retry/backoff for 429). + - Filters plans to active ones; excludes Enterprise for non `aws-admin` accounts but re-injects the account’s assigned plan so UI always reflects truth. + - Ensures there is at least one subscription entry; if none returned, synthesizes one from `user.account.plan`. + - Plan selection: + - Requires a chosen payment method if any exist. + - `handleSelectPlan(planId)` → `createSubscription({ plan_id, payment_method })` → reload data. + - Cancellation: + - `handleCancelSubscription` calls `cancelSubscription(currentSubscription.id)` and refreshes state. + - Tabs: + - `plan` shows current subscription summary. + - `upgrade` lists available plans (filtered) with select buttons; highlights defaults via UI styling. + + ## Data Structures / Models Involved (no code) + - `Plan`: name, slug, price, billing cycle, features, `is_active`. + - `Subscription`: `id`, `plan`, `status`. + + ## Execution Flow + - `useEffect` → `loadData` (controlled sequence + throttling gaps) → set plans/subscriptions/paymentMethods. + - Select plan → API call → toast → reload. + - Cancel → API call → toast → reload. + + ## Cross-Module Interactions + - Shares payment method selection with credit purchase; uses `useAuthStore` for account plan fallbacks. + + ## State Transitions + - `activeTab` controls view; `planLoadingId`/`purchaseLoadingId` track in-flight plan or package actions. + - Subscription status rendered from backend; cancellation triggers status change server-side. + + ## Error Handling + - Central `handleBillingError` surfaces message + toast. + - Throttle (429) handled with delayed retry for subscriptions; errors suppressed if retry fails to keep page usable. + + ## Tenancy Rules + - All subscription APIs are account-scoped by the backend; client does not switch tenants. + + ## Billing Rules (if applicable) + - Requires a payment method before subscription create/update. + - Enterprise plans hidden for non-admin tenants to prevent accidental selection. + + ## Background Tasks / Schedulers (if applicable) + - None client-side; no polling beyond initial load. + + ## Key Design Considerations + - Guardrails to avoid showing empty state when account already has a plan (synthesizes subscription from account plan). + - Prefers bank/manual methods when selecting plan for safer checkout in absence of Stripe/PayPal integrations. + + ## How Developers Should Work With This Module + - To expose new plan attributes, extend plan cards in the upgrade tab and ensure `Plan` type carries those fields. + - Keep enterprise-plan visibility rules aligned with backend authorization; avoid client-side-only blocking for sensitive plans. diff --git a/master-docs/30-frontend/automation/AUTOMATION-COMPONENTS.md b/master-docs/30-frontend/automation/AUTOMATION-COMPONENTS.md index e69de29b..b3957dd5 100644 --- a/master-docs/30-frontend/automation/AUTOMATION-COMPONENTS.md +++ b/master-docs/30-frontend/automation/AUTOMATION-COMPONENTS.md @@ -0,0 +1,70 @@ + # Automation Components + + ## Purpose + Describe the reusable UI components that compose the automation dashboard: stage cards, current processing card, run history, activity log, and configuration modal. These components visualize pipeline state, history, and settings sourced from automation services. + + ## Code Locations (exact paths) + - Stage cards & layout: `frontend/src/pages/Automation/AutomationPage.tsx` (renders stage cards from `STAGE_CONFIG`) + - Current run card: `frontend/src/components/Automation/CurrentProcessingCard.tsx` + - Activity log: `frontend/src/components/Automation/ActivityLog.tsx` + - Run history: `frontend/src/components/Automation/RunHistory.tsx` + - Config modal: `frontend/src/components/Automation/ConfigModal.tsx` + - Shared UI: `frontend/src/components/common/{ComponentCard,PageMeta,DebugSiteSelector}`, `frontend/src/components/dashboard/EnhancedMetricCard` + + ## High-Level Responsibilities + - Stage cards: show each of the 7 pipeline stages with icon/color/status derived from pipeline overview. + - CurrentProcessingCard: surface active run details, stage name, status, percent, timestamps, and controls (Pause/Resume). + - ActivityLog: list recent automation events (from run log feed). + - RunHistory: show prior runs with status and timestamps. + - ConfigModal: edit and persist automation configuration per site. + + ## Detailed Behavior + - Stage Cards: + - Built from `STAGE_CONFIG` array (keywords→clusters, clusters→ideas, ideas→tasks, tasks→content, content→image prompts, image prompts→images, manual review). + - Status/progress comes from `pipelineOverview.stages` provided by `automationService.getPipelineOverview`. + - CurrentProcessingCard: + - Receives `currentRun` and shows status; displays pause/resume buttons wired to page handlers that call `automationService.pause/resume`. + - Hidden when no current run; toggled by `showProcessingCard`. + - RunHistory: + - Takes run list (from `automationService.getCurrentRun` payload history) and renders chronological entries. + - ActivityLog: + - Displays textual log entries for the active run; consumes run log data supplied by the page. + - ConfigModal: + - Opens from page button; on save calls `automationService.updateConfig(activeSite.id, newConfig)`; merges into local config and refreshes pipeline/metrics. + + ## Data Structures / Models Involved (no code) + - `AutomationRun` (id, status, stage, progress, started_at/ended_at). + - `PipelineStage` array with stage identifiers, names, progress. + - `AutomationConfig` fields shown in modal (intervals/gates/etc., defined server-side). + + ## Execution Flow + - Page loads run + pipeline → passes data into stage cards, processing card, history, activity log. + - User opens ConfigModal → submit triggers updateConfig → page reloads pipeline/metrics/run to reflect new settings. + - Pause/Resume buttons on CurrentProcessingCard call page handlers, which in turn call automationService. + + ## Cross-Module Interactions + - Components depend on site context from `useSiteStore` and data from automationService; no direct planner/writer calls (metrics happen in page). + + ## State Transitions + - Components are pure renderers; state (visibility, selected config) managed by `AutomationPage`. + + ## Error Handling + - Errors in save/pause/resume are surfaced by the page via toasts; components render based on provided props. + + ## Tenancy Rules + - All data passed in is already scoped to `activeSite`; components do not alter scoping. + + ## Billing Rules (if applicable) + - None inside components; Run Now credit gating handled at page level. + + ## Background Tasks / Schedulers (if applicable) + - None; updates driven by page polling interval. + + ## Key Design Considerations + - Separation of concerns: components stay presentational; network calls remain in page. + - Stage cards use color/icon metadata for fast visual scanning of pipeline status. + + ## How Developers Should Work With This Module + - Add new stages by extending `STAGE_CONFIG` and ensuring pipeline overview includes the new stage id/status. + - Extend ConfigModal fields in sync with backend `AutomationConfig`; persist via automationService. + - Keep CurrentProcessingCard controls minimal; any new action should call automationService and refresh run/pipeline afterward. diff --git a/master-docs/30-frontend/automation/AUTOMATION-PAGE.md b/master-docs/30-frontend/automation/AUTOMATION-PAGE.md index e69de29b..d5771a39 100644 --- a/master-docs/30-frontend/automation/AUTOMATION-PAGE.md +++ b/master-docs/30-frontend/automation/AUTOMATION-PAGE.md @@ -0,0 +1,72 @@ + # Automation Page (AI Automation Pipeline Dashboard) + + ## Purpose + Provide a site-scoped dashboard to configure, run, pause, resume, and monitor the 7-stage automation pipeline. Surfaces pipeline overview, current run status, metrics, history, configuration modal, and credit sufficiency checks. + + ## Code Locations (exact paths) + - Primary page: `frontend/src/pages/Automation/AutomationPage.tsx` + - Service: `frontend/src/services/automationService` (config, current run, estimate, pipeline overview, runNow, pause, resume, publishWithoutReview) + - Metrics sources: `frontend/src/services/api` (`fetchKeywords`, `fetchClusters`, `fetchContentIdeas`, `fetchTasks`, `fetchContent`, `fetchContentImages`) + - UI components: `frontend/src/components/Automation/{ActivityLog,ConfigModal,RunHistory,CurrentProcessingCard}`, `frontend/src/components/dashboard/EnhancedMetricCard`, `frontend/src/components/common/ComponentCard`, `frontend/src/components/common/PageMeta`, `frontend/src/components/common/DebugSiteSelector` + + ## High-Level Responsibilities + - Load automation config, current run, credit estimate, and pipeline overview for the active site. + - Poll current run and pipeline status while running/paused; refresh metrics regularly during runs. + - Provide controls: Run Now, Pause, Resume, Save Config, Publish Without Review. + - Show per-stage cards, current processing card, run history, and activity log. + + ## Detailed Behavior + - Site binding: requires `useSiteStore.activeSite`; without it, page shows a “select a site” message. + - Initial load (`loadData`): parallel calls to `getConfig`, `getCurrentRun`, `estimate`, `getPipelineOverview` plus low-level metrics (keywords/clusters/ideas/tasks/content/images counts) per site_id. + - Polling: 5s interval; if run status is `running`/`paused`, refresh run, pipeline, and metrics; otherwise refresh pipeline only. + - Metrics: counts for keywords (total/new/mapped), clusters (total/new/mapped), ideas (total/new/queued/completed), tasks (total), content (total/draft/review/published), images (total/pending). + - Stage cards: derived from `STAGE_CONFIG` array representing 7 pipeline stages including manual review gate. + - Actions: + - Run Now: checks credit sufficiency from `estimate`; blocks if insufficient. + - Pause/Resume: call automationService with site + run_id, then refresh run/pipeline/metrics. + - Save Config: persists partial config, updates local state, reloads pipeline/metrics/run. + - Publish Without Review: calls `publishWithoutReview` with confirmation prompt, then reloads. + - UI states: `loading` spinner until initial fetch; `showProcessingCard` toggled on when a run exists. + + ## Data Structures / Models Involved (no code) + - `AutomationConfig`: site-level automation settings (intervals, gates, etc.). + - `AutomationRun`: run_id, status (`running`, `paused`, etc.), stage info. + - `PipelineStage`: stage list with status/progress. + - Metrics DTOs from planner/writer/image endpoints (`count` fields only). + + ## Execution Flow + - `useEffect` with site dependency → `loadData`. + - Interval polling while active runs → `loadCurrentRun`, `loadPipelineOverview`, `loadMetrics`. + - User actions dispatch to automationService; toasts for success/error; follow-up refreshes. + + ## Cross-Module Interactions + - Pulls planner/writer/image counts to present authoritative pipeline context. + - Uses `useSiteStore` for tenant/site scoping; credit checks rely on billing estimate API. + + ## State Transitions + - `currentRun.status` drives polling and control availability. + - `showProcessingCard` turns on when a run exists. + - `estimate.sufficient` gates Run Now action. + + ## Error Handling + - Toast-based error reporting; fetch failures log to console but keep page usable. + - Metrics fetch wrapped in try/catch; failure degrades to missing metrics without blocking the rest. + + ## Tenancy Rules + - All automation service calls require `activeSite.id`; backend enforces account/site scoping. No client override for other tenants. + + ## Billing Rules (if applicable) + - Run Now is blocked when `estimate.sufficient` is false; shows required vs current credits. + + ## Background Tasks / Schedulers (if applicable) + - Client-side polling (5s) during active runs; no background scheduling beyond that. + + ## Key Design Considerations + - Polling keeps UI aligned with long-running Celery pipeline states. + - Credit gate prevents user-initiated runs that would immediately fail server-side. + - Metrics are read-only mirrors of backend aggregates to align UI with planner/writer state. + + ## How Developers Should Work With This Module + - When adding new stages, extend `STAGE_CONFIG` and ensure backend pipeline overview includes them. + - Keep polling interval modest; if adding heavier metrics, consider staggering fetches to avoid rate limits. + - Wire new config fields through `AutomationConfig` type, `ConfigModal`, and `updateConfig` payload. diff --git a/master-docs/30-frontend/sites/SITE-DASHBOARD.md b/master-docs/30-frontend/sites/SITE-DASHBOARD.md index e69de29b..b41debc8 100644 --- a/master-docs/30-frontend/sites/SITE-DASHBOARD.md +++ b/master-docs/30-frontend/sites/SITE-DASHBOARD.md @@ -0,0 +1,59 @@ + # Site Dashboard + + ## Purpose + Provide a per-site overview showing basic site info, stats (pages/content/integrations/deployments), and quick actions for navigation. Serves as the landing view for a specific site id. + + ## Code Locations (exact paths) + - Page: `frontend/src/pages/Sites/Dashboard.tsx` + - API: `frontend/src/services/api` (`fetchAPI` for `/v1/auth/sites/{id}/`; `fetchSiteStats` placeholder) + - UI: `frontend/src/components/common/{PageMeta,PageHeader,ComponentCard}`, `frontend/src/components/dashboard/EnhancedMetricCard`, `frontend/src/components/sites/SiteProgressWidget`, `frontend/src/components/ui/{card,button}` + + ## High-Level Responsibilities + - Load a single site’s record (name/slug/type/hosting/domain/status) and display a card-based dashboard. + - Show metric cards for pages, integrations, deployments, and content counts (placeholder stats until backend endpoint exists). + - Provide quick actions to manage pages, sync, publish, or open settings. + + ## Detailed Behavior + - Routing: expects `:id` param; uses `useParams` and `fetchAPI(/v1/auth/sites/{id}/)` to load site. + - Stats: `fetchSiteStats` currently returns placeholder structure (zeros); TODO noted to back with a real endpoint. + - If site not found: shows “Site not found” with Back to Sites button. + - Metric cards (EnhancedMetricCard) link to pages list, published filter, draft filter, integrations tab, deployments preview, and content list. + - Quick actions: buttons for Manage Pages, View Integrations, View Content, Deployments (navigations). + - Header shows site name, slug, type, hosting, optional domain, and Settings button linking to `/sites/{id}/settings`. + - Loading state: spinner until data fetch completes. + + ## Data Structures / Models Involved (no code) + - Site DTO: `id`, `name`, `slug`, `site_type`, `hosting_type`, `status`, `is_active`, `domain`. + - Stats DTO (placeholder): counts for pages/content/integrations/deployments/views/visitors. + + ## Execution Flow + - `useEffect` on `siteId` → `loadSiteData` (Promise.all for site + stats) → set `site` + `stats`. + - Renders cards and actions with derived links. + + ## Cross-Module Interactions + - Integrations quick action routes to Site Settings integrations tab. + - Content and pages links route to module pages that rely on planner/writer data scoped by site. + + ## State Transitions + - `loading` gates initial fetch; absent site shows fallback card. + + ## Error Handling + - Errors during fetch show toast (`Failed to load site data`) and stop loading spinner. + + ## Tenancy Rules + - Site fetch is account-scoped server-side; client only supplies the site id from the router. + + ## Billing Rules (if applicable) + - None on this page. + + ## Background Tasks / Schedulers (if applicable) + - None; no polling. + + ## Key Design Considerations + - Uses placeholder stats until backend supplies real site analytics; structure prepared for drop-in replacement. + - Quick navigation shortcuts reduce hops for site-level operations. + + ## How Developers Should Work With This Module + - Replace `fetchSiteStats` with a real endpoint and map returned fields into `statCards`. + - Add new quick actions by extending the ComponentCard grid with navigation handlers. + - Keep links consistent with router paths in `App.tsx` to avoid broken navigation. diff --git a/master-docs/30-frontend/sites/SITE-FLOW.md b/master-docs/30-frontend/sites/SITE-FLOW.md index e69de29b..47af4097 100644 --- a/master-docs/30-frontend/sites/SITE-FLOW.md +++ b/master-docs/30-frontend/sites/SITE-FLOW.md @@ -0,0 +1,61 @@ + # Site Flow (Listing & Management) + + ## Purpose + Provide the tenant-facing list and management surface for all sites, including creation, navigation to settings/integrations, viewing, editing, and deletion. This is the primary entry point before drilling into site dashboard or settings. + + ## Code Locations (exact paths) + - Page: `frontend/src/pages/Sites/Manage.tsx` (titled “Site Management”) + - API: `frontend/src/services/api` (`fetchAPI` for sites collection and integration checks) + - UI: `frontend/src/components/ui/{card,button,badge}`, `frontend/src/components/sites/SiteTypeBadge`, `frontend/src/components/ui/badge/Badge` + + ## High-Level Responsibilities + - List all account sites with status, type, hosting, and integration hints. + - Offer actions per site: view, edit, settings, delete, manage WordPress integration. + - Provide entry point for creating new sites (routes to builder). + + ## Detailed Behavior + - On mount: `loadSites()` → GET `/v1/auth/sites/`; if hosting_type is `wordpress`, performs an additional call to `/v1/integration/integrations/?site={id}&platform=wordpress` to flag `has_wordpress_integration`. + - Renders grid of cards: + - Shows name, slug, active badge, hosting badge, site type, optional WP integration button (“Connect WordPress” / “Manage Integration”). + - Footer actions: View (`/sites/{id}`), Edit (`/sites/{id}/edit`), Settings (`/sites/{id}/settings`), Delete (DELETE `/v1/auth/sites/{id}/` after confirm). + - Empty state: if no sites, shows CTA to create first site (`/sites/builder`). + - Create button in header also routes to `/sites/builder`. + + ## Data Structures / Models Involved (no code) + - Site list DTO: id, name, slug, site_type, hosting_type, status, is_active, page_count, integration_count. + - Integration probe response: `results` length indicates WP connection presence. + + ## Execution Flow + - `useEffect` → `loadSites` → set `sites`. + - Per-site integration check executed inside `Promise.all` when hosting_type is `wordpress`. + - Action buttons call navigation or delete; delete triggers reload. + + ## Cross-Module Interactions + - Integrations tab reachable via settings link; integration existence probed via integration module API. + - Builder route (`/sites/builder`) ties into site creation flow (outside this component). + + ## State Transitions + - `loading` gates initial render; after deletion, reload resets state. + - WP integration flag stored per site for button labeling. + + ## Error Handling + - Load errors show toast (`Failed to load sites`); delete errors show toast. + - Integration probe failures fall back to `has_wordpress_integration: false` without blocking list. + + ## Tenancy Rules + - Backend scopes `/v1/auth/sites/` to current account; page does not switch tenants. + + ## Billing Rules (if applicable) + - None on this page; plan limits enforced server-side when creating/managing sites. + + ## Background Tasks / Schedulers (if applicable) + - None; no polling. + + ## Key Design Considerations + - Integration probe is conditional to reduce extra calls for non-WordPress hosting. + - Delete confirmation prevents accidental site removal. + + ## How Developers Should Work With This Module + - When adding new hosting types or platforms, adjust integration probe logic and `SiteTypeBadge`. + - Keep routes aligned with `App.tsx`; ensure permissions enforced server-side for edit/delete. + - If adding pagination/filtering, wire through `fetchAPI` params and preserve integration detection per page. diff --git a/master-docs/30-frontend/sites/SITE-SETTINGS-PAGE.md b/master-docs/30-frontend/sites/SITE-SETTINGS-PAGE.md index e69de29b..a2d76a69 100644 --- a/master-docs/30-frontend/sites/SITE-SETTINGS-PAGE.md +++ b/master-docs/30-frontend/sites/SITE-SETTINGS-PAGE.md @@ -0,0 +1,76 @@ + # Site Settings Page + + ## Purpose + Advanced site management surface to edit site metadata, SEO fields, industry/sector selection, activation, and WordPress integration. Supports site switching, tabbed sections, and content-type syncing for WP. + + ## Code Locations (exact paths) + - Page: `frontend/src/pages/Sites/Settings.tsx` + - APIs: `frontend/src/services/api` (`fetchAPI`, `fetchSites`, `fetchIndustries`, dynamic imports for `fetchAccountSetting`, `fetchSiteSectors`), `frontend/src/services/integration.api` (`integrationApi.getWordPressIntegration`) + - UI: `frontend/src/components/common/{PageMeta,PageHeader}`, `frontend/src/components/ui/{card,button,badge}`, `frontend/src/components/form/{Label,SelectDropdown,Checkbox,TextArea}`, `frontend/src/components/ui/dropdown/{Dropdown,DropdownItem}`, `frontend/src/components/sites/WordPressIntegrationForm` + + ## High-Level Responsibilities + - Load a specific site by id and allow editing of general fields (name, slug, URL, type, hosting, active flag). + - Manage SEO metadata (meta/OG/schema fields) per site. + - Select industry and sectors for the site; load available industries and current sector assignments. + - Manage WordPress integration (fetch existing integration, display form, load content types when tab active). + - Provide site switcher dropdown for quickly navigating between active sites. + + ## Detailed Behavior + - Routing: uses `:id` param; optional `tab` query (`general`, `integrations`, `content-types`) controls active tab. + - Initial loads: + - `loadSite` → GET `/v1/auth/sites/{id}/`; seeds formData with site fields plus SEO metadata (from `seo_metadata` or `metadata`). + - `loadIntegrations` → `integrationApi.getWordPressIntegration(siteId)`; ignores missing integration. + - `loadIndustries` → fetches industries (and sectors) for selection. + - `loadSites` → fetches active sites for selector; `loadUserPreferences` → account setting `user_preferences` (industry/sector defaults). + - When site + industries loaded → `loadSiteSectors` to map sector slugs to selections. + - Tabs: + - General: edits site core fields and SEO fields; save handler persists via `fetchAPI` PATCH/PUT (not fully shown in snippet but formData bound for submission). + - Integrations: shows WordPress integration form, uses `integrationApi` to fetch/update connection; `integrationLoading` guards state. + - Content-types: when tab active and WP integration exists, calls `loadContentTypes` to fetch available structures from WP. + - Industry/Sector selection: + - Industries from backend; sectors filtered by selected industry; selected sector slugs stored in state. + - `loadSiteSectors` pulls current site sectors; tries to infer industry from sectors if not set. + - Site switcher: dropdown using `fetchSites` results; navigates to new site settings route preserving tab query. + + ## Data Structures / Models Involved (no code) + - Site DTO: id, name, slug, domain/url, site_type, hosting_type, is_active, industry_slug, seo_metadata fields. + - Industry DTO: slug, name, sectors[] (each sector has slug/name). + - WordPress integration DTO (`SiteIntegration`): platform, credentials/config, status (from `integration.api`). + + ## Execution Flow + - `useEffect` on siteId → clear integration/content types → load site, integration, industries. + - `useEffect` on activeTab & integration → load content types when needed. + - `useEffect` on site+industries → load site sectors. + - Dropdown selection → navigate to other site settings URL; closes selector. + + ## Cross-Module Interactions + - Integration tab depends on integration service; content-types fetch hits integration endpoints. + - Sector selection aligns with planner/writer scoping by site/sector. + + ## State Transitions + - `activeTab` from query or state; `loading`, `saving`, `integrationLoading`, `sitesLoading` gate respective sections. + - Form data is controlled; when site changes, state resets. + + ## Error Handling + - Toast errors for site load failure and integration load; silent handling for user preference load failures. + - Integration fetch failures simply show no integration; no hard failure. + + ## Tenancy Rules + - All requests are account-scoped server-side; page uses site id route but does not allow cross-tenant access. + - Site selector filters to `is_active` sites from the same account. + + ## Billing Rules (if applicable) + - None directly; industry/sector constraints enforced server-side (plan limits) not in this component. + + ## Background Tasks / Schedulers (if applicable) + - None; no polling. + + ## Key Design Considerations + - Separates tabs to avoid heavy content-type loads unless needed. + - Avoids assuming integration exists; handles absent integration gracefully. + - Keeps SEO fields in formData for a single-save payload. + + ## How Developers Should Work With This Module + - When adding new site fields, extend `formData` initialization from `loadSite` and include inputs plus save payload. + - Back the placeholder stats/content-types with real endpoints, ensuring they key off `siteId`. + - Preserve query-param tab handling when adding new tabs. diff --git a/master-docs/30-frontend/writer/CONTENT-EDITOR.md b/master-docs/30-frontend/writer/CONTENT-EDITOR.md index e69de29b..c3dc4d71 100644 --- a/master-docs/30-frontend/writer/CONTENT-EDITOR.md +++ b/master-docs/30-frontend/writer/CONTENT-EDITOR.md @@ -0,0 +1,53 @@ + # Content Editor Page + + ## Purpose + Display a single content record with full details for review/read-only inspection. Acts as the per-record viewer for content generated or managed by the Writer module. + + ## Code Locations (exact paths) + - Page: `frontend/src/pages/Writer/ContentView.tsx` + - Template: `frontend/src/templates/ContentViewTemplate` (renders the actual layout/content fields) + - API: `frontend/src/services/api` (`fetchContentById`) + + ## High-Level Responsibilities + - Fetch a specific content item by id from route param and render it via `ContentViewTemplate`. + - Validate id parameter, handle not-found/error states, and provide back navigation to content list. + + ## Detailed Behavior + - Route: `/writer/content/:id`. + - On mount: validates `id` is numeric; on invalid, shows toast and redirects to `/writer/content`. + - Fetches content via `fetchContentById(contentId)`; sets `content` state and clears `loading`. + - Errors: shows toast (`Failed to load content`) and leaves `content` null. + - Back action: `onBack` navigates to `/writer/content`. + - Page metadata: sets document title/description via `PageMeta`. + + ## Data Structures / Models Involved (no code) + - Content DTO from writer API (includes title, body/html, status, external_url, etc.; structure defined server-side). + + ## Execution Flow + - `useEffect` → validate id → fetch content → update state → render template. + - Template receives `{ content, loading, onBack }`. + + ## Cross-Module Interactions + - Navigation back to writer drafts list; no direct cross-module calls. + + ## State Transitions + - `loading` toggles during fetch; `content` set on success; invalid id triggers navigation away. + + ## Error Handling + - Toast errors for missing/invalid id or fetch failure; console logs errors. + + ## Tenancy Rules + - Backend enforces account/site/sector scoping; client only supplies id from route. + + ## Billing Rules (if applicable) + - None within this view; generation/updates handled elsewhere. + + ## Background Tasks / Schedulers (if applicable) + - None. + + ## Key Design Considerations + - Strict id validation avoids bad requests. + - Keeps view read-only; editing handled elsewhere (not in this component). + + ## How Developers Should Work With This Module + - If adding inline editing, extend `ContentViewTemplate` and add PATCH/PUT calls; keep id validation and error handling intact. diff --git a/master-docs/30-frontend/writer/IMAGE-EDITOR.md b/master-docs/30-frontend/writer/IMAGE-EDITOR.md index e69de29b..651f01ee 100644 --- a/master-docs/30-frontend/writer/IMAGE-EDITOR.md +++ b/master-docs/30-frontend/writer/IMAGE-EDITOR.md @@ -0,0 +1,70 @@ + # Image Editor Page (Images List & Generation) + + ## Purpose + Manage generated images linked to writer content: list images grouped by content, filter/search, trigger generation, update statuses, and download/view images. Uses table template patterns similar to content/tasks. + + ## Code Locations (exact paths) + - Page: `frontend/src/pages/Writer/Images.tsx` + - API: `frontend/src/services/api` (`fetchContentImages`, `fetchImageGenerationSettings`, `generateImages`, `bulkUpdateImagesStatus`, `deleteContent`, `bulkDeleteContent`) + - Config: `frontend/src/config/pages/images.config` (columns/filters) + - UI components: `frontend/src/components/common/ImageQueueModal`, `frontend/src/components/common/SingleRecordStatusUpdateModal`, `frontend/src/components/common/PageHeader`, `frontend/src/components/navigation/ModuleNavigationTabs`, `frontend/src/components/ui/modal` + - Hooks: `frontend/src/hooks/useResourceDebug` (AI logs toggle), `frontend/src/hooks/useProgressModal` (via modal usage pattern) + + ## High-Level Responsibilities + - Fetch and render content-image groups with client-side filtering/search/sorting/pagination. + - Trigger image generation for selected content with queue modal and provider/model selection. + - Update image status in bulk and delete images/content. + - Provide AI function log visibility when resource debug is enabled. + + ## Detailed Behavior + - Data load: `loadImages` calls `fetchContentImages({})`, applies client-side search (`content_title`) and status filter (`overall_status`), sorts (default `content_title`), paginates client-side (page size 10), and adds `id` field mirroring `content_id` for selection. + - Filters: search text; status dropdown; sort controls; pagination tracked in local state. + - Actions: + - Generate images: opens `ImageQueueModal`, builds queue items, calls `generateImages` with provider/model; tracks taskId/model/provider state; shows AI logs when resource debug enabled. + - Bulk status update: `bulkUpdateImagesStatus` on selected ids. + - Delete (single/bulk): uses `deleteContent`/`bulkDeleteContent`. + - Download/view: handled by row actions in config (template-driven). + - Navigation: writer tabs rendered via `ModuleNavigationTabs` (Queue/Drafts/Images/Review/Published). + - Resource debug: AI logs captured only when `useResourceDebug` returns true; `addAiLog` appends logs for visibility. + - Loading UX: `loading` + `showContent` gating; debounced search resets page as needed. + + ## Data Structures / Models Involved (no code) + - `ContentImagesGroup`: content_id, content_title, overall_status, images[]. + - `ContentImage`: individual image entries with URL/status/prompt (from API). + - Generation settings: provider/model options from `fetchImageGenerationSettings`. + + ## Execution Flow + - `useEffect` → `loadImages`. + - Filters/sort/page changes → recompute client-side subsets. + - Generate action → open modal → call `generateImages` → optionally log steps → reload images. + - Status update/delete actions → call API → reload. + + ## Cross-Module Interactions + - Tied to writer content records; delete actions use writer content endpoints. + - AI generation leverages shared API endpoints that consume credits server-side. + + ## State Transitions + - `loading`/`showContent` manage render timing; modal open states for queue/status/image viewer. + - `aiLogs` maintained only when debug is enabled. + + ## Error Handling + - Toast errors for load/generate/update/delete; generation errors also recorded in AI logs when enabled. + - Debounced search handles errors gracefully by keeping prior data until reload. + + ## Tenancy Rules + - Backend enforces account/site/sector; client passes no explicit tenant fields beyond any default filters in API layer. + + ## Billing Rules (if applicable) + - Image generation consumes credits on backend; page performs no credit gating. + + ## Background Tasks / Schedulers (if applicable) + - None; generation is user-triggered, and polling is not used here. + + ## Key Design Considerations + - Client-side pagination used because API returns grouped images; keeps UI responsive without extra endpoints. + - Resource debug toggle avoids unnecessary log storage unless explicitly enabled. + + ## How Developers Should Work With This Module + - If server adds server-side pagination/filtering, remove client-side slicing and pass filters to API. + - Extend row actions by updating `images.config` with handlers wired to new behaviors. + - Keep generation flow in sync with backend provider/model options; surface credit estimates if backend exposes them. diff --git a/master-docs/30-frontend/writer/WRITER-MAIN-PAGE.md b/master-docs/30-frontend/writer/WRITER-MAIN-PAGE.md index e69de29b..54bbd328 100644 --- a/master-docs/30-frontend/writer/WRITER-MAIN-PAGE.md +++ b/master-docs/30-frontend/writer/WRITER-MAIN-PAGE.md @@ -0,0 +1,53 @@ + # Writer Main Page (Queue/Drafts/Images Navigation) + + ## Purpose + Serve as the entry surface for writer workflows, routing users to task queue, drafts, images, review, and published content views. It relies on shared table templates and writer navigation tabs defined in the content/tasks/images pages. + + ## Code Locations (exact paths) + - Key pages under Writer: + - Tasks (queue): `frontend/src/pages/Writer/Tasks.tsx` + - Content drafts/list: `frontend/src/pages/Writer/Content.tsx` + - Images: `frontend/src/pages/Writer/Images.tsx` + - Content view: `frontend/src/pages/Writer/ContentView.tsx` + - Navigation tabs defined inside Tasks/Content/Images pages (`writerTabs`). + + ## High-Level Responsibilities + - Present writer-specific navigation and headers. + - Delegate to module pages that implement task creation, AI generation, content listing, and image management. + + ## Detailed Behavior + - Writer pages use shared navigation tabs (`writerTabs`: Queue, Drafts, Images, Review, Published) rendered via `ModuleNavigationTabs`. + - Each page sets a header (`PageHeader`) with badge icon and binds to the tab component for consistent navigation. + - State (filters, pagination, selections) is managed within each page; there is no global writer-state container beyond shared stores (`useSectorStore`, `usePageSizeStore`). + + ## Data Structures / Models Involved (no code) + - Task, Content, ContentImage DTOs from `frontend/src/services/api` (writer endpoints). + + ## Execution Flow + - User enters writer area via route (e.g., `/writer/tasks` or `/writer/content`). + - Navigation tabs switch routes; each route mounts its page and fetches data (tasks/content/images). + + ## Cross-Module Interactions + - Sector/site scoping via `useSectorStore` and backend query params in API calls. + - Optimization and image generation actions route into optimizer or image generation APIs. + + ## State Transitions + - Per-page loading/filters; navigation changes unmount current page and mount target page. + + ## Error Handling + - Each page uses toasts for API errors; no shared error surface beyond per-page banners. + + ## Tenancy Rules + - Backend filters by account/site/sector; pages pass sector context via filters or rely on server defaults. + + ## Billing Rules (if applicable) + - None on navigation; individual actions (AI generation) consume credits on backend. + + ## Background Tasks / Schedulers (if applicable) + - None at the navigation level. + + ## Key Design Considerations + - Tabs keep writer UX consistent; each page owns its data loading to avoid cross-coupling. + + ## How Developers Should Work With This Module + - When adding a new writer view (e.g., “Outlines”), add a tab entry and a route in `App.tsx`, and implement the page with the same header/tab pattern. diff --git a/master-docs/40-product/ACCOUNT-LIFECYCLE.md b/master-docs/40-product/ACCOUNT-LIFECYCLE.md index e69de29b..1174a6ef 100644 --- a/master-docs/40-product/ACCOUNT-LIFECYCLE.md +++ b/master-docs/40-product/ACCOUNT-LIFECYCLE.md @@ -0,0 +1,67 @@ + # Account Lifecycle + + ## Purpose + Explain how accounts are created, activated, associated with plans/subscriptions, how users access them, and how account state affects the platform. + + ## Code Locations (exact paths) + - Models: `backend/igny8_core/auth/models.py` (`Account`, `User`, `Plan`, `Subscription`, `Site`, `SiteUserAccess`) + - Auth middleware: `backend/igny8_core/auth/middleware.py` (sets `request.account`, validates plan status) + - Permissions: `backend/igny8_core/auth/permissions.py` + - Account settings API: `backend/igny8_core/modules/system/views.py` (account settings), `auth/views.py` (team) + - Frontend account settings: `frontend/src/pages/account/AccountSettingsPage.tsx` + - Frontend auth store: `frontend/src/store/authStore.ts` + + ## High-Level Responsibilities + - Maintain tenant identity (`Account`) and link users, sites, sectors, plans, subscriptions, and credits. + - Control access via account state and user roles. + - Provide account-level settings (name, billing email/address, tax info). + + ## Detailed Behavior + - Account fields: `name`, `slug`, `owner`, `plan`, `credits`, `status` (`active`, `suspended`, `trial`, `cancelled`), timestamps. + - User fields: extends Django user with `account` FK and `role` (`developer`, `owner`, `admin`, `editor`, `viewer`, `system_bot`). + - Plan/subscription: `Plan` defines limits and included credits; `Subscription` links account to plan and holds billing cycle/status; `Account.plan` references current plan. + - Sites: `Site` attaches to account; `SiteUserAccess` grants non-admin users site-level access. + - Middleware: `AccountContextMiddleware` resolves account from JWT or session, ensures account is active, and attaches plan information; rejects access for inactive accounts. + - Settings update: `AccountSettings` API exposes billing fields consumed by `AccountSettingsPage`. + + ## Data Structures / Models Involved (no code) + - `Account`, `User`, `Plan`, `Subscription`, `Site`, `SiteUserAccess`, `Sector`, `CreditTransaction`, `CreditUsageLog`. + + ## Execution Flow + - Signup/Login → JWT issued with `account_id` → middleware sets `request.account`. + - Requests pass through permissions tied to roles and account context. + - Plan changes/subscriptions update `Account.plan` and `Subscription` records (via billing endpoints and `invoice_service`/`payment_service`). + - Account settings updates persist via system/account settings endpoint and reflect in billing data. + + ## Cross-Module Interactions + - Billing uses `Account.credits` and plan linkage; `credit_service` deducts/credits per action. + - Sites/Planner/Writer/Automation all inherit account scoping via base viewsets and base models. + - Integration (WordPress) uses account-scoped `Site.wp_api_key` for API key auth. + + ## State Transitions (if applicable) + - `status`: trial → active → (suspended/cancelled) based on plan/payment handling (enforced via middleware checks). + - `plan` changes when subscriptions are created/updated; `Subscription.status` drives billing state. + - Site activation independent per site; account state still gates overall access. + + ## Error Handling + - Middleware rejects requests for inactive/suspended accounts; API returns unified error response. + - Permissions reject unauthorized roles before hitting business logic. + + ## Tenancy Rules + - Every model inherits account (direct FK or via base models); every request resolves `request.account`. + - Querysets in viewsets filter by `account` (and site/sector where applicable); public reads limited to specific cases (e.g., active site by slug). + + ## Billing Rules (if applicable) + - Plan inclusion and subscription determine credit grants; credits stored on `Account`; deductions logged via `CreditTransaction`/`CreditUsageLog`. + + ## Background Tasks / Schedulers (if applicable) + - None specific to account state; billing/automation tasks run with account context provided by Celery payloads. + + ## Key Design Considerations + - Account is the tenant boundary; all downstream objects carry account FK for isolation. + - Middleware centralizes account activation/plan validation to avoid per-view duplication. + + ## How Developers Should Work With This Module + - When adding new resources, inherit from `AccountBaseModel`/`SiteSectorBaseModel` and use base viewsets to enforce scoping. + - On new plan features/limits, extend `Plan` and ensure middleware/permissions or service checks enforce them. + - Preserve slug immutability for stability across integrations. diff --git a/master-docs/40-product/BILLING-LIFECYCLE.md b/master-docs/40-product/BILLING-LIFECYCLE.md index e69de29b..d85aaaa7 100644 --- a/master-docs/40-product/BILLING-LIFECYCLE.md +++ b/master-docs/40-product/BILLING-LIFECYCLE.md @@ -0,0 +1,69 @@ + # Billing Lifecycle + + ## Purpose + Detail how credits, plans, subscriptions, invoices, and payments flow through the system, and where deductions/additions are enforced. + + ## Code Locations (exact paths) + - Models: `backend/igny8_core/business/billing/models.py` (`CreditTransaction`, `CreditUsageLog`, `CreditCostConfig`, `Invoice`, `Payment`, `CreditPackage`, `PaymentMethodConfig`, `AccountPaymentMethod`) + - Services: `backend/igny8_core/business/billing/services/credit_service.py`, `services/invoice_service.py`, `services/payment_service.py` + - Views/Endpoints: `backend/igny8_core/modules/billing/views.py` + - Frontend billing pages: `frontend/src/pages/account/{AccountBillingPage.tsx,PlansAndBillingPage.tsx,PurchaseCreditsPage.tsx}` + + ## High-Level Responsibilities + - Track credit balance, record usage, and enforce costs for AI/automation actions. + - Manage plans/subscriptions, generate invoices, and record payments. + - Expose account-scoped billing data (balance, usage, invoices, payments, packages, payment methods). + + ## Detailed Behavior + - Credit balance: stored on `Account.credits`; `CreditService` adds/deducts via `CreditTransaction` and logs usage in `CreditUsageLog`. + - Costs: read from `CreditCostConfig` or hardcoded `CREDIT_COSTS` fallbacks inside `credit_service.py`. + - Deductions: called from AI/automation flows to ensure sufficient credits; `InsufficientCreditsError` thrown on shortage. + - Plans/Subscriptions: `Plan` defines price, billing_cycle, included_credits, max sites/sectors/users; `Subscription` links account to plan. Plan selection handled via billing endpoints and frontend plan tabs. + - Invoices: `InvoiceService` creates invoices for subscriptions and credit packages; generates unique invoice numbers and line items. + - Payments: `PaymentService` records payments; manual payments stored with status (pending/processing/etc.); payment methods configured via `PaymentMethodConfig`/`AccountPaymentMethod`. + - Packages: `CreditPackage` defines purchasable credit bundles; purchasing triggers invoice/payment flows. + + ## Data Structures / Models Involved (no code) + - Billing: `CreditTransaction`, `CreditUsageLog`, `CreditCostConfig`, `Invoice`, `Payment`, `CreditPackage`, `PaymentMethodConfig`, `AccountPaymentMethod`. + - Plan: `Plan`, `Subscription` (in `auth/models.py`). + - Account linkage: `Account` holds `credits` and `plan`. + + ## Execution Flow + - Balance/Usage read: billing endpoints return account-scoped balance and usage summaries. + - AI/Automation call: service invokes `CreditService.deduct_credits_for_action` → creates `CreditTransaction` + `CreditUsageLog`. + - Purchase credits: frontend calls billing endpoint → `CreditService.add_credits` + `InvoiceService` for package; payment recorded via `PaymentService` (manual/other). + - Subscription change: endpoint updates `Subscription` + `Account.plan`, generates invoice as needed; payment recorded if required. + - Invoice download: `modules/billing/views.py` exposes invoice retrieval; frontend uses `downloadInvoicePDF`. + + ## Cross-Module Interactions + - Automation/writer AI calls depend on credit checks; insufficient balance blocks operations. + - Account settings feed invoice billing details; payments/invoices are tied to account. + + ## State Transitions (if applicable) + - Payment status: `pending` → `processing/pending_approval` → `succeeded/completed` or `failed/refunded/cancelled/void/uncollectible`. + - Subscription status: `active`/`cancelled` (persisted in `Subscription`); plan reference on account updates accordingly. + + ## Error Handling + - `InsufficientCreditsError` for low balance; surfaces to API as error response. + - Payment/Invoice service raise validation errors (e.g., already paid invoice). + + ## Tenancy Rules + - All billing queries filter by `request.user.account`; viewsets inherit from `AccountModelViewSet`. + - Payment methods, invoices, payments, credit transactions are account-scoped; no cross-tenant access. + + ## Billing Rules (if applicable) + - Costs derived from config constants or `CreditCostConfig`. + - Included credits and limits come from `Plan`; extra purchases via `CreditPackage`. + - Credits must be available before AI/automation runs proceed. + + ## Background Tasks / Schedulers (if applicable) + - None dedicated; billing operations are request-driven. Automation tasks carry account context to deduct credits within Celery runs. + + ## Key Design Considerations + - CreditService centralizes deductions to avoid scattered logic. + - Manual payment flow avoids storing sensitive data; relies on transaction_reference and admin approval. + + ## How Developers Should Work With This Module + - When adding new billable actions, route them through `CreditService` with a defined cost key. + - To change pricing, update `CreditCostConfig` data or the `CREDIT_COSTS` fallback map. + - Keep all billing endpoints inheriting `AccountModelViewSet` to maintain isolation. diff --git a/master-docs/40-product/CONTENT-LIFECYCLE.md b/master-docs/40-product/CONTENT-LIFECYCLE.md index e69de29b..6fc8b53a 100644 --- a/master-docs/40-product/CONTENT-LIFECYCLE.md +++ b/master-docs/40-product/CONTENT-LIFECYCLE.md @@ -0,0 +1,73 @@ + # Content Lifecycle + + ## Purpose + Trace content from keyword intake through clustering, idea generation, task creation, AI writing, image generation, and publishing/sync, mapping to backend services and frontend pages. + + ## Code Locations (exact paths) + - Planner models: `backend/igny8_core/business/planning/models.py` (`Keywords`, `Clusters`, `ContentIdeas`) + - Writer models: `backend/igny8_core/business/content/models.py` (`Tasks`, `Content`, `Images`, `ContentTaxonomy`) + - ViewSets: `backend/igny8_core/modules/planner/views.py`; `backend/igny8_core/modules/writer/views.py` + - Automation orchestration: `backend/igny8_core/business/automation/services/automation_service.py`; Celery tasks `business/automation/tasks.py` + - WordPress sync/publish: `backend/igny8_core/business/integration/services/integration_service.py`, `modules/integration/views.py` + - Frontend pages: `frontend/src/pages/Planner/*`, `frontend/src/pages/Writer/*`, `frontend/src/pages/Automation/AutomationPage.tsx`, `frontend/src/pages/Sites/*` + + ## High-Level Responsibilities + - Collect and cluster keywords, generate ideas, create tasks, generate content and images, then publish/sync. + - Support manual steps via Planner/Writer pages and automated runs via Automation pipeline. + - Maintain site/sector scoping and credit enforcement throughout AI operations. + + ## Detailed Behavior + - Keywords → Clusters: `KeywordViewSet` (upload/create) and `ClusterViewSet` (manual or AI auto_cluster). Models inherit `SiteSectorBaseModel` for scoping. + - Clusters → ContentIdeas: `ContentIdeasViewSet` with AI generation endpoint; ideas track status. + - Ideas/Tasks: `Tasks` model holds brief/keywords/structure/status; created manually or by automation stage. + - Tasks → Content: Writer endpoints generate content (AI) and update `Content` records; statuses managed in writer views. + - Content → Images: `ImagesViewSet` handles image generation and storage; images linked to tasks/content. + - Publishing: Integration service sends content to WordPress via `/api/v1/integration` endpoints; WP plugin responds with IDs and syncs status back. + - Automation: `automation_service.run_automation` executes 7 stages with delays/retries/credit estimates; run status tracked in `AutomationRun`. + + ## Data Structures / Models Involved (no code) + - Planner: `Keywords`, `Clusters`, `ContentIdeas`. + - Writer: `Tasks`, `Content`, `Images`, `ContentTaxonomy`. + - Automation: `AutomationConfig`, `AutomationRun`. + - Integration: `SiteIntegration`. + + ## Execution Flow + - Manual path: upload keywords → cluster (auto/manual) → generate ideas → create tasks → generate content → generate images → publish to WP → WP syncs status back. + - Automated path: Automation pipeline stages perform the same transitions in sequence, logging progress and deducting credits. + - Each stage uses DRF viewsets; automation uses Celery tasks (`run_automation_task`, `resume_automation_task`) and logger for trace files. + + ## Cross-Module Interactions + - Automation consumes planner/writer endpoints and triggers integration publish. + - Integration relies on site’s WP API key (`Site.wp_api_key`) and `SiteIntegration` config for credentials/URLs. + - Billing deducts credits for AI operations through `CreditService`. + + ## State Transitions (if applicable) + - Keywords: status (active/inactive/used) per model. + - Clusters: status (active/archived). + - ContentIdeas: status (draft/approved/in_progress/completed). + - Tasks: status simplified (queued/completed) in Stage 1; content status (draft/published) and image status stored on related models. + - AutomationRun: status across run lifecycle (created/running/paused/completed/failed). + + ## Error Handling + - Viewsets rely on unified DRF error responses; automation logger records failures per stage; Celery retries configured where applicable. + - WordPress sync errors surfaced via integration service responses. + + ## Tenancy Rules + - All planner/writer/automation/integration models inherit account/site/sector via base models; viewsets filter by `request.account` and query params. + + ## Billing Rules (if applicable) + - AI clustering/idea/content/image generation deduct credits via `CreditService` cost map; automation blocks run if estimated credits insufficient. + + ## Background Tasks / Schedulers (if applicable) + - Automation scheduler: `check_scheduled_automations` enqueues runs; `run_automation_task`/`resume_automation_task` execute pipeline in Celery. + - AI tasks run async via Celery wrappers (`ai/tasks.py`). + + ## Key Design Considerations + - Single source of truth for tenancy via base models; prevents cross-tenant data exposure. + - Automation mirrors manual flow; manual pages remain usable when automation is paused or disabled. + - WordPress publish/sync uses API key auth to avoid storing user credentials beyond site integration config. + + ## How Developers Should Work With This Module + - When adding new content stages, update both manual endpoints and automation stages. + - Ensure every new action that calls AI is wired through `CreditService` and respects site/sector filters. + - For new publishing targets, extend `SiteIntegration` and integration service while keeping site/account scoping. diff --git a/master-docs/40-product/PERMISSION-MATRIX.md b/master-docs/40-product/PERMISSION-MATRIX.md index e69de29b..f1e5e2bd 100644 --- a/master-docs/40-product/PERMISSION-MATRIX.md +++ b/master-docs/40-product/PERMISSION-MATRIX.md @@ -0,0 +1,54 @@ + # Permission Matrix + + ## Purpose + Define which roles can perform which actions across major modules, based on backend permission classes and role checks, so feature teams know the enforced access patterns. + + ## Code Locations (exact paths) + - Roles: `backend/igny8_core/auth/models.py` (`User.role`) + - Permission classes: `backend/igny8_core/auth/permissions.py` (`IsOwnerOrAdmin`, `IsEditorOrAbove`, `IsViewerOrAbove`, `AccountPermission`) + - Base viewsets: `backend/igny8_core/api/base.py` (Account/Site/Sector scoping; unified responses) + - Middleware: `backend/igny8_core/auth/middleware.py` (ensures account context and plan status) + - Frontend guards: `frontend/src/App.tsx` (`ProtectedRoute`, `ModuleGuard`), `frontend/src/store/authStore.ts` (role in state) + + ## Roles + - `developer` + - `owner` + - `admin` + - `editor` + - `viewer` + - `system_bot` + + ## High-Level Rules (backend) + - Owner/Admin: full CRUD on account resources; pass `IsOwnerOrAdmin`. + - Editor: edit/create content within account; passes `IsEditorOrAbove`. + - Viewer: read-only access; passes `IsViewerOrAbove`. + - AccountPermission: checks authenticated user and role presence; used as base for many endpoints. + - SiteUserAccess: for non-admins, site membership governs visibility in `SiteViewSet` list. + + ## Module Access (derived from permission classes) + - Auth/Account: team management/settings via owner/admin; viewers can read limited data. + - Planner (keywords/clusters/ideas): editor+ for write; viewer for read (via `IsEditorOrAbove` on actions). + - Writer (tasks/content/images): editor+ for write (generate/update/delete); viewer for read. + - Automation: editor+ to configure/run/pause/resume; viewer can read status/history where permitted. + - Integration: editor+ to connect/test/sync; viewer can typically read integration status. + - Billing: owner/admin for invoices/payments/payment methods/plan changes; others read balance/usage where exposed. + - System settings: owner/admin for global/account settings; viewer may read select public prompts/strategies if exposed by viewset permissions. + + ## Tenancy Enforcement + - Middleware sets `request.account`; base viewsets filter by account/site/sector; permissions applied after scoping. + - API key auth (WordPress) sets account from `Site.wp_api_key`, bypassing user roles for plugin calls but still scoped to site/account. + + ## Execution Flow + - Request → JWT/API key auth → `AccountContextMiddleware` (account + plan validation) → DRF permission class → queryset filtering by base viewset → action logic. + + ## Error Handling + - Permission denials return unified error response via DRF handler; 403 for insufficient role; 401 for unauthenticated. + + ## Cross-Module Interactions + - Role checks consistent via shared permission classes; modules rely on same `IsEditorOrAbove`/`IsOwnerOrAdmin`. + - Frontend `ModuleGuard` mirrors backend by restricting route access based on `authStore.user.role` and `allowedModules`. + + ## How Developers Should Work With This Module + - When adding endpoints, choose the strictest applicable permission (`IsOwnerOrAdmin` > `IsEditorOrAbove` > `IsViewerOrAbove`). + - Keep AccountPermission (auth required) on all tenant resources; avoid anonymous access except intentional public endpoints (e.g., site slug read). + - If creating module-specific roles, extend permission classes centrally to avoid drift between modules. diff --git a/master-docs/40-product/ROLE-DEFINITIONS.md b/master-docs/40-product/ROLE-DEFINITIONS.md index e69de29b..f3bddb35 100644 --- a/master-docs/40-product/ROLE-DEFINITIONS.md +++ b/master-docs/40-product/ROLE-DEFINITIONS.md @@ -0,0 +1,36 @@ + # Role Definitions + + ## Purpose + Define each role available in IGNY8, what it can access, and where those permissions are enforced. + + ## Code Locations (exact paths) + - Role field: `backend/igny8_core/auth/models.py` (`User.role`) + - Permission classes: `backend/igny8_core/auth/permissions.py` (`IsOwnerOrAdmin`, `IsEditorOrAbove`, `IsViewerOrAbove`, `AccountPermission`) + - Frontend guards: `frontend/src/App.tsx` (`ProtectedRoute`, `ModuleGuard`), `frontend/src/store/authStore.ts` (role persisted in state) + + ## Roles and Capabilities (backend-enforced) + - `developer`: Treated as admin-level for debugging; passes owner/admin checks. + - `owner`: Full tenant control (users, billing, plans, sites, automation). + - `admin`: Nearly full control; same as owner for most modules except potential business policy differences (not differentiated in code). + - `editor`: Can create/update planner/writer/automation configs; cannot perform owner/admin-only billing or team admin actions. + - `viewer`: Read-only access where `IsViewerOrAbove` is applied; cannot mutate resources. + - `system_bot`: Reserved for system automation; not typically used via UI; expected to bypass some checks when invoked server-side (still scoped to account). + + ## Enforcement Points + - DRF permissions: Applied per view/action using classes above; combined with account/site/sector scoping from base viewsets. + - Middleware: `AccountContextMiddleware` must set `request.account` and reject inactive accounts before permissions run. + - Site scoping: For non-admins, `SiteUserAccess` limits visible sites in `SiteViewSet`. + + ## Cross-Module Impact + - Planner/Writer/Automation: `IsEditorOrAbove` gates writes; viewers read only. + - Billing: owner/admin (and developer) expected for invoices/payments/plan changes; editors/viewers not permitted to mutate. + - Integration: editor+ for connect/test/sync; viewer may read status where allowed. + - System settings: owner/admin for account/system settings; others limited. + + ## Tenancy Rules + - Roles operate within the resolved `request.account`; there is no cross-tenant role scope. API key flows bypass user roles but remain account-scoped. + + ## How Developers Should Work With This Module + - When adding a new endpoint, pick the minimal role needed and reuse existing permission classes. + - If introducing finer-grained roles, extend `permissions.py` centrally and update `ModuleGuard` to mirror backend expectations. + - Keep `User.role` values consistent; avoid hardcoding role strings outside permission helpers. diff --git a/master-docs/40-product/SITE-LIFECYCLE.md b/master-docs/40-product/SITE-LIFECYCLE.md index e69de29b..86fc5a4a 100644 --- a/master-docs/40-product/SITE-LIFECYCLE.md +++ b/master-docs/40-product/SITE-LIFECYCLE.md @@ -0,0 +1,82 @@ + # Site Lifecycle + + ## Purpose + Describe how sites are created, configured, activated, integrated, and used across planner/writer/automation flows, including sector selection and WordPress connectivity. + + ## Code Locations (exact paths) + - Models: `backend/igny8_core/auth/models.py` (`Site`, `Sector`, `Industry`, `IndustrySector`, `SiteUserAccess`) + - ViewSet: `backend/igny8_core/auth/views.py` (`SiteViewSet`) + - Routing: `backend/igny8_core/auth/urls.py` (paths under `/api/v1/auth/sites/`) + - Sector selection logic: `SiteViewSet.select_sectors` action + - Frontend pages: `frontend/src/pages/Sites/{Manage.tsx,Dashboard.tsx,Settings.tsx}` + - WordPress integration service: `backend/igny8_core/business/integration/services/integration_service.py`, `modules/integration/views.py` + + ## High-Level Responsibilities + - Create and manage sites per account, with optional industry and sector assignments. + - Enforce plan-based sector limits and site activation. + - Provide public read-by-slug for active sites (renderer support). + - Connect to external platforms (WordPress) using API key–based integration. + + ## Detailed Behavior + - Creation (`POST /auth/sites/`): Authenticated user; account auto-set from request; slug stored; no single-active constraint. + - Listing (`GET /auth/sites/`): + - Unauthenticated + `slug` param: returns active site matching slug (public read). + - Admin/developer: all sites. + - Owner/admin: sites for user’s account. + - Others: sites where `SiteUserAccess` exists. + - Activation (`POST /auth/sites/{id}/set_active/`): Marks site active; does not auto-deactivate others. + - Sector selection (`POST /auth/sites/{id}/select_sectors/`): + - Requires `industry_slug`; optional `sector_slugs`. + - Validates industry active; enforces `get_max_sectors_limit` from plan. + - Deactivates sectors not provided; ensures sectors belong to industry; creates missing site sectors; returns site + sectors. + - Sectors listing (`GET /auth/sites/{id}/sectors/`): Active sectors for site. + - WordPress integration: `Site.wp_api_key` used by API key auth; `SiteIntegration` stores credentials/config; integration endpoints test connection and sync structure/content. + - Frontend: + - Manage list (`Sites/Manage.tsx`): cards with hosting/type, WP integration button, CRUD, navigation to settings. + - Dashboard (`Sites/Dashboard.tsx`): site overview, stats placeholders, quick actions. + - Settings (`Sites/Settings.tsx`): edit core fields, SEO metadata, industry/sector selection, WP integration tab and content-types tab. + + ## Data Structures / Models Involved (no code) + - `Site`: account FK, name, slug, domain/wordpress fields, industry, hosting_type/site_type, status, is_active, `wp_api_key`. + - `Sector`: account/site FK, name, industry_sector FK, status. + - `Industry`/`IndustrySector`: reference data for industry/sector taxonomy. + - `SiteIntegration`: platform, credentials_json, config_json, flags. + - `SiteUserAccess`: grants non-admin access to specific sites. + + ## Execution Flow + - Create site → optionally set industry/sectors via `select_sectors` → activate → connect WordPress (integration endpoints) → use site across planner/writer/automation via site/sector filters. + - Renderer/public slug access: unauthenticated `GET /auth/sites/?slug=...` returns active site for public consumption. + + ## Cross-Module Interactions + - Planner/Writer/Automation viewsets inherit `SiteSectorModelViewSet` to require `site_id`/`sector_id` filters and attach site/sector to records. + - Integration module uses `Site.wp_api_key` to authenticate WP plugin requests and `SiteIntegration` for config/credentials. + - Plan limits (max sectors) enforced during `select_sectors`. + + ## State Transitions (if applicable) + - Site `status`/`is_active` set via activation endpoint. + - Sectors activated/deactivated by `select_sectors`; industry assignment updated. + - Integration connection status managed via integration service responses (not persisted on Site). + + ## Error Handling + - 400 on missing industry slug or exceeding sector limits; 404 for missing site/industry; 500 if account missing on site during sector selection. + - Integration connection errors returned from integration service; frontend shows toasts. + + ## Tenancy Rules + - `SiteViewSet` filters by account via `AccountModelViewSet`; public slug read is the only unauthenticated path and limited to active site. + - `SiteUserAccess` governs non-admin user visibility. + + ## Billing Rules (if applicable) + - Sector limit enforced from plan (`get_max_sectors_limit`); additional sites/sectors constrained by plan. + + ## Background Tasks / Schedulers (if applicable) + - None for site management; integration sync tasks may run via integration services but are triggered, not scheduled here. + + ## Key Design Considerations + - Public slug read enables site renderer without auth while keeping other operations tenant-scoped. + - Sector selection centralizes industry/plan validation and avoids orphaned sectors. + - WP integration uses API key to avoid sharing user passwords. + + ## How Developers Should Work With This Module + - When adding new hosting/platform types, extend Site model fields plus integration handling. + - Keep all site-scoped resources inheriting site/sector base models and viewsets to maintain isolation. + - If adding analytics/stats, expose a dedicated endpoint and wire to Dashboard page cards. diff --git a/master-docs/40-product/USER-FLOW-OVERVIEW.md b/master-docs/40-product/USER-FLOW-OVERVIEW.md index e69de29b..f6f8087a 100644 --- a/master-docs/40-product/USER-FLOW-OVERVIEW.md +++ b/master-docs/40-product/USER-FLOW-OVERVIEW.md @@ -0,0 +1,77 @@ + # User Flow Overview + + ## Purpose + Describe the end-to-end journey from signup through planning, writing, automation, publishing, and billing within IGNY8, mapping each step to concrete backend/frontend modules so engineers can navigate without scanning code. + + ## Code Locations (exact paths) + - Auth & account: `backend/igny8_core/auth/{views.py,serializers.py,models.py,middleware.py}`, `frontend/src/store/authStore.ts`, routes in `frontend/src/App.tsx` + - Planner: `backend/igny8_core/modules/planner/views.py`, `business/planning/models.py`, `frontend/src/pages/Planner/*` + - Writer: `backend/igny8_core/modules/writer/views.py`, `business/content/models.py`, `frontend/src/pages/Writer/*` + - Automation: `backend/igny8_core/business/automation/{services/automation_service.py,tasks.py,views.py}`, `frontend/src/pages/Automation/AutomationPage.tsx` + - Publishing/Integration: `backend/igny8_core/business/integration/{models.py,services/integration_service.py}`, `modules/integration/views.py`, WordPress plugin endpoints consumed via `/api/v1/integration/`; `frontend/src/pages/Sites/*` + - Billing: `backend/igny8_core/business/billing/{models.py,services/credit_service.py,services/invoice_service.py,views.py}`, `frontend/src/pages/account/*` + - Tenancy enforcement: `backend/igny8_core/auth/middleware.py`, `backend/igny8_core/api/base.py` + + ## High-Level Responsibilities + - Authenticate users and attach account context to every request. + - Let users plan keywords/clusters/ideas, create tasks and content, optionally automate all seven pipeline stages. + - Manage sites/sectors and connect WordPress for publishing/sync. + - Track credits, plans, subscriptions, invoices, and payments through billing services. + + ## Detailed Behavior + - Signup/Login: `auth/views.py` issues JWTs; `AccountContextMiddleware` sets `request.account`; frontend `authStore` stores tokens and refreshes them. + - Planning: `KeywordViewSet`, `ClusterViewSet`, `ContentIdeasViewSet` create/list/update scoped by `SiteSectorModelViewSet` filters; frontend planner pages drive these endpoints. + - Writing: `TasksViewSet`, `ContentViewSet`, `ImagesViewSet` manage tasks/content/images; AI generation endpoints trigger Celery-backed functions. + - Automation: `AutomationViewSet` + `automation_service.py` orchestrate 7 stages (keywords→clusters→ideas→tasks→content→image-prompts→images/manual review) with pause/resume and credit estimation; Celery tasks `run_automation_task`/`resume_automation_task` execute runs. + - Publishing/Integration: `IntegrationViewSet` handles WP connection tests and sync; WP plugin sends/receives data via API key; content publish endpoints in writer module update WordPress via integration services. + - Billing: credit balances and costs computed in `credit_service.py`; invoices via `invoice_service.py`; payments via `payment_service.py`; endpoints in `modules/billing/views.py` feed frontend billing pages. + + ## Data Structures / Models Involved (no code) + - Tenancy: `Account`, `Site`, `Sector` (plus `SiteUserAccess`), base models `AccountBaseModel`/`SiteSectorBaseModel`. + - Planning: `Keywords`, `Clusters`, `ContentIdeas`. + - Writing: `Tasks`, `Content`, `Images`. + - Automation: `AutomationConfig`, `AutomationRun`. + - Billing: `CreditTransaction`, `CreditUsageLog`, `CreditCostConfig`, `Invoice`, `Payment`, `CreditPackage`, `Subscription`, `Plan`. + - Integration: `SiteIntegration`. + + ## Execution Flow + 1) User signs in → JWT stored → `AccountContextMiddleware` populates `request.account`. + 2) Tenant creates sites/sectors (`SiteViewSet`), selects industry/sectors, optionally connects WordPress. + 3) Planner: upload/enter keywords → cluster → generate ideas (manual or via automation/AI functions). + 4) Writer: create tasks from ideas, generate content/images (manual endpoints or automation stages). + 5) Publish: send to WordPress via integration endpoints or automation publish step; WP plugin syncs back statuses. + 6) Automation (optional): run 7-stage pipeline via `AutomationViewSet` + Celery tasks; pause/resume supported. + 7) Billing: credits deducted per AI/pipeline usage (`credit_service`), invoices/payments recorded; users view in billing pages. + + ## Cross-Module Interactions + - Automation invokes planner/writer endpoints/services and logs credit estimates. + - Billing hooks into automation and writer AI calls via credit deduction utilities. + - Integration uses site/account context and WP API key for sync; writer publish flows depend on integration configuration. + + ## State Transitions (if applicable) + - Account status (`active/suspended/trial/cancelled`) governs access; plan/subscription affect limits. + - Tasks/content/images status transitions handled in writer endpoints; automation run status moves through `created/running/paused/completed/failed`. + - Site activation via `set_active`; sectors toggled via `select_sectors`. + + ## Error Handling + - Unified API responses via `api/response.py`; DRF exception handler configured in settings; frontend shows toasts/banners. + - Automation errors logged via `automation_logger`; tasks wrapped in Celery with retries where defined. + + ## Tenancy Rules + - `AccountContextMiddleware` sets `request.account`; base viewsets filter by account/site/sector; API key auth sets account from `Site.wp_api_key`; public site slug reads limited to active sites. + + ## Billing Rules (if applicable) + - AI/automation uses `CreditService.deduct_credits_for_action`; credit balance required before runs; plans/subscriptions define included credits and sector/site limits. + + ## Background Tasks / Schedulers (if applicable) + - Celery tasks: automation runs (`run_automation_task`, `resume_automation_task`), scheduler `check_scheduled_automations`, AI functions (`ai/tasks.py`), publishing tasks (`tasks/wordpress_publishing.py`). + + ## Key Design Considerations + - Strong tenant isolation via middleware + filtered querysets. + - Automation relies on polling and Celery; frontend automation page polls every 5s during runs. + - Billing is authoritative in backend; frontend is read-only except initiating purchases/subscriptions. + + ## How Developers Should Work With This Module + - Trace user-visible flows by following router → page component → service call → backend viewset. + - When adding steps, ensure credit deductions and tenancy filters are applied in the corresponding backend service/viewset. + - Keep WP integration changes consistent with API key auth and `SiteIntegration` schema. diff --git a/master-docs/50-infra/BACKUP-AND-RECOVERY.md b/master-docs/50-infra/BACKUP-AND-RECOVERY.md index e69de29b..78e7d3f6 100644 --- a/master-docs/50-infra/BACKUP-AND-RECOVERY.md +++ b/master-docs/50-infra/BACKUP-AND-RECOVERY.md @@ -0,0 +1,59 @@ + # Backup and Recovery + + ## Purpose + Describe what needs backup and how to restore based on the runtime components present (Postgres DB, Redis, logs, sites data). No automated backup scripts are versioned here; guidance is derived from files in the repo. + + ## Code Locations (exact paths) + - Database defaults: `backend/igny8_core/settings.py` (Postgres/SQLite selection via env) + - Compose mounts: `docker-compose.app.yml` (backend logs volume, sites data mounts) + - Example DB dumps present: `backend/backup_postgres_20251120_232816.sql`, `backend/db_backup_20251120_232646.sqlite3` + + ## High-Level Responsibilities + - Preserve database state (Postgres primary target). + - Preserve published sites data (`/data/app/sites-data`) consumed by sites renderer. + - Preserve application logs for audit/debug. + + ## Detailed Behavior + - Primary datastore: Postgres (preferred). `settings.py` falls back to SQLite only in DEBUG/no-env scenarios; production should use Postgres to enable reliable backups. + - Redis: used as Celery broker/result backend; not a system of record—no backups required beyond availability. + - Files to keep: + - Postgres DB: dump regularly (e.g., `pg_dump`); sample dumps in `backend/` illustrate format. + - Sites data: mounted read-only into sites renderer from `/data/app/sites-data`; back up that directory to retain deployed static sites. + - Logs: stored under `/data/app/logs` and `backend/logs/publish-sync-logs`; optional for troubleshooting. + - Static assets: `backend/staticfiles` can be regenerated via `collectstatic`; not required to back up if build pipeline exists. + + ## Data Structures / Models Involved (no code) + - All Django models persist in Postgres; automation logs/files live in `logs/` directory. + + ## Execution Flow (Suggested) + - Postgres backup: `pg_dump -Fc -h -U > igny8_.dump` + - Sites data backup: archive `/data/app/sites-data` (and any uploads stored under `/data/app/igny8/sites` if used). + - Logs backup: optional tar of `/data/app/logs` and `backend/logs/publish-sync-logs`. + - Restore Postgres: `pg_restore -c -d -h -U igny8_.dump` (ensure DB created and app stopped before restore). + + ## Cross-Module Interactions + - Billing, tenancy, automation data all live in the DB; restoring DB restores these states. + - Sites renderer consumes `/data/app/sites-data`; ensure it matches DB state if site URLs/records reference deployed assets. + + ## State Transitions (if applicable) + - After restore, run `python manage.py migrate` to align schema, then restart backend/worker/beat. + + ## Error Handling + - Backups should be verified via `pg_restore --list` or test restores; failed restores will surface at DB connection time. + + ## Tenancy Rules + - Restoring DB restores tenant isolation; ensure backups are tenant-wide and protected. + + ## Billing Rules (if applicable) + - Billing data (credits, invoices, payments) is DB-resident; backups must be handled securely due to financial data sensitivity. + + ## Background Tasks / Schedulers (if applicable) + - Stop Celery beat/worker during restore to avoid tasks running on partial data. + + ## Key Design Considerations + - Use Postgres in all non-local environments to avoid SQLite backups. + - Keep DB credentials and dumps secure; avoid storing secrets in dumps. + + ## How Developers Should Work With This Module + - Add automated backup scripts (cron/Portainer stacks) external to this repo; document their paths and retention when added. + - Remove legacy `.sql` dumps from repo in production to avoid stale data exposure; rely on managed backups instead. diff --git a/master-docs/50-infra/CI-CD-PIPELINE.md b/master-docs/50-infra/CI-CD-PIPELINE.md index e69de29b..38a85d15 100644 --- a/master-docs/50-infra/CI-CD-PIPELINE.md +++ b/master-docs/50-infra/CI-CD-PIPELINE.md @@ -0,0 +1,62 @@ + # CI/CD Pipeline + + ## Purpose + Document the build-and-deploy sequence using the artifacts present in the repo. No CI config is versioned here; this describes the expected pipeline based on Dockerfiles and compose. + + ## Code Locations (exact paths) + - Compose stack definition: `docker-compose.app.yml` + - Backend image definition: `backend/Dockerfile` + - Frontend image definition: `frontend/Dockerfile` (production build via Caddy) + - Backend config: `backend/igny8_core/settings.py` + + ## High-Level Responsibilities + - Build backend and frontend images from repo sources. + - Push images to registry (external step, not defined here). + - Deploy via `docker-compose.app.yml` consuming those images on the target host/Portainer. + + ## Detailed Behavior + - Build steps (as indicated in compose comments): + - Backend: `docker build -t igny8-backend:latest -f backend/Dockerfile backend` + - Frontend app: `docker build -t igny8-frontend-dev:latest -f frontend/Dockerfile.dev frontend` + - Marketing: `docker build -t igny8-marketing-dev:latest -f frontend/Dockerfile.marketing.dev frontend` + - Sites renderer: `docker build -t igny8-sites-dev:latest -f sites/Dockerfile.dev sites` + - Deploy steps: + - Ensure external infra stack (Postgres/Redis/network `igny8_net`) is running. + - Pull/receive built images on target. + - Run `docker compose -f docker-compose.app.yml -p igny8-app up -d`. + - Healthcheck: backend service health gated by `/api/v1/system/status/`; frontend depends_on backend health. + - No automated migrations are defined in compose; run `python manage.py migrate` inside backend container before switching traffic if schema changes are present. + + ## Data Structures / Models Involved (no code) + - None; pipeline operates on container images. + + ## Execution Flow + - CI: build images → (optional tests not defined here) → push to registry. + - CD: pull images on host → `docker compose up -d` using new tags → healthcheck ensures backend ready. + + ## Cross-Module Interactions + - Celery worker/beat use the same backend image; ensure it is rebuilt alongside backend changes. + + ## State Transitions (if applicable) + - Service rollout occurs when new containers start; existing volumes (code mounts in compose) mean host filesystem is source of truth in current compose. + + ## Error Handling + - Build failures visible during `docker build`. + - Deploy failures captured via Docker/Portainer logs; unhealthy backend blocks dependent services. + + ## Tenancy Rules + - Not altered by CI/CD; enforced at runtime by application. + + ## Billing Rules (if applicable) + - None in pipeline. + + ## Background Tasks / Schedulers (if applicable) + - Celery beat/worker containers start via compose; ensure images include latest task code. + + ## Key Design Considerations + - Current compose uses local bind mounts (`/data/app/igny8/...`) which expect code present on host; in an immutable-image pipeline, remove bind mounts and bake code into images instead. + - External infra stack separation requires coordination to ensure DB/Redis available during deploy. + + ## How Developers Should Work With This Module + - If introducing automated CI, codify the build steps above and add migrations/test stages. + - When changing service names or ports, update compose and healthcheck references consistently. diff --git a/master-docs/50-infra/DEPLOYMENT-GUIDE.md b/master-docs/50-infra/DEPLOYMENT-GUIDE.md index e69de29b..c97dc33e 100644 --- a/master-docs/50-infra/DEPLOYMENT-GUIDE.md +++ b/master-docs/50-infra/DEPLOYMENT-GUIDE.md @@ -0,0 +1,74 @@ + # Deployment Guide + + ## Purpose + Describe how to deploy the IGNY8 stack using the provided Dockerfiles and `docker-compose.app.yml`, including service wiring and required external dependencies. + + ## Code Locations (exact paths) + - App compose stack: `docker-compose.app.yml` + - Backend image: `backend/Dockerfile` + - Frontend image: `frontend/Dockerfile` + - Backend settings/env: `backend/igny8_core/settings.py` + + ## High-Level Responsibilities + - Build images for backend, frontend, marketing, and sites renderer. + - Bring up the app stack (backend, frontend, marketing, sites, Celery worker, Celery beat) on the shared external network. + - Rely on external infra services (Postgres, Redis) defined outside this repo (referenced in compose comments). + + ## Detailed Behavior + - Backend container: + - Image `igny8-backend:latest` from `backend/Dockerfile` (Python 3.11 slim, installs `requirements.txt`, runs Gunicorn on 8010). + - Mounted volumes: `/data/app/igny8/backend` (code), `/data/app/igny8` (shared), `/data/app/logs` (logs). + - Env vars for DB/Redis and security flags (USE_SECURE_COOKIES/PROXY, DEBUG, SECRET_KEY). + - Healthcheck hits `http://localhost:8010/api/v1/system/status/`. + - Frontend container: + - Image `igny8-frontend-dev:latest` from `frontend/Dockerfile.dev` (built separately; serves via Vite dev server on 5173 exposed as 8021). + - Env `VITE_BACKEND_URL`. + - Marketing dev and Sites renderer containers: images `igny8-marketing-dev:latest` and `igny8-sites-dev:latest`, ports 8023→5174 and 8024→5176; Sites mounts `/data/app/igny8/sites` and `/data/app/sites-data`. + - Celery worker/beat: + - Use `igny8-backend:latest`, commands `celery -A igny8_core worker` and `celery -A igny8_core beat`. + - Share same DB/Redis env and code volumes. + - Network: `igny8_net` marked `external: true`; compose expects Postgres and Redis running in another stack (`/data/app/docker-compose.yml` per comment). + - Ports: backend 8011→8010, frontend 8021→5173, marketing 8023→5174, sites 8024→5176. + + ## Data Structures / Models Involved (no code) + - Not model-specific; relies on runtime env (Postgres DB, Redis broker). + + ## Execution Flow + - Build images: + - `docker build -t igny8-backend:latest -f backend/Dockerfile backend` + - `docker build -t igny8-frontend-dev:latest -f frontend/Dockerfile.dev frontend` + - `docker build -t igny8-marketing-dev:latest -f frontend/Dockerfile.marketing.dev frontend` + - `docker build -t igny8-sites-dev:latest -f sites/Dockerfile.dev sites` + - Ensure external infra stack (Postgres, Redis, network `igny8_net`) is up. + - Run: `docker compose -f docker-compose.app.yml -p igny8-app up -d`. + - Healthcheck will mark backend healthy before frontend depends_on proceeds. + + ## Cross-Module Interactions + - Backend depends on Postgres/Redis; Celery worker/beat rely on same env to process automation/AI tasks. + - Frontend depends on backend health; sites renderer reads deployed sites from `/data/app/sites-data`. + + ## State Transitions (if applicable) + - Container lifecycle managed by Docker restart policy (`restart: always`). + + ## Error Handling + - Backend healthcheck fails if status endpoint not reachable; container marked unhealthy causing depends_on wait. + - Gunicorn exit surfaces via Docker logs; no auto-restart beyond Docker restart policy. + + ## Tenancy Rules + - Enforced in application layer via middleware; deployment does not alter tenancy. + + ## Billing Rules (if applicable) + - None at deployment layer. + + ## Background Tasks / Schedulers (if applicable) + - Celery beat runs schedules (e.g., automation scheduler) using same image and env. + + ## Key Design Considerations + - Compose uses images (not builds) to avoid accidental rebuilds in Portainer; images must exist beforehand. + - External network requirement means infra stack must pre-create `igny8_net` and services. + - Backend served by Gunicorn with 4 workers per compose command; adjust via compose if scaling container horizontally. + + ## How Developers Should Work With This Module + - When changing env vars, update `docker-compose.app.yml` and keep parity with `settings.py`. + - For new services (e.g., monitoring), add to compose and attach to `igny8_net`. + - Keep healthcheck endpoint stable (`/api/v1/system/status/`) or update compose accordingly. diff --git a/master-docs/50-infra/ENVIRONMENT-SETUP.md b/master-docs/50-infra/ENVIRONMENT-SETUP.md index e69de29b..c7879d2d 100644 --- a/master-docs/50-infra/ENVIRONMENT-SETUP.md +++ b/master-docs/50-infra/ENVIRONMENT-SETUP.md @@ -0,0 +1,72 @@ + # Environment Setup + + ## Purpose + Outline required runtime dependencies, environment variables, and local setup steps derived from the codebase configuration. + + ## Code Locations (exact paths) + - Django settings/env: `backend/igny8_core/settings.py` + - Backend dependencies: `backend/requirements.txt` + - Backend image provisioning: `backend/Dockerfile` + - Frontend env/build: `frontend/Dockerfile`, `frontend/package.json`, `frontend/vite.config.ts` + - Compose stack env: `docker-compose.app.yml` + + ## High-Level Responsibilities + - Provide prerequisites for backend (Python 3.11, Postgres, Redis) and frontend (Node 18). + - Enumerate environment variables consumed by backend and compose files. + - Describe local or containerized setup flows. + + ## Detailed Behavior + - Backend settings require: + - `SECRET_KEY`, `DEBUG`, `USE_SECURE_COOKIES`, `USE_SECURE_PROXY_HEADER`. + - Database: `DATABASE_URL` or `DB_HOST`, `DB_NAME`, `DB_USER`, `DB_PASSWORD`, `DB_PORT`; falls back to SQLite in DEBUG if none provided. + - Redis/Celery: `CELERY_BROKER_URL`, `CELERY_RESULT_BACKEND` default to `redis://{REDIS_HOST}:{REDIS_PORT}/0`. + - JWT: `JWT_SECRET_KEY`, expiry defaults (15m access, 30d refresh). + - CORS: allowed origins include local ports (5173/5174/5176/8024) and `app.igny8.com`. + - Stripe/PayPal keys optional (`STRIPE_PUBLIC_KEY`, `STRIPE_SECRET_KEY`, `PAYPAL_*`). + - Backend Dockerfile installs system deps (gcc, libpq-dev) and pip installs `requirements.txt`; runs `collectstatic` (best-effort). + - Frontend expects `VITE_BACKEND_URL` (compose sets to `https://api.igny8.com/api`); build via `npm install` then `npm run build` (Dockerfile). + - Compose injects DB/Redis env to backend/worker/beat and secure cookie/proxy flags for production use. + + ## Data Structures / Models Involved (no code) + - Not model-specific; environment affects DB connections, auth, CORS, Celery, billing keys. + + ## Execution Flow + - Local (backend): + - `python -m venv .venv && source .venv/bin/activate` + - `pip install -r backend/requirements.txt` + - Set env vars (DB/REDIS/JWT/CORS/SECRET_KEY). + - `python backend/manage.py migrate` then `python backend/manage.py runserver 8010`. + - Local (frontend): + - `npm install` in `frontend/` + - Set `VITE_BACKEND_URL` + - `npm run dev -- --host --port 5173` + - Containerized: + - Build images per Dockerfiles; run `docker compose -f docker-compose.app.yml up -d` with external Postgres/Redis and network `igny8_net`. + + ## Cross-Module Interactions + - Celery uses same Redis host/port as defined in env; automation tasks rely on this. + - CORS/secure cookie flags must align with frontend host to allow auth. + + ## State Transitions (if applicable) + - `DEBUG` toggles throttling bypass (`IGNY8_DEBUG_THROTTLE`) and SQLite fallback. + + ## Error Handling + - Missing DB env → falls back to SQLite in DEBUG; in prod must set Postgres vars. + - Healthcheck in compose will fail if env misconfigured and backend cannot start. + + ## Tenancy Rules + - Unchanged by env; account context enforced in middleware once app runs. + + ## Billing Rules (if applicable) + - Stripe/PayPal keys optional; without them payment flows are disabled/pending. + + ## Background Tasks / Schedulers (if applicable) + - Celery broker/backend must be reachable; worker/beat require the same env set. + + ## Key Design Considerations + - Prefer Postgres in all shared/test/prod; SQLite only for local development. + - Keep SECRET_KEY/JWT keys distinct and secret in production. + + ## How Developers Should Work With This Module + - Add new env variables in `settings.py` with safe defaults; document in this file. + - Mirror envs in compose and deployment systems (Portainer/CI) to avoid drift. diff --git a/master-docs/50-infra/LOGGING-AND-MONITORING.md b/master-docs/50-infra/LOGGING-AND-MONITORING.md index e69de29b..a209e84e 100644 --- a/master-docs/50-infra/LOGGING-AND-MONITORING.md +++ b/master-docs/50-infra/LOGGING-AND-MONITORING.md @@ -0,0 +1,62 @@ + # Logging and Monitoring + + ## Purpose + Explain runtime logging, request tracing, and resource tracking implemented in code, plus where logs are written. + + ## Code Locations (exact paths) + - Django logging config: `backend/igny8_core/settings.py` (`LOGGING`, `PUBLISH_SYNC_LOG_DIR`) + - Request ID middleware: `backend/igny8_core/middleware/request_id.py` + - Resource tracking middleware: `backend/igny8_core/middleware/resource_tracker.py` + - Publish/sync logging categories: `publish_sync`, `wordpress_api`, `webhooks` loggers in `settings.py` + + ## High-Level Responsibilities + - Attach request IDs to each request/response for traceability. + - Optionally track CPU/memory/I/O per request for authenticated admin/developer users. + - Persist publish/sync logs to rotating files and console. + + ## Detailed Behavior + - Request ID: + - `RequestIDMiddleware` assigns a UUID per request; adds header `X-Request-ID` (exposed via CORS `x-resource-tracking-id`). + - Resource tracking: + - `ResourceTrackingMiddleware` measures CPU/memory/I/O deltas per request for authenticated admin/developer users; stores metrics in cache; can be toggled via custom header `x-debug-resource-tracking`. + - Logging configuration (`settings.py`): + - Log directory: `logs/publish-sync-logs` (auto-created). + - Handlers: console + rotating file handlers (10 MB, 10 backups) for `publish_sync.log`, `wordpress-api.log`, `webhooks.log`. + - Formatters: verbose general formatter; publish_sync formatter with timestamp/level/message. + - Loggers: `publish_sync`, `wordpress_api`, `webhooks` all INFO level, non-propagating. + - Static files and general Django logging fall back to default console (not customized beyond above). + + ## Data Structures / Models Involved (no code) + - None; logging is operational. + + ## Execution Flow + - Incoming request → RequestIDMiddleware → AccountContextMiddleware → ResourceTrackingMiddleware → view. + - Views/services log using standard `logging.getLogger('publish_sync')` etc.; output to console and rotating files. + + ## Cross-Module Interactions + - WordPress integration and publish flows should log to `wordpress_api`/`publish_sync` to capture sync diagnostics. + - CORS exposes request ID header so frontend can correlate with logs. + + ## State Transitions (if applicable) + - Log rotation occurs when files exceed 10 MB (backupCount 10). + + ## Error Handling + - Logging failures fall back to console; middleware errors propagate via unified exception handler. + + ## Tenancy Rules + - Request ID and resource tracking apply to all requests; data remains in application logs scoped by account within log content. + + ## Billing Rules (if applicable) + - None. + + ## Background Tasks / Schedulers (if applicable) + - Celery tasks can log using same loggers; no dedicated Celery handler configured beyond console. + + ## Key Design Considerations + - Request ID early in middleware to ensure downstream logs include correlation ID. + - Resource tracking limited to privileged users to avoid overhead for all traffic. + + ## How Developers Should Work With This Module + - Use named loggers (`publish_sync`, `wordpress_api`, `webhooks`) for relevant flows to keep logs organized. + - When adding new log domains, extend `LOGGING` in `settings.py` with dedicated handlers/formatters as needed. + - Surface request IDs in error responses to aid log correlation (already enabled via unified handler). diff --git a/master-docs/50-infra/SCALING-AND-LOAD-BALANCING.md b/master-docs/50-infra/SCALING-AND-LOAD-BALANCING.md index e69de29b..9324a5a6 100644 --- a/master-docs/50-infra/SCALING-AND-LOAD-BALANCING.md +++ b/master-docs/50-infra/SCALING-AND-LOAD-BALANCING.md @@ -0,0 +1,63 @@ + # Scaling and Load Balancing + + ## Purpose + Outline how services can be scaled based on existing Docker/Gunicorn/Celery configuration and external dependencies. + + ## Code Locations (exact paths) + - Compose stack: `docker-compose.app.yml` + - Backend process model: `docker-compose.app.yml` (Gunicorn command), `backend/Dockerfile` + - Celery worker/beat commands: `docker-compose.app.yml` + - Celery settings: `backend/igny8_core/settings.py` (Celery config) + + ## High-Level Responsibilities + - Describe horizontal/vertical scaling levers for backend and workers. + - Note reliance on external Postgres/Redis and shared network. + + ## Detailed Behavior + - Backend container runs Gunicorn with `--workers 4 --timeout 120` (from compose). Scaling options: + - Increase workers via compose command args. + - Run additional backend containers on the same `igny8_net` behind a reverse proxy (proxy not defined here; compose assumes external Caddy/infra stack handles routing). + - Celery: + - Worker command `celery -A igny8_core worker --loglevel=info --concurrency=4`; concurrency can be increased per container or by adding more worker replicas. + - Beat runs separately to schedule tasks. + - Broker/backend: Redis from external infra; settings enforce prefetch multiplier 1 and task soft/hard time limits (25/30 min). + - Frontend/marketing/sites: + - Served via dev servers in current compose; for production, build static assets (frontend Dockerfile uses Caddy). Can scale by running additional containers behind the proxy. + - Network: + - `igny8_net` is external; load balancer (e.g., Caddy/infra) not defined in this repo but must front multiple backend/frontend replicas if added. + + ## Data Structures / Models Involved (no code) + - None; operational scaling only. + + ## Execution Flow + - Scaling out: start additional containers with the same images on `igny8_net`; configure external proxy to round-robin to backend instances. + - Scaling up: raise Gunicorn worker count or Celery concurrency in compose overrides. + + ## Cross-Module Interactions + - Celery workload includes automation/AI tasks; ensure Redis/Postgres sized accordingly when increasing concurrency. + - Request ID and resource tracking middleware remain per-instance; logs aggregate by container. + + ## State Transitions (if applicable) + - New replicas join network immediately; no shared session storage configured (stateless JWT APIs), so backend replicas are safe behind load balancer. + + ## Error Handling + - If backend not reachable, healthcheck fails and depends_on blocks frontend start. + - Celery tasks exceeding time limits are terminated per settings. + + ## Tenancy Rules + - Unchanged by scaling; tenancy enforced per request in each instance. + + ## Billing Rules (if applicable) + - None; credit deductions occur in application code regardless of scale. + + ## Background Tasks / Schedulers (if applicable) + - Celery beat should remain single active scheduler; if running multiple beats, use external lock (not implemented) to avoid duplicate schedules. + + ## Key Design Considerations + - Ensure reverse proxy/ingress (outside this repo) balances across backend replicas and terminates TLS. + - Keep Redis/Postgres highly available and sized for additional connections when scaling workers/backends. + + ## How Developers Should Work With This Module + - Use compose override or Portainer to adjust worker counts; validate resource limits. + - Avoid multiple Celery beat instances unless coordination is added. + - When introducing production-ready load balancing, add proxy config to infra repo and keep ports consistent with compose. diff --git a/master-docs/90-misc/MIGRATION-NOTES.md b/master-docs/90-misc/MIGRATION-NOTES.md index e69de29b..ebd0e2ba 100644 --- a/master-docs/90-misc/MIGRATION-NOTES.md +++ b/master-docs/90-misc/MIGRATION-NOTES.md @@ -0,0 +1,55 @@ + # Migration Notes + + ## Purpose + Track significant schema or data migrations, with references to the scripts present in the repo that have been used to adjust data or structure. + + ## Code Locations (exact paths) + - Migration/utility scripts (repo root `backend/`): + - `verify_migrations.py`, `verify_status_fixes.py`, `verify_taxonomy.py` + - `fix_*` scripts (e.g., `fix_cluster_status.py`, `fix_content_types.py`, `fix_integration_site_url.py`, `fix_sync.py`, `fix_taxonomy_relationships.py`) + - `rename_fields_migration.sql` + - `inject_test_data.py`, `sync_idea_status.py`, `check_recent_keywords.py`, `check_api_response.py`, `diagnose_generate_content.py` + - Django migrations: `backend/igny8_core/**/migrations/` (per app) + + ## High-Level Responsibilities + - Capture when ad-hoc fixes or manual SQL were required so future migrations can account for existing state. + - Provide guidance on verifying migrations after code changes. + + ## Detailed Behavior + - Python fix/verify scripts operate against live DBs to patch or validate data (names indicate target domain: clusters/content/integration/taxonomy). + - `rename_fields_migration.sql` suggests a manual SQL rename was performed; ensure corresponding Django migration exists or is applied. + - `verify_*` scripts check consistency after migrations or data changes. + - These scripts are not automatically executed; they are run manually as needed. + + ## Execution Flow (Recommended) + - Prefer standard Django migrations first: `python manage.py makemigrations && python manage.py migrate`. + - For data issues covered by fix scripts: + - Review script purpose and run in controlled environment (staging) before production. + - Take DB backup before executing manual SQL or fix scripts. + - After applying migrations/fixes, run verify scripts to confirm expected state. + + ## Cross-Module Interactions + - Scripts touch planner/writer/integration data; impacts automation and billing indirectly if content/status changes alter usage. + + ## State Transitions (if applicable) + - DB schema changes via migrations; data patches via scripts. + + ## Error Handling + - Scripts log/print outputs; failures will surface during execution; no centralized handler. + + ## Tenancy Rules + - All scripts should respect account/site scoping; review code before running to ensure filters exist or add them. + + ## Billing Rules (if applicable) + - None directly; data corrections may affect content/automation counts but not credit logs. + + ## Background Tasks / Schedulers (if applicable) + - None; scripts are on-demand. + + ## Key Design Considerations + - Keep fix/verify scripts version-controlled but treat them as one-off tools; remove or archive obsolete scripts. + - Ensure Django migrations are the source of truth for schema; manual SQL should be mirrored in migrations. + + ## How Developers Should Work With This Module + - Before running any fix/verify script, read it and constrain to target account/site if possible. + - Add notes here when new manual interventions occur, including date/purpose and scripts used. diff --git a/master-docs/90-misc/TEMPORARY-NOTES.md b/master-docs/90-misc/TEMPORARY-NOTES.md index e69de29b..5c6617e5 100644 --- a/master-docs/90-misc/TEMPORARY-NOTES.md +++ b/master-docs/90-misc/TEMPORARY-NOTES.md @@ -0,0 +1,11 @@ + # Temporary Notes + + ## Purpose + Provide a space for short-lived operational notes or in-progress decisions; should be cleared or moved to canonical docs once finalized. + + ## Current Guidance + - No active temporary notes documented. Use this file to capture time-bound decisions, experiments, or rollout instructions, then archive or delete after completion. + + ## How Developers Should Work With This Module + - Add date-stamped bullet points for any temporary guidance. + - When resolved, move durable learnings into the relevant module docs and prune from here.