docs re-org

docs/10-BACKEND/MODELS.md

@@ -0,0 +1,110 @@
+# Domain Models
+
+## Purpose
+Describe the key backend models, their responsibilities, constraints, and tenancy rules, grounded in current implementations.
+
+## Code Locations (exact paths)
+- Tenancy bases and identity: `backend/igny8_core/auth/models.py`
+- Planner models: `backend/igny8_core/business/planning/models.py`
+- Writer/content models: `backend/igny8_core/business/content/models.py`
+- Automation models: `backend/igny8_core/business/automation/models.py`
+- Billing models: `backend/igny8_core/business/billing/models.py`
+- Integration models: `backend/igny8_core/business/integration/models.py`
+- Publishing models: `backend/igny8_core/business/publishing/models.py`
+- Optimization models: `backend/igny8_core/business/optimization/models.py`
+
+## High-Level Responsibilities
+- Provide tenant-scoped storage for planner (keywords/clusters/ideas), writer (tasks/content/taxonomies/images/attributes), automation runs/config, billing (credits, invoices, payments), integration metadata, publishing records, and optimization tasks.
+- Enforce account/site/sector alignment via base classes and save-time validation.
+- Track external platform links (WordPress/Shopify/custom), credit usage, and publishing/optimization state.
+
+## Detailed Behavior
+### Tenancy Bases (auth/models.py)
+- `AccountBaseModel`: adds `account`, timestamps, and indexes; all tenant models inherit.
+- `SiteSectorBaseModel`: extends with `site` and `sector`; save enforces site → account alignment and sector belonging to the same site; raises validation errors if mismatched.
+
+### Planner (business/planning/models.py)
+- `Clusters`: tenant/site/sector-scoped keyword group; tracks counts, volume, mapped pages, status (`new/mapped`), disable flag; unique per site/sector by name; soft-deletable.
+- `Keywords`: tenant/site/sector-scoped keyword tied to a global `SeedKeyword`; optional overrides for volume/difficulty/attributes; optional cluster link (same sector enforced); validation ensures seed keyword industry/sector matches site/sector; status (`new/mapped`), disable flag; soft-deletable.
+- `ContentIdeas`: ideas tied to clusters and optional keywords; tracks status (`new/queued/completed`), content type/structure, estimated word count; soft-deletable.
+
+### Writer / Content (business/content/models.py)
+- `Tasks`: queue items for content generation; tied to cluster (required) and optional idea/taxonomy; content type/structure, keywords text, target word count, status (`queued/completed`); soft-deletable.
+- `Content`: generated or imported content; stores HTML, word count, SEO fields, cluster link, content type/structure, taxonomy M2M, external IDs/URLs/metadata, sync status, source (`igny8/wordpress`), and status (`draft/review/published`); soft-deletable.
+- `ContentTaxonomy`: simplified taxonomy (category/tag) with external taxonomy/ID, sync status, description, count, metadata; unique per site by slug/type and by external ID/taxonomy.
+- `Images`: images linked to content or task; auto-populates account/site/sector from the linked object; tracks type, URL/path, prompt, status, position; soft-deletable.
+- `ContentClusterMap`: maps content/tasks to clusters with role (`hub/supporting/attribute`) and source (`blueprint/manual/import`); auto-populates tenant context from linked content/task; unique per content+cluster+role.
+- `ContentAttribute` (alias `ContentAttributeMap`): tenant/site/sector-scoped attributes for content/task/cluster; typed (`product_spec/service_modifier/semantic_facet`), with optional external IDs, sources, and metadata; auto-populates tenant context from linked content/task.
+
+### Automation (business/automation/models.py)
+- `AutomationConfig`: per-site config with enable flag, frequency (`daily/weekly/monthly`), scheduled time, batch sizes per stage, within/between-stage delays, and next/last run timestamps.
+- `AutomationRun`: tracks each run with trigger (`manual/scheduled`), status (`running/paused/cancelled/completed/failed`), current stage, pause/cancel timestamps, start/end, total credits used, per-stage JSON results, and optional error message.
+
+### Billing (business/billing/models.py)
+- `CreditTransaction`: ledger of credit changes (purchase/subscription/refund/deduction/adjustment) with balance-after and metadata.
+- `CreditUsageLog`: detailed AI usage log with operation type (clustering/idea/content/image/reparse/legacy names), credits used, optional cost/model/tokens, related object references, and metadata.
+- `CreditCostConfig`: admin-configurable credit costs per operation with unit (per request/words/items/images), display metadata, active flag, audit fields, and previous cost tracking.
+- `Invoice`: tenant invoice with amounts, status (`draft/pending/paid/void/uncollectible`), dates, subscription link, line items JSON, payment metadata, Stripe IDs, notes; helper properties mirror legacy fields.
+- `Payment`: payment records per invoice with status lifecycle (pending/processing/succeeded/completed/failed/refunded/cancelled), method (Stripe/PayPal/bank/local wallet/manual), provider references, manual notes/approval fields, failure reason, timestamps, metadata.
+- `CreditPackage`: one-time credit bundles with price, discount, Stripe/PayPal IDs, active/featured flags, description/features, sort order.
+- `PaymentMethodConfig`: per-country payment-method availability and display/instruction fields; includes bank/local wallet metadata; unique per country+method.
+- `AccountPaymentMethod`: account-level payment metadata (non-sensitive) with type, display name, default/enabled/verified flags, country code, instructions, metadata; unique per account+display name.
+
+### Integration (business/integration/models.py)
+- `SiteIntegration`: tenant/site-specific integration config with platform (`wordpress/shopify/custom`), platform type (`cms/ecommerce/custom_api`), config JSON, credentials JSON, active/sync flags, sync status, last sync/error, timestamps; unique per site+platform.
+- `SyncEvent`: event log per integration/site with event/action types, success flag, optional content/external IDs, details JSON, error, duration, and timestamps; indexed for debugging feeds.
+
+### Publishing (business/publishing/models.py)
+- `PublishingRecord`: tracks content publishing to destinations (wordpress/sites/shopify) with destination IDs/URLs, status (`pending/publishing/published/failed`), timestamps, errors, metadata; site/sector scoped via base.
+- `DeploymentRecord`: tracks site deployments (sites renderer) with version/deployed_version, status (`pending/deploying/deployed/failed/rolled_back`), deployment URL, error, metadata, timestamps; site/sector scoped.
+
+### Optimization (business/optimization/models.py)
+- `OptimizationTask`: content optimization runs with before/after scores and HTML, status (`pending/running/completed/failed`), credits used, metadata; auto-sets account from content; tenant scoped.
+
+## Execution Flow
+- Tenant context is inherited from base models; many save methods propagate account/site/sector from related entities (e.g., Images, ContentClusterMap, ContentAttribute).
+- Planner → Writer linkage: Keywords and Clusters feed ContentIdeas; Tasks reference clusters/ideas; Content references clusters and taxonomies; Images/Attributes link to Tasks/Content.
+- Automation runs reference planner/writer models and record per-stage outputs; configs control batching/delays.
+- Billing logs and cost configs govern credit debits triggered by services (see services doc).
+- Integration/publishing models bind site integrations and publishing deployments to site-scoped content.
+- Optimization tasks attach to content and capture before/after artifacts.
+
+## Cross-Module Interactions
+- Planner and writer share clusters/ideas/tasks/content relationships.
+- Billing models are invoked by services during AI/automation/image/content operations.
+- Integration events reference content IDs and external IDs for sync traces.
+- Publishing records reference writer content; deployment records reference sites.
+- Optimization tasks reference writer content and can influence publishing readiness downstream.
+
+## State Transitions (if applicable)
+- Soft-delete is available on planner keywords/clusters/ideas and writer tasks/content/images via `SoftDeletableModel`.
+- Status fields track lifecycle: planner (`new/mapped/queued/completed`), writer tasks (`queued/completed`), content (`draft/review/published`), automation (`running/paused/cancelled/completed/failed`), publishing/deployment statuses, payment/invoice statuses, optimization statuses.
+
+## Error Handling
+- Save-time validation in `SiteSectorBaseModel` and `Keywords` ensures tenant/site/sector alignment and industry/sector matching.
+- Unique constraints prevent duplicate clusters/keywords per site/sector and overlapping taxonomies/external IDs.
+- Automation runs store error messages and partial stage results; publishing/deployment records store error text.
+
+## Tenancy Rules
+- All models shown are tenant scoped via `AccountBaseModel` or `SiteSectorBaseModel`; save hooks propagate context from related objects where needed.
+- Privileged roles can bypass filtering at the viewset layer, but persisted records retain account/site/sector ownership.
+
+## Billing Rules (if applicable)
+- Credits reside on `Account`; transactions/usage logs record debits/credits; cost configs define per-operation pricing.
+- Invoices/payments/credit packages configure monetary flows; payment methods can be toggled per country or per account.
+
+## Background Tasks / Schedulers (if applicable)
+- Automation configs drive scheduled runs; automation runs record stage outputs and timing.
+- Publishing/optimization tasks may be executed async via services/Celery (see services doc).
+
+## Key Design Considerations
+- Tenant isolation is encoded at the model layer via base classes and validation, ensuring downstream services inherit scoping.
+- Cross-module links (clusters ↔ tasks ↔ content ↔ publishing/optimization) keep content lifecycle traceable.
+- Billing and integration models include metadata fields to avoid schema churn while capturing provider-specific details.
+
+## How Developers Should Work With This Module
+- Inherit new tenant models from `AccountBaseModel` or `SiteSectorBaseModel` to enforce scoping automatically.
+- Validate cross-entity alignment (site/sector/industry) when relating planner and writer records.
+- Use existing status fields/choices when extending lifecycles; preserve unique constraints when adding fields.
+- When integrating new providers, extend or add models parallel to `SiteIntegration`/`SyncEvent` and keep platform-specific data in JSON fields.
+

docs/10-BACKEND/OVERVIEW.md

@@ -0,0 +1,82 @@
+# Backend Architecture
+
+## Purpose
+Explain how the backend is structured, wired, and executed across Django/DRF, middleware, routing, async processing, and logging.
+
+## Code Locations (exact paths)
+- Settings and app wiring: `backend/igny8_core/settings.py`
+- URL routing: `backend/igny8_core/urls.py`
+- Middleware: `backend/igny8_core/middleware/request_id.py`, `backend/igny8_core/auth/middleware.py`, `backend/igny8_core/middleware/resource_tracker.py`
+- Auth stack: `backend/igny8_core/api/authentication.py`, `backend/igny8_core/auth/utils.py`
+- Base viewsets and unified responses: `backend/igny8_core/api/base.py`
+- Domain models: `backend/igny8_core/auth/models.py` plus `backend/igny8_core/business/*/models.py`
+- Async/Celery config: `backend/igny8_core/settings.py`
+- Logging: `backend/igny8_core/settings.py` (publish/webhook logs), automation logging (`backend/igny8_core/business/automation/services/automation_logger.py`)
+
+## High-Level Responsibilities
+- Django/DRF API surface under `/api/v1/*` for planner, writer, system, billing, automation, linker, optimizer, publisher, and integration modules.
+- Middleware establishes request IDs, tenant context, and optional resource tracking before views run.
+- DRF configuration standardizes pagination, filtering, authentication, throttling, and exception handling.
+- Celery + Redis provide async execution for AI, automation, publishing, and other long-running tasks.
+- Logging and CORS/security settings are centralized in settings.
+
+## Detailed Behavior
+- Apps registered in `INSTALLED_APPS` include auth, AI framework, planner, writer, system, billing, automation, optimization, publishing, integration, linker, optimizer, publisher. Custom admin config is loaded first.
+- Middleware order: security → WhiteNoise → CORS → session/common/CSRF → Django auth → `RequestIDMiddleware` → `AccountContextMiddleware` (tenant + plan enforcement) → `ResourceTrackingMiddleware` (opt-in for admin/developer with header) → messages → clickjacking.
+- Routing (`urls.py`): admin plus CSV helpers for industries/seed keywords, then `/api/v1/` routers for auth, account, planner, writer, system, billing (user/admin), automation, linker, optimizer, publisher, integration. OpenAPI served at `/api/docs` and `/api/redoc`.
+- DRF defaults: unified exception handler (unless env disables), custom pagination, filtering/search/ordering, auth stack (API key → JWT → CSRF-exempt session → basic), scoped throttling per operation class, drf-spectacular schema generation with tag ordering.
+- CORS: IGNY8 domains and local dev ports allowed; credentials enabled; request/resource tracking headers exposed.
+- Celery: Redis broker/result; JSON serialization; task/soft time limits; single-prefetch; sentinel/SSL toggles via env.
+- Logging: console plus rotating files for publish/sync, WordPress API, webhooks; request IDs from middleware surface in responses; automation has dedicated file-based logger per run.
+
+## Data Structures / Models Involved (no code)
+- Tenancy/identity: `Account`, `Plan`, `Subscription`, `Site`, `Sector`, `User`, `SiteUserAccess`, `PasswordResetToken` (in auth models).
+- Domain models across planner, writer, billing, automation, publishing, integration, optimization (see domain models doc).
+- Middleware uses request-scoped `request.account`, `request.site`, and `request.request_id`.
+
+## Execution Flow
+1) Request enters middleware stack; IDs and tenant context are set; plan is verified.  
+2) DRF auth classes authenticate user/api-key/JWT/session.  
+3) Base viewsets filter by tenant/site/sector and handle unified responses.  
+4) Serializers validate; database writes enforce tenant alignment via model bases.  
+5) Optional Celery tasks are enqueued for long-running work; responses remain synchronous JSON with pagination/throttle headers when applicable.  
+6) Logging writes to configured handlers; request/resource IDs are added to responses.
+
+## Cross-Module Interactions
+- Tenant context from middleware is consumed by all module viewsets.  
+- AI and automation invoke Celery tasks and AI functions; billing services deduct credits used by automation/AI flows.  
+- Integration/publishing modules rely on API key auth to set `request.site` for WordPress and other platforms.
+
+## State Transitions (if applicable)
+- Account/plan validity is checked per request; system accounts are protected from deletion.  
+- Soft-delete is available on many models via `SoftDeletableModel`.  
+- Request lifecycle includes request ID creation, optional resource tracking, and unified response formatting.
+
+## Error Handling
+- Custom DRF exception handler wraps errors with `success=false`.  
+- `AccountContextMiddleware` blocks requests lacking account or active plan (403/402 JSON).  
+- Base viewsets wrap validation/404/500 errors into unified payloads.
+
+## Tenancy Rules
+- Request-level `account` (and `site`/`sector` where applicable) is injected by middleware/auth; base viewsets enforce filtering.  
+- Admin/developer/system-account users bypass tenant filtering; system accounts are guarded from deletion.
+
+## Billing Rules (if applicable)
+- Billing env keys configured in settings; plan enforcement occurs in middleware; credit debits handled by billing services during operations.
+
+## Background Tasks / Schedulers (if applicable)
+- Celery worker/beat use Redis URLs from settings; task limits and JSON serialization enforced.  
+- Automation, AI, publishing, and optimization tasks run async; automation logger writes per-run files.
+
+## Key Design Considerations
+- Middleware-first tenant enforcement ensures isolation.  
+- Unified API standards (responses, throttles, schema) keep clients consistent.  
+- Celery offloads expensive AI/automation/publishing work with bounded execution.  
+- Logging and request IDs aid observability; resource tracking is opt-in for admins.
+
+## How Developers Should Work With This Module
+- Register new apps in `INSTALLED_APPS` and route them under `/api/v1/*`.  
+- Use existing middleware order; add new middleware only if it does not break account/request ID handling.  
+- Inherit from base viewsets for scoping and response consistency.  
+- Use Celery for long-running tasks; respect Redis/task time-limit settings.  
+- Keep env vars (DB, JWT, Redis, Stripe/PayPal) set per `settings.py`; avoid hardcoded secrets.  

docs/10-BACKEND/SERVICES.md

@@ -0,0 +1,96 @@
+# Services and Modules
+
+## Purpose
+Describe the backend service layer and module wiring that orchestrate domain models, AI/automation, billing, integration, and publishing.
+
+## Code Locations (exact paths)
+- Automation services: `backend/igny8_core/business/automation/services/automation_service.py`, `automation_logger.py`
+- Billing services: `backend/igny8_core/business/billing/services/credit_service.py`, `invoice_service.py`, `payment_service.py`
+- Integration service: `backend/igny8_core/business/integration/services/integration_service.py`
+- Additional service directories (see module-specific docs for details):
+  - Planner: `backend/igny8_core/business/planning/services/*`
+  - Content/Writer: `backend/igny8_core/business/content/services/*`
+  - Automation tasks: `backend/igny8_core/business/automation/tasks.py`
+  - Integration sync: `backend/igny8_core/business/integration/services/*`
+  - Publishing: `backend/igny8_core/business/publishing/services/*`
+  - Optimization: `backend/igny8_core/business/optimization/services/*`
+  - Linking: `backend/igny8_core/business/linking/services/*`
+
+## High-Level Responsibilities
+- Orchestrate multi-stage automation that chains planner and writer operations using AI and credits.
+- Manage credit pricing, balance checks, deductions, and ledger logging.
+- Generate invoices and handle billing documents.
+- Create, update, test, and list site integrations for external platforms.
+- Provide domain-specific service hooks for planner, writer, publishing, linking, and optimization flows (behavior documented in module-specific files).
+
+## Detailed Behavior
+### Automation
+- `AutomationService` enforces single concurrent run per site via cache lock, estimates required credits, checks account balance (with buffer), and creates an `AutomationRun`. It sequences stages (keywords→clusters, clusters→ideas, ideas→tasks/content, image prompt generation, image generation queue) using AI functions (`AutoClusterFunction`, `GenerateIdeasFunction`, `GenerateContentFunction`, `GenerateImagePromptsFunction`) through `AIEngine`, and the Celery image queue (`process_image_generation_queue`). It supports pause/cancel checks mid-stage, records partial progress, and advances `current_stage` with per-stage result JSON and credit tallies. Batch sizes and delays respect `AutomationConfig`.
+- `AutomationLogger` creates per-run directories (and optional shared mirrors), generates run IDs, writes main/stage logs, mirrors to shared folders when configured, and emits structured JSONL trace events for run start, progress, completion, and errors.
+
+### Billing
+- `CreditService` computes credit cost by first consulting `CreditCostConfig` (unit-aware for words/items/images) and falling back to constants. It checks balances, deducts credits atomically, updates account balance, writes `CreditTransaction`, and logs usage in `CreditUsageLog` with optional model/token metadata. It can also add credits and provides legacy check helpers.
+- `InvoiceService` generates invoice numbers per account/month, creates invoices for subscriptions or credit packages (adding line items and computing totals), supports custom invoices, and marks invoices paid or void. PDF generation is currently a placeholder.
+- `PaymentService` (see file) processes payments against invoices and updates statuses; details are documented in the module file.
+
+### Integration
+- `IntegrationService` creates, updates, deletes, fetches, and lists `SiteIntegration` records, setting account/site automatically. It can test connections per platform (WordPress, Shopify) and delegates to platform-specific test helpers inside the service; unimplemented platforms raise `NotImplementedError`. Credentials are set via `set_credentials`, which currently stores JSON as-is.
+- Additional sync services (`content_sync_service.py`, `sync_service.py`, `sync_metadata_service.py`, `sync_health_service.py`) coordinate publish/sync flows and health checks; see module docs for specifics.
+
+### Other Service Areas (structure)
+- Planner services (`clustering_service.py`, `ideas_service.py`) handle clustering/idea logic.
+- Content services (`content_generation_service.py`, `content_pipeline_service.py`, `metadata_mapping_service.py`, `validation_service.py`) manage content generation, pipelines, metadata mapping, and validation.
+- Publishing services (`publisher_service.py`, `deployment_service.py`, `adapters/`) manage publishing/deployment flows and destination adapters.
+- Optimization services (`analyzer.py`, `optimizer_service.py`) analyze and optimize content.
+- Linking services (`candidate_engine.py`, `injection_engine.py`, `linker_service.py`) prepare and apply link suggestions.
+- Automation tasks (`business/automation/tasks.py`) provide Celery entrypoints for automation runs.
+
+## Execution Flow
+- Automation: `AutomationService.start_automation` acquires lock → credit estimate/check → create `AutomationRun` → stage methods query planner/writer models, call AI functions via `AIEngine`, respect batch sizes/delays, and update run state/logs → credits are tallied from `AITaskLog` differences.
+- Billing: operations call `CreditService.check_*`/`deduct_*` before AI or content operations; invoices are created through `InvoiceService` and payments processed via `PaymentService`.
+- Integration: API endpoints invoke `IntegrationService` to persist integrations, retrieve lists, and run connection tests; sync services handle subsequent data movement.
+- Other domains: planner/content/publishing/linking/optimization services orchestrate their models and, where applicable, AI or external adapters; see domain docs for invocation points.
+
+## Cross-Module Interactions
+- Automation stages consume planner (Keywords/Clusters/ContentIdeas) and writer (Tasks/Content/Images) data and rely on credit usage logs from AI tasks.
+- Billing services are used by automation/AI flows to enforce credit availability and record deductions.
+- Integration services connect site data and publishing/sync flows to external platforms; publishing services depend on integration metadata when targeting destinations.
+- Planner/content services feed data used by publishing and optimization tasks.
+
+## State Transitions (if applicable)
+- Automation runs move through stages, can pause/cancel, and record partial progress with stage result JSON.
+- Credit balances mutate through add/deduct operations; transactions/usage logs capture each change.
+- Invoices progress through draft/pending/paid/void and payments through their status lifecycle.
+- Integrations toggle active/sync flags and update sync status/errors.
+
+## Error Handling
+- Automation: checks for concurrent runs; validates minimum keywords; pauses/cancels mid-stage; writes stage error messages; releases locks on failures.
+- Billing: raises when credit cost unknown or balance insufficient; wraps changes in atomic transactions.
+- Integration: platform test errors are logged and returned with `success=false`; unsupported platforms raise `NotImplementedError`.
+- Invoice service prevents voiding paid invoices and returns placeholder PDF until implemented.
+
+## Tenancy Rules
+- Services operate on tenant-scoped models; constructors typically receive account/site or derive them from models. Integration creation sets account from site; credit operations mutate `Account.credits`.
+- Privileged role bypass applies at the viewset layer; persisted records maintain account/site ownership.
+
+## Billing Rules (if applicable)
+- Costs resolved via `CreditCostConfig` (preferred) or constants; units can be per request, words (100/200), item, or image.
+- Deduct operations both adjust `Account.credits` and log to `CreditTransaction` and `CreditUsageLog`.
+- Invoice creation links to subscriptions or credit packages and uses account billing email/plan pricing.
+
+## Background Tasks / Schedulers (if applicable)
+- Automation uses Celery for image generation and may be triggered by scheduled runs (frequency/time in `AutomationConfig`).
+- Other long-running tasks (publishing, optimization, sync) are handled via their Celery tasks/adapters in respective service modules.
+
+## Key Design Considerations
+- Automation enforces exclusivity per site and accounts for credit sufficiency before starting.
+- Logging (AutomationLogger) produces per-run artifacts and structured traces for observability.
+- Credit handling is centralized to keep ledger and usage logs consistent and atomic.
+- Integration services abstract platform handling and allow per-platform test logic.
+
+## How Developers Should Work With This Module
+- Reuse `AutomationService` for running/continuing automation; respect locks and stage APIs.
+- Use `CreditService` before AI/content-heavy operations to check/deduct/add credits and to log usage.
+- Create invoices via `InvoiceService` helpers rather than constructing manually; update invoice/payment status through service methods.
+- Manage integrations through `IntegrationService` (create/update/test/list) and extend platform-specific tests as needed.
+- For domain-specific flows (planner/content/publishing/linking/optimization), place orchestration in the existing service modules and keep tenant context explicit.

docs/10-BACKEND/automation/AUTOMATION-REFERENCE.md

@@ -0,0 +1,91 @@
+# Automation Module Reference
+
+## Purpose
+Document how the automation module orchestrates multi-stage AI pipelines, exposes API endpoints, enforces tenancy/credits, and manages runs, configs, and logging.
+
+## Code Locations (exact paths)
+- Models: `backend/igny8_core/business/automation/models.py`
+- Services: `backend/igny8_core/business/automation/services/automation_service.py`, `automation_logger.py`
+- Tasks (Celery): `backend/igny8_core/business/automation/tasks.py`
+- API views and routing: `backend/igny8_core/business/automation/views.py`, `urls.py`
+- Supporting AI functions: `backend/igny8_core/ai/functions/auto_cluster.py`, `generate_ideas.py`, `generate_content.py`, `generate_image_prompts.py`, image queue in `backend/igny8_core/ai/tasks.py`
+- Tenancy/auth context: `backend/igny8_core/auth/middleware.py`, `backend/igny8_core/api/base.py`
+
+## High-Level Responsibilities
+- Maintain per-site automation configs (batch sizes, delays, schedule, enable flag) and track run state with detailed per-stage results.
+- Provide APIs to configure, trigger, pause/resume/cancel, inspect, and log automation runs.
+- Execute seven sequential stages that transform planner/writer data via AI and local operations, with credit checks and pause/cancel handling.
+- Enforce tenant/site scoping on all automation resources and API operations.
+
+## Detailed Behavior
+- `AutomationConfig` stores enablement, frequency, scheduled time, batch sizes for stages 1–6, and within/between-stage delays. Config is created lazily per site.
+- `AutomationRun` captures run metadata: trigger type (manual/scheduled), status (`running/paused/cancelled/completed/failed`), current stage, pause/cancel timestamps, per-stage JSON results, total credits used, and error message.
+- `AutomationService` orchestrates the pipeline:
+  - Locks per site via cache (`automation_lock_{site.id}`) to prevent concurrent runs.
+  - Estimates credits before start and requires a 20% buffer over the estimate against `Account.credits`.
+  - Creates `AutomationRun` with generated `run_id` and logs start via `AutomationLogger`.
+  - Executes stages in order; each stage logs start/progress/complete, applies within/between-stage delays from config, and writes stage result JSON (counts, credits, timestamps, partial flags).
+  - Pause/cancel checks occur inside loops; state is persisted so resumed runs continue from the recorded stage.
+  - Stage credit usage is derived from AI task logs difference before/after the stage.
+- API layer (`AutomationViewSet`):
+  - `config`/`update_config` read/write `AutomationConfig` for a given `site_id` (scoped to the user’s account).
+  - `run_now` triggers `AutomationService.start_automation` and enqueues Celery `run_automation_task`.
+  - `current_run`, `history`, `logs`, `current_processing`, `estimate`, `pipeline_overview` expose run status, history, logs, credit estimates, and per-stage pending counts.
+  - `pause`, `resume`, `cancel` endpoints update run status and enqueue resume tasks when needed.
+- Celery tasks:
+  - `check_scheduled_automations` scans enabled configs hourly and triggers runs when frequency/time matches and no recent run exists.
+  - `run_automation_task` performs full pipeline execution.
+  - `resume_automation_task`/`continue_automation_task` continue a paused run from its recorded stage.
+
+## Data Structures / Models Involved (no code)
+- `AutomationConfig`, `AutomationRun` (automation state).
+- Planner models: `Keywords`, `Clusters`, `ContentIdeas`.
+- Writer models: `Tasks`, `Content`, `Images`.
+- AI task log (`AITaskLog`) for credit usage measurement.
+- Tenancy entities: `Account`, `Site` (scoping every query).
+
+## Execution Flow
+- API call → DRF auth → tenant/site resolved → viewset method → `AutomationService` operations → Celery task (for long-running execution).
+- Pipeline stages run in-process inside Celery workers, reading planner/writer data, invoking AI functions, updating models, logging progress, and writing stage results to `AutomationRun`.
+- Completion (or failure) updates run status and releases the site lock.
+
+## Cross-Module Interactions
+- Planner/writer models supply inputs and receive outputs (clusters, ideas, tasks, content, images).
+- AI engine executes clustering, idea generation, content generation, and image prompt generation; image rendering uses the AI image queue.
+- Billing credits are checked against `Account.credits`; credit usage is inferred from AI task logs (deduction logic handled in billing services when those AI calls occur).
+- Integration/publishing modules consume content/images produced downstream (outside automation).
+
+## State Transitions
+- Run status moves through `running` → (`paused`/`cancelled`/`failed`/`completed`); `current_stage` increments after each stage finishes; partial flags and timestamps mark mid-stage exits.
+- Config changes take effect on the next run; pause/resume toggles update run timestamps.
+
+## Error Handling
+- Start blocks if a run is already active for the site or cache lock is held.
+- Stage loops log and continue on per-batch/item errors; pause/cancel results are persisted mid-stage.
+- Failures in Celery run mark `AutomationRun` as failed, store error message, timestamp completion, and release the lock.
+- API endpoints return 400 for missing params or invalid state transitions, 404 for unknown runs, 500 on unexpected errors.
+
+## Tenancy Rules
+- All automation queries filter by `site` tied to the authenticated user’s `account`; config/run creation sets `account` and `site` explicitly.
+- API endpoints fetch `Site` with `account=request.user.account`; automation locks are per site.
+- No cross-tenant access; privileged role bypass is handled by DRF auth/permissions upstream.
+
+## Billing Rules
+- Start requires `Account.credits` ≥ 1.2× estimated credits; otherwise a 400 is returned.
+- Credits actually deducted by AI tasks are reflected via AI task logs and billing services (outside this module); automation aggregates usage per stage in `AutomationRun`.
+
+## Background Tasks / Schedulers
+- Hourly `check_scheduled_automations` respects config frequency/time and last run; skips if a run is already active.
+- Pipeline execution and resume steps run inside Celery tasks; within-stage sleeps apply delays from config.
+
+## Key Design Considerations
+- Single-run-per-site enforced via cache lock to prevent overlapping credit use or data contention.
+- Pause/resume/cancel is cooperative, checked inside stage loops, with partial results persisted.
+- Stage-by-stage logging and result JSON make pipeline progress observable and resumable.
+- Configurable batch sizes and delays balance throughput and API/credit usage.
+
+## How Developers Should Work With This Module
+- Use `AutomationService.start_automation` for new runs; never bypass the cache lock or credit check.
+- When extending stages, preserve pause/cancel checks, result recording, and credit delta calculation.
+- Add new API actions through `AutomationViewSet` if they manipulate automation state; keep site/account scoping.
+- For new schedulers, reuse the lock pattern and `AutomationConfig` fields, and update `next_run_at` appropriately.

docs/10-BACKEND/automation/PIPELINE-STAGES.md

@@ -0,0 +1,102 @@
+# Automation Pipeline Stages
+
+## Purpose
+Detail the seven pipeline stages executed by `AutomationService`, including inputs, queries, validations, delays, credit handling, and state recording.
+
+## Code Locations (exact paths)
+- Orchestration: `backend/igny8_core/business/automation/services/automation_service.py`
+- Models: `backend/igny8_core/business/automation/models.py`
+- AI functions: `backend/igny8_core/ai/functions/auto_cluster.py`, `generate_ideas.py`, `generate_content.py`, `generate_image_prompts.py`
+- Image queue: `backend/igny8_core/ai/tasks.py` (`process_image_generation_queue`)
+- Stage entrypoints: `backend/igny8_core/business/automation/tasks.py` (Celery `run_automation_task`, `resume_automation_task`)
+
+## High-Level Responsibilities
+- Execute a fixed seven-stage sequence that moves data from planner keywords through content with images and into manual review readiness.
+- Enforce batch sizes/delays from `AutomationConfig`, support pause/cancel, and write per-stage results into `AutomationRun`.
+- Track credit deltas per stage using AI task log counts.
+
+## Detailed Behavior
+Across all stages:
+- Each stage logs start/progress/complete via `AutomationLogger`, respects `within_stage_delay` between batches/items, and `between_stage_delay` between stages.
+- Pause/cancel is checked inside loops; on pause/cancel, the stage records partial counts, credits, elapsed time, and reason, then exits.
+- Credits used per stage are computed from `AITaskLog` count delta relative to stage start.
+
+### Stage 1: Keywords → Clusters (AI)
+- Input query: `Keywords` where `site=current`, `status='new'`, `cluster__isnull=True`, `disabled=False`.
+- Validation: `validate_minimum_keywords` requires at least 5 keywords; if not valid, stage is skipped with result noting skip reason and `current_stage` advances to 2.
+- Processing: Batch size = `stage_1_batch_size` (capped to total). For each batch, calls `AIEngine.execute(AutoClusterFunction, payload={'ids': batch})`; waits on task ID; logs per-batch progress. Errors are logged and skipped; pipeline continues.
+- Result: counts keywords processed, clusters created since run start, batches, credits used, time elapsed; sets `current_stage=2`.
+
+### Stage 2: Clusters → Ideas (AI)
+- Pre-check: warns if any `Keywords` still pending from Stage 1.
+- Input query: `Clusters` where `site=current`, `status='new'`, `disabled=False`.
+- Processing: Iterates clusters one-by-one; for each, calls `AIEngine.execute(GenerateIdeasFunction, payload={'cluster_id': cluster.id})`; waits on task ID; logs progress. Errors are logged and skipped.
+- Result: counts clusters processed, ideas created since run start, credits used, time elapsed; sets `current_stage=3`.
+
+### Stage 3: Ideas → Tasks (Local)
+- Pre-check: warns if clusters remain without ideas.
+- Input query: `ContentIdeas` where `site=current`, `status='new'`.
+- Processing: Batched by `stage_3_batch_size`. For each idea, builds keyword string (M2M keywords or `target_keywords`) and creates `Tasks` with queued status, copying account/site/sector, cluster, content type/structure, and description. Idea status set to `queued`.
+- Result: ideas processed, tasks created, batches, time elapsed (credits 0 because local); sets `current_stage=4`.
+
+### Stage 4: Tasks → Content (AI)
+- Pre-check: warns if `ContentIdeas` remain `new`.
+- Input query: `Tasks` where `site=current`, `status='queued'`.
+- Processing: Batched by `stage_4_batch_size`. Uses `GenerateContentFunction` via `AIEngine` per batch (payload contains task IDs). Waits on task IDs, logs progress, continues on errors. Tracks total words by summing generated content word_count.
+- Result: tasks processed, content created count, total_words, credits used, time elapsed; sets `current_stage=5`.
+
+### Stage 5: Content → Image Prompts (AI)
+- Input query: `Content` where `site=current`, `status='draft'`, with zero images (annotated count=0).
+- Processing: Batched by `stage_5_batch_size`. For each batch, calls `GenerateImagePromptsFunction` via `AIEngine` (payload content IDs). Waits on task IDs, logs progress; continues on errors.
+- Result: content processed, prompts created (from AI task logs), credits used, time elapsed; sets `current_stage=6`.
+
+### Stage 6: Image Prompts → Images (AI image queue)
+- Input query: `Images` where `site=current`, `status='pending'`.
+- Processing: Iterates pending images; for each, enqueues `process_image_generation_queue.delay(image_ids=[id], account_id, content_id)` when Celery is available, or calls directly in sync fallback. Waits on task IDs with continue-on-error to avoid blocking the stage. Logs progress per image; applies within-stage delay between images.
+- Result: images processed, images generated (status `generated` since run start), content moved to `review`, credits used, time elapsed; sets `current_stage=7`.
+
+### Stage 7: Manual Review Gate (Count-only)
+- Input query: `Content` where `site=current`, `status='review'`.
+- Processing: Counts review-ready content, logs IDs (truncated), marks run `status='completed'`, sets `completed_at`, and releases the site lock.
+- Result: ready_for_review count and content IDs stored in `stage_7_result`.
+
+## Execution Flow
+- Celery task `run_automation_task` instantiates `AutomationService.from_run_id` and calls stages 1→7 sequentially.
+- Stage transitions update `AutomationRun.current_stage`; between-stage delays applied via `between_stage_delay`.
+- Resume path (`resume_automation_task`) starts from the recorded `current_stage` and continues through remaining stages.
+
+## Cross-Module Interactions
+- Planner: Stage 1/2 use `Keywords`/`Clusters`; Stage 3 converts `ContentIdeas` into `Tasks`.
+- Writer: Stages 4–6 create `Content` and `Images` and move content toward review.
+- AI engine and functions are invoked in Stages 1, 2, 4, 5; Stage 6 uses the AI image queue.
+- Billing: Credits are consumed by AI calls; automation records deltas per stage from AI task logs.
+
+## State Transitions
+- `AutomationRun.status` moves to `completed` at Stage 7; can be set to `failed` on exceptions or `cancelled` via API; `paused` can be set mid-run and resumed.
+- `current_stage` increments after each successful stage; partial stage results include a `partial` flag and stop reason.
+- Domain models change status along the pipeline (`Keywords` → clusters, `Clusters` → ideas, `ContentIdeas` → queued/tasks, `Tasks` → completed/content, `Content` → draft/review, `Images` → generated).
+
+## Error Handling
+- Each stage logs errors and continues to next batch/item; pause/cancel checks short-circuit with partial results saved.
+- Task wait helper tolerates Celery backend errors; can continue on error when flagged.
+- Stage start may be skipped with explicit skip reason (e.g., insufficient keywords).
+
+## Tenancy Rules
+- All queries filter by `site` (and implicit account via tenancy bases); account/site set on created `Tasks` and inherited on `Images` and other records through model save hooks.
+- Locks and runs are per site; API scoping requires the authenticated user’s account to own the site.
+
+## Billing Rules
+- Start requires sufficient credits (1.2× estimate). Credits used are inferred from AI task log counts per stage; actual deductions occur in AI/billing services invoked by the AI functions.
+
+## Background Tasks / Schedulers
+- Entire stage chain runs inside Celery workers; within-stage sleeps respect config delays; between-stage sleeps applied after each stage.
+
+## Key Design Considerations
+- Idempotent, resume-capable progression with partial state persisted in `AutomationRun`.
+- Configurable batch sizes/delays mitigate rate limits and manage credit burn.
+- Continue-on-error semantics prevent single failures from stopping the pipeline while still recording issues.
+
+## How Developers Should Work With This Module
+- When modifying stages, keep pause/cancel checks, stage result recording, and credit delta calculation.
+- Add new AI stages by wiring through `AIEngine.execute` and the task wait helper; ensure queries are site-scoped and statuses updated.
+- For new items types, add pending queries and status transitions consistent with existing patterns.

docs/10-BACKEND/automation/SCHEDULER.md

@@ -0,0 +1,75 @@
+# Automation Scheduler
+
+## Purpose
+Describe how scheduled runs are detected, triggered, and resumed using Celery tasks and automation configs.
+
+## Code Locations (exact paths)
+- Celery tasks: `backend/igny8_core/business/automation/tasks.py`
+- Models: `backend/igny8_core/business/automation/models.py`
+- Service invoked: `backend/igny8_core/business/automation/services/automation_service.py`
+
+## High-Level Responsibilities
+- Periodically scan enabled automation configs to start scheduled runs.
+- Prevent overlapping runs per site via cache locks and active run checks.
+- Resume paused runs from their recorded stage.
+
+## Detailed Behavior
+- `check_scheduled_automations` (Celery, hourly):
+  - Iterates `AutomationConfig` with `is_enabled=True`.
+  - Frequency rules:
+    - `daily`: run when current hour matches `scheduled_time.hour`.
+    - `weekly`: run Mondays at the scheduled hour.
+    - `monthly`: run on the 1st of the month at the scheduled hour.
+  - Skips if `last_run_at` is within ~23 hours or if an `AutomationRun` with `status='running'` exists for the site.
+  - On trigger: instantiates `AutomationService(account, site)`, calls `start_automation(trigger_type='scheduled')`, updates `last_run_at` and `next_run_at` (via `_calculate_next_run`), saves config, and enqueues `run_automation_task.delay(run_id)`.
+  - Exceptions are logged per site; lock release is handled by the service on failure paths.
+- `run_automation_task`:
+  - Loads service via `from_run_id`, runs stages 1–7 sequentially.
+  - On exception: marks run failed, records error/completed_at, and deletes site lock.
+- `resume_automation_task` / alias `continue_automation_task`:
+  - Loads service via `from_run_id`, uses `current_stage` to continue remaining stages.
+  - On exception: marks run failed, records error/completed_at.
+- `_calculate_next_run`:
+  - Computes next run datetime based on frequency and `scheduled_time`, resetting seconds/microseconds; handles month rollover for monthly frequency.
+
+## Data Structures / Models Involved (no code)
+- `AutomationConfig`: contains schedule fields (`frequency`, `scheduled_time`, `last_run_at`, `next_run_at`, `is_enabled`).
+- `AutomationRun`: records run status/stage used during resume/failure handling.
+
+## Execution Flow
+1) Celery beat (or cron) invokes `check_scheduled_automations` hourly.  
+2) Eligible configs spawn new runs via `AutomationService.start_automation` (includes lock + credit check).  
+3) `run_automation_task` executes the pipeline asynchronously.  
+4) Paused runs can be resumed by enqueueing `resume_automation_task`/`continue_automation_task`, which restart at `current_stage`.  
+5) Failures set run status to `failed` and release locks.
+
+## Cross-Module Interactions
+- Uses planner/writer data inside the pipeline (see pipeline doc); billing/credits enforced at start.
+- Locking is done via Django cache, independent of other modules but prevents concurrent Celery runs per site.
+
+## State Transitions
+- Config timestamps (`last_run_at`, `next_run_at`) update on scheduled launch.
+- Run status changes to `failed` on task exceptions; to `completed` at stage 7; to `paused/cancelled` via API.
+
+## Error Handling
+- Scheduled start is skipped with log messages if recently run or already running.
+- Exceptions during run execution mark the run failed, record error message, set `completed_at`, and release the cache lock.
+
+## Tenancy Rules
+- Configs and runs are site- and account-scoped; scheduler uses stored account/site from the config; no cross-tenant scheduling.
+
+## Billing Rules
+- Start uses `AutomationService.start_automation`, which enforces credit sufficiency before scheduling the Celery execution.
+
+## Background Tasks / Schedulers
+- Hourly `check_scheduled_automations` plus the long-running `run_automation_task` and resume tasks run in Celery workers.
+
+## Key Design Considerations
+- Hourly scan with coarse matching keeps implementation simple while honoring per-site schedules.
+- Cache lock and active-run checks prevent double-starts from overlapping schedules or manual triggers.
+- Resume task reuses the same stage methods to keep behavior consistent between fresh and resumed runs.
+
+## How Developers Should Work With This Module
+- When adding new frequencies, extend `check_scheduled_automations` and `_calculate_next_run` consistently.
+- Ensure Celery beat (or an equivalent scheduler) runs `check_scheduled_automations` hourly in production.
+- Preserve lock acquisition and failure handling when modifying task flows to avoid orphaned locks.

docs/10-BACKEND/billing/BILLING-REFERENCE.md

@@ -0,0 +1,98 @@
+# Billing Module Reference
+
+## Purpose
+Explain how billing is implemented: invoices, payments, credit packages, credit history/usage, payment methods, and limits, as exposed by billing services and API viewsets.
+
+## Code Locations (exact paths)
+- Models: `backend/igny8_core/business/billing/models.py`
+- Services: `backend/igny8_core/business/billing/services/credit_service.py`, `invoice_service.py`, `payment_service.py`
+- API (business billing): `backend/igny8_core/business/billing/views.py`, `backend/igny8_core/business/billing/urls.py`
+- API (core billing endpoints): `backend/igny8_core/modules/billing/views.py`, `backend/igny8_core/modules/billing/urls.py`
+- Plan metadata (included credits/limits): `backend/igny8_core/auth/models.py` (`Plan`, `Account`)
+
+## High-Level Responsibilities
+- Maintain credit ledger, usage logs, configurable per-operation costs, invoices, payments, credit packages, country-level payment method configs, and account-scoped payment methods.
+- Expose endpoints for invoices/payments/credit packages/transactions, credit balance/usage/limits, and admin billing operations.
+- Deduct or add credits atomically while recording both transactions and usage logs.
+- Provide manual and (placeholder) Stripe/PayPal payment flows.
+
+## Detailed Behavior
+- Invoices (`InvoiceViewSet`):
+  - `list`/`retrieve`: fetch account-scoped invoices; include totals, line items, billing period, dates.
+  - `download_pdf`: returns PDF bytes from `InvoiceService.generate_pdf` (currently placeholder text payload).
+- Payments (`PaymentViewSet`):
+  - `list`: account-scoped payments with amounts, methods, status, invoice linkage, timestamps.
+  - `available_methods`: delegates to `PaymentService.get_available_payment_methods` (country/config-driven; returns methods plus metadata).
+  - `create_manual_payment`: creates a manual payment for an invoice (bank/local wallet); requires invoice_id, method, reference; rejects already-paid invoices; returns pending-approval status.
+- Account payment methods (`AccountPaymentMethodViewSet`):
+  - CRUD for account-level payment metadata (non-sensitive). `perform_create`/`perform_update` enforce one default by resetting others when `is_default` true or none exists.
+  - `set_default` toggles default within the account.
+- Credit packages (`CreditPackageViewSet`):
+  - `list`: active packages with credits/price/discount/featured/sort order.
+  - `purchase`: creates invoice via `InvoiceService.create_credit_package_invoice`; returns next action depending on requested payment method (`stripe`/`paypal` placeholders, manual fallback returns invoice info).
+- Credit transactions (`CreditTransactionViewSet` in business billing views; also registered under `credits/transactions` in URLs): lists credit ledger entries per account.
+- Core billing endpoints (`modules/billing/views.py`):
+  - `CreditBalanceViewSet.list`: returns current credits, plan monthly credits, credits used this month.
+  - `CreditUsageViewSet`: paginated credit usage logs with filters (operation_type, date range); `summary` aggregates by operation/model and totals (credits, USD); `limits` returns plan/account limits and usage (users/sites/etc.) based on `Plan` fields.
+- Routing (`business/billing/urls.py`):
+  - Routers for invoices, payments, credit packages, transactions, payment methods, and canonical credit endpoints (balance/usage/transactions). Legacy available-methods endpoint exposed at `/payment-methods/available/`.
+
+## Data Structures / Models Involved (no code)
+- `CreditTransaction`: ledger with type, amount (positive/negative), balance_after, metadata.
+- `CreditUsageLog`: per-AI-operation usage with operation_type, credits_used, cost_usd, model_used, tokens, related object references, metadata.
+- `CreditCostConfig`: per-operation cost config with unit, display metadata, active flag, audit of previous cost.
+- `Invoice`: invoice_number, status, amounts (subtotal/tax/total), currency, billing_period, line_items JSON, subscription link, payment metadata, timestamps.
+- `Payment`: status lifecycle, method (stripe/paypal/bank/local wallet/manual), provider references, manual notes/approval, failure reason, metadata.
+- `CreditPackage`, `PaymentMethodConfig` (country/method availability + instructions/bank/local wallet data), `AccountPaymentMethod` (account-scoped payment metadata).
+- Plan/account credits and limits: `Plan.included_credits`, `Plan.max_users/sites/industries/author_profiles`, `Account.credits`.
+
+## Execution Flow
+- Credits:
+  - Operations call `CreditService.get_credit_cost` → `check_credits`/`deduct_credits` or `deduct_credits_for_operation` → updates `Account.credits`, writes `CreditTransaction` and `CreditUsageLog`.
+  - Costs resolved from `CreditCostConfig` when active; otherwise constants. Units support per request, per 100/200 words, per item/image/idea.
+- Invoicing/Payments:
+  - Credit package purchase → invoice creation → next-action response (manual vs pending Stripe/PayPal integration).
+  - Manual payment submission → `PaymentService.create_manual_payment` persists pending approval.
+- Credit balance/usage:
+  - Balance endpoint computes plan credits and month-to-date usage (from `CreditUsageLog`).
+  - Usage endpoint filters logs; summary aggregates by operation/model and totals; limits endpoint computes plan-based limits and usage (users/sites/etc.) per account.
+
+## Cross-Module Interactions
+- AI/automation/planner/writer flows trigger credit checks/deductions through `CreditService`; usage logs can be filtered by operation_type (clustering, idea_generation, content_generation, image_generation, optimization, reparse).
+- Plan limits surfaced in billing limits endpoint affect account/user/site management elsewhere.
+- Invoice/payment metadata may be used by frontend billing UI; manual payments require admin follow-up.
+
+## State Transitions
+- Credits mutate `Account.credits`; ledger captures each change; usage logs accumulate per operation.
+- Invoices move through statuses (`draft/pending/paid/void/uncollectible`); payments through (`pending/pending_approval/processing/succeeded/completed/failed/refunded/cancelled`).
+- Credit packages and payment methods have active/default flags; cost configs can be toggled active/inactive.
+
+## Error Handling
+- Insufficient credits raise `InsufficientCreditsError`; API returns 402 when caught (e.g., content generation, credit deduct flows).
+- Manual payment rejects missing fields or already-paid invoices.
+- Usage/balance endpoints return empty data when account/plan missing instead of hard errors.
+- Validation errors on site/sector/account scoping propagate from base viewsets and model constraints.
+
+## Tenancy Rules
+- Account scoping enforced via `AccountModelViewSet` or explicit account filtering in viewsets; all billing data is per-account.
+- Payment methods, invoices, payments, transactions, usage logs, and balances are filtered to `request.user.account` (or `request.account` from middleware).
+- Plan data comes from the account’s associated plan; no cross-tenant visibility.
+
+## Billing Rules
+- Credit costs configurable via `CreditCostConfig`; fallback constants apply when no config.
+- Credit deductions and ledger entries are atomic; usage logs record the same operation with optional model/token metadata.
+- Balance endpoint includes plan monthly credits; limits endpoint reports plan/user/site limits but does not enforce them here.
+
+## Background Tasks / Schedulers
+- None specific to billing in this module; Stripe/PayPal integrations are placeholders. Any future webhooks should persist to these models and adjust credits/invoices/payments accordingly.
+
+## Key Design Considerations
+- Credit handling is centralized in `CreditService` to keep ledger and usage logs consistent.
+- Usage/balance endpoints are read-mostly, with graceful handling when plan/account data is missing.
+- Manual payments support bank/local wallet flows without storing sensitive data; account payment methods store display metadata only.
+
+## How Developers Should Work With This Module
+- Use `CreditService` for all credit checks/deductions/additions; do not mutate `Account.credits` directly.
+- When adding new AI operations, add `operation_type` to `CreditUsageLog.OPERATION_TYPE_CHOICES` and `CreditCostConfig` seed/constants, then use `deduct_credits_for_operation`.
+- Extend payment flows by implementing Stripe/PayPal handlers in `PaymentService` and wiring webhooks to update `Payment`/`Invoice`.
+- Use existing serializers/viewsets when exposing billing data; keep queries scoped to `request.account`.

docs/10-BACKEND/billing/CREDITS-SYSTEM.md

@@ -0,0 +1,80 @@
+# Credit System
+
+## Purpose
+Detail how credits are priced, checked, deducted, logged, and reported across the platform.
+
+## Code Locations (exact paths)
+- Credit logic: `backend/igny8_core/business/billing/services/credit_service.py`
+- Credit costs config: `backend/igny8_core/business/billing/models.py` (`CreditCostConfig`)
+- Ledger and usage logs: `backend/igny8_core/business/billing/models.py` (`CreditTransaction`, `CreditUsageLog`)
+- API endpoints: `backend/igny8_core/modules/billing/views.py` (`CreditBalanceViewSet`, `CreditUsageViewSet`)
+- Constants and exceptions: `backend/igny8_core/business/billing/constants.py`, `exceptions.py`
+- Plan included credits: `backend/igny8_core/auth/models.py` (`Plan.included_credits`, `Account.credits`)
+
+## High-Level Responsibilities
+- Compute operation costs (configurable per operation and unit) and enforce sufficient balance before AI/content actions.
+- Deduct or add credits atomically while writing both transaction and usage log records.
+- Provide balance, usage history, summaries, and limits endpoints for clients.
+- Allow admin-configurable cost overrides via `CreditCostConfig`.
+
+## Detailed Behavior
+- Cost resolution (`CreditService.get_credit_cost`):
+  - First checks `CreditCostConfig` for the operation (active records). Supports units: `per_request`, `per_100_words`, `per_200_words`, `per_item`, `per_image`. Applies amounts when provided (e.g., words, items, images).
+  - Falls back to hardcoded `CREDIT_COSTS` constants; raises `CreditCalculationError` for unknown operations.
+  - Legacy variable costs: content_generation per 100 words, optimization per 200 words, image_generation per image, idea_generation per idea.
+- Balance enforcement:
+  - `check_credits`/`check_credits_legacy` raise `InsufficientCreditsError` when balance insufficient.
+  - `deduct_credits_for_operation` computes cost, checks balance, builds a default description per operation when not supplied, then calls `deduct_credits`.
+- Deduction (`deduct_credits`):
+  - Atomic: decrements `Account.credits`, writes `CreditTransaction` with negative amount and balance_after, writes `CreditUsageLog` capturing operation_type, cost_usd/model/tokens/related object references, metadata.
+- Add credits (`add_credits`):
+  - Atomic: increments `Account.credits`, writes `CreditTransaction` with positive amount and balance_after.
+- Logging:
+  - `CreditTransaction` is the authoritative ledger; `CreditUsageLog` tracks per-operation usage for analytics and summaries.
+  - `CreditCostConfig` tracks `previous_cost` on save when cost changes.
+- Reporting endpoints (modules/billing):
+  - `CreditBalanceViewSet.list`: returns current credits, plan monthly credits, credits used this month (from `CreditUsageLog`), and remaining credits.
+  - `CreditUsageViewSet.list`: paginated usage logs filtered by operation_type and date range, account-scoped.
+  - `CreditUsageViewSet.summary`: aggregates credits and USD cost by operation and model over a date range (defaults to current month).
+  - `CreditUsageViewSet.limits`: returns plan limits (users/sites/industries/author profiles) and current usage counts, plus credits info; returns empty limits if account/plan missing.
+
+## Data Structures / Models Involved (no code)
+- `CreditCostConfig`: operation_type, credits_cost, unit, display_name, description, is_active, previous_cost, updated_by, timestamps.
+- `CreditTransaction`: transaction_type (purchase/subscription/refund/deduction/adjustment), amount (+/-), balance_after, description, metadata, reference_id, created_at.
+- `CreditUsageLog`: operation_type (clustering/idea_generation/content_generation/image_generation/reparse/legacy names), credits_used, cost_usd, model_used, tokens_in/out, related object type/id, metadata, created_at.
+- `Account.credits`: current balance; `Plan.included_credits`: monthly included credits.
+
+## Execution Flow
+- Caller requests an operation → `get_credit_cost` computes cost → `check_credits` ensures balance → `deduct_credits` mutates balance and writes ledger + usage → upstream operation proceeds (AI/content).
+- Balance/usage endpoints read from `Account`, `CreditUsageLog`, `Plan` to present current balances, month-to-date usage, summaries, and limits.
+
+## Cross-Module Interactions
+- AI/automation/planner/writer services call `CreditService` before expensive operations; automation aggregates credits used per stage from AI task logs, while billing logs capture actual deductions at AI call sites.
+- Plan limits (users/sites/industries/authors) referenced in billing limits endpoint inform account management elsewhere.
+
+## State Transitions
+- Balance changes recorded per transaction; usage accumulates per operation. Cost configs can be toggled active/inactive and updated with audit of previous cost.
+
+## Error Handling
+- Unknown operation types raise `CreditCalculationError`; insufficient balance raises `InsufficientCreditsError` (mapped to HTTP 402 in callers).
+- Balance/usage endpoints fall back to zeros/empty when account/plan missing rather than erroring.
+
+## Tenancy Rules
+- All credit operations and queries are account-scoped; viewsets filter by `request.account`/`request.user.account`. No cross-tenant access.
+
+## Billing Rules
+- Costs derived from `CreditCostConfig` > constants; units must match supplied amount. Ledger and usage logs are mandatory for every deduction/addition.
+
+## Background Tasks / Schedulers
+- None dedicated; credit usage often originates from AI Celery tasks but logging happens at the service call site.
+
+## Key Design Considerations
+- Centralized credit math and atomic DB updates prevent drift between balance, ledger, and usage logs.
+- Configurable costs allow runtime tuning without code changes.
+- Reporting endpoints are read-only and tolerant of missing plan data to keep dashboards resilient.
+
+## How Developers Should Work With This Module
+- Always route credit checks/deductions/additions through `CreditService`.
+- When adding new AI operations, register operation_type in `CreditUsageLog`/constants and optionally seed `CreditCostConfig`.
+- Include amount (words/items/images) when calling `deduct_credits_for_operation` so unit-based pricing applies.
+- Use balance/usage endpoints for UI/analytics; avoid custom queries that bypass account scoping.

docs/10-BACKEND/planner/PLANNER-REFERENCE.md

@@ -0,0 +1,80 @@
+# Planner Overview
+
+## Purpose
+Explain how the planner module manages keywords, clusters, and content ideas, including tenancy, validation, and API behaviors.
+
+## Code Locations (exact paths)
+- Models: `backend/igny8_core/business/planning/models.py`
+- Serializers: `backend/igny8_core/modules/planner/serializers.py`
+- Views: `backend/igny8_core/modules/planner/views.py`
+- URLs: `backend/igny8_core/modules/planner/urls.py`
+- Services: `backend/igny8_core/business/planning/services/clustering_service.py`, `ideas_service.py`
+
+## High-Level Responsibilities
+- Store site/sector-scoped keywords, clusters, and content ideas linked to global seed keywords.
+- Provide CRUD and bulk operations for keywords/clusters/ideas with search, filter, and ordering.
+- Validate industry/sector alignment between seed keywords and site/sector.
+- Feed downstream writer/automation stages with structured ideas and clusters.
+
+## Detailed Behavior
+- Models (site/sector scoped via `SiteSectorBaseModel` and soft-delete):
+  - `Keywords`: references global `SeedKeyword`; optional volume/difficulty overrides; optional cluster; status `new/mapped`; disabled flag; validation ensures seed keyword industry/sector match the site/sector.
+  - `Clusters`: topic groups with counts, volume, mapped_pages, status `new/mapped`; unique per site/sector by name.
+  - `ContentIdeas`: ideas tied to clusters and optional keyword objects; status `new/queued/completed`; content_type/structure and estimated_word_count; disabled flag.
+- Serializers:
+  - `KeywordSerializer` enforces `seed_keyword_id` on create; exposes seed keyword-derived fields (keyword/volume/difficulty/intent) read-only; optional overrides; includes site_id/sector_id write-only; provides cluster/sector names via getters; optional `attribute_values` when `USE_SITE_BUILDER_REFACTOR` flag is enabled.
+  - `ClusterSerializer` exposes site/sector IDs, sector name, read-only counts/volume/mapped_pages.
+  - `ContentIdeasSerializer` exposes cluster/sector names, content type/structure, keyword/target fields, site/sector write-only.
+- Views (`KeywordViewSet`, `ClusterViewSet`, `ContentIdeasViewSet`):
+  - Inherit `SiteSectorModelViewSet` for tenant/site/sector filtering and unified responses.
+  - Permissions: `IsAuthenticatedAndActive` + viewer-or-above (search/list CRUD).
+  - Pagination: `CustomPageNumberPagination`; throttle scope `planner`.
+  - Filters/Search/Ordering: keywords searchable by seed_keyword.keyword; filter by status, cluster_id, seed intent; ordering by created_at, volume, difficulty; custom range filters for difficulty/volume; clusters filter by status; ideas filter by status/cluster.
+  - Creation requires explicit `site_id` and `sector_id`; viewsets validate site/sector existence and alignment; set `account/site/sector` explicitly on save.
+  - Bulk endpoints:
+    - Keywords: `bulk_delete`, two definitions of `bulk_update` (status update) present; `bulk_add_from_seed` to create Keywords from SeedKeywords with site/sector validation.
+    - Clusters: bulk update endpoints in view (not shown above) adjust status.
+    - Ideas: creation and status updates follow standard CRUD; not shown as bulk endpoints in code snippet.
+  - Error handling: viewsets wrap errors into unified responses; return 400/404/500 as appropriate.
+- Services:
+  - `ClusteringService` and `IdeasService` (not detailed here) provide additional business logic for clustering/idea generation when invoked by automation or endpoints.
+
+## Data Structures / Models Involved (no code)
+- `Keywords`, `Clusters`, `ContentIdeas`; global `SeedKeyword`.
+- Tenancy entities: `Account`, `Site`, `Sector`.
+
+## Execution Flow
+- Requests → DRF auth → `SiteSectorModelViewSet` filters by account/site/sector → serializers validate seed/site/sector alignment → models persisted → responses returned via unified helpers; bulk actions operate on filtered querysets.
+
+## Cross-Module Interactions
+- Automation stages 1–3 consume planner data (keywords → clusters → ideas → tasks).
+- Writer tasks reference clusters/ideas produced here.
+- Billing credits may be checked upstream for AI clustering/idea generation (via services invoked in automation or planner actions).
+
+## State Transitions
+- Keywords: `new` → `mapped`; Clusters: `new` → `mapped`; ContentIdeas: `new` → `queued` → `completed`.
+- Disabled flag can exclude records from processes.
+
+## Error Handling
+- Validation enforces seed/site/sector industry alignment; missing site/sector on create raises validation errors.
+- Bulk operations return 400 on missing IDs/status; general errors return unified 500 responses.
+
+## Tenancy Rules
+- All planner models inherit `SiteSectorBaseModel`; queries are filtered by account/site/sector; admin/developer/system roles can bypass filtering via base viewset logic.
+- Create requires explicit site and sector; viewsets fetch and set account from site/user/middleware.
+
+## Billing Rules
+- None directly in the planner endpoints; credit costs for clustering/idea generation are enforced where those services are called (automation or planner service usage).
+
+## Background Tasks / Schedulers
+- Planner endpoints run synchronously; automation may batch planner data into AI tasks via Celery (documented in automation).
+
+## Key Design Considerations
+- Strict site/sector validation prevents cross-tenant or cross-site leaks.
+- Seed keyword linkage centralizes keyword metadata; overrides allow per-site tuning.
+- Pagination/filtering/search are optimized for large keyword sets.
+
+## How Developers Should Work With This Module
+- Use provided viewsets and serializers; ensure site_id/sector_id are supplied on create.
+- When adding fields, extend serializers with read-only/write-only behavior consistent with current patterns.
+- For new bulk operations, follow existing patterns: validate IDs, operate on filtered queryset, return unified responses.

docs/10-BACKEND/writer/WRITER-REFERENCE.md

@@ -0,0 +1,82 @@
+# Writer Overview
+
+## Purpose
+Describe how the writer module manages tasks, content, images, and taxonomies, including validations, tenancy, and API behaviors.
+
+## Code Locations (exact paths)
+- Models: `backend/igny8_core/business/content/models.py`
+- Serializers: `backend/igny8_core/modules/writer/serializers.py`
+- Views: `backend/igny8_core/modules/writer/views.py`
+- URLs: `backend/igny8_core/modules/writer/urls.py`
+- Services: `backend/igny8_core/business/content/services/content_generation_service.py`, `validation_service.py`, `metadata_mapping_service.py`
+- Automation linkages: `backend/igny8_core/business/automation/services/automation_service.py` (stages 3–6)
+
+## High-Level Responsibilities
+- Manage tenant/site/sector-scoped tasks, generated/imported content, images, and taxonomies.
+- Provide CRUD and bulk operations for tasks and images; content generation triggers; taxonomy CRUD.
+- Enforce site/sector alignment on creation and propagate account/site/sector to related records.
+- Feed automation and publishing flows with content and images, and integrate with billing credits for AI operations.
+
+## Detailed Behavior
+- Models (all inherit tenant/site/sector bases; many soft-deletable):
+  - `Tasks`: queued work items tied to clusters/ideas/taxonomy; content type/structure; keywords text; word_count target; status `queued/completed`.
+  - `Content`: generated/imported content with HTML, SEO fields, cluster link, content type/structure, taxonomy M2M, external IDs/URLs/metadata, sync status, source (`igny8/wordpress`), status (`draft/review/published`), word_count, timestamps.
+  - `Images`: linked to content or task; auto-sets account/site/sector from provider; tracks type, URL/path, prompt, status, position.
+  - `ContentTaxonomy`: site/sector taxonomy terms (category/tag) with external taxonomy/id, sync status, description, count, metadata; unique per site by slug+type and by external id+taxonomy.
+  - Supporting: `ContentClusterMap`, `ContentAttribute` (documented in domain models).
+- Serializers:
+  - `TasksSerializer`: requires cluster/content_type/content_structure on create; exposes cluster/sector names; write-only site_id/sector_id; defaults status to queued if absent.
+  - `ImagesSerializer`: exposes task/content titles; read-only derived fields; creation requires site/sector (validated, with fallback to request user active site default sector); sets account/site/sector via base perform_create.
+  - `ContentSerializer`: exposes cluster/sector names; taxonomy terms grouped (tags/categories); image status flags; write-only site_id/sector_id; validates presence of required fields and status transitions.
+  - Group serializers for images support grouped responses by content.
+- Views:
+  - `TasksViewSet`: filters/search/order; bulk_delete, bulk_update status; `auto_generate_content` triggers `ContentGenerationService.generate_content` for up to 10 tasks, enforcing account presence and credit checks (returns 402 on `InsufficientCreditsError`); inherits tenant/site/sector filtering.
+  - `ImagesViewSet`: CRUD with ordering/filtering; perform_create enforces site/sector and populates context; `serve_image_file` serves local files with basic checks.
+  - `ContentViewSet`: CRUD for content; ordering/filtering; handles taxonomy assignments; exposes status/SEO fields; uses base scoping.
+  - `ContentTaxonomyViewSet`: CRUD for taxonomy terms (category/tag) scoped to site/sector; inherits base scoping.
+- Services:
+  - `ContentGenerationService`: invoked by `auto_generate_content`; orchestrates AI content generation for tasks, returns async task_id when queued or synchronous result; raises `InsufficientCreditsError` on low balance.
+  - `ContentValidationService`, `MetadataMappingService`: additional processing/validation/mapping (used in writer flows and publishing).
+- Automation linkage:
+  - Stage 3 creates tasks from ideas; Stage 4 generates content; Stage 5 generates image prompts; Stage 6 generates images and moves content to review.
+
+## Data Structures / Models Involved (no code)
+- `Tasks`, `Content`, `Images`, `ContentTaxonomy`, plus `ContentClusterMap`/`ContentAttribute`.
+- Tenancy entities: `Account`, `Site`, `Sector`.
+
+## Execution Flow
+- Requests → DRF auth → `SiteSectorModelViewSet` filtering → serializer validation (site/sector required on create) → model save → unified response.
+- `auto_generate_content` validates task IDs/account, calls service → service may enqueue Celery AI tasks or run synchronously → response with task_id or result.
+- Images creation auto-inherits account/site/sector from linked task/content; local file serving bypasses account filter for read-only access.
+
+## Cross-Module Interactions
+- Planner clusters/ideas feed tasks; tasks/content/images feed publishing and automation stages.
+- Billing credits enforced via `ContentGenerationService` when AI is used; automation aggregates credits used per stage from AI task logs.
+- Integration/publishing sync uses content/taxonomy/image data when pushing to external platforms.
+
+## State Transitions
+- Tasks: `queued` → `completed`; Content: `draft` → `review` → `published`; Images: `pending` → `generated`; Taxonomies created/updated/deleted via CRUD.
+
+## Error Handling
+- Viewsets return unified errors; bulk operations validate IDs/status; `auto_generate_content` returns 400/404/402/500 depending on validation, missing tasks, insufficient credits, or unexpected errors.
+- Images file serving returns 404 on missing file/image.
+
+## Tenancy Rules
+- Base viewset filters enforce account/site/sector scoping; admins/developers/system can bypass filtering; persisted records retain tenant/site/sector ownership.
+- Creation requires site/sector; account set from request/site/user; Images perform_create validates presence of site/sector.
+
+## Billing Rules
+- AI content generation is guarded by credit checks; InsufficientCredits returns HTTP 402; credit logging occurs in billing services at AI call sites.
+
+## Background Tasks / Schedulers
+- AI generation may run via Celery when `ContentGenerationService` returns a task_id; automation stages run inside Celery workers for bulk flows.
+
+## Key Design Considerations
+- Strict site/sector requirements prevent cross-site contamination.
+- Bulk limits (max 10 tasks in auto_generate_content) to control credit burn and processing load.
+- Image serving permits public access via endpoint while still performing existence checks.
+
+## How Developers Should Work With This Module
+- Use existing serializers/viewsets; keep site/sector required on create.
+- When extending AI generation, ensure billing checks (`CreditService`) and account/site scoping are preserved.
+- For new bulk actions, follow current patterns: validate input, operate on filtered queryset, return unified responses.