docs re-org

This commit is contained in:
IGNY8 VPS (Salman)
2025-12-09 13:26:35 +00:00
parent 4d13a57068
commit 6a4f95c35a
231 changed files with 11353 additions and 31152 deletions

View File

@@ -0,0 +1,148 @@
# Automation Module (Code-Sourced, Dec 2025)
Single canonical reference for IGNY8 automation (backend, frontend, and runtime behavior). Replaces all prior automation docs in this folder.
---
## 1) What Automation Does
- Runs the 7-stage pipeline across Planner/Writer:
1) Keywords → Clusters (AI)
2) Clusters → Ideas (AI)
3) Ideas → Tasks (Local)
4) Tasks → Content (AI)
5) Content → Image Prompts (AI)
6) Image Prompts → Images (AI)
7) Manual Review Gate (Manual)
- Per-site, per-account isolation. One run at a time per site; guarded by cache lock `automation_lock_{site_id}`.
- Scheduling via Celery beat (`automation.check_scheduled_automations`); execution via Celery tasks (`run_automation_task`, `resume_automation_task` / `continue_automation_task`).
---
## 2) Backend API (behavior + payloads)
Base: `/api/v1/automation/` (auth required; site must belong to users account).
- `GET config?site_id=`: returns or creates config with enable flag, frequency (`daily|weekly|monthly`), scheduled_time, stage_1..6 batch sizes, delays (`within_stage_delay`, `between_stage_delay`), last_run_at, next_run_at.
- `PUT update_config?site_id=`: same fields as above, updates in-place.
- `POST run_now?site_id=`: starts a manual run; enqueues `run_automation_task`. Fails if a run is already active or lock exists.
- `GET current_run?site_id=`: current running/paused run with status, current_stage, totals, and stage_1..7_result blobs (counts, credits, partial flags, skip reasons).
- `GET pipeline_overview?site_id=`: per-stage status counts and “pending” numbers for UI cards.
- `GET current_processing?site_id=&run_id=`: live processing snapshot for an active run; null if not running.
- `POST pause|resume|cancel?site_id=&run_id=`: pause after current item; resume from saved `current_stage`; cancel after current item and stamp cancelled_at/completed_at.
- `GET history?site_id=`: last 20 runs (id, status, trigger, timestamps, total_credits_used, current_stage).
- `GET logs?run_id=&lines=100`: tail of the per-run activity log written by AutomationLogger.
- `GET estimate?site_id=`: estimated_credits, current_balance, sufficient (balance >= 1.2x estimate).
Error behaviors:
- Missing site_id/run_id → 400.
- Site not in account → 404.
- Run not found → 404 on run-specific endpoints.
- Already running / lock held → 400 on run_now.
---
## 3) Data Model (runtime state)
- `AutomationConfig` (one per site): enable flag, schedule (frequency, time), batch sizes per stage (16), delays (within-stage, between-stage), last_run_at, next_run_at.
- `AutomationRun`: run_id, trigger_type (manual/scheduled), status (running/paused/cancelled/completed/failed), current_stage, timestamps (start/pause/resume/cancel/complete), total_credits_used, per-stage result JSON (stage_1_result … stage_7_result), error_message.
- Activity logs: one file per run via AutomationLogger; streamed through the `logs` endpoint.
---
## 4) How Execution Works (AutomationService)
- Start: grabs cache lock `automation_lock_{site_id}`, estimates credits, enforces 1.2x balance check, creates AutomationRun and log file.
- AI functions used: Stage 1 `AutoClusterFunction`; Stage 2 `GenerateIdeasFunction`; Stage 4 `GenerateContentFunction`; Stage 5 `GenerateImagePromptsFunction`; Stage 6 uses `process_image_generation_queue` (not the partial `generate_images` AI function).
- Stage flow (per code):
- Stage 1 Keywords → Clusters: require ≥5 keywords (validate_minimum_keywords); batch by config; AIEngine clustering; records keywords_processed, clusters_created, batches, credits, time; skips if insufficient keywords.
- Stage 2 Clusters → Ideas: batch by config; AIEngine ideas; records ideas_created.
- Stage 3 Ideas → Tasks: local conversion of queued ideas to tasks; batches by config; no AI.
- Stage 4 Tasks → Content: batch by config; AIEngine content; records content count + word totals.
- Stage 5 Content → Image Prompts: batch by config; AIEngine image-prompts into Images (featured + in-article).
- Stage 6 Image Prompts → Images: uses `process_image_generation_queue` with provider/model from IntegrationSettings; updates Images status.
- Stage 7 Manual Review Gate: marks ready-for-review counts; no AI.
- Control: each stage checks `_check_should_stop` (paused/cancelled); saves partial progress (counts, credits) before returning; resume continues from `current_stage`.
- Credits: upfront estimate check (1.2x buffer) before starting; AIEngine per-call pre-checks and post-SAVE deductions; `total_credits_used` accumulates.
- Locks: acquired on start; cleared on completion or failure; also cleared on fatal errors in tasks.
- Errors: any unhandled exception marks run failed, sets error_message, logs error, clears lock; pipeline_overview/history reflect status.
- Stage result fields (persisted):
- S1: keywords_processed, clusters_created, batches_run, credits_used, skipped/partial flags, time_elapsed.
- S2: clusters_processed, ideas_created, batches_run, credits_used.
- S3: ideas_processed, tasks_created, batches_run.
- S4: tasks_processed, content_created, total_words, batches_run, credits_used.
- S5: content_processed, prompts_created, batches_run, credits_used.
- S6: images_processed, images_generated, batches_run.
- S7: ready_for_review counts.
Batching & delays:
- Configurable per site; stage_1..6 batch sizes control how many items per batch; `within_stage_delay` pauses between batches; `between_stage_delay` between stages.
Scheduling:
- `check_scheduled_automations` runs hourly; respects frequency/time and last_run_at (~23h guard); skips if a run is active; sets next_run_at; starts `run_automation_task`.
Celery execution:
- `run_automation_task` runs stages 1→7 sequentially for a run_id; failures mark run failed and clear lock.
- `resume_automation_task` / `continue_automation_task` continue from saved `current_stage`.
- Workers need access to cache (locks) and IntegrationSettings (models/providers).
Image pipeline specifics:
- Stage 5 writes prompts to Images (featured + ordered in-article).
- Stage 6 generates images via queue helper; AI `generate_images` remains partial/broken and is not used by automation.
---
## 5) Frontend Behavior (AutomationPage)
- Route: `/automation`.
- What the user can do: run now, pause, resume, cancel; edit config (enable/schedule, batch sizes, delays); view activity log; view history; watch live processing card and pipeline cards update.
- Polling: every ~5s while a run is running/paused for current_run, pipeline_overview, metrics, current_processing; lighter polling when idle.
- Metrics: fetched via low-level endpoints (keywords/clusters/ideas/tasks/content/images) for authoritative counts.
- States shown: running, paused, cancelled, failed, completed; processing card shown when a run exists; pipeline cards use “pending” counts from pipeline_overview.
- Activity log: pulled from `logs` endpoint; shown in UI for live tailing.
---
## 6) Configuration & Dependencies
- Needs IntegrationSettings for AI models and image providers (OpenAI/runware).
- Requires Celery beat and workers; cache backend required for locks.
- Tenant scoping everywhere: site + account filtering on all automation queries.
---
## 7) Known Limitations and Gaps
- `generate_images` AI function is partial/broken; automation uses queue helper instead.
- Pause/Cancel stop after the current item; no mid-item abort.
- Batch defaults are conservative (e.g., stage_2=1, stage_4=1); tune per site for throughput.
- Stage 7 is manual; no automated review step.
- No automated test suite observed for automation pipeline (stage transitions, pause/resume/cancel, scheduling guards, credit estimation/deduction).
- Enhancements to consider: fix or replace `generate_images`; add mid-item abort; surface lock status/owner; broaden batch defaults after validation; add operator-facing doc in app; add tests.
---
## 8) Field/Behavior Quick Tables
### Pipeline “pending” definitions (pipeline_overview)
- Stage 1: Keywords with status `new`, cluster is null, not disabled.
- Stage 2: Clusters status `new`, not disabled, with no ideas.
- Stage 3: ContentIdeas status `new`.
- Stage 4: Tasks status `queued`.
- Stage 5: Content status `draft` with zero images.
- Stage 6: Images status `pending`.
- Stage 7: Content status `review`.
### Stage result fields (stored on AutomationRun)
- S1: keywords_processed, clusters_created, batches_run, credits_used, skipped, partial, time_elapsed.
- S2: clusters_processed, ideas_created, batches_run, credits_used.
- S3: ideas_processed, tasks_created, batches_run.
- S4: tasks_processed, content_created, total_words, batches_run, credits_used.
- S5: content_processed, prompts_created, batches_run, credits_used.
- S6: images_processed, images_generated, batches_run.
- S7: ready_for_review.
### Credit handling
- Pre-run: estimate_credits * 1.2 vs account.credits (fails if insufficient).
- Per AI call: AIEngine pre-check credits; post-SAVE deduction with cost/tokens tracked; total_credits_used aggregates deductions.
### Logging
- Per-run log file via AutomationLogger; accessed with `GET logs?run_id=&lines=`; includes stage start/progress/errors and batch info.
### Polling (frontend)
- Active run: ~5s cadence for current_run, pipeline_overview, metrics, current_processing, logs tail.
- Idle: lighter polling (current_run/pipeline_overview) to show readiness and pending counts.

View File

@@ -0,0 +1,73 @@
# Content Lifecycle
## Purpose
Trace content from keyword intake through clustering, idea generation, task creation, AI writing, image generation, and publishing/sync, mapping to backend services and frontend pages.
## Code Locations (exact paths)
- Planner models: `backend/igny8_core/business/planning/models.py` (`Keywords`, `Clusters`, `ContentIdeas`)
- Writer models: `backend/igny8_core/business/content/models.py` (`Tasks`, `Content`, `Images`, `ContentTaxonomy`)
- ViewSets: `backend/igny8_core/modules/planner/views.py`; `backend/igny8_core/modules/writer/views.py`
- Automation orchestration: `backend/igny8_core/business/automation/services/automation_service.py`; Celery tasks `business/automation/tasks.py`
- WordPress sync/publish: `backend/igny8_core/business/integration/services/integration_service.py`, `modules/integration/views.py`
- Frontend pages: `frontend/src/pages/Planner/*`, `frontend/src/pages/Writer/*`, `frontend/src/pages/Automation/AutomationPage.tsx`, `frontend/src/pages/Sites/*`
## High-Level Responsibilities
- Collect and cluster keywords, generate ideas, create tasks, generate content and images, then publish/sync.
- Support manual steps via Planner/Writer pages and automated runs via Automation pipeline.
- Maintain site/sector scoping and credit enforcement throughout AI operations.
## Detailed Behavior
- Keywords → Clusters: `KeywordViewSet` (upload/create) and `ClusterViewSet` (manual or AI auto_cluster). Models inherit `SiteSectorBaseModel` for scoping.
- Clusters → ContentIdeas: `ContentIdeasViewSet` with AI generation endpoint; ideas track status.
- Ideas/Tasks: `Tasks` model holds brief/keywords/structure/status; created manually or by automation stage.
- Tasks → Content: Writer endpoints generate content (AI) and update `Content` records; statuses managed in writer views.
- Content → Images: `ImagesViewSet` handles image generation and storage; images linked to tasks/content.
- Publishing: Integration service sends content to WordPress via `/api/v1/integration` endpoints; WP plugin responds with IDs and syncs status back.
- Automation: `automation_service.run_automation` executes 7 stages with delays/retries/credit estimates; run status tracked in `AutomationRun`.
## Data Structures / Models Involved (no code)
- Planner: `Keywords`, `Clusters`, `ContentIdeas`.
- Writer: `Tasks`, `Content`, `Images`, `ContentTaxonomy`.
- Automation: `AutomationConfig`, `AutomationRun`.
- Integration: `SiteIntegration`.
## Execution Flow
- Manual path: upload keywords → cluster (auto/manual) → generate ideas → create tasks → generate content → generate images → publish to WP → WP syncs status back.
- Automated path: Automation pipeline stages perform the same transitions in sequence, logging progress and deducting credits.
- Each stage uses DRF viewsets; automation uses Celery tasks (`run_automation_task`, `resume_automation_task`) and logger for trace files.
## Cross-Module Interactions
- Automation consumes planner/writer endpoints and triggers integration publish.
- Integration relies on sites WP API key (`Site.wp_api_key`) and `SiteIntegration` config for credentials/URLs.
- Billing deducts credits for AI operations through `CreditService`.
## State Transitions (if applicable)
- Keywords: status (active/inactive/used) per model.
- Clusters: status (active/archived).
- ContentIdeas: status (draft/approved/in_progress/completed).
- Tasks: status simplified (queued/completed) in Stage 1; content status (draft/published) and image status stored on related models.
- AutomationRun: status across run lifecycle (created/running/paused/completed/failed).
## Error Handling
- Viewsets rely on unified DRF error responses; automation logger records failures per stage; Celery retries configured where applicable.
- WordPress sync errors surfaced via integration service responses.
## Tenancy Rules
- All planner/writer/automation/integration models inherit account/site/sector via base models; viewsets filter by `request.account` and query params.
## Billing Rules (if applicable)
- AI clustering/idea/content/image generation deduct credits via `CreditService` cost map; automation blocks run if estimated credits insufficient.
## Background Tasks / Schedulers (if applicable)
- Automation scheduler: `check_scheduled_automations` enqueues runs; `run_automation_task`/`resume_automation_task` execute pipeline in Celery.
- AI tasks run async via Celery wrappers (`ai/tasks.py`).
## Key Design Considerations
- Single source of truth for tenancy via base models; prevents cross-tenant data exposure.
- Automation mirrors manual flow; manual pages remain usable when automation is paused or disabled.
- WordPress publish/sync uses API key auth to avoid storing user credentials beyond site integration config.
## How Developers Should Work With This Module
- When adding new content stages, update both manual endpoints and automation stages.
- Ensure every new action that calls AI is wired through `CreditService` and respects site/sector filters.
- For new publishing targets, extend `SiteIntegration` and integration service while keeping site/account scoping.

View File

@@ -0,0 +1,69 @@
# Billing Lifecycle
## Purpose
Detail how credits, plans, subscriptions, invoices, and payments flow through the system, and where deductions/additions are enforced.
## Code Locations (exact paths)
- Models: `backend/igny8_core/business/billing/models.py` (`CreditTransaction`, `CreditUsageLog`, `CreditCostConfig`, `Invoice`, `Payment`, `CreditPackage`, `PaymentMethodConfig`, `AccountPaymentMethod`)
- Services: `backend/igny8_core/business/billing/services/credit_service.py`, `services/invoice_service.py`, `services/payment_service.py`
- Views/Endpoints: `backend/igny8_core/modules/billing/views.py`
- Frontend billing pages: `frontend/src/pages/account/{AccountBillingPage.tsx,PlansAndBillingPage.tsx,PurchaseCreditsPage.tsx}`
## High-Level Responsibilities
- Track credit balance, record usage, and enforce costs for AI/automation actions.
- Manage plans/subscriptions, generate invoices, and record payments.
- Expose account-scoped billing data (balance, usage, invoices, payments, packages, payment methods).
## Detailed Behavior
- Credit balance: stored on `Account.credits`; `CreditService` adds/deducts via `CreditTransaction` and logs usage in `CreditUsageLog`.
- Costs: read from `CreditCostConfig` or hardcoded `CREDIT_COSTS` fallbacks inside `credit_service.py`.
- Deductions: called from AI/automation flows to ensure sufficient credits; `InsufficientCreditsError` thrown on shortage.
- Plans/Subscriptions: `Plan` defines price, billing_cycle, included_credits, max sites/sectors/users; `Subscription` links account to plan. Plan selection handled via billing endpoints and frontend plan tabs.
- Invoices: `InvoiceService` creates invoices for subscriptions and credit packages; generates unique invoice numbers and line items.
- Payments: `PaymentService` records payments; manual payments stored with status (pending/processing/etc.); payment methods configured via `PaymentMethodConfig`/`AccountPaymentMethod`.
- Packages: `CreditPackage` defines purchasable credit bundles; purchasing triggers invoice/payment flows.
## Data Structures / Models Involved (no code)
- Billing: `CreditTransaction`, `CreditUsageLog`, `CreditCostConfig`, `Invoice`, `Payment`, `CreditPackage`, `PaymentMethodConfig`, `AccountPaymentMethod`.
- Plan: `Plan`, `Subscription` (in `auth/models.py`).
- Account linkage: `Account` holds `credits` and `plan`.
## Execution Flow
- Balance/Usage read: billing endpoints return account-scoped balance and usage summaries.
- AI/Automation call: service invokes `CreditService.deduct_credits_for_action` → creates `CreditTransaction` + `CreditUsageLog`.
- Purchase credits: frontend calls billing endpoint → `CreditService.add_credits` + `InvoiceService` for package; payment recorded via `PaymentService` (manual/other).
- Subscription change: endpoint updates `Subscription` + `Account.plan`, generates invoice as needed; payment recorded if required.
- Invoice download: `modules/billing/views.py` exposes invoice retrieval; frontend uses `downloadInvoicePDF`.
## Cross-Module Interactions
- Automation/writer AI calls depend on credit checks; insufficient balance blocks operations.
- Account settings feed invoice billing details; payments/invoices are tied to account.
## State Transitions (if applicable)
- Payment status: `pending``processing/pending_approval``succeeded/completed` or `failed/refunded/cancelled/void/uncollectible`.
- Subscription status: `active`/`cancelled` (persisted in `Subscription`); plan reference on account updates accordingly.
## Error Handling
- `InsufficientCreditsError` for low balance; surfaces to API as error response.
- Payment/Invoice service raise validation errors (e.g., already paid invoice).
## Tenancy Rules
- All billing queries filter by `request.user.account`; viewsets inherit from `AccountModelViewSet`.
- Payment methods, invoices, payments, credit transactions are account-scoped; no cross-tenant access.
## Billing Rules (if applicable)
- Costs derived from config constants or `CreditCostConfig`.
- Included credits and limits come from `Plan`; extra purchases via `CreditPackage`.
- Credits must be available before AI/automation runs proceed.
## Background Tasks / Schedulers (if applicable)
- None dedicated; billing operations are request-driven. Automation tasks carry account context to deduct credits within Celery runs.
## Key Design Considerations
- CreditService centralizes deductions to avoid scattered logic.
- Manual payment flow avoids storing sensitive data; relies on transaction_reference and admin approval.
## How Developers Should Work With This Module
- When adding new billable actions, route them through `CreditService` with a defined cost key.
- To change pricing, update `CreditCostConfig` data or the `CREDIT_COSTS` fallback map.
- Keep all billing endpoints inheriting `AccountModelViewSet` to maintain isolation.

View File

@@ -0,0 +1,77 @@
# User Flow Overview
## Purpose
Describe the end-to-end journey from signup through planning, writing, automation, publishing, and billing within IGNY8, mapping each step to concrete backend/frontend modules so engineers can navigate without scanning code.
## Code Locations (exact paths)
- Auth & account: `backend/igny8_core/auth/{views.py,serializers.py,models.py,middleware.py}`, `frontend/src/store/authStore.ts`, routes in `frontend/src/App.tsx`
- Planner: `backend/igny8_core/modules/planner/views.py`, `business/planning/models.py`, `frontend/src/pages/Planner/*`
- Writer: `backend/igny8_core/modules/writer/views.py`, `business/content/models.py`, `frontend/src/pages/Writer/*`
- Automation: `backend/igny8_core/business/automation/{services/automation_service.py,tasks.py,views.py}`, `frontend/src/pages/Automation/AutomationPage.tsx`
- Publishing/Integration: `backend/igny8_core/business/integration/{models.py,services/integration_service.py}`, `modules/integration/views.py`, WordPress plugin endpoints consumed via `/api/v1/integration/`; `frontend/src/pages/Sites/*`
- Billing: `backend/igny8_core/business/billing/{models.py,services/credit_service.py,services/invoice_service.py,views.py}`, `frontend/src/pages/account/*`
- Tenancy enforcement: `backend/igny8_core/auth/middleware.py`, `backend/igny8_core/api/base.py`
## High-Level Responsibilities
- Authenticate users and attach account context to every request.
- Let users plan keywords/clusters/ideas, create tasks and content, optionally automate all seven pipeline stages.
- Manage sites/sectors and connect WordPress for publishing/sync.
- Track credits, plans, subscriptions, invoices, and payments through billing services.
## Detailed Behavior
- Signup/Login: `auth/views.py` issues JWTs; `AccountContextMiddleware` sets `request.account`; frontend `authStore` stores tokens and refreshes them.
- Planning: `KeywordViewSet`, `ClusterViewSet`, `ContentIdeasViewSet` create/list/update scoped by `SiteSectorModelViewSet` filters; frontend planner pages drive these endpoints.
- Writing: `TasksViewSet`, `ContentViewSet`, `ImagesViewSet` manage tasks/content/images; AI generation endpoints trigger Celery-backed functions.
- Automation: `AutomationViewSet` + `automation_service.py` orchestrate 7 stages (keywords→clusters→ideas→tasks→content→image-prompts→images/manual review) with pause/resume and credit estimation; Celery tasks `run_automation_task`/`resume_automation_task` execute runs.
- Publishing/Integration: `IntegrationViewSet` handles WP connection tests and sync; WP plugin sends/receives data via API key; content publish endpoints in writer module update WordPress via integration services.
- Billing: credit balances and costs computed in `credit_service.py`; invoices via `invoice_service.py`; payments via `payment_service.py`; endpoints in `modules/billing/views.py` feed frontend billing pages.
## Data Structures / Models Involved (no code)
- Tenancy: `Account`, `Site`, `Sector` (plus `SiteUserAccess`), base models `AccountBaseModel`/`SiteSectorBaseModel`.
- Planning: `Keywords`, `Clusters`, `ContentIdeas`.
- Writing: `Tasks`, `Content`, `Images`.
- Automation: `AutomationConfig`, `AutomationRun`.
- Billing: `CreditTransaction`, `CreditUsageLog`, `CreditCostConfig`, `Invoice`, `Payment`, `CreditPackage`, `Subscription`, `Plan`.
- Integration: `SiteIntegration`.
## Execution Flow
1) User signs in → JWT stored → `AccountContextMiddleware` populates `request.account`.
2) Tenant creates sites/sectors (`SiteViewSet`), selects industry/sectors, optionally connects WordPress.
3) Planner: upload/enter keywords → cluster → generate ideas (manual or via automation/AI functions).
4) Writer: create tasks from ideas, generate content/images (manual endpoints or automation stages).
5) Publish: send to WordPress via integration endpoints or automation publish step; WP plugin syncs back statuses.
6) Automation (optional): run 7-stage pipeline via `AutomationViewSet` + Celery tasks; pause/resume supported.
7) Billing: credits deducted per AI/pipeline usage (`credit_service`), invoices/payments recorded; users view in billing pages.
## Cross-Module Interactions
- Automation invokes planner/writer endpoints/services and logs credit estimates.
- Billing hooks into automation and writer AI calls via credit deduction utilities.
- Integration uses site/account context and WP API key for sync; writer publish flows depend on integration configuration.
## State Transitions (if applicable)
- Account status (`active/suspended/trial/cancelled`) governs access; plan/subscription affect limits.
- Tasks/content/images status transitions handled in writer endpoints; automation run status moves through `created/running/paused/completed/failed`.
- Site activation via `set_active`; sectors toggled via `select_sectors`.
## Error Handling
- Unified API responses via `api/response.py`; DRF exception handler configured in settings; frontend shows toasts/banners.
- Automation errors logged via `automation_logger`; tasks wrapped in Celery with retries where defined.
## Tenancy Rules
- `AccountContextMiddleware` sets `request.account`; base viewsets filter by account/site/sector; API key auth sets account from `Site.wp_api_key`; public site slug reads limited to active sites.
## Billing Rules (if applicable)
- AI/automation uses `CreditService.deduct_credits_for_action`; credit balance required before runs; plans/subscriptions define included credits and sector/site limits.
## Background Tasks / Schedulers (if applicable)
- Celery tasks: automation runs (`run_automation_task`, `resume_automation_task`), scheduler `check_scheduled_automations`, AI functions (`ai/tasks.py`), publishing tasks (`tasks/wordpress_publishing.py`).
## Key Design Considerations
- Strong tenant isolation via middleware + filtered querysets.
- Automation relies on polling and Celery; frontend automation page polls every 5s during runs.
- Billing is authoritative in backend; frontend is read-only except initiating purchases/subscriptions.
## How Developers Should Work With This Module
- Trace user-visible flows by following router → page component → service call → backend viewset.
- When adding steps, ensure credit deductions and tenancy filters are applied in the corresponding backend service/viewset.
- Keep WP integration changes consistent with API key auth and `SiteIntegration` schema.

File diff suppressed because it is too large Load Diff