diff --git a/README.md b/README.md index a1fa0d83..75746ca2 100644 --- a/README.md +++ b/README.md @@ -25,13 +25,13 @@ IGNY8 is a full-stack SaaS platform that combines AI-powered content generation ## Repository Structure -This monorepo contains two main applications: +This monorepo contains two main applications and documentation: ``` igny8/ ├── backend/ # Django REST API + Celery ├── frontend/ # React + Vite SPA -├── master-docs/ # Architecture documentation +├── docs/ # Documentation index and topic folders └── docker-compose.app.yml # Docker deployment config ``` @@ -210,14 +210,20 @@ The WordPress bridge plugin (`igny8-wp-integration`) creates a bidirectional con ## Documentation -Comprehensive documentation is available in the `master-docs/` directory: +Start here: [docs/README.md](./docs/README.md) (index of all topics). -- **[MASTER_REFERENCE.md](./MASTER_REFERENCE.md)** - Complete system architecture and navigation -- **[API-COMPLETE-REFERENCE.md](./master-docs/API-COMPLETE-REFERENCE.md)** - Full API documentation -- **[02-APPLICATION-ARCHITECTURE.md](./master-docs/02-APPLICATION-ARCHITECTURE.md)** - System design -- **[04-BACKEND-IMPLEMENTATION.md](./master-docs/04-BACKEND-IMPLEMENTATION.md)** - Backend details -- **[03-FRONTEND-ARCHITECTURE.md](./master-docs/03-FRONTEND-ARCHITECTURE.md)** - Frontend details -- **[WORDPRESS-PLUGIN-INTEGRATION.md](./master-docs/WORDPRESS-PLUGIN-INTEGRATION.md)** - Plugin integration guide +Common entry points: +- App architecture: `docs/igny8-app/IGNY8-APP-ARCHITECTURE.md` +- Backend architecture: `docs/backend/IGNY8-BACKEND-ARCHITECTURE.md` +- Planner backend detail: `docs/backend/IGNY8-PLANNER-BACKEND.md` +- Writer backend detail: `docs/backend/IGNY8-WRITER-BACKEND.md` +- Automation: `docs/automation/AUTOMATION-REFERENCE.md` +- Tech stack: `docs/tech-stack/00-SYSTEM-ARCHITECTURE-MASTER-REFERENCE.md` +- API: `docs/API/API-COMPLETE-REFERENCE-LATEST.md` +- Billing & Credits: `docs/billing/billing-account-final-plan-2025-12-05.md` +- App guides: `docs/igny8-app/` (planner/writer workflows, taxonomy, feature modification) +- WordPress: `docs/wp/` (plugin integration and sync) +- Docs changelog: `docs/CHANGELOG.md` --- diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md new file mode 100644 index 00000000..22d53b62 --- /dev/null +++ b/docs/CHANGELOG.md @@ -0,0 +1,19 @@ +# IGNY8 Docs Changelog (Dec 2025) + +## 2025-12-07 +### Features captured (code-sourced) +- Added backend-only architecture doc (`docs/backend/IGNY8-BACKEND-ARCHITECTURE.md`) reflecting current namespaces, billing models, integrations, and webhooks. +- Added planner backend doc (`docs/backend/IGNY8-PLANNER-BACKEND.md`) and writer backend doc (`docs/backend/IGNY8-WRITER-BACKEND.md`) from live code. +- Replaced automation folder with single canonical `docs/automation/AUTOMATION-REFERENCE.md` (code-sourced pipeline, APIs, models, tasks, frontend). +- Multi-tenant foundation via `AccountBaseModel` / `SiteSectorBaseModel` with tenant-scoped billing fields on `Account`. +- Backend namespaces wired in `backend/igny8_core/urls.py` covering auth, account, planner, writer, system, billing (tenant), admin billing, automation, linker, optimizer, publisher, integration. +- Billing tenant surface: invoices, payments (including manual submission + available methods), credit-packages, credit-transactions, payment-methods CRUD/default, credits balance/usage/transactions. +- Billing admin surface under `/api/v1/admin/`: stats, users credit adjustments, credit costs, invoices/payments/pending approvals, payment-method configs, account payment methods. +- WordPress integration webhooks (`/api/v1/integration/webhooks/wordpress/status|metadata/`) and Gitea webhook (`/api/v1/system/webhook/`). +- Frontend routes mapped for Planner, Writer, Automation, Linker, Optimizer, Thinker, Billing module, Account pages, and Admin billing/management; sidebar paths defined in `frontend/src/layout/AppSidebar.tsx`. + +### Issues / gaps to fix +- `docs/user-flow/` is empty; needs end-to-end flows (account, billing, content) based on current routes and APIs. +- Billing/account documentation must be aligned to the live namespaces and models captured in `docs/igny8-app/IGNY8-APP-ARCHITECTURE.md` (avoid legacy/retired paths). +- Add API request/response samples for billing/admin endpoints to the billing docs to match the current serializers (not yet documented here). + diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..12017547 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,30 @@ +# IGNY8 Documentation Index (Dec 2025) + +Purpose: single entry point for all project docs with live links to the current sources. This index reflects the codebase as implemented, not legacy placeholders. + +## Codebase Snapshot +- **Backend (Django/DRF):** Multi-tenant models inherit `AccountBaseModel`/`SiteSectorBaseModel` for automatic tenant scoping (`backend/igny8_core/auth/models.py`). API namespaces are wired in `backend/igny8_core/urls.py`: `/api/v1/auth/`, `/api/v1/account/`, `/api/v1/planner/`, `/api/v1/writer/`, `/api/v1/system/`, `/api/v1/billing/` (tenant billing/invoices/credits), `/api/v1/admin/` (billing admin + credit costs/stats), `/api/v1/automation/`, `/api/v1/linker/`, `/api/v1/optimizer/`, `/api/v1/publisher/`, `/api/v1/integration/`. +- **Frontend (React/Vite):** Routing lives in `frontend/src/App.tsx` with lazy-loaded pages; account/billing pages at `/account/plans`, `/account/billing`, `/account/purchase-credits`, `/account/settings`, `/account/team`, `/account/usage`, plus tenant billing module routes under `/billing/*` and admin billing under `/admin/*`. Sidebar labels/paths are defined in `frontend/src/layout/AppSidebar.tsx`. + +## Document Map +- **App Architecture (code-sourced):** `docs/igny8-app/IGNY8-APP-ARCHITECTURE.md` +- **Backend Architecture (code-sourced):** `docs/backend/IGNY8-BACKEND-ARCHITECTURE.md` +- **Planner Backend Detail:** `docs/backend/IGNY8-PLANNER-BACKEND.md` +- **Writer Backend Detail:** `docs/backend/IGNY8-WRITER-BACKEND.md` +- **Platform / Tech Stack:** `docs/tech-stack/00-SYSTEM-ARCHITECTURE-MASTER-REFERENCE.md` +- **API:** `docs/API/01-IGNY8-REST-API-COMPLETE-REFERENCE.md`, `docs/API/API-COMPLETE-REFERENCE-LATEST.md` +- **Billing & Credits (canonical):** `docs/billing/billing-account-final-plan-2025-12-05.md`, `docs/billing/credits-system-audit-and-improvement-plan.md` +- **App Feature Guides:** `docs/igny8-app/02-PLANNER-WRITER-WORKFLOW-TECHNICAL-GUIDE.md`, `docs/igny8-app/05-WRITER-IMAGES-PAGE-SYSTEM-DESIGN.md`, `docs/igny8-app/06-FEATURE-MODIFICATION-DEVELOPER-GUIDE.md`, `docs/igny8-app/KEYWORDS-CLUSTERS-IDEAS-COMPLETE-MAPPING.md`, `docs/igny8-app/app-packaging-backaup-plan.md`, `docs/igny8-app/TAXONOMY/*`, `docs/igny8-app/status-related-temporary/*` +- **Automation:** `docs/automation/` (automation-plan, implementation analysis, UX improvements, stage-6 image generation fix, cluster validation fix plan) +- **AI Functions:** `docs/ai/AI-FUNCTIONS-COMPLETE-REFERENCE.md` +- **Automation:** `docs/automation/AUTOMATION-REFERENCE.md` +- **WordPress Integration:** `docs/wp/` (API integration guide, bidirectional sync reference, publishing field mapping, refactor/implementation summaries, deployment fixes) +- **User Flow Drafts:** root-level `user-flow-plan*.md` (in-progress drafts); `docs/user-flow/` is currently empty and needs coverage. +- **Docs Changelog:** `docs/CHANGELOG.md` + +## Known Documentation Gaps to Close +- App-level architecture summary sourced from code (backend modules and frontend routes), including tenant scoping rules. +- Billing/account accuracy pass: align docs to current endpoints in `igny8_core/urls.py` and admin aliases in `modules/billing/admin_urls.py`, plus the active frontend routes in `App.tsx`. +- Frontend route/UX map for account/billing/admin sections tied to sidebar paths. +- Multi-tenancy guardrails doc (account/site/sector isolation, role expectations) based on `auth/models.py`. + diff --git a/docs/ai/AI-FUNCTIONS-COMPLETE-REFERENCE.md b/docs/ai/AI-FUNCTIONS-COMPLETE-REFERENCE.md index afa4e3b4..cd466d0c 100644 --- a/docs/ai/AI-FUNCTIONS-COMPLETE-REFERENCE.md +++ b/docs/ai/AI-FUNCTIONS-COMPLETE-REFERENCE.md @@ -1065,6 +1065,11 @@ Frontend can listen to these events via: - Centralized progress tracking - Easy to add new AI functions (inherit from BaseAIFunction) +### Current gaps vs code (Dec 2025) +- AIEngine now performs a credit pre-check before the AI call (still deducts after SAVE); this is not reflected in earlier notes. +- `generate_images` implementation is partially broken: it expects task IDs (not image IDs), tries to read `task.content` (field does not exist), and uses the `extract_image_prompts` prompt path; credit estimation also looks for `image_ids`. Treat it as partial/needs fix. +- AIEngine includes messaging/cost maps for `generate_site_structure` (extra function beyond the documented six); not presently documented above. + --- ## Automation Integration diff --git a/docs/automation/AUTOMATION-IMPLEMENTATION-ANALYSIS-CORRECTED.md b/docs/automation/AUTOMATION-IMPLEMENTATION-ANALYSIS-CORRECTED.md deleted file mode 100644 index ad50863f..00000000 --- a/docs/automation/AUTOMATION-IMPLEMENTATION-ANALYSIS-CORRECTED.md +++ /dev/null @@ -1,1053 +0,0 @@ -# IGNY8 Automation Implementation - Complete Analysis -**Date:** December 3, 2025 -**Version:** 2.0 - CORRECTED AFTER FULL CODEBASE AUDIT -**Based on:** Complete actual codebase analysis (backend + frontend) + automation-plan.md comparison - ---- - -## Executive Summary - -**IMPLEMENTATION STATUS: 95% COMPLETE AND FULLY FUNCTIONAL** ✅ - -The IGNY8 automation system is **FULLY IMPLEMENTED AND WORKING** in production. The previous documentation incorrectly stated the frontend route was missing and migrations weren't run. After thorough codebase analysis, this is WRONG. - -### ✅ VERIFIED WORKING COMPONENTS - -| Component | Status | Evidence | -|-----------|--------|----------| -| **Backend Models** | ✅ 100% Complete | `AutomationConfig`, `AutomationRun` fully implemented | -| **Backend Service** | ✅ 100% Complete | All 7 stages working in `AutomationService` (830 lines) | -| **REST API** | ✅ 100% Complete | 10 endpoints working (9 planned + 1 bonus) | -| **Celery Tasks** | ✅ 100% Complete | Scheduling and execution working | -| **Frontend Page** | ✅ 100% Complete | `AutomationPage.tsx` (643 lines) fully functional | -| **Frontend Route** | ✅ REGISTERED | `/automation` route EXISTS in `App.tsx` line 264 | -| **Frontend Service** | ✅ 100% Complete | `automationService.ts` with all 10 API methods | -| **Frontend Components** | ✅ 100% Complete | StageCard, ActivityLog, RunHistory, ConfigModal | -| **Sidebar Navigation** | ✅ REGISTERED | Automation menu item in `AppSidebar.tsx` line 132 | -| **Distributed Locking** | ✅ Working | Redis-based concurrent run prevention | -| **Credit Management** | ✅ Working | Automatic deduction via AIEngine | -| **Real-time Updates** | ✅ Working | 5-second polling with live status | - -### ⚠️ MINOR GAPS (Non-Breaking) - -| Item | Status | Impact | Fix Needed | -|------|--------|--------|------------| -| Stage 6 Image Generation | ⚠️ Partial | Low - structure exists, API integration may need testing | Test/complete image provider API calls | -| Word Count Calculation | ⚠️ Estimate only | Cosmetic - reports estimated (tasks * 2500) not actual | Use `Sum('word_count')` in Stage 4 | -| AutomationLogger Paths | ⚠️ Needs testing | Low - works but file paths may need production validation | Test in production environment | - -**Overall Grade: A- (95/100)** - ---- - -## Table of Contents -1. [Frontend Implementation](#frontend-implementation) -2. [Backend Implementation](#backend-implementation) -3. [7-Stage Pipeline Deep Dive](#7-stage-pipeline-deep-dive) -4. [API Endpoints](#api-endpoints) -5. [Configuration & Settings](#configuration--settings) -6. [Comparison vs Plan](#comparison-vs-plan) -7. [Gaps & Recommendations](#gaps--recommendations) - ---- - -## Frontend Implementation - -### 1. AutomationPage.tsx (643 lines) - -**Location:** `frontend/src/pages/Automation/AutomationPage.tsx` - -**Route Registration:** ✅ `/automation` route EXISTS in `App.tsx` line 264: -```tsx -{/* Automation Module */} - - - -} /> -``` - -**Sidebar Navigation:** ✅ Menu item REGISTERED in `AppSidebar.tsx` line 128-135: -```tsx -// Add Automation (always available if Writer is enabled) -if (account.writer_enabled) { - mainNav.push({ - name: "Automation", - path: "/automation", - icon: BoltIcon, - }); -} -``` - -**Page Features:** -- Real-time polling (5-second interval) -- Schedule & controls (Enable/Disable, Run Now, Pause, Resume) -- Pipeline overview with 7 stage cards -- Current run details with live progress -- Activity log viewer (real-time logs) -- Run history table -- Configuration modal - -**Key Hooks & State:** -```tsx -const [config, setConfig] = useState(null); -const [currentRun, setCurrentRun] = useState(null); -const [pipelineOverview, setPipelineOverview] = useState([]); -const [estimate, setEstimate] = useState<{ estimated_credits, current_balance, sufficient } | null>(null); - -// Real-time polling -useEffect(() => { - const interval = setInterval(() => { - if (currentRun && (currentRun.status === 'running' || currentRun.status === 'paused')) { - loadCurrentRun(); - } else { - loadPipelineOverview(); - } - }, 5000); - return () => clearInterval(interval); -}, [currentRun?.status]); -``` - -**Stage Configuration (7 stages with icons):** -```tsx -const STAGE_CONFIG = [ - { icon: ListIcon, color: 'from-blue-500 to-blue-600', name: 'Keywords → Clusters' }, - { icon: GroupIcon, color: 'from-purple-500 to-purple-600', name: 'Clusters → Ideas' }, - { icon: CheckCircleIcon, color: 'from-indigo-500 to-indigo-600', name: 'Ideas → Tasks' }, - { icon: PencilIcon, color: 'from-green-500 to-green-600', name: 'Tasks → Content' }, - { icon: FileIcon, color: 'from-amber-500 to-amber-600', name: 'Content → Image Prompts' }, - { icon: FileTextIcon, color: 'from-pink-500 to-pink-600', name: 'Image Prompts → Images' }, - { icon: PaperPlaneIcon, color: 'from-teal-500 to-teal-600', name: 'Manual Review Gate' }, -]; -``` - -**Pipeline Overview UI:** -- Combines Stages 3 & 4 into one card (Ideas → Tasks → Content) -- Shows pending counts from `pipeline_overview` endpoint -- Displays live run results when active -- Color-coded status (Active=Blue, Complete=Green, Ready=Purple, Empty=Gray) -- Stage 7 shown separately as "Manual Review Gate" with warning (automation stops here) - ---- - -### 2. Frontend Service (automationService.ts) - -**Location:** `frontend/src/services/automationService.ts` - -**Complete API Client with 10 Methods:** - -```typescript -export const automationService = { - // 1. Get config - getConfig: async (siteId: number): Promise => { - return fetchAPI(buildUrl('/config/', { site_id: siteId })); - }, - - // 2. Update config - updateConfig: async (siteId: number, config: Partial): Promise => { - await fetchAPI(buildUrl('/update_config/', { site_id: siteId }), { - method: 'PUT', - body: JSON.stringify(config), - }); - }, - - // 3. Run now - runNow: async (siteId: number): Promise<{ run_id: string; message: string }> => { - return fetchAPI(buildUrl('/run_now/', { site_id: siteId }), { method: 'POST' }); - }, - - // 4. Get current run - getCurrentRun: async (siteId: number): Promise<{ run: AutomationRun | null }> => { - return fetchAPI(buildUrl('/current_run/', { site_id: siteId })); - }, - - // 5. Pause - pause: async (runId: string): Promise => { - await fetchAPI(buildUrl('/pause/', { run_id: runId }), { method: 'POST' }); - }, - - // 6. Resume - resume: async (runId: string): Promise => { - await fetchAPI(buildUrl('/resume/', { run_id: runId }), { method: 'POST' }); - }, - - // 7. Get history - getHistory: async (siteId: number): Promise => { - const response = await fetchAPI(buildUrl('/history/', { site_id: siteId })); - return response.runs; - }, - - // 8. Get logs - getLogs: async (runId: string, lines: number = 100): Promise => { - const response = await fetchAPI(buildUrl('/logs/', { run_id: runId, lines })); - return response.log; - }, - - // 9. Estimate credits - estimate: async (siteId: number): Promise<{ - estimated_credits: number; - current_balance: number; - sufficient: boolean; - }> => { - return fetchAPI(buildUrl('/estimate/', { site_id: siteId })); - }, - - // 10. Get pipeline overview (BONUS - not in plan) - getPipelineOverview: async (siteId: number): Promise<{ stages: PipelineStage[] }> => { - return fetchAPI(buildUrl('/pipeline_overview/', { site_id: siteId })); - }, -}; -``` - -**TypeScript Interfaces:** -```typescript -export interface AutomationConfig { - is_enabled: boolean; - frequency: 'daily' | 'weekly' | 'monthly'; - scheduled_time: string; - stage_1_batch_size: number; - stage_2_batch_size: number; - stage_3_batch_size: number; - stage_4_batch_size: number; - stage_5_batch_size: number; - stage_6_batch_size: number; - last_run_at: string | null; - next_run_at: string | null; -} - -export interface AutomationRun { - run_id: string; - status: 'running' | 'paused' | 'completed' | 'failed'; - current_stage: number; - trigger_type: 'manual' | 'scheduled'; - started_at: string; - total_credits_used: number; - stage_1_result: StageResult | null; - stage_2_result: StageResult | null; - stage_3_result: StageResult | null; - stage_4_result: StageResult | null; - stage_5_result: StageResult | null; - stage_6_result: StageResult | null; - stage_7_result: StageResult | null; -} - -export interface PipelineStage { - number: number; - name: string; - pending: number; - type: 'AI' | 'Local' | 'Manual'; -} -``` - ---- - -### 3. Frontend Components - -**a) StageCard.tsx** -- Shows individual stage status -- Color-coded by state (pending/active/complete) -- Displays pending counts from pipeline overview -- Shows run results when stage completes - -**b) ActivityLog.tsx** -- Real-time log viewer -- Polls logs every 3 seconds -- Configurable line count (50/100/200/500) -- Terminal-style display (monospace, dark bg) - -**c) RunHistory.tsx** -- Table of past automation runs -- Columns: Run ID, Status, Trigger, Started, Completed, Credits, Stage -- Color-coded status badges -- Responsive table design - -**d) ConfigModal.tsx** -- Edit automation configuration -- Enable/disable toggle -- Frequency selector (daily/weekly/monthly) -- Scheduled time picker -- Batch size inputs for all 6 AI stages (1-6) - ---- - -## Backend Implementation - -### 1. Database Models - -**File:** `backend/igny8_core/business/automation/models.py` (106 lines) - -**AutomationConfig Model:** -```python -class AutomationConfig(models.Model): - """Per-site automation configuration""" - - FREQUENCY_CHOICES = [ - ('daily', 'Daily'), - ('weekly', 'Weekly'), - ('monthly', 'Monthly'), - ] - - account = models.ForeignKey(Account, on_delete=models.CASCADE) - site = models.OneToOneField(Site, on_delete=models.CASCADE) # ONE config per site - - is_enabled = models.BooleanField(default=False) - frequency = models.CharField(max_length=20, choices=FREQUENCY_CHOICES, default='daily') - scheduled_time = models.TimeField(default='02:00') - - # Batch sizes per stage - stage_1_batch_size = models.IntegerField(default=20) - stage_2_batch_size = models.IntegerField(default=1) - stage_3_batch_size = models.IntegerField(default=20) - stage_4_batch_size = models.IntegerField(default=1) - stage_5_batch_size = models.IntegerField(default=1) - stage_6_batch_size = models.IntegerField(default=1) - - last_run_at = models.DateTimeField(null=True, blank=True) - next_run_at = models.DateTimeField(null=True, blank=True) - - class Meta: - db_table = 'igny8_automation_configs' -``` - -**AutomationRun Model:** -```python -class AutomationRun(models.Model): - """Tracks automation execution""" - - STATUS_CHOICES = [ - ('running', 'Running'), - ('paused', 'Paused'), - ('completed', 'Completed'), - ('failed', 'Failed'), - ] - - TRIGGER_CHOICES = [ - ('manual', 'Manual'), - ('scheduled', 'Scheduled'), - ] - - run_id = models.UUIDField(default=uuid.uuid4, unique=True) - account = models.ForeignKey(Account, on_delete=models.CASCADE) - site = models.ForeignKey(Site, on_delete=models.CASCADE) - - status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='running') - current_stage = models.IntegerField(default=1) - trigger_type = models.CharField(max_length=20, choices=TRIGGER_CHOICES) - - started_at = models.DateTimeField(auto_now_add=True) - completed_at = models.DateTimeField(null=True, blank=True) - - total_credits_used = models.IntegerField(default=0) - - # Results for each stage (JSON) - stage_1_result = models.JSONField(null=True, blank=True) - stage_2_result = models.JSONField(null=True, blank=True) - stage_3_result = models.JSONField(null=True, blank=True) - stage_4_result = models.JSONField(null=True, blank=True) - stage_5_result = models.JSONField(null=True, blank=True) - stage_6_result = models.JSONField(null=True, blank=True) - stage_7_result = models.JSONField(null=True, blank=True) - - class Meta: - db_table = 'igny8_automation_runs' -``` - ---- - -### 2. AutomationService (830 lines) - -**File:** `backend/igny8_core/business/automation/services/automation_service.py` - -**Key Methods:** - -```python -class AutomationService: - def __init__(self, account, site): - self.account = account - self.site = site - self.logger = AutomationLogger(site.id) - - def start_automation(self, trigger_type='manual'): - """ - Main entry point - starts 7-stage pipeline - - 1. Check credits - 2. Acquire distributed lock (Redis) - 3. Create AutomationRun record - 4. Execute stages 1-7 sequentially - 5. Update run status - 6. Release lock - """ - - # Credit check - estimate = self.estimate_credits() - if not estimate['sufficient']: - raise ValueError('Insufficient credits') - - # Distributed lock - lock_key = f'automation_run_{self.site.id}' - lock = redis_client.lock(lock_key, timeout=3600) - if not lock.acquire(blocking=False): - raise ValueError('Automation already running for this site') - - try: - # Create run record - run = AutomationRun.objects.create( - account=self.account, - site=self.site, - trigger_type=trigger_type, - status='running', - current_stage=1 - ) - - # Execute stages - for stage in range(1, 8): - if run.status == 'paused': - break - - result = self._execute_stage(stage, run) - setattr(run, f'stage_{stage}_result', result) - run.current_stage = stage + 1 - run.save() - - # Mark complete - run.status = 'completed' - run.completed_at = timezone.now() - run.save() - - return run - - finally: - lock.release() - - def _execute_stage(self, stage_num, run): - """Execute individual stage""" - if stage_num == 1: - return self.run_stage_1() - elif stage_num == 2: - return self.run_stage_2() - # ... stages 3-7 - - def run_stage_1(self): - """ - Stage 1: Keywords → Clusters (AI) - - 1. Get unmapped keywords (status='new') - 2. Batch by stage_1_batch_size - 3. Call AutoClusterFunction via AIEngine - 4. Update Keywords.cluster_id and Keywords.status='mapped' - """ - - config = AutomationConfig.objects.get(site=self.site) - batch_size = config.stage_1_batch_size - - keywords = Keywords.objects.filter( - site=self.site, - status='new' - )[:batch_size] - - if not keywords: - return {'keywords_processed': 0, 'clusters_created': 0} - - # Call AI function - from igny8_core.ai.tasks import run_ai_task - result = run_ai_task( - function_name='auto_cluster', - payload={'ids': [k.id for k in keywords]}, - account_id=self.account.id - ) - - return { - 'keywords_processed': len(keywords), - 'clusters_created': result.get('count', 0), - 'credits_used': result.get('credits_used', 0) - } - - def run_stage_2(self): - """ - Stage 2: Clusters → Ideas (AI) - - 1. Get clusters with status='active' and ideas_count=0 - 2. Process ONE cluster at a time (batch_size=1 recommended) - 3. Call GenerateIdeasFunction - 4. Update Cluster.status='mapped' - """ - - config = AutomationConfig.objects.get(site=self.site) - batch_size = config.stage_2_batch_size - - clusters = Clusters.objects.filter( - site=self.site, - status='active' - ).annotate( - ideas_count=Count('contentideas') - ).filter(ideas_count=0)[:batch_size] - - if not clusters: - return {'clusters_processed': 0, 'ideas_created': 0} - - from igny8_core.ai.tasks import run_ai_task - result = run_ai_task( - function_name='generate_ideas', - payload={'ids': [c.id for c in clusters]}, - account_id=self.account.id - ) - - return { - 'clusters_processed': len(clusters), - 'ideas_created': result.get('count', 0), - 'credits_used': result.get('credits_used', 0) - } - - def run_stage_3(self): - """ - Stage 3: Ideas → Tasks (LOCAL - No AI) - - 1. Get ideas with status='new' - 2. Create Tasks records - 3. Update ContentIdeas.status='in_progress' - """ - - config = AutomationConfig.objects.get(site=self.site) - batch_size = config.stage_3_batch_size - - ideas = ContentIdeas.objects.filter( - site=self.site, - status='new' - )[:batch_size] - - tasks_created = 0 - for idea in ideas: - Tasks.objects.create( - title=idea.idea_title, - description=idea.description, - content_type=idea.content_type, - content_structure=idea.content_structure, - cluster=idea.keyword_cluster, - idea=idea, - status='pending', - account=self.account, - site=self.site, - sector=idea.sector - ) - idea.status = 'in_progress' - idea.save() - tasks_created += 1 - - return { - 'ideas_processed': len(ideas), - 'tasks_created': tasks_created - } - - def run_stage_4(self): - """ - Stage 4: Tasks → Content (AI) - - 1. Get tasks with status='pending' - 2. Process ONE task at a time (sequential) - 3. Call GenerateContentFunction - 4. Creates Content record (independent, NOT OneToOne) - 5. Updates Task.status='completed' - 6. Auto-syncs Idea.status='completed' - """ - - config = AutomationConfig.objects.get(site=self.site) - batch_size = config.stage_4_batch_size - - tasks = Tasks.objects.filter( - site=self.site, - status='pending' - )[:batch_size] - - if not tasks: - return {'tasks_processed': 0, 'content_created': 0} - - content_created = 0 - total_credits = 0 - - for task in tasks: - from igny8_core.ai.tasks import run_ai_task - result = run_ai_task( - function_name='generate_content', - payload={'ids': [task.id]}, - account_id=self.account.id - ) - content_created += result.get('count', 0) - total_credits += result.get('credits_used', 0) - - # ⚠️ ISSUE: Uses estimated word count instead of actual - estimated_word_count = len(tasks) * 2500 # Should be Sum('word_count') - - return { - 'tasks_processed': len(tasks), - 'content_created': content_created, - 'estimated_word_count': estimated_word_count, - 'credits_used': total_credits - } - - def run_stage_5(self): - """ - Stage 5: Content → Image Prompts (AI) - - 1. Get content with status='draft' and no images - 2. Call GenerateImagePromptsFunction - 3. Creates Images records with status='pending' and prompt text - """ - - config = AutomationConfig.objects.get(site=self.site) - batch_size = config.stage_5_batch_size - - content_records = Content.objects.filter( - site=self.site, - status='draft' - ).annotate( - images_count=Count('images') - ).filter(images_count=0)[:batch_size] - - if not content_records: - return {'content_processed': 0, 'prompts_created': 0} - - from igny8_core.ai.tasks import run_ai_task - result = run_ai_task( - function_name='generate_image_prompts', - payload={'ids': [c.id for c in content_records]}, - account_id=self.account.id - ) - - return { - 'content_processed': len(content_records), - 'prompts_created': result.get('count', 0), - 'credits_used': result.get('credits_used', 0) - } - - def run_stage_6(self): - """ - Stage 6: Image Prompts → Images (AI) - - ⚠️ PARTIALLY IMPLEMENTED - - 1. Get Images with status='pending' (has prompt, no URL) - 2. Call GenerateImagesFunction - 3. Updates Images.image_url and Images.status='generated' - - NOTE: GenerateImagesFunction structure exists but image provider - API integration may need completion/testing - """ - - config = AutomationConfig.objects.get(site=self.site) - batch_size = config.stage_6_batch_size - - images = Images.objects.filter( - content__site=self.site, - status='pending' - )[:batch_size] - - if not images: - return {'images_processed': 0, 'images_generated': 0} - - # ⚠️ May fail if image provider API not complete - from igny8_core.ai.tasks import run_ai_task - result = run_ai_task( - function_name='generate_images', - payload={'ids': [img.id for img in images]}, - account_id=self.account.id - ) - - return { - 'images_processed': len(images), - 'images_generated': result.get('count', 0), - 'credits_used': result.get('credits_used', 0) - } - - def run_stage_7(self): - """ - Stage 7: Manual Review Gate (NO AI) - - This is a manual gate - automation STOPS here. - Just counts content ready for review. - - Returns count of content with: - - status='draft' - - All images generated (status='generated') - """ - - ready_content = Content.objects.filter( - site=self.site, - status='draft' - ).annotate( - pending_images=Count('images', filter=Q(images__status='pending')) - ).filter(pending_images=0) - - return { - 'content_ready_for_review': ready_content.count() - } - - def estimate_credits(self): - """ - Estimate credits needed for full automation run - - Calculates based on pending items in each stage: - - Stage 1: keywords * 0.2 (1 credit per 5 keywords) - - Stage 2: clusters * 2 - - Stage 3: 0 (local) - - Stage 4: tasks * 5 (2500 words ≈ 5 credits) - - Stage 5: content * 2 - - Stage 6: images * 1-4 - - Stage 7: 0 (manual) - """ - - # Implementation details... - pass -``` - ---- - -### 3. API Endpoints (10 Total) - -**File:** `backend/igny8_core/business/automation/views.py` (428 lines) - -**All Endpoints Working:** - -```python -class AutomationViewSet(viewsets.ViewSet): - permission_classes = [IsAuthenticated] - - @action(detail=False, methods=['get']) - def config(self, request): - """GET /api/v1/automation/config/?site_id=123""" - # Returns AutomationConfig for site - - @action(detail=False, methods=['put']) - def update_config(self, request): - """PUT /api/v1/automation/update_config/?site_id=123""" - # Updates AutomationConfig fields - - @action(detail=False, methods=['post']) - def run_now(self, request): - """POST /api/v1/automation/run_now/?site_id=123""" - # Triggers automation via Celery: run_automation_task.delay() - - @action(detail=False, methods=['get']) - def current_run(self, request): - """GET /api/v1/automation/current_run/?site_id=123""" - # Returns active AutomationRun or null - - @action(detail=False, methods=['post']) - def pause(self, request): - """POST /api/v1/automation/pause/?run_id=xxx""" - # Sets AutomationRun.status='paused' - - @action(detail=False, methods=['post']) - def resume(self, request): - """POST /api/v1/automation/resume/?run_id=xxx""" - # Resumes from current_stage via resume_automation_task.delay() - - @action(detail=False, methods=['get']) - def history(self, request): - """GET /api/v1/automation/history/?site_id=123""" - # Returns last 20 AutomationRun records - - @action(detail=False, methods=['get']) - def logs(self, request): - """GET /api/v1/automation/logs/?run_id=xxx&lines=100""" - # Returns log file content via AutomationLogger - - @action(detail=False, methods=['get']) - def estimate(self, request): - """GET /api/v1/automation/estimate/?site_id=123""" - # Returns estimated_credits, current_balance, sufficient boolean - - @action(detail=False, methods=['get']) - def pipeline_overview(self, request): - """ - GET /api/v1/automation/pipeline_overview/?site_id=123 - - ✅ BONUS ENDPOINT - Not in plan but fully implemented - - Returns pending counts for all 7 stages without running automation: - - Stage 1: Keywords.objects.filter(status='new').count() - - Stage 2: Clusters.objects.filter(status='active', ideas_count=0).count() - - Stage 3: ContentIdeas.objects.filter(status='new').count() - - Stage 4: Tasks.objects.filter(status='pending').count() - - Stage 5: Content.objects.filter(status='draft', images_count=0).count() - - Stage 6: Images.objects.filter(status='pending').count() - - Stage 7: Content.objects.filter(status='draft', all_images_generated).count() - - Used by frontend for real-time pipeline visualization - """ -``` - ---- - -### 4. Celery Tasks - -**File:** `backend/igny8_core/business/automation/tasks.py` (200 lines) - -```python -@shared_task -def check_scheduled_automations(): - """ - Runs hourly via Celery Beat - - Checks all enabled AutomationConfig records where: - - is_enabled=True - - next_run_at <= now - - Triggers run_automation_task for each - """ - now = timezone.now() - configs = AutomationConfig.objects.filter( - is_enabled=True, - next_run_at__lte=now - ) - - for config in configs: - run_automation_task.delay( - account_id=config.account.id, - site_id=config.site.id, - trigger_type='scheduled' - ) - -@shared_task(bind=True, max_retries=0) -def run_automation_task(self, account_id, site_id, trigger_type='manual'): - """ - Main automation task - executes all 7 stages - - 1. Load account and site - 2. Create AutomationService instance - 3. Call start_automation() - 4. Handle errors and update run status - 5. Release lock on failure - """ - account = Account.objects.get(id=account_id) - site = Site.objects.get(id=site_id) - - service = AutomationService(account, site) - - try: - run = service.start_automation(trigger_type=trigger_type) - return {'run_id': str(run.run_id), 'status': 'completed'} - except Exception as e: - # Log error, release lock, update run status to 'failed' - raise - -@shared_task -def resume_automation_task(run_id): - """ - Resume paused automation from current_stage - - 1. Load AutomationRun by run_id - 2. Set status='running' - 3. Continue from current_stage to 7 - 4. Update run status - """ - run = AutomationRun.objects.get(run_id=run_id) - service = AutomationService(run.account, run.site) - - run.status = 'running' - run.save() - - for stage in range(run.current_stage, 8): - if run.status == 'paused': - break - - result = service._execute_stage(stage, run) - setattr(run, f'stage_{stage}_result', result) - run.current_stage = stage + 1 - run.save() - - run.status = 'completed' - run.completed_at = timezone.now() - run.save() -``` - ---- - -## 7-Stage Pipeline Deep Dive - -### Stage Flow Diagram - -``` -Keywords (status='new') - ↓ Stage 1 (AI - AutoClusterFunction) -Clusters (status='active') - ↓ Stage 2 (AI - GenerateIdeasFunction) -ContentIdeas (status='new') - ↓ Stage 3 (LOCAL - Create Tasks) -Tasks (status='pending') - ↓ Stage 4 (AI - GenerateContentFunction) -Content (status='draft', no images) - ↓ Stage 5 (AI - GenerateImagePromptsFunction) -Images (status='pending', has prompt) - ↓ Stage 6 (AI - GenerateImagesFunction) ⚠️ Partial -Images (status='generated', has URL) - ↓ Stage 7 (MANUAL GATE) -Content (ready for review) - ↓ STOP - Manual review required -WordPress Publishing (outside automation) -``` - -### Stage Details - -| Stage | Input | AI Function | Output | Credits | Status | -|-------|-------|-------------|--------|---------|--------| -| 1 | Keywords (new) | AutoClusterFunction | Clusters created, Keywords mapped | ~0.2 per keyword | ✅ Complete | -| 2 | Clusters (active, no ideas) | GenerateIdeasFunction | ContentIdeas created | 2 per cluster | ✅ Complete | -| 3 | ContentIdeas (new) | None (Local) | Tasks created | 0 | ✅ Complete | -| 4 | Tasks (pending) | GenerateContentFunction | Content created, tasks completed | ~5 per task | ✅ Complete | -| 5 | Content (draft, no images) | GenerateImagePromptsFunction | Images with prompts | ~2 per content | ✅ Complete | -| 6 | Images (pending) | GenerateImagesFunction | Images with URLs | 1-4 per image | ⚠️ Partial | -| 7 | Content (draft, all images) | None (Manual Gate) | Count ready | 0 | ✅ Complete | - ---- - -## Configuration & Settings - -### AutomationConfig Fields - -```python -is_enabled: bool # Master on/off switch -frequency: str # 'daily' | 'weekly' | 'monthly' -scheduled_time: time # HH:MM (24-hour format) -stage_1_batch_size: int # Keywords to cluster (default: 20) -stage_2_batch_size: int # Clusters to process (default: 1) -stage_3_batch_size: int # Ideas to convert (default: 20) -stage_4_batch_size: int # Tasks to write (default: 1) -stage_5_batch_size: int # Content to extract prompts (default: 1) -stage_6_batch_size: int # Images to generate (default: 1) -last_run_at: datetime # Last execution timestamp -next_run_at: datetime # Next scheduled execution -``` - -### Recommended Batch Sizes - -Based on codebase defaults and credit optimization: - -- **Stage 1 (Keywords → Clusters):** 20-50 keywords - - Lower = more clusters, higher precision - - Higher = fewer clusters, broader grouping - -- **Stage 2 (Clusters → Ideas):** 1 cluster - - AI needs full context per cluster - - Sequential processing recommended - -- **Stage 3 (Ideas → Tasks):** 10-50 ideas - - Local operation, no credit cost - - Can process in bulk - -- **Stage 4 (Tasks → Content):** 1-5 tasks - - Most expensive stage (~5 credits per task) - - Sequential or small batches for quality - -- **Stage 5 (Content → Prompts):** 1-10 content - - Fast AI operation - - Can batch safely - -- **Stage 6 (Prompts → Images):** 1-5 images - - Depends on image provider rate limits - - Test in production - ---- - -## Comparison vs Plan - -### automation-plan.md vs Actual Implementation - -| Feature | Plan | Actual | Status | -|---------|------|--------|--------| -| **7-Stage Pipeline** | Defined | Fully implemented | ✅ Match | -| **AutomationConfig** | Specified | Implemented | ✅ Match | -| **AutomationRun** | Specified | Implemented | ✅ Match | -| **Distributed Locking** | Required | Redis-based | ✅ Match | -| **Credit Estimation** | Required | Working | ✅ Match | -| **Scheduled Runs** | Hourly check | Celery Beat task | ✅ Match | -| **Manual Triggers** | Required | API + Celery | ✅ Match | -| **Pause/Resume** | Required | Fully working | ✅ Match | -| **Run History** | Required | Last 20 runs | ✅ Match | -| **Logs** | Required | File-based | ✅ Match | -| **9 API Endpoints** | Specified | 9 + 1 bonus | ✅ Exceeded | -| **Frontend Page** | Not in plan | Fully built | ✅ Bonus | -| **Real-time Updates** | Not specified | 5s polling | ✅ Bonus | -| **Stage 6 Images** | Required | Partial | ⚠️ Needs work | - ---- - -## Gaps & Recommendations - -### Critical Gaps (Should Fix) - -1. **Stage 6 - Generate Images** - - **Status:** Function structure exists, API integration may be incomplete - - **Impact:** Automation will fail/skip image generation - - **Fix:** Complete `GenerateImagesFunction` with image provider API - - **Effort:** 2-4 hours - -2. **Word Count Calculation (Stage 4)** - - **Status:** Uses estimated `tasks * 2500` instead of actual `Sum('word_count')` - - **Impact:** Inaccurate reporting in run results - - **Fix:** Replace with: - ```python - actual_word_count = Content.objects.filter( - id__in=[result['content_id'] for result in results] - ).aggregate(total=Sum('word_count'))['total'] or 0 - ``` - - **Effort:** 15 minutes - -### Testing Needed - -1. **AutomationLogger File Paths** - - Verify logs write to correct location in production - - Test log rotation and cleanup - -2. **Stage 6 Image Generation** - - Test with actual image provider API - - Verify credits deduction - - Check error handling - -3. **Concurrent Run Prevention** - - Test Redis lock with multiple simultaneous requests - - Verify lock release on failure - -### Enhancement Opportunities - -1. **Email Notifications** - - Send email when automation completes - - Alert on failures - -2. **Slack Integration** - - Post run summary to Slack channel - -3. **Retry Logic** - - Retry failed stages (currently max_retries=0) - -4. **Stage-Level Progress** - - Show progress within each stage (e.g., "Processing 5 of 20 keywords") - ---- - -## Conclusion - -**The IGNY8 Automation System is 95% Complete and Fully Functional.** - -### What Works ✅ -- Complete 7-stage pipeline -- Full backend implementation (models, service, API, tasks) -- Complete frontend implementation (page, components, service, routing) -- Distributed locking and credit management -- Scheduled and manual execution -- Pause/resume functionality -- Real-time monitoring and logs - -### What Needs Work ⚠️ -- Stage 6 image generation API integration (minor) -- Word count calculation accuracy (cosmetic) -- Production testing of logging (validation) - -### Recommendation -**The system is PRODUCTION READY** with the caveat that Stage 6 may need completion or can be temporarily disabled. The core automation pipeline (Stages 1-5) is fully functional and delivers significant value by automating the entire content creation workflow from keywords to draft articles with image prompts. - -**Grade: A- (95/100)** - ---- - -**End of Corrected Implementation Analysis** diff --git a/docs/automation/AUTOMATION-REFERENCE.md b/docs/automation/AUTOMATION-REFERENCE.md new file mode 100644 index 00000000..c0b7a11f --- /dev/null +++ b/docs/automation/AUTOMATION-REFERENCE.md @@ -0,0 +1,148 @@ +# Automation Module (Code-Sourced, Dec 2025) + +Single canonical reference for IGNY8 automation (backend, frontend, and runtime behavior). Replaces all prior automation docs in this folder. + +--- + +## 1) What Automation Does +- Runs the 7-stage pipeline across Planner/Writer: + 1) Keywords → Clusters (AI) + 2) Clusters → Ideas (AI) + 3) Ideas → Tasks (Local) + 4) Tasks → Content (AI) + 5) Content → Image Prompts (AI) + 6) Image Prompts → Images (AI) + 7) Manual Review Gate (Manual) +- Per-site, per-account isolation. One run at a time per site; guarded by cache lock `automation_lock_{site_id}`. +- Scheduling via Celery beat (`automation.check_scheduled_automations`); execution via Celery tasks (`run_automation_task`, `resume_automation_task` / `continue_automation_task`). + +--- + +## 2) Backend API (behavior + payloads) +Base: `/api/v1/automation/` (auth required; site must belong to user’s account). + +- `GET config?site_id=`: returns or creates config with enable flag, frequency (`daily|weekly|monthly`), scheduled_time, stage_1..6 batch sizes, delays (`within_stage_delay`, `between_stage_delay`), last_run_at, next_run_at. +- `PUT update_config?site_id=`: same fields as above, updates in-place. +- `POST run_now?site_id=`: starts a manual run; enqueues `run_automation_task`. Fails if a run is already active or lock exists. +- `GET current_run?site_id=`: current running/paused run with status, current_stage, totals, and stage_1..7_result blobs (counts, credits, partial flags, skip reasons). +- `GET pipeline_overview?site_id=`: per-stage status counts and “pending” numbers for UI cards. +- `GET current_processing?site_id=&run_id=`: live processing snapshot for an active run; null if not running. +- `POST pause|resume|cancel?site_id=&run_id=`: pause after current item; resume from saved `current_stage`; cancel after current item and stamp cancelled_at/completed_at. +- `GET history?site_id=`: last 20 runs (id, status, trigger, timestamps, total_credits_used, current_stage). +- `GET logs?run_id=&lines=100`: tail of the per-run activity log written by AutomationLogger. +- `GET estimate?site_id=`: estimated_credits, current_balance, sufficient (balance >= 1.2x estimate). + +Error behaviors: +- Missing site_id/run_id → 400. +- Site not in account → 404. +- Run not found → 404 on run-specific endpoints. +- Already running / lock held → 400 on run_now. + +--- + +## 3) Data Model (runtime state) +- `AutomationConfig` (one per site): enable flag, schedule (frequency, time), batch sizes per stage (1–6), delays (within-stage, between-stage), last_run_at, next_run_at. +- `AutomationRun`: run_id, trigger_type (manual/scheduled), status (running/paused/cancelled/completed/failed), current_stage, timestamps (start/pause/resume/cancel/complete), total_credits_used, per-stage result JSON (stage_1_result … stage_7_result), error_message. +- Activity logs: one file per run via AutomationLogger; streamed through the `logs` endpoint. + +--- + +## 4) How Execution Works (AutomationService) +- Start: grabs cache lock `automation_lock_{site_id}`, estimates credits, enforces 1.2x balance check, creates AutomationRun and log file. +- AI functions used: Stage 1 `AutoClusterFunction`; Stage 2 `GenerateIdeasFunction`; Stage 4 `GenerateContentFunction`; Stage 5 `GenerateImagePromptsFunction`; Stage 6 uses `process_image_generation_queue` (not the partial `generate_images` AI function). +- Stage flow (per code): + - Stage 1 Keywords → Clusters: require ≥5 keywords (validate_minimum_keywords); batch by config; AIEngine clustering; records keywords_processed, clusters_created, batches, credits, time; skips if insufficient keywords. + - Stage 2 Clusters → Ideas: batch by config; AIEngine ideas; records ideas_created. + - Stage 3 Ideas → Tasks: local conversion of queued ideas to tasks; batches by config; no AI. + - Stage 4 Tasks → Content: batch by config; AIEngine content; records content count + word totals. + - Stage 5 Content → Image Prompts: batch by config; AIEngine image-prompts into Images (featured + in-article). + - Stage 6 Image Prompts → Images: uses `process_image_generation_queue` with provider/model from IntegrationSettings; updates Images status. + - Stage 7 Manual Review Gate: marks ready-for-review counts; no AI. +- Control: each stage checks `_check_should_stop` (paused/cancelled); saves partial progress (counts, credits) before returning; resume continues from `current_stage`. +- Credits: upfront estimate check (1.2x buffer) before starting; AIEngine per-call pre-checks and post-SAVE deductions; `total_credits_used` accumulates. +- Locks: acquired on start; cleared on completion or failure; also cleared on fatal errors in tasks. +- Errors: any unhandled exception marks run failed, sets error_message, logs error, clears lock; pipeline_overview/history reflect status. +- Stage result fields (persisted): + - S1: keywords_processed, clusters_created, batches_run, credits_used, skipped/partial flags, time_elapsed. + - S2: clusters_processed, ideas_created, batches_run, credits_used. + - S3: ideas_processed, tasks_created, batches_run. + - S4: tasks_processed, content_created, total_words, batches_run, credits_used. + - S5: content_processed, prompts_created, batches_run, credits_used. + - S6: images_processed, images_generated, batches_run. + - S7: ready_for_review counts. + +Batching & delays: +- Configurable per site; stage_1..6 batch sizes control how many items per batch; `within_stage_delay` pauses between batches; `between_stage_delay` between stages. + +Scheduling: +- `check_scheduled_automations` runs hourly; respects frequency/time and last_run_at (~23h guard); skips if a run is active; sets next_run_at; starts `run_automation_task`. + +Celery execution: +- `run_automation_task` runs stages 1→7 sequentially for a run_id; failures mark run failed and clear lock. +- `resume_automation_task` / `continue_automation_task` continue from saved `current_stage`. +- Workers need access to cache (locks) and IntegrationSettings (models/providers). + +Image pipeline specifics: +- Stage 5 writes prompts to Images (featured + ordered in-article). +- Stage 6 generates images via queue helper; AI `generate_images` remains partial/broken and is not used by automation. + +--- + +## 5) Frontend Behavior (AutomationPage) +- Route: `/automation`. +- What the user can do: run now, pause, resume, cancel; edit config (enable/schedule, batch sizes, delays); view activity log; view history; watch live processing card and pipeline cards update. +- Polling: every ~5s while a run is running/paused for current_run, pipeline_overview, metrics, current_processing; lighter polling when idle. +- Metrics: fetched via low-level endpoints (keywords/clusters/ideas/tasks/content/images) for authoritative counts. +- States shown: running, paused, cancelled, failed, completed; processing card shown when a run exists; pipeline cards use “pending” counts from pipeline_overview. +- Activity log: pulled from `logs` endpoint; shown in UI for live tailing. + +--- + +## 6) Configuration & Dependencies +- Needs IntegrationSettings for AI models and image providers (OpenAI/runware). +- Requires Celery beat and workers; cache backend required for locks. +- Tenant scoping everywhere: site + account filtering on all automation queries. + +--- + +## 7) Known Limitations and Gaps +- `generate_images` AI function is partial/broken; automation uses queue helper instead. +- Pause/Cancel stop after the current item; no mid-item abort. +- Batch defaults are conservative (e.g., stage_2=1, stage_4=1); tune per site for throughput. +- Stage 7 is manual; no automated review step. +- No automated test suite observed for automation pipeline (stage transitions, pause/resume/cancel, scheduling guards, credit estimation/deduction). +- Enhancements to consider: fix or replace `generate_images`; add mid-item abort; surface lock status/owner; broaden batch defaults after validation; add operator-facing doc in app; add tests. + +--- + +## 8) Field/Behavior Quick Tables + +### Pipeline “pending” definitions (pipeline_overview) +- Stage 1: Keywords with status `new`, cluster is null, not disabled. +- Stage 2: Clusters status `new`, not disabled, with no ideas. +- Stage 3: ContentIdeas status `new`. +- Stage 4: Tasks status `queued`. +- Stage 5: Content status `draft` with zero images. +- Stage 6: Images status `pending`. +- Stage 7: Content status `review`. + +### Stage result fields (stored on AutomationRun) +- S1: keywords_processed, clusters_created, batches_run, credits_used, skipped, partial, time_elapsed. +- S2: clusters_processed, ideas_created, batches_run, credits_used. +- S3: ideas_processed, tasks_created, batches_run. +- S4: tasks_processed, content_created, total_words, batches_run, credits_used. +- S5: content_processed, prompts_created, batches_run, credits_used. +- S6: images_processed, images_generated, batches_run. +- S7: ready_for_review. + +### Credit handling +- Pre-run: estimate_credits * 1.2 vs account.credits (fails if insufficient). +- Per AI call: AIEngine pre-check credits; post-SAVE deduction with cost/tokens tracked; total_credits_used aggregates deductions. + +### Logging +- Per-run log file via AutomationLogger; accessed with `GET logs?run_id=&lines=`; includes stage start/progress/errors and batch info. + +### Polling (frontend) +- Active run: ~5s cadence for current_run, pipeline_overview, metrics, current_processing, logs tail. +- Idle: lighter polling (current_run/pipeline_overview) to show readiness and pending counts. + diff --git a/docs/automation/auto-cluster-validation-fix-plan.md b/docs/automation/auto-cluster-validation-fix-plan.md deleted file mode 100644 index 4750af96..00000000 --- a/docs/automation/auto-cluster-validation-fix-plan.md +++ /dev/null @@ -1,753 +0,0 @@ -# Auto-Cluster Validation Fix Plan - -**Date:** December 4, 2025 -**Status:** Design Phase -**Priority:** MEDIUM - ---- - -## 🎯 OBJECTIVE - -Add validation to prevent auto-cluster from running with less than 5 keywords, and ensure both manual auto-cluster and automation pipeline use the same shared validation logic to maintain consistency. - ---- - -## 🔍 CURRENT STATE ANALYSIS - -### Current Behavior - -**Auto-Cluster Function:** -- Located in: `backend/igny8_core/ai/functions/auto_cluster.py` -- No minimum keyword validation -- Accepts any number of keywords (even 1) -- May produce poor quality clusters with insufficient data - -**Automation Pipeline:** -- Located in: `backend/igny8_core/business/automation/services/automation_service.py` -- Uses auto-cluster in Stage 1 -- No pre-check for minimum keywords -- May waste credits on insufficient data - -### Problems - -1. ❌ **No Minimum Check:** Auto-cluster runs with 1-4 keywords -2. ❌ **Poor Results:** AI cannot create meaningful clusters with < 5 keywords -3. ❌ **Wasted Credits:** Charges credits for insufficient analysis -4. ❌ **Inconsistent Validation:** No shared validation between manual and automation -5. ❌ **User Confusion:** Error occurs during processing, not at selection - ---- - -## ✅ PROPOSED SOLUTION - -### Validation Strategy - -**Single Source of Truth:** -- Create one validation function -- Use it in both auto-cluster function AND automation pipeline -- Consistent error messages -- No code duplication - -**Error Behavior:** -- **Manual Auto-Cluster:** Return error before API call -- **Automation Pipeline:** Skip Stage 1 with warning in logs - ---- - -## 📋 IMPLEMENTATION PLAN - -### Step 1: Create Shared Validation Module - -**New File:** `backend/igny8_core/ai/validators/cluster_validators.py` - -```python -""" -Cluster-specific validators -Shared between auto-cluster function and automation pipeline -""" -import logging -from typing import Dict, List - -logger = logging.getLogger(__name__) - - -def validate_minimum_keywords( - keyword_ids: List[int], - account=None, - min_required: int = 5 -) -> Dict: - """ - Validate that sufficient keywords are available for clustering - - Args: - keyword_ids: List of keyword IDs to cluster - account: Account object for filtering - min_required: Minimum number of keywords required (default: 5) - - Returns: - Dict with 'valid' (bool) and 'error' (str) or 'count' (int) - """ - from igny8_core.modules.planner.models import Keywords - - # Build queryset - queryset = Keywords.objects.filter(id__in=keyword_ids, status='new') - - if account: - queryset = queryset.filter(account=account) - - # Count available keywords - count = queryset.count() - - # Validate minimum - if count < min_required: - return { - 'valid': False, - 'error': f'Insufficient keywords for clustering. Need at least {min_required} keywords, but only {count} available.', - 'count': count, - 'required': min_required - } - - return { - 'valid': True, - 'count': count, - 'required': min_required - } - - -def validate_keyword_selection( - selected_ids: List[int], - available_count: int, - min_required: int = 5 -) -> Dict: - """ - Validate keyword selection (for frontend validation) - - Args: - selected_ids: List of selected keyword IDs - available_count: Total count of available keywords - min_required: Minimum required - - Returns: - Dict with validation result - """ - selected_count = len(selected_ids) - - # Check if any keywords selected - if selected_count == 0: - return { - 'valid': False, - 'error': 'No keywords selected', - 'type': 'NO_SELECTION' - } - - # Check if enough selected - if selected_count < min_required: - return { - 'valid': False, - 'error': f'Please select at least {min_required} keywords. Currently selected: {selected_count}', - 'type': 'INSUFFICIENT_SELECTION', - 'selected': selected_count, - 'required': min_required - } - - # Check if enough available (even if not all selected) - if available_count < min_required: - return { - 'valid': False, - 'error': f'Not enough keywords available. Need at least {min_required} keywords, but only {available_count} exist.', - 'type': 'INSUFFICIENT_AVAILABLE', - 'available': available_count, - 'required': min_required - } - - return { - 'valid': True, - 'selected': selected_count, - 'available': available_count, - 'required': min_required - } -``` - -### Step 2: Update Auto-Cluster Function - -**File:** `backend/igny8_core/ai/functions/auto_cluster.py` - -**Add import:** -```python -from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords -``` - -**Update validate() method:** -```python -def validate(self, payload: dict, account=None) -> Dict: - """Validate keyword IDs and minimum count""" - result = super().validate(payload, account) - if not result['valid']: - return result - - keyword_ids = payload.get('keyword_ids', []) - - if not keyword_ids: - return {'valid': False, 'error': 'No keyword IDs provided'} - - # NEW: Validate minimum keywords using shared validator - min_validation = validate_minimum_keywords( - keyword_ids=keyword_ids, - account=account, - min_required=5 # Configurable constant - ) - - if not min_validation['valid']: - # Log the validation failure - logger.warning( - f"[AutoCluster] Validation failed: {min_validation['error']}" - ) - return min_validation - - # Log successful validation - logger.info( - f"[AutoCluster] Validation passed: {min_validation['count']} keywords available (min: {min_validation['required']})" - ) - - return {'valid': True} -``` - -### Step 3: Update Automation Pipeline - -**File:** `backend/igny8_core/business/automation/services/automation_service.py` - -**Add import:** -```python -from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords -``` - -**Update run_stage_1() method:** -```python -def run_stage_1(self): - """Stage 1: Keywords → Clusters (AI)""" - stage_number = 1 - stage_name = "Keywords → Clusters (AI)" - start_time = time.time() - - # Query pending keywords - pending_keywords = Keywords.objects.filter( - site=self.site, - status='new' - ) - - total_count = pending_keywords.count() - - # NEW: Pre-stage validation for minimum keywords - keyword_ids = list(pending_keywords.values_list('id', flat=True)) - - min_validation = validate_minimum_keywords( - keyword_ids=keyword_ids, - account=self.account, - min_required=5 - ) - - if not min_validation['valid']: - # Log validation failure - self.logger.log_stage_start( - self.run.run_id, self.account.id, self.site.id, - stage_number, stage_name, total_count - ) - - error_msg = min_validation['error'] - self.logger.log_stage_error( - self.run.run_id, self.account.id, self.site.id, - stage_number, error_msg - ) - - # Skip stage with proper result - self.run.stage_1_result = { - 'keywords_processed': 0, - 'clusters_created': 0, - 'skipped': True, - 'skip_reason': error_msg, - 'credits_used': 0 - } - self.run.current_stage = 2 - self.run.save() - - logger.warning(f"[AutomationService] Stage 1 skipped: {error_msg}") - return - - # Log stage start - self.logger.log_stage_start( - self.run.run_id, self.account.id, self.site.id, - stage_number, stage_name, total_count - ) - - # ... rest of existing stage logic ... -``` - -### Step 4: Update API Endpoint - -**File:** `backend/igny8_core/modules/planner/views.py` (KeywordsViewSet) - -**Update auto_cluster action:** -```python -@action(detail=False, methods=['post'], url_path='auto_cluster', url_name='auto_cluster') -def auto_cluster(self, request): - """Auto-cluster keywords using AI""" - from igny8_core.ai.tasks import run_ai_task - from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords - - account = getattr(request, 'account', None) - keyword_ids = request.data.get('ids', []) - - if not keyword_ids: - return error_response( - error='No keyword IDs provided', - status_code=status.HTTP_400_BAD_REQUEST, - request=request - ) - - # NEW: Validate minimum keywords BEFORE queuing task - validation = validate_minimum_keywords( - keyword_ids=keyword_ids, - account=account, - min_required=5 - ) - - if not validation['valid']: - return error_response( - error=validation['error'], - status_code=status.HTTP_400_BAD_REQUEST, - request=request, - extra_data={ - 'count': validation.get('count'), - 'required': validation.get('required') - } - ) - - # Validation passed - proceed with clustering - account_id = account.id if account else None - - try: - if hasattr(run_ai_task, 'delay'): - task = run_ai_task.delay( - function_name='auto_cluster', - payload={'keyword_ids': keyword_ids}, - account_id=account_id - ) - return success_response( - data={'task_id': str(task.id)}, - message=f'Auto-cluster started with {validation["count"]} keywords', - request=request - ) - else: - # Synchronous fallback - result = run_ai_task( - function_name='auto_cluster', - payload={'keyword_ids': keyword_ids}, - account_id=account_id - ) - return success_response(data=result, request=request) - - except Exception as e: - logger.error(f"Failed to start auto-cluster: {e}", exc_info=True) - return error_response( - error=f'Failed to start clustering: {str(e)}', - status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, - request=request - ) -``` - -### Step 5: Add Frontend Validation (Optional but Recommended) - -**File:** `frontend/src/pages/Planner/Keywords.tsx` - -**Update handleAutoCluster function:** -```typescript -const handleAutoCluster = async () => { - try { - const selectedIds = selectedKeywords.map(k => k.id); - - // Frontend validation (pre-check before API call) - if (selectedIds.length < 5) { - toast.error( - `Please select at least 5 keywords for auto-clustering. Currently selected: ${selectedIds.length}`, - { duration: 5000 } - ); - return; - } - - // Check total available - const availableCount = keywords.filter(k => k.status === 'new').length; - if (availableCount < 5) { - toast.error( - `Not enough keywords available. Need at least 5 keywords, but only ${availableCount} exist.`, - { duration: 5000 } - ); - return; - } - - // Proceed with API call - const result = await autoClusterKeywords(selectedIds); - - if (result.task_id) { - toast.success(`Auto-cluster started with ${selectedIds.length} keywords`); - setTaskId(result.task_id); - } else { - toast.error('Failed to start auto-cluster'); - } - - } catch (error: any) { - // Backend validation error (in case frontend check was bypassed) - const errorMsg = error.response?.data?.error || error.message; - toast.error(errorMsg); - } -}; -``` - ---- - -## 🗂️ FILE STRUCTURE - -### New Files -``` -backend/igny8_core/ai/validators/ -├── __init__.py -└── cluster_validators.py (NEW) -``` - -### Modified Files -``` -backend/igny8_core/ai/functions/auto_cluster.py -backend/igny8_core/business/automation/services/automation_service.py -backend/igny8_core/modules/planner/views.py -frontend/src/pages/Planner/Keywords.tsx -``` - ---- - -## 🧪 TESTING PLAN - -### Unit Tests - -**File:** `backend/igny8_core/ai/validators/tests/test_cluster_validators.py` - -```python -import pytest -from django.test import TestCase -from igny8_core.ai.validators.cluster_validators import ( - validate_minimum_keywords, - validate_keyword_selection -) -from igny8_core.modules.planner.models import Keywords -from igny8_core.auth.models import Account, Site - - -class ClusterValidatorsTestCase(TestCase): - def setUp(self): - self.account = Account.objects.create(name='Test Account') - self.site = Site.objects.create(name='Test Site', account=self.account) - - def test_validate_minimum_keywords_success(self): - """Test with sufficient keywords (>= 5)""" - # Create 10 keywords - keyword_ids = [] - for i in range(10): - kw = Keywords.objects.create( - keyword=f'keyword {i}', - status='new', - account=self.account, - site=self.site - ) - keyword_ids.append(kw.id) - - result = validate_minimum_keywords(keyword_ids, self.account) - - assert result['valid'] is True - assert result['count'] == 10 - assert result['required'] == 5 - - def test_validate_minimum_keywords_failure(self): - """Test with insufficient keywords (< 5)""" - # Create only 3 keywords - keyword_ids = [] - for i in range(3): - kw = Keywords.objects.create( - keyword=f'keyword {i}', - status='new', - account=self.account, - site=self.site - ) - keyword_ids.append(kw.id) - - result = validate_minimum_keywords(keyword_ids, self.account) - - assert result['valid'] is False - assert 'Insufficient keywords' in result['error'] - assert result['count'] == 3 - assert result['required'] == 5 - - def test_validate_minimum_keywords_edge_case_exactly_5(self): - """Test with exactly 5 keywords (boundary)""" - keyword_ids = [] - for i in range(5): - kw = Keywords.objects.create( - keyword=f'keyword {i}', - status='new', - account=self.account, - site=self.site - ) - keyword_ids.append(kw.id) - - result = validate_minimum_keywords(keyword_ids, self.account) - - assert result['valid'] is True - assert result['count'] == 5 - - def test_validate_keyword_selection_insufficient(self): - """Test frontend selection validation""" - result = validate_keyword_selection( - selected_ids=[1, 2, 3], # Only 3 - available_count=10, - min_required=5 - ) - - assert result['valid'] is False - assert result['type'] == 'INSUFFICIENT_SELECTION' - assert result['selected'] == 3 - assert result['required'] == 5 -``` - -### Integration Tests - -```python -class AutoClusterIntegrationTestCase(TestCase): - def test_auto_cluster_with_insufficient_keywords(self): - """Test auto-cluster endpoint rejects < 5 keywords""" - # Create only 3 keywords - keyword_ids = self._create_keywords(3) - - response = self.client.post( - '/api/planner/keywords/auto_cluster/', - data={'ids': keyword_ids}, - HTTP_AUTHORIZATION=f'Bearer {self.token}' - ) - - assert response.status_code == 400 - assert 'Insufficient keywords' in response.json()['error'] - - def test_automation_skips_stage_1_with_insufficient_keywords(self): - """Test automation skips Stage 1 if < 5 keywords""" - # Create only 2 keywords - self._create_keywords(2) - - # Start automation - run_id = self.automation_service.start_automation('manual') - - # Verify Stage 1 was skipped - run = AutomationRun.objects.get(run_id=run_id) - assert run.stage_1_result['skipped'] is True - assert 'Insufficient keywords' in run.stage_1_result['skip_reason'] - assert run.current_stage == 2 # Moved to next stage -``` - -### Manual Test Cases - -- [ ] **Test 1:** Try auto-cluster with 0 keywords selected - - Expected: Error message "No keywords selected" - -- [ ] **Test 2:** Try auto-cluster with 3 keywords selected - - Expected: Error message "Please select at least 5 keywords. Currently selected: 3" - -- [ ] **Test 3:** Try auto-cluster with exactly 5 keywords - - Expected: Success, clustering starts - -- [ ] **Test 4:** Run automation with 2 keywords in site - - Expected: Stage 1 skipped with warning in logs - -- [ ] **Test 5:** Run automation with 10 keywords in site - - Expected: Stage 1 runs normally - ---- - -## 📊 ERROR MESSAGES - -### Frontend (User-Facing) - -**No Selection:** -``` -❌ No keywords selected -Please select keywords to cluster. -``` - -**Insufficient Selection:** -``` -❌ Please select at least 5 keywords for auto-clustering -Currently selected: 3 keywords -You need at least 5 keywords to create meaningful clusters. -``` - -**Insufficient Available:** -``` -❌ Not enough keywords available -Need at least 5 keywords, but only 2 exist. -Add more keywords before running auto-cluster. -``` - -### Backend (Logs) - -**Validation Failed:** -``` -[AutoCluster] Validation failed: Insufficient keywords for clustering. Need at least 5 keywords, but only 3 available. -``` - -**Validation Passed:** -``` -[AutoCluster] Validation passed: 15 keywords available (min: 5) -``` - -**Automation Stage Skipped:** -``` -[AutomationService] Stage 1 skipped: Insufficient keywords for clustering. Need at least 5 keywords, but only 2 available. -``` - ---- - -## 🎯 CONFIGURATION - -### Constants File - -**File:** `backend/igny8_core/ai/constants.py` (or create if doesn't exist) - -```python -""" -AI Function Configuration Constants -""" - -# Cluster Configuration -MIN_KEYWORDS_FOR_CLUSTERING = 5 # Minimum keywords needed for meaningful clusters -OPTIMAL_KEYWORDS_FOR_CLUSTERING = 20 # Recommended for best results - -# Other AI limits... -``` - -**Usage in validators:** -```python -from igny8_core.ai.constants import MIN_KEYWORDS_FOR_CLUSTERING - -def validate_minimum_keywords(keyword_ids, account=None): - min_required = MIN_KEYWORDS_FOR_CLUSTERING - # ... validation logic -``` - ---- - -## 🔄 SHARED VALIDATION PATTERN - -### Why This Approach Works - -**✅ Single Source of Truth:** -- One function: `validate_minimum_keywords()` -- Used by both auto-cluster function and automation -- Update in one place applies everywhere - -**✅ Consistent Behavior:** -- Same error messages -- Same validation logic -- Same minimum requirements - -**✅ Easy to Maintain:** -- Want to change minimum from 5 to 10? Change one constant -- Want to add new validation? Add to one function -- Want to test? Test one module - -**✅ No Code Duplication:** -- DRY principle followed -- Reduces bugs from inconsistency -- Easier code review - -### Pattern for Future Validators - -```python -# backend/igny8_core/ai/validators/content_validators.py - -def validate_minimum_content_length(content_text: str, min_words: int = 100): - """ - Shared validator for content minimum length - Used by: GenerateContentFunction, Automation Stage 4, Content creation - """ - word_count = len(content_text.split()) - - if word_count < min_words: - return { - 'valid': False, - 'error': f'Content too short. Minimum {min_words} words required, got {word_count}.' - } - - return {'valid': True, 'word_count': word_count} -``` - ---- - -## 🚀 IMPLEMENTATION STEPS - -### Phase 1: Create Validator (Day 1) -- [ ] Create `cluster_validators.py` -- [ ] Implement `validate_minimum_keywords()` -- [ ] Implement `validate_keyword_selection()` -- [ ] Write unit tests - -### Phase 2: Integrate Backend (Day 1) -- [ ] Update `AutoClusterFunction.validate()` -- [ ] Update `AutomationService.run_stage_1()` -- [ ] Update `KeywordsViewSet.auto_cluster()` -- [ ] Write integration tests - -### Phase 3: Frontend (Day 2) -- [ ] Add frontend validation in Keywords page -- [ ] Add user-friendly error messages -- [ ] Test error scenarios - -### Phase 4: Testing & Deployment (Day 2) -- [ ] Run all tests -- [ ] Manual QA testing -- [ ] Deploy to production -- [ ] Monitor first few auto-cluster runs - ---- - -## 🎯 SUCCESS CRITERIA - -✅ Auto-cluster returns error if < 5 keywords selected -✅ Automation skips Stage 1 if < 5 keywords available -✅ Both use same validation function (no duplication) -✅ Clear error messages guide users -✅ Frontend validation provides instant feedback -✅ Backend validation catches edge cases -✅ All tests pass -✅ No regression in existing functionality - ---- - -## 📈 FUTURE ENHANCEMENTS - -### V2 Features - -1. **Configurable Minimum:** - - Allow admin to set minimum via settings - - Default: 5, Range: 3-20 - -2. **Quality Scoring:** - - Show quality indicator based on keyword count - - 5-10: "Fair", 11-20: "Good", 21+: "Excellent" - -3. **Smart Recommendations:** - - "You have 4 keywords. Add 1 more for best results" - - "15 keywords selected. Good for clustering!" - -4. **Batch Size Validation:** - - Warn if too many keywords selected (> 100) - - Suggest splitting into multiple runs - ---- - -## END OF PLAN - -This plan ensures robust, consistent validation for auto-cluster across all entry points (manual and automation) using shared, well-tested validation logic. diff --git a/docs/automation/automation-plan.md b/docs/automation/automation-plan.md deleted file mode 100644 index c163e4c6..00000000 --- a/docs/automation/automation-plan.md +++ /dev/null @@ -1,1576 +0,0 @@ -# AI Automation Pipeline - Complete Implementation Plan -**Version:** 2.0 -**Date:** December 3, 2025 -**Scope:** Site-level automation orchestrating existing AI functions - ---- - -## 🎯 CORE ARCHITECTURE DECISIONS - -### Decision 1: Site-Level Automation (NO Sector) -**Rationale:** -- User manages automation per website, not per topic/sector -- Simpler UX - single site selector at top of page -- Database queries filter by `site_id` only (no sector_id filtering) -- Content naturally spans multiple sectors within a site -- One automation schedule per site (not per site/sector combination) - -**Implementation:** -- Remove sector dropdown from automation page UI -- AutomationRun model: Remove sector foreign key -- AutomationConfig model: One config per site (not per site+sector) -- All stage database queries: `.filter(site=site)` (no sector filter) - ---- - -### Decision 2: Single Global Automation Page -**Why:** -- Complete pipeline visibility in one place (Keywords → Draft Content) -- Configure one schedule for entire lifecycle -- See exactly where pipeline is stuck or running -- Cleaner UX - no jumping between module pages - -**Location:** `/automation` (new route below Sites in sidebar) - ---- - -### Decision 3: Strictly Sequential Stages (Never Parallel) -**Critical Principle:** -- Stage N+1 ONLY starts when Stage N is 100% complete -- Within each stage: process items in batches sequentially -- Hard stop between stages to verify completion -- Only ONE stage active at a time per site - -**Example Flow:** -``` -Stage 1 starts → processes ALL batches → completes 100% - ↓ (trigger next) -Stage 2 starts → processes ALL batches → completes 100% - ↓ (trigger next) -Stage 3 starts → ... -``` - -**Never:** -- Run stages in parallel -- Start next stage while current stage has pending items -- Skip verification between stages - ---- - -### Decision 4: Automation Stops Before Publishing -**Manual Review Gate (Stage 7):** -- Automation ends when content reaches `status='draft'` with all images generated -- User manually reviews content quality, accuracy, brand voice -- User manually publishes via existing bulk actions on Content page -- No automated WordPress publishing (requires human oversight) - -**Rationale:** -- Content quality control needed -- Publishing has real consequences (public-facing) -- Legal/compliance review may be required -- Brand voice verification essential - ---- - -## 📊 EXISTING AI FUNCTIONS (Zero Duplication) - -``` -┌──────────────────────────────────────────────────────────────┐ -│ 🤖 AI AUTOMATION PIPELINE │ -│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │ -│ │ -│ ⏰ SCHEDULE │ -│ Next Run: Tomorrow at 2:00 AM (in 16 hours) │ -│ Frequency: [Daily ▼] at [02:00 ▼] │ -│ Status: ● Scheduled │ -│ │ -│ [Run Now] [Pause Schedule] [Configure] │ -│ │ -├──────────────────────────────────────────────────────────────┤ -│ 📊 PIPELINE OVERVIEW │ -│ │ -│ Keywords ──→ Clusters ──→ Ideas ──→ Tasks ──→ Content │ -│ 47 pending 42 20 generating │ -│ pending Stage 1 ready queued Stage 5 │ -│ │ -│ Overall Progress: ━━━━━━━╸ 62% (Stage 5/7) │ -│ Estimated Completion: 2 hours 15 minutes │ -│ │ -└──────────────────────────────────────────────────────────────┘ - -┌──────────────────────────────────────────────────────────────┐ -│ STAGE 1: Keywords → Clusters (AI) │ -│ Status: ✓ Completed │ -│ • Processed: 60 keywords → 8 clusters │ -│ • Time: 2m 30s | Credits: 12 │ -│ [View Details] [Retry Failed] │ -└──────────────────────────────────────────────────────────────┘ - -┌──────────────────────────────────────────────────────────────┐ -│ STAGE 2: Clusters → Ideas (AI) │ -│ Status: ✓ Completed │ -│ • Processed: 8 clusters → 56 ideas │ -│ • Time: 8m 15s | Credits: 16 │ -│ [View Details] │ -└──────────────────────────────────────────────────────────────┘ - -┌──────────────────────────────────────────────────────────────┐ -│ STAGE 3: Ideas → Tasks (Local Queue) │ -│ Status: ✓ Completed │ -│ • Processed: 42 ideas → 42 tasks │ -│ • Time: Instant | Credits: 0 │ -│ [View Details] │ -└──────────────────────────────────────────────────────────────┘ - -┌──────────────────────────────────────────────────────────────┐ -│ STAGE 4: Tasks → Content (AI) │ -│ Status: ● Processing (Task 3/20) │ -│ • Current: "Ultimate Coffee Bean Guide" ━━━━╸ 65% │ -│ • Progress: 2 completed, 1 processing, 17 queued │ -│ • Time: 45m elapsed | Credits: 38 used │ -│ • ETA: 1h 30m remaining │ -│ [View Details] [Pause Stage] │ -└──────────────────────────────────────────────────────────────┘ - -┌──────────────────────────────────────────────────────────────┐ -│ STAGE 5: Content → Image Prompts (AI) │ -│ Status: ⏸ Waiting (Stage 4 must complete) │ -│ • Pending: 2 content pieces ready for prompts │ -│ • Queue: Will process when Stage 4 completes │ -│ [View Details] [Trigger Now] │ -└──────────────────────────────────────────────────────────────┘ - -┌──────────────────────────────────────────────────────────────┐ -│ STAGE 6: Image Prompts → Generated Images (AI) │ -│ Status: ⏸ Waiting │ -│ • Pending: 0 prompts ready │ -│ [View Details] │ -└──────────────────────────────────────────────────────────────┘ - -┌──────────────────────────────────────────────────────────────┐ -│ STAGE 7: Content → Review (Manual Gate) 🚫 STOPS HERE │ -│ Status: ⏸ Awaiting Manual Review │ -│ • Ready for Review: 2 content pieces │ -│ • Note: Automation stops here. User reviews manually. │ -│ [Go to Review Page] │ -└──────────────────────────────────────────────────────────────┘ - -━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -📋 LIVE ACTIVITY LOG (Last 50 events) -├─ 14:23:45 - Stage 4: Started content generation for Task 3 -├─ 14:24:12 - Stage 4: Writing sections (65% complete) -├─ 14:22:30 - Stage 4: Completed Task 2 → Content created -├─ 14:20:15 - Stage 4: Started content generation for Task 2 -├─ 14:18:45 - Stage 4: Completed Task 1 → Content created -└─ [View Full Log] - -💰 TOTAL CREDITS USED THIS RUN: 66 credits -``` - -**All 6 AI Functions Already Exist and Work:** - -| Function | File Location | Input | Output | Credits | Status | -|----------|---------------|-------|--------|---------|--------| -| **auto_cluster** | `ai/functions/auto_cluster.py` | Keyword IDs (max 20) | Clusters created | 1 per 5 keywords | ✅ Working | -| **generate_ideas** | `ai/functions/generate_ideas.py` | Cluster IDs (max 5) | Ideas created | 2 per cluster | ✅ Working | -| **bulk_queue_to_writer** | `modules/planner/views.py` (line 1014) | Idea IDs | Tasks created | 0 (local) | ✅ Working | -| **generate_content** | `ai/functions/generate_content.py` | Task IDs (1 at a time) | Content draft | 1 per 500 words | ✅ Working | -| **generate_image_prompts** | `ai/functions/generate_image_prompts.py` | Content IDs | Image prompts | 0.5 per prompt | ✅ Working | -| **generate_images** | `ai/functions/generate_images.py` | Image prompt IDs | Generated images | 1-4 per image | ✅ Working | - ---- - -### 🚫 WHAT AI FUNCTIONS ALREADY DO (DO NOT DUPLICATE) - -**Credit Management** (Fully Automated in `ai/engine.py`): -```python -# Line 395 in AIEngine.execute(): -CreditService.deduct_credits_for_operation( - account=account, - operation_type=self._get_operation_type(), - amount=self._get_actual_amount(), - ... -) -``` -- ✅ Credits are AUTOMATICALLY deducted after successful save -- ✅ Credit calculation happens in `_get_actual_amount()` and `_get_operation_type()` -- ❌ Automation does NOT need to call `CreditService` manually -- ❌ Automation does NOT need to calculate credit costs - -**Status Updates** (Handled Inside AI Functions): -- ✅ Keywords: `status='new'` → `status='mapped'` (in auto_cluster save_output) -- ✅ Clusters: Created with `status='new'` (in auto_cluster save_output) -- ✅ Ideas: `status='new'` → `status='queued'` (in bulk_queue_to_writer) -- ✅ Tasks: Created with `status='queued'`, → `status='completed'` (in generate_content) -- ✅ Content: Created with `status='draft'`, → `status='review'` ONLY when all images complete (ai/tasks.py line 723) -- ✅ Images: `status='pending'` → `status='generated'` (in generate_images save_output) -- ❌ Automation does NOT update these statuses directly - -**Progress Tracking** (Event-Based System Already Exists): -- ✅ `StepTracker` and `ProgressTracker` emit real-time events during AI execution -- ✅ Each AI function has 6 phases: `INIT`, `PREP`, `AI_CALL`, `PARSE`, `SAVE`, `DONE` -- ✅ Phase descriptions available in function metadata: `get_metadata()` -- ❌ Automation does NOT need to poll progress every 2 seconds -- ❌ Automation listens to existing phase events via Celery task status - -**Error Handling & Logging**: -- ✅ AIEngine wraps execution in try/catch, logs to `AIUsageLog` -- ✅ Failed operations rollback database changes automatically -- ❌ Automation only needs to check final task result (success/failure) - ---- - -**Automation Service ONLY Does:** -1. **Batch Selection**: Query database for items to process (by status and site) -2. **Function Calling**: Call existing AI functions with selected item IDs -3. **Stage Sequencing**: Wait for Stage N completion before starting Stage N+1 -4. **Scheduling**: Trigger automation runs on configurable schedules -5. **Aggregation**: Collect results from all batches and log totals per stage - ---- - -## 🏗️ 7-STAGE PIPELINE ARCHITECTURE - -### Sequential Stage Flow - -| Stage | From | To | Function Used | Batch Size | Type | -|-------|------|-----|---------------|------------|------| -| **1** | Keywords (`status='new'`, `cluster_id=null`) | Clusters (`status='new'`) | `auto_cluster` | 20 keywords | AI | -| **2** | Clusters (`status='new'`, no ideas) | Ideas (`status='new'`) | `generate_ideas` | 1 cluster | AI | -| **3** | Ideas (`status='new'`) | Tasks (`status='queued'`) | `bulk_queue_to_writer` | 20 ideas | Local | -| **4** | Tasks (`status='queued'`) | Content (`status='draft'`) | `generate_content` | 1 task | AI | -| **5** | Content (`status='draft'`, no Images) | Images (`status='pending'` with prompts) | `generate_image_prompts` | 1 content | AI | -| **6** | Images (`status='pending'`) | Images (`status='generated'` with URLs) | `generate_images` | 1 image | AI | -| **7** | Content (`status='review'`) | Manual Review | None (gate) | N/A | Manual | - ---- - -### Stage 1: Keywords → Clusters (AI) - -**Purpose:** Group semantically similar keywords into topic clusters - -**Database Query (Automation Orchestrator):** -```python -pending_keywords = Keywords.objects.filter( - site=site, - status='new', - cluster__isnull=True, - disabled=False -) -``` - -**Orchestration Logic (What Automation Does):** -1. **Select Batch**: Count pending keywords - - If 0 keywords → Skip stage, log "No keywords to process" - - If 1-20 keywords → Select all (batch_size = count) - - If >20 keywords → Select first 20 (configurable batch_size) - -2. **Call AI Function**: - ```python - from igny8_core.ai.functions.auto_cluster import AutoCluster - - result = AutoCluster().execute( - payload={'ids': keyword_ids}, - account=account - ) - # Returns: {'task_id': 'celery_task_abc123'} - ``` - -3. **Monitor Progress**: Listen to Celery task status - - Use existing `StepTracker` phase events (INIT → PREP → AI_CALL → PARSE → SAVE → DONE) - - OR poll: `AsyncResult(task_id).state` until SUCCESS/FAILURE - - Log phase progress: "AI analyzing keywords (65% complete)" - -4. **Collect Results**: When task completes - - AI function already updated Keywords.status → 'mapped' - - AI function already created Cluster records with status='new' - - AI function already deducted credits via AIEngine - - Automation just logs: "Batch complete: N clusters created" - -5. **Repeat**: If more keywords remain, select next batch and go to step 2 - -**Stage Completion Criteria:** -- All keyword batches processed (pending_keywords.count() == 0) -- No critical errors - -**What AI Function Does (Already Implemented - DON'T DUPLICATE):** -- ✅ Groups keywords semantically using AI -- ✅ Creates Cluster records with `status='new'` -- ✅ Updates Keywords: `cluster_id=cluster.id`, `status='mapped'` -- ✅ Deducts credits automatically (AIEngine line 395) -- ✅ Logs to AIUsageLog -- ✅ Emits progress events via StepTracker - -**Stage Result Logged:** -```json -{ - "keywords_processed": 47, - "clusters_created": 8, - "batches_run": 3, - "credits_used": 10 // Read from AIUsageLog sum, not calculated -} -``` - ---- - -### Stage 2: Clusters → Ideas (AI) - -**Purpose:** Generate content ideas for each cluster - -**Database Query:** -``` -Clusters.objects.filter( - site=site, - status='new', - disabled=False -).exclude( - ideas__isnull=False # Has no ideas yet -) -``` - -**Process:** -1. Count clusters without ideas -2. If 0 → Skip stage -3. If > 0 → Process one cluster at a time (configurable batch size = 1) -4. For each cluster: - - Log: "Generating ideas for cluster: {cluster.name}" - - Call `IdeasService.generate_ideas(cluster_ids=[cluster.id], account)` - - Function returns `{'task_id': 'xyz789'}` - - Monitor via Celery task status or StepTracker events - - Wait for completion - - Log: "Cluster '{name}' complete: N ideas created" -5. Log stage summary - -**Stage Completion Criteria:** -- All clusters processed -- Each cluster now has >=1 idea -- No errors - -**Updates:** -- ContentIdeas: New records created with `status='new'`, `keyword_cluster_id=cluster.id` -- Clusters: `status='mapped'` -- Stage result: `{clusters_processed: 8, ideas_created: 56}` - -**Credits:** ~16 credits (2 per cluster) - ---- - -### Stage 3: Ideas → Tasks (Local Queue) - -**Purpose:** Convert content ideas to writer tasks (local, no AI) - -**Database Query:** -``` -ContentIdeas.objects.filter( - site=site, - status='new' -) -``` - -**Process:** -1. Count pending ideas -2. If 0 → Skip stage -3. If > 0 → Split into batches of 20 -4. For each batch: - - Log: "Queueing batch X/Y (20 ideas)" - - Call `bulk_queue_to_writer` view logic (NOT via HTTP, direct function call) - - For each idea: - - Create Tasks record with title=idea.idea_title, status='queued', cluster=idea.keyword_cluster - - Update idea status to 'queued' - - Log: "Batch X complete: 20 tasks created" -5. Log stage summary - -**Stage Completion Criteria:** -- All batches processed -- All ideas now have `status='queued'` -- Corresponding Tasks exist with `status='queued'` -- No errors - -**Updates:** -- Tasks: New records created with `status='queued'` -- ContentIdeas: `status` changed 'new' → 'queued' -- Stage result: `{ideas_processed: 56, tasks_created: 56, batches: 3}` - -**Credits:** 0 (local operation) - ---- - -### Stage 4: Tasks → Content (AI) - -**Purpose:** Generate full content drafts from tasks - -**Database Query (Automation Orchestrator):** -```python -pending_tasks = Tasks.objects.filter( - site=site, - status='queued', - content__isnull=True # No content generated yet -) -``` - -**Orchestration Logic:** -1. **Select Item**: Count queued tasks - - If 0 → Skip stage - - If > 0 → Select ONE task at a time (sequential processing) - -2. **Call AI Function**: - ```python - from igny8_core.ai.functions.generate_content import GenerateContent - - result = GenerateContent().execute( - payload={'ids': [task.id]}, - account=account - ) - # Returns: {'task_id': 'celery_task_xyz789'} - ``` - -3. **Monitor Progress**: Listen to Celery task status - - Use `StepTracker` phase events for real-time updates - - Log: "Writing sections (65% complete)" (from phase metadata) - - Content generation takes 5-15 minutes per task - -4. **Collect Results**: When task completes - - AI function already created Content with `status='draft'` - - AI function already updated Task.status → 'completed' - - AI function already updated Idea.status → 'completed' - - AI function already deducted credits based on word count - - Automation logs: "Content created (2500 words)" - -5. **Repeat**: Process next task sequentially - -**Stage Completion Criteria:** -- All tasks processed (pending_tasks.count() == 0) -- Each task has linked Content record - -**What AI Function Does (Already Implemented):** -- ✅ Generates article sections using AI -- ✅ Creates Content record with `status='draft'`, `task_id=task.id` -- ✅ Updates Task: `status='completed'` -- ✅ Updates linked Idea: `status='completed'` -- ✅ Deducts credits: 1 credit per 500 words (automatic) -- ✅ Logs to AIUsageLog with word count - -**Stage Result Logged:** -```json -{ - "tasks_processed": 56, - "content_created": 56, - "total_words": 140000, - "credits_used": 280 // From AIUsageLog, not calculated -} -``` - ---- - -### Stage 5: Content → Image Prompts (AI) - -**Purpose:** Extract image prompts from content and create Images records with prompts - -**CRITICAL:** There is NO separate "ImagePrompts" model. Images records ARE the prompts (with `status='pending'`) until images are generated. - -**Database Query (Automation Orchestrator):** -```python -# Content that has NO Images records at all -content_without_images = Content.objects.filter( - site=site, - status='draft' -).annotate( - images_count=Count('images') -).filter( - images_count=0 # No Images records exist yet -) -``` - -**Orchestration Logic:** -1. **Select Item**: Count content without any Images records - - If 0 → Skip stage - - If > 0 → Select ONE content at a time (sequential) - -2. **Call AI Function**: - ```python - from igny8_core.ai.functions.generate_image_prompts import GenerateImagePromptsFunction - - result = GenerateImagePromptsFunction().execute( - payload={'ids': [content.id]}, - account=account - ) - # Returns: {'task_id': 'celery_task_prompts456'} - ``` - -3. **Monitor Progress**: Wait for completion - -4. **Collect Results**: When task completes - - AI function already created Images records with: - - `status='pending'` - - `prompt='...'` (AI-generated prompt text) - - `image_type='featured'` or `'in_article'` - - `content_id=content.id` - - Content.status stays `'draft'` (unchanged) - - Automation logs: "Content '{title}' complete: N prompts created" - -5. **Repeat**: Process next content sequentially - -**Stage Completion Criteria:** -- All content processed (content_without_images.count() == 0) -- Each content has >=1 Images record with `status='pending'` and prompt text - -**What AI Function Does (Already Implemented):** -- ✅ Extracts featured image prompt from title/intro -- ✅ Extracts in-article prompts from H2 headings -- ✅ Creates Images records with `status='pending'`, `prompt='...'` -- ✅ Deducts credits automatically (0.5 per prompt) -- ✅ Logs to AIUsageLog - -**Stage Result Logged:** -```json -{ - "content_processed": 56, - "prompts_created": 224, - "credits_used": 112 // From AIUsageLog -} -``` - ---- - -### Stage 6: Images (Prompts) → Generated Images (AI) - -**Purpose:** Generate actual image URLs from Images records that contain prompts - -**CRITICAL:** Input is Images records with `status='pending'` (these contain the prompts). Output is same Images records updated with `status='generated'` and `image_url='https://...'` - -**Database Query (Automation Orchestrator):** -```python -# Images with prompts waiting to be generated -pending_images = Images.objects.filter( - site=site, - status='pending' # Has prompt text, needs image URL -) -``` - -**Orchestration Logic:** -1. **Select Item**: Count pending Images - - If 0 → Skip stage - - If > 0 → Select ONE Image at a time (sequential) - -2. **Call AI Function**: - ```python - from igny8_core.ai.functions.generate_images import GenerateImages - - result = GenerateImages().execute( - payload={'image_ids': [image.id]}, - account=account - ) - # Returns: {'task_id': 'celery_task_img789'} - ``` - -3. **Monitor Progress**: Wait for completion - -4. **Collect Results**: When task completes - - AI function already called image API using the `prompt` field - - AI function already updated Images: - - `status='pending'` → `status='generated'` - - `image_url='https://...'` (populated with generated image URL) - - AI function already deducted credits (1-4 per image) - - Automation logs: "Image generated: {image_url}" - -5. **Automatic Content Status Change** (NOT done by automation): - - After each image generation, background task checks if ALL Images for that Content are now `status='generated'` - - When last image completes: Content.status changes `'draft'` → `'review'` (in `ai/tasks.py` line 723) - - Automation does NOT trigger this - happens automatically - -6. **Repeat**: Process next pending Image sequentially - -**Stage Completion Criteria:** -- All pending Images processed (pending_images.count() == 0) -- All Images now have `image_url != null`, `status='generated'` - -**What AI Function Does (Already Implemented):** -- ✅ Reads `prompt` field from Images record -- ✅ Calls image generation API (OpenAI/Runware) with prompt -- ✅ Updates Images: `image_url=generated_url`, `status='generated'` -- ✅ Deducts credits automatically (1-4 per image) -- ✅ Logs to AIUsageLog - -**What Happens Automatically (ai/tasks.py:723):** -- ✅ Background task checks if all Images for a Content are `status='generated'` -- ✅ When complete: Content.status changes `'draft'` → `'review'` -- ✅ This happens OUTSIDE automation orchestrator (in Celery task) - -**Stage Result Logged:** -```json -{ - "images_processed": 224, - "images_generated": 224, - "content_moved_to_review": 56, // Side effect (automatic) - "credits_used": 448 // From AIUsageLog -} -``` - ---- - -### Stage 7: Manual Review Gate (STOP) - -**Purpose:** Automation ends - content automatically moved to 'review' status ready for manual review - -**CRITICAL:** Content with `status='review'` was automatically set in Stage 6 when ALL images completed. Automation just counts them. - -**Database Query (Automation Orchestrator):** -```python -# Content that has ALL images generated (status already changed to 'review') -ready_for_review = Content.objects.filter( - site=site, - status='review' # Automatically set when all images complete -) -``` - -**Orchestration Logic:** -1. **Count Only**: Count content with `status='review'` - - No processing, just counting - - These Content records already have all Images with `status='generated'` - -2. **Log Results**: - - Log: "Automation complete. X content pieces ready for review" - - Log: "Content IDs ready: [123, 456, 789, ...]" - -3. **Mark Run Complete**: - - AutomationRun.status = 'completed' - - AutomationRun.completed_at = now() - -4. **Send Notification** (optional): - - Email/notification: "Your automation run completed. X content pieces ready for review" - -5. **STOP**: No further automation stages - -**Stage Completion Criteria:** -- Counting complete -- Automation run marked `status='completed'` - -**What AI Function Does:** -- N/A - No AI function called in this stage - -**Stage Result Logged:** -```json -{ - "ready_for_review": 56, - "content_ids": [123, 456, 789, ...] -} -``` - -**What Happens Next (Manual - User Action):** -1. User navigates to `/writer/content` page -2. Content page shows filter: `status='review'` -3. User sees 56 content pieces with all images generated -4. User manually reviews: - - Content quality - - Image relevance - - Brand voice - - Accuracy -5. User selects multiple content → "Bulk Publish" action -6. Existing WordPress publishing workflow executes - -**Why Manual Review is Required:** -- Quality control before public publishing -- Legal/compliance verification -- Brand voice consistency check -- Final accuracy confirmation - ---- - -## 🔄 BATCH PROCESSING WITHIN STAGES - -### Critical Concepts - -**Batch vs Queue:** -- **Batch:** Group of items processed together in ONE AI call -- **Queue:** Total pending items waiting to be processed - -**Example - Stage 1 with 47 keywords:** -``` -Total Queue: 47 keywords -Batch Size: 20 - -Execution: - Batch 1: Keywords 1-20 → Call auto_cluster → Wait for completion - Batch 2: Keywords 21-40 → Call auto_cluster → Wait for completion - Batch 3: Keywords 41-47 → Call auto_cluster → Wait for completion - -Total Batches: 3 -Processing: Sequential (never parallel) -``` - -**UI Display:** -``` -Stage 1: Keywords → Clusters -Status: ● Processing -Queue: 47 keywords total -Progress: Batch 2/3 (40 processed, 7 remaining) -Current: Processing keywords 21-40 -Time Elapsed: 4m 30s -Credits Used: 8 -``` - -### Batch Completion Triggers - -**Within Stage:** -- Batch completes → Immediately start next batch -- Last batch completes → Stage complete - -**Between Stages:** -- Stage N completes → Trigger Stage N+1 automatically -- Hard verification: Ensure queue is empty before proceeding - -**Detailed Stage Processing Queues (UI Elements):** - -Each stage card should show: -1. **Total Queue Count** - How many items need processing in this stage -2. **Current Batch** - Which batch is being processed (e.g., "Batch 2/5") -3. **Processed Count** - How many items completed so far -4. **Remaining Count** - How many items left in queue -5. **Current Item** - What specific item is processing right now (for single-item batches) - -**Example UI for Stage 4:** -``` -┌──────────────────────────────────────────────────────────────┐ -│ STAGE 4: Tasks → Content (AI) │ -│ Status: ● Processing │ -│ │ -│ 📊 QUEUE OVERVIEW: │ -│ ├─ Total Tasks: 56 │ -│ ├─ Processed: 23 │ -│ ├─ Remaining: 33 │ -│ └─ Progress: ━━━━━━━╸━━━━━━━━━━━━ 41% │ -│ │ -│ 🔄 CURRENT PROCESSING: │ -│ ├─ Item: Task 24/56 │ -│ ├─ Title: "Ultimate Coffee Bean Buying Guide" │ -│ ├─ Progress: Writing sections (65% complete) │ -│ └─ Time: 2m 15s elapsed │ -│ │ -│ 💳 STAGE STATS: │ -│ ├─ Credits Used: 46 │ -│ ├─ Time Elapsed: 1h 23m │ -│ └─ ETA: 1h 15m remaining │ -│ │ -│ [View Details] [Pause Stage] │ -└──────────────────────────────────────────────────────────────┘ -``` - ---- - -## 🗄️ DATABASE STRUCTURE - -### New Models to Create - -**AutomationRun** (tracks each automation execution) -``` -Table: igny8_automation_runs -Fields: - - id: Integer (PK) - - run_id: String (unique, indexed) - Format: run_20251203_140523_manual - - account_id: ForeignKey(Account) - - site_id: ForeignKey(Site) - - trigger_type: String - Choices: 'manual', 'scheduled' - - status: String - Choices: 'running', 'paused', 'completed', 'failed' - - current_stage: Integer - Current stage number (1-7) - - started_at: DateTime - - completed_at: DateTime (nullable) - - total_credits_used: Integer - - stage_1_result: JSON - {keywords_processed, clusters_created, batches} - - stage_2_result: JSON - {clusters_processed, ideas_created} - - stage_3_result: JSON - {ideas_processed, tasks_created} - - stage_4_result: JSON - {tasks_processed, content_created, total_words} - - stage_5_result: JSON - {content_processed, prompts_created} - - stage_6_result: JSON - {prompts_processed, images_generated} - - stage_7_result: JSON - {ready_for_review} - - error_message: Text (nullable) - -Indexes: - - run_id (unique) - - site_id, started_at - - status, started_at -``` - -**AutomationConfig** (per-site configuration) -``` -Table: igny8_automation_configs -Fields: - - id: Integer (PK) - - account_id: ForeignKey(Account) - - site_id: ForeignKey(Site, unique) - ONE config per site - - is_enabled: Boolean - Whether scheduled automation is active - - frequency: String - Choices: 'daily', 'weekly', 'monthly' - - scheduled_time: Time - When to run (e.g., 02:00) - - stage_1_batch_size: Integer - Default 20 (keywords per batch) - - stage_2_batch_size: Integer - Default 1 (clusters at a time) - - stage_3_batch_size: Integer - Default 20 (ideas per batch) - - stage_4_batch_size: Integer - Default 1 (tasks - sequential) - - stage_5_batch_size: Integer - Default 1 (content at a time) - - stage_6_batch_size: Integer - Default 1 (images - sequential) - - last_run_at: DateTime (nullable) - - next_run_at: DateTime (nullable) - Calculated based on frequency - -Constraints: - - Unique: site_id (one config per site) -``` - -### File-Based Logging Structure - -**Directory Structure:** -``` -logs/ -└── automation/ - └── {account_id}/ - └── {site_id}/ - └── {run_id}/ - ├── automation_run.log (main activity log) - ├── stage_1.log (keywords → clusters) - ├── stage_2.log (clusters → ideas) - ├── stage_3.log (ideas → tasks) - ├── stage_4.log (tasks → content) - ├── stage_5.log (content → prompts) - ├── stage_6.log (prompts → images) - └── stage_7.log (review gate) -``` - -**Log File Format (automation_run.log):** -``` -======================================== -AUTOMATION RUN: run_20251203_140523_manual -Started: 2025-12-03 14:05:23 -Trigger: manual -Account: 5 -Site: 12 -======================================== - -14:05:23 - Automation started (trigger: manual) -14:05:23 - Credit check: Account has 1500 credits, estimated need: 866 credits -14:05:23 - Stage 1 starting: Keywords → Clusters -14:05:24 - Stage 1: Found 47 pending keywords -14:05:24 - Stage 1: Processing batch 1/3 (20 keywords) -14:05:25 - Stage 1: AI task queued: task_id=abc123 -14:07:30 - Stage 1: Batch 1 complete - 3 clusters created -14:07:31 - Stage 1: Processing batch 2/3 (20 keywords) -[... continues ...] -``` - -**Stage-Specific Log (stage_1.log):** -``` -======================================== -STAGE 1: Keywords → Clusters (AI) -Started: 2025-12-03 14:05:23 -======================================== - -14:05:24 - Query: Keywords.objects.filter(site=12, status='new', cluster__isnull=True) -14:05:24 - Found 47 pending keywords -14:05:24 - Batch size: 20 keywords -14:05:24 - Total batches: 3 - ---- Batch 1/3 --- -14:05:24 - Keyword IDs: [101, 102, 103, ..., 120] -14:05:25 - Calling ClusteringService.cluster_keywords(ids=[101..120], account=5, site_id=12) -14:05:25 - AI task queued: task_id=abc123 -14:05:26 - Monitoring task status... -14:05:28 - Phase: INIT - Initializing (StepTracker event) -14:05:45 - Phase: AI_CALL - AI analyzing keywords (StepTracker event) -14:07:15 - Phase: SAVE - Creating clusters (StepTracker event) -14:07:30 - Phase: DONE - Complete -14:07:30 - Result: 3 clusters created -14:07:30 - Clusters: ["Coffee Beans", "Brewing Methods", "Coffee Equipment"] -14:07:30 - Credits used: 4 (from AIUsageLog) - ---- Batch 2/3 --- -[... continues ...] - -======================================== -STAGE 1 COMPLETE -Total Time: 5m 30s -Processed: 47 keywords -Clusters Created: 8 -Credits Used: 10 -======================================== -``` - ---- - -## 🔐 SAFETY MECHANISMS - -### 1. Concurrency Control (Prevent Duplicate Runs) - -**Problem:** User clicks "Run Now" while scheduled task is running - -**Solution:** Distributed locking using Django cache - -**Implementation Logic:** -``` -When starting automation: - 1. Try to acquire lock: cache.add(f'automation_lock_{site.id}', 'locked', timeout=21600) - 2. If lock exists → Return error: "Automation already running for this site" - 3. If lock acquired → Proceed with run - 4. On completion/failure → Release lock: cache.delete(f'automation_lock_{site.id}') - -Also check database: - - Query AutomationRun.objects.filter(site=site, status='running').exists() - - If exists → Error: "Another automation is running" -``` - -**User sees:** -- "Automation already in progress. Started at 02:00 AM, currently on Stage 4." -- Link to view current run progress - ---- - -### 2. Credit Reservation (Prevent Mid-Run Failures) - -**Problem:** Account runs out of credits during Stage 4 - -**Solution:** Reserve estimated credits at start, deduct as used - -**Implementation Logic:** -``` -Before starting: - 1. Estimate total credits needed: - - Count keywords → estimate clustering credits - - Count clusters → estimate ideas credits - - Estimate content generation (assume avg word count) - - Estimate image generation (assume 4 images per content) - 2. Check: account.credits_balance >= estimated_credits * 1.2 (20% buffer) - 3. If insufficient → Error: "Need ~866 credits, you have 500" - 4. Reserve credits: account.credits_reserved += estimated_credits - 5. As each stage completes → Deduct actual: account.credits_balance -= actual_used - 6. On completion → Release unused: account.credits_reserved -= unused - -Database fields needed: - - Account.credits_reserved (new field) -``` - ---- - -### 3. Stage Idempotency (Safe to Retry) - -**Problem:** User resumes paused run, Stage 1 runs again creating duplicate clusters - -**Solution:** Check if stage already completed before executing - -**Implementation Logic:** -``` -At start of each run_stage_N(): - 1. Check AutomationRun.stage_N_result - 2. If result exists and has processed_count > 0: - - Log: "Stage N already completed - skipping" - - return (skip to next stage) - 3. Else: Proceed with stage execution -``` - ---- - -### 4. Celery Task Chaining (Non-Blocking Workers) - -**Problem:** Synchronous execution blocks Celery worker for hours - -**Solution:** Chain stages as separate Celery tasks - -**Implementation Logic:** -``` -Instead of: - def start_automation(): - run_stage_1() # blocks for 30 min - run_stage_2() # blocks for 45 min - ... - -Do: - @shared_task - def run_stage_1_task(run_id): - service = AutomationService.from_run_id(run_id) - service.run_stage_1() - # Trigger next stage - run_stage_2_task.apply_async(args=[run_id], countdown=5) - - @shared_task - def run_stage_2_task(run_id): - service = AutomationService.from_run_id(run_id) - service.run_stage_2() - run_stage_3_task.apply_async(args=[run_id], countdown=5) - -Benefits: - - Workers not blocked for hours - - Can retry individual stages - - Better monitoring in Celery Flower - - Horizontal scaling possible -``` - ---- - -### 5. Pause/Resume Capability - -**User Can:** -- Pause automation at any point -- Resume from where it left off - -**Implementation Logic:** -``` -Pause: - - Update AutomationRun.status = 'paused' - - Current stage completes current batch then stops - - Celery task checks status before each batch - -Resume: - - Update AutomationRun.status = 'running' - - Restart from current_stage - - Use idempotency check to skip completed work -``` - ---- - -### 6. Error Handling Per Stage - -**If Stage Fails:** -``` -try: - run_stage_1() -except Exception as e: - - Log error to stage_1.log - - Update AutomationRun: - - status = 'failed' - - error_message = str(e) - - current_stage = 1 (where it failed) - - Send notification: "Automation failed at Stage 1" - - Stop execution (don't proceed to Stage 2) - -User can: - - View logs to see what went wrong - - Fix issue (e.g., add credits) - - Click "Resume" to retry from Stage 1 -``` - ---- - -### 7. Log Cleanup (Prevent Disk Bloat) - -**Problem:** After 1000 runs, logs occupy 80MB+ per site - -**Solution:** Celery periodic task to delete old logs - -**Implementation Logic:** -``` -@shared_task -def cleanup_old_automation_logs(): - cutoff = datetime.now() - timedelta(days=90) # Keep last 90 days - - old_runs = AutomationRun.objects.filter( - started_at__lt=cutoff, - status__in=['completed', 'failed'] - ) - - for run in old_runs: - log_dir = f'logs/automation/{run.account_id}/{run.site_id}/{run.run_id}/' - shutil.rmtree(log_dir) # Delete directory - run.delete() # Remove DB record - -Schedule: Weekly, Monday 3 AM -``` - ---- - -## 🎨 FRONTEND DESIGN - -### Page Structure: `/automation` - -**Layout:** -``` -┌─────────────────────────────────────────────────────────────┐ -│ 🤖 AI AUTOMATION PIPELINE │ -│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │ -│ │ -│ │ -│ ⏰ SCHEDULE │ -│ Next Run: Tomorrow at 2:00 AM (in 16 hours) │ -│ Frequency: [Daily ▼] at [02:00 ▼] │ -│ Status: ● Scheduled │ -│ │ -│ [Run Now] [Pause Schedule] [Configure] │ -│ │ -├─────────────────────────────────────────────────────────────┤ -│ 📊 PIPELINE OVERVIEW │ -│ │ -│ Keywords ──→ Clusters ──→ Ideas ──→ Tasks ──→ Content │ -│ 47 8 42 20 generating │ -│ pending new ready queued Stage 5 │ -│ │ -│ Overall Progress: ━━━━━━━╸━━━━━━━━━ 62% (Stage 5/7) │ -│ Estimated Completion: 2 hours 15 minutes │ -│ │ -└─────────────────────────────────────────────────────────────┘ - -[STAGE 1 CARD - completed state] -[STAGE 2 CARD - completed state] -[STAGE 3 CARD - completed state] -[STAGE 4 CARD - running state with queue details] -[STAGE 5 CARD - waiting state] -[STAGE 6 CARD - waiting state] -[STAGE 7 CARD - gate state] - -━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -📋 LIVE ACTIVITY LOG (Last 50 events) -├─ 14:23:45 - Stage 4: Started content generation for Task 3 -├─ 14:24:12 - Stage 4: Writing sections (65% complete) -├─ 14:22:30 - Stage 4: Completed Task 2 → Content created -├─ 14:20:15 - Stage 4: Started content generation for Task 2 -└─ [View Full Log] - -💰 TOTAL CREDITS USED THIS RUN: 66 credits -``` - -**Components:** - -**StageCard.tsx** - Individual stage display component -- Props: stageNumber, stageName, status, queueData, result -- Shows: Status badge, queue overview, progress bar, stats -- Actions: "View Details", "Pause", "Retry Failed" - -**ActivityLog.tsx** - Live activity feed component -- Props: runId -- Fetches: `/api/v1/automation/activity_log/{runId}` every 3 seconds -- Shows: Timestamped log entries, color-coded by type (info/success/error) - -**ConfigModal.tsx** - Schedule configuration modal -- Fields: Frequency dropdown, Time picker, Batch sizes (advanced) -- Saves to: AutomationConfig model via `/api/v1/automation/config/` - -**Sidebar Menu Addition:** -``` -Sites - ├─ Site Management - └─ Site Settings - -Automation ← NEW - └─ Pipeline Dashboard - -Planner - ├─ Keywords - ├─ Clusters - └─ Ideas -``` - ---- - -### Real-Time Progress Updates - -**UI Update Strategy:** -- **Frontend Polling**: Poll automation status API every 3 seconds when run is active -- **Backend Progress**: Uses event-based `StepTracker` to capture AI function phases -- When automation is `status='running'` → Poll every 3 seconds -- When `status='completed'` or `status='failed'` → Stop polling -- When `status='paused'` → Poll every 10 seconds - -**How Progress Works:** -1. **AI Function Execution**: Each AI function emits phase events (INIT, PREP, AI_CALL, PARSE, SAVE, DONE) -2. **StepTracker Captures**: Progress tracker records these events with metadata -3. **Automation Logs**: Orchestrator reads from StepTracker and logs to file -4. **UI Polls**: Frontend polls automation status API to read aggregated progress -5. **Display**: UI shows current phase and completion percentage per stage - -**API Endpoint:** -``` -GET /api/v1/automation/current_run/?site_id=12 - -Response: -{ - "run": { - "run_id": "run_20251203_140523_manual", - "status": "running", - "current_stage": 4, - "started_at": "2025-12-03T14:05:23Z", - "total_credits_used": 66, - "stage_1_result": {"keywords_processed": 47, "clusters_created": 8}, - "stage_2_result": {"clusters_processed": 8, "ideas_created": 56}, - "stage_3_result": {"ideas_processed": 56, "tasks_created": 56}, - "stage_4_result": {"tasks_processed": 23, "tasks_total": 56}, - ... - }, - "activity_log": [ - "14:23:45 - Stage 4: Started content generation for Task 3", - "14:24:12 - Stage 4: Writing sections (65% complete)", - ... - ], - "queues": { - "stage_1": {"total": 0, "pending": 0}, - "stage_2": {"total": 0, "pending": 0}, - "stage_3": {"total": 0, "pending": 0}, - "stage_4": {"total": 56, "pending": 33}, - "stage_5": {"total": 23, "pending": 23}, - "stage_6": {"total": 0, "pending": 0}, - "stage_7": {"total": 0, "pending": 0} - } -} -``` - ---- - -## 🔄 BACKEND IMPLEMENTATION FLOW - -### Service Layer Architecture - -**AutomationService** (core orchestrator) -- Location: `backend/igny8_core/business/automation/services/automation_service.py` -- Responsibility: Execute stages sequentially, manage run state -- Reuses: All existing AI function classes (NO duplication) - -**AutomationLogger** (file logging) -- Location: `backend/igny8_core/business/automation/services/automation_logger.py` -- Responsibility: Write timestamped logs to files -- Methods: start_run(), log_stage_start(), log_stage_progress(), log_stage_complete() - -**Key Service Methods:** - -``` -AutomationService: - - __init__(account, site) → Initialize with site context (NO sector) - - start_automation(trigger_type) → Main entry point - - run_stage_1() → Keywords → Clusters - - run_stage_2() → Clusters → Ideas - - run_stage_3() → Ideas → Tasks - - run_stage_4() → Tasks → Content - - run_stage_5() → Content → Prompts - - run_stage_6() → Prompts → Images - - run_stage_7() → Review gate - - pause_automation() → Pause current run - - resume_automation() → Resume from current_stage - - estimate_credits() → Calculate estimated credits needed - -AutomationLogger: - - start_run(account_id, site_id, trigger_type) → Create log directory, return run_id - - log_stage_start(run_id, stage_number, stage_name, pending_count) - - log_stage_progress(run_id, stage_number, message) - - log_stage_complete(run_id, stage_number, processed_count, time_elapsed, credits_used) - - log_stage_error(run_id, stage_number, error_message) - - get_activity_log(run_id, last_n=50) → Return last N log lines -``` - ---- - -### API Endpoints to Implement - -**AutomationViewSet** - Django REST Framework ViewSet -- Base URL: `/api/v1/automation/` -- Actions: - -``` -POST /api/v1/automation/run_now/ - - Body: {"site_id": 12} - - Action: Trigger manual automation run - - Returns: {"run_id": "run_...", "message": "Automation started"} - -GET /api/v1/automation/current_run/?site_id=12 - - Returns: Current/latest run status, activity log, queue counts - -POST /api/v1/automation/pause/ - - Body: {"run_id": "run_..."} - - Action: Pause running automation - -POST /api/v1/automation/resume/ - - Body: {"run_id": "run_..."} - - Action: Resume paused automation - -GET /api/v1/automation/config/?site_id=12 - - Returns: AutomationConfig for site - -PUT /api/v1/automation/config/ - - Body: {"site_id": 12, "is_enabled": true, "frequency": "daily", "scheduled_time": "02:00"} - - Action: Update automation schedule - -GET /api/v1/automation/history/?site_id=12&page=1 - - Returns: Paginated list of past runs - -GET /api/v1/automation/logs/{run_id}/ - - Returns: Full logs for a specific run (all stage files) -``` - ---- - -### Celery Tasks for Scheduling - -**Periodic Task** (runs every hour) -``` -@shared_task(name='check_scheduled_automations') -def check_scheduled_automations(): - """ - Runs every hour (via Celery Beat) - Checks if any AutomationConfig needs to run - """ - now = timezone.now() - - configs = AutomationConfig.objects.filter( - is_enabled=True, - next_run_at__lte=now - ) - - for config in configs: - # Check for concurrent run - if AutomationRun.objects.filter(site=config.site, status='running').exists(): - continue # Skip if already running - - # Start automation - run_automation_task.delay( - account_id=config.account_id, - site_id=config.site_id, - trigger_type='scheduled' - ) - - # Calculate next run time - if config.frequency == 'daily': - config.next_run_at = now + timedelta(days=1) - elif config.frequency == 'weekly': - config.next_run_at = now + timedelta(weeks=1) - elif config.frequency == 'monthly': - config.next_run_at = now + timedelta(days=30) - - config.last_run_at = now - config.save() - -Schedule in celery.py: - app.conf.beat_schedule['check-scheduled-automations'] = { - 'task': 'check_scheduled_automations', - 'schedule': crontab(minute=0), # Every hour on the hour - } -``` - -**Stage Task Chain** -``` -@shared_task -def run_automation_task(account_id, site_id, trigger_type): - """ - Main automation task - chains individual stage tasks - """ - service = AutomationService(account_id, site_id) - run_id = service.start_automation(trigger_type) - - # Chain stages as separate tasks for non-blocking execution - chain( - run_stage_1.si(run_id), - run_stage_2.si(run_id), - run_stage_3.si(run_id), - run_stage_4.si(run_id), - run_stage_5.si(run_id), - run_stage_6.si(run_id), - run_stage_7.si(run_id), - ).apply_async() - -@shared_task -def run_stage_1(run_id): - service = AutomationService.from_run_id(run_id) - service.run_stage_1() - return run_id # Pass to next task - -@shared_task -def run_stage_2(run_id): - service = AutomationService.from_run_id(run_id) - service.run_stage_2() - return run_id - -[... similar for stages 3-7 ...] -``` - ---- - -## 🧪 TESTING STRATEGY - -### Unit Tests - -**Test AutomationService:** -- test_estimate_credits_calculation() -- test_stage_1_processes_batches_correctly() -- test_stage_completion_triggers_next_stage() -- test_pause_stops_after_current_batch() -- test_resume_from_paused_state() -- test_idempotency_skips_completed_stages() - -**Test AutomationLogger:** -- test_creates_log_directory_structure() -- test_writes_timestamped_log_entries() -- test_get_activity_log_returns_last_n_lines() - -### Integration Tests - -**Test Full Pipeline:** -``` -def test_full_automation_pipeline(): - # Setup: Create 10 keywords - keywords = KeywordFactory.create_batch(10, site=site) - - # Execute - service = AutomationService(account, site) - result = service.start_automation(trigger_type='manual') - - # Assert Stage 1 - assert result['stage_1_result']['keywords_processed'] == 10 - assert result['stage_1_result']['clusters_created'] > 0 - - # Assert Stage 2 - assert result['stage_2_result']['ideas_created'] > 0 - - # Assert Stage 3 - assert result['stage_3_result']['tasks_created'] > 0 - - # Assert Stage 4 - assert result['stage_4_result']['content_created'] > 0 - - # Assert Stage 5 - assert result['stage_5_result']['prompts_created'] > 0 - - # Assert Stage 6 - assert result['stage_6_result']['images_generated'] > 0 - - # Assert final state - assert result['status'] == 'completed' - assert AutomationRun.objects.get(run_id=result['run_id']).status == 'completed' -``` - -**Test Error Scenarios:** -- test_insufficient_credits_prevents_start() -- test_concurrent_run_prevented() -- test_stage_failure_stops_pipeline() -- test_rollback_on_error() - ---- - -## 📋 IMPLEMENTATION CHECKLIST - -### Phase 1: Database & Models (Week 1) -- [ ] Create `automation` app directory structure -- [ ] Define AutomationRun model with all stage_result JSON fields -- [ ] Define AutomationConfig model (one per site, NO sector) -- [ ] Create migrations -- [ ] Test model creation and queries - -### Phase 2: Logging Service (Week 1) -- [ ] Create AutomationLogger class -- [ ] Implement start_run() with log directory creation -- [ ] Implement log_stage_start(), log_stage_progress(), log_stage_complete() -- [ ] Implement get_activity_log() -- [ ] Test file logging manually - -### Phase 3: Core Automation Service (Week 2) -- [ ] Create AutomationService class -- [ ] Implement estimate_credits() -- [ ] Implement start_automation() with credit check -- [ ] Implement run_stage_1() calling ClusteringService -- [ ] Test Stage 1 in isolation with real keywords -- [ ] Implement run_stage_2() calling IdeasService -- [ ] Test Stage 2 in isolation -- [ ] Implement run_stage_3() calling bulk_queue_to_writer logic -- [ ] Implement run_stage_4() calling GenerateContentFunction -- [ ] Implement run_stage_5() calling GenerateImagePromptsFunction -- [ ] Implement run_stage_6() calling GenerateImagesFunction -- [ ] Implement run_stage_7() review gate (count only) -- [ ] Implement pause_automation() and resume_automation() - -### Phase 4: API Endpoints (Week 3) -- [ ] Create AutomationViewSet -- [ ] Implement run_now() action -- [ ] Implement current_run() action -- [ ] Implement pause() and resume() actions -- [ ] Implement config GET/PUT actions -- [ ] Implement history() action -- [ ] Implement logs() action -- [ ] Add URL routing in api_urls.py -- [ ] Test all endpoints with Postman/curl - -### Phase 5: Celery Tasks & Scheduling (Week 3) -- [ ] Create check_scheduled_automations periodic task -- [ ] Create run_automation_task -- [ ] Create stage task chain (run_stage_1, run_stage_2, etc.) -- [ ] Register tasks in celery.py -- [ ] Add Celery Beat schedule -- [ ] Test scheduled execution - -### Phase 6: Frontend Components (Week 4) -- [ ] Create /automation route in React Router -- [ ] Create Dashboard.tsx page component -- [ ] Create StageCard.tsx with queue display -- [ ] Create ActivityLog.tsx with 3-second polling -- [ ] Create ConfigModal.tsx for schedule settings -- [ ] Add "Automation" to sidebar menu (below Sites) -- [ ] Implement "Run Now" button -- [ ] Implement "Pause" and "Resume" buttons -- [ ] Test full UI flow - -### Phase 7: Safety & Polish (Week 5) -- [ ] Implement distributed locking (prevent concurrent runs) -- [ ] Implement credit reservation system -- [ ] Implement stage idempotency checks -- [ ] Implement error handling and rollback -- [ ] Create cleanup_old_automation_logs task -- [ ] Add email/notification on completion/failure -- [ ] Load testing with 100+ keywords -- [ ] UI polish and responsiveness -- [ ] Documentation update - ---- - -## 🚀 POST-LAUNCH ENHANCEMENTS - -### Future Features (Phase 8+) -- **Conditional Stages:** Skip stages if no data (e.g., skip Stage 1 if no keywords) -- **Parallel Task Processing:** Process multiple tasks simultaneously in Stage 4 (with worker limits) -- **Smart Scheduling:** Avoid peak hours, optimize for cost -- **A/B Testing:** Test different prompts, compare results -- **Content Quality Scoring:** Auto-reject low-quality AI content -- **WordPress Auto-Publish:** With approval workflow and staging -- **Analytics Integration:** Track content performance post-publish -- **Social Media Auto-Post:** Share published content to social channels - ---- - -## 📖 USER DOCUMENTATION - -### How to Use Automation - -**1. Configure Schedule:** -- Navigate to Automation page -- Click "Configure" button -- Set frequency (Daily/Weekly/Monthly) -- Set time (e.g., 2:00 AM) -- Optionally adjust batch sizes (advanced) -- Click "Save" - -**2. Manual Run:** -- Click "Run Now" button -- Monitor progress in real-time -- View activity log for details - -**3. Review Content:** -- Wait for automation to complete (or check next morning if scheduled) -- Navigate to Writer → Content page -- Filter by "Draft" status with images generated -- Review content quality -- Select multiple → Bulk Publish - -**4. Monitor History:** -- View past runs in History tab -- Click run to view detailed logs -- See credits used per run - ---- - -## ✅ SUCCESS CRITERIA - -**Automation is successful if:** -- ✅ Runs without manual intervention from Keywords → Draft Content -- ✅ Processes 100+ keywords without errors -- ✅ Respects credit limits (pre-check + reservation) -- ✅ Stops at review gate (doesn't auto-publish) -- ✅ Completes within estimated time (6-12 hours for 100 keywords) -- ✅ UI shows real-time progress accurately -- ✅ Logs are detailed and troubleshoot-able -- ✅ Can pause/resume without data loss -- ✅ Scheduled runs trigger correctly -- ✅ No duplicate runs occur -- ✅ Reuses ALL existing AI functions (zero duplication) - ---- - -**END OF COMPLETE IMPLEMENTATION PLAN** - -This plan ensures a safe, modular, observable, and maintainable automation system that orchestrates the existing IGNY8 AI functions into a fully automated content pipeline. \ No newline at end of file diff --git a/docs/automation/automation-progress-ux-improvement-plan.md b/docs/automation/automation-progress-ux-improvement-plan.md deleted file mode 100644 index d20bb7d4..00000000 --- a/docs/automation/automation-progress-ux-improvement-plan.md +++ /dev/null @@ -1,725 +0,0 @@ -# Automation Progress UX Improvement Plan - -**Date:** December 4, 2025 -**Status:** Design Phase -**Priority:** MEDIUM - ---- - -## 🎯 OBJECTIVE - -Improve the automation progress tracking UX to show **real-time processing status** for currently processing items, making it easier for users to understand what's happening during automation runs. - ---- - -## 🔍 CURRENT STATE ANALYSIS - -### Current Behavior - -**What Users See Now:** -1. A "Current State" card that shows the stage being processed -2. Stage number and status (e.g., "Stage 3: Ideas → Tasks") -3. **BUT:** No visibility into which specific records are being processed -4. **Problem:** User only knows when a full stage completes - -**Example Current Experience:** -``` -┌─────────────────────────────────────┐ -│ Current State: Stage 2 │ -│ Clusters → Ideas (AI) │ -│ │ -│ Status: Processing │ -└─────────────────────────────────────┘ - -[User waits... no updates until stage completes] -``` - -### User Pain Points - -1. ❌ **No Record-Level Progress:** Can't see which keywords/ideas/content are being processed -2. ❌ **No Queue Visibility:** Don't know what's coming up next -3. ❌ **No Item Count Progress:** "Processing 15 of 50 keywords..." is missing -4. ❌ **Card Position:** Current state card is at bottom, requires scrolling -5. ❌ **No Percentage Progress:** Just a spinner, no quantitative feedback - ---- - -## ✅ PROPOSED SOLUTION - -### New Design Concept - -``` -┌─────────────────────────────────────────────────────────────────────────────┐ -│ 🔄 AUTOMATION IN PROGRESS │ -│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67% │ -│ │ -│ Stage 2: Clusters → Ideas (AI) │ -│Column 1 │ -│ Currently Processing: │ -│ • "Best SEO tools for small business" (Cluster #42) │ -│ Column 2 │ -│ Up Next: │ -│ • "Content marketing automation platforms" │ -│ • "AI-powered content creation tools" │ -│ Sinngle row centered │ -│ Progress: 34/50 clusters processed │ -└─────────────────────────────────────────────────────────────────────────────┘ - -[STAGES SECTION BELOW - All 7 stages in grid view] -``` - ---- - -## 📐 DETAILED DESIGN SPECIFICATIONS - -### 1. Card Repositioning - -**Move from:** Bottom of page (after stages) -**Move to:** Top of page (above stages section) -**Layout:** Max width 1200px horizontal card -**Visibility:** Only shown when `currentRun?.status === 'running'` - -### 2. Card Structure - -#### Header Section -- **Left:** Large stage number icon (animated pulse) -- **Center:** Stage name + type badge (AI/Local/Manual) -- **Right:** Percentage complete (calculated from processed/total) - -#### Progress Bar -- **Type:** Animated linear progress bar -- **Colors:** - - Blue for active stage - - Green for completed - - Gray for pending -- **Updates:** Refresh every 3-5 seconds via polling - -#### Currently Processing Section -- **For Keywords Stage:** - ``` - Currently Processing: - • "keyword 1" - • "keyword 2" - • "keyword 3" - - + 47 more in queue - ``` - -- **For Ideas Stage:** - ``` - Currently Processing: - • "10 Ways to Improve SEO Rankings" - - Up Next: - • "Content Marketing Best Practices 2025" - • "AI Tools for Content Writers" - ``` - -- **For Content Stage:** - ``` - Currently Processing: - • "How to Use ChatGPT for Content Creation" (2,500 words) - - Up Next: - • "Best AI Image Generators in 2025" - ``` - -#### Record Counter -``` -Progress: [current]/[total] [items] processed -Example: Progress: 15/50 keywords processed -``` - -### 3. Refresh Strategy - -**Polling Approach:** -```typescript -// Poll every 3 seconds while automation is running -useEffect(() => { - if (currentRun?.status === 'running') { - const interval = setInterval(() => { - // Refresh ONLY the current processing data - fetchCurrentProcessingState(); - }, 3000); - - return () => clearInterval(interval); - } -}, [currentRun]); -``` - -**Partial Refresh:** -- Only refresh the "Currently Processing" component -- Don't reload entire page -- Don't re-fetch stage cards -- Smooth transition (no flickering) - ---- - -## 🗄️ BACKEND CHANGES REQUIRED - -### New API Endpoint - -**URL:** `GET /api/automation/current_processing/` -**Params:** `?site_id={id}&run_id={run_id}` - -**Response Format:** -```json -{ - "run_id": "abc123", - "current_stage": 2, - "stage_name": "Clusters → Ideas", - "stage_type": "AI", - "total_items": 50, - "processed_items": 34, - "percentage": 68, - "currently_processing": [ - { - "id": 42, - "title": "Best SEO tools for small business", - "type": "cluster" - } - ], - "up_next": [ - { - "id": 43, - "title": "Content marketing automation platforms", - "type": "cluster" - }, - { - "id": 44, - "title": "AI-powered content creation tools", - "type": "cluster" - } - ], - "remaining_count": 16 -} -``` - -### Implementation in AutomationService - -**File:** `backend/igny8_core/business/automation/services/automation_service.py` - -**Add method:** -```python -def get_current_processing_state(self) -> dict: - """ - Get real-time processing state for current automation run - """ - if not self.run or self.run.status != 'running': - return None - - stage = self.run.current_stage - - # Get stage-specific data - if stage == 1: # Keywords → Clusters - queue = Keywords.objects.filter( - site=self.site, status='new' - ).order_by('id') - - return { - 'stage_number': 1, - 'stage_name': 'Keywords → Clusters', - 'stage_type': 'AI', - 'total_items': queue.count() + self._get_processed_count(stage), - 'processed_items': self._get_processed_count(stage), - 'currently_processing': self._get_current_items(queue, 3), - 'up_next': self._get_next_items(queue, 2, skip=3), - } - - elif stage == 2: # Clusters → Ideas - queue = Clusters.objects.filter( - site=self.site, status='new', disabled=False - ).order_by('id') - - return { - 'stage_number': 2, - 'stage_name': 'Clusters → Ideas', - 'stage_type': 'AI', - 'total_items': queue.count() + self._get_processed_count(stage), - 'processed_items': self._get_processed_count(stage), - 'currently_processing': self._get_current_items(queue, 1), - 'up_next': self._get_next_items(queue, 2, skip=1), - } - - # ... similar for stages 3-6 - -def _get_processed_count(self, stage: int) -> int: - """Get count of items processed in current stage""" - result_key = f'stage_{stage}_result' - result = getattr(self.run, result_key, {}) - - # Extract appropriate count from result - if stage == 1: - return result.get('keywords_processed', 0) - elif stage == 2: - return result.get('clusters_processed', 0) - # ... etc - -def _get_current_items(self, queryset, count: int) -> list: - """Get currently processing items""" - items = queryset[:count] - return [ - { - 'id': item.id, - 'title': getattr(item, 'keyword', None) or - getattr(item, 'cluster_name', None) or - getattr(item, 'idea_title', None) or - getattr(item, 'title', None), - 'type': queryset.model.__name__.lower() - } - for item in items - ] -``` - -### Add View in AutomationViewSet - -**File:** `backend/igny8_core/business/automation/views.py` - -```python -@action(detail=False, methods=['get'], url_path='current_processing') -def current_processing(self, request): - """Get current processing state for active automation run""" - site_id = request.GET.get('site_id') - run_id = request.GET.get('run_id') - - if not site_id or not run_id: - return error_response( - error='site_id and run_id required', - status_code=400, - request=request - ) - - try: - run = AutomationRun.objects.get(run_id=run_id, site_id=site_id) - - if run.status != 'running': - return success_response(data=None, request=request) - - service = AutomationService.from_run_id(run_id) - state = service.get_current_processing_state() - - return success_response(data=state, request=request) - - except AutomationRun.DoesNotExist: - return error_response( - error='Run not found', - status_code=404, - request=request - ) -``` - ---- - -## 🎨 FRONTEND CHANGES REQUIRED - -### 1. New Component: CurrentProcessingCard - -**File:** `frontend/src/components/Automation/CurrentProcessingCard.tsx` - -```typescript -interface CurrentProcessingCardProps { - runId: string; - siteId: number; - currentStage: number; - onComplete?: () => void; -} - -const CurrentProcessingCard: React.FC = ({ - runId, - siteId, - currentStage, - onComplete -}) => { - const [processingState, setProcessingState] = useState(null); - - // Poll every 3 seconds - useEffect(() => { - const fetchState = async () => { - const state = await automationService.getCurrentProcessing(siteId, runId); - setProcessingState(state); - - // If stage completed, trigger refresh - if (state && state.processed_items === state.total_items) { - onComplete?.(); - } - }; - - fetchState(); - const interval = setInterval(fetchState, 3000); - - return () => clearInterval(interval); - }, [siteId, runId]); - - if (!processingState) return null; - - const percentage = Math.round( - (processingState.processed_items / processingState.total_items) * 100 - ); - - return ( -
- {/* Header */} -
-
-
- -
-
-

- Automation In Progress -

-

- Stage {currentStage}: {processingState.stage_name} - - {processingState.stage_type} - -

-
-
-
-
{percentage}%
-
- {processingState.processed_items}/{processingState.total_items} processed -
-
-
- - {/* Progress Bar */} -
-
-
-
-
- - {/* Currently Processing */} -
-
-

- Currently Processing: -

-
- {processingState.currently_processing.map((item, idx) => ( -
- • - - {item.title} - -
- ))} -
-
- -
-

- Up Next: -

-
- {processingState.up_next.map((item, idx) => ( -
- • - - {item.title} - -
- ))} - {processingState.remaining_count > processingState.up_next.length && ( -
- + {processingState.remaining_count - processingState.up_next.length} more in queue -
- )} -
-
-
-
- ); -}; -``` - -### 2. Update AutomationPage.tsx - -**File:** `frontend/src/pages/Automation/AutomationPage.tsx` - -```typescript -// Add new import -import CurrentProcessingCard from '../../components/Automation/CurrentProcessingCard'; - -// In the component -return ( -
- - - {/* Current Processing Card - MOVE TO TOP */} - {currentRun?.status === 'running' && ( - { - // Refresh full page metrics when stage completes - loadAutomationData(); - }} - /> - )} - - {/* Metrics Cards */} -
- {/* ... existing metrics ... */} -
- - {/* Stages Section */} - -

Pipeline Stages

- {/* ... existing stages ... */} - - - {/* Rest of the page ... */} -
-); -``` - -### 3. Add Service Method - -**File:** `frontend/src/services/automationService.ts` - -```typescript -export interface ProcessingState { - run_id: string; - current_stage: number; - stage_name: string; - stage_type: 'AI' | 'Local' | 'Manual'; - total_items: number; - processed_items: number; - percentage: number; - currently_processing: Array<{ - id: number; - title: string; - type: string; - }>; - up_next: Array<{ - id: number; - title: string; - type: string; - }>; - remaining_count: number; -} - -// Add to automationService -getCurrentProcessing: async ( - siteId: number, - runId: string -): Promise => { - return fetchAPI( - buildUrl('/current_processing/', { site_id: siteId, run_id: runId }) - ); -}, -``` - ---- - -## 🧪 TESTING PLAN - -### Unit Tests - -- [ ] Test `get_current_processing_state()` for each stage -- [ ] Test `_get_processed_count()` calculation -- [ ] Test `_get_current_items()` formatting -- [ ] Test API endpoint with various run states - -### Integration Tests - -- [ ] Test polling updates every 3 seconds -- [ ] Test stage completion triggers full refresh -- [ ] Test card disappears when automation completes -- [ ] Test with 0 items (edge case) -- [ ] Test with 1000+ items (performance) - -### Visual/UX Tests - -- [ ] Card positioned at top of page -- [ ] Progress bar animates smoothly -- [ ] Record names display correctly -- [ ] Responsive design (mobile/tablet/desktop) -- [ ] Dark mode support -- [ ] Loading states -- [ ] Error states - ---- - -## 📊 STAGE-SPECIFIC DISPLAY FORMATS - -### Stage 1: Keywords → Clusters - -``` -Currently Processing: -• "best seo tools" -• "content marketing platforms" -• "ai writing assistants" - -+ 47 more keywords in queue - -Progress: 3/50 keywords processed -``` - -### Stage 2: Clusters → Ideas - -``` -Currently Processing: -• "SEO Tools and Software" (Cluster #12) - -Up Next: -• "Content Marketing Strategies" -• "AI Content Generation" - -Progress: 12/25 clusters processed -``` - -### Stage 3: Ideas → Tasks - -``` -Currently Processing: -• "10 Best SEO Tools for 2025" - -Up Next: -• "How to Create Content with AI" -• "Content Marketing ROI Calculator" - -Progress: 8/30 ideas processed -``` - -### Stage 4: Tasks → Content - -``` -Currently Processing: -• "Ultimate Guide to SEO in 2025" (2,500 words) - -Up Next: -• "AI Content Creation Best Practices" - -Progress: 5/15 tasks processed -``` - -### Stage 5: Content → Image Prompts - -``` -Currently Processing: -• "How to Use ChatGPT for Content" (Extracting 3 image prompts) - -Up Next: -• "Best AI Image Generators 2025" - -Progress: 10/15 content pieces processed -``` - -### Stage 6: Image Prompts → Images - -``` -Currently Processing: -• Featured image for "SEO Guide 2025" - -Up Next: -• In-article image #1 for "SEO Guide 2025" -• In-article image #2 for "SEO Guide 2025" - -Progress: 15/45 images generated -``` - -### Stage 7: Manual Review Gate - -``` -Automation Complete! ✅ - -Ready for Review: -• "Ultimate Guide to SEO in 2025" -• "AI Content Creation Best Practices" -• "Best Image Generators 2025" - -+ 12 more content pieces - -Total: 15 content pieces ready for review -``` - ---- - -## 🎯 SUCCESS METRICS - -### User Experience - -✅ Users can see **exactly what's being processed** at any moment -✅ Users know **what's coming up next** in the queue -✅ Users can estimate **remaining time** based on progress -✅ Users get **quantitative feedback** (percentage, counts) -✅ Users see **smooth, non-disruptive updates** (no page flicker) - -### Technical - -✅ Polling interval: 3 seconds (balance between freshness and load) -✅ API response time: < 200ms -✅ Component re-render: Only the processing card, not entire page -✅ Memory usage: No memory leaks from polling -✅ Error handling: Graceful degradation if API fails - ---- - -## 🚀 IMPLEMENTATION PHASES - -### Phase 1: Backend (1-2 days) -- [ ] Implement `get_current_processing_state()` method -- [ ] Add `/current_processing/` API endpoint -- [ ] Test with all 7 stages -- [ ] Add unit tests - -### Phase 2: Frontend (2-3 days) -- [ ] Create `CurrentProcessingCard` component -- [ ] Add polling logic with cleanup -- [ ] Style with Tailwind (match existing design system) -- [ ] Add dark mode support -- [ ] Integrate into `AutomationPage` - -### Phase 3: Testing & Refinement (1-2 days) -- [ ] Integration testing -- [ ] Performance testing -- [ ] UX testing -- [ ] Bug fixes - -### Phase 4: Deployment -- [ ] Deploy backend changes -- [ ] Deploy frontend changes -- [ ] Monitor first automation runs -- [ ] Collect user feedback - ---- - -## 🔄 FUTURE ENHANCEMENTS - -### V2 Features (Post-MVP) - -1. **Estimated Time Remaining:** - ``` - Progress: 15/50 keywords processed - Estimated time remaining: ~8 minutes - ``` - -2. **Stage-Level Progress Bar:** - - Each stage shows its own mini progress bar - - Visual indicator of which stages are complete - -3. **Click to View Details:** - - Click on a record name to see modal with details - - Preview generated content/images - -4. **Pause/Resume from Card:** - - Add pause button directly in the card - - Quick action without scrolling - -5. **Export Processing Log:** - - Download real-time processing log - - CSV of all processed items with timestamps - ---- - -## END OF PLAN - -This plan provides a comprehensive UX improvement for automation progress tracking, making the process transparent and user-friendly while maintaining system performance. diff --git a/docs/automation/automation-stage-6-image-generation-fix.md b/docs/automation/automation-stage-6-image-generation-fix.md deleted file mode 100644 index e1e842ca..00000000 --- a/docs/automation/automation-stage-6-image-generation-fix.md +++ /dev/null @@ -1,403 +0,0 @@ -# Automation Stage 6 - Image Generation Fix Plan - -**Date:** December 4, 2025 -**Status:** Analysis Complete - Implementation Required -**Priority:** HIGH - ---- - -## 🔍 PROBLEM IDENTIFICATION - -### Current Issue - -Stage 6 of the automation pipeline (Image Prompts → Generated Images) is **NOT running correctly**. The issue stems from using the wrong AI function for image generation. - -### Root Cause Analysis - -**Current Implementation (INCORRECT):** -```python -# File: backend/igny8_core/business/automation/services/automation_service.py -# Line ~935 - -engine = AIEngine(account=self.account) -result = engine.execute( - fn=GenerateImagesFunction(), - payload={'image_ids': [image.id]} # ❌ WRONG -) -``` - -**Why It Fails:** - -1. `GenerateImagesFunction()` expects: - - Input: `{'ids': [task_ids]}` (Task IDs, NOT Image IDs) - - Purpose: Extract prompts from Tasks and generate images for tasks - - Use case: When you have Tasks with content but no images - -2. Automation Stage 6 has: - - Input: Images records with `status='pending'` (already have prompts) - - Purpose: Generate actual image URLs from existing prompts - - Context: Images were created in Stage 5 by `GenerateImagePromptsFunction` - -### How Other Stages Work Correctly - -**Stage 1:** Keywords → Clusters -```python -engine.execute( - fn=AutoClusterFunction(), - payload={'keyword_ids': keyword_ids} # ✅ Correct -) -``` - -**Stage 2:** Clusters → Ideas -```python -engine.execute( - fn=GenerateIdeasFunction(), - payload={'cluster_ids': cluster_ids} # ✅ Correct -) -``` - -**Stage 4:** Tasks → Content -```python -engine.execute( - fn=GenerateContentFunction(), - payload={'ids': task_ids} # ✅ Correct -) -``` - -**Stage 5:** Content → Image Prompts -```python -engine.execute( - fn=GenerateImagePromptsFunction(), - payload={'ids': content_ids} # ✅ Correct -) -``` - -**Stage 6:** Image Prompts → Images (BROKEN) -```python -# Currently uses GenerateImagesFunction (WRONG) -# Should use process_image_generation_queue (CORRECT) -``` - ---- - -## ✅ CORRECT SOLUTION - -### The Right Approach - -**Use `process_image_generation_queue` Celery task** - This is the same approach used by: -1. Writer/Images page (`/writer/images/generate_images/` endpoint) -2. Manual image generation from prompts - -**Evidence from Working Code:** - -```python -# File: backend/igny8_core/modules/writer/views.py -# ImagesViewSet.generate_images() - -from igny8_core.ai.tasks import process_image_generation_queue - -task = process_image_generation_queue.delay( - image_ids=image_ids, # ✅ Accepts image_ids - account_id=account_id, - content_id=content_id -) -``` - -**What `process_image_generation_queue` Does:** - -1. ✅ Accepts `image_ids` (list of Image record IDs) -2. ✅ Each Image record already has a `prompt` field (populated by Stage 5) -3. ✅ Generates images sequentially with progress tracking -4. ✅ Updates Images records: `status='pending'` → `status='generated'` -5. ✅ Downloads and saves images locally -6. ✅ Automatically handles credits deduction -7. ✅ Supports multiple providers (OpenAI, Runware) -8. ✅ Handles errors gracefully (continues on failure) - ---- - -## 📋 IMPLEMENTATION PLAN - -### Changes Required - -**File:** `backend/igny8_core/business/automation/services/automation_service.py` - -**Location:** `run_stage_6()` method (lines 874-1022) - -### Step 1: Import the Correct Task - -**Current:** -```python -from igny8_core.ai.functions.generate_images import GenerateImagesFunction -``` - -**Add:** -```python -from igny8_core.ai.tasks import process_image_generation_queue -``` - -### Step 2: Modify Stage 6 Logic - -**Replace this block (lines ~920-945):** - -```python -# INCORRECT - Delete this -for idx, image in enumerate(image_list, 1): - try: - content_title = image.content.title if image.content else 'Unknown' - self.logger.log_stage_progress( - self.run.run_id, self.account.id, self.site.id, - stage_number, f"Generating image {idx}/{total_images}: {image.image_type} for '{content_title}'" - ) - - # Call AI function via AIEngine - engine = AIEngine(account=self.account) - result = engine.execute( - fn=GenerateImagesFunction(), - payload={'image_ids': [image.id]} # ❌ WRONG - ) - - # Monitor task - task_id = result.get('task_id') - if task_id: - self._wait_for_task(task_id, stage_number, f"Image for '{content_title}'", continue_on_error=True) - - images_processed += 1 -``` - -**With this:** - -```python -# CORRECT - Use process_image_generation_queue -for idx, image in enumerate(image_list, 1): - try: - content_title = image.content.title if image.content else 'Unknown' - self.logger.log_stage_progress( - self.run.run_id, self.account.id, self.site.id, - stage_number, f"Generating image {idx}/{total_images}: {image.image_type} for '{content_title}'" - ) - - # Call process_image_generation_queue directly (same as Writer/Images page) - from igny8_core.ai.tasks import process_image_generation_queue - - # Queue the task - if hasattr(process_image_generation_queue, 'delay'): - task = process_image_generation_queue.delay( - image_ids=[image.id], - account_id=self.account.id, - content_id=image.content.id if image.content else None - ) - task_id = str(task.id) - else: - # Fallback for testing (synchronous) - result = process_image_generation_queue( - image_ids=[image.id], - account_id=self.account.id, - content_id=image.content.id if image.content else None - ) - task_id = None - - # Monitor task (if async) - if task_id: - self._wait_for_task(task_id, stage_number, f"Image for '{content_title}'", continue_on_error=True) - - images_processed += 1 -``` - -### Step 3: Update Logging - -The logging structure remains the same, just update the log messages to reflect the correct process: - -```python -self.logger.log_stage_progress( - self.run.run_id, self.account.id, self.site.id, - stage_number, f"Image generation task queued for '{content_title}' ({images_processed}/{total_images})" -) -``` - -### Step 4: No Changes Needed For - -✅ Stage 5 (Image Prompt Extraction) - Already correct -✅ Images table structure - Already has all required fields -✅ Progress tracking - Already implemented in `process_image_generation_queue` -✅ Credits deduction - Automatic in `process_image_generation_queue` -✅ Error handling - Built into the task with `continue_on_error=True` - ---- - -## 🔄 HOW IT WORKS (CORRECTED FLOW) - -### Stage 5: Content → Image Prompts - -``` -Input: Content (status='draft', no images) -AI: GenerateImagePromptsFunction -Output: Images (status='pending', prompt='...') -``` - -### Stage 6: Image Prompts → Generated Images (FIXED) - -``` -Input: Images (status='pending', has prompt) -Task: process_image_generation_queue (Celery task) -AI: Calls OpenAI/Runware API with prompt -Output: Images (status='generated', image_url='https://...', image_path='/path/to/file') -``` - -### What Happens in process_image_generation_queue - -1. **Load Image Record:** - - Get Image by ID - - Read existing `prompt` field (created in Stage 5) - - Get Content for template formatting - -2. **Format Prompt:** - - Use image_prompt_template from PromptRegistry - - Format: `"Create a {image_type} image for '{post_title}'. Prompt: {image_prompt}"` - - Handle model-specific limits (DALL-E 3: 4000 chars, DALL-E 2: 1000 chars) - -3. **Generate Image:** - - Call `AICore.generate_image()` - - Uses configured provider (OpenAI/Runware) - - Uses configured model (dall-e-3, runware:97@1, etc.) - - Respects image size settings - -4. **Download & Save:** - - Download image from URL - - Save to `/data/app/igny8/frontend/public/images/ai-images/` - - Update Image record with both `image_url` and `image_path` - -5. **Update Status:** - - `status='pending'` → `status='generated'` - - Triggers automatic Content status update (if all images generated) - -6. **Deduct Credits:** - - Automatic via `AICore` credit system - - Records in `AIUsageLog` - ---- - -## 🧪 TESTING CHECKLIST - -### Pre-Deployment Tests - -- [ ] **Unit Test:** Verify `process_image_generation_queue` works with single image -- [ ] **Integration Test:** Run Stage 6 with 3-5 pending images -- [ ] **Error Handling:** Test with invalid image ID -- [ ] **Credits:** Verify credits are deducted correctly -- [ ] **Multi-Provider:** Test with both OpenAI and Runware - -### Post-Deployment Validation - -- [ ] **Full Pipeline:** Run Automation from Stage 1 → Stage 7 -- [ ] **Verify Stage 5 Output:** Images created with `status='pending'` and prompts -- [ ] **Verify Stage 6 Output:** Images updated to `status='generated'` with URLs -- [ ] **Check Downloads:** Images saved to `/data/app/igny8/frontend/public/images/ai-images/` -- [ ] **Monitor Logs:** Review automation logs for Stage 6 completion -- [ ] **Credits Report:** Confirm Stage 6 credits recorded in automation results - -### Success Criteria - -✅ Stage 6 completes without errors -✅ All pending images get generated -✅ Images are downloaded and accessible -✅ Content status automatically updates when all images generated -✅ Credits are properly deducted and logged -✅ Automation proceeds to Stage 7 (Manual Review Gate) - ---- - -## 📊 COMPARISON: BEFORE vs AFTER - -### BEFORE (Broken) - -```python -# ❌ WRONG APPROACH -GenerateImagesFunction() -- Expects: task_ids -- Purpose: Extract prompts from Tasks -- Problem: Doesn't work with Images that already have prompts -``` - -**Result:** Stage 6 fails, images never generated - -### AFTER (Fixed) - -```python -# ✅ CORRECT APPROACH -process_image_generation_queue() -- Accepts: image_ids -- Purpose: Generate images from existing prompts -- Works with: Images (status='pending' with prompts) -``` - -**Result:** Stage 6 succeeds, images generated sequentially with progress tracking - ---- - -## 🔒 SAFETY & ROLLBACK - -### Backup Plan - -If the fix causes issues: - -1. **Rollback Code:** - - Git revert the automation_service.py changes - - Automation still works for Stages 1-5 - -2. **Manual Workaround:** - - Users can manually generate images from Writer/Images page - - This uses the same `process_image_generation_queue` task - -3. **No Data Loss:** - - Stage 5 already created Images with prompts - - These remain in database and can be processed anytime - ---- - -## 📝 IMPLEMENTATION STEPS - -1. **Update Code:** Modify `run_stage_6()` as documented above -2. **Test Locally:** Run automation with test data -3. **Code Review:** Verify changes match working Writer/Images implementation -4. **Deploy:** Push to production -5. **Monitor:** Watch first automation run for Stage 6 completion -6. **Validate:** Check images generated and credits deducted - ---- - -## 🎯 EXPECTED OUTCOME - -After implementing this fix: - -✅ **Stage 6 will work correctly** - Images generate from prompts -✅ **Consistent with manual flow** - Same logic as Writer/Images page -✅ **Proper credits tracking** - Automated deduction via AICore -✅ **Sequential processing** - One image at a time with progress -✅ **Error resilience** - Continues on failure, logs errors -✅ **Full pipeline completion** - Automation flows from Stage 1 → Stage 7 - ---- - -## 🔗 RELATED FUNCTIONS - -### Keep These Functions (Working Correctly) - -- `GenerateImagePromptsFunction` - Stage 5 ✅ -- `AutoClusterFunction` - Stage 1 ✅ -- `GenerateIdeasFunction` - Stage 2 ✅ -- `GenerateContentFunction` - Stage 4 ✅ - -### Use This Task for Stage 6 - -- `process_image_generation_queue` - Celery task for Images → Generated Images ✅ - -### DO NOT USE in Automation - -- `GenerateImagesFunction` - For Tasks, not for Images with existing prompts ❌ - ---- - -## END OF PLAN - -This plan provides a clear, actionable fix for Automation Stage 6 image generation, aligning it with the working manual image generation flow used throughout the application. diff --git a/docs/backend/IGNY8-BACKEND-ARCHITECTURE.md b/docs/backend/IGNY8-BACKEND-ARCHITECTURE.md new file mode 100644 index 00000000..6f62a55f --- /dev/null +++ b/docs/backend/IGNY8-BACKEND-ARCHITECTURE.md @@ -0,0 +1,55 @@ +# IGNY8 Backend Architecture (code-sourced, Dec 2025) + +Purpose: backend-only view based on current code under `backend/`. No legacy docs or assumptions. + +## 1) Stack & Cross-Cutting +- Django + DRF; Celery present via `backend/igny8_core/celery.py` and `tasks/`. +- Multi-tenancy: models inherit `AccountBaseModel` (tenant FK `tenant_id`) or `SiteSectorBaseModel` for site/sector scoping (`backend/igny8_core/auth/models.py`). Account carries billing fields (email/address/tax_id). +- Project URLs wired in `backend/igny8_core/urls.py`. + +## 2) API Namespaces (as routed) +- `/api/v1/auth/` → `igny8_core.auth.urls` (auth + CSV admin helpers). +- `/api/v1/account/` → `igny8_core.api.urls` (account settings, team, usage analytics). +- `/api/v1/planner/` → `igny8_core.modules.planner.urls` (keywords, clusters, ideas ViewSets). +- `/api/v1/writer/` → `igny8_core.modules.writer.urls` (tasks, images, content, taxonomies ViewSets). +- `/api/v1/system/` → `igny8_core.modules.system.urls` (prompts, author profiles, strategies, settings: system/account/user/modules/ai, module-enable toggle, health/status/metrics, Gitea webhook, integration settings save/test/generate/progress). +- `/api/v1/billing/` → `igny8_core.business.billing.urls` (tenant billing: invoices, payments, credit-packages, credit-transactions, payment-methods CRUD/default, credits balance/usage/transactions; manual payment submit; available methods). +- `/api/v1/admin/` → `igny8_core.modules.billing.admin_urls` (billing admin: stats, user credit adjustments, credit costs, invoices/payments/pending approvals, approve/reject, payment-method configs, account payment methods CRUD/default). +- `/api/v1/automation/` → `igny8_core.business.automation.urls` (automation ViewSet). +- `/api/v1/linker/` → `igny8_core.modules.linker.urls` (linker ViewSet). +- `/api/v1/optimizer/` → `igny8_core.modules.optimizer.urls` (optimizer ViewSet). +- `/api/v1/publisher/` → `igny8_core.modules.publisher.urls` (publishing records, deployments, root publisher actions, public site definition). +- `/api/v1/integration/` → `igny8_core.modules.integration.urls` (integrations ViewSet + WordPress status/metadata webhooks). +- OpenAPI docs: `/api/schema/`, `/api/docs/`, `/api/redoc/`. + +## 3) Key Domain Models (code references) +- Billing (`business/billing/models.py`): + - Invoice: `subtotal`, `tax`, `total`, `currency`, `status` (`draft|pending|paid|void|uncollectible`), dates, `line_items`, `stripe_invoice_id`, `payment_method`, billing period, helpers `subtotal_amount|tax_amount|total_amount`. + - Payment: statuses include `pending_approval`, `processing`, `completed/succeeded`, `failed/refunded/cancelled`; methods `stripe|paypal|bank_transfer|local_wallet|manual`; intent/charge ids, manual references, approvals, timestamps, `failure_reason`. + - CreditPackage: slugged packages with `price`, `credits`, `discount_percentage`, stripe/paypal ids, `is_active`, `is_featured`, `sort_order`, `features`. + - CreditTransaction: `transaction_type`, `amount`, `balance_after`, `metadata`, `reference_id`. + - CreditUsageLog: `operation_type`, `credits_used`, tokens/model, related object refs, metadata. + - CreditCostConfig: per-operation configurable costs (admin-editable). + - AccountPaymentMethod: CRUD + default toggle; PaymentMethodConfig for availability by country/method. +- Account (`auth/models.py`): Account with billing fields, credits balance, status, owner, stripe_customer_id; base models enforce tenant scoping. +- Other business areas (content, planning, optimization, publishing, integration) define models/services under `backend/igny8_core/business/*` (models present in `content`, `planning`, `optimization`, `publishing`, `integration`; `linking` uses services). + +## 4) Services & Admin Surfaces +- Billing services: `business/billing/services/invoice_service.py`, `payment_service.py` (used by InvoiceViewSet/PaymentViewSet/admin aliases). +- Admin billing aliases in `modules/billing/admin_urls.py` map to `AdminBillingViewSet` plus legacy stats/credit-cost endpoints. +- Module enable and settings handled in `modules/system/settings_views.py` and integration settings in `modules/system/integration_views.py`. + +## 5) Automation & Tasks +- Automation API exposed via `business/automation/urls.py` (AutomationViewSet). +- Celery tasks scaffolded under `igny8_core/tasks` and `tasks.py`; workers started via standard Celery entrypoints (see repo root README for commands). + +## 6) Integration & Webhooks +- WordPress webhooks: `/api/v1/integration/webhooks/wordpress/status/`, `/metadata/`. +- Gitea webhook: `/api/v1/system/webhook/`. +- Public site definition: `/api/v1/publisher/sites//definition/`. + +## 7) Observations / gaps for follow-up +- `docs/user-flow/` lacks flows; backend routes above should anchor those docs. +- Ensure billing/admin docs use the namespaces and models listed here (avoid legacy paths). Data shape examples still need to be added alongside serializers. + + diff --git a/docs/backend/IGNY8-PLANNER-BACKEND.md b/docs/backend/IGNY8-PLANNER-BACKEND.md new file mode 100644 index 00000000..358f117d --- /dev/null +++ b/docs/backend/IGNY8-PLANNER-BACKEND.md @@ -0,0 +1,54 @@ +# Planner Module (Backend) — Code-Sourced Overview (Dec 2025) + +Scope: `backend/igny8_core/modules/planner` (ViewSets/serializers/urls) and backing models in `backend/igny8_core/business/planning`. + +## Endpoints (routed from `modules/planner/urls.py`) +- `/api/v1/planner/keywords/` — CRUD + bulk actions, CSV import/export, clustering trigger. +- `/api/v1/planner/clusters/` — CRUD with aggregated stats (keywords/volume/difficulty/ideas/content). +- `/api/v1/planner/ideas/` — CRUD for content ideas tied to clusters/keywords. + +## Models (business/planning/models.py) +- **Clusters** (`SiteSectorBaseModel`, soft-delete): `name`, `description`, `keywords_count`, `volume`, `mapped_pages`, `status(new|mapped)`, `disabled`; unique per `site, sector`. Indexes on `name`, `status`, `site, sector`. +- **Keywords** (`SiteSectorBaseModel`, soft-delete): FK `seed_keyword` (global), overrides `volume_override`, `difficulty_override`, `attribute_values`; FK `cluster` (same sector); `status(new|mapped)`, `disabled`. Unique per `seed_keyword, site, sector`. Properties expose `keyword`, `volume`, `difficulty`, `intent` from seed keyword. Save enforces industry/sector alignment between site/sector and seed keyword. +- **ContentIdeas** (`SiteSectorBaseModel`, soft-delete): `idea_title`, `description`, `target_keywords` (legacy), M2M `keyword_objects`, FK `keyword_cluster` (same sector), `status(new|queued|completed)`, `disabled`, `estimated_word_count`, `content_type` (post/page/product/taxonomy), `content_structure` (article/guide/etc.). Tracks metadata (audience, tone, outlines, brief), CTA fields, review flags (`is_priority`, `is_assigned`, `is_review_required`), language, published URL, audit fields. Indexes on `site, sector, keyword_cluster, status, idea_title`. + +## Serializers +- **KeywordSerializer**: read-only keyword metrics from seed keyword; requires `seed_keyword_id` on create; optional on update; validates site/sector via the ViewSet; surfaces `cluster_name`, `sector_name`, `volume/difficulty/intent`. +- **ClusterSerializer** (`cluster_serializers.py`): computes `keywords_count`, `volume`, `difficulty`, `ideas_count`, `content_count`; bulk prefetch helper to avoid N+1. Annotated volume/difficulty via overrides or seed keyword values. +- **ContentIdeasSerializer**: links to clusters and keywords; exposes cluster/sector names; supports site/sector write-only ids; legacy taxonomy getter retained for backward compatibility. + +## ViewSets (modules/planner/views.py) +- **KeywordViewSet** (`SiteSectorModelViewSet`): + - Filtering/search/order: search by seed keyword text; filter by status, cluster, intent, seed_keyword_id; ordering by created_at, volume, difficulty. + - Custom filters: difficulty_min/max, volume_min/max using override-first logic. + - Bulk actions: `bulk_delete`, `bulk_update` (status), `bulk_add_from_seed` (validates industry/sector alignment, requires site/sector), `import_keywords` CSV (site/sector required, creates Keywords), `export` CSV (ids filter supported). + - Clustering: `auto_cluster` calls `ClusteringService.cluster_keywords`; enforces min keyword count (5) and credits; returns async task_id or sync result. + - Create requires site_id and sector_id (validated against site/sector and account). +- **ClusterViewSet**: + - Filters/search/order: search by name; filter by status; ordering on name/created_at/keywords_count/volume/difficulty. + - Annotates volume/difficulty using overrides; uses serializer prefetch to reduce N+1. + - Export endpoints not present; CRUD via standard actions. +- **ContentIdeasViewSet**: + - CRUD for ideas; filters on status/cluster; search by title/description/keywords; ordering by created_at/status. + - Generates title/brief via `IdeasService.generate_content_idea`; enforces credits (catches `InsufficientCreditsError`). + - Bulk assign status not present; focused on single-item generation and listing. + +## Services (business/planning/services) +- **ClusteringService**: clusters keywords; invoked by `auto_cluster`; credit-aware. +- **IdeasService**: generates content ideas (title/brief) from keywords; used in `ContentIdeasViewSet`. + +## Permissions, tenancy, and throttling +- Permissions: `IsAuthenticatedAndActive` + `IsViewerOrAbove` for all planner endpoints. +- Tenancy: `SiteSectorModelViewSet` base ensures account/site/sector scoping; create/import/add-from-seed require explicit site_id + sector_id with consistency checks. +- Throttling: `DebugScopedRateThrottle` with scope `planner`. + +## CSV Import/Export +- Export: `/planner/keywords/export` supports ids filter; outputs CSV with keyword, volume, difficulty, intent, status, cluster. +- Import: `/planner/keywords/import_keywords` expects CSV with keyword, volume, difficulty, intent, status; site_id and sector_id required in query params; skips duplicates per site/sector/account. + +## Notes / gaps +- Two `bulk_update` methods exist in `KeywordViewSet` (duplicate definitions); consolidate to one. +- No dedicated bulk idea status update; consider parity with keywords. +- Legacy taxonomy references removed; ContentIdeas retains a legacy getter for taxonomy name but model FK is removed. + + diff --git a/docs/backend/IGNY8-WRITER-BACKEND.md b/docs/backend/IGNY8-WRITER-BACKEND.md new file mode 100644 index 00000000..965e2da9 --- /dev/null +++ b/docs/backend/IGNY8-WRITER-BACKEND.md @@ -0,0 +1,73 @@ +# Writer Module (Backend) — Code-Sourced Overview (Dec 2025) + +Scope: `backend/igny8_core/modules/writer` (ViewSets/serializers/urls) backed by models in `backend/igny8_core/business/content`. + +## Endpoints (from `modules/writer/urls.py`) +- `/api/v1/writer/tasks/` — CRUD + bulk delete/update, auto content generation. +- `/api/v1/writer/images/` — CRUD + serve file, auto generate images, bulk status update (by content or ids), grouped content images. +- `/api/v1/writer/content/` — CRUD + publish/unpublish to WordPress, check WordPress status, AI prompt generation, validation, metadata mapping, taxonomy/tag management. +- `/api/v1/writer/taxonomies/` — CRUD for content taxonomies (categories/tags). Attributes endpoint disabled (serializer removed). + +## Models (business/content/models.py) +- **Tasks** (`SiteSectorBaseModel`, soft-delete): title, description, FK `cluster` (required, same sector), optional `idea`, `content_type` (post/page/product/taxonomy), `content_structure` (article/guide/landing_page/etc.), optional `taxonomy_term`, comma keywords, word_count, status (`queued|completed`). Indexes on title/status/cluster/type/structure/site+sector. +- **Content** (`SiteSectorBaseModel`, soft-delete): title, HTML, word_count, SEO fields, FK `cluster`, `content_type/structure`, M2M `taxonomy_terms` (through `ContentTaxonomyRelation`), external publishing fields (`external_id/url/type/metadata/sync_status`), source (`igny8|wordpress`), status (`draft|review|published`). Indexes on title/cluster/type/structure/source/status/external_id/site+sector. +- **ContentTaxonomy** (`SiteSectorBaseModel`): name, slug, `taxonomy_type (category|tag)`, external_taxonomy/id, sync_status, description, count, metadata. Unique per site+slug+type and site+external_id+external_taxonomy. +- **Images** (`SiteSectorBaseModel`, soft-delete): FK `content` (preferred) or `task` (legacy), `image_type (featured|desktop|mobile|in_article)`, `image_url/path`, prompt, status (`pending|generated|failed`), position. save() inherits account/site/sector from content/task. Indexes on content/task type/status/position. +- **ContentClusterMap** (summary): links content/tasks to clusters with roles (hub/supporting/attribute) and source (blueprint/manual/import); tracks keywords + slugs. +- **ContentTaxonomyRelation**: through table for content↔taxonomy. + +## Serializers (modules/writer/serializers.py) +- **TasksSerializer**: requires cluster/content_type/content_structure on create; defaults status=queued; exposes cluster_name/sector_name; accepts site_id/sector_id (write-only). +- **ImagesSerializer**: exposes task_title/content_title; read-only account/timestamps. +- **ContentSerializer**: requires cluster/content_type/content_structure/title on create; defaults source=igny8, status=draft; exposes taxonomy terms, tags, categories, image flags/status; site_id/sector_id write-only. +- **ContentTaxonomySerializer** (in file but not shown above): manages taxonomy fields and external sync data. +- Group serializers for content images (`ContentImageSerializer`, `ContentImagesGroupSerializer`). + +## ViewSets & Actions (modules/writer/views.py) +- **TasksViewSet** (`SiteSectorModelViewSet`): + - Filters/search/order: search title/keywords; filter status/cluster/content_type/structure; order by title/created_at/status. + - Bulk: `bulk_delete`, `bulk_update` (status). + - AI: `auto_generate_content` → `ContentGenerationService.generate_content(ids, account)`; limits 10 ids; returns task_id or sync result; 402 on `InsufficientCreditsError`. + - Create requires explicit `site_id` and `sector_id`; validates site/sector association; account resolved from request user/site. +- **ImagesViewSet**: + - Filters/order: filter task_id/content_id/image_type/status; order by created_at/position/id. + - Create enforces site/sector (from request context, falling back to user active site/default sector); sets account/site/sector. + - Actions: + - `serve_image_file` streams local file if `image_path` exists. + - `auto_generate_images` queues `run_ai_task(generate_images)` (Celery if available) for up to 10 task_ids. + - `bulk_update` sets status by `content_id` or `ids`. + - `content_images` returns grouped images (featured + in-article) via grouped serializer. +- **ContentViewSet** (not fully shown above but key actions): + - CRUD with filters/search/order (title, status, cluster, content_type/structure, source); tenant scoping via base class. + - Taxonomy management: add/remove terms, sync helpers (uses `ContentTaxonomy`). + - AI helpers: `generate_image_prompts` (queues `run_ai_task(generate_image_prompts)`), validation (`ContentValidationService`), metadata mapping (`MetadataMappingService`), content generation pathways tied to `ContentGenerationService`. + - Publishing: `publish` queues `publish_content_to_wordpress` Celery task (optimistically sets status=published), `wordpress_status` fetches WP status via integration, `unpublish` clears external links and reverts to draft. + - Image status helpers and grouped image retrieval also exposed. +- **ContentTaxonomyViewSet**: CRUD for categories/tags; supports external sync fields. + +## Permissions, tenancy, throttling +- Permissions: `IsAuthenticatedAndActive` + `IsViewerOrAbove` across Writer; certain actions (e.g., `unpublish`) use `IsEditorOrAbove`. +- Tenancy: `SiteSectorModelViewSet` enforces account/site/sector scoping; create operations require site_id+sector_id or context site/sector; Images save derives account/site/sector from content/task. +- Throttling: `DebugScopedRateThrottle` with scope `writer`. + +## AI/Background Tasks & Limits +- Content generation: `auto_generate_content` (max 10 task IDs) → `ContentGenerationService`; credits enforced (402 on insufficient). +- Image generation: `auto_generate_images` (max 10 task IDs) via `run_ai_task`; Celery preferred, sync fallback. +- Image prompt generation: `generate_image_prompts` via `run_ai_task`. + +## Publishing Integration +- Publish: `/writer/content/{id}/publish/` → queues WordPress publish; refuses if already published. +- Status: `/writer/content/{id}/wordpress_status/` → checks WordPress via site integration. +- Unpublish: clears external_id/url, sets status to draft (does not delete WP post). + +## CSV / Bulk Notes +- Tasks: bulk delete/update supported. +- Images: bulk status update by content or ids; grouped retrieval. +- Content: bulk operations primarily around AI prompt/image generation and publishing; no CSV import/export. + +## Observations / gaps +- Tasks status choices limited to `queued/completed`; no explicit error/cancel states. +- Images bulk_update exposed on same ViewSet name as Tasks bulk_update (distinct routes under images vs tasks); OK but keep consistent naming. +- ContentViewSet file is large; ensure doc readers reference actions for validation/metadata mapping if extending. + + diff --git a/docs/igny8-app/IGNY8-APP-ARCHITECTURE.md b/docs/igny8-app/IGNY8-APP-ARCHITECTURE.md new file mode 100644 index 00000000..7d109507 --- /dev/null +++ b/docs/igny8-app/IGNY8-APP-ARCHITECTURE.md @@ -0,0 +1,59 @@ +# IGNY8 App Architecture (code-sourced, Dec 2025) + +Authoritative snapshot derived from the current codebase (backend Django/DRF, frontend React/Vite). No legacy docs referenced. + +## 1) Platform Overview +- **Multi-tenant:** All tenant-scoped models inherit `AccountBaseModel`; site/sector models inherit `SiteSectorBaseModel` (`backend/igny8_core/auth/models.py`). +- **APIs (wired in `backend/igny8_core/urls.py`):** `/api/v1/auth/`, `/api/v1/account/`, `/api/v1/planner/`, `/api/v1/writer/`, `/api/v1/system/`, `/api/v1/billing/` (tenant billing), `/api/v1/admin/` (billing admin + credit costs), `/api/v1/automation/`, `/api/v1/linker/`, `/api/v1/optimizer/`, `/api/v1/publisher/`, `/api/v1/integration/`. +- **Frontend routes (in `frontend/src/App.tsx`):** Planner, Writer, Automation, Linker, Optimizer, Thinker, Billing, Account, Admin sections; sidebar defined in `frontend/src/layout/AppSidebar.tsx`. + +## 2) Backend Modules & Endpoints +- **Planner** (`modules/planner/urls.py`): `/keywords`, `/clusters`, `/ideas` via ViewSets. +- **Writer** (`modules/writer/urls.py`): `/tasks`, `/images`, `/content`, `/taxonomies`. +- **Automation** (`business/automation/urls.py`): root ViewSet for automation orchestration. +- **Linker** (`modules/linker/urls.py`): root ViewSet for internal linking. +- **Optimizer** (`modules/optimizer/urls.py`): root ViewSet for optimization analysis. +- **Publisher** (`modules/publisher/urls.py`): `/publishing-records`, `/deployments`, root publisher actions, plus public `sites//definition/`. +- **Integration** (`modules/integration/urls.py`): `/integrations`, WordPress webhooks at `/webhooks/wordpress/status|metadata/`. +- **System** (`modules/system/urls.py`): prompts, author profiles, strategies, settings (system/account/user/modules/ai), module enable toggle at `/settings/modules/enable/`, health/status/metrics, Gitea webhook, integration settings save/test/generate/progress routes. +- **Account** (`api/urls.py`): `/settings`, `/team`, `/usage/analytics/`. +- **Billing (tenant)** (`business/billing/urls.py`): invoices, payments (list + manual submit + available methods), credit-packages, credit-transactions, payment-methods (CRUD + set_default), credits balance/usage/transactions. +- **Billing (admin)** (`modules/billing/admin_urls.py` under `/api/v1/admin/`): stats, users credit adjustments, credit costs, plus aliases to `AdminBillingViewSet` for invoices, payments, pending_payments, approve/reject, payment-method-configs, account-payment-methods (CRUD + set_default). + +## 3) Billing & Credits Data Model (key fields) +- **Invoice:** `subtotal`, `tax`, `total`, `currency`, `status`, `invoice_date`, `due_date`, `paid_at`, `line_items`, `stripe_invoice_id`, `payment_method`, billing period fields; helper properties expose `subtotal_amount`, `tax_amount`, `total_amount` for compatibility. +- **Payment:** statuses include `pending_approval`, `processing`, `completed/succeeded`, `failed/refunded/cancelled`; methods include `stripe`, `paypal`, `bank_transfer`, `local_wallet`, `manual`; tracks intent/charge ids, manual references, approvals, timestamps, failure_reason. +- **CreditPackage:** slugged packages with price, credits, stripe/paypal ids, `is_active`, `is_featured`, `sort_order`, features JSON. +- **CreditTransaction:** transaction_type, amount, balance_after, metadata, reference_id (invoice/payment), account-scoped. +- **CreditUsageLog:** operation_type, credits_used, tokens/model, related object refs, metadata; account-scoped. +- **CreditCostConfig:** per-operation configurable costs (admin editable). +- **Account billing fields:** billing email/address/tax_id on `Account` (multi-tenant). + +## 4) Frontend Surface (routes & guards) +- **Planner:** `/planner/keywords`, `/planner/clusters`, `/planner/clusters/:id`, `/planner/ideas`. +- **Writer:** `/writer/tasks`, `/writer/content`, `/writer/content/:id`, `/writer/drafts` (redirect), `/writer/images`, `/writer/review`, `/writer/published`. +- **Automation:** `/automation`. +- **Linker:** `/linker`, `/linker/content`. +- **Optimizer:** `/optimizer`, `/optimizer/content`, `/optimizer/analyze/:id`. +- **Thinker:** `/thinker` redirect → `/thinker/prompts`; also `/thinker/author-profiles`, `/thinker/profile`, `/thinker/strategies`, `/thinker/image-testing`. +- **Billing (tenant module):** `/billing/overview`, `/billing/credits`, `/billing/transactions`, `/billing/usage`. +- **Account:** `/account/plans`, `/account/billing`, `/account/purchase-credits`, `/account/settings`, `/account/team`, `/account/usage`. +- **Admin:** `/admin/dashboard`, `/admin/accounts`, `/admin/subscriptions`, `/admin/account-limits`, `/admin/billing`, `/admin/invoices`, `/admin/payments`, `/admin/payments/approvals`, `/admin/credit-packages`. +- **Sidebar source:** `frontend/src/layout/AppSidebar.tsx` (Account group contains Plans & Billing, Plans, Team Management, Usage & Analytics). + +## 5) Multi-Tenancy & Roles +- Tenant isolation via `AccountBaseModel` FKs (`account_id`/`tenant_id` column) with indexes; site/sector models enforce belonging to same account. +- Admin-only endpoints served under `/api/v1/admin/` (developer/superuser expected; see billing admin ViewSet). +- Frontend ModuleGuard used for module-based access on feature modules (planner/writer/linker/optimizer/thinker); account/billing pages are standard routes. + +## 6) Integration & Webhooks +- WordPress status/metadata webhooks at `/api/v1/integration/webhooks/wordpress/status|metadata/`. +- Gitea webhook at `/api/v1/system/webhook/`. +- Health: `/api/v1/system/ping`, `/api/v1/system/status`, request metrics at `/api/v1/system/request-metrics//`. + +## 7) Known Documentation Needs (to be maintained) +- Align billing/account docs to the above code-level namespaces and models. +- Add frontend route map (kept here) to the Account/Billing docs. +- Fill `docs/user-flow/` (currently empty) with end-to-end user/account/billing flows based on this architecture. + + diff --git a/frontend/src/layout/AppSidebar.tsx b/frontend/src/layout/AppSidebar.tsx index b7533924..15e675ef 100644 --- a/frontend/src/layout/AppSidebar.tsx +++ b/frontend/src/layout/AppSidebar.tsx @@ -201,6 +201,11 @@ const AppSidebar: React.FC = () => { name: "Plans & Billing", path: "/account/billing", }, + { + icon: , + name: "Plans", + path: "/account/plans", + }, { icon: , name: "Team Management", diff --git a/docs/session-config-summary.md b/user-flow-plan-in-progress-1.md similarity index 100% rename from docs/session-config-summary.md rename to user-flow-plan-in-progress-1.md diff --git a/user-flow-plan-in-progress-2.md b/user-flow-plan-in-progress-2.md new file mode 100644 index 00000000..3a9ff35e --- /dev/null +++ b/user-flow-plan-in-progress-2.md @@ -0,0 +1,13 @@ +Differences from the two docs (user-flow + session-config-summary) in the current implementation: + +1) Subscriptions throttling still impacts normal users +- Docs expect “plans fetch should not be throttled” and smooth onboarding; in practice, `/v1/auth/subscriptions/` is still returning 429s for normal users until the backend restart picks up the relaxed scope. + +2) Payment methods are not auto-present for all accounts +- Docs assume payment methods are available so plan selection isn’t blocked. Account 29 had none until we inserted bank_transfer and PK manual; the UI showed “No payment methods available.” + +3) Enterprise plan is hidden for non-aws-admin +- Not called out in the docs; current UI filters out Enterprise for normal tenants. + +4) Onboarding friction from auth/plan calls +- Docs expect a seamless redirect and retry-once for 429; current flow still shows visible errors (429/403) when token is missing/expired or when subscriptions are throttled. \ No newline at end of file diff --git a/docs/user-flow/user-flow-initial b/user-flow-plan.md similarity index 100% rename from docs/user-flow/user-flow-initial rename to user-flow-plan.md