docs 1
This commit is contained in:
IGNY8 VPS (Salman)
24
README.md
24
README.md
@@ -25,13 +25,13 @@ IGNY8 is a full-stack SaaS platform that combines AI-powered content generation
|
||||
|
||||
## Repository Structure
|
||||
|
||||
This monorepo contains two main applications:
|
||||
This monorepo contains two main applications and documentation:
|
||||
|
||||
```
|
||||
igny8/
|
||||
├── backend/ # Django REST API + Celery
|
||||
├── frontend/ # React + Vite SPA
|
||||
├── master-docs/ # Architecture documentation
|
||||
├── docs/ # Documentation index and topic folders
|
||||
└── docker-compose.app.yml # Docker deployment config
|
||||
```
|
||||
|
||||
@@ -210,14 +210,20 @@ The WordPress bridge plugin (`igny8-wp-integration`) creates a bidirectional con
|
||||
|
||||
## Documentation
|
||||
|
||||
Comprehensive documentation is available in the `master-docs/` directory:
|
||||
Start here: [docs/README.md](./docs/README.md) (index of all topics).
|
||||
|
||||
- **[MASTER_REFERENCE.md](./MASTER_REFERENCE.md)** - Complete system architecture and navigation
|
||||
- **[API-COMPLETE-REFERENCE.md](./master-docs/API-COMPLETE-REFERENCE.md)** - Full API documentation
|
||||
- **[02-APPLICATION-ARCHITECTURE.md](./master-docs/02-APPLICATION-ARCHITECTURE.md)** - System design
|
||||
- **[04-BACKEND-IMPLEMENTATION.md](./master-docs/04-BACKEND-IMPLEMENTATION.md)** - Backend details
|
||||
- **[03-FRONTEND-ARCHITECTURE.md](./master-docs/03-FRONTEND-ARCHITECTURE.md)** - Frontend details
|
||||
- **[WORDPRESS-PLUGIN-INTEGRATION.md](./master-docs/WORDPRESS-PLUGIN-INTEGRATION.md)** - Plugin integration guide
|
||||
Common entry points:
|
||||
- App architecture: `docs/igny8-app/IGNY8-APP-ARCHITECTURE.md`
|
||||
- Backend architecture: `docs/backend/IGNY8-BACKEND-ARCHITECTURE.md`
|
||||
- Planner backend detail: `docs/backend/IGNY8-PLANNER-BACKEND.md`
|
||||
- Writer backend detail: `docs/backend/IGNY8-WRITER-BACKEND.md`
|
||||
- Automation: `docs/automation/AUTOMATION-REFERENCE.md`
|
||||
- Tech stack: `docs/tech-stack/00-SYSTEM-ARCHITECTURE-MASTER-REFERENCE.md`
|
||||
- API: `docs/API/API-COMPLETE-REFERENCE-LATEST.md`
|
||||
- Billing & Credits: `docs/billing/billing-account-final-plan-2025-12-05.md`
|
||||
- App guides: `docs/igny8-app/` (planner/writer workflows, taxonomy, feature modification)
|
||||
- WordPress: `docs/wp/` (plugin integration and sync)
|
||||
- Docs changelog: `docs/CHANGELOG.md`
|
||||
|
||||
---
|
||||
|
||||
|
||||
19
docs/CHANGELOG.md
Normal file
19
docs/CHANGELOG.md
Normal file
@@ -0,0 +1,19 @@
|
||||
# IGNY8 Docs Changelog (Dec 2025)
|
||||
|
||||
## 2025-12-07
|
||||
### Features captured (code-sourced)
|
||||
- Added backend-only architecture doc (`docs/backend/IGNY8-BACKEND-ARCHITECTURE.md`) reflecting current namespaces, billing models, integrations, and webhooks.
|
||||
- Added planner backend doc (`docs/backend/IGNY8-PLANNER-BACKEND.md`) and writer backend doc (`docs/backend/IGNY8-WRITER-BACKEND.md`) from live code.
|
||||
- Replaced automation folder with single canonical `docs/automation/AUTOMATION-REFERENCE.md` (code-sourced pipeline, APIs, models, tasks, frontend).
|
||||
- Multi-tenant foundation via `AccountBaseModel` / `SiteSectorBaseModel` with tenant-scoped billing fields on `Account`.
|
||||
- Backend namespaces wired in `backend/igny8_core/urls.py` covering auth, account, planner, writer, system, billing (tenant), admin billing, automation, linker, optimizer, publisher, integration.
|
||||
- Billing tenant surface: invoices, payments (including manual submission + available methods), credit-packages, credit-transactions, payment-methods CRUD/default, credits balance/usage/transactions.
|
||||
- Billing admin surface under `/api/v1/admin/`: stats, users credit adjustments, credit costs, invoices/payments/pending approvals, payment-method configs, account payment methods.
|
||||
- WordPress integration webhooks (`/api/v1/integration/webhooks/wordpress/status|metadata/`) and Gitea webhook (`/api/v1/system/webhook/`).
|
||||
- Frontend routes mapped for Planner, Writer, Automation, Linker, Optimizer, Thinker, Billing module, Account pages, and Admin billing/management; sidebar paths defined in `frontend/src/layout/AppSidebar.tsx`.
|
||||
|
||||
### Issues / gaps to fix
|
||||
- `docs/user-flow/` is empty; needs end-to-end flows (account, billing, content) based on current routes and APIs.
|
||||
- Billing/account documentation must be aligned to the live namespaces and models captured in `docs/igny8-app/IGNY8-APP-ARCHITECTURE.md` (avoid legacy/retired paths).
|
||||
- Add API request/response samples for billing/admin endpoints to the billing docs to match the current serializers (not yet documented here).
|
||||
|
||||
30
docs/README.md
Normal file
30
docs/README.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# IGNY8 Documentation Index (Dec 2025)
|
||||
|
||||
Purpose: single entry point for all project docs with live links to the current sources. This index reflects the codebase as implemented, not legacy placeholders.
|
||||
|
||||
## Codebase Snapshot
|
||||
- **Backend (Django/DRF):** Multi-tenant models inherit `AccountBaseModel`/`SiteSectorBaseModel` for automatic tenant scoping (`backend/igny8_core/auth/models.py`). API namespaces are wired in `backend/igny8_core/urls.py`: `/api/v1/auth/`, `/api/v1/account/`, `/api/v1/planner/`, `/api/v1/writer/`, `/api/v1/system/`, `/api/v1/billing/` (tenant billing/invoices/credits), `/api/v1/admin/` (billing admin + credit costs/stats), `/api/v1/automation/`, `/api/v1/linker/`, `/api/v1/optimizer/`, `/api/v1/publisher/`, `/api/v1/integration/`.
|
||||
- **Frontend (React/Vite):** Routing lives in `frontend/src/App.tsx` with lazy-loaded pages; account/billing pages at `/account/plans`, `/account/billing`, `/account/purchase-credits`, `/account/settings`, `/account/team`, `/account/usage`, plus tenant billing module routes under `/billing/*` and admin billing under `/admin/*`. Sidebar labels/paths are defined in `frontend/src/layout/AppSidebar.tsx`.
|
||||
|
||||
## Document Map
|
||||
- **App Architecture (code-sourced):** `docs/igny8-app/IGNY8-APP-ARCHITECTURE.md`
|
||||
- **Backend Architecture (code-sourced):** `docs/backend/IGNY8-BACKEND-ARCHITECTURE.md`
|
||||
- **Planner Backend Detail:** `docs/backend/IGNY8-PLANNER-BACKEND.md`
|
||||
- **Writer Backend Detail:** `docs/backend/IGNY8-WRITER-BACKEND.md`
|
||||
- **Platform / Tech Stack:** `docs/tech-stack/00-SYSTEM-ARCHITECTURE-MASTER-REFERENCE.md`
|
||||
- **API:** `docs/API/01-IGNY8-REST-API-COMPLETE-REFERENCE.md`, `docs/API/API-COMPLETE-REFERENCE-LATEST.md`
|
||||
- **Billing & Credits (canonical):** `docs/billing/billing-account-final-plan-2025-12-05.md`, `docs/billing/credits-system-audit-and-improvement-plan.md`
|
||||
- **App Feature Guides:** `docs/igny8-app/02-PLANNER-WRITER-WORKFLOW-TECHNICAL-GUIDE.md`, `docs/igny8-app/05-WRITER-IMAGES-PAGE-SYSTEM-DESIGN.md`, `docs/igny8-app/06-FEATURE-MODIFICATION-DEVELOPER-GUIDE.md`, `docs/igny8-app/KEYWORDS-CLUSTERS-IDEAS-COMPLETE-MAPPING.md`, `docs/igny8-app/app-packaging-backaup-plan.md`, `docs/igny8-app/TAXONOMY/*`, `docs/igny8-app/status-related-temporary/*`
|
||||
- **Automation:** `docs/automation/` (automation-plan, implementation analysis, UX improvements, stage-6 image generation fix, cluster validation fix plan)
|
||||
- **AI Functions:** `docs/ai/AI-FUNCTIONS-COMPLETE-REFERENCE.md`
|
||||
- **Automation:** `docs/automation/AUTOMATION-REFERENCE.md`
|
||||
- **WordPress Integration:** `docs/wp/` (API integration guide, bidirectional sync reference, publishing field mapping, refactor/implementation summaries, deployment fixes)
|
||||
- **User Flow Drafts:** root-level `user-flow-plan*.md` (in-progress drafts); `docs/user-flow/` is currently empty and needs coverage.
|
||||
- **Docs Changelog:** `docs/CHANGELOG.md`
|
||||
|
||||
## Known Documentation Gaps to Close
|
||||
- App-level architecture summary sourced from code (backend modules and frontend routes), including tenant scoping rules.
|
||||
- Billing/account accuracy pass: align docs to current endpoints in `igny8_core/urls.py` and admin aliases in `modules/billing/admin_urls.py`, plus the active frontend routes in `App.tsx`.
|
||||
- Frontend route/UX map for account/billing/admin sections tied to sidebar paths.
|
||||
- Multi-tenancy guardrails doc (account/site/sector isolation, role expectations) based on `auth/models.py`.
|
||||
|
||||
@@ -1065,6 +1065,11 @@ Frontend can listen to these events via:
|
||||
- Centralized progress tracking
|
||||
- Easy to add new AI functions (inherit from BaseAIFunction)
|
||||
|
||||
### Current gaps vs code (Dec 2025)
|
||||
- AIEngine now performs a credit pre-check before the AI call (still deducts after SAVE); this is not reflected in earlier notes.
|
||||
- `generate_images` implementation is partially broken: it expects task IDs (not image IDs), tries to read `task.content` (field does not exist), and uses the `extract_image_prompts` prompt path; credit estimation also looks for `image_ids`. Treat it as partial/needs fix.
|
||||
- AIEngine includes messaging/cost maps for `generate_site_structure` (extra function beyond the documented six); not presently documented above.
|
||||
|
||||
---
|
||||
|
||||
## Automation Integration
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
148
docs/automation/AUTOMATION-REFERENCE.md
Normal file
148
docs/automation/AUTOMATION-REFERENCE.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Automation Module (Code-Sourced, Dec 2025)
|
||||
|
||||
Single canonical reference for IGNY8 automation (backend, frontend, and runtime behavior). Replaces all prior automation docs in this folder.
|
||||
|
||||
---
|
||||
|
||||
## 1) What Automation Does
|
||||
- Runs the 7-stage pipeline across Planner/Writer:
|
||||
1) Keywords → Clusters (AI)
|
||||
2) Clusters → Ideas (AI)
|
||||
3) Ideas → Tasks (Local)
|
||||
4) Tasks → Content (AI)
|
||||
5) Content → Image Prompts (AI)
|
||||
6) Image Prompts → Images (AI)
|
||||
7) Manual Review Gate (Manual)
|
||||
- Per-site, per-account isolation. One run at a time per site; guarded by cache lock `automation_lock_{site_id}`.
|
||||
- Scheduling via Celery beat (`automation.check_scheduled_automations`); execution via Celery tasks (`run_automation_task`, `resume_automation_task` / `continue_automation_task`).
|
||||
|
||||
---
|
||||
|
||||
## 2) Backend API (behavior + payloads)
|
||||
Base: `/api/v1/automation/` (auth required; site must belong to user’s account).
|
||||
|
||||
- `GET config?site_id=`: returns or creates config with enable flag, frequency (`daily|weekly|monthly`), scheduled_time, stage_1..6 batch sizes, delays (`within_stage_delay`, `between_stage_delay`), last_run_at, next_run_at.
|
||||
- `PUT update_config?site_id=`: same fields as above, updates in-place.
|
||||
- `POST run_now?site_id=`: starts a manual run; enqueues `run_automation_task`. Fails if a run is already active or lock exists.
|
||||
- `GET current_run?site_id=`: current running/paused run with status, current_stage, totals, and stage_1..7_result blobs (counts, credits, partial flags, skip reasons).
|
||||
- `GET pipeline_overview?site_id=`: per-stage status counts and “pending” numbers for UI cards.
|
||||
- `GET current_processing?site_id=&run_id=`: live processing snapshot for an active run; null if not running.
|
||||
- `POST pause|resume|cancel?site_id=&run_id=`: pause after current item; resume from saved `current_stage`; cancel after current item and stamp cancelled_at/completed_at.
|
||||
- `GET history?site_id=`: last 20 runs (id, status, trigger, timestamps, total_credits_used, current_stage).
|
||||
- `GET logs?run_id=&lines=100`: tail of the per-run activity log written by AutomationLogger.
|
||||
- `GET estimate?site_id=`: estimated_credits, current_balance, sufficient (balance >= 1.2x estimate).
|
||||
|
||||
Error behaviors:
|
||||
- Missing site_id/run_id → 400.
|
||||
- Site not in account → 404.
|
||||
- Run not found → 404 on run-specific endpoints.
|
||||
- Already running / lock held → 400 on run_now.
|
||||
|
||||
---
|
||||
|
||||
## 3) Data Model (runtime state)
|
||||
- `AutomationConfig` (one per site): enable flag, schedule (frequency, time), batch sizes per stage (1–6), delays (within-stage, between-stage), last_run_at, next_run_at.
|
||||
- `AutomationRun`: run_id, trigger_type (manual/scheduled), status (running/paused/cancelled/completed/failed), current_stage, timestamps (start/pause/resume/cancel/complete), total_credits_used, per-stage result JSON (stage_1_result … stage_7_result), error_message.
|
||||
- Activity logs: one file per run via AutomationLogger; streamed through the `logs` endpoint.
|
||||
|
||||
---
|
||||
|
||||
## 4) How Execution Works (AutomationService)
|
||||
- Start: grabs cache lock `automation_lock_{site_id}`, estimates credits, enforces 1.2x balance check, creates AutomationRun and log file.
|
||||
- AI functions used: Stage 1 `AutoClusterFunction`; Stage 2 `GenerateIdeasFunction`; Stage 4 `GenerateContentFunction`; Stage 5 `GenerateImagePromptsFunction`; Stage 6 uses `process_image_generation_queue` (not the partial `generate_images` AI function).
|
||||
- Stage flow (per code):
|
||||
- Stage 1 Keywords → Clusters: require ≥5 keywords (validate_minimum_keywords); batch by config; AIEngine clustering; records keywords_processed, clusters_created, batches, credits, time; skips if insufficient keywords.
|
||||
- Stage 2 Clusters → Ideas: batch by config; AIEngine ideas; records ideas_created.
|
||||
- Stage 3 Ideas → Tasks: local conversion of queued ideas to tasks; batches by config; no AI.
|
||||
- Stage 4 Tasks → Content: batch by config; AIEngine content; records content count + word totals.
|
||||
- Stage 5 Content → Image Prompts: batch by config; AIEngine image-prompts into Images (featured + in-article).
|
||||
- Stage 6 Image Prompts → Images: uses `process_image_generation_queue` with provider/model from IntegrationSettings; updates Images status.
|
||||
- Stage 7 Manual Review Gate: marks ready-for-review counts; no AI.
|
||||
- Control: each stage checks `_check_should_stop` (paused/cancelled); saves partial progress (counts, credits) before returning; resume continues from `current_stage`.
|
||||
- Credits: upfront estimate check (1.2x buffer) before starting; AIEngine per-call pre-checks and post-SAVE deductions; `total_credits_used` accumulates.
|
||||
- Locks: acquired on start; cleared on completion or failure; also cleared on fatal errors in tasks.
|
||||
- Errors: any unhandled exception marks run failed, sets error_message, logs error, clears lock; pipeline_overview/history reflect status.
|
||||
- Stage result fields (persisted):
|
||||
- S1: keywords_processed, clusters_created, batches_run, credits_used, skipped/partial flags, time_elapsed.
|
||||
- S2: clusters_processed, ideas_created, batches_run, credits_used.
|
||||
- S3: ideas_processed, tasks_created, batches_run.
|
||||
- S4: tasks_processed, content_created, total_words, batches_run, credits_used.
|
||||
- S5: content_processed, prompts_created, batches_run, credits_used.
|
||||
- S6: images_processed, images_generated, batches_run.
|
||||
- S7: ready_for_review counts.
|
||||
|
||||
Batching & delays:
|
||||
- Configurable per site; stage_1..6 batch sizes control how many items per batch; `within_stage_delay` pauses between batches; `between_stage_delay` between stages.
|
||||
|
||||
Scheduling:
|
||||
- `check_scheduled_automations` runs hourly; respects frequency/time and last_run_at (~23h guard); skips if a run is active; sets next_run_at; starts `run_automation_task`.
|
||||
|
||||
Celery execution:
|
||||
- `run_automation_task` runs stages 1→7 sequentially for a run_id; failures mark run failed and clear lock.
|
||||
- `resume_automation_task` / `continue_automation_task` continue from saved `current_stage`.
|
||||
- Workers need access to cache (locks) and IntegrationSettings (models/providers).
|
||||
|
||||
Image pipeline specifics:
|
||||
- Stage 5 writes prompts to Images (featured + ordered in-article).
|
||||
- Stage 6 generates images via queue helper; AI `generate_images` remains partial/broken and is not used by automation.
|
||||
|
||||
---
|
||||
|
||||
## 5) Frontend Behavior (AutomationPage)
|
||||
- Route: `/automation`.
|
||||
- What the user can do: run now, pause, resume, cancel; edit config (enable/schedule, batch sizes, delays); view activity log; view history; watch live processing card and pipeline cards update.
|
||||
- Polling: every ~5s while a run is running/paused for current_run, pipeline_overview, metrics, current_processing; lighter polling when idle.
|
||||
- Metrics: fetched via low-level endpoints (keywords/clusters/ideas/tasks/content/images) for authoritative counts.
|
||||
- States shown: running, paused, cancelled, failed, completed; processing card shown when a run exists; pipeline cards use “pending” counts from pipeline_overview.
|
||||
- Activity log: pulled from `logs` endpoint; shown in UI for live tailing.
|
||||
|
||||
---
|
||||
|
||||
## 6) Configuration & Dependencies
|
||||
- Needs IntegrationSettings for AI models and image providers (OpenAI/runware).
|
||||
- Requires Celery beat and workers; cache backend required for locks.
|
||||
- Tenant scoping everywhere: site + account filtering on all automation queries.
|
||||
|
||||
---
|
||||
|
||||
## 7) Known Limitations and Gaps
|
||||
- `generate_images` AI function is partial/broken; automation uses queue helper instead.
|
||||
- Pause/Cancel stop after the current item; no mid-item abort.
|
||||
- Batch defaults are conservative (e.g., stage_2=1, stage_4=1); tune per site for throughput.
|
||||
- Stage 7 is manual; no automated review step.
|
||||
- No automated test suite observed for automation pipeline (stage transitions, pause/resume/cancel, scheduling guards, credit estimation/deduction).
|
||||
- Enhancements to consider: fix or replace `generate_images`; add mid-item abort; surface lock status/owner; broaden batch defaults after validation; add operator-facing doc in app; add tests.
|
||||
|
||||
---
|
||||
|
||||
## 8) Field/Behavior Quick Tables
|
||||
|
||||
### Pipeline “pending” definitions (pipeline_overview)
|
||||
- Stage 1: Keywords with status `new`, cluster is null, not disabled.
|
||||
- Stage 2: Clusters status `new`, not disabled, with no ideas.
|
||||
- Stage 3: ContentIdeas status `new`.
|
||||
- Stage 4: Tasks status `queued`.
|
||||
- Stage 5: Content status `draft` with zero images.
|
||||
- Stage 6: Images status `pending`.
|
||||
- Stage 7: Content status `review`.
|
||||
|
||||
### Stage result fields (stored on AutomationRun)
|
||||
- S1: keywords_processed, clusters_created, batches_run, credits_used, skipped, partial, time_elapsed.
|
||||
- S2: clusters_processed, ideas_created, batches_run, credits_used.
|
||||
- S3: ideas_processed, tasks_created, batches_run.
|
||||
- S4: tasks_processed, content_created, total_words, batches_run, credits_used.
|
||||
- S5: content_processed, prompts_created, batches_run, credits_used.
|
||||
- S6: images_processed, images_generated, batches_run.
|
||||
- S7: ready_for_review.
|
||||
|
||||
### Credit handling
|
||||
- Pre-run: estimate_credits * 1.2 vs account.credits (fails if insufficient).
|
||||
- Per AI call: AIEngine pre-check credits; post-SAVE deduction with cost/tokens tracked; total_credits_used aggregates deductions.
|
||||
|
||||
### Logging
|
||||
- Per-run log file via AutomationLogger; accessed with `GET logs?run_id=&lines=`; includes stage start/progress/errors and batch info.
|
||||
|
||||
### Polling (frontend)
|
||||
- Active run: ~5s cadence for current_run, pipeline_overview, metrics, current_processing, logs tail.
|
||||
- Idle: lighter polling (current_run/pipeline_overview) to show readiness and pending counts.
|
||||
|
||||
@@ -1,753 +0,0 @@
|
||||
# Auto-Cluster Validation Fix Plan
|
||||
|
||||
**Date:** December 4, 2025
|
||||
**Status:** Design Phase
|
||||
**Priority:** MEDIUM
|
||||
|
||||
---
|
||||
|
||||
## 🎯 OBJECTIVE
|
||||
|
||||
Add validation to prevent auto-cluster from running with less than 5 keywords, and ensure both manual auto-cluster and automation pipeline use the same shared validation logic to maintain consistency.
|
||||
|
||||
---
|
||||
|
||||
## 🔍 CURRENT STATE ANALYSIS
|
||||
|
||||
### Current Behavior
|
||||
|
||||
**Auto-Cluster Function:**
|
||||
- Located in: `backend/igny8_core/ai/functions/auto_cluster.py`
|
||||
- No minimum keyword validation
|
||||
- Accepts any number of keywords (even 1)
|
||||
- May produce poor quality clusters with insufficient data
|
||||
|
||||
**Automation Pipeline:**
|
||||
- Located in: `backend/igny8_core/business/automation/services/automation_service.py`
|
||||
- Uses auto-cluster in Stage 1
|
||||
- No pre-check for minimum keywords
|
||||
- May waste credits on insufficient data
|
||||
|
||||
### Problems
|
||||
|
||||
1. ❌ **No Minimum Check:** Auto-cluster runs with 1-4 keywords
|
||||
2. ❌ **Poor Results:** AI cannot create meaningful clusters with < 5 keywords
|
||||
3. ❌ **Wasted Credits:** Charges credits for insufficient analysis
|
||||
4. ❌ **Inconsistent Validation:** No shared validation between manual and automation
|
||||
5. ❌ **User Confusion:** Error occurs during processing, not at selection
|
||||
|
||||
---
|
||||
|
||||
## ✅ PROPOSED SOLUTION
|
||||
|
||||
### Validation Strategy
|
||||
|
||||
**Single Source of Truth:**
|
||||
- Create one validation function
|
||||
- Use it in both auto-cluster function AND automation pipeline
|
||||
- Consistent error messages
|
||||
- No code duplication
|
||||
|
||||
**Error Behavior:**
|
||||
- **Manual Auto-Cluster:** Return error before API call
|
||||
- **Automation Pipeline:** Skip Stage 1 with warning in logs
|
||||
|
||||
---
|
||||
|
||||
## 📋 IMPLEMENTATION PLAN
|
||||
|
||||
### Step 1: Create Shared Validation Module
|
||||
|
||||
**New File:** `backend/igny8_core/ai/validators/cluster_validators.py`
|
||||
|
||||
```python
|
||||
"""
|
||||
Cluster-specific validators
|
||||
Shared between auto-cluster function and automation pipeline
|
||||
"""
|
||||
import logging
|
||||
from typing import Dict, List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def validate_minimum_keywords(
|
||||
keyword_ids: List[int],
|
||||
account=None,
|
||||
min_required: int = 5
|
||||
) -> Dict:
|
||||
"""
|
||||
Validate that sufficient keywords are available for clustering
|
||||
|
||||
Args:
|
||||
keyword_ids: List of keyword IDs to cluster
|
||||
account: Account object for filtering
|
||||
min_required: Minimum number of keywords required (default: 5)
|
||||
|
||||
Returns:
|
||||
Dict with 'valid' (bool) and 'error' (str) or 'count' (int)
|
||||
"""
|
||||
from igny8_core.modules.planner.models import Keywords
|
||||
|
||||
# Build queryset
|
||||
queryset = Keywords.objects.filter(id__in=keyword_ids, status='new')
|
||||
|
||||
if account:
|
||||
queryset = queryset.filter(account=account)
|
||||
|
||||
# Count available keywords
|
||||
count = queryset.count()
|
||||
|
||||
# Validate minimum
|
||||
if count < min_required:
|
||||
return {
|
||||
'valid': False,
|
||||
'error': f'Insufficient keywords for clustering. Need at least {min_required} keywords, but only {count} available.',
|
||||
'count': count,
|
||||
'required': min_required
|
||||
}
|
||||
|
||||
return {
|
||||
'valid': True,
|
||||
'count': count,
|
||||
'required': min_required
|
||||
}
|
||||
|
||||
|
||||
def validate_keyword_selection(
|
||||
selected_ids: List[int],
|
||||
available_count: int,
|
||||
min_required: int = 5
|
||||
) -> Dict:
|
||||
"""
|
||||
Validate keyword selection (for frontend validation)
|
||||
|
||||
Args:
|
||||
selected_ids: List of selected keyword IDs
|
||||
available_count: Total count of available keywords
|
||||
min_required: Minimum required
|
||||
|
||||
Returns:
|
||||
Dict with validation result
|
||||
"""
|
||||
selected_count = len(selected_ids)
|
||||
|
||||
# Check if any keywords selected
|
||||
if selected_count == 0:
|
||||
return {
|
||||
'valid': False,
|
||||
'error': 'No keywords selected',
|
||||
'type': 'NO_SELECTION'
|
||||
}
|
||||
|
||||
# Check if enough selected
|
||||
if selected_count < min_required:
|
||||
return {
|
||||
'valid': False,
|
||||
'error': f'Please select at least {min_required} keywords. Currently selected: {selected_count}',
|
||||
'type': 'INSUFFICIENT_SELECTION',
|
||||
'selected': selected_count,
|
||||
'required': min_required
|
||||
}
|
||||
|
||||
# Check if enough available (even if not all selected)
|
||||
if available_count < min_required:
|
||||
return {
|
||||
'valid': False,
|
||||
'error': f'Not enough keywords available. Need at least {min_required} keywords, but only {available_count} exist.',
|
||||
'type': 'INSUFFICIENT_AVAILABLE',
|
||||
'available': available_count,
|
||||
'required': min_required
|
||||
}
|
||||
|
||||
return {
|
||||
'valid': True,
|
||||
'selected': selected_count,
|
||||
'available': available_count,
|
||||
'required': min_required
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Update Auto-Cluster Function
|
||||
|
||||
**File:** `backend/igny8_core/ai/functions/auto_cluster.py`
|
||||
|
||||
**Add import:**
|
||||
```python
|
||||
from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
|
||||
```
|
||||
|
||||
**Update validate() method:**
|
||||
```python
|
||||
def validate(self, payload: dict, account=None) -> Dict:
|
||||
"""Validate keyword IDs and minimum count"""
|
||||
result = super().validate(payload, account)
|
||||
if not result['valid']:
|
||||
return result
|
||||
|
||||
keyword_ids = payload.get('keyword_ids', [])
|
||||
|
||||
if not keyword_ids:
|
||||
return {'valid': False, 'error': 'No keyword IDs provided'}
|
||||
|
||||
# NEW: Validate minimum keywords using shared validator
|
||||
min_validation = validate_minimum_keywords(
|
||||
keyword_ids=keyword_ids,
|
||||
account=account,
|
||||
min_required=5 # Configurable constant
|
||||
)
|
||||
|
||||
if not min_validation['valid']:
|
||||
# Log the validation failure
|
||||
logger.warning(
|
||||
f"[AutoCluster] Validation failed: {min_validation['error']}"
|
||||
)
|
||||
return min_validation
|
||||
|
||||
# Log successful validation
|
||||
logger.info(
|
||||
f"[AutoCluster] Validation passed: {min_validation['count']} keywords available (min: {min_validation['required']})"
|
||||
)
|
||||
|
||||
return {'valid': True}
|
||||
```
|
||||
|
||||
### Step 3: Update Automation Pipeline
|
||||
|
||||
**File:** `backend/igny8_core/business/automation/services/automation_service.py`
|
||||
|
||||
**Add import:**
|
||||
```python
|
||||
from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
|
||||
```
|
||||
|
||||
**Update run_stage_1() method:**
|
||||
```python
|
||||
def run_stage_1(self):
|
||||
"""Stage 1: Keywords → Clusters (AI)"""
|
||||
stage_number = 1
|
||||
stage_name = "Keywords → Clusters (AI)"
|
||||
start_time = time.time()
|
||||
|
||||
# Query pending keywords
|
||||
pending_keywords = Keywords.objects.filter(
|
||||
site=self.site,
|
||||
status='new'
|
||||
)
|
||||
|
||||
total_count = pending_keywords.count()
|
||||
|
||||
# NEW: Pre-stage validation for minimum keywords
|
||||
keyword_ids = list(pending_keywords.values_list('id', flat=True))
|
||||
|
||||
min_validation = validate_minimum_keywords(
|
||||
keyword_ids=keyword_ids,
|
||||
account=self.account,
|
||||
min_required=5
|
||||
)
|
||||
|
||||
if not min_validation['valid']:
|
||||
# Log validation failure
|
||||
self.logger.log_stage_start(
|
||||
self.run.run_id, self.account.id, self.site.id,
|
||||
stage_number, stage_name, total_count
|
||||
)
|
||||
|
||||
error_msg = min_validation['error']
|
||||
self.logger.log_stage_error(
|
||||
self.run.run_id, self.account.id, self.site.id,
|
||||
stage_number, error_msg
|
||||
)
|
||||
|
||||
# Skip stage with proper result
|
||||
self.run.stage_1_result = {
|
||||
'keywords_processed': 0,
|
||||
'clusters_created': 0,
|
||||
'skipped': True,
|
||||
'skip_reason': error_msg,
|
||||
'credits_used': 0
|
||||
}
|
||||
self.run.current_stage = 2
|
||||
self.run.save()
|
||||
|
||||
logger.warning(f"[AutomationService] Stage 1 skipped: {error_msg}")
|
||||
return
|
||||
|
||||
# Log stage start
|
||||
self.logger.log_stage_start(
|
||||
self.run.run_id, self.account.id, self.site.id,
|
||||
stage_number, stage_name, total_count
|
||||
)
|
||||
|
||||
# ... rest of existing stage logic ...
|
||||
```
|
||||
|
||||
### Step 4: Update API Endpoint
|
||||
|
||||
**File:** `backend/igny8_core/modules/planner/views.py` (KeywordsViewSet)
|
||||
|
||||
**Update auto_cluster action:**
|
||||
```python
|
||||
@action(detail=False, methods=['post'], url_path='auto_cluster', url_name='auto_cluster')
|
||||
def auto_cluster(self, request):
|
||||
"""Auto-cluster keywords using AI"""
|
||||
from igny8_core.ai.tasks import run_ai_task
|
||||
from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
|
||||
|
||||
account = getattr(request, 'account', None)
|
||||
keyword_ids = request.data.get('ids', [])
|
||||
|
||||
if not keyword_ids:
|
||||
return error_response(
|
||||
error='No keyword IDs provided',
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
request=request
|
||||
)
|
||||
|
||||
# NEW: Validate minimum keywords BEFORE queuing task
|
||||
validation = validate_minimum_keywords(
|
||||
keyword_ids=keyword_ids,
|
||||
account=account,
|
||||
min_required=5
|
||||
)
|
||||
|
||||
if not validation['valid']:
|
||||
return error_response(
|
||||
error=validation['error'],
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
request=request,
|
||||
extra_data={
|
||||
'count': validation.get('count'),
|
||||
'required': validation.get('required')
|
||||
}
|
||||
)
|
||||
|
||||
# Validation passed - proceed with clustering
|
||||
account_id = account.id if account else None
|
||||
|
||||
try:
|
||||
if hasattr(run_ai_task, 'delay'):
|
||||
task = run_ai_task.delay(
|
||||
function_name='auto_cluster',
|
||||
payload={'keyword_ids': keyword_ids},
|
||||
account_id=account_id
|
||||
)
|
||||
return success_response(
|
||||
data={'task_id': str(task.id)},
|
||||
message=f'Auto-cluster started with {validation["count"]} keywords',
|
||||
request=request
|
||||
)
|
||||
else:
|
||||
# Synchronous fallback
|
||||
result = run_ai_task(
|
||||
function_name='auto_cluster',
|
||||
payload={'keyword_ids': keyword_ids},
|
||||
account_id=account_id
|
||||
)
|
||||
return success_response(data=result, request=request)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start auto-cluster: {e}", exc_info=True)
|
||||
return error_response(
|
||||
error=f'Failed to start clustering: {str(e)}',
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
request=request
|
||||
)
|
||||
```
|
||||
|
||||
### Step 5: Add Frontend Validation (Optional but Recommended)
|
||||
|
||||
**File:** `frontend/src/pages/Planner/Keywords.tsx`
|
||||
|
||||
**Update handleAutoCluster function:**
|
||||
```typescript
|
||||
const handleAutoCluster = async () => {
|
||||
try {
|
||||
const selectedIds = selectedKeywords.map(k => k.id);
|
||||
|
||||
// Frontend validation (pre-check before API call)
|
||||
if (selectedIds.length < 5) {
|
||||
toast.error(
|
||||
`Please select at least 5 keywords for auto-clustering. Currently selected: ${selectedIds.length}`,
|
||||
{ duration: 5000 }
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Check total available
|
||||
const availableCount = keywords.filter(k => k.status === 'new').length;
|
||||
if (availableCount < 5) {
|
||||
toast.error(
|
||||
`Not enough keywords available. Need at least 5 keywords, but only ${availableCount} exist.`,
|
||||
{ duration: 5000 }
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Proceed with API call
|
||||
const result = await autoClusterKeywords(selectedIds);
|
||||
|
||||
if (result.task_id) {
|
||||
toast.success(`Auto-cluster started with ${selectedIds.length} keywords`);
|
||||
setTaskId(result.task_id);
|
||||
} else {
|
||||
toast.error('Failed to start auto-cluster');
|
||||
}
|
||||
|
||||
} catch (error: any) {
|
||||
// Backend validation error (in case frontend check was bypassed)
|
||||
const errorMsg = error.response?.data?.error || error.message;
|
||||
toast.error(errorMsg);
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🗂️ FILE STRUCTURE
|
||||
|
||||
### New Files
|
||||
```
|
||||
backend/igny8_core/ai/validators/
|
||||
├── __init__.py
|
||||
└── cluster_validators.py (NEW)
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
```
|
||||
backend/igny8_core/ai/functions/auto_cluster.py
|
||||
backend/igny8_core/business/automation/services/automation_service.py
|
||||
backend/igny8_core/modules/planner/views.py
|
||||
frontend/src/pages/Planner/Keywords.tsx
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 TESTING PLAN
|
||||
|
||||
### Unit Tests
|
||||
|
||||
**File:** `backend/igny8_core/ai/validators/tests/test_cluster_validators.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from django.test import TestCase
|
||||
from igny8_core.ai.validators.cluster_validators import (
|
||||
validate_minimum_keywords,
|
||||
validate_keyword_selection
|
||||
)
|
||||
from igny8_core.modules.planner.models import Keywords
|
||||
from igny8_core.auth.models import Account, Site
|
||||
|
||||
|
||||
class ClusterValidatorsTestCase(TestCase):
|
||||
def setUp(self):
|
||||
self.account = Account.objects.create(name='Test Account')
|
||||
self.site = Site.objects.create(name='Test Site', account=self.account)
|
||||
|
||||
def test_validate_minimum_keywords_success(self):
|
||||
"""Test with sufficient keywords (>= 5)"""
|
||||
# Create 10 keywords
|
||||
keyword_ids = []
|
||||
for i in range(10):
|
||||
kw = Keywords.objects.create(
|
||||
keyword=f'keyword {i}',
|
||||
status='new',
|
||||
account=self.account,
|
||||
site=self.site
|
||||
)
|
||||
keyword_ids.append(kw.id)
|
||||
|
||||
result = validate_minimum_keywords(keyword_ids, self.account)
|
||||
|
||||
assert result['valid'] is True
|
||||
assert result['count'] == 10
|
||||
assert result['required'] == 5
|
||||
|
||||
def test_validate_minimum_keywords_failure(self):
|
||||
"""Test with insufficient keywords (< 5)"""
|
||||
# Create only 3 keywords
|
||||
keyword_ids = []
|
||||
for i in range(3):
|
||||
kw = Keywords.objects.create(
|
||||
keyword=f'keyword {i}',
|
||||
status='new',
|
||||
account=self.account,
|
||||
site=self.site
|
||||
)
|
||||
keyword_ids.append(kw.id)
|
||||
|
||||
result = validate_minimum_keywords(keyword_ids, self.account)
|
||||
|
||||
assert result['valid'] is False
|
||||
assert 'Insufficient keywords' in result['error']
|
||||
assert result['count'] == 3
|
||||
assert result['required'] == 5
|
||||
|
||||
def test_validate_minimum_keywords_edge_case_exactly_5(self):
|
||||
"""Test with exactly 5 keywords (boundary)"""
|
||||
keyword_ids = []
|
||||
for i in range(5):
|
||||
kw = Keywords.objects.create(
|
||||
keyword=f'keyword {i}',
|
||||
status='new',
|
||||
account=self.account,
|
||||
site=self.site
|
||||
)
|
||||
keyword_ids.append(kw.id)
|
||||
|
||||
result = validate_minimum_keywords(keyword_ids, self.account)
|
||||
|
||||
assert result['valid'] is True
|
||||
assert result['count'] == 5
|
||||
|
||||
def test_validate_keyword_selection_insufficient(self):
|
||||
"""Test frontend selection validation"""
|
||||
result = validate_keyword_selection(
|
||||
selected_ids=[1, 2, 3], # Only 3
|
||||
available_count=10,
|
||||
min_required=5
|
||||
)
|
||||
|
||||
assert result['valid'] is False
|
||||
assert result['type'] == 'INSUFFICIENT_SELECTION'
|
||||
assert result['selected'] == 3
|
||||
assert result['required'] == 5
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
```python
|
||||
class AutoClusterIntegrationTestCase(TestCase):
|
||||
def test_auto_cluster_with_insufficient_keywords(self):
|
||||
"""Test auto-cluster endpoint rejects < 5 keywords"""
|
||||
# Create only 3 keywords
|
||||
keyword_ids = self._create_keywords(3)
|
||||
|
||||
response = self.client.post(
|
||||
'/api/planner/keywords/auto_cluster/',
|
||||
data={'ids': keyword_ids},
|
||||
HTTP_AUTHORIZATION=f'Bearer {self.token}'
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert 'Insufficient keywords' in response.json()['error']
|
||||
|
||||
def test_automation_skips_stage_1_with_insufficient_keywords(self):
|
||||
"""Test automation skips Stage 1 if < 5 keywords"""
|
||||
# Create only 2 keywords
|
||||
self._create_keywords(2)
|
||||
|
||||
# Start automation
|
||||
run_id = self.automation_service.start_automation('manual')
|
||||
|
||||
# Verify Stage 1 was skipped
|
||||
run = AutomationRun.objects.get(run_id=run_id)
|
||||
assert run.stage_1_result['skipped'] is True
|
||||
assert 'Insufficient keywords' in run.stage_1_result['skip_reason']
|
||||
assert run.current_stage == 2 # Moved to next stage
|
||||
```
|
||||
|
||||
### Manual Test Cases
|
||||
|
||||
- [ ] **Test 1:** Try auto-cluster with 0 keywords selected
|
||||
- Expected: Error message "No keywords selected"
|
||||
|
||||
- [ ] **Test 2:** Try auto-cluster with 3 keywords selected
|
||||
- Expected: Error message "Please select at least 5 keywords. Currently selected: 3"
|
||||
|
||||
- [ ] **Test 3:** Try auto-cluster with exactly 5 keywords
|
||||
- Expected: Success, clustering starts
|
||||
|
||||
- [ ] **Test 4:** Run automation with 2 keywords in site
|
||||
- Expected: Stage 1 skipped with warning in logs
|
||||
|
||||
- [ ] **Test 5:** Run automation with 10 keywords in site
|
||||
- Expected: Stage 1 runs normally
|
||||
|
||||
---
|
||||
|
||||
## 📊 ERROR MESSAGES
|
||||
|
||||
### Frontend (User-Facing)
|
||||
|
||||
**No Selection:**
|
||||
```
|
||||
❌ No keywords selected
|
||||
Please select keywords to cluster.
|
||||
```
|
||||
|
||||
**Insufficient Selection:**
|
||||
```
|
||||
❌ Please select at least 5 keywords for auto-clustering
|
||||
Currently selected: 3 keywords
|
||||
You need at least 5 keywords to create meaningful clusters.
|
||||
```
|
||||
|
||||
**Insufficient Available:**
|
||||
```
|
||||
❌ Not enough keywords available
|
||||
Need at least 5 keywords, but only 2 exist.
|
||||
Add more keywords before running auto-cluster.
|
||||
```
|
||||
|
||||
### Backend (Logs)
|
||||
|
||||
**Validation Failed:**
|
||||
```
|
||||
[AutoCluster] Validation failed: Insufficient keywords for clustering. Need at least 5 keywords, but only 3 available.
|
||||
```
|
||||
|
||||
**Validation Passed:**
|
||||
```
|
||||
[AutoCluster] Validation passed: 15 keywords available (min: 5)
|
||||
```
|
||||
|
||||
**Automation Stage Skipped:**
|
||||
```
|
||||
[AutomationService] Stage 1 skipped: Insufficient keywords for clustering. Need at least 5 keywords, but only 2 available.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 CONFIGURATION
|
||||
|
||||
### Constants File
|
||||
|
||||
**File:** `backend/igny8_core/ai/constants.py` (or create if doesn't exist)
|
||||
|
||||
```python
|
||||
"""
|
||||
AI Function Configuration Constants
|
||||
"""
|
||||
|
||||
# Cluster Configuration
|
||||
MIN_KEYWORDS_FOR_CLUSTERING = 5 # Minimum keywords needed for meaningful clusters
|
||||
OPTIMAL_KEYWORDS_FOR_CLUSTERING = 20 # Recommended for best results
|
||||
|
||||
# Other AI limits...
|
||||
```
|
||||
|
||||
**Usage in validators:**
|
||||
```python
|
||||
from igny8_core.ai.constants import MIN_KEYWORDS_FOR_CLUSTERING
|
||||
|
||||
def validate_minimum_keywords(keyword_ids, account=None):
|
||||
min_required = MIN_KEYWORDS_FOR_CLUSTERING
|
||||
# ... validation logic
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 SHARED VALIDATION PATTERN
|
||||
|
||||
### Why This Approach Works
|
||||
|
||||
**✅ Single Source of Truth:**
|
||||
- One function: `validate_minimum_keywords()`
|
||||
- Used by both auto-cluster function and automation
|
||||
- Update in one place applies everywhere
|
||||
|
||||
**✅ Consistent Behavior:**
|
||||
- Same error messages
|
||||
- Same validation logic
|
||||
- Same minimum requirements
|
||||
|
||||
**✅ Easy to Maintain:**
|
||||
- Want to change minimum from 5 to 10? Change one constant
|
||||
- Want to add new validation? Add to one function
|
||||
- Want to test? Test one module
|
||||
|
||||
**✅ No Code Duplication:**
|
||||
- DRY principle followed
|
||||
- Reduces bugs from inconsistency
|
||||
- Easier code review
|
||||
|
||||
### Pattern for Future Validators
|
||||
|
||||
```python
|
||||
# backend/igny8_core/ai/validators/content_validators.py
|
||||
|
||||
def validate_minimum_content_length(content_text: str, min_words: int = 100):
|
||||
"""
|
||||
Shared validator for content minimum length
|
||||
Used by: GenerateContentFunction, Automation Stage 4, Content creation
|
||||
"""
|
||||
word_count = len(content_text.split())
|
||||
|
||||
if word_count < min_words:
|
||||
return {
|
||||
'valid': False,
|
||||
'error': f'Content too short. Minimum {min_words} words required, got {word_count}.'
|
||||
}
|
||||
|
||||
return {'valid': True, 'word_count': word_count}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 IMPLEMENTATION STEPS
|
||||
|
||||
### Phase 1: Create Validator (Day 1)
|
||||
- [ ] Create `cluster_validators.py`
|
||||
- [ ] Implement `validate_minimum_keywords()`
|
||||
- [ ] Implement `validate_keyword_selection()`
|
||||
- [ ] Write unit tests
|
||||
|
||||
### Phase 2: Integrate Backend (Day 1)
|
||||
- [ ] Update `AutoClusterFunction.validate()`
|
||||
- [ ] Update `AutomationService.run_stage_1()`
|
||||
- [ ] Update `KeywordsViewSet.auto_cluster()`
|
||||
- [ ] Write integration tests
|
||||
|
||||
### Phase 3: Frontend (Day 2)
|
||||
- [ ] Add frontend validation in Keywords page
|
||||
- [ ] Add user-friendly error messages
|
||||
- [ ] Test error scenarios
|
||||
|
||||
### Phase 4: Testing & Deployment (Day 2)
|
||||
- [ ] Run all tests
|
||||
- [ ] Manual QA testing
|
||||
- [ ] Deploy to production
|
||||
- [ ] Monitor first few auto-cluster runs
|
||||
|
||||
---
|
||||
|
||||
## 🎯 SUCCESS CRITERIA
|
||||
|
||||
✅ Auto-cluster returns error if < 5 keywords selected
|
||||
✅ Automation skips Stage 1 if < 5 keywords available
|
||||
✅ Both use same validation function (no duplication)
|
||||
✅ Clear error messages guide users
|
||||
✅ Frontend validation provides instant feedback
|
||||
✅ Backend validation catches edge cases
|
||||
✅ All tests pass
|
||||
✅ No regression in existing functionality
|
||||
|
||||
---
|
||||
|
||||
## 📈 FUTURE ENHANCEMENTS
|
||||
|
||||
### V2 Features
|
||||
|
||||
1. **Configurable Minimum:**
|
||||
- Allow admin to set minimum via settings
|
||||
- Default: 5, Range: 3-20
|
||||
|
||||
2. **Quality Scoring:**
|
||||
- Show quality indicator based on keyword count
|
||||
- 5-10: "Fair", 11-20: "Good", 21+: "Excellent"
|
||||
|
||||
3. **Smart Recommendations:**
|
||||
- "You have 4 keywords. Add 1 more for best results"
|
||||
- "15 keywords selected. Good for clustering!"
|
||||
|
||||
4. **Batch Size Validation:**
|
||||
- Warn if too many keywords selected (> 100)
|
||||
- Suggest splitting into multiple runs
|
||||
|
||||
---
|
||||
|
||||
## END OF PLAN
|
||||
|
||||
This plan ensures robust, consistent validation for auto-cluster across all entry points (manual and automation) using shared, well-tested validation logic.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,725 +0,0 @@
|
||||
# Automation Progress UX Improvement Plan
|
||||
|
||||
**Date:** December 4, 2025
|
||||
**Status:** Design Phase
|
||||
**Priority:** MEDIUM
|
||||
|
||||
---
|
||||
|
||||
## 🎯 OBJECTIVE
|
||||
|
||||
Improve the automation progress tracking UX to show **real-time processing status** for currently processing items, making it easier for users to understand what's happening during automation runs.
|
||||
|
||||
---
|
||||
|
||||
## 🔍 CURRENT STATE ANALYSIS
|
||||
|
||||
### Current Behavior
|
||||
|
||||
**What Users See Now:**
|
||||
1. A "Current State" card that shows the stage being processed
|
||||
2. Stage number and status (e.g., "Stage 3: Ideas → Tasks")
|
||||
3. **BUT:** No visibility into which specific records are being processed
|
||||
4. **Problem:** User only knows when a full stage completes
|
||||
|
||||
**Example Current Experience:**
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ Current State: Stage 2 │
|
||||
│ Clusters → Ideas (AI) │
|
||||
│ │
|
||||
│ Status: Processing │
|
||||
└─────────────────────────────────────┘
|
||||
|
||||
[User waits... no updates until stage completes]
|
||||
```
|
||||
|
||||
### User Pain Points
|
||||
|
||||
1. ❌ **No Record-Level Progress:** Can't see which keywords/ideas/content are being processed
|
||||
2. ❌ **No Queue Visibility:** Don't know what's coming up next
|
||||
3. ❌ **No Item Count Progress:** "Processing 15 of 50 keywords..." is missing
|
||||
4. ❌ **Card Position:** Current state card is at bottom, requires scrolling
|
||||
5. ❌ **No Percentage Progress:** Just a spinner, no quantitative feedback
|
||||
|
||||
---
|
||||
|
||||
## ✅ PROPOSED SOLUTION
|
||||
|
||||
### New Design Concept
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ 🔄 AUTOMATION IN PROGRESS │
|
||||
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67% │
|
||||
│ │
|
||||
│ Stage 2: Clusters → Ideas (AI) │
|
||||
│Column 1 │
|
||||
│ Currently Processing: │
|
||||
│ • "Best SEO tools for small business" (Cluster #42) │
|
||||
│ Column 2 │
|
||||
│ Up Next: │
|
||||
│ • "Content marketing automation platforms" │
|
||||
│ • "AI-powered content creation tools" │
|
||||
│ Sinngle row centered │
|
||||
│ Progress: 34/50 clusters processed │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
[STAGES SECTION BELOW - All 7 stages in grid view]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📐 DETAILED DESIGN SPECIFICATIONS
|
||||
|
||||
### 1. Card Repositioning
|
||||
|
||||
**Move from:** Bottom of page (after stages)
|
||||
**Move to:** Top of page (above stages section)
|
||||
**Layout:** Max width 1200px horizontal card
|
||||
**Visibility:** Only shown when `currentRun?.status === 'running'`
|
||||
|
||||
### 2. Card Structure
|
||||
|
||||
#### Header Section
|
||||
- **Left:** Large stage number icon (animated pulse)
|
||||
- **Center:** Stage name + type badge (AI/Local/Manual)
|
||||
- **Right:** Percentage complete (calculated from processed/total)
|
||||
|
||||
#### Progress Bar
|
||||
- **Type:** Animated linear progress bar
|
||||
- **Colors:**
|
||||
- Blue for active stage
|
||||
- Green for completed
|
||||
- Gray for pending
|
||||
- **Updates:** Refresh every 3-5 seconds via polling
|
||||
|
||||
#### Currently Processing Section
|
||||
- **For Keywords Stage:**
|
||||
```
|
||||
Currently Processing:
|
||||
• "keyword 1"
|
||||
• "keyword 2"
|
||||
• "keyword 3"
|
||||
|
||||
+ 47 more in queue
|
||||
```
|
||||
|
||||
- **For Ideas Stage:**
|
||||
```
|
||||
Currently Processing:
|
||||
• "10 Ways to Improve SEO Rankings"
|
||||
|
||||
Up Next:
|
||||
• "Content Marketing Best Practices 2025"
|
||||
• "AI Tools for Content Writers"
|
||||
```
|
||||
|
||||
- **For Content Stage:**
|
||||
```
|
||||
Currently Processing:
|
||||
• "How to Use ChatGPT for Content Creation" (2,500 words)
|
||||
|
||||
Up Next:
|
||||
• "Best AI Image Generators in 2025"
|
||||
```
|
||||
|
||||
#### Record Counter
|
||||
```
|
||||
Progress: [current]/[total] [items] processed
|
||||
Example: Progress: 15/50 keywords processed
|
||||
```
|
||||
|
||||
### 3. Refresh Strategy
|
||||
|
||||
**Polling Approach:**
|
||||
```typescript
|
||||
// Poll every 3 seconds while automation is running
|
||||
useEffect(() => {
|
||||
if (currentRun?.status === 'running') {
|
||||
const interval = setInterval(() => {
|
||||
// Refresh ONLY the current processing data
|
||||
fetchCurrentProcessingState();
|
||||
}, 3000);
|
||||
|
||||
return () => clearInterval(interval);
|
||||
}
|
||||
}, [currentRun]);
|
||||
```
|
||||
|
||||
**Partial Refresh:**
|
||||
- Only refresh the "Currently Processing" component
|
||||
- Don't reload entire page
|
||||
- Don't re-fetch stage cards
|
||||
- Smooth transition (no flickering)
|
||||
|
||||
---
|
||||
|
||||
## 🗄️ BACKEND CHANGES REQUIRED
|
||||
|
||||
### New API Endpoint
|
||||
|
||||
**URL:** `GET /api/automation/current_processing/`
|
||||
**Params:** `?site_id={id}&run_id={run_id}`
|
||||
|
||||
**Response Format:**
|
||||
```json
|
||||
{
|
||||
"run_id": "abc123",
|
||||
"current_stage": 2,
|
||||
"stage_name": "Clusters → Ideas",
|
||||
"stage_type": "AI",
|
||||
"total_items": 50,
|
||||
"processed_items": 34,
|
||||
"percentage": 68,
|
||||
"currently_processing": [
|
||||
{
|
||||
"id": 42,
|
||||
"title": "Best SEO tools for small business",
|
||||
"type": "cluster"
|
||||
}
|
||||
],
|
||||
"up_next": [
|
||||
{
|
||||
"id": 43,
|
||||
"title": "Content marketing automation platforms",
|
||||
"type": "cluster"
|
||||
},
|
||||
{
|
||||
"id": 44,
|
||||
"title": "AI-powered content creation tools",
|
||||
"type": "cluster"
|
||||
}
|
||||
],
|
||||
"remaining_count": 16
|
||||
}
|
||||
```
|
||||
|
||||
### Implementation in AutomationService
|
||||
|
||||
**File:** `backend/igny8_core/business/automation/services/automation_service.py`
|
||||
|
||||
**Add method:**
|
||||
```python
|
||||
def get_current_processing_state(self) -> dict:
|
||||
"""
|
||||
Get real-time processing state for current automation run
|
||||
"""
|
||||
if not self.run or self.run.status != 'running':
|
||||
return None
|
||||
|
||||
stage = self.run.current_stage
|
||||
|
||||
# Get stage-specific data
|
||||
if stage == 1: # Keywords → Clusters
|
||||
queue = Keywords.objects.filter(
|
||||
site=self.site, status='new'
|
||||
).order_by('id')
|
||||
|
||||
return {
|
||||
'stage_number': 1,
|
||||
'stage_name': 'Keywords → Clusters',
|
||||
'stage_type': 'AI',
|
||||
'total_items': queue.count() + self._get_processed_count(stage),
|
||||
'processed_items': self._get_processed_count(stage),
|
||||
'currently_processing': self._get_current_items(queue, 3),
|
||||
'up_next': self._get_next_items(queue, 2, skip=3),
|
||||
}
|
||||
|
||||
elif stage == 2: # Clusters → Ideas
|
||||
queue = Clusters.objects.filter(
|
||||
site=self.site, status='new', disabled=False
|
||||
).order_by('id')
|
||||
|
||||
return {
|
||||
'stage_number': 2,
|
||||
'stage_name': 'Clusters → Ideas',
|
||||
'stage_type': 'AI',
|
||||
'total_items': queue.count() + self._get_processed_count(stage),
|
||||
'processed_items': self._get_processed_count(stage),
|
||||
'currently_processing': self._get_current_items(queue, 1),
|
||||
'up_next': self._get_next_items(queue, 2, skip=1),
|
||||
}
|
||||
|
||||
# ... similar for stages 3-6
|
||||
|
||||
def _get_processed_count(self, stage: int) -> int:
|
||||
"""Get count of items processed in current stage"""
|
||||
result_key = f'stage_{stage}_result'
|
||||
result = getattr(self.run, result_key, {})
|
||||
|
||||
# Extract appropriate count from result
|
||||
if stage == 1:
|
||||
return result.get('keywords_processed', 0)
|
||||
elif stage == 2:
|
||||
return result.get('clusters_processed', 0)
|
||||
# ... etc
|
||||
|
||||
def _get_current_items(self, queryset, count: int) -> list:
|
||||
"""Get currently processing items"""
|
||||
items = queryset[:count]
|
||||
return [
|
||||
{
|
||||
'id': item.id,
|
||||
'title': getattr(item, 'keyword', None) or
|
||||
getattr(item, 'cluster_name', None) or
|
||||
getattr(item, 'idea_title', None) or
|
||||
getattr(item, 'title', None),
|
||||
'type': queryset.model.__name__.lower()
|
||||
}
|
||||
for item in items
|
||||
]
|
||||
```
|
||||
|
||||
### Add View in AutomationViewSet
|
||||
|
||||
**File:** `backend/igny8_core/business/automation/views.py`
|
||||
|
||||
```python
|
||||
@action(detail=False, methods=['get'], url_path='current_processing')
|
||||
def current_processing(self, request):
|
||||
"""Get current processing state for active automation run"""
|
||||
site_id = request.GET.get('site_id')
|
||||
run_id = request.GET.get('run_id')
|
||||
|
||||
if not site_id or not run_id:
|
||||
return error_response(
|
||||
error='site_id and run_id required',
|
||||
status_code=400,
|
||||
request=request
|
||||
)
|
||||
|
||||
try:
|
||||
run = AutomationRun.objects.get(run_id=run_id, site_id=site_id)
|
||||
|
||||
if run.status != 'running':
|
||||
return success_response(data=None, request=request)
|
||||
|
||||
service = AutomationService.from_run_id(run_id)
|
||||
state = service.get_current_processing_state()
|
||||
|
||||
return success_response(data=state, request=request)
|
||||
|
||||
except AutomationRun.DoesNotExist:
|
||||
return error_response(
|
||||
error='Run not found',
|
||||
status_code=404,
|
||||
request=request
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎨 FRONTEND CHANGES REQUIRED
|
||||
|
||||
### 1. New Component: CurrentProcessingCard
|
||||
|
||||
**File:** `frontend/src/components/Automation/CurrentProcessingCard.tsx`
|
||||
|
||||
```typescript
|
||||
interface CurrentProcessingCardProps {
|
||||
runId: string;
|
||||
siteId: number;
|
||||
currentStage: number;
|
||||
onComplete?: () => void;
|
||||
}
|
||||
|
||||
const CurrentProcessingCard: React.FC<CurrentProcessingCardProps> = ({
|
||||
runId,
|
||||
siteId,
|
||||
currentStage,
|
||||
onComplete
|
||||
}) => {
|
||||
const [processingState, setProcessingState] = useState<ProcessingState | null>(null);
|
||||
|
||||
// Poll every 3 seconds
|
||||
useEffect(() => {
|
||||
const fetchState = async () => {
|
||||
const state = await automationService.getCurrentProcessing(siteId, runId);
|
||||
setProcessingState(state);
|
||||
|
||||
// If stage completed, trigger refresh
|
||||
if (state && state.processed_items === state.total_items) {
|
||||
onComplete?.();
|
||||
}
|
||||
};
|
||||
|
||||
fetchState();
|
||||
const interval = setInterval(fetchState, 3000);
|
||||
|
||||
return () => clearInterval(interval);
|
||||
}, [siteId, runId]);
|
||||
|
||||
if (!processingState) return null;
|
||||
|
||||
const percentage = Math.round(
|
||||
(processingState.processed_items / processingState.total_items) * 100
|
||||
);
|
||||
|
||||
return (
|
||||
<div className="bg-blue-50 dark:bg-blue-900/20 border-2 border-blue-500 rounded-lg p-6 mb-6">
|
||||
{/* Header */}
|
||||
<div className="flex items-center justify-between mb-4">
|
||||
<div className="flex items-center gap-3">
|
||||
<div className="animate-pulse">
|
||||
<BoltIcon className="w-8 h-8 text-blue-600" />
|
||||
</div>
|
||||
<div>
|
||||
<h2 className="text-2xl font-bold text-gray-900 dark:text-white">
|
||||
Automation In Progress
|
||||
</h2>
|
||||
<p className="text-sm text-gray-600 dark:text-gray-400">
|
||||
Stage {currentStage}: {processingState.stage_name}
|
||||
<span className="ml-2 px-2 py-0.5 bg-blue-100 dark:bg-blue-900 text-blue-700 dark:text-blue-300 rounded text-xs">
|
||||
{processingState.stage_type}
|
||||
</span>
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
<div className="text-right">
|
||||
<div className="text-4xl font-bold text-blue-600">{percentage}%</div>
|
||||
<div className="text-sm text-gray-600 dark:text-gray-400">
|
||||
{processingState.processed_items}/{processingState.total_items} processed
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Progress Bar */}
|
||||
<div className="mb-6">
|
||||
<div className="w-full bg-gray-200 dark:bg-gray-700 rounded-full h-3">
|
||||
<div
|
||||
className="bg-blue-600 h-3 rounded-full transition-all duration-500"
|
||||
style={{ width: `${percentage}%` }}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Currently Processing */}
|
||||
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
|
||||
<div>
|
||||
<h3 className="text-sm font-semibold text-gray-700 dark:text-gray-300 mb-2">
|
||||
Currently Processing:
|
||||
</h3>
|
||||
<div className="space-y-1">
|
||||
{processingState.currently_processing.map((item, idx) => (
|
||||
<div key={idx} className="flex items-start gap-2 text-sm">
|
||||
<span className="text-blue-600 mt-1">•</span>
|
||||
<span className="text-gray-800 dark:text-gray-200 font-medium">
|
||||
{item.title}
|
||||
</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<h3 className="text-sm font-semibold text-gray-700 dark:text-gray-300 mb-2">
|
||||
Up Next:
|
||||
</h3>
|
||||
<div className="space-y-1">
|
||||
{processingState.up_next.map((item, idx) => (
|
||||
<div key={idx} className="flex items-start gap-2 text-sm">
|
||||
<span className="text-gray-400 mt-1">•</span>
|
||||
<span className="text-gray-600 dark:text-gray-400">
|
||||
{item.title}
|
||||
</span>
|
||||
</div>
|
||||
))}
|
||||
{processingState.remaining_count > processingState.up_next.length && (
|
||||
<div className="text-xs text-gray-500 mt-2">
|
||||
+ {processingState.remaining_count - processingState.up_next.length} more in queue
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
```
|
||||
|
||||
### 2. Update AutomationPage.tsx
|
||||
|
||||
**File:** `frontend/src/pages/Automation/AutomationPage.tsx`
|
||||
|
||||
```typescript
|
||||
// Add new import
|
||||
import CurrentProcessingCard from '../../components/Automation/CurrentProcessingCard';
|
||||
|
||||
// In the component
|
||||
return (
|
||||
<div className="p-6">
|
||||
<PageMeta title="Automation" description="AI automation pipeline" />
|
||||
|
||||
{/* Current Processing Card - MOVE TO TOP */}
|
||||
{currentRun?.status === 'running' && (
|
||||
<CurrentProcessingCard
|
||||
runId={currentRun.run_id}
|
||||
siteId={selectedSite.id}
|
||||
currentStage={currentRun.current_stage}
|
||||
onComplete={() => {
|
||||
// Refresh full page metrics when stage completes
|
||||
loadAutomationData();
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
|
||||
{/* Metrics Cards */}
|
||||
<div className="grid grid-cols-1 md:grid-cols-3 gap-6 mb-6">
|
||||
{/* ... existing metrics ... */}
|
||||
</div>
|
||||
|
||||
{/* Stages Section */}
|
||||
<ComponentCard>
|
||||
<h2 className="text-xl font-semibold mb-4">Pipeline Stages</h2>
|
||||
{/* ... existing stages ... */}
|
||||
</ComponentCard>
|
||||
|
||||
{/* Rest of the page ... */}
|
||||
</div>
|
||||
);
|
||||
```
|
||||
|
||||
### 3. Add Service Method
|
||||
|
||||
**File:** `frontend/src/services/automationService.ts`
|
||||
|
||||
```typescript
|
||||
export interface ProcessingState {
|
||||
run_id: string;
|
||||
current_stage: number;
|
||||
stage_name: string;
|
||||
stage_type: 'AI' | 'Local' | 'Manual';
|
||||
total_items: number;
|
||||
processed_items: number;
|
||||
percentage: number;
|
||||
currently_processing: Array<{
|
||||
id: number;
|
||||
title: string;
|
||||
type: string;
|
||||
}>;
|
||||
up_next: Array<{
|
||||
id: number;
|
||||
title: string;
|
||||
type: string;
|
||||
}>;
|
||||
remaining_count: number;
|
||||
}
|
||||
|
||||
// Add to automationService
|
||||
getCurrentProcessing: async (
|
||||
siteId: number,
|
||||
runId: string
|
||||
): Promise<ProcessingState | null> => {
|
||||
return fetchAPI(
|
||||
buildUrl('/current_processing/', { site_id: siteId, run_id: runId })
|
||||
);
|
||||
},
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 TESTING PLAN
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- [ ] Test `get_current_processing_state()` for each stage
|
||||
- [ ] Test `_get_processed_count()` calculation
|
||||
- [ ] Test `_get_current_items()` formatting
|
||||
- [ ] Test API endpoint with various run states
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- [ ] Test polling updates every 3 seconds
|
||||
- [ ] Test stage completion triggers full refresh
|
||||
- [ ] Test card disappears when automation completes
|
||||
- [ ] Test with 0 items (edge case)
|
||||
- [ ] Test with 1000+ items (performance)
|
||||
|
||||
### Visual/UX Tests
|
||||
|
||||
- [ ] Card positioned at top of page
|
||||
- [ ] Progress bar animates smoothly
|
||||
- [ ] Record names display correctly
|
||||
- [ ] Responsive design (mobile/tablet/desktop)
|
||||
- [ ] Dark mode support
|
||||
- [ ] Loading states
|
||||
- [ ] Error states
|
||||
|
||||
---
|
||||
|
||||
## 📊 STAGE-SPECIFIC DISPLAY FORMATS
|
||||
|
||||
### Stage 1: Keywords → Clusters
|
||||
|
||||
```
|
||||
Currently Processing:
|
||||
• "best seo tools"
|
||||
• "content marketing platforms"
|
||||
• "ai writing assistants"
|
||||
|
||||
+ 47 more keywords in queue
|
||||
|
||||
Progress: 3/50 keywords processed
|
||||
```
|
||||
|
||||
### Stage 2: Clusters → Ideas
|
||||
|
||||
```
|
||||
Currently Processing:
|
||||
• "SEO Tools and Software" (Cluster #12)
|
||||
|
||||
Up Next:
|
||||
• "Content Marketing Strategies"
|
||||
• "AI Content Generation"
|
||||
|
||||
Progress: 12/25 clusters processed
|
||||
```
|
||||
|
||||
### Stage 3: Ideas → Tasks
|
||||
|
||||
```
|
||||
Currently Processing:
|
||||
• "10 Best SEO Tools for 2025"
|
||||
|
||||
Up Next:
|
||||
• "How to Create Content with AI"
|
||||
• "Content Marketing ROI Calculator"
|
||||
|
||||
Progress: 8/30 ideas processed
|
||||
```
|
||||
|
||||
### Stage 4: Tasks → Content
|
||||
|
||||
```
|
||||
Currently Processing:
|
||||
• "Ultimate Guide to SEO in 2025" (2,500 words)
|
||||
|
||||
Up Next:
|
||||
• "AI Content Creation Best Practices"
|
||||
|
||||
Progress: 5/15 tasks processed
|
||||
```
|
||||
|
||||
### Stage 5: Content → Image Prompts
|
||||
|
||||
```
|
||||
Currently Processing:
|
||||
• "How to Use ChatGPT for Content" (Extracting 3 image prompts)
|
||||
|
||||
Up Next:
|
||||
• "Best AI Image Generators 2025"
|
||||
|
||||
Progress: 10/15 content pieces processed
|
||||
```
|
||||
|
||||
### Stage 6: Image Prompts → Images
|
||||
|
||||
```
|
||||
Currently Processing:
|
||||
• Featured image for "SEO Guide 2025"
|
||||
|
||||
Up Next:
|
||||
• In-article image #1 for "SEO Guide 2025"
|
||||
• In-article image #2 for "SEO Guide 2025"
|
||||
|
||||
Progress: 15/45 images generated
|
||||
```
|
||||
|
||||
### Stage 7: Manual Review Gate
|
||||
|
||||
```
|
||||
Automation Complete! ✅
|
||||
|
||||
Ready for Review:
|
||||
• "Ultimate Guide to SEO in 2025"
|
||||
• "AI Content Creation Best Practices"
|
||||
• "Best Image Generators 2025"
|
||||
|
||||
+ 12 more content pieces
|
||||
|
||||
Total: 15 content pieces ready for review
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 SUCCESS METRICS
|
||||
|
||||
### User Experience
|
||||
|
||||
✅ Users can see **exactly what's being processed** at any moment
|
||||
✅ Users know **what's coming up next** in the queue
|
||||
✅ Users can estimate **remaining time** based on progress
|
||||
✅ Users get **quantitative feedback** (percentage, counts)
|
||||
✅ Users see **smooth, non-disruptive updates** (no page flicker)
|
||||
|
||||
### Technical
|
||||
|
||||
✅ Polling interval: 3 seconds (balance between freshness and load)
|
||||
✅ API response time: < 200ms
|
||||
✅ Component re-render: Only the processing card, not entire page
|
||||
✅ Memory usage: No memory leaks from polling
|
||||
✅ Error handling: Graceful degradation if API fails
|
||||
|
||||
---
|
||||
|
||||
## 🚀 IMPLEMENTATION PHASES
|
||||
|
||||
### Phase 1: Backend (1-2 days)
|
||||
- [ ] Implement `get_current_processing_state()` method
|
||||
- [ ] Add `/current_processing/` API endpoint
|
||||
- [ ] Test with all 7 stages
|
||||
- [ ] Add unit tests
|
||||
|
||||
### Phase 2: Frontend (2-3 days)
|
||||
- [ ] Create `CurrentProcessingCard` component
|
||||
- [ ] Add polling logic with cleanup
|
||||
- [ ] Style with Tailwind (match existing design system)
|
||||
- [ ] Add dark mode support
|
||||
- [ ] Integrate into `AutomationPage`
|
||||
|
||||
### Phase 3: Testing & Refinement (1-2 days)
|
||||
- [ ] Integration testing
|
||||
- [ ] Performance testing
|
||||
- [ ] UX testing
|
||||
- [ ] Bug fixes
|
||||
|
||||
### Phase 4: Deployment
|
||||
- [ ] Deploy backend changes
|
||||
- [ ] Deploy frontend changes
|
||||
- [ ] Monitor first automation runs
|
||||
- [ ] Collect user feedback
|
||||
|
||||
---
|
||||
|
||||
## 🔄 FUTURE ENHANCEMENTS
|
||||
|
||||
### V2 Features (Post-MVP)
|
||||
|
||||
1. **Estimated Time Remaining:**
|
||||
```
|
||||
Progress: 15/50 keywords processed
|
||||
Estimated time remaining: ~8 minutes
|
||||
```
|
||||
|
||||
2. **Stage-Level Progress Bar:**
|
||||
- Each stage shows its own mini progress bar
|
||||
- Visual indicator of which stages are complete
|
||||
|
||||
3. **Click to View Details:**
|
||||
- Click on a record name to see modal with details
|
||||
- Preview generated content/images
|
||||
|
||||
4. **Pause/Resume from Card:**
|
||||
- Add pause button directly in the card
|
||||
- Quick action without scrolling
|
||||
|
||||
5. **Export Processing Log:**
|
||||
- Download real-time processing log
|
||||
- CSV of all processed items with timestamps
|
||||
|
||||
---
|
||||
|
||||
## END OF PLAN
|
||||
|
||||
This plan provides a comprehensive UX improvement for automation progress tracking, making the process transparent and user-friendly while maintaining system performance.
|
||||
@@ -1,403 +0,0 @@
|
||||
# Automation Stage 6 - Image Generation Fix Plan
|
||||
|
||||
**Date:** December 4, 2025
|
||||
**Status:** Analysis Complete - Implementation Required
|
||||
**Priority:** HIGH
|
||||
|
||||
---
|
||||
|
||||
## 🔍 PROBLEM IDENTIFICATION
|
||||
|
||||
### Current Issue
|
||||
|
||||
Stage 6 of the automation pipeline (Image Prompts → Generated Images) is **NOT running correctly**. The issue stems from using the wrong AI function for image generation.
|
||||
|
||||
### Root Cause Analysis
|
||||
|
||||
**Current Implementation (INCORRECT):**
|
||||
```python
|
||||
# File: backend/igny8_core/business/automation/services/automation_service.py
|
||||
# Line ~935
|
||||
|
||||
engine = AIEngine(account=self.account)
|
||||
result = engine.execute(
|
||||
fn=GenerateImagesFunction(),
|
||||
payload={'image_ids': [image.id]} # ❌ WRONG
|
||||
)
|
||||
```
|
||||
|
||||
**Why It Fails:**
|
||||
|
||||
1. `GenerateImagesFunction()` expects:
|
||||
- Input: `{'ids': [task_ids]}` (Task IDs, NOT Image IDs)
|
||||
- Purpose: Extract prompts from Tasks and generate images for tasks
|
||||
- Use case: When you have Tasks with content but no images
|
||||
|
||||
2. Automation Stage 6 has:
|
||||
- Input: Images records with `status='pending'` (already have prompts)
|
||||
- Purpose: Generate actual image URLs from existing prompts
|
||||
- Context: Images were created in Stage 5 by `GenerateImagePromptsFunction`
|
||||
|
||||
### How Other Stages Work Correctly
|
||||
|
||||
**Stage 1:** Keywords → Clusters
|
||||
```python
|
||||
engine.execute(
|
||||
fn=AutoClusterFunction(),
|
||||
payload={'keyword_ids': keyword_ids} # ✅ Correct
|
||||
)
|
||||
```
|
||||
|
||||
**Stage 2:** Clusters → Ideas
|
||||
```python
|
||||
engine.execute(
|
||||
fn=GenerateIdeasFunction(),
|
||||
payload={'cluster_ids': cluster_ids} # ✅ Correct
|
||||
)
|
||||
```
|
||||
|
||||
**Stage 4:** Tasks → Content
|
||||
```python
|
||||
engine.execute(
|
||||
fn=GenerateContentFunction(),
|
||||
payload={'ids': task_ids} # ✅ Correct
|
||||
)
|
||||
```
|
||||
|
||||
**Stage 5:** Content → Image Prompts
|
||||
```python
|
||||
engine.execute(
|
||||
fn=GenerateImagePromptsFunction(),
|
||||
payload={'ids': content_ids} # ✅ Correct
|
||||
)
|
||||
```
|
||||
|
||||
**Stage 6:** Image Prompts → Images (BROKEN)
|
||||
```python
|
||||
# Currently uses GenerateImagesFunction (WRONG)
|
||||
# Should use process_image_generation_queue (CORRECT)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ CORRECT SOLUTION
|
||||
|
||||
### The Right Approach
|
||||
|
||||
**Use `process_image_generation_queue` Celery task** - This is the same approach used by:
|
||||
1. Writer/Images page (`/writer/images/generate_images/` endpoint)
|
||||
2. Manual image generation from prompts
|
||||
|
||||
**Evidence from Working Code:**
|
||||
|
||||
```python
|
||||
# File: backend/igny8_core/modules/writer/views.py
|
||||
# ImagesViewSet.generate_images()
|
||||
|
||||
from igny8_core.ai.tasks import process_image_generation_queue
|
||||
|
||||
task = process_image_generation_queue.delay(
|
||||
image_ids=image_ids, # ✅ Accepts image_ids
|
||||
account_id=account_id,
|
||||
content_id=content_id
|
||||
)
|
||||
```
|
||||
|
||||
**What `process_image_generation_queue` Does:**
|
||||
|
||||
1. ✅ Accepts `image_ids` (list of Image record IDs)
|
||||
2. ✅ Each Image record already has a `prompt` field (populated by Stage 5)
|
||||
3. ✅ Generates images sequentially with progress tracking
|
||||
4. ✅ Updates Images records: `status='pending'` → `status='generated'`
|
||||
5. ✅ Downloads and saves images locally
|
||||
6. ✅ Automatically handles credits deduction
|
||||
7. ✅ Supports multiple providers (OpenAI, Runware)
|
||||
8. ✅ Handles errors gracefully (continues on failure)
|
||||
|
||||
---
|
||||
|
||||
## 📋 IMPLEMENTATION PLAN
|
||||
|
||||
### Changes Required
|
||||
|
||||
**File:** `backend/igny8_core/business/automation/services/automation_service.py`
|
||||
|
||||
**Location:** `run_stage_6()` method (lines 874-1022)
|
||||
|
||||
### Step 1: Import the Correct Task
|
||||
|
||||
**Current:**
|
||||
```python
|
||||
from igny8_core.ai.functions.generate_images import GenerateImagesFunction
|
||||
```
|
||||
|
||||
**Add:**
|
||||
```python
|
||||
from igny8_core.ai.tasks import process_image_generation_queue
|
||||
```
|
||||
|
||||
### Step 2: Modify Stage 6 Logic
|
||||
|
||||
**Replace this block (lines ~920-945):**
|
||||
|
||||
```python
|
||||
# INCORRECT - Delete this
|
||||
for idx, image in enumerate(image_list, 1):
|
||||
try:
|
||||
content_title = image.content.title if image.content else 'Unknown'
|
||||
self.logger.log_stage_progress(
|
||||
self.run.run_id, self.account.id, self.site.id,
|
||||
stage_number, f"Generating image {idx}/{total_images}: {image.image_type} for '{content_title}'"
|
||||
)
|
||||
|
||||
# Call AI function via AIEngine
|
||||
engine = AIEngine(account=self.account)
|
||||
result = engine.execute(
|
||||
fn=GenerateImagesFunction(),
|
||||
payload={'image_ids': [image.id]} # ❌ WRONG
|
||||
)
|
||||
|
||||
# Monitor task
|
||||
task_id = result.get('task_id')
|
||||
if task_id:
|
||||
self._wait_for_task(task_id, stage_number, f"Image for '{content_title}'", continue_on_error=True)
|
||||
|
||||
images_processed += 1
|
||||
```
|
||||
|
||||
**With this:**
|
||||
|
||||
```python
|
||||
# CORRECT - Use process_image_generation_queue
|
||||
for idx, image in enumerate(image_list, 1):
|
||||
try:
|
||||
content_title = image.content.title if image.content else 'Unknown'
|
||||
self.logger.log_stage_progress(
|
||||
self.run.run_id, self.account.id, self.site.id,
|
||||
stage_number, f"Generating image {idx}/{total_images}: {image.image_type} for '{content_title}'"
|
||||
)
|
||||
|
||||
# Call process_image_generation_queue directly (same as Writer/Images page)
|
||||
from igny8_core.ai.tasks import process_image_generation_queue
|
||||
|
||||
# Queue the task
|
||||
if hasattr(process_image_generation_queue, 'delay'):
|
||||
task = process_image_generation_queue.delay(
|
||||
image_ids=[image.id],
|
||||
account_id=self.account.id,
|
||||
content_id=image.content.id if image.content else None
|
||||
)
|
||||
task_id = str(task.id)
|
||||
else:
|
||||
# Fallback for testing (synchronous)
|
||||
result = process_image_generation_queue(
|
||||
image_ids=[image.id],
|
||||
account_id=self.account.id,
|
||||
content_id=image.content.id if image.content else None
|
||||
)
|
||||
task_id = None
|
||||
|
||||
# Monitor task (if async)
|
||||
if task_id:
|
||||
self._wait_for_task(task_id, stage_number, f"Image for '{content_title}'", continue_on_error=True)
|
||||
|
||||
images_processed += 1
|
||||
```
|
||||
|
||||
### Step 3: Update Logging
|
||||
|
||||
The logging structure remains the same, just update the log messages to reflect the correct process:
|
||||
|
||||
```python
|
||||
self.logger.log_stage_progress(
|
||||
self.run.run_id, self.account.id, self.site.id,
|
||||
stage_number, f"Image generation task queued for '{content_title}' ({images_processed}/{total_images})"
|
||||
)
|
||||
```
|
||||
|
||||
### Step 4: No Changes Needed For
|
||||
|
||||
✅ Stage 5 (Image Prompt Extraction) - Already correct
|
||||
✅ Images table structure - Already has all required fields
|
||||
✅ Progress tracking - Already implemented in `process_image_generation_queue`
|
||||
✅ Credits deduction - Automatic in `process_image_generation_queue`
|
||||
✅ Error handling - Built into the task with `continue_on_error=True`
|
||||
|
||||
---
|
||||
|
||||
## 🔄 HOW IT WORKS (CORRECTED FLOW)
|
||||
|
||||
### Stage 5: Content → Image Prompts
|
||||
|
||||
```
|
||||
Input: Content (status='draft', no images)
|
||||
AI: GenerateImagePromptsFunction
|
||||
Output: Images (status='pending', prompt='...')
|
||||
```
|
||||
|
||||
### Stage 6: Image Prompts → Generated Images (FIXED)
|
||||
|
||||
```
|
||||
Input: Images (status='pending', has prompt)
|
||||
Task: process_image_generation_queue (Celery task)
|
||||
AI: Calls OpenAI/Runware API with prompt
|
||||
Output: Images (status='generated', image_url='https://...', image_path='/path/to/file')
|
||||
```
|
||||
|
||||
### What Happens in process_image_generation_queue
|
||||
|
||||
1. **Load Image Record:**
|
||||
- Get Image by ID
|
||||
- Read existing `prompt` field (created in Stage 5)
|
||||
- Get Content for template formatting
|
||||
|
||||
2. **Format Prompt:**
|
||||
- Use image_prompt_template from PromptRegistry
|
||||
- Format: `"Create a {image_type} image for '{post_title}'. Prompt: {image_prompt}"`
|
||||
- Handle model-specific limits (DALL-E 3: 4000 chars, DALL-E 2: 1000 chars)
|
||||
|
||||
3. **Generate Image:**
|
||||
- Call `AICore.generate_image()`
|
||||
- Uses configured provider (OpenAI/Runware)
|
||||
- Uses configured model (dall-e-3, runware:97@1, etc.)
|
||||
- Respects image size settings
|
||||
|
||||
4. **Download & Save:**
|
||||
- Download image from URL
|
||||
- Save to `/data/app/igny8/frontend/public/images/ai-images/`
|
||||
- Update Image record with both `image_url` and `image_path`
|
||||
|
||||
5. **Update Status:**
|
||||
- `status='pending'` → `status='generated'`
|
||||
- Triggers automatic Content status update (if all images generated)
|
||||
|
||||
6. **Deduct Credits:**
|
||||
- Automatic via `AICore` credit system
|
||||
- Records in `AIUsageLog`
|
||||
|
||||
---
|
||||
|
||||
## 🧪 TESTING CHECKLIST
|
||||
|
||||
### Pre-Deployment Tests
|
||||
|
||||
- [ ] **Unit Test:** Verify `process_image_generation_queue` works with single image
|
||||
- [ ] **Integration Test:** Run Stage 6 with 3-5 pending images
|
||||
- [ ] **Error Handling:** Test with invalid image ID
|
||||
- [ ] **Credits:** Verify credits are deducted correctly
|
||||
- [ ] **Multi-Provider:** Test with both OpenAI and Runware
|
||||
|
||||
### Post-Deployment Validation
|
||||
|
||||
- [ ] **Full Pipeline:** Run Automation from Stage 1 → Stage 7
|
||||
- [ ] **Verify Stage 5 Output:** Images created with `status='pending'` and prompts
|
||||
- [ ] **Verify Stage 6 Output:** Images updated to `status='generated'` with URLs
|
||||
- [ ] **Check Downloads:** Images saved to `/data/app/igny8/frontend/public/images/ai-images/`
|
||||
- [ ] **Monitor Logs:** Review automation logs for Stage 6 completion
|
||||
- [ ] **Credits Report:** Confirm Stage 6 credits recorded in automation results
|
||||
|
||||
### Success Criteria
|
||||
|
||||
✅ Stage 6 completes without errors
|
||||
✅ All pending images get generated
|
||||
✅ Images are downloaded and accessible
|
||||
✅ Content status automatically updates when all images generated
|
||||
✅ Credits are properly deducted and logged
|
||||
✅ Automation proceeds to Stage 7 (Manual Review Gate)
|
||||
|
||||
---
|
||||
|
||||
## 📊 COMPARISON: BEFORE vs AFTER
|
||||
|
||||
### BEFORE (Broken)
|
||||
|
||||
```python
|
||||
# ❌ WRONG APPROACH
|
||||
GenerateImagesFunction()
|
||||
- Expects: task_ids
|
||||
- Purpose: Extract prompts from Tasks
|
||||
- Problem: Doesn't work with Images that already have prompts
|
||||
```
|
||||
|
||||
**Result:** Stage 6 fails, images never generated
|
||||
|
||||
### AFTER (Fixed)
|
||||
|
||||
```python
|
||||
# ✅ CORRECT APPROACH
|
||||
process_image_generation_queue()
|
||||
- Accepts: image_ids
|
||||
- Purpose: Generate images from existing prompts
|
||||
- Works with: Images (status='pending' with prompts)
|
||||
```
|
||||
|
||||
**Result:** Stage 6 succeeds, images generated sequentially with progress tracking
|
||||
|
||||
---
|
||||
|
||||
## 🔒 SAFETY & ROLLBACK
|
||||
|
||||
### Backup Plan
|
||||
|
||||
If the fix causes issues:
|
||||
|
||||
1. **Rollback Code:**
|
||||
- Git revert the automation_service.py changes
|
||||
- Automation still works for Stages 1-5
|
||||
|
||||
2. **Manual Workaround:**
|
||||
- Users can manually generate images from Writer/Images page
|
||||
- This uses the same `process_image_generation_queue` task
|
||||
|
||||
3. **No Data Loss:**
|
||||
- Stage 5 already created Images with prompts
|
||||
- These remain in database and can be processed anytime
|
||||
|
||||
---
|
||||
|
||||
## 📝 IMPLEMENTATION STEPS
|
||||
|
||||
1. **Update Code:** Modify `run_stage_6()` as documented above
|
||||
2. **Test Locally:** Run automation with test data
|
||||
3. **Code Review:** Verify changes match working Writer/Images implementation
|
||||
4. **Deploy:** Push to production
|
||||
5. **Monitor:** Watch first automation run for Stage 6 completion
|
||||
6. **Validate:** Check images generated and credits deducted
|
||||
|
||||
---
|
||||
|
||||
## 🎯 EXPECTED OUTCOME
|
||||
|
||||
After implementing this fix:
|
||||
|
||||
✅ **Stage 6 will work correctly** - Images generate from prompts
|
||||
✅ **Consistent with manual flow** - Same logic as Writer/Images page
|
||||
✅ **Proper credits tracking** - Automated deduction via AICore
|
||||
✅ **Sequential processing** - One image at a time with progress
|
||||
✅ **Error resilience** - Continues on failure, logs errors
|
||||
✅ **Full pipeline completion** - Automation flows from Stage 1 → Stage 7
|
||||
|
||||
---
|
||||
|
||||
## 🔗 RELATED FUNCTIONS
|
||||
|
||||
### Keep These Functions (Working Correctly)
|
||||
|
||||
- `GenerateImagePromptsFunction` - Stage 5 ✅
|
||||
- `AutoClusterFunction` - Stage 1 ✅
|
||||
- `GenerateIdeasFunction` - Stage 2 ✅
|
||||
- `GenerateContentFunction` - Stage 4 ✅
|
||||
|
||||
### Use This Task for Stage 6
|
||||
|
||||
- `process_image_generation_queue` - Celery task for Images → Generated Images ✅
|
||||
|
||||
### DO NOT USE in Automation
|
||||
|
||||
- `GenerateImagesFunction` - For Tasks, not for Images with existing prompts ❌
|
||||
|
||||
---
|
||||
|
||||
## END OF PLAN
|
||||
|
||||
This plan provides a clear, actionable fix for Automation Stage 6 image generation, aligning it with the working manual image generation flow used throughout the application.
|
||||
55
docs/backend/IGNY8-BACKEND-ARCHITECTURE.md
Normal file
55
docs/backend/IGNY8-BACKEND-ARCHITECTURE.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# IGNY8 Backend Architecture (code-sourced, Dec 2025)
|
||||
|
||||
Purpose: backend-only view based on current code under `backend/`. No legacy docs or assumptions.
|
||||
|
||||
## 1) Stack & Cross-Cutting
|
||||
- Django + DRF; Celery present via `backend/igny8_core/celery.py` and `tasks/`.
|
||||
- Multi-tenancy: models inherit `AccountBaseModel` (tenant FK `tenant_id`) or `SiteSectorBaseModel` for site/sector scoping (`backend/igny8_core/auth/models.py`). Account carries billing fields (email/address/tax_id).
|
||||
- Project URLs wired in `backend/igny8_core/urls.py`.
|
||||
|
||||
## 2) API Namespaces (as routed)
|
||||
- `/api/v1/auth/` → `igny8_core.auth.urls` (auth + CSV admin helpers).
|
||||
- `/api/v1/account/` → `igny8_core.api.urls` (account settings, team, usage analytics).
|
||||
- `/api/v1/planner/` → `igny8_core.modules.planner.urls` (keywords, clusters, ideas ViewSets).
|
||||
- `/api/v1/writer/` → `igny8_core.modules.writer.urls` (tasks, images, content, taxonomies ViewSets).
|
||||
- `/api/v1/system/` → `igny8_core.modules.system.urls` (prompts, author profiles, strategies, settings: system/account/user/modules/ai, module-enable toggle, health/status/metrics, Gitea webhook, integration settings save/test/generate/progress).
|
||||
- `/api/v1/billing/` → `igny8_core.business.billing.urls` (tenant billing: invoices, payments, credit-packages, credit-transactions, payment-methods CRUD/default, credits balance/usage/transactions; manual payment submit; available methods).
|
||||
- `/api/v1/admin/` → `igny8_core.modules.billing.admin_urls` (billing admin: stats, user credit adjustments, credit costs, invoices/payments/pending approvals, approve/reject, payment-method configs, account payment methods CRUD/default).
|
||||
- `/api/v1/automation/` → `igny8_core.business.automation.urls` (automation ViewSet).
|
||||
- `/api/v1/linker/` → `igny8_core.modules.linker.urls` (linker ViewSet).
|
||||
- `/api/v1/optimizer/` → `igny8_core.modules.optimizer.urls` (optimizer ViewSet).
|
||||
- `/api/v1/publisher/` → `igny8_core.modules.publisher.urls` (publishing records, deployments, root publisher actions, public site definition).
|
||||
- `/api/v1/integration/` → `igny8_core.modules.integration.urls` (integrations ViewSet + WordPress status/metadata webhooks).
|
||||
- OpenAPI docs: `/api/schema/`, `/api/docs/`, `/api/redoc/`.
|
||||
|
||||
## 3) Key Domain Models (code references)
|
||||
- Billing (`business/billing/models.py`):
|
||||
- Invoice: `subtotal`, `tax`, `total`, `currency`, `status` (`draft|pending|paid|void|uncollectible`), dates, `line_items`, `stripe_invoice_id`, `payment_method`, billing period, helpers `subtotal_amount|tax_amount|total_amount`.
|
||||
- Payment: statuses include `pending_approval`, `processing`, `completed/succeeded`, `failed/refunded/cancelled`; methods `stripe|paypal|bank_transfer|local_wallet|manual`; intent/charge ids, manual references, approvals, timestamps, `failure_reason`.
|
||||
- CreditPackage: slugged packages with `price`, `credits`, `discount_percentage`, stripe/paypal ids, `is_active`, `is_featured`, `sort_order`, `features`.
|
||||
- CreditTransaction: `transaction_type`, `amount`, `balance_after`, `metadata`, `reference_id`.
|
||||
- CreditUsageLog: `operation_type`, `credits_used`, tokens/model, related object refs, metadata.
|
||||
- CreditCostConfig: per-operation configurable costs (admin-editable).
|
||||
- AccountPaymentMethod: CRUD + default toggle; PaymentMethodConfig for availability by country/method.
|
||||
- Account (`auth/models.py`): Account with billing fields, credits balance, status, owner, stripe_customer_id; base models enforce tenant scoping.
|
||||
- Other business areas (content, planning, optimization, publishing, integration) define models/services under `backend/igny8_core/business/*` (models present in `content`, `planning`, `optimization`, `publishing`, `integration`; `linking` uses services).
|
||||
|
||||
## 4) Services & Admin Surfaces
|
||||
- Billing services: `business/billing/services/invoice_service.py`, `payment_service.py` (used by InvoiceViewSet/PaymentViewSet/admin aliases).
|
||||
- Admin billing aliases in `modules/billing/admin_urls.py` map to `AdminBillingViewSet` plus legacy stats/credit-cost endpoints.
|
||||
- Module enable and settings handled in `modules/system/settings_views.py` and integration settings in `modules/system/integration_views.py`.
|
||||
|
||||
## 5) Automation & Tasks
|
||||
- Automation API exposed via `business/automation/urls.py` (AutomationViewSet).
|
||||
- Celery tasks scaffolded under `igny8_core/tasks` and `tasks.py`; workers started via standard Celery entrypoints (see repo root README for commands).
|
||||
|
||||
## 6) Integration & Webhooks
|
||||
- WordPress webhooks: `/api/v1/integration/webhooks/wordpress/status/`, `/metadata/`.
|
||||
- Gitea webhook: `/api/v1/system/webhook/`.
|
||||
- Public site definition: `/api/v1/publisher/sites/<site_id>/definition/`.
|
||||
|
||||
## 7) Observations / gaps for follow-up
|
||||
- `docs/user-flow/` lacks flows; backend routes above should anchor those docs.
|
||||
- Ensure billing/admin docs use the namespaces and models listed here (avoid legacy paths). Data shape examples still need to be added alongside serializers.
|
||||
|
||||
|
||||
54
docs/backend/IGNY8-PLANNER-BACKEND.md
Normal file
54
docs/backend/IGNY8-PLANNER-BACKEND.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# Planner Module (Backend) — Code-Sourced Overview (Dec 2025)
|
||||
|
||||
Scope: `backend/igny8_core/modules/planner` (ViewSets/serializers/urls) and backing models in `backend/igny8_core/business/planning`.
|
||||
|
||||
## Endpoints (routed from `modules/planner/urls.py`)
|
||||
- `/api/v1/planner/keywords/` — CRUD + bulk actions, CSV import/export, clustering trigger.
|
||||
- `/api/v1/planner/clusters/` — CRUD with aggregated stats (keywords/volume/difficulty/ideas/content).
|
||||
- `/api/v1/planner/ideas/` — CRUD for content ideas tied to clusters/keywords.
|
||||
|
||||
## Models (business/planning/models.py)
|
||||
- **Clusters** (`SiteSectorBaseModel`, soft-delete): `name`, `description`, `keywords_count`, `volume`, `mapped_pages`, `status(new|mapped)`, `disabled`; unique per `site, sector`. Indexes on `name`, `status`, `site, sector`.
|
||||
- **Keywords** (`SiteSectorBaseModel`, soft-delete): FK `seed_keyword` (global), overrides `volume_override`, `difficulty_override`, `attribute_values`; FK `cluster` (same sector); `status(new|mapped)`, `disabled`. Unique per `seed_keyword, site, sector`. Properties expose `keyword`, `volume`, `difficulty`, `intent` from seed keyword. Save enforces industry/sector alignment between site/sector and seed keyword.
|
||||
- **ContentIdeas** (`SiteSectorBaseModel`, soft-delete): `idea_title`, `description`, `target_keywords` (legacy), M2M `keyword_objects`, FK `keyword_cluster` (same sector), `status(new|queued|completed)`, `disabled`, `estimated_word_count`, `content_type` (post/page/product/taxonomy), `content_structure` (article/guide/etc.). Tracks metadata (audience, tone, outlines, brief), CTA fields, review flags (`is_priority`, `is_assigned`, `is_review_required`), language, published URL, audit fields. Indexes on `site, sector, keyword_cluster, status, idea_title`.
|
||||
|
||||
## Serializers
|
||||
- **KeywordSerializer**: read-only keyword metrics from seed keyword; requires `seed_keyword_id` on create; optional on update; validates site/sector via the ViewSet; surfaces `cluster_name`, `sector_name`, `volume/difficulty/intent`.
|
||||
- **ClusterSerializer** (`cluster_serializers.py`): computes `keywords_count`, `volume`, `difficulty`, `ideas_count`, `content_count`; bulk prefetch helper to avoid N+1. Annotated volume/difficulty via overrides or seed keyword values.
|
||||
- **ContentIdeasSerializer**: links to clusters and keywords; exposes cluster/sector names; supports site/sector write-only ids; legacy taxonomy getter retained for backward compatibility.
|
||||
|
||||
## ViewSets (modules/planner/views.py)
|
||||
- **KeywordViewSet** (`SiteSectorModelViewSet`):
|
||||
- Filtering/search/order: search by seed keyword text; filter by status, cluster, intent, seed_keyword_id; ordering by created_at, volume, difficulty.
|
||||
- Custom filters: difficulty_min/max, volume_min/max using override-first logic.
|
||||
- Bulk actions: `bulk_delete`, `bulk_update` (status), `bulk_add_from_seed` (validates industry/sector alignment, requires site/sector), `import_keywords` CSV (site/sector required, creates Keywords), `export` CSV (ids filter supported).
|
||||
- Clustering: `auto_cluster` calls `ClusteringService.cluster_keywords`; enforces min keyword count (5) and credits; returns async task_id or sync result.
|
||||
- Create requires site_id and sector_id (validated against site/sector and account).
|
||||
- **ClusterViewSet**:
|
||||
- Filters/search/order: search by name; filter by status; ordering on name/created_at/keywords_count/volume/difficulty.
|
||||
- Annotates volume/difficulty using overrides; uses serializer prefetch to reduce N+1.
|
||||
- Export endpoints not present; CRUD via standard actions.
|
||||
- **ContentIdeasViewSet**:
|
||||
- CRUD for ideas; filters on status/cluster; search by title/description/keywords; ordering by created_at/status.
|
||||
- Generates title/brief via `IdeasService.generate_content_idea`; enforces credits (catches `InsufficientCreditsError`).
|
||||
- Bulk assign status not present; focused on single-item generation and listing.
|
||||
|
||||
## Services (business/planning/services)
|
||||
- **ClusteringService**: clusters keywords; invoked by `auto_cluster`; credit-aware.
|
||||
- **IdeasService**: generates content ideas (title/brief) from keywords; used in `ContentIdeasViewSet`.
|
||||
|
||||
## Permissions, tenancy, and throttling
|
||||
- Permissions: `IsAuthenticatedAndActive` + `IsViewerOrAbove` for all planner endpoints.
|
||||
- Tenancy: `SiteSectorModelViewSet` base ensures account/site/sector scoping; create/import/add-from-seed require explicit site_id + sector_id with consistency checks.
|
||||
- Throttling: `DebugScopedRateThrottle` with scope `planner`.
|
||||
|
||||
## CSV Import/Export
|
||||
- Export: `/planner/keywords/export` supports ids filter; outputs CSV with keyword, volume, difficulty, intent, status, cluster.
|
||||
- Import: `/planner/keywords/import_keywords` expects CSV with keyword, volume, difficulty, intent, status; site_id and sector_id required in query params; skips duplicates per site/sector/account.
|
||||
|
||||
## Notes / gaps
|
||||
- Two `bulk_update` methods exist in `KeywordViewSet` (duplicate definitions); consolidate to one.
|
||||
- No dedicated bulk idea status update; consider parity with keywords.
|
||||
- Legacy taxonomy references removed; ContentIdeas retains a legacy getter for taxonomy name but model FK is removed.
|
||||
|
||||
|
||||
73
docs/backend/IGNY8-WRITER-BACKEND.md
Normal file
73
docs/backend/IGNY8-WRITER-BACKEND.md
Normal file
@@ -0,0 +1,73 @@
|
||||
# Writer Module (Backend) — Code-Sourced Overview (Dec 2025)
|
||||
|
||||
Scope: `backend/igny8_core/modules/writer` (ViewSets/serializers/urls) backed by models in `backend/igny8_core/business/content`.
|
||||
|
||||
## Endpoints (from `modules/writer/urls.py`)
|
||||
- `/api/v1/writer/tasks/` — CRUD + bulk delete/update, auto content generation.
|
||||
- `/api/v1/writer/images/` — CRUD + serve file, auto generate images, bulk status update (by content or ids), grouped content images.
|
||||
- `/api/v1/writer/content/` — CRUD + publish/unpublish to WordPress, check WordPress status, AI prompt generation, validation, metadata mapping, taxonomy/tag management.
|
||||
- `/api/v1/writer/taxonomies/` — CRUD for content taxonomies (categories/tags). Attributes endpoint disabled (serializer removed).
|
||||
|
||||
## Models (business/content/models.py)
|
||||
- **Tasks** (`SiteSectorBaseModel`, soft-delete): title, description, FK `cluster` (required, same sector), optional `idea`, `content_type` (post/page/product/taxonomy), `content_structure` (article/guide/landing_page/etc.), optional `taxonomy_term`, comma keywords, word_count, status (`queued|completed`). Indexes on title/status/cluster/type/structure/site+sector.
|
||||
- **Content** (`SiteSectorBaseModel`, soft-delete): title, HTML, word_count, SEO fields, FK `cluster`, `content_type/structure`, M2M `taxonomy_terms` (through `ContentTaxonomyRelation`), external publishing fields (`external_id/url/type/metadata/sync_status`), source (`igny8|wordpress`), status (`draft|review|published`). Indexes on title/cluster/type/structure/source/status/external_id/site+sector.
|
||||
- **ContentTaxonomy** (`SiteSectorBaseModel`): name, slug, `taxonomy_type (category|tag)`, external_taxonomy/id, sync_status, description, count, metadata. Unique per site+slug+type and site+external_id+external_taxonomy.
|
||||
- **Images** (`SiteSectorBaseModel`, soft-delete): FK `content` (preferred) or `task` (legacy), `image_type (featured|desktop|mobile|in_article)`, `image_url/path`, prompt, status (`pending|generated|failed`), position. save() inherits account/site/sector from content/task. Indexes on content/task type/status/position.
|
||||
- **ContentClusterMap** (summary): links content/tasks to clusters with roles (hub/supporting/attribute) and source (blueprint/manual/import); tracks keywords + slugs.
|
||||
- **ContentTaxonomyRelation**: through table for content↔taxonomy.
|
||||
|
||||
## Serializers (modules/writer/serializers.py)
|
||||
- **TasksSerializer**: requires cluster/content_type/content_structure on create; defaults status=queued; exposes cluster_name/sector_name; accepts site_id/sector_id (write-only).
|
||||
- **ImagesSerializer**: exposes task_title/content_title; read-only account/timestamps.
|
||||
- **ContentSerializer**: requires cluster/content_type/content_structure/title on create; defaults source=igny8, status=draft; exposes taxonomy terms, tags, categories, image flags/status; site_id/sector_id write-only.
|
||||
- **ContentTaxonomySerializer** (in file but not shown above): manages taxonomy fields and external sync data.
|
||||
- Group serializers for content images (`ContentImageSerializer`, `ContentImagesGroupSerializer`).
|
||||
|
||||
## ViewSets & Actions (modules/writer/views.py)
|
||||
- **TasksViewSet** (`SiteSectorModelViewSet`):
|
||||
- Filters/search/order: search title/keywords; filter status/cluster/content_type/structure; order by title/created_at/status.
|
||||
- Bulk: `bulk_delete`, `bulk_update` (status).
|
||||
- AI: `auto_generate_content` → `ContentGenerationService.generate_content(ids, account)`; limits 10 ids; returns task_id or sync result; 402 on `InsufficientCreditsError`.
|
||||
- Create requires explicit `site_id` and `sector_id`; validates site/sector association; account resolved from request user/site.
|
||||
- **ImagesViewSet**:
|
||||
- Filters/order: filter task_id/content_id/image_type/status; order by created_at/position/id.
|
||||
- Create enforces site/sector (from request context, falling back to user active site/default sector); sets account/site/sector.
|
||||
- Actions:
|
||||
- `serve_image_file` streams local file if `image_path` exists.
|
||||
- `auto_generate_images` queues `run_ai_task(generate_images)` (Celery if available) for up to 10 task_ids.
|
||||
- `bulk_update` sets status by `content_id` or `ids`.
|
||||
- `content_images` returns grouped images (featured + in-article) via grouped serializer.
|
||||
- **ContentViewSet** (not fully shown above but key actions):
|
||||
- CRUD with filters/search/order (title, status, cluster, content_type/structure, source); tenant scoping via base class.
|
||||
- Taxonomy management: add/remove terms, sync helpers (uses `ContentTaxonomy`).
|
||||
- AI helpers: `generate_image_prompts` (queues `run_ai_task(generate_image_prompts)`), validation (`ContentValidationService`), metadata mapping (`MetadataMappingService`), content generation pathways tied to `ContentGenerationService`.
|
||||
- Publishing: `publish` queues `publish_content_to_wordpress` Celery task (optimistically sets status=published), `wordpress_status` fetches WP status via integration, `unpublish` clears external links and reverts to draft.
|
||||
- Image status helpers and grouped image retrieval also exposed.
|
||||
- **ContentTaxonomyViewSet**: CRUD for categories/tags; supports external sync fields.
|
||||
|
||||
## Permissions, tenancy, throttling
|
||||
- Permissions: `IsAuthenticatedAndActive` + `IsViewerOrAbove` across Writer; certain actions (e.g., `unpublish`) use `IsEditorOrAbove`.
|
||||
- Tenancy: `SiteSectorModelViewSet` enforces account/site/sector scoping; create operations require site_id+sector_id or context site/sector; Images save derives account/site/sector from content/task.
|
||||
- Throttling: `DebugScopedRateThrottle` with scope `writer`.
|
||||
|
||||
## AI/Background Tasks & Limits
|
||||
- Content generation: `auto_generate_content` (max 10 task IDs) → `ContentGenerationService`; credits enforced (402 on insufficient).
|
||||
- Image generation: `auto_generate_images` (max 10 task IDs) via `run_ai_task`; Celery preferred, sync fallback.
|
||||
- Image prompt generation: `generate_image_prompts` via `run_ai_task`.
|
||||
|
||||
## Publishing Integration
|
||||
- Publish: `/writer/content/{id}/publish/` → queues WordPress publish; refuses if already published.
|
||||
- Status: `/writer/content/{id}/wordpress_status/` → checks WordPress via site integration.
|
||||
- Unpublish: clears external_id/url, sets status to draft (does not delete WP post).
|
||||
|
||||
## CSV / Bulk Notes
|
||||
- Tasks: bulk delete/update supported.
|
||||
- Images: bulk status update by content or ids; grouped retrieval.
|
||||
- Content: bulk operations primarily around AI prompt/image generation and publishing; no CSV import/export.
|
||||
|
||||
## Observations / gaps
|
||||
- Tasks status choices limited to `queued/completed`; no explicit error/cancel states.
|
||||
- Images bulk_update exposed on same ViewSet name as Tasks bulk_update (distinct routes under images vs tasks); OK but keep consistent naming.
|
||||
- ContentViewSet file is large; ensure doc readers reference actions for validation/metadata mapping if extending.
|
||||
|
||||
|
||||
59
docs/igny8-app/IGNY8-APP-ARCHITECTURE.md
Normal file
59
docs/igny8-app/IGNY8-APP-ARCHITECTURE.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# IGNY8 App Architecture (code-sourced, Dec 2025)
|
||||
|
||||
Authoritative snapshot derived from the current codebase (backend Django/DRF, frontend React/Vite). No legacy docs referenced.
|
||||
|
||||
## 1) Platform Overview
|
||||
- **Multi-tenant:** All tenant-scoped models inherit `AccountBaseModel`; site/sector models inherit `SiteSectorBaseModel` (`backend/igny8_core/auth/models.py`).
|
||||
- **APIs (wired in `backend/igny8_core/urls.py`):** `/api/v1/auth/`, `/api/v1/account/`, `/api/v1/planner/`, `/api/v1/writer/`, `/api/v1/system/`, `/api/v1/billing/` (tenant billing), `/api/v1/admin/` (billing admin + credit costs), `/api/v1/automation/`, `/api/v1/linker/`, `/api/v1/optimizer/`, `/api/v1/publisher/`, `/api/v1/integration/`.
|
||||
- **Frontend routes (in `frontend/src/App.tsx`):** Planner, Writer, Automation, Linker, Optimizer, Thinker, Billing, Account, Admin sections; sidebar defined in `frontend/src/layout/AppSidebar.tsx`.
|
||||
|
||||
## 2) Backend Modules & Endpoints
|
||||
- **Planner** (`modules/planner/urls.py`): `/keywords`, `/clusters`, `/ideas` via ViewSets.
|
||||
- **Writer** (`modules/writer/urls.py`): `/tasks`, `/images`, `/content`, `/taxonomies`.
|
||||
- **Automation** (`business/automation/urls.py`): root ViewSet for automation orchestration.
|
||||
- **Linker** (`modules/linker/urls.py`): root ViewSet for internal linking.
|
||||
- **Optimizer** (`modules/optimizer/urls.py`): root ViewSet for optimization analysis.
|
||||
- **Publisher** (`modules/publisher/urls.py`): `/publishing-records`, `/deployments`, root publisher actions, plus public `sites/<id>/definition/`.
|
||||
- **Integration** (`modules/integration/urls.py`): `/integrations`, WordPress webhooks at `/webhooks/wordpress/status|metadata/`.
|
||||
- **System** (`modules/system/urls.py`): prompts, author profiles, strategies, settings (system/account/user/modules/ai), module enable toggle at `/settings/modules/enable/`, health/status/metrics, Gitea webhook, integration settings save/test/generate/progress routes.
|
||||
- **Account** (`api/urls.py`): `/settings`, `/team`, `/usage/analytics/`.
|
||||
- **Billing (tenant)** (`business/billing/urls.py`): invoices, payments (list + manual submit + available methods), credit-packages, credit-transactions, payment-methods (CRUD + set_default), credits balance/usage/transactions.
|
||||
- **Billing (admin)** (`modules/billing/admin_urls.py` under `/api/v1/admin/`): stats, users credit adjustments, credit costs, plus aliases to `AdminBillingViewSet` for invoices, payments, pending_payments, approve/reject, payment-method-configs, account-payment-methods (CRUD + set_default).
|
||||
|
||||
## 3) Billing & Credits Data Model (key fields)
|
||||
- **Invoice:** `subtotal`, `tax`, `total`, `currency`, `status`, `invoice_date`, `due_date`, `paid_at`, `line_items`, `stripe_invoice_id`, `payment_method`, billing period fields; helper properties expose `subtotal_amount`, `tax_amount`, `total_amount` for compatibility.
|
||||
- **Payment:** statuses include `pending_approval`, `processing`, `completed/succeeded`, `failed/refunded/cancelled`; methods include `stripe`, `paypal`, `bank_transfer`, `local_wallet`, `manual`; tracks intent/charge ids, manual references, approvals, timestamps, failure_reason.
|
||||
- **CreditPackage:** slugged packages with price, credits, stripe/paypal ids, `is_active`, `is_featured`, `sort_order`, features JSON.
|
||||
- **CreditTransaction:** transaction_type, amount, balance_after, metadata, reference_id (invoice/payment), account-scoped.
|
||||
- **CreditUsageLog:** operation_type, credits_used, tokens/model, related object refs, metadata; account-scoped.
|
||||
- **CreditCostConfig:** per-operation configurable costs (admin editable).
|
||||
- **Account billing fields:** billing email/address/tax_id on `Account` (multi-tenant).
|
||||
|
||||
## 4) Frontend Surface (routes & guards)
|
||||
- **Planner:** `/planner/keywords`, `/planner/clusters`, `/planner/clusters/:id`, `/planner/ideas`.
|
||||
- **Writer:** `/writer/tasks`, `/writer/content`, `/writer/content/:id`, `/writer/drafts` (redirect), `/writer/images`, `/writer/review`, `/writer/published`.
|
||||
- **Automation:** `/automation`.
|
||||
- **Linker:** `/linker`, `/linker/content`.
|
||||
- **Optimizer:** `/optimizer`, `/optimizer/content`, `/optimizer/analyze/:id`.
|
||||
- **Thinker:** `/thinker` redirect → `/thinker/prompts`; also `/thinker/author-profiles`, `/thinker/profile`, `/thinker/strategies`, `/thinker/image-testing`.
|
||||
- **Billing (tenant module):** `/billing/overview`, `/billing/credits`, `/billing/transactions`, `/billing/usage`.
|
||||
- **Account:** `/account/plans`, `/account/billing`, `/account/purchase-credits`, `/account/settings`, `/account/team`, `/account/usage`.
|
||||
- **Admin:** `/admin/dashboard`, `/admin/accounts`, `/admin/subscriptions`, `/admin/account-limits`, `/admin/billing`, `/admin/invoices`, `/admin/payments`, `/admin/payments/approvals`, `/admin/credit-packages`.
|
||||
- **Sidebar source:** `frontend/src/layout/AppSidebar.tsx` (Account group contains Plans & Billing, Plans, Team Management, Usage & Analytics).
|
||||
|
||||
## 5) Multi-Tenancy & Roles
|
||||
- Tenant isolation via `AccountBaseModel` FKs (`account_id`/`tenant_id` column) with indexes; site/sector models enforce belonging to same account.
|
||||
- Admin-only endpoints served under `/api/v1/admin/` (developer/superuser expected; see billing admin ViewSet).
|
||||
- Frontend ModuleGuard used for module-based access on feature modules (planner/writer/linker/optimizer/thinker); account/billing pages are standard routes.
|
||||
|
||||
## 6) Integration & Webhooks
|
||||
- WordPress status/metadata webhooks at `/api/v1/integration/webhooks/wordpress/status|metadata/`.
|
||||
- Gitea webhook at `/api/v1/system/webhook/`.
|
||||
- Health: `/api/v1/system/ping`, `/api/v1/system/status`, request metrics at `/api/v1/system/request-metrics/<request_id>/`.
|
||||
|
||||
## 7) Known Documentation Needs (to be maintained)
|
||||
- Align billing/account docs to the above code-level namespaces and models.
|
||||
- Add frontend route map (kept here) to the Account/Billing docs.
|
||||
- Fill `docs/user-flow/` (currently empty) with end-to-end user/account/billing flows based on this architecture.
|
||||
|
||||
|
||||
@@ -201,6 +201,11 @@ const AppSidebar: React.FC = () => {
|
||||
name: "Plans & Billing",
|
||||
path: "/account/billing",
|
||||
},
|
||||
{
|
||||
icon: <DollarLineIcon />,
|
||||
name: "Plans",
|
||||
path: "/account/plans",
|
||||
},
|
||||
{
|
||||
icon: <UserIcon />,
|
||||
name: "Team Management",
|
||||
|
||||
13
user-flow-plan-in-progress-2.md
Normal file
13
user-flow-plan-in-progress-2.md
Normal file
@@ -0,0 +1,13 @@
|
||||
Differences from the two docs (user-flow + session-config-summary) in the current implementation:
|
||||
|
||||
1) Subscriptions throttling still impacts normal users
|
||||
- Docs expect “plans fetch should not be throttled” and smooth onboarding; in practice, `/v1/auth/subscriptions/` is still returning 429s for normal users until the backend restart picks up the relaxed scope.
|
||||
|
||||
2) Payment methods are not auto-present for all accounts
|
||||
- Docs assume payment methods are available so plan selection isn’t blocked. Account 29 had none until we inserted bank_transfer and PK manual; the UI showed “No payment methods available.”
|
||||
|
||||
3) Enterprise plan is hidden for non-aws-admin
|
||||
- Not called out in the docs; current UI filters out Enterprise for normal tenants.
|
||||
|
||||
4) Onboarding friction from auth/plan calls
|
||||
- Docs expect a seamless redirect and retry-once for 429; current flow still shows visible errors (429/403) when token is missing/expired or when subscriptions are throttled.
|
||||
Reference in New Issue
Block a user