diff --git a/Pre-Launch-Pending.md b/Pre-Launch-Pending.md new file mode 100644 index 00000000..97a7e5e2 --- /dev/null +++ b/Pre-Launch-Pending.md @@ -0,0 +1,556 @@ +--- + +# **APP LAUNCH PLAN: ITEMS 1 to 3** + +# **Item 1: Status Modules for AI Planner and Writer** + +### **Objective** + +Ensure Planner and Writer status tracking is consistent, accurate, and fully functional across backend and frontend. + +### **Work Scope** + +#### **A. Backend** + +* Standardize status states for: + + * Planner tasks + * Writer tasks + * Queue processing + * AI function progress +* Create unified status enums and error codes. +* Ensure all AI jobs push accurate progress updates during: + + * Keyword extraction + * Clustering + * Idea generation + * Content generation + * Image generation +* Ensure all status updates propagate through WebSockets or async polling. +* Log transitions for debugging. + +#### **B. Frontend** + +* Display accurate status for: + + * Each task + * Each queue item + * Each stage of execution +* Ensure Planner and Writer modals show: + + * Correct live progress + * Correct messaging + * Correct completion and failure notices +* Refactor old status code to match backend changes. + +#### **C. QA** + +* Test all workflows manually on: + + * Keywords + * Clusters + * Ideas + * Content + * Automation wizard +* Test edge cases: + + * Stuck tasks + * Failed tasks + * Incorrect counts + * Race conditions +* Validate real-time UI updates. + +--- + +# **Item 2: Credits, Billing, Pricing Logic, and Usage Limits** + +### **Objective** + +Define credit costs per AI function and define plan limits for counts and usage. + +### **Work Scope** + +#### **A. Define Credit Cost per AI Function** + +For each AI operation, decide exact credit cost: + +* Keyword extraction +* Clustering +* SAG mapping +* Idea generation +* Content generation (per word length) +* Image generation +* Any additional operations + +This defines package pricing and resource consumption. + +#### **B. Define Plan-Based Usage Limits** + +Define limits that each plan can have: + +* Maximum keywords saved +* Maximum clusters +* Maximum ideas +* Maximum content pieces +* Maximum images +* Max tasks queued at a time +* Limits for daily or monthly usage caps +* Per-account or per-user limits + +These limits sit **on top of credit allocation**, or credit alone may cap output. Decision required. + +#### **C. Backend Implementation** + +* Add limit checks in API for create actions. +* Add counters for monthly/daily usage. +* Add logic to block or warn when limits reached. +* Integrate credit deduction rules per function. + +#### **D. Frontend Implementation** + +* Show plan limits clearly. +* Show remaining credits and remaining records. +* Show warnings when nearing limits. +* Unify progress counters in dashboards. + +#### **E. Pricing Plan Optimization** + +* After defining credit values and limits: + + * Create Starter, Growth, Pro, and Enterprise plans. + * Set competitive credit allocations. + * Set record-based limits and overage policies. + * Ensure predictable cost for users and profitability for platform. + +--- + +# **Item 3: Prompt Improvement and Model Optimization** + +### **Objective** + +Fully redesign AI prompts for extreme accuracy, consistent output, faster processing, and correct word count. Improve content quality, clustering quality, and image prompts. + +### **Work Scope** + +#### **A. SAG Clustering Prompt** + +Handled separately later but included as part of item 3. + +Goal: + +* Convert IGNY8 semantic architecture into an AI-usable compact prompt. +* Include examples. +* Produce consistent clustering across all modules. + +(This will be done in next phase.) + +--- + +#### **B. Idea Generation Prompt (Improve Existing Prompt)** + +Improvements needed: + +* Increase semantic alignment with cluster intent. +* Produce stronger, more realistic editorial ideas. +* Improve JSON formatting reliability. +* Better keyword mapping. +* Ensure structure variation and non-repetitive logic. +* Produce cluster hub + proper supporting ideas with strict editorial rules. + +Output: +A refined prompt for idea generation with optimized structure, clarity, and output consistency. + +--- + +#### **C. Content Generation Prompt (Improve Existing Prompt)** + +Improvements needed: + +1. **Precise word-count adaptation** + + * 500, 1000, 1500 words + * Automatic proportional scaling +2. **Better editorial quality** + + * Richer depth + * More variation in style and examples + * No generic flavor +3. **Smarter structure control** + + * Paragraphs first + * Then list/table + * Never start with list + * Proper subsection placement +4. **Strict SEO rules** + + * Primary keyword in title, intro, and two H2s + * Natural secondary keywords +5. **HTML consistency** + + * Clean tags + * No broken JSON strings +6. **Better image prompts** + + * Clear + * Topic relevant + * Frontend-friendly + +Output: +A set of optimized content prompts for 500, 1000, 1500 versions. + +--- + +# **Summary of Deliverables for Items 1–3** + +### **Item 1 Deliverables** + +* Unified backend status engine +* Clean frontend progress modals +* Error-proof workflow states +* Automation wizard corrections + +### **Item 2 Deliverables** + +* Full credit cost table per function +* Pricing plan limits and counters +* Backend logic for limit enforcement +* Frontend display of usage and credit consumption + +### **Item 3 Deliverables** + +* Rewritten idea-generation prompt +* Rewritten content-generation prompt (3 lengths) +* Improved image prompt format +* Separate SAG clustering prompt (later) + +--- + +# 4. Page Flow and Workflow UX Improvements + +Improve the overall navigation, page structure, workflow clarity, and helper systems across all main user-facing areas. + +## 4.1 Improve Page Flows Across Main Menu Groups + +### Groups to cover + +* Setup +* Workflow +* Account +* Settings + +### Required Enhancements + +| Area | Improvement | +| -------------------- | ------------------------------------------------------------------------------------ | +| Navigation hierarchy | Standardize all menu groups and subpages with consistent UX patterns. | +| Linear workflows | Ensure planner and writer flows follow a clear step-by-step progression. | +| Breadcrumbs | Add contextual breadcrumbs for deeper sections. | +| Page orientation | Each page requires a header with title, short description, and quick action buttons. | +| Cross-navigation | Add "Back to previous step", "Continue", and "View progress" controls. | + +--- + +## 4.2 Workflow UX Improvements + +### Planner and Writer workflows need: + +* Step banners +* Clear state indicators +* Autosave indicators +* Better use of helper components + +### Helper Elements Required + +| Component | Purpose | +| -------------------- | ------------------------------------------------------------------ | +| Helper notifications | Display small contextual guidance when entering a new screen. | +| Tooltips | For icons, complex actions, and configuration options. | +| Inline guidance | Small text under fields explaining purpose or required formatting. | +| Step banners | Persistent “Step X of Y” headers guiding the user. | + +--- + +## 4.3 Notification System + +### Requirements + +* Event-driven notifications for task progress, completion, failure, errors, and actions required. +* Unified notification dropdown accessible via bell icon in the header. + +### Notification Types + +| Type | Examples | +| --------------- | ------------------------------------------------------- | +| Progress | Planner or writer job started, queued, processing. | +| Success | Content generated, workflow completed, export success. | +| Failure | API errors, workflow failure, validation issues. | +| Action required | Missing settings, incomplete fields, failed publishing. | + +### Dropdown Features + +* Chronological list +* Read/unread states +* Links to relevant pages or tasks +* Icon-based status indicators (info, success, warning, danger) + +--- + +## 4.4 Global and Page-level Metrics + +### Bottom Metrics Panel (per workflow page) + +* Move detailed metrics to the bottom of Planner and Writer workflows. +* Collapsible panel. +* Shows counts, progress, keyword metrics, SEO density, etc. + +### Header Metrics (global) + +Improved header-level metrics visible everywhere: + +* Tasks in progress +* Completed tasks +* Failed tasks +* Credits remaining +* Usage summary +* Notification count + +--- + +## 4.5 Tenant-level Settings Reorganization + +### Problem + +Settings are scattered and inconsistent. + +### Solution + +Create a centralized **Tenant Settings Hub** (superuser excluded). + +### New Structure + +| Category | Subpages | +| ------------------- | ----------------------------------------------------- | +| General | Basic account settings, display options. | +| Publishing | WordPress connection, publishing defaults, SEO rules. | +| Pagination | Control pagination for listings, workflows, results. | +| Branding | Logo, colors, styling preferences. | +| Workflow Defaults | Planner, writer, auto settings, task behaviors. | +| Integrations | API keys, external platforms, plugins. | +| Notifications | Type preferences, email alerts, push alerts. | +| Account and Billing | Plan, credits, invoices, team access. | + +--- + +# 5. WordPress Content Template Improvements and Redesign + +## 5.1 Image Reuse Logic + +Automatically reuse available images based on count. + +| Available Images | Behavior | +| ---------------- | ------------------------------------------------------------------ | +| 1 | Reuse multiple times across intro, middle, and final sections. | +| 2 | Alternate image placement across sections. | +| 3 | Use each twice for balanced distribution. | +| More than 3 | Use each once until exhausted, then reuse based on article length. | + +--- + +## 5.2 Placement Rules + +Images must appear at predictable, meaningful points. + +| Placement | Purpose | +| ------------------------ | ----------------------------------- | +| After introduction | Establish visual tone. | +| Before major H2 sections | Break reading monotony. | +| Mid-article | Reset attention, add visual pacing. | +| Before conclusion | Support final argument or CTA. | + +--- + +## 5.3 Template Versions by Article Length + +### A. 500-word Template + +* Intro (1 paragraph) +* H2 Section 1 (1 paragraph) +* H2 Section 2 (1–2 paragraphs) +* Conclusion (1 paragraph) +* Images: 2 to 3 placements + +### B. 1000-word Template + +* Intro (2 paragraphs) +* H2 Section 1 (2 paragraphs) +* H2 Section 2 (2 paragraphs) +* H2 Section 3 (2 paragraphs) +* Conclusion (2 paragraphs) +* Images: 3 to 5 placements + +### C. 1500-word Template + +* Intro (2–3 paragraphs) +* H2 Section 1 (3 paragraphs) +* H2 Section 2 (3 paragraphs) +* H2 Section 3 (3 paragraphs) +* Optional additional H2 or H3 subsections +* Conclusion with CTA +* Images: 4 to 7 placements + +--- + +## 5.4 Cluster Hub Landing Page Template + +This single template is reused for: + +* Cluster hubs +* Category pages +* Tag archive pages + +### Core Structure + +| Section | Description | +| ---------------------- | ------------------------------------------------- | +| Hero section | Term title, description, and optional hero image. | +| Featured visual | Illustration or banner for cluster identity. | +| Overview block | Short intro text about the topic group. | +| Subcluster cards | If hierarchical categories exist. | +| Latest articles list | Auto-populated. | +| Popular articles block | Based on views or tagging. | +| CTA block | Newsletter or lead magnet. | +| Footer navigation | Internal linking anchors. | + +### Conditional Variants + +| Term Type | Variation | +| ----------- | -------------------------------------- | +| Category | Shows category tree and subcategories. | +| Tag | Shows related tag cloud. | +| Cluster Hub | Includes deeper cluster overview map. | + +--- + +# 6. Frontend Marketing Site + +## 6.1 Update Marketing Site Content + +### Pages to Update + +| Page | Required Work | +| -------------- | ------------------------------------------------------------------------ | +| Home | Rewrite hero, features, workflows, CTAs, add visuals. | +| Features | List key modules: planner, writer, metrics, notifications, settings hub. | +| Product pages | Dedicated pages for each workflow module. | +| Use-case pages | For agencies, e-commerce, content teams. | +| About | Mission, philosophy, architecture reasoning. | +| Contact | Updated contact and support flow. | + +--- + +## 6.2 Update Pricing Page and Plans + +### Required Upgrades + +| Area | Details | +| ------------------- | ------------------------------------------------------- | +| Plan definitions | Clearly define Free, Starter, Growth, Enterprise tiers. | +| Usage-based pricing | Explain credit system and workflow limits. | +| Add-ons | Extra users, domains, integrations. | +| Comparison tables | Feature-by-feature breakdown. | +| Billing cycles | Monthly and annual toggles. | +| Global currencies | USD baseline, PK optional. | +| FAQs under pricing | Immediate resolution for common objections. | + +--- + +# 7. Documentation + +## 7.1 In-app Documentation + +### Areas to update + +| Section | Required Documentation | +| ------------------ | ------------------------------------------------- | +| Planner workflow | Step-by-step guides, tooltips, screenshots. | +| Writer workflow | Drafting, editing, optimization, publishing. | +| Tenant settings | Publishing, pagination, defaults, integrations. | +| Metrics | Definitions, data refresh rules, interpretation. | +| Notifications | Event types, triggers, statuses, troubleshooting. | +| Account management | Team roles, permissions, billing, usage. | + +### Placement + +* Floating help icon +* Embedded guides per workflow +* Onboarding tutorials + +--- + +## 7.2 Marketing Site Documentation + +### Includes + +| Area | Content Needed | +| ------------------- | -------------------------------------- | +| API documentation | Endpoints, auth, requests, responses. | +| Integration guides | WordPress setup, API keys, automation. | +| Workflow guides | Planner-to-writer SEO flow. | +| AI usage | Credit consumption and behavior specs. | +| Notification events | JSON payloads, event types, callbacks. | +| Data privacy | Storage, security, backups. | + +--- + +## 7.3 FAQ Updates + +### Categories + +* General +* Pricing +* Content generation +* Integrations +* Settings +* Technical troubleshooting +* Data handling + +### Structure + +* Accordion layout +* Search bar +* Support CTA + + +--- +Known bugs and issues +1. AI functions +Manual run on pages +For pages: Keywords, Cluster, and Tasks, for clustering, idea generation, content generation: + +The texts shown in manual AI function run for Planner and Writer progress modals: + +Text is not accurate. +Needs to be fixed for all different modals based on the specific function running. +Process to fix: + +Ask AI agent to build a table of current texts. + +Then optimize these texts with: + +Better wording. +Dynamic variables. +Automation (Manual Run for automation wizard on automation page) +Wrong queue items. + +Missing queue items. + +Progress bar does not progress properly. + +Total in queue and processed counts are buggy for many stages. + +Stage cards: + +Realtime metrics not optimized. +Should be more robust and more user friendly. + +user sometimes log out atumatically , diff --git a/docs/PRE-LAUNCH/ITEM-1-STATUS-MODULES.md b/docs/PRE-LAUNCH/ITEM-1-STATUS-MODULES.md new file mode 100644 index 00000000..e10be020 --- /dev/null +++ b/docs/PRE-LAUNCH/ITEM-1-STATUS-MODULES.md @@ -0,0 +1,547 @@ +# Item 1: Status Modules for AI Planner and Writer + +**Priority:** Critical +**Target:** Production Launch +**Last Updated:** December 11, 2025 + +--- + +## Overview + +Standardize and fix status tracking across Planner and Writer modules to ensure consistent, accurate, and real-time progress updates for all AI operations. This includes backend status management, frontend progress displays, WebSocket/polling systems, and error handling. + +--- + +## Current Implementation Analysis + +### Backend Status System + +#### Status Fields in Models + +**Location:** `backend/igny8_core/business/planning/models.py` and `backend/igny8_core/business/content/models.py` + +| Model | Current Status Choices | Issues | +|-------|----------------------|---------| +| **Keywords** | `new`, `mapped` | Limited states, no processing states | +| **Clusters** | `new`, `mapped` | No active/processing states | +| **ContentIdeas** | `new`, `queued`, `completed` | Missing `processing`, `failed` states | +| **Tasks** | `queued`, `completed` | Missing intermediate states (processing, failed, cancelled) | +| **Content** | `draft`, `review`, `published` | Separate from generation status | +| **AutomationRun** | `running`, `paused`, `cancelled`, `completed`, `failed` | Well-defined but isolated | + +**Key Problem:** Status choices are inconsistent across models. Some models track generation status, others track editorial status, causing confusion. + +--- + +### AI Function Progress Tracking + +**Location:** `backend/igny8_core/ai/tracker.py`, `backend/igny8_core/ai/engine.py` + +#### Progress Phases + +All AI functions emit standard progress phases: + +| Phase | Default Label | Percentage Range | Purpose | +|-------|--------------|------------------|---------| +| `INIT` | Initializing... | 0-10% | Validation, setup | +| `PREP` | Preparing data... | 10-20% | Data loading, preparation | +| `AI_CALL` | Processing with AI... | 20-80% | AI model execution | +| `PARSE` | Parsing response... | 80-90% | Response validation, parsing | +| `SAVE` | Saving results... | 90-100% | Database operations | +| `DONE` | Complete! | 100% | Final state | + +**Implementation:** +- `StepTracker` class emits step-level events +- `ProgressTracker` class manages Celery task state updates +- `AIEngine` orchestrates phase transitions + +**Issues:** +- Phase messages are generic and don't reflect actual counts +- Percentage calculations sometimes inconsistent +- No clear error state handling in phases + +--- + +### Frontend Progress Modal System + +**Location:** `frontend/src/hooks/useProgressModal.ts`, `frontend/src/components/common/ProgressModal.tsx` + +#### Current Modal Implementation + +**Hook:** `useProgressModal` +- Manages modal state, task polling, progress updates +- Polls backend every 1 second for task status +- Maps backend phases to user-friendly messages +- Tracks step logs for debugging + +**Modal Component:** `ProgressModal` +- Displays progress bar with percentage +- Shows current phase and message +- Lists step-by-step progress visually +- Provides cancel/close actions + +#### Step Mapping Logic + +**Function:** `getStepInfo()` in `useProgressModal.ts` + +Maps backend phases to: +- User-friendly messages with dynamic counts +- Adjusted percentages for smooth progression +- Function-specific labels (clustering vs. ideas vs. content) + +**Problems Identified:** +1. **Inaccurate counts in messages**: Extracts counts from messages using regex, often missing or incorrect +2. **Generic phase labels**: Messages don't always reflect what's actually happening +3. **Inconsistent percentage progression**: Jumps or stalls during AI_CALL phase +4. **Status text not aligned**: Labels in modal don't match backend phase names + +--- + +### Progress Display by Page + +| Page | Location | Progress Display | Issues | +|------|----------|------------------|--------| +| Keywords | `frontend/src/pages/Planner/Keywords.tsx` | ProgressModal, AI logs | Clustering progress text inaccurate | +| Clusters | `frontend/src/pages/Planner/Clusters.tsx` | ProgressModal only | No cluster generation function exists | +| Ideas | `frontend/src/pages/Planner/Ideas.tsx` | ProgressModal, reload on complete | Text shows generic "preparing clusters" | +| Tasks | `frontend/src/pages/Writer/Tasks.tsx` | ProgressModal, AI logs | Content generation text inaccurate | +| Content | `frontend/src/pages/Writer/Content.tsx` | ProgressModal | Missing real-time updates | +| Images | `frontend/src/pages/Writer/Images.tsx` | ImageQueueModal | Uses separate queue-based system | + +--- + +### Known Bugs and Issues + +From `Pre-Launch-Pending.md` Known Bugs section: + +#### Manual AI Function Run Modals + +**Affected Pages:** Keywords, Clusters, Ideas, Tasks + +**Issues:** +- Progress modal texts are not accurate for specific functions +- Dynamic variables not properly populated with counts +- Generic messages instead of function-specific ones + +**Example:** +- Clustering: Should say "Mapping 45 keywords into clusters" but shows "Preparing data..." +- Ideas: Should say "Generating ideas for 3 clusters" but shows generic text + +--- + +#### Automation Wizard Progress + +**Page:** Automation (`frontend/src/pages/Automation/AutomationWizard.tsx`) + +**Issues:** +- Wrong queue items displayed +- Missing queue items for certain stages +- Progress bar doesn't progress properly +- Total in queue and processed counts buggy across stages +- Stage cards: Real-time metrics not optimized + +--- + +### WebSocket vs Polling + +**Current Implementation:** Polling-based + +**Location:** `useProgressModal.ts` - polls task status every 1 second + +**Note:** Code mentions WebSockets but current implementation uses REST API polling via `/api/v1/...` endpoints + +**Improvement Needed:** Consider WebSocket implementation for real-time updates without polling overhead + +--- + +## Required Changes + +### A. Backend Status Standardization + +#### 1. Unified Status Enum + +**Create:** `backend/igny8_core/common/status_enums.py` + +Define standard status choices for all modules: + +``` +GENERATION_STATUS_CHOICES = [ + ('pending', 'Pending'), + ('queued', 'Queued'), + ('processing', 'Processing'), + ('completed', 'Completed'), + ('failed', 'Failed'), + ('cancelled', 'Cancelled'), +] + +CONTENT_STATUS_CHOICES = [ + ('draft', 'Draft'), + ('review', 'Review'), + ('published', 'Published'), + ('archived', 'Archived'), +] +``` + +**Action:** Add `generation_status` field to models that undergo AI processing: +- Keywords (for extraction/clustering operations) +- Clusters (for cluster generation, if implemented) +- ContentIdeas (for idea generation) +- Tasks (for content generation) +- Content (separate from editorial status) + +#### 2. Enhanced Progress Details + +**Update:** `backend/igny8_core/ai/tracker.py` + +**Modifications:** +- `StepTracker.update_step()` should include: + - `items_total`: Total count of items being processed + - `items_processed`: Current count of items processed + - `current_item_name`: Name/title of current item + - `estimated_remaining_seconds`: Estimated time remaining + +**Example Progress Payload:** +```json +{ + "phase": "AI_CALL", + "message": "Clustering 45 keywords", + "percentage": 45, + "items_total": 45, + "items_processed": 20, + "current_item": "best seo tools", + "estimated_seconds_remaining": 120 +} +``` + +#### 3. Function-Specific Progress Messages + +**Update:** Each AI function's phase messages + +**Files to modify:** +- `backend/igny8_core/ai/functions/auto_cluster.py` +- `backend/igny8_core/ai/functions/generate_ideas.py` +- `backend/igny8_core/ai/functions/generate_content.py` +- `backend/igny8_core/ai/functions/generate_image_prompts.py` +- `backend/igny8_core/ai/functions/generate_images.py` + +**Changes per function:** + +| Function | Phase | Accurate Message Template | +|----------|-------|---------------------------| +| auto_cluster | PREP | "Loading {count} keywords for clustering" | +| auto_cluster | AI_CALL | "Analyzing keyword relationships ({processed}/{total})" | +| auto_cluster | SAVE | "Creating {cluster_count} clusters" | +| generate_ideas | PREP | "Preparing {count} cluster(s) for idea generation" | +| generate_ideas | AI_CALL | "Generating ideas for cluster: {cluster_name}" | +| generate_ideas | SAVE | "Created {idea_count} content ideas" | +| generate_content | PREP | "Preparing task: {task_title}" | +| generate_content | AI_CALL | "Writing {word_count}-word article" | +| generate_content | SAVE | "Saving article: {title}" | +| generate_image_prompts | PREP | "Mapping content for {count} image prompts" | +| generate_image_prompts | AI_CALL | "Writing featured image prompt" | +| generate_image_prompts | PARSE | "Writing {count} in-article image prompts" | +| generate_images | PREP | "Preparing {count} images for generation" | +| generate_images | AI_CALL | "Generating image {current}/{total}: {prompt}" | + +**Implementation:** Use `self.tracker.update_step()` with dynamic values from actual data + +--- + +### B. Frontend Progress Display Improvements + +#### 1. Update ProgressModal Messages + +**File:** `frontend/src/components/common/ProgressModal.tsx` + +**Function:** `getSuccessMessage()` + +**Current Issues:** +- Generic success messages +- Count extraction from logs is unreliable +- Doesn't distinguish between function types clearly + +**Required Changes:** +| Function | Current Success Message | Improved Success Message | +|----------|------------------------|--------------------------| +| Clustering | "Clustering complete" | "45 keywords mapped into 8 clusters" | +| Ideas | "Content ideas created successfully" | "Generated 12 content ideas for 3 clusters" | +| Content | "Article drafted successfully" | "1,245-word article drafted successfully" | +| Image Prompts | Generic text | "1 Featured Image + 4 In-article Image Prompts ready" | +| Images | "Images generated successfully" | "5 images generated successfully" | + +**Implementation:** Extract counts from backend `details` field, not from message strings + +--- + +#### 2. Improve Step Mapping Logic + +**File:** `frontend/src/hooks/useProgressModal.ts` + +**Function:** `getStepInfo()` + +**Problems:** +- Complex regex-based count extraction from messages +- Percentage calculation doesn't match backend +- Messages manually constructed instead of using backend messages + +**Solution:** +- **Rely on backend `details` field** for counts: + ```typescript + const { items_total, items_processed, current_item } = progress.details; + ``` +- Use backend message directly when it contains proper counts +- Only enhance message format, don't reconstruct it + +**Example Refactor:** +```typescript +// OLD: Complex regex extraction +const keywordCount = extractNumber(/(\d+)\s+keyword/i, message); + +// NEW: Use backend details +const percentage = details.percentage; +const message = details.message; // Already has counts +const itemName = details.current_item || ''; +``` + +--- + +#### 3. Real-Time Updates Per Page + +**Pages to update:** Keywords, Ideas, Tasks, Content + +**Current behavior:** Modal closes, then data reloads + +**Improved behavior:** +- Show inline status badge in table rows during processing +- Update table row in real-time as status changes +- Auto-reload specific rows instead of entire table +- Show notification when task completes in background + +**Implementation approach:** +- Subscribe to task updates while modal is open +- Update table state with new status +- Add `status` badge column to tables showing: + - 🔄 Processing (animated) + - ✅ Completed + - ❌ Failed + - ⏸️ Paused + +--- + +### C. Automation Wizard Progress Fixes + +**File:** `frontend/src/pages/Automation/AutomationWizard.tsx` + +**Issues identified:** +1. Wrong queue items displayed +2. Missing queue items +3. Progress bar inaccurate +4. Stage counts buggy + +**Required fixes:** + +#### 1. Stage Progress Cards + +**Current location:** Stage cards display in wizard + +**Problems:** +- Counts mismatch between backend and frontend +- "In Queue" vs "Processed" numbers don't add up +- Real-time updates missing + +**Fix:** +- Pull stage results from `AutomationRun.stage_X_result` JSON fields +- Display accurate counts: + ``` + Stage 1: Clustering + - Keywords Loaded: 45 + - Clusters Created: 8 + - Status: Completed + ``` + +#### 2. Queue Item Display + +**Fix queue item tracking:** +- Each stage should show items being processed +- Items should appear in queue before processing +- Items should move to "Completed" list after processing +- Failed items should show in "Failed" section with retry option + +**Data structure per stage:** +```json +{ + "pending": ["item1", "item2"], + "processing": ["item3"], + "completed": ["item4", "item5"], + "failed": [{"item": "item6", "error": "..."}] +} +``` + +#### 3. Progress Bar Calculation + +**Fix:** +- Base percentage on actual completed stages +- Within-stage percentage based on items processed +- Formula: `(completed_stages * 100 / 7) + (current_stage_progress / 7)` + +--- + +### D. Error State Handling + +#### Backend Error Logging + +**Location:** `backend/igny8_core/ai/engine.py` + +**Current:** Errors logged but status not always propagated + +**Improvement:** +- Always set task state to `FAILURE` on error +- Include detailed error in state metadata +- Log error with context (function, input data, trace) + +**Error payload structure:** +```json +{ + "status": "error", + "error_type": "InsufficientCreditsError", + "error_message": "Insufficient credits. Required: 45, Available: 20", + "phase": "INIT", + "timestamp": "2025-12-11T10:30:00Z" +} +``` + +#### Frontend Error Display + +**Files:** All progress modals + +**Current:** Generic "Task failed" message + +**Improvement:** +- Show specific error type +- Display user-friendly error message +- Provide actionable next steps +- Show "Retry" button for recoverable errors + +**Error message mapping:** + +| Error Type | User Message | Action | +|------------|--------------|--------| +| InsufficientCreditsError | "Not enough credits. You need X credits." | "Purchase Credits" button | +| ValidationError | "Invalid data: {details}" | "Fix Data" button | +| AIProviderError | "AI service temporarily unavailable" | "Retry" button | +| TimeoutError | "Request timed out. Try again." | "Retry" button | + +--- + +## QA Testing Checklist + +### Manual AI Function Tests + +Test each function on each page: + +| Page | Function | Test Case | Expected Result | +|------|----------|-----------|-----------------| +| Keywords | Clustering | Select 50 keywords, run clustering | Modal shows "Clustering 50 keywords", creates clusters, shows "50 keywords mapped into X clusters" | +| Ideas | Generate Ideas | Select 3 clusters, generate ideas | Modal shows "Generating ideas for 3 clusters", creates ideas, shows "Generated X ideas for 3 clusters" | +| Tasks | Generate Content | Select 1 task, generate content | Modal shows "Writing {word_count}-word article", creates content, shows "{word_count}-word article drafted" | +| Tasks | Generate Image Prompts | Select 1 content, generate prompts | Modal shows "Mapping content for X prompts", shows "1 Featured + X In-article prompts ready" | +| Images | Generate Images | Select content with prompts, generate | ImageQueueModal shows each image generating, completes all | + +### Edge Case Tests + +| Scenario | Expected Behavior | +|----------|-------------------| +| Task cancelled mid-process | Status updates to "cancelled", modal shows cancellation message | +| Task fails due to insufficient credits | Error modal with credit purchase link | +| Task fails due to validation error | Error modal with specific validation issue | +| Multiple tasks running simultaneously | Each has independent progress tracking | +| Network disconnection during task | Polling resumes when reconnected, catches up with status | +| Task completes while modal is closed | Table updates status automatically, notification shown | + +### Automation Wizard Tests + +| Stage | Test Case | Expected Result | +|-------|-----------|-----------------| +| Stage 1 | Run clustering on 100 keywords | Shows batch progress, creates clusters, displays accurate counts | +| Stage 2 | Generate ideas for 10 clusters | Shows cluster-by-cluster progress, creates ideas | +| Stage 3 | Convert ideas to tasks | Shows conversion progress, creates tasks | +| Stage 4 | Generate content for tasks | Shows content generation per task | +| Stage 5 | Generate image prompts | Shows prompt creation progress | +| Stage 6 | Generate images | Shows image generation progress | +| Stage 7 | Final review | Shows completed content ready for publishing | +| Pause/Resume | Pause at stage 3, resume | Continues from where it left off | +| Cancel | Cancel at stage 4 | Stops processing, shows cancelled status | + +--- + +## Implementation Priority + +### Phase 1: Backend Fixes (Week 1) +1. ✅ Create unified status enums +2. ✅ Add `generation_status` fields to models +3. ✅ Update AI function progress messages with accurate counts +4. ✅ Enhance tracker to include item counts and current item + +### Phase 2: Frontend Display (Week 1-2) +1. ✅ Update ProgressModal success messages +2. ✅ Refactor step mapping logic to use backend details +3. ✅ Add real-time status badges to table rows +4. ✅ Improve error message display + +### Phase 3: Automation Wizard (Week 2) +1. ✅ Fix stage progress cards +2. ✅ Fix queue item tracking +3. ✅ Fix progress bar calculation +4. ✅ Add pause/resume/cancel controls + +### Phase 4: QA & Polish (Week 2-3) +1. ✅ Run all manual tests +2. ✅ Run all edge case tests +3. ✅ Run automation wizard tests +4. ✅ Fix any discovered issues +5. ✅ Performance optimization + +--- + +## Success Metrics + +- ✅ All progress modals show accurate counts and messages +- ✅ Progress percentages match actual work completed +- ✅ Automation wizard shows correct queue states +- ✅ Error messages are specific and actionable +- ✅ Real-time updates work consistently +- ✅ No user-reported confusion about task status +- ✅ All QA test cases pass + +--- + +## Related Files Reference + +### Backend +- `backend/igny8_core/business/planning/models.py` - Keywords, Clusters, ContentIdeas models +- `backend/igny8_core/business/content/models.py` - Tasks, Content models +- `backend/igny8_core/business/automation/models.py` - AutomationRun model +- `backend/igny8_core/ai/tracker.py` - ProgressTracker, StepTracker +- `backend/igny8_core/ai/engine.py` - AIEngine orchestration +- `backend/igny8_core/ai/functions/*.py` - All AI function implementations + +### Frontend +- `frontend/src/hooks/useProgressModal.ts` - Progress modal hook +- `frontend/src/components/common/ProgressModal.tsx` - Progress modal component +- `frontend/src/components/common/ImageQueueModal.tsx` - Image generation modal +- `frontend/src/pages/Planner/Keywords.tsx` - Keywords page with clustering +- `frontend/src/pages/Planner/Ideas.tsx` - Ideas page with idea generation +- `frontend/src/pages/Writer/Tasks.tsx` - Tasks page with content generation +- `frontend/src/pages/Writer/Content.tsx` - Content page +- `frontend/src/pages/Writer/Images.tsx` - Images page +- `frontend/src/pages/Automation/AutomationWizard.tsx` - Automation wizard + +--- + +## Notes + +- Consider WebSocket implementation for production to reduce polling overhead +- Status badges in tables should be subtle and not distract from main content +- Error messages should prioritize user understanding over technical accuracy +- Progress messages should be encouraging and informative +- Real-time updates should not cause UI flickering or performance issues diff --git a/docs/PRE-LAUNCH/ITEM-2-CREDITS-BILLING-PRICING.md b/docs/PRE-LAUNCH/ITEM-2-CREDITS-BILLING-PRICING.md new file mode 100644 index 00000000..03b2ba44 --- /dev/null +++ b/docs/PRE-LAUNCH/ITEM-2-CREDITS-BILLING-PRICING.md @@ -0,0 +1,628 @@ +# Item 2: Credits, Billing, Pricing Logic, and Usage Limits + +**Priority:** Critical +**Target:** Production Launch +**Last Updated:** December 11, 2025 + +--- + +## Overview + +Define and implement a comprehensive credit cost system, plan-based usage limits, and billing logic for all AI operations. This includes setting credit costs per function, establishing plan tiers with limits, and implementing enforcement mechanisms across backend and frontend. + +--- + +## Current Implementation Analysis + +### Credit System Architecture + +**Location:** `backend/igny8_core/business/billing/` + +#### Credit Models + +| Model | Purpose | Key Fields | +|-------|---------|------------| +| **CreditTransaction** | Tracks all credit additions/deductions | `transaction_type`, `amount`, `balance_after`, `description` | +| **CreditUsageLog** | Detailed log per AI operation | `operation_type`, `credits_used`, `cost_usd`, `model_used`, `tokens_input`, `tokens_output` | +| **CreditCostConfig** | Admin-configurable credit costs | `operation_type`, `credits_cost`, `unit`, `display_name` | + +**Credit Transaction Types:** +- `purchase` - Credit purchase +- `subscription` - Monthly subscription renewal +- `refund` - Credit refund +- `deduction` - Usage deduction (AI operations) +- `adjustment` - Manual admin adjustment + +#### Credit Service + +**Location:** `backend/igny8_core/business/billing/services/credit_service.py` + +**Methods:** +- `get_credit_cost(operation_type, amount)` - Calculate cost for operation +- `check_credits(account, operation_type, amount)` - Validate sufficient credits +- `deduct_credits(account, amount, operation_type, ...)` - Deduct and log +- `deduct_credits_for_operation(...)` - Convenience method with auto-calculation + +**Logic:** +1. Checks database `CreditCostConfig` first +2. Falls back to hardcoded `CREDIT_COSTS` constants +3. Applies unit-based calculation (per 100 words, per image, etc.) +4. Validates sufficient balance before deduction +5. Creates both `CreditTransaction` and `CreditUsageLog` records + +--- + +### Current Credit Costs + +**Location:** `backend/igny8_core/business/billing/constants.py` + +| Operation | Current Cost | Unit | Notes | +|-----------|--------------|------|-------| +| `clustering` | 10 credits | per request | Clusters all submitted keywords | +| `idea_generation` | 15 credits | per request | Ideas for one cluster | +| `content_generation` | 1 credit | per 100 words | Word-count based | +| `image_prompt_extraction` | 2 credits | per content | Extract prompts from content | +| `image_generation` | 5 credits | per image | Generate single image | +| `linking` | 8 credits | per content | Internal linking (NEW) | +| `optimization` | 1 credit | per 200 words | Content optimization (NEW) | +| `site_structure_generation` | 50 credits | per site | Site blueprint (Phase 7) | +| `site_page_generation` | 20 credits | per page | Page generation (Phase 7) | + +**Legacy Aliases:** +- `ideas` → `idea_generation` +- `content` → 3 credits fixed (legacy) +- `images` → `image_generation` +- `reparse` → 1 credit + +**Issues with Current Costs:** +1. **Not optimized for profitability** - Costs may not reflect actual AI provider costs +2. **Arbitrary values** - No clear formula based on model costs, processing time, or value +3. **Inconsistent granularity** - Some per-request, some per-word, some per-item +4. **No differentiation by quality** - Same cost regardless of model quality (GPT-4 vs GPT-3.5) + +--- + +### Plan Model and Limits + +**Location:** `backend/igny8_core/auth/models.py` - `Plan` model + +#### Current Plan Structure + +| Field | Purpose | Current State | +|-------|---------|---------------| +| `name`, `slug` | Plan identification | ✅ Implemented | +| `price`, `billing_cycle` | Pricing | ✅ Monthly/Annual support | +| `included_credits` | Monthly credit allocation | ✅ Implemented | +| `extra_credit_price` | Per-credit overage cost | ✅ Default $0.01 | +| `allow_credit_topup` | Can buy more credits | ✅ Boolean flag | +| `auto_credit_topup_threshold` | Auto-buy trigger | ✅ Optional | +| `auto_credit_topup_amount` | Auto-buy amount | ✅ Optional | +| `max_users` | Users per account | ✅ Implemented | +| `max_sites` | Sites per account | ✅ Implemented | +| `max_industries` | Industries/sectors limit | ✅ Optional | +| `max_author_profiles` | Writing styles limit | ✅ Default 5 | + +**What's MISSING:** +- ❌ Max keywords limit +- ❌ Max clusters limit +- ❌ Max ideas limit +- ❌ Max content pieces limit +- ❌ Max images limit +- ❌ Max tasks in queue limit +- ❌ Daily/monthly usage caps (beyond credits) +- ❌ Per-user vs per-account limits distinction + +--- + +### Account Credit Balance + +**Location:** `backend/igny8_core/auth/models.py` - `Account` model + +**Field:** `credits` (IntegerField with MinValueValidator(0)) + +**Current Behavior:** +- Credits deducted on AI operation completion +- Credit balance checked before operation starts +- `InsufficientCreditsError` raised if balance < required + +**No Implementation For:** +- Credit expiration dates +- Credit rollover rules (monthly vs annual) +- Negative balance prevention (hard stop vs warning) +- Credit reserve for pending operations + +--- + +## Pricing Plan Requirements + +### Recommended Plan Tiers + +Based on industry standards and target market: + +| Plan | Monthly Price | Annual Price | Included Credits | Target User | +|------|---------------|--------------|------------------|-------------| +| **Free** | $0 | $0 | 50 | Trial users, hobbyists | +| **Starter** | $29 | $299 (15% off) | 500 | Solo creators, small blogs | +| **Growth** | $99 | $1,019 (15% off) | 2,000 | Growing sites, agencies | +| **Pro** | $299 | $3,077 (15% off) | 7,500 | Power users, large agencies | +| **Enterprise** | Custom | Custom | Custom | Enterprise clients | + +**Free Plan Considerations:** +- Should be marked `is_internal=True` to hide from public pricing +- Limits should be strict enough to encourage upgrade +- Should not include advanced features (automation, API access) + +--- + +### Usage Limits Per Plan + +**Proposed Limits** (to be finalized): + +| Limit Type | Free | Starter | Growth | Pro | Enterprise | +|------------|------|---------|--------|-----|------------| +| **Monthly Credits** | 50 | 500 | 2,000 | 7,500 | Custom | +| **Max Users** | 1 | 2 | 5 | 15 | Unlimited | +| **Max Sites** | 1 | 3 | 10 | 50 | Unlimited | +| **Max Keywords (saved)** | 100 | 1,000 | 5,000 | 25,000 | Unlimited | +| **Max Clusters** | 20 | 100 | 500 | 2,500 | Unlimited | +| **Max Ideas (saved)** | 50 | 500 | 2,500 | 12,500 | Unlimited | +| **Max Content Pieces** | 25 | 250 | 1,250 | 6,250 | Unlimited | +| **Max Images** | 25 | 250 | 1,250 | 6,250 | Unlimited | +| **Max Queue Size** | 5 | 20 | 50 | 200 | Unlimited | +| **Automation Enabled** | ❌ | ❌ | ✅ | ✅ | ✅ | +| **API Access** | ❌ | ❌ | ✅ | ✅ | ✅ | +| **Priority Support** | ❌ | ❌ | ❌ | ✅ | ✅ | + +**Notes:** +- "Unlimited" means no hard limit, but still subject to fair use policy +- Limits apply per account (across all sites in account) +- Deleted items don't count toward limits (soft-delete system) + +--- + +### Credit Cost Optimization Strategy + +**Goal:** Define credit costs that: +1. Cover AI provider costs + margin +2. Are competitive with market rates +3. Encourage usage without abuse +4. Scale predictably with usage + +#### Recommended Credit Cost Revisions + +**Analysis Required:** +- [ ] Calculate actual AI provider costs per operation (OpenAI, Runware, etc.) +- [ ] Add 30-50% margin for infrastructure, support, and profit +- [ ] Compare with competitor pricing (Jasper, Copy.ai, Writesonic) +- [ ] Test with sample use cases to ensure plan value + +**Proposed Adjustments** (pending analysis): + +| Operation | Current | Proposed | Reasoning | +|-----------|---------|----------|-----------| +| Clustering | 10 | **8** | Lower barrier for discovery phase | +| Idea Generation | 15 | **12** | Encourage ideation before writing | +| Content (100 words) | 1 | **1.5** | Reflect actual GPT-4 costs | +| Image Prompts | 2 | **3** | More complex extraction logic | +| Image Generation | 5 | **6** | Runware/DALL-E costs increasing | +| Optimization | 1 per 200 words | **0.5 per 100 words** | Encourage optimization usage | + +**Variable Costs by Model Quality:** +- **Option:** Charge more for GPT-4 vs GPT-3.5, DALL-E 3 vs DALL-E 2 +- **Implementation:** Add `model_tier` multiplier in `get_credit_cost()` + +--- + +## Required Implementation + +### A. Expand Plan Model with Usage Limits + +**File:** `backend/igny8_core/auth/models.py` - `Plan` model + +**Add Fields:** + +```python +# Content Creation Limits (NULL = unlimited) +max_keywords = models.IntegerField( + null=True, blank=True, + validators=[MinValueValidator(1)], + help_text="Maximum keywords saved per account" +) +max_clusters = models.IntegerField( + null=True, blank=True, + validators=[MinValueValidator(1)], + help_text="Maximum clusters per account" +) +max_ideas = models.IntegerField( + null=True, blank=True, + validators=[MinValueValidator(1)], + help_text="Maximum content ideas saved per account" +) +max_content = models.IntegerField( + null=True, blank=True, + validators=[MinValueValidator(1)], + help_text="Maximum content pieces per account" +) +max_images = models.IntegerField( + null=True, blank=True, + validators=[MinValueValidator(1)], + help_text="Maximum images per account" +) + +# Queue and Rate Limits +max_queue_size = models.IntegerField( + default=10, + validators=[MinValueValidator(1)], + help_text="Maximum concurrent items in queue" +) +max_daily_ai_requests = models.IntegerField( + null=True, blank=True, + validators=[MinValueValidator(1)], + help_text="Maximum AI requests per day (prevents abuse)" +) +max_monthly_content_generated = models.IntegerField( + null=True, blank=True, + validators=[MinValueValidator(1)], + help_text="Maximum content pieces generated per month" +) + +# Feature Access Flags +allow_automation = models.BooleanField( + default=False, + help_text="Enable automation wizard" +) +allow_api_access = models.BooleanField( + default=False, + help_text="Enable API access" +) +allow_bulk_operations = models.BooleanField( + default=True, + help_text="Enable bulk actions (delete, export, etc.)" +) +``` + +**Migration:** Create Django migration to add these fields with default NULL values + +--- + +### B. Create Limit Enforcement Service + +**File:** `backend/igny8_core/business/billing/services/limit_service.py` (NEW) + +**Service Class:** `LimitService` + +**Methods to Implement:** + +| Method | Purpose | Returns | +|--------|---------|---------| +| `check_keyword_limit(account)` | Check if can add more keywords | `bool` or raises `LimitExceededError` | +| `check_cluster_limit(account)` | Check if can add more clusters | `bool` or raises `LimitExceededError` | +| `check_idea_limit(account)` | Check if can add more ideas | `bool` or raises `LimitExceededError` | +| `check_content_limit(account)` | Check if can add more content | `bool` or raises `LimitExceededError` | +| `check_image_limit(account)` | Check if can add more images | `bool` or raises `LimitExceededError` | +| `check_queue_limit(account)` | Check queue capacity | `bool` or raises `LimitExceededError` | +| `check_daily_request_limit(account)` | Check daily AI request quota | `bool` or raises `LimitExceededError` | +| `get_usage_stats(account)` | Get current usage counts | `dict` with all counters | +| `get_limit_stats(account)` | Get limits and remaining capacity | `dict` with limits | + +**Implementation Logic:** + +```python +def check_keyword_limit(account): + plan = account.plan + if plan.max_keywords is None: + return True # Unlimited + + current_count = Keywords.objects.filter( + account=account, + deleted_at__isnull=True # Exclude soft-deleted + ).count() + + if current_count >= plan.max_keywords: + raise LimitExceededError( + f"Keyword limit reached ({plan.max_keywords}). Upgrade your plan." + ) + + return True +``` + +**Exception:** `LimitExceededError` (inherit from `BillingException`) + +--- + +### C. Integrate Limit Checks in API Views + +**Files to Update:** + +- `backend/igny8_core/modules/planner/views.py` - KeywordsViewSet +- `backend/igny8_core/modules/planner/views.py` - ClustersViewSet +- `backend/igny8_core/modules/planner/views.py` - ContentIdeasViewSet +- `backend/igny8_core/modules/writer/views.py` - TasksViewSet +- `backend/igny8_core/modules/writer/views.py` - ContentViewSet + +**Integration Points:** + +| ViewSet | Action | Check to Add | +|---------|--------|--------------| +| KeywordsViewSet | `create()` | `LimitService.check_keyword_limit(account)` | +| KeywordsViewSet | `bulk_create()` | Check limit with proposed count | +| ClustersViewSet | `create()` | `LimitService.check_cluster_limit(account)` | +| ContentIdeasViewSet | `create()` | `LimitService.check_idea_limit(account)` | +| TasksViewSet | `create()` | `LimitService.check_content_limit(account)` + `check_queue_limit()` | +| ContentViewSet | `create()` | `LimitService.check_content_limit(account)` | + +**Example Integration:** + +```python +def create(self, request, *args, **kwargs): + account = request.user.account + + # Check limit before creating + try: + LimitService.check_keyword_limit(account) + except LimitExceededError as e: + return Response({ + 'error': str(e), + 'error_code': 'LIMIT_EXCEEDED', + 'upgrade_url': '/pricing' + }, status=403) + + # Proceed with creation + return super().create(request, *args, **kwargs) +``` + +--- + +### D. Add Usage Tracking and Counter Cache + +**Optimization:** Instead of counting records on every request, cache counts + +**Implementation Options:** + +#### Option 1: Add Counter Fields to Account Model + +```python +# Add to Account model +keyword_count = models.IntegerField(default=0) +cluster_count = models.IntegerField(default=0) +idea_count = models.IntegerField(default=0) +content_count = models.IntegerField(default=0) +image_count = models.IntegerField(default=0) +``` + +**Update counters in signals:** +- `post_save` signal: increment counter +- `post_delete` signal: decrement counter +- Periodic reconciliation task to fix drift + +#### Option 2: Cache Usage Stats (Recommended) + +Use Django cache with 5-minute TTL: + +```python +def get_cached_usage_stats(account): + cache_key = f'usage_stats_{account.id}' + stats = cache.get(cache_key) + + if stats is None: + stats = { + 'keywords': Keywords.objects.filter(account=account, deleted_at__isnull=True).count(), + 'clusters': Clusters.objects.filter(account=account, deleted_at__isnull=True).count(), + # ... etc + } + cache.set(cache_key, stats, 300) # 5 minutes + + return stats +``` + +**Invalidate cache on:** +- Create operations +- Delete operations +- Soft-delete operations + +--- + +### E. Frontend Limit Display + +#### 1. Usage Dashboard Widget + +**Location:** `frontend/src/components/dashboard/UsageLimitsWidget.tsx` (NEW) + +**Display:** +- Current usage vs limit for each resource +- Progress bars with color coding: + - Green: < 70% used + - Yellow: 70-90% used + - Red: > 90% used +- "Upgrade Plan" button when approaching limits + +**Example UI:** + +``` +Usage & Limits +━━━━━━━━━━━━━━━━━━━━━━━━━━━ +Keywords: 750 / 1,000 ████████░░ 75% +Clusters: 45 / 100 ████░░░░░░ 45% +Ideas: 380 / 500 ███████░░░ 76% +Content: 120 / 250 ████░░░░░░ 48% + +[Upgrade Plan] +``` + +#### 2. Inline Warnings + +**Show warnings when approaching limits:** + +- At 80%: Yellow badge "Approaching limit" +- At 90%: Orange warning "Near limit - Upgrade recommended" +- At 100%: Red error "Limit reached - Upgrade required" + +**Display in:** +- Header metrics +- Page headers +- Before bulk operations +- In forms (disable submit if limit reached) + +#### 3. Create/Import Dialogs + +**Add limit check before showing form:** + +```typescript +const handleCreateKeyword = () => { + const stats = usageStats; // from API + const limit = account.plan.max_keywords; + + if (limit && stats.keywords >= limit) { + toast.error('Keyword limit reached. Upgrade your plan.'); + navigate('/settings/billing'); + return; + } + + setShowCreateModal(true); +}; +``` + +#### 4. Upgrade Prompts + +**When limit error occurs:** +- Show modal with: + - Current plan + - Current limit + - Recommended plan + - Benefits of upgrading + - "Upgrade Now" CTA + +--- + +### F. Credit Cost Configuration UI (Admin) + +**Location:** Django Admin or custom Admin Panel page + +**Feature:** Allow superusers to edit credit costs without code changes + +**Admin Interface:** +- List all operations with current costs +- Edit cost, unit, and display name +- Track change history (previous_cost field) +- Enable/disable operations +- Preview impact on sample use cases + +**Models Used:** +- `CreditCostConfig` - Admin-editable costs +- Falls back to `CREDIT_COSTS` constants if not configured + +--- + +## Testing Requirements + +### Limit Enforcement Tests + +| Test Case | Expected Result | +|-----------|-----------------| +| Create keyword at limit | Error: "Keyword limit reached" | +| Create keyword below limit | Success | +| Create 10 keywords via bulk import at limit | Error with count blocked | +| Delete keyword then create | Success (count decremented) | +| Soft-delete keyword then restore | Counts update correctly | +| Upgrade plan mid-session | New limits apply immediately | + +### Credit Deduction Tests + +| Test Case | Expected Result | +|-----------|-----------------| +| Generate content with sufficient credits | Content created, credits deducted | +| Generate content with insufficient credits | Error: "Insufficient credits" | +| Generate content at exact credit balance | Success, balance = 0 | +| Generate multiple items in queue | Each deducts credits sequentially | +| Credit deduction failure mid-operation | Transaction rolled back, no partial deduction | + +### Plan Limit Tests + +| Plan | Test Case | Expected Result | +|------|-----------|-----------------| +| Free | Create 101st keyword (limit: 100) | Blocked | +| Starter | Create 6 queue items (limit: 5) | Blocked | +| Growth | Enable automation | Success (has access) | +| Pro | Create unlimited keywords | Success (no limit) | +| Enterprise | All operations | No limits enforced | + +--- + +## Pricing Page Updates + +**Location:** `frontend/src/pages/marketing/Pricing.tsx` + +### Required Elements + +1. **Plan Comparison Table** + - All tiers side-by-side + - Feature checkmarks + - Highlight "Most Popular" plan + - Monthly/Annual toggle with savings badge + +2. **Usage Limits Display** + - Show key limits per plan + - Use "Unlimited" label for null limits + - Tooltip explanations for complex limits + +3. **Credit System Explanation** + - What credits are + - How they're consumed + - How to buy more + - Credit rollover rules + +4. **FAQ Section** + - "What happens when I run out of credits?" + - "Can I change plans mid-month?" + - "Do unused credits roll over?" + - "What's included in Enterprise?" + +5. **Calculator Widget** (Optional) + - Estimate monthly usage + - Recommend plan based on needs + - Show credit consumption breakdown + +--- + +## Success Metrics + +- ✅ All AI operations enforce credit checks +- ✅ All create operations enforce limit checks +- ✅ Credit costs reflect actual provider costs + margin +- ✅ Plans are competitively priced +- ✅ Usage dashboard shows accurate counts +- ✅ Limit warnings prevent user frustration +- ✅ Upgrade flow is clear and frictionless +- ✅ Admin can adjust costs without code changes +- ✅ All tests pass + +--- + +## Related Files Reference + +### Backend +- `backend/igny8_core/auth/models.py` - Account, Plan models +- `backend/igny8_core/business/billing/models.py` - Credit models +- `backend/igny8_core/business/billing/constants.py` - Credit costs +- `backend/igny8_core/business/billing/services/credit_service.py` - Credit logic +- `backend/igny8_core/business/billing/services/limit_service.py` - **NEW** Limit enforcement +- `backend/igny8_core/modules/planner/views.py` - Planner API views +- `backend/igny8_core/modules/writer/views.py` - Writer API views + +### Frontend +- `frontend/src/components/dashboard/UsageLimitsWidget.tsx` - **NEW** Usage display +- `frontend/src/pages/marketing/Pricing.tsx` - Pricing page +- `frontend/src/pages/Planner/*.tsx` - Planner pages (add limit checks) +- `frontend/src/pages/Writer/*.tsx` - Writer pages (add limit checks) +- `frontend/src/services/api.ts` - API service (handle limit errors) + +--- + +## Notes + +- Limits should be enforced at API level, not just UI level +- Consider "soft limits" with warnings vs "hard limits" with blocks +- Credit expiration and rollover rules need business decision +- Enterprise pricing needs custom quote system +- Monitor actual usage patterns to optimize costs and limits +- A/B test different pricing tiers to maximize conversion diff --git a/docs/PRE-LAUNCH/ITEM-3-PROMPT-OPTIMIZATION.md b/docs/PRE-LAUNCH/ITEM-3-PROMPT-OPTIMIZATION.md new file mode 100644 index 00000000..e303c221 --- /dev/null +++ b/docs/PRE-LAUNCH/ITEM-3-PROMPT-OPTIMIZATION.md @@ -0,0 +1,713 @@ +# Item 3: Prompt Improvement and Model Optimization + +**Priority:** High +**Target:** Production Launch +**Last Updated:** December 11, 2025 + +--- + +## Overview + +Redesign and optimize all AI prompts for clustering, idea generation, content generation, and image prompt extraction to achieve: +- Extreme accuracy and consistent outputs +- Faster processing with optimized token usage +- Correct word count adherence (500, 1000, 1500 words) +- Improved clustering quality and idea relevance +- Better image prompt clarity and relevance + +--- + +## Current Prompt System Architecture + +### Prompt Registry + +**Location:** `backend/igny8_core/ai/prompts.py` + +**Class:** `PromptRegistry` + +**Hierarchy** (resolution order): +1. Task-level `prompt_override` (if exists on specific task) +2. Database prompt from `AIPrompt` model (account-specific) +3. Default fallback from `PromptRegistry.DEFAULT_PROMPTS` + +**Storage:** +- Default prompts: Hardcoded in `prompts.py` +- Account overrides: `system_aiprompt` database table +- Task overrides: `prompt_override` field on task object + +--- + +## Current Prompts Analysis + +### 1. Clustering Prompt + +**Function:** `auto_cluster` +**File:** `backend/igny8_core/ai/functions/auto_cluster.py` +**Prompt Key:** `'clustering'` + +#### Current Prompt Structure + +**Approach:** Semantic strategist + intent-driven clustering + +**Key Instructions:** +- Return single JSON with "clusters" array +- Each cluster: name, description, keywords[] +- Multi-dimensional grouping (intent, use-case, function, persona, context) +- Model real search behavior and user journeys +- Avoid superficial groupings and duplicates +- 3-10 keywords per cluster + +**Strengths:** +✅ Clear JSON output format +✅ Detailed grouping logic with dimensions +✅ Emphasis on semantic strength over keyword matching +✅ User journey modeling (Problem → Solution, General → Specific) + +**Issues:** +❌ Very long prompt (~400+ tokens) - may confuse model +❌ No examples provided - model must guess formatting +❌ Doesn't specify what to do with outliers explicitly +❌ No guidance on cluster count (outputs variable) +❌ Description length not constrained + +**Real-World Performance Issues:** +- Sometimes creates too many small clusters (1-2 keywords each) +- Inconsistent cluster naming convention +- Descriptions sometimes generic ("Keywords related to...") + +--- + +### 2. Idea Generation Prompt + +**Function:** `generate_ideas` +**File:** `backend/igny8_core/ai/functions/generate_ideas.py` +**Prompt Key:** `'ideas'` + +#### Current Prompt Structure + +**Approach:** SEO-optimized content ideas + outlines + +**Key Instructions:** +- Input: Clusters + Keywords +- Output: JSON "ideas" array +- 1 cluster_hub + 2-4 supporting ideas per cluster +- Fields: title, description, content_type, content_structure, cluster_id, estimated_word_count, covered_keywords +- Outline format: intro (hook + 2 paragraphs), 5-8 H2 sections with 2-3 H3s each +- Content mixing: paragraphs, lists, tables, blockquotes +- No bullets/lists at start +- Professional tone, no generic phrasing + +**Strengths:** +✅ Detailed outline structure +✅ Content mixing guidance (lists, tables, blockquotes) +✅ Clear JSON format +✅ Tone guidelines + +**Issues:** +❌ Very complex prompt (600+ tokens) +❌ Outline format too prescriptive (might limit creativity) +❌ No examples provided +❌ Estimated word count often inaccurate (too high or too low) +❌ "hook" guidance unclear (what makes a good hook?) +❌ Content structure validation not enforced + +**Real-World Performance Issues:** +- Generated ideas sometimes too similar within cluster +- Outlines don't always respect structure types (e.g., "review" vs "guide") +- covered_keywords field sometimes empty or incorrect +- cluster_hub vs supporting ideas distinction unclear + +--- + +### 3. Content Generation Prompt + +**Function:** `generate_content` +**File:** `backend/igny8_core/ai/functions/generate_content.py` +**Prompt Key:** `'content_generation'` + +#### Current Prompt Structure + +**Approach:** Editorial content strategist + +**Key Instructions:** +- Output: JSON {title, content (HTML)} +- Introduction: 1 italic hook (30-40 words) + 2 paragraphs (50-60 words each), no headings +- H2 sections: 5-8 total, 250-300 words each +- Section format: 2 narrative paragraphs → list/table → optional closing paragraph → 2-3 subsections +- Vary list/table types +- Never start section with list/table +- Tone: professional, no passive voice, no generic intros +- Keyword usage: natural in title, intro, headings + +**Strengths:** +✅ Detailed structure guidance +✅ Strong tone/style rules +✅ HTML output format +✅ Keyword integration guidance + +**Issues:** +❌ **Word count not mentioned in prompt** - critical flaw +❌ No guidance on 500 vs 1000 vs 1500 word versions +❌ Hook word count (30-40) + paragraph counts (50-60 × 2) don't scale proportionally +❌ Section word count (250-300) doesn't adapt to total target +❌ No example output +❌ Content structure (article vs guide vs review) not clearly differentiated +❌ Table column guidance missing (what columns? how many?) + +**Real-World Performance Issues:** +- **Output length wildly inconsistent** (generates 800 words when asked for 1500) +- Introductions sometimes have headings despite instructions +- Lists appear at start of sections +- Table structure unclear (random columns) +- Doesn't adapt content density to word count + +--- + +### 4. Image Prompt Extraction + +**Function:** `generate_image_prompts` +**File:** `backend/igny8_core/ai/functions/generate_image_prompts.py` +**Prompt Key:** `'image_prompt_extraction'` + +#### Current Prompt Structure + +**Approach:** Extract visual descriptions from article + +**Key Instructions:** +- Input: article title + content +- Output: JSON {featured_prompt, in_article_prompts[]} +- Extract featured image (main topic) +- Extract up to {max_images} in-article images +- Each prompt detailed for image generation (visual elements, style, mood, composition) + +**Strengths:** +✅ Clear structure +✅ Separates featured vs in-article +✅ Emphasizes detail in descriptions + +**Issues:** +❌ No guidance on what makes a good image prompt +❌ No style/mood specifications +❌ Doesn't specify where in article to place images +❌ No examples +❌ "Detailed enough" is subjective + +**Real-World Performance Issues:** +- Prompts sometimes too generic ("Image of a person using a laptop") +- No context from article content (extracts irrelevant visuals) +- Featured image prompt sometimes identical to in-article prompt +- No guidance on image diversity (all similar) + +--- + +### 5. Image Generation Template + +**Prompt Key:** `'image_prompt_template'` + +#### Current Template + +**Approach:** Template-based prompt assembly + +**Format:** +``` +Create a high-quality {image_type} image... "{post_title}"... {image_prompt}... +Focus on realistic, well-composed scene... lifestyle/editorial web content... +Avoid text, watermarks, logos... **not blurry.** +``` + +**Issues:** +❌ {image_type} not always populated +❌ "high-quality" and "not blurry" redundant/unclear +❌ No style guidance (photographic, illustration, 3D, etc.) +❌ No aspect ratio specification + +--- + +## Required Improvements + +### A. Clustering Prompt Redesign + +#### Goals +- Reduce prompt length by 30-40% +- Add 2-3 concrete examples +- Enforce consistent cluster count (5-15 clusters ideal) +- Standardize cluster naming (title case, descriptive) +- Limit description to 20-30 words + +#### Proposed Structure + +**Section 1: Role & Task** (50 tokens) +- Clear, concise role definition +- Task: group keywords into intent-driven clusters + +**Section 2: Output Format with Example** (100 tokens) +- JSON structure +- Show 1 complete example cluster +- Specify exact field requirements + +**Section 3: Clustering Rules** (150 tokens) +- List 5-7 key rules (bullet format) +- Keyword-first approach +- Intent dimensions (brief) +- Quality thresholds (3-10 keywords per cluster) +- No duplicates + +**Section 4: Quality Checklist** (50 tokens) +- Checklist of 4-5 validation points +- Model self-validates before output + +**Total:** ~350 tokens (vs current ~420) + +#### Example Output Format to Include + +```json +{ + "clusters": [ + { + "name": "Organic Bedding Benefits", + "description": "Health, eco-friendly, and comfort aspects of organic cotton bedding materials", + "keywords": ["organic sheets", "eco-friendly bedding", "chemical-free cotton", "hypoallergenic sheets", "sustainable bedding"] + } + ] +} +``` + +--- + +### B. Idea Generation Prompt Redesign + +#### Goals +- Simplify outline structure (less prescriptive) +- Add examples of cluster_hub vs supporting ideas +- Better covered_keywords extraction +- Adaptive word count estimation +- Content structure differentiation + +#### Proposed Structure + +**Section 1: Role & Objective** (40 tokens) +- SEO content strategist +- Task: generate content ideas from clusters + +**Section 2: Output Format with Examples** (150 tokens) +- Show 1 cluster_hub example +- Show 1 supporting idea example +- Highlight key differences + +**Section 3: Idea Generation Rules** (100 tokens) +- 1 cluster_hub (comprehensive, authoritative) +- 2-4 supporting ideas (specific angles) +- Word count: 1500-2200 for hubs, 1000-1500 for supporting +- covered_keywords: extract from cluster keywords + +**Section 4: Outline Guidance** (100 tokens) +- Simplified: Intro + 5-8 sections + Conclusion +- Section types by content_structure: + - article: narrative + data + - guide: step-by-step + tips + - review: pros/cons + comparison + - listicle: numbered + categories + - comparison: side-by-side + verdict + +**Total:** ~390 tokens (vs current ~610) + +--- + +### C. Content Generation Prompt Redesign + +**Most Critical Improvement:** Word Count Adherence + +#### Goals +- **Primary:** Generate exact word count (±5% tolerance) +- Scale structure proportionally to word count +- Differentiate content structures clearly +- Improve HTML quality and consistency +- Better keyword integration + +#### Proposed Adaptive Word Count System + +**Word Count Targets:** +- 500 words: Short-form (5 sections × 80 words + intro/outro 60 words) +- 1000 words: Standard (6 sections × 140 words + intro/outro 120 words) +- 1500 words: Long-form (7 sections × 180 words + intro/outro 180 words) + +**Prompt Variable Replacement:** + +Before sending to AI, calculate: +- `{TARGET_WORD_COUNT}` - from task.word_count +- `{INTRO_WORDS}` - 60 / 120 / 180 based on target +- `{SECTION_COUNT}` - 5 / 6 / 7 based on target +- `{SECTION_WORDS}` - 80 / 140 / 180 based on target +- `{HOOK_WORDS}` - 25 / 35 / 45 based on target + +#### Proposed Structure + +**Section 1: Role & Objective** (30 tokens) +``` +You are an editorial content writer. Generate a {TARGET_WORD_COUNT}-word article... +``` + +**Section 2: Word Count Requirements** (80 tokens) +``` +CRITICAL: The content must be exactly {TARGET_WORD_COUNT} words (±5% tolerance). + +Structure breakdown: +- Introduction: {INTRO_WORDS} words total + - Hook (italic): {HOOK_WORDS} words + - Paragraphs: 2 × ~{INTRO_WORDS/2} words each +- Main Sections: {SECTION_COUNT} H2 sections + - Each section: {SECTION_WORDS} words +- Conclusion: 60 words + +Word count validation: Count words in final output and adjust if needed. +``` + +**Section 3: Content Flow & HTML** (120 tokens) +- Detailed structure per section +- HTML tag usage (

,

,

,