igny8/part2-dev/PHASE-0-4-FOUNDATION-TO-LINKER-OPTIMIZER.md

PHASES 0-4: FOUNDATION TO LINKER & OPTIMIZER

Complete Implementation Documentation - IMPLEMENTED ✅

Status: All Phases Complete
Last Updated: 2025-01-XX
Implementation Period: Foundation through Linker & Optimizer

---

TABLE OF CONTENTS

1. Executive Summary
2. Phase 0: Foundation & Credit System
3. Phase 1: Service Layer Refactoring
4. Phase 2: Automation System
5. Phase 3: Site Builder
6. Phase 4: Linker & Optimizer
7. Cross-Phase Integration
8. Database Migrations Summary
9. Testing Strategy

---

EXECUTIVE SUMMARY

This document covers the complete implementation of Phases 0-4, establishing the foundation for the IGNY8 platform:

• Phase 0: Credit-only system replacing plan limits, module enable/disable functionality
• Phase 1: Service layer architecture with business logic extraction
• Phase 2: Automation rules and scheduled tasks system
• Phase 3: Site Builder with AI-powered structure generation
• Phase 4: Internal linking and content optimization with multiple entry points

All phases are fully implemented and integrated into the production system.

---

PHASE 0: FOUNDATION & CREDIT SYSTEM

Status: ✅ IMPLEMENTED

Purpose

Migrate from plan-based limits to a credit-only model while preserving all existing functionality. Establish module enable/disable capabilities and comprehensive credit tracking.

Key Objectives

• Migrate from plan-based limits to credit-only system
• Implement module enable/disable functionality
• Add credit cost tracking for all operations
• Preserve all existing functionality
• Update frontend to show credits instead of limits

Models

ModuleSettings Model

Purpose: Control module availability per account

Key Fields:

• planner_enabled - Boolean flag for Planner module
• writer_enabled - Boolean flag for Writer module
• automation_enabled - Boolean flag for Automation module
• site_builder_enabled - Boolean flag for Site Builder module
• linker_enabled - Boolean flag for Linker module
• optimizer_enabled - Boolean flag for Optimizer module

Workflow: Accounts can enable/disable modules through settings. Disabled modules don't appear in UI and don't load routes.

Plan Model Updates

Purpose: Simplified plan model with only credit allocation

Fields Removed:

• All limit fields (max_keywords, max_clusters, max_content_ideas, daily_content_tasks, monthly_word_count_limit, daily_image_generation_limit, monthly_image_count)

Fields Retained:

• monthly_credits - Credits allocated per month
• support_level - Support tier
• billing_cycle - Billing frequency
• price - Plan price
• features - JSON field for future use

CreditUsageLog Model

Purpose: Track all credit usage with detailed metadata

Key Fields:

• operation_type - Type of operation (clustering, content_generation, linking, optimization, etc.)
• credits_used - Credits consumed
• related_object_type - Related model type
• related_object_id - Related model ID
• metadata - JSON field for operation-specific data

Services

CreditService

Purpose: Centralized credit management and validation

Key Functions:

• check_credits(account, operation_type, amount=None) - Validates sufficient credits before operation
• deduct_credits(account, operation_type, amount=None) - Deducts credits after successful operation
• get_credit_cost(operation_type, amount=None) - Calculates credit cost for operation
• add_credits(account, amount, reason) - Adds credits (for replenishment)

Credit Cost Constants:

• clustering: 10 credits per request
• idea_generation: 15 credits per cluster → ideas request
• content_generation: 1 credit per 100 words
• image_generation: 5 credits per image
• linking: 8 credits per content piece
• optimization: 1 credit per 200 words
• site_structure_generation: 50 credits per site blueprint
• site_page_generation: 20 credits per page

Workflow: All operations check credits before execution, deduct credits after success, and log usage for audit trail.

Workflows

Module Enable/Disable Workflow

1. User accesses Settings → Modules
2. User toggles module enabled/disabled state
3. Frontend checks module status before loading routes
4. Disabled modules filtered from sidebar navigation
5. API validates module status before processing requests

Credit Check Workflow

1. Operation initiated (e.g., content generation)
2. Service calls CreditService.check_credits()
3. System validates account has sufficient credits
4. If insufficient, raises InsufficientCreditsError
5. If sufficient, operation proceeds
6. After success, CreditService.deduct_credits() called
7. Usage logged in CreditUsageLog

Monthly Credit Replenishment Workflow

1. Celery Beat task runs on 1st of each month
2. Task queries all active accounts
3. For each account, adds plan.monthly_credits to balance
4. Logs replenishment in CreditUsageLog

Database Migrations

Migration: igny8_core_auth.0014_remove_plan_operation_limits_phase0

• Purpose: Remove all plan limit fields
• Status: ✅ Applied
• Risk: LOW - Fields were unused, migration removes them safely

Migration: Credit tracking fields already included in billing.0001_initial

• related_object_type, related_object_id, metadata fields exist in CreditUsageLog

Frontend Implementation

Module Settings UI:

• Settings page with toggle switches for each module
• Module status checked before route loading
• Disabled modules hidden from sidebar

Credit Display:

• Credit balance shown prominently in header
• Credit costs displayed per operation
• Usage history filtered by operation type

---

PHASE 1: SERVICE LAYER REFACTORING

Status: ✅ IMPLEMENTED

Purpose

Extract business logic from ViewSets into reusable services, creating a clean separation between API layer and business logic.

Key Objectives

• Create business/ folder structure
• Move models from modules/ to business/
• Extract business logic from ViewSets to services
• Keep ViewSets as thin wrappers
• Preserve all existing API functionality

Architecture

Business Layer Structure

business/
├── content/              # Content business logic
│   ├── models.py        # Content, Tasks, Images
│   └── services/
│       ├── content_generation_service.py
│       ├── content_pipeline_service.py
│       └── content_versioning_service.py
│
├── planning/             # Planning business logic
│   ├── models.py        # Keywords, Clusters, Ideas
│   └── services/
│       ├── clustering_service.py
│       └── ideas_service.py
│
├── billing/             # Billing business logic
│   ├── models.py        # Credits, Transactions
│   └── services/
│       └── credit_service.py
│
└── automation/          # Automation business logic
    ├── models.py
    └── services/
        └── automation_service.py

Services

ContentGenerationService

Purpose: Unified content generation logic

Key Functions:

• generate_content(task, account) - Generate content for a task
• regenerate_content(content_id, account) - Regenerate existing content
• batch_generate_content(task_ids, account) - Generate multiple content pieces

Workflow:

1. Check credits via CreditService.check_credits()
2. Generate content using AI engine
3. Save content to database
4. Deduct credits via CreditService.deduct_credits()
5. Return generated content

ClusteringService

Purpose: Keyword clustering using AI

Key Functions:

• cluster_keywords(keyword_ids, account) - Cluster keywords into groups
• update_cluster(cluster_id, keyword_ids, account) - Update cluster membership

Workflow:

1. Check credits for clustering operation
2. Call AI function to generate clusters
3. Create Cluster records
4. Associate keywords with clusters
5. Deduct credits and log usage

IdeasService

Purpose: Generate content ideas from clusters

Key Functions:

• generate_ideas(cluster_ids, account) - Generate ideas for clusters
• regenerate_ideas(cluster_id, account) - Regenerate ideas for a cluster

Workflow:

1. Check credits for idea generation
2. Call AI function with cluster context
3. Create ContentIdeas records
4. Associate ideas with clusters
5. Deduct credits and log usage

Model Migrations

Content Models: Moved from modules/writer/models.py to business/content/models.py

• Content model
• Tasks model
• Images model

Planning Models: Moved from modules/planner/models.py to business/planning/models.py

• Keywords model
• Clusters model
• ContentIdeas model

Billing Models: Moved from modules/billing/models.py to business/billing/models.py

• CreditTransaction model
• CreditUsageLog model

Migration Strategy: Models kept same app_label to avoid data migration complexity. Only import paths changed.

ViewSet Refactoring

Before: ViewSets contained 50-100+ lines of business logic

After: ViewSets delegate to services:

# Example: ClusterViewSet
@action(detail=False, methods=['post'])
def auto_generate_ideas(self, request):
    cluster_ids = request.data.get('cluster_ids')
    account = request.account
    
    # Delegate to service
    ideas = self.ideas_service.generate_ideas(cluster_ids, account)
    
    # Serialize and return
    serializer = ContentIdeasSerializer(ideas, many=True)
    return Response(serializer.data)

Workflows

Content Generation Workflow

1. User creates task in Writer module
2. User triggers content generation
3. ViewSet receives request, delegates to ContentGenerationService
4. Service checks credits, generates content, deducts credits
5. ViewSet serializes and returns response

Keyword Clustering Workflow

1. User selects keywords in Planner module
2. User triggers clustering
3. ViewSet delegates to ClusteringService
4. Service checks credits, calls AI, creates clusters
5. ViewSet returns cluster data

---

PHASE 2: AUTOMATION SYSTEM

Status: ✅ IMPLEMENTED

Purpose

Implement automation rules and scheduled tasks to enable users to automate repetitive workflows.

Key Objectives

• Create AutomationRule and ScheduledTask models
• Build AutomationService with rule execution engine
• Implement Celery Beat scheduled tasks
• Create automation API endpoints
• Build automation UI (Dashboard, Rules, History)

Models

AutomationRule Model

Purpose: Store automation rule definitions

Key Fields:

• name - Rule name
• description - Rule description
• trigger - Trigger type (schedule, keyword_added, cluster_created, idea_created, content_generated, task_created)
• conditions - JSON field for condition evaluation
• actions - JSON field for actions to execute
• schedule - JSON field for cron-like schedule (for scheduled triggers)
• is_active - Whether rule is active
• max_executions_per_day - Daily execution limit
• credit_limit_per_execution - Credit limit per execution
• last_executed_at - Last execution timestamp
• execution_count_today - Daily execution counter

Workflow: Rules evaluate conditions, execute actions when triggered, respect execution limits and credit constraints.

ScheduledTask Model

Purpose: Track scheduled task executions

Key Fields:

• automation_rule - ForeignKey to AutomationRule
• scheduled_at - Scheduled execution time
• executed_at - Actual execution time
• status - Task status (pending, running, completed, failed, skipped)
• result - JSON field for execution results
• error_message - Error message if failed
• credits_used - Credits consumed

Workflow: Tasks scheduled by Celery Beat, executed by AutomationService, results stored for audit trail.

Services

AutomationService

Purpose: Execute automation rules with condition evaluation and action execution

Key Functions:

• execute_rule(rule, context=None) - Execute an automation rule
• evaluate_conditions(rule, context) - Evaluate rule conditions
• execute_actions(rule, context) - Execute rule actions
• check_execution_limits(rule) - Validate execution limits

Workflow:

1. Check if rule is active
2. Check execution limits (daily limit, credit limit)
3. Evaluate conditions against context
4. If conditions met, execute actions
5. Update rule tracking (last_executed_at, execution_count_today)
6. Create ScheduledTask record
7. Return execution results

RuleEngine

Purpose: Orchestrate rule execution

Key Functions:

• execute_rule(rule, context) - Orchestrate full rule execution
• validate_rule(rule) - Validate rule configuration

ConditionEvaluator

Purpose: Evaluate rule conditions

Supported Operators:

• eq - Equals
• ne - Not equals
• gt - Greater than
• gte - Greater than or equal
• lt - Less than
• lte - Less than or equal
• in - In list
• contains - Contains substring

Example Condition:

{
  "field": "status",
  "operator": "eq",
  "value": "draft"
}

ActionExecutor

Purpose: Execute rule actions

Supported Actions:

• generate_ideas - Generate content ideas from clusters
• generate_content - Generate content from tasks
• cluster_keywords - Cluster keywords
• optimize_content - Optimize content

Example Action:

{
  "type": "generate_ideas",
  "params": {
    "cluster_ids": [1, 2, 3]
  }
}

Celery Beat Tasks

Scheduled Automation Task

Purpose: Execute scheduled automation rules

Schedule: Every 15 minutes

Workflow:

1. Query all active rules with trigger='schedule'
2. For each rule, check if schedule matches current time
3. If matches, execute rule via AutomationService
4. Log execution in ScheduledTask

Monthly Credit Replenishment Task

Purpose: Add monthly credits to all active accounts

Schedule: 1st day of month at midnight

Workflow:

1. Query all active accounts
2. For each account, add plan.monthly_credits to balance
3. Log replenishment in CreditUsageLog with operation_type='monthly_replenishment'

Database Migrations

Migration: automation.0001_initial

• Purpose: Create AutomationRule and ScheduledTask tables
• Status: ✅ Applied
• Tables Created: igny8_automation_rules, igny8_scheduled_tasks

API Endpoints

AutomationRuleViewSet:

• GET /api/v1/automation/rules/ - List rules
• POST /api/v1/automation/rules/ - Create rule
• GET /api/v1/automation/rules/{id}/ - Get rule
• PUT /api/v1/automation/rules/{id}/ - Update rule
• DELETE /api/v1/automation/rules/{id}/ - Delete rule
• POST /api/v1/automation/rules/{id}/execute/ - Manually execute rule
• POST /api/v1/automation/rules/{id}/test/ - Test rule conditions

ScheduledTaskViewSet:

• GET /api/v1/automation/scheduled-tasks/ - List scheduled tasks
• GET /api/v1/automation/scheduled-tasks/{id}/ - Get scheduled task

Frontend Implementation

Automation Dashboard:

• Active rules count
• Recent executions
• Success/failure rates
• Credit usage from automation
• Quick actions (create rule, view history)

Rules Management:

• List all rules
• Create new rule (wizard)
• Edit existing rule
• Enable/disable rule
• Delete rule
• Test rule
• Manual execution

Scheduled Tasks History:

• List scheduled tasks
• Filter by status, rule, date
• View execution results
• View error messages
• Retry failed tasks

---

PHASE 3: SITE BUILDER

Status: ✅ IMPLEMENTED

Purpose

Build Site Builder for creating sites via wizard with AI-powered structure generation, preview canvas, and file management.

Key Objectives

• Create Site Builder wizard for site creation
• Generate site structure using AI
• Build preview canvas for site editing
• Create shared component library
• Support multiple layouts and templates
• Enable file management for site assets

Models

SiteBlueprint Model

Purpose: Store site structure definitions

Key Fields:

• name - Site name
• description - Site description
• config_json - Wizard configuration (business_type, style, objectives)
• structure_json - AI-generated structure (pages, layout, theme)
• status - Status (draft, generating, ready, deployed)
• hosting_type - Hosting platform (igny8_sites, wordpress, shopify, multi)
• version - Blueprint version
• deployed_version - Currently deployed version

Workflow: Blueprint created via wizard, structure generated by AI, pages created from structure, content generated for pages.

PageBlueprint Model

Purpose: Store page definitions within a site

Key Fields:

• site_blueprint - ForeignKey to SiteBlueprint
• slug - Page slug
• title - Page title
• type - Page type (home, about, services, products, blog, contact, custom)
• blocks_json - Page content blocks
• status - Status (draft, generating, ready)
• order - Page order in navigation

Workflow: Pages created from AI-generated structure, content generated via Writer module, pages rendered in preview.

Services

StructureGenerationService

Purpose: Generate site structure using AI

Key Functions:

• generate_structure(site_blueprint, business_brief, objectives, style) - Generate site structure
• _create_page_blueprints(site_blueprint, structure) - Create PageBlueprint records from structure

Workflow:

1. Check credits for structure generation (50 credits)
2. Update blueprint status to 'generating'
3. Call AI function GenerateSiteStructureFunction
4. Parse AI response JSON
5. Update blueprint with structure JSON
6. Create/update/delete PageBlueprint records based on structure
7. Update blueprint status to 'ready'
8. Deduct credits

PageGenerationService

Purpose: Generate page content from blueprints

Key Functions:

• generate_page_content(page_blueprint, account) - Generate content for a page
• regenerate_page(page_blueprint, account) - Regenerate page content

Workflow:

1. Check credits for page generation (20 credits per page)
2. Create Writer task from page blueprint
3. Use ContentGenerationService to generate content
4. Update page blueprint status
5. Deduct credits

SiteBuilderFileService

Purpose: Manage site files and assets

Key Functions:

• get_user_accessible_sites(user) - Get sites user can access
• check_file_access(user, site_id) - Validate user access to site files
• upload_file(user, site_id, file, folder='images') - Upload file to site assets
• delete_file(user, site_id, file_path) - Delete file from site assets
• list_files(user, site_id, folder='images') - List files in site folder
• check_storage_quota(site_id, file_size) - Validate storage quota

File Structure:

/data/app/sites-data/clients/{site_id}/v{version}/
├── site.json              # Site definition
├── pages/                 # Page definitions
│   ├── home.json
│   ├── about.json
│   └── ...
└── assets/                # User-managed files
    ├── images/
    ├── documents/
    └── media/

Access Control:

• Owner/Admin: Full access to all account sites
• Editor: Access to granted sites (via SiteUserAccess)
• Viewer: Read-only access to granted sites

AI Functions

GenerateSiteStructureFunction

Purpose: Generate site structure from business brief

Operation Type: site_structure_generation Credit Cost: 50 credits

Workflow:

1. Build prompt from business brief, objectives, style preferences
2. Call AI with prompt
3. Parse JSON response
4. Validate structure format
5. Return structure JSON

Structure Format:

{
  "pages": [
    {
      "slug": "home",
      "title": "Home",
      "type": "home",
      "blocks": [...]
    }
  ],
  "layout": "default",
  "theme": {...}
}

Database Migrations

Migration: site_building.0001_initial

• Purpose: Create SiteBlueprint and PageBlueprint tables
• Status: ✅ Applied
• Tables Created: igny8_site_blueprints, igny8_page_blueprints
• Dependencies: igny8_core_auth.0014_remove_plan_operation_limits_phase0, writer.0009_add_content_site_source_fields

API Endpoints

SiteBuilderViewSet:

• GET /api/v1/site-builder/blueprints/ - List site blueprints
• POST /api/v1/site-builder/blueprints/ - Create site blueprint
• GET /api/v1/site-builder/blueprints/{id}/ - Get site blueprint
• PUT /api/v1/site-builder/blueprints/{id}/ - Update site blueprint
• DELETE /api/v1/site-builder/blueprints/{id}/ - Delete site blueprint
• POST /api/v1/site-builder/blueprints/{id}/generate_structure/ - Generate site structure

PageBlueprintViewSet:

• GET /api/v1/site-builder/pages/ - List page blueprints
• POST /api/v1/site-builder/pages/ - Create page blueprint
• GET /api/v1/site-builder/pages/{id}/ - Get page blueprint
• PUT /api/v1/site-builder/pages/{id}/ - Update page blueprint
• DELETE /api/v1/site-builder/pages/{id}/ - Delete page blueprint
• POST /api/v1/site-builder/pages/{id}/generate_content/ - Generate page content
• POST /api/v1/site-builder/pages/{id}/regenerate/ - Regenerate page content

SiteAssetView:

• GET /api/v1/site-builder/assets/ - List files
• POST /api/v1/site-builder/assets/ - Upload file
• DELETE /api/v1/site-builder/assets/ - Delete file

Frontend Implementation

Site Builder Container (site-builder/):

• Standalone Vite + React + TypeScript application
• Docker container on port 8025:5175
• Routed to builder.igny8.com via Caddy reverse proxy

Wizard Steps:

• Step 1: Business Details (Business type, industry, audience)
• Step 2: Business Brief (Description textarea)
• Step 3: Objectives (Multiple objectives with add/remove)
• Step 4: Style Preferences (Palette, typography, personality)

Preview Canvas:

• Live preview of generated site structure
• Page navigation
• Block rendering

Site Dashboard:

• List all site blueprints
• Create new blueprint
• View blueprint details
• Generate structure
• Deploy site

Shared Component Library (frontend/src/components/shared/):

• Blocks: HeroBlock, FeatureGridBlock, StatsPanel
• Layouts: DefaultLayout, MinimalLayout
• Templates: MarketingTemplate, LandingTemplate

State Management:

• builderStore.ts - Wizard state (currentStep, wizardData, activeBlueprint)
• siteDefinitionStore.ts - Site preview state (siteStructure, pages, activePageSlug)

---

PHASE 4: LINKER & OPTIMIZER

Status: ✅ IMPLEMENTED

Purpose

Add internal linking and content optimization as post-processing stages with multiple entry points for different content sources.

Key Objectives

• Add internal linking to content
• Add content optimization
• Support multiple entry points (Writer, WordPress Sync, 3rd Party, Manual)
• Create content pipeline service
• Build UI for linker and optimizer

Content Model Extensions

Purpose: Track content source and sync status

Fields Added to Content Model:

• source - Content source (igny8, wordpress, shopify, custom)
• sync_status - Sync status (native, imported, synced)
• external_id - External platform ID
• external_url - External platform URL
• sync_metadata - Platform-specific sync metadata
• internal_links - JSON field for internal links added by linker
• linker_version - Version of linker processing
• optimizer_version - Version of optimizer processing
• optimization_scores - JSON field for optimization scores (SEO, readability, engagement)

Workflow: Content source tracked from creation, sync status updated during import/sync, linking and optimization versions incremented after processing.

Models

OptimizationTask Model

Purpose: Track optimization operations

Key Fields:

• content - ForeignKey to Content
• scores_before - Optimization scores before optimization
• scores_after - Optimization scores after optimization
• html_before - HTML content before optimization
• html_after - HTML content after optimization
• status - Task status (pending, running, completed, failed)
• credits_used - Credits used for optimization
• metadata - Additional metadata

Workflow: Task created before optimization, scores calculated before/after, HTML stored for comparison, credits logged.

Services

LinkerService

Purpose: Add internal links to content

Key Functions:

• process(content_id) - Process content for linking
• batch_process(content_ids) - Process multiple content items

Workflow:

1. Check credits for linking (8 credits per content)
2. Find link candidates using CandidateEngine
3. Inject links using InjectionEngine
4. Update content with links and increment linker_version
5. Deduct credits and log usage

CandidateEngine

Purpose: Find relevant content for linking

Key Functions:

• find_candidates(content) - Find link candidates based on content relevance
• _find_relevant_content(content) - Query relevant content
• _score_candidates(content, candidates) - Score candidates by relevance

Workflow:

1. Extract keywords from content
2. Query content with similar keywords
3. Score candidates by relevance (keyword overlap, semantic similarity)
4. Return top candidates

InjectionEngine

Purpose: Inject links into HTML content

Key Functions:

• inject_links(content, candidates) - Inject links into content HTML
• _inject_link_into_html(html, link_data) - Inject single link

Workflow:

1. Parse HTML content
2. Find anchor text candidates
3. Inject links at appropriate positions
4. Respect max links per content
5. Prevent duplicate links
6. Return updated HTML

OptimizerService

Purpose: Optimize content with multiple entry points

Key Functions:

• optimize_from_writer(content_id) - Entry Point 1: Writer → Optimizer
• optimize_from_wordpress_sync(content_id) - Entry Point 2: WordPress Sync → Optimizer
• optimize_from_external_sync(content_id) - Entry Point 3: External Sync → Optimizer
• optimize_manual(content_id) - Entry Point 4: Manual Selection → Optimizer
• optimize(content) - Unified optimization logic

Workflow:

1. Check credits for optimization (1 credit per 200 words)
2. Analyze content before optimization using ContentAnalyzer
3. Call AI function OptimizeContentFunction to optimize content
4. Analyze optimized content
5. Create OptimizationTask record
6. Update content with optimized HTML and scores
7. Increment optimizer_version
8. Deduct credits and log usage

ContentAnalyzer

Purpose: Analyze content quality

Key Functions:

• analyze(content) - Analyze content and return scores
• _calculate_seo_score(content) - Calculate SEO score
• _calculate_readability_score(content) - Calculate readability score
• _calculate_engagement_score(content) - Calculate engagement score

Scoring:

• SEO Score: Keyword density, meta tags, heading structure
• Readability Score: Sentence length, paragraph length, Flesch reading ease
• Engagement Score: Call-to-action presence, internal links, formatting
• Overall Score: Weighted average of all scores

ContentPipelineService

Purpose: Orchestrate content processing pipeline

Key Functions:

• process_writer_content(content_id, stages=['linking', 'optimization']) - Full pipeline for Writer content
• process_synced_content(content_id, stages=['optimization']) - Optimization-only for synced content

Pipeline Workflow:

1. Writer Content → Linker → Optimizer → Publish
2. Synced Content → Optimizer → Publish

Content States:

• draft - Generated, not processed
• linked - Links added, ready for optimization
• optimized - Optimized, ready for review
• review - Ready for publishing
• published - Published to destination(s)

AI Functions

OptimizeContentFunction

Purpose: Optimize content for SEO, readability, and engagement

Operation Type: optimize_content Credit Cost: 1 credit per 200 words

Workflow:

1. Build prompt with content, target keywords, optimization goals
2. Call AI with prompt
3. Parse optimized HTML from response
4. Return optimized content

Optimization Goals:

• SEO: Keyword optimization, meta tags, heading structure
• Readability: Sentence structure, paragraph length, clarity
• Engagement: Call-to-actions, formatting, internal links

Database Migrations

Migration: writer.0009_add_content_site_source_fields

• Purpose: Add Phase 4 fields to Content model
• Status: ✅ Applied
• Fields Added: source, sync_status, external_id, external_url, sync_metadata, internal_links, linker_version, optimizer_version, optimization_scores
• Indexes Added: source, sync_status, source+sync_status

Migration: optimization.0001_initial

• Purpose: Create OptimizationTask table
• Status: ✅ Applied
• Table Created: igny8_optimization_tasks
• Dependencies: igny8_core_auth.0013_remove_ai_cost_per_request, writer.0009_add_content_site_source_fields

API Endpoints

LinkerViewSet:

• POST /api/v1/linker/process/ - Process content for linking
• POST /api/v1/linker/batch_process/ - Process multiple content items

OptimizerViewSet:

• POST /api/v1/optimizer/optimize/ - Optimize content (auto-detects entry point)
• POST /api/v1/optimizer/batch_optimize/ - Batch optimize multiple content items
• POST /api/v1/optimizer/analyze/ - Analyze content without optimizing

Throttle Rates:

• Linker: 30 requests per minute
• Optimizer: 10 requests per minute

Frontend Implementation

Linker Pages:

• Dashboard - Linker overview with stats and quick actions
• ContentList - Content list with link processing actions

Optimizer Pages:

• Dashboard - Optimizer overview with stats and quick actions
• ContentSelector - Content selector with batch optimization and filters
• AnalysisPreview - Analysis preview page for content scores

Shared Components:

• SourceBadge - Displays content source (igny8, wordpress, shopify, custom)
• SyncStatusBadge - Displays sync status (native, imported, synced)
• ContentFilter - Filters content by source and sync status
• LinkResults - Displays linking results with links added count
• OptimizationScores - Displays optimization scores (SEO, readability, engagement, overall)
• ScoreComparison - Compares before/after optimization scores

Writer Integration:

• Source and sync status columns in Content table
• Source and sync status filters
• "Optimize" action button
• "Send to Optimizer" action

---

CROSS-PHASE INTEGRATION

Service Dependencies

Phase 1 Services Used by Phase 2-4:

• CreditService - Used by all phases for credit checks
• ContentGenerationService - Used by Phase 3 for page generation
• ClusteringService - Used by Phase 2 automation actions
• IdeasService - Used by Phase 2 automation actions

Phase 2 Integration:

• Automation rules can trigger Phase 1 services (clustering, ideas, content generation)
• Automation can trigger Phase 3 site structure generation
• Automation can trigger Phase 4 linking and optimization

Phase 3 Integration:

• Site Builder uses Phase 1 ContentGenerationService for page content
• Site Builder uses Phase 0 CreditService for credit checks
• Site pages stored as Content records (Phase 1)

Phase 4 Integration:

• Linker and Optimizer work on all Content records regardless of source
• ContentPipelineService orchestrates Writer → Linker → Optimizer pipeline
• Optimizer can optimize Site Builder page content

Workflow Integration

Complete Content Workflow:

1. User creates content in Writer (Phase 1)
2. Content flows to Linker (Phase 4)
3. Linked content flows to Optimizer (Phase 4)
4. Optimized content ready for publishing
5. Automation rules can trigger any step (Phase 2)

Site Builder Workflow:

1. User creates site blueprint via wizard (Phase 3)
2. AI generates site structure (Phase 3)
3. Pages created from structure (Phase 3)
4. Page content generated via Writer (Phase 1)
5. Pages can be optimized via Optimizer (Phase 4)

---

DATABASE MIGRATIONS SUMMARY

Phase 0 Migrations

Migration: igny8_core_auth.0014_remove_plan_operation_limits_phase0

• Status: ✅ Applied
• Purpose: Remove plan limit fields
• Risk: LOW

Phase 1 Migrations

Status: No new migrations required

• Models moved but kept same app_label
• Only import paths changed

Phase 2 Migrations

Migration: automation.0001_initial

• Status: ✅ Applied
• Purpose: Create AutomationRule and ScheduledTask tables
• Tables: igny8_automation_rules, igny8_scheduled_tasks

Phase 3 Migrations

Migration: site_building.0001_initial

• Status: ✅ Applied
• Purpose: Create SiteBlueprint and PageBlueprint tables
• Tables: igny8_site_blueprints, igny8_page_blueprints
• Dependencies: igny8_core_auth.0014_remove_plan_operation_limits_phase0, writer.0009_add_content_site_source_fields

Phase 4 Migrations

Migration: writer.0009_add_content_site_source_fields

• Status: ✅ Applied
• Purpose: Add Phase 4 fields to Content model
• Fields: source, sync_status, external_id, external_url, sync_metadata, internal_links, linker_version, optimizer_version, optimization_scores

Migration: optimization.0001_initial

• Status: ✅ Applied
• Purpose: Create OptimizationTask table
• Table: igny8_optimization_tasks
• Dependencies: igny8_core_auth.0013_remove_ai_cost_per_request, writer.0009_add_content_site_source_fields

Migration Dependencies

Phase 0: igny8_core_auth.0014_remove_plan_operation_limits_phase0
    ↓
Phase 3: site_building.0001_initial (depends on Phase 0)
    ↓
Phase 4: optimization.0001_initial (depends on writer.0009, which is applied)

---

TESTING STRATEGY

Phase 0 Testing

Credit System Tests:

• All existing features work with credit checks
• Credit deduction happens correctly
• Insufficient credits show clear error
• Usage logging tracks all operations
• Frontend shows credit balance, not limits

Module Settings Tests:

• Disabled modules don't appear in sidebar
• Disabled modules don't load routes
• Disabled modules return 403/404 appropriately
• Module settings persist correctly

Phase 1 Testing

Service Tests:

• Services can be tested independently
• Services handle errors correctly
• Services check credits before operations
• Services deduct credits after operations

API Compatibility Tests:

• All existing API endpoints work identically
• Response formats unchanged
• No breaking changes for frontend
• All ViewSet actions work correctly

Phase 2 Testing

Automation Service Tests:

• Rules execute correctly
• Conditions evaluate correctly
• Actions execute correctly
• Execution limits enforced
• Credit checks work

Scheduled Tasks Tests:

• Scheduled tasks run on time
• Credit replenishment works monthly
• Task status tracking works

Phase 3 Testing

Site Builder Tests:

• Site Builder wizard works end-to-end
• Structure generation creates valid blueprints
• Preview renders correctly
• Page generation reuses existing content service
• File management works correctly

Component Library Tests:

• Components render correctly
• Components work in Site Builder
• Components work in Sites Renderer

Phase 4 Testing

Linker Tests:

• Linker finds appropriate link candidates
• Links inject correctly into content
• Credit deduction works
• Batch processing works

Optimizer Tests:

• Optimizer works from Writer entry point
• Optimizer works from WordPress sync entry point
• Optimizer works from 3rd party sync entry point
• Optimizer works from manual selection
• Content source tracking works
• Pipeline orchestrates correctly

Backend Test Coverage:

• 10 test files for Phase 4
• 70+ test cases
• All services tested
• All API endpoints tested

Frontend Test Coverage:

• 7 test files for Phase 4
• 30+ test cases
• All components tested
• All pages tested

---

SUCCESS CRITERIA

Phase 0 Success Criteria

✅ All existing features work with credit checks
✅ Credit deduction happens correctly for all operations
✅ Insufficient credits show clear error messages
✅ Usage logging tracks all operations
✅ Frontend shows credit balance, not limits
✅ Module settings enable/disable modules correctly
✅ Disabled modules don't appear in UI
✅ No breaking changes for existing users

Phase 1 Success Criteria

✅ Services are testable independently
✅ Business logic extracted from ViewSets
✅ ViewSets are thin wrappers that delegate to services
✅ All models moved to business layer
✅ All imports updated correctly
✅ Services handle credit checks and business rules

Phase 2 Success Criteria

✅ Automation rules execute correctly
✅ Scheduled tasks run on time
✅ Credit replenishment works monthly
✅ UI shows automation status
✅ Rules can be created, edited, deleted
✅ Execution history is tracked
✅ All automation respects credit limits

Phase 3 Success Criteria

✅ Site Builder wizard works end-to-end
✅ Structure generation creates valid blueprints
✅ Preview renders correctly
✅ Page generation reuses existing content service
✅ File management works correctly
✅ Shared components work across all apps
✅ Multiple layouts supported

Phase 4 Success Criteria

✅ Writer → Linker handover works
✅ Optimizer works from all entry points
✅ Content source tracking works
✅ Pipeline orchestrates correctly
✅ UI shows content sources and filters
✅ API endpoints functional and tested
✅ AI function integrated and working
✅ Credit deduction working at all stages
✅ Frontend UI complete with all dashboards and selectors
✅ Writer integration complete with badges and filters
✅ Navigation and routing complete
✅ Backend tests complete (10 test files)
✅ Frontend tests complete (7 test files)

---

END OF PHASES 0-4 DOCUMENTATION