igny8/part2-dev/PHASE-0-4-FOUNDATION-TO-LINKER-OPTIMIZER.md

# PHASES 0-4: FOUNDATION TO LINKER & OPTIMIZER
**Complete Implementation Documentation - IMPLEMENTED ✅**

**Status**: All Phases Complete  
**Last Updated**: 2025-01-XX  
**Implementation Period**: Foundation through Linker & Optimizer

---

## TABLE OF CONTENTS

1. [Executive Summary](#executive-summary)
2. [Phase 0: Foundation & Credit System](#phase-0-foundation--credit-system)
3. [Phase 1: Service Layer Refactoring](#phase-1-service-layer-refactoring)
4. [Phase 2: Automation System](#phase-2-automation-system)
5. [Phase 3: Site Builder](#phase-3-site-builder)
6. [Phase 4: Linker & Optimizer](#phase-4-linker--optimizer)
7. [Cross-Phase Integration](#cross-phase-integration)
8. [Database Migrations Summary](#database-migrations-summary)
9. [Testing Strategy](#testing-strategy)

---

## EXECUTIVE SUMMARY

This document covers the complete implementation of Phases 0-4, establishing the foundation for the IGNY8 platform:

- **Phase 0**: Credit-only system replacing plan limits, module enable/disable functionality
- **Phase 1**: Service layer architecture with business logic extraction
- **Phase 2**: Automation rules and scheduled tasks system
- **Phase 3**: Site Builder with AI-powered structure generation
- **Phase 4**: Internal linking and content optimization with multiple entry points

All phases are **fully implemented** and integrated into the production system.

---

## PHASE 0: FOUNDATION & CREDIT SYSTEM

**Status**: ✅ **IMPLEMENTED**

### Purpose

Migrate from plan-based limits to a credit-only model while preserving all existing functionality. Establish module enable/disable capabilities and comprehensive credit tracking.

### Key Objectives

- Migrate from plan-based limits to credit-only system
- Implement module enable/disable functionality
- Add credit cost tracking for all operations
- Preserve all existing functionality
- Update frontend to show credits instead of limits

### Models

#### ModuleSettings Model
**Purpose**: Control module availability per account

**Key Fields**:
- `planner_enabled` - Boolean flag for Planner module
- `writer_enabled` - Boolean flag for Writer module
- `automation_enabled` - Boolean flag for Automation module
- `site_builder_enabled` - Boolean flag for Site Builder module
- `linker_enabled` - Boolean flag for Linker module
- `optimizer_enabled` - Boolean flag for Optimizer module

**Workflow**: Accounts can enable/disable modules through settings. Disabled modules don't appear in UI and don't load routes.

#### Plan Model Updates
**Purpose**: Simplified plan model with only credit allocation

**Fields Removed**:
- All limit fields (`max_keywords`, `max_clusters`, `max_content_ideas`, `daily_content_tasks`, `monthly_word_count_limit`, `daily_image_generation_limit`, `monthly_image_count`)

**Fields Retained**:
- `monthly_credits` - Credits allocated per month
- `support_level` - Support tier
- `billing_cycle` - Billing frequency
- `price` - Plan price
- `features` - JSON field for future use

#### CreditUsageLog Model
**Purpose**: Track all credit usage with detailed metadata

**Key Fields**:
- `operation_type` - Type of operation (clustering, content_generation, linking, optimization, etc.)
- `credits_used` - Credits consumed
- `related_object_type` - Related model type
- `related_object_id` - Related model ID
- `metadata` - JSON field for operation-specific data

### Services

#### CreditService
**Purpose**: Centralized credit management and validation

**Key Functions**:
- `check_credits(account, operation_type, amount=None)` - Validates sufficient credits before operation
- `deduct_credits(account, operation_type, amount=None)` - Deducts credits after successful operation
- `get_credit_cost(operation_type, amount=None)` - Calculates credit cost for operation
- `add_credits(account, amount, reason)` - Adds credits (for replenishment)

**Credit Cost Constants**:
- `clustering`: 10 credits per request
- `idea_generation`: 15 credits per cluster → ideas request
- `content_generation`: 1 credit per 100 words
- `image_generation`: 5 credits per image
- `linking`: 8 credits per content piece
- `optimization`: 1 credit per 200 words
- `site_structure_generation`: 50 credits per site blueprint
- `site_page_generation`: 20 credits per page

**Workflow**: All operations check credits before execution, deduct credits after success, and log usage for audit trail.

### Workflows

#### Module Enable/Disable Workflow
1. User accesses Settings → Modules
2. User toggles module enabled/disabled state
3. Frontend checks module status before loading routes
4. Disabled modules filtered from sidebar navigation
5. API validates module status before processing requests

#### Credit Check Workflow
1. Operation initiated (e.g., content generation)
2. Service calls `CreditService.check_credits()`
3. System validates account has sufficient credits
4. If insufficient, raises `InsufficientCreditsError`
5. If sufficient, operation proceeds
6. After success, `CreditService.deduct_credits()` called
7. Usage logged in `CreditUsageLog`

#### Monthly Credit Replenishment Workflow
1. Celery Beat task runs on 1st of each month
2. Task queries all active accounts
3. For each account, adds `plan.monthly_credits` to balance
4. Logs replenishment in `CreditUsageLog`

### Database Migrations

**Migration**: `igny8_core_auth.0014_remove_plan_operation_limits_phase0`
- **Purpose**: Remove all plan limit fields
- **Status**: ✅ Applied
- **Risk**: LOW - Fields were unused, migration removes them safely

**Migration**: Credit tracking fields already included in `billing.0001_initial`
- `related_object_type`, `related_object_id`, `metadata` fields exist in `CreditUsageLog`

### Frontend Implementation

**Module Settings UI**:
- Settings page with toggle switches for each module
- Module status checked before route loading
- Disabled modules hidden from sidebar

**Credit Display**:
- Credit balance shown prominently in header
- Credit costs displayed per operation
- Usage history filtered by operation type

---

## PHASE 1: SERVICE LAYER REFACTORING

**Status**: ✅ **IMPLEMENTED**

### Purpose

Extract business logic from ViewSets into reusable services, creating a clean separation between API layer and business logic.

### Key Objectives

- Create `business/` folder structure
- Move models from `modules/` to `business/`
- Extract business logic from ViewSets to services
- Keep ViewSets as thin wrappers
- Preserve all existing API functionality

### Architecture

#### Business Layer Structure
```
business/
├── content/              # Content business logic
│   ├── models.py        # Content, Tasks, Images
│   └── services/
│       ├── content_generation_service.py
│       ├── content_pipeline_service.py
│       └── content_versioning_service.py
│
├── planning/             # Planning business logic
│   ├── models.py        # Keywords, Clusters, Ideas
│   └── services/
│       ├── clustering_service.py
│       └── ideas_service.py
│
├── billing/             # Billing business logic
│   ├── models.py        # Credits, Transactions
│   └── services/
│       └── credit_service.py
│
└── automation/          # Automation business logic
    ├── models.py
    └── services/
        └── automation_service.py
```

### Services

#### ContentGenerationService
**Purpose**: Unified content generation logic

**Key Functions**:
- `generate_content(task, account)` - Generate content for a task
- `regenerate_content(content_id, account)` - Regenerate existing content
- `batch_generate_content(task_ids, account)` - Generate multiple content pieces

**Workflow**:
1. Check credits via `CreditService.check_credits()`
2. Generate content using AI engine
3. Save content to database
4. Deduct credits via `CreditService.deduct_credits()`
5. Return generated content

#### ClusteringService
**Purpose**: Keyword clustering using AI

**Key Functions**:
- `cluster_keywords(keyword_ids, account)` - Cluster keywords into groups
- `update_cluster(cluster_id, keyword_ids, account)` - Update cluster membership

**Workflow**:
1. Check credits for clustering operation
2. Call AI function to generate clusters
3. Create Cluster records
4. Associate keywords with clusters
5. Deduct credits and log usage

#### IdeasService
**Purpose**: Generate content ideas from clusters

**Key Functions**:
- `generate_ideas(cluster_ids, account)` - Generate ideas for clusters
- `regenerate_ideas(cluster_id, account)` - Regenerate ideas for a cluster

**Workflow**:
1. Check credits for idea generation
2. Call AI function with cluster context
3. Create ContentIdeas records
4. Associate ideas with clusters
5. Deduct credits and log usage

### Model Migrations

**Content Models**: Moved from `modules/writer/models.py` to `business/content/models.py`
- `Content` model
- `Tasks` model
- `Images` model

**Planning Models**: Moved from `modules/planner/models.py` to `business/planning/models.py`
- `Keywords` model
- `Clusters` model
- `ContentIdeas` model

**Billing Models**: Moved from `modules/billing/models.py` to `business/billing/models.py`
- `CreditTransaction` model
- `CreditUsageLog` model

**Migration Strategy**: Models kept same `app_label` to avoid data migration complexity. Only import paths changed.

### ViewSet Refactoring

**Before**: ViewSets contained 50-100+ lines of business logic

**After**: ViewSets delegate to services:
```python
# Example: ClusterViewSet
@action(detail=False, methods=['post'])
def auto_generate_ideas(self, request):
    cluster_ids = request.data.get('cluster_ids')
    account = request.account
    
    # Delegate to service
    ideas = self.ideas_service.generate_ideas(cluster_ids, account)
    
    # Serialize and return
    serializer = ContentIdeasSerializer(ideas, many=True)
    return Response(serializer.data)
```

### Workflows

#### Content Generation Workflow
1. User creates task in Writer module
2. User triggers content generation
3. ViewSet receives request, delegates to `ContentGenerationService`
4. Service checks credits, generates content, deducts credits
5. ViewSet serializes and returns response

#### Keyword Clustering Workflow
1. User selects keywords in Planner module
2. User triggers clustering
3. ViewSet delegates to `ClusteringService`
4. Service checks credits, calls AI, creates clusters
5. ViewSet returns cluster data

---

## PHASE 2: AUTOMATION SYSTEM

**Status**: ✅ **IMPLEMENTED**

### Purpose

Implement automation rules and scheduled tasks to enable users to automate repetitive workflows.

### Key Objectives

- Create AutomationRule and ScheduledTask models
- Build AutomationService with rule execution engine
- Implement Celery Beat scheduled tasks
- Create automation API endpoints
- Build automation UI (Dashboard, Rules, History)

### Models

#### AutomationRule Model
**Purpose**: Store automation rule definitions

**Key Fields**:
- `name` - Rule name
- `description` - Rule description
- `trigger` - Trigger type (schedule, keyword_added, cluster_created, idea_created, content_generated, task_created)
- `conditions` - JSON field for condition evaluation
- `actions` - JSON field for actions to execute
- `schedule` - JSON field for cron-like schedule (for scheduled triggers)
- `is_active` - Whether rule is active
- `max_executions_per_day` - Daily execution limit
- `credit_limit_per_execution` - Credit limit per execution
- `last_executed_at` - Last execution timestamp
- `execution_count_today` - Daily execution counter

**Workflow**: Rules evaluate conditions, execute actions when triggered, respect execution limits and credit constraints.

#### ScheduledTask Model
**Purpose**: Track scheduled task executions

**Key Fields**:
- `automation_rule` - ForeignKey to AutomationRule
- `scheduled_at` - Scheduled execution time
- `executed_at` - Actual execution time
- `status` - Task status (pending, running, completed, failed, skipped)
- `result` - JSON field for execution results
- `error_message` - Error message if failed
- `credits_used` - Credits consumed

**Workflow**: Tasks scheduled by Celery Beat, executed by AutomationService, results stored for audit trail.

### Services

#### AutomationService
**Purpose**: Execute automation rules with condition evaluation and action execution

**Key Functions**:
- `execute_rule(rule, context=None)` - Execute an automation rule
- `evaluate_conditions(rule, context)` - Evaluate rule conditions
- `execute_actions(rule, context)` - Execute rule actions
- `check_execution_limits(rule)` - Validate execution limits

**Workflow**:
1. Check if rule is active
2. Check execution limits (daily limit, credit limit)
3. Evaluate conditions against context
4. If conditions met, execute actions
5. Update rule tracking (last_executed_at, execution_count_today)
6. Create ScheduledTask record
7. Return execution results

#### RuleEngine
**Purpose**: Orchestrate rule execution

**Key Functions**:
- `execute_rule(rule, context)` - Orchestrate full rule execution
- `validate_rule(rule)` - Validate rule configuration

#### ConditionEvaluator
**Purpose**: Evaluate rule conditions

**Supported Operators**:
- `eq` - Equals
- `ne` - Not equals
- `gt` - Greater than
- `gte` - Greater than or equal
- `lt` - Less than
- `lte` - Less than or equal
- `in` - In list
- `contains` - Contains substring

**Example Condition**:
```json
{
  "field": "status",
  "operator": "eq",
  "value": "draft"
}
```

#### ActionExecutor
**Purpose**: Execute rule actions

**Supported Actions**:
- `generate_ideas` - Generate content ideas from clusters
- `generate_content` - Generate content from tasks
- `cluster_keywords` - Cluster keywords
- `optimize_content` - Optimize content

**Example Action**:
```json
{
  "type": "generate_ideas",
  "params": {
    "cluster_ids": [1, 2, 3]
  }
}
```

### Celery Beat Tasks

#### Scheduled Automation Task
**Purpose**: Execute scheduled automation rules

**Schedule**: Every 15 minutes

**Workflow**:
1. Query all active rules with `trigger='schedule'`
2. For each rule, check if schedule matches current time
3. If matches, execute rule via `AutomationService`
4. Log execution in ScheduledTask

#### Monthly Credit Replenishment Task
**Purpose**: Add monthly credits to all active accounts

**Schedule**: 1st day of month at midnight

**Workflow**:
1. Query all active accounts
2. For each account, add `plan.monthly_credits` to balance
3. Log replenishment in `CreditUsageLog` with `operation_type='monthly_replenishment'`

### Database Migrations

**Migration**: `automation.0001_initial`
- **Purpose**: Create AutomationRule and ScheduledTask tables
- **Status**: ✅ Applied
- **Tables Created**: `igny8_automation_rules`, `igny8_scheduled_tasks`

### API Endpoints

**AutomationRuleViewSet**:
- `GET /api/v1/automation/rules/` - List rules
- `POST /api/v1/automation/rules/` - Create rule
- `GET /api/v1/automation/rules/{id}/` - Get rule
- `PUT /api/v1/automation/rules/{id}/` - Update rule
- `DELETE /api/v1/automation/rules/{id}/` - Delete rule
- `POST /api/v1/automation/rules/{id}/execute/` - Manually execute rule
- `POST /api/v1/automation/rules/{id}/test/` - Test rule conditions

**ScheduledTaskViewSet**:
- `GET /api/v1/automation/scheduled-tasks/` - List scheduled tasks
- `GET /api/v1/automation/scheduled-tasks/{id}/` - Get scheduled task

### Frontend Implementation

**Automation Dashboard**:
- Active rules count
- Recent executions
- Success/failure rates
- Credit usage from automation
- Quick actions (create rule, view history)

**Rules Management**:
- List all rules
- Create new rule (wizard)
- Edit existing rule
- Enable/disable rule
- Delete rule
- Test rule
- Manual execution

**Scheduled Tasks History**:
- List scheduled tasks
- Filter by status, rule, date
- View execution results
- View error messages
- Retry failed tasks

---

## PHASE 3: SITE BUILDER

**Status**: ✅ **IMPLEMENTED**

### Purpose

Build Site Builder for creating sites via wizard with AI-powered structure generation, preview canvas, and file management.

### Key Objectives

- Create Site Builder wizard for site creation
- Generate site structure using AI
- Build preview canvas for site editing
- Create shared component library
- Support multiple layouts and templates
- Enable file management for site assets

### Models

#### SiteBlueprint Model
**Purpose**: Store site structure definitions

**Key Fields**:
- `name` - Site name
- `description` - Site description
- `config_json` - Wizard configuration (business_type, style, objectives)
- `structure_json` - AI-generated structure (pages, layout, theme)
- `status` - Status (draft, generating, ready, deployed)
- `hosting_type` - Hosting platform (igny8_sites, wordpress, shopify, multi)
- `version` - Blueprint version
- `deployed_version` - Currently deployed version

**Workflow**: Blueprint created via wizard, structure generated by AI, pages created from structure, content generated for pages.

#### PageBlueprint Model
**Purpose**: Store page definitions within a site

**Key Fields**:
- `site_blueprint` - ForeignKey to SiteBlueprint
- `slug` - Page slug
- `title` - Page title
- `type` - Page type (home, about, services, products, blog, contact, custom)
- `blocks_json` - Page content blocks
- `status` - Status (draft, generating, ready)
- `order` - Page order in navigation

**Workflow**: Pages created from AI-generated structure, content generated via Writer module, pages rendered in preview.

### Services

#### StructureGenerationService
**Purpose**: Generate site structure using AI

**Key Functions**:
- `generate_structure(site_blueprint, business_brief, objectives, style)` - Generate site structure
- `_create_page_blueprints(site_blueprint, structure)` - Create PageBlueprint records from structure

**Workflow**:
1. Check credits for structure generation (50 credits)
2. Update blueprint status to 'generating'
3. Call AI function `GenerateSiteStructureFunction`
4. Parse AI response JSON
5. Update blueprint with structure JSON
6. Create/update/delete PageBlueprint records based on structure
7. Update blueprint status to 'ready'
8. Deduct credits

#### PageGenerationService
**Purpose**: Generate page content from blueprints

**Key Functions**:
- `generate_page_content(page_blueprint, account)` - Generate content for a page
- `regenerate_page(page_blueprint, account)` - Regenerate page content

**Workflow**:
1. Check credits for page generation (20 credits per page)
2. Create Writer task from page blueprint
3. Use ContentGenerationService to generate content
4. Update page blueprint status
5. Deduct credits

#### SiteBuilderFileService
**Purpose**: Manage site files and assets

**Key Functions**:
- `get_user_accessible_sites(user)` - Get sites user can access
- `check_file_access(user, site_id)` - Validate user access to site files
- `upload_file(user, site_id, file, folder='images')` - Upload file to site assets
- `delete_file(user, site_id, file_path)` - Delete file from site assets
- `list_files(user, site_id, folder='images')` - List files in site folder
- `check_storage_quota(site_id, file_size)` - Validate storage quota

**File Structure**:
```
/data/app/sites-data/clients/{site_id}/v{version}/
├── site.json              # Site definition
├── pages/                 # Page definitions
│   ├── home.json
│   ├── about.json
│   └── ...
└── assets/                # User-managed files
    ├── images/
    ├── documents/
    └── media/
```

**Access Control**:
- Owner/Admin: Full access to all account sites
- Editor: Access to granted sites (via SiteUserAccess)
- Viewer: Read-only access to granted sites

### AI Functions

#### GenerateSiteStructureFunction
**Purpose**: Generate site structure from business brief

**Operation Type**: `site_structure_generation`
**Credit Cost**: 50 credits

**Workflow**:
1. Build prompt from business brief, objectives, style preferences
2. Call AI with prompt
3. Parse JSON response
4. Validate structure format
5. Return structure JSON

**Structure Format**:
```json
{
  "pages": [
    {
      "slug": "home",
      "title": "Home",
      "type": "home",
      "blocks": [...]
    }
  ],
  "layout": "default",
  "theme": {...}
}
```

### Database Migrations

**Migration**: `site_building.0001_initial`
- **Purpose**: Create SiteBlueprint and PageBlueprint tables
- **Status**: ✅ Applied
- **Tables Created**: `igny8_site_blueprints`, `igny8_page_blueprints`
- **Dependencies**: `igny8_core_auth.0014_remove_plan_operation_limits_phase0`, `writer.0009_add_content_site_source_fields`

### API Endpoints

**SiteBuilderViewSet**:
- `GET /api/v1/site-builder/blueprints/` - List site blueprints
- `POST /api/v1/site-builder/blueprints/` - Create site blueprint
- `GET /api/v1/site-builder/blueprints/{id}/` - Get site blueprint
- `PUT /api/v1/site-builder/blueprints/{id}/` - Update site blueprint
- `DELETE /api/v1/site-builder/blueprints/{id}/` - Delete site blueprint
- `POST /api/v1/site-builder/blueprints/{id}/generate_structure/` - Generate site structure

**PageBlueprintViewSet**:
- `GET /api/v1/site-builder/pages/` - List page blueprints
- `POST /api/v1/site-builder/pages/` - Create page blueprint
- `GET /api/v1/site-builder/pages/{id}/` - Get page blueprint
- `PUT /api/v1/site-builder/pages/{id}/` - Update page blueprint
- `DELETE /api/v1/site-builder/pages/{id}/` - Delete page blueprint
- `POST /api/v1/site-builder/pages/{id}/generate_content/` - Generate page content
- `POST /api/v1/site-builder/pages/{id}/regenerate/` - Regenerate page content

**SiteAssetView**:
- `GET /api/v1/site-builder/assets/` - List files
- `POST /api/v1/site-builder/assets/` - Upload file
- `DELETE /api/v1/site-builder/assets/` - Delete file

### Frontend Implementation

**Site Builder Container** (`site-builder/`):
- Standalone Vite + React + TypeScript application
- Docker container on port 8025:5175
- Routed to `builder.igny8.com` via Caddy reverse proxy

**Wizard Steps**:
- Step 1: Business Details (Business type, industry, audience)
- Step 2: Business Brief (Description textarea)
- Step 3: Objectives (Multiple objectives with add/remove)
- Step 4: Style Preferences (Palette, typography, personality)

**Preview Canvas**:
- Live preview of generated site structure
- Page navigation
- Block rendering

**Site Dashboard**:
- List all site blueprints
- Create new blueprint
- View blueprint details
- Generate structure
- Deploy site

**Shared Component Library** (`frontend/src/components/shared/`):
- **Blocks**: HeroBlock, FeatureGridBlock, StatsPanel
- **Layouts**: DefaultLayout, MinimalLayout
- **Templates**: MarketingTemplate, LandingTemplate

**State Management**:
- `builderStore.ts` - Wizard state (currentStep, wizardData, activeBlueprint)
- `siteDefinitionStore.ts` - Site preview state (siteStructure, pages, activePageSlug)

---

## PHASE 4: LINKER & OPTIMIZER

**Status**: ✅ **IMPLEMENTED**

### Purpose

Add internal linking and content optimization as post-processing stages with multiple entry points for different content sources.

### Key Objectives

- Add internal linking to content
- Add content optimization
- Support multiple entry points (Writer, WordPress Sync, 3rd Party, Manual)
- Create content pipeline service
- Build UI for linker and optimizer

### Content Model Extensions

**Purpose**: Track content source and sync status

**Fields Added to Content Model**:
- `source` - Content source (igny8, wordpress, shopify, custom)
- `sync_status` - Sync status (native, imported, synced)
- `external_id` - External platform ID
- `external_url` - External platform URL
- `sync_metadata` - Platform-specific sync metadata
- `internal_links` - JSON field for internal links added by linker
- `linker_version` - Version of linker processing
- `optimizer_version` - Version of optimizer processing
- `optimization_scores` - JSON field for optimization scores (SEO, readability, engagement)

**Workflow**: Content source tracked from creation, sync status updated during import/sync, linking and optimization versions incremented after processing.

### Models

#### OptimizationTask Model
**Purpose**: Track optimization operations

**Key Fields**:
- `content` - ForeignKey to Content
- `scores_before` - Optimization scores before optimization
- `scores_after` - Optimization scores after optimization
- `html_before` - HTML content before optimization
- `html_after` - HTML content after optimization
- `status` - Task status (pending, running, completed, failed)
- `credits_used` - Credits used for optimization
- `metadata` - Additional metadata

**Workflow**: Task created before optimization, scores calculated before/after, HTML stored for comparison, credits logged.

### Services

#### LinkerService
**Purpose**: Add internal links to content

**Key Functions**:
- `process(content_id)` - Process content for linking
- `batch_process(content_ids)` - Process multiple content items

**Workflow**:
1. Check credits for linking (8 credits per content)
2. Find link candidates using CandidateEngine
3. Inject links using InjectionEngine
4. Update content with links and increment linker_version
5. Deduct credits and log usage

#### CandidateEngine
**Purpose**: Find relevant content for linking

**Key Functions**:
- `find_candidates(content)` - Find link candidates based on content relevance
- `_find_relevant_content(content)` - Query relevant content
- `_score_candidates(content, candidates)` - Score candidates by relevance

**Workflow**:
1. Extract keywords from content
2. Query content with similar keywords
3. Score candidates by relevance (keyword overlap, semantic similarity)
4. Return top candidates

#### InjectionEngine
**Purpose**: Inject links into HTML content

**Key Functions**:
- `inject_links(content, candidates)` - Inject links into content HTML
- `_inject_link_into_html(html, link_data)` - Inject single link

**Workflow**:
1. Parse HTML content
2. Find anchor text candidates
3. Inject links at appropriate positions
4. Respect max links per content
5. Prevent duplicate links
6. Return updated HTML

#### OptimizerService
**Purpose**: Optimize content with multiple entry points

**Key Functions**:
- `optimize_from_writer(content_id)` - Entry Point 1: Writer → Optimizer
- `optimize_from_wordpress_sync(content_id)` - Entry Point 2: WordPress Sync → Optimizer
- `optimize_from_external_sync(content_id)` - Entry Point 3: External Sync → Optimizer
- `optimize_manual(content_id)` - Entry Point 4: Manual Selection → Optimizer
- `optimize(content)` - Unified optimization logic

**Workflow**:
1. Check credits for optimization (1 credit per 200 words)
2. Analyze content before optimization using ContentAnalyzer
3. Call AI function `OptimizeContentFunction` to optimize content
4. Analyze optimized content
5. Create OptimizationTask record
6. Update content with optimized HTML and scores
7. Increment optimizer_version
8. Deduct credits and log usage

#### ContentAnalyzer
**Purpose**: Analyze content quality

**Key Functions**:
- `analyze(content)` - Analyze content and return scores
- `_calculate_seo_score(content)` - Calculate SEO score
- `_calculate_readability_score(content)` - Calculate readability score
- `_calculate_engagement_score(content)` - Calculate engagement score

**Scoring**:
- SEO Score: Keyword density, meta tags, heading structure
- Readability Score: Sentence length, paragraph length, Flesch reading ease
- Engagement Score: Call-to-action presence, internal links, formatting
- Overall Score: Weighted average of all scores

#### ContentPipelineService
**Purpose**: Orchestrate content processing pipeline

**Key Functions**:
- `process_writer_content(content_id, stages=['linking', 'optimization'])` - Full pipeline for Writer content
- `process_synced_content(content_id, stages=['optimization'])` - Optimization-only for synced content

**Pipeline Workflow**:
1. Writer Content → Linker → Optimizer → Publish
2. Synced Content → Optimizer → Publish

**Content States**:
- `draft` - Generated, not processed
- `linked` - Links added, ready for optimization
- `optimized` - Optimized, ready for review
- `review` - Ready for publishing
- `published` - Published to destination(s)

### AI Functions

#### OptimizeContentFunction
**Purpose**: Optimize content for SEO, readability, and engagement

**Operation Type**: `optimize_content`
**Credit Cost**: 1 credit per 200 words

**Workflow**:
1. Build prompt with content, target keywords, optimization goals
2. Call AI with prompt
3. Parse optimized HTML from response
4. Return optimized content

**Optimization Goals**:
- SEO: Keyword optimization, meta tags, heading structure
- Readability: Sentence structure, paragraph length, clarity
- Engagement: Call-to-actions, formatting, internal links

### Database Migrations

**Migration**: `writer.0009_add_content_site_source_fields`
- **Purpose**: Add Phase 4 fields to Content model
- **Status**: ✅ Applied
- **Fields Added**: source, sync_status, external_id, external_url, sync_metadata, internal_links, linker_version, optimizer_version, optimization_scores
- **Indexes Added**: source, sync_status, source+sync_status

**Migration**: `optimization.0001_initial`
- **Purpose**: Create OptimizationTask table
- **Status**: ✅ Applied
- **Table Created**: `igny8_optimization_tasks`
- **Dependencies**: `igny8_core_auth.0013_remove_ai_cost_per_request`, `writer.0009_add_content_site_source_fields`

### API Endpoints

**LinkerViewSet**:
- `POST /api/v1/linker/process/` - Process content for linking
- `POST /api/v1/linker/batch_process/` - Process multiple content items

**OptimizerViewSet**:
- `POST /api/v1/optimizer/optimize/` - Optimize content (auto-detects entry point)
- `POST /api/v1/optimizer/batch_optimize/` - Batch optimize multiple content items
- `POST /api/v1/optimizer/analyze/` - Analyze content without optimizing

**Throttle Rates**:
- Linker: 30 requests per minute
- Optimizer: 10 requests per minute

### Frontend Implementation

**Linker Pages**:
- Dashboard - Linker overview with stats and quick actions
- ContentList - Content list with link processing actions

**Optimizer Pages**:
- Dashboard - Optimizer overview with stats and quick actions
- ContentSelector - Content selector with batch optimization and filters
- AnalysisPreview - Analysis preview page for content scores

**Shared Components**:
- SourceBadge - Displays content source (igny8, wordpress, shopify, custom)
- SyncStatusBadge - Displays sync status (native, imported, synced)
- ContentFilter - Filters content by source and sync status
- LinkResults - Displays linking results with links added count
- OptimizationScores - Displays optimization scores (SEO, readability, engagement, overall)
- ScoreComparison - Compares before/after optimization scores

**Writer Integration**:
- Source and sync status columns in Content table
- Source and sync status filters
- "Optimize" action button
- "Send to Optimizer" action

---

## CROSS-PHASE INTEGRATION

### Service Dependencies

**Phase 1 Services Used by Phase 2-4**:
- `CreditService` - Used by all phases for credit checks
- `ContentGenerationService` - Used by Phase 3 for page generation
- `ClusteringService` - Used by Phase 2 automation actions
- `IdeasService` - Used by Phase 2 automation actions

**Phase 2 Integration**:
- Automation rules can trigger Phase 1 services (clustering, ideas, content generation)
- Automation can trigger Phase 3 site structure generation
- Automation can trigger Phase 4 linking and optimization

**Phase 3 Integration**:
- Site Builder uses Phase 1 ContentGenerationService for page content
- Site Builder uses Phase 0 CreditService for credit checks
- Site pages stored as Content records (Phase 1)

**Phase 4 Integration**:
- Linker and Optimizer work on all Content records regardless of source
- ContentPipelineService orchestrates Writer → Linker → Optimizer pipeline
- Optimizer can optimize Site Builder page content

### Workflow Integration

**Complete Content Workflow**:
1. User creates content in Writer (Phase 1)
2. Content flows to Linker (Phase 4)
3. Linked content flows to Optimizer (Phase 4)
4. Optimized content ready for publishing
5. Automation rules can trigger any step (Phase 2)

**Site Builder Workflow**:
1. User creates site blueprint via wizard (Phase 3)
2. AI generates site structure (Phase 3)
3. Pages created from structure (Phase 3)
4. Page content generated via Writer (Phase 1)
5. Pages can be optimized via Optimizer (Phase 4)

---

## DATABASE MIGRATIONS SUMMARY

### Phase 0 Migrations

**Migration**: `igny8_core_auth.0014_remove_plan_operation_limits_phase0`
- **Status**: ✅ Applied
- **Purpose**: Remove plan limit fields
- **Risk**: LOW

### Phase 1 Migrations

**Status**: No new migrations required
- Models moved but kept same `app_label`
- Only import paths changed

### Phase 2 Migrations

**Migration**: `automation.0001_initial`
- **Status**: ✅ Applied
- **Purpose**: Create AutomationRule and ScheduledTask tables
- **Tables**: `igny8_automation_rules`, `igny8_scheduled_tasks`

### Phase 3 Migrations

**Migration**: `site_building.0001_initial`
- **Status**: ✅ Applied
- **Purpose**: Create SiteBlueprint and PageBlueprint tables
- **Tables**: `igny8_site_blueprints`, `igny8_page_blueprints`
- **Dependencies**: `igny8_core_auth.0014_remove_plan_operation_limits_phase0`, `writer.0009_add_content_site_source_fields`

### Phase 4 Migrations

**Migration**: `writer.0009_add_content_site_source_fields`
- **Status**: ✅ Applied
- **Purpose**: Add Phase 4 fields to Content model
- **Fields**: source, sync_status, external_id, external_url, sync_metadata, internal_links, linker_version, optimizer_version, optimization_scores

**Migration**: `optimization.0001_initial`
- **Status**: ✅ Applied
- **Purpose**: Create OptimizationTask table
- **Table**: `igny8_optimization_tasks`
- **Dependencies**: `igny8_core_auth.0013_remove_ai_cost_per_request`, `writer.0009_add_content_site_source_fields`

### Migration Dependencies

```
Phase 0: igny8_core_auth.0014_remove_plan_operation_limits_phase0
    ↓
Phase 3: site_building.0001_initial (depends on Phase 0)
    ↓
Phase 4: optimization.0001_initial (depends on writer.0009, which is applied)
```

---

## TESTING STRATEGY

### Phase 0 Testing

**Credit System Tests**:
- All existing features work with credit checks
- Credit deduction happens correctly
- Insufficient credits show clear error
- Usage logging tracks all operations
- Frontend shows credit balance, not limits

**Module Settings Tests**:
- Disabled modules don't appear in sidebar
- Disabled modules don't load routes
- Disabled modules return 403/404 appropriately
- Module settings persist correctly

### Phase 1 Testing

**Service Tests**:
- Services can be tested independently
- Services handle errors correctly
- Services check credits before operations
- Services deduct credits after operations

**API Compatibility Tests**:
- All existing API endpoints work identically
- Response formats unchanged
- No breaking changes for frontend
- All ViewSet actions work correctly

### Phase 2 Testing

**Automation Service Tests**:
- Rules execute correctly
- Conditions evaluate correctly
- Actions execute correctly
- Execution limits enforced
- Credit checks work

**Scheduled Tasks Tests**:
- Scheduled tasks run on time
- Credit replenishment works monthly
- Task status tracking works

### Phase 3 Testing

**Site Builder Tests**:
- Site Builder wizard works end-to-end
- Structure generation creates valid blueprints
- Preview renders correctly
- Page generation reuses existing content service
- File management works correctly

**Component Library Tests**:
- Components render correctly
- Components work in Site Builder
- Components work in Sites Renderer

### Phase 4 Testing

**Linker Tests**:
- Linker finds appropriate link candidates
- Links inject correctly into content
- Credit deduction works
- Batch processing works

**Optimizer Tests**:
- Optimizer works from Writer entry point
- Optimizer works from WordPress sync entry point
- Optimizer works from 3rd party sync entry point
- Optimizer works from manual selection
- Content source tracking works
- Pipeline orchestrates correctly

**Backend Test Coverage**:
- 10 test files for Phase 4
- 70+ test cases
- All services tested
- All API endpoints tested

**Frontend Test Coverage**:
- 7 test files for Phase 4
- 30+ test cases
- All components tested
- All pages tested

---

## SUCCESS CRITERIA

### Phase 0 Success Criteria

✅ All existing features work with credit checks  
✅ Credit deduction happens correctly for all operations  
✅ Insufficient credits show clear error messages  
✅ Usage logging tracks all operations  
✅ Frontend shows credit balance, not limits  
✅ Module settings enable/disable modules correctly  
✅ Disabled modules don't appear in UI  
✅ No breaking changes for existing users

### Phase 1 Success Criteria

✅ Services are testable independently  
✅ Business logic extracted from ViewSets  
✅ ViewSets are thin wrappers that delegate to services  
✅ All models moved to business layer  
✅ All imports updated correctly  
✅ Services handle credit checks and business rules

### Phase 2 Success Criteria

✅ Automation rules execute correctly  
✅ Scheduled tasks run on time  
✅ Credit replenishment works monthly  
✅ UI shows automation status  
✅ Rules can be created, edited, deleted  
✅ Execution history is tracked  
✅ All automation respects credit limits

### Phase 3 Success Criteria

✅ Site Builder wizard works end-to-end  
✅ Structure generation creates valid blueprints  
✅ Preview renders correctly  
✅ Page generation reuses existing content service  
✅ File management works correctly  
✅ Shared components work across all apps  
✅ Multiple layouts supported

### Phase 4 Success Criteria

✅ Writer → Linker handover works  
✅ Optimizer works from all entry points  
✅ Content source tracking works  
✅ Pipeline orchestrates correctly  
✅ UI shows content sources and filters  
✅ API endpoints functional and tested  
✅ AI function integrated and working  
✅ Credit deduction working at all stages  
✅ Frontend UI complete with all dashboards and selectors  
✅ Writer integration complete with badges and filters  
✅ Navigation and routing complete  
✅ Backend tests complete (10 test files)  
✅ Frontend tests complete (7 test files)

---

**END OF PHASES 0-4 DOCUMENTATION**