igny8/docs/planning/phases/PHASE-4-LINKER-OPTIMIZER.md

# PHASE 4: LINKER & OPTIMIZER
**Detailed Implementation Plan**

**Goal**: Add linking and optimization as post-processing stages with multiple entry points.

**Timeline**: 4-5 weeks  
**Priority**: MEDIUM  
**Dependencies**: Phase 1

---

## TABLE OF CONTENTS

1. [Overview](#overview)
2. [Content Workflow & Entry Points](#content-workflow--entry-points)
3. [Content Model Extensions](#content-model-extensions)
4. [Linker Implementation](#linker-implementation)
5. [Optimizer Implementation](#optimizer-implementation)
6. [Content Pipeline Service](#content-pipeline-service)
7. [Linker & Optimizer APIs](#linker--optimizer-apis)
8. [Linker & Optimizer UI](#linker--optimizer-ui)
9. [Testing & Validation](#testing--validation)
10. [Implementation Checklist](#implementation-checklist)

---

## OVERVIEW

### Objectives
- ✅ Add internal linking to content
- ✅ Add content optimization
- ✅ Support multiple entry points (Writer, WordPress Sync, 3rd Party, Manual)
- ✅ Create content pipeline service
- ✅ Build UI for linker and optimizer

### Key Principles
- **Multiple Entry Points**: Optimizer works from any content source
- **Unified Content Model**: All content stored in same model with source tracking
- **Pipeline Orchestration**: Linker → Optimizer → Publish workflow
- **Source Agnostic**: Optimizer works on any content regardless of source

---

## CONTENT WORKFLOW & ENTRY POINTS

### 4.0 Content Workflow & Entry Points

**Content Sources**:
1. **IGNY8 Generated** - Content created via Writer module
2. **WordPress Synced** - Content synced from WordPress via plugin
3. **3rd Party Synced** - Content synced from external sources (Shopify, custom APIs)

**Workflow Entry Points**:
```
Entry Point 1: Writer → Linker → Optimizer → Publish
Entry Point 2: WordPress Sync → Optimizer → Publish
Entry Point 3: 3rd Party Sync → Optimizer → Publish
Entry Point 4: Manual Selection → Linker/Optimizer
```

**Content Storage Strategy**:
- All content stored in unified `Content` model
- `source` field: `'igny8'`, `'wordpress'`, `'shopify'`, `'custom'`
- `sync_status` field: `'native'`, `'imported'`, `'synced'`

---

## CONTENT MODEL EXTENSIONS

### 4.1 Content Model Extensions

**Purpose**: Add fields to track content source and sync status.

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Add source field** | `domain/content/models.py` | Phase 1 | Track content source |
| **Add sync_status field** | `domain/content/models.py` | Phase 1 | Track sync status |
| **Add external_id field** | `domain/content/models.py` | Phase 1 | Store external platform ID |
| **Add sync_metadata field** | `domain/content/models.py` | Phase 1 | Store platform-specific metadata |

**Content Model Extensions**:
```python
# domain/content/models.py
class Content(SiteSectorBaseModel):
    # Existing fields...
    
    # NEW: Source tracking
    source = models.CharField(
        max_length=50,
        choices=[
            ('igny8', 'IGNY8 Generated'),
            ('wordpress', 'WordPress Synced'),
            ('shopify', 'Shopify Synced'),
            ('custom', 'Custom API Synced'),
        ],
        default='igny8'
    )
    
    # NEW: Sync status
    sync_status = models.CharField(
        max_length=50,
        choices=[
            ('native', 'Native IGNY8 Content'),
            ('imported', 'Imported from External'),
            ('synced', 'Synced from External'),
        ],
        default='native'
    )
    
    # NEW: External reference
    external_id = models.CharField(max_length=255, blank=True, null=True)
    external_url = models.URLField(blank=True, null=True)
    sync_metadata = models.JSONField(default=dict)
    
    # NEW: Linking fields
    internal_links = models.JSONField(default=list)
    linker_version = models.IntegerField(default=0)
    
    # NEW: Optimization fields
    optimizer_version = models.IntegerField(default=0)
    optimization_scores = models.JSONField(default=dict)
```

---

## LINKER IMPLEMENTATION

### 4.2 Linker Models

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **InternalLink Model** | `domain/linking/models.py` | Phase 1 | Store link relationships |
| **LinkGraph Model** | `domain/linking/models.py` | Phase 1 | Store link graph |

### 4.3 Linker Service

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **LinkerService** | `domain/linking/services/linker_service.py` | Phase 1, ContentService | Main linking service |
| **Link Candidate Engine** | `domain/linking/services/candidate_engine.py` | Phase 1 | Find link candidates |
| **Link Injection Engine** | `domain/linking/services/injection_engine.py` | Phase 1 | Inject links into content |

**LinkerService**:
```python
# domain/linking/services/linker_service.py
class LinkerService:
    def process(self, content_id):
        """Process content for linking"""
        content = Content.objects.get(id=content_id)
        
        # Check credits
        credit_service.check_credits(content.account, 'linking')
        
        # Find link candidates
        candidates = self.candidate_engine.find_candidates(content)
        
        # Inject links
        linked_content = self.injection_engine.inject_links(content, candidates)
        
        # Update content
        content.internal_links = linked_content['links']
        content.linker_version += 1
        content.save()
        
        # Deduct credits
        credit_service.deduct_credits(content.account, 'linking')
        
        return content
```

---

## OPTIMIZER IMPLEMENTATION

### 4.5 Optimizer Models

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **OptimizationTask Model** | `domain/optimization/models.py` | Phase 1 | Store optimization results |
| **OptimizationScores Model** | `domain/optimization/models.py` | Phase 1 | Store optimization scores |

### 4.6 Optimizer Service (Multiple Entry Points)

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **OptimizerService** | `domain/optimization/services/optimizer_service.py` | Phase 1, ContentService | Main optimization service |
| **Content Analyzer** | `domain/optimization/services/analyzer.py` | Phase 1 | Analyze content quality |
| **Optimization AI Function** | `infrastructure/ai/functions/optimize_content.py` | Existing AI framework | AI optimization function |

**OptimizerService**:
```python
# domain/optimization/services/optimizer_service.py
class OptimizerService:
    def optimize_from_writer(self, content_id):
        """Entry Point 1: Writer → Optimizer"""
        content = Content.objects.get(id=content_id, source='igny8')
        return self.optimize(content)
    
    def optimize_from_wordpress_sync(self, content_id):
        """Entry Point 2: WordPress Sync → Optimizer"""
        content = Content.objects.get(id=content_id, source='wordpress')
        return self.optimize(content)
    
    def optimize_from_external_sync(self, content_id):
        """Entry Point 3: External Sync → Optimizer"""
        content = Content.objects.get(id=content_id, source__in=['shopify', 'custom'])
        return self.optimize(content)
    
    def optimize_manual(self, content_id):
        """Entry Point 4: Manual Selection → Optimizer"""
        content = Content.objects.get(id=content_id)
        return self.optimize(content)
    
    def optimize(self, content):
        """Unified optimization logic"""
        # Check credits
        credit_service.check_credits(content.account, 'optimization', content.word_count)
        
        # Analyze content
        scores_before = self.analyzer.analyze(content)
        
        # Optimize content
        optimized = self.ai_function.optimize(content)
        
        # Analyze optimized content
        scores_after = self.analyzer.analyze(optimized)
        
        # Store optimization task
        OptimizationTask.objects.create(
            content=content,
            scores_before=scores_before,
            scores_after=scores_after,
            html_before=content.html_content,
            html_after=optimized['html_content'],
        )
        
        # Update content
        content.optimizer_version += 1
        content.optimization_scores = scores_after
        content.save()
        
        # Deduct credits
        credit_service.deduct_credits(content.account, 'optimization', content.word_count)
        
        return content
```

---

## CONTENT PIPELINE SERVICE

### 4.7 Content Pipeline Service

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **ContentPipelineService** | `domain/content/services/content_pipeline_service.py` | LinkerService, OptimizerService | Orchestrate content pipeline |

**Pipeline Workflow States**:
```
Content States:
- 'draft' → Generated, not processed
- 'linked' → Links added, ready for optimization
- 'optimized' → Optimized, ready for review
- 'review' → Ready for publishing
- 'published' → Published to destination(s)
```

**ContentPipelineService**:
```python
# domain/content/services/content_pipeline_service.py
class ContentPipelineService:
    def process_writer_content(self, content_id, stages=['linking', 'optimization']):
        """Writer → Linker → Optimizer pipeline"""
        content = Content.objects.get(id=content_id, source='igny8')
        
        if 'linking' in stages:
            content = linker_service.process(content.id)
        
        if 'optimization' in stages:
            content = optimizer_service.optimize_from_writer(content.id)
        
        return content
    
    def process_synced_content(self, content_id, stages=['optimization']):
        """Synced Content → Optimizer (skip linking if needed)"""
        content = Content.objects.get(id=content_id)
        
        if 'optimization' in stages:
            content = optimizer_service.optimize_manual(content.id)
        
        return content
```

---

## LINKER & OPTIMIZER APIs

### 4.8 Linker & Optimizer APIs

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Linker ViewSet** | `modules/linker/views.py` | LinkerService | API for linker operations |
| **Optimizer ViewSet** | `modules/optimizer/views.py` | OptimizerService | API for optimizer operations |

---

## LINKER & OPTIMIZER UI

### 4.9 Linker & Optimizer UI

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Linker Dashboard** | `frontend/src/pages/Linker/Dashboard.tsx` | NEW | Linker overview |
| **Optimizer Dashboard** | `frontend/src/pages/Optimizer/Dashboard.tsx` | NEW | Optimizer overview |
| **Content Selection UI** | `frontend/src/components/optimizer/ContentSelector.tsx` | NEW | Select content for optimization |
| **Source Badge Component** | `frontend/src/components/content/SourceBadge.tsx` | NEW | Show content source |

**Optimizer UI Features**:
- Show content source (IGNY8, WordPress, Shopify badge)
- Show sync status (Native, Synced, Imported badge)
- Entry point selection (from Writer, from Sync, Manual)
- Content list with source filters
- "Send to Optimizer" button (works for any source)

### 4.10 Content Filtering & Display

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Content Filter Component** | `frontend/src/components/content/ContentFilter.tsx` | NEW | Filter content by source |
| **Source Filter** | `frontend/src/components/content/SourceFilter.tsx` | NEW | Filter by source |
| **Sync Status Filter** | `frontend/src/components/content/SyncStatusFilter.tsx` | NEW | Filter by sync status |

---

## TESTING & VALIDATION

### 4.11 Testing

**Test Cases**:
- ✅ Writer → Linker handover works
- ✅ Linker finds appropriate link candidates
- ✅ Links inject correctly into content
- ✅ Optimizer works from Writer entry point
- ✅ Optimizer works from WordPress sync entry point
- ✅ Optimizer works from 3rd party sync entry point
- ✅ Optimizer works from manual selection
- ✅ Synced content stored correctly with source flags
- ✅ Content filtering works (by source, sync_status)
- ✅ Pipeline orchestrates correctly
- ✅ All entry points use same optimization logic

---

## IMPLEMENTATION CHECKLIST

### Backend Tasks

- [ ] Extend Content model with source/sync fields
- [ ] Create `domain/linking/models.py`
- [ ] Create LinkerService
- [ ] Create `domain/optimization/models.py`
- [ ] Create OptimizerService
- [ ] Create optimization AI function
- [ ] Create ContentPipelineService
- [ ] Create Linker ViewSet
- [ ] Create Optimizer ViewSet
- [ ] Create content sync service (for Phase 6)

### Frontend Tasks

- [ ] Create Linker Dashboard
- [ ] Create Optimizer Dashboard
- [ ] Create content selection UI
- [ ] Create source badge component
- [ ] Create content filters
- [ ] Update content list with filters

---

## SUCCESS CRITERIA

- ✅ Writer → Linker handover works
- ✅ Optimizer works from all entry points
- ✅ Content source tracking works
- ✅ Pipeline orchestrates correctly
- ✅ UI shows content sources and filters

---

**END OF PHASE 4 DOCUMENT**