Files
igny8/docs/planning/PHASE-3-4-IMPLEMENTATION-PLAN.md
IGNY8 VPS (Salman) 9930728e8a Add source tracking and sync status fields to Content model; update services module
- Introduced new fields in the Content model for source tracking and sync status, including external references and optimization fields.
- Updated the services module to include new content generation and pipeline services for better organization and clarity.
2025-11-17 11:15:15 +00:00

867 lines
25 KiB
Markdown

# PHASE 3 & 4 IMPLEMENTATION PLAN
**Detailed Configuration Plan for Site Builder & Linker/Optimizer**
**Created**: 2025-01-XX
**Status**: Planning Phase
---
## TABLE OF CONTENTS
1. [Overview](#overview)
2. [Phase 3: Site Builder Implementation Plan](#phase-3-site-builder-implementation-plan)
3. [Phase 4: Linker & Optimizer Implementation Plan](#phase-4-linker--optimizer-implementation-plan)
4. [Integration Points](#integration-points)
5. [File Structure](#file-structure)
6. [Dependencies & Order](#dependencies--order)
7. [Testing Strategy](#testing-strategy)
---
## OVERVIEW
### Implementation Approach
- **Phase 3**: Build Site Builder with wizard, AI structure generation, and file management
- **Phase 4**: Implement Linker and Optimizer as post-processing stages with multiple entry points
- **Shared Components**: Create global component library for reuse across apps
- **Integration**: Ensure seamless integration with existing Phase 1 & 2 services
### Key Principles
- **Service Layer Pattern**: All business logic in services (Phase 1 pattern)
- **Credit-Aware**: All operations check credits before execution
- **Multiple Entry Points**: Optimizer works from Writer, WordPress sync, 3rd party, manual
- **Component Reuse**: Shared components across Site Builder, Sites Renderer, Main App
---
## PHASE 3: SITE BUILDER IMPLEMENTATION PLAN
### 3.1 Backend Structure
#### Business Layer (`business/site_building/`)
```
business/site_building/
├── __init__.py
├── models.py # SiteBlueprint, PageBlueprint
├── migrations/
│ └── 0001_initial.py
└── services/
├── __init__.py
├── file_management_service.py
├── structure_generation_service.py
└── page_generation_service.py
```
**Models to Create**:
1. **SiteBlueprint** (`business/site_building/models.py`)
- Fields:
- `name`, `description`
- `config_json` (wizard choices: business_type, style, objectives)
- `structure_json` (AI-generated structure: pages, layout, theme)
- `status` (draft, generating, ready, deployed)
- `hosting_type` (igny8_sites, wordpress, shopify, multi)
- `version`, `deployed_version`
- Inherits from `SiteSectorBaseModel`
2. **PageBlueprint** (`business/site_building/models.py`)
- Fields:
- `site_blueprint` (ForeignKey)
- `slug`, `title`
- `type` (home, about, services, products, blog, contact, custom)
- `blocks_json` (page content blocks)
- `status` (draft, generating, ready)
- `order`
- Inherits from `SiteSectorBaseModel`
#### Services to Create
1. **FileManagementService** (`business/site_building/services/file_management_service.py`)
```python
class SiteBuilderFileService:
def get_user_accessible_sites(self, user)
def check_file_access(self, user, site_id)
def upload_file(self, user, site_id, file, folder='images')
def delete_file(self, user, site_id, file_path)
def list_files(self, user, site_id, folder='images')
def check_storage_quota(self, site_id, file_size)
```
2. **StructureGenerationService** (`business/site_building/services/structure_generation_service.py`)
```python
class StructureGenerationService:
def __init__(self):
self.ai_function = GenerateSiteStructureFunction()
self.credit_service = CreditService()
def generate_structure(self, site_blueprint, business_brief, objectives, style)
def _create_page_blueprints(self, site_blueprint, structure)
```
3. **PageGenerationService** (`business/site_building/services/page_generation_service.py`)
```python
class PageGenerationService:
def __init__(self):
self.content_service = ContentGenerationService()
self.credit_service = CreditService()
def generate_page_content(self, page_blueprint, account)
def regenerate_page(self, page_blueprint, account)
```
#### AI Functions (`infrastructure/ai/functions/`)
1. **GenerateSiteStructureFunction** (`infrastructure/ai/functions/generate_site_structure.py`)
- Operation type: `site_structure_generation`
- Credit cost: 50 credits (from constants)
- Generates site structure JSON from business brief
#### API Layer (`modules/site_builder/`)
```
modules/site_builder/
├── __init__.py
├── views.py # SiteBuilderViewSet, PageBlueprintViewSet, FileUploadView
├── serializers.py # SiteBlueprintSerializer, PageBlueprintSerializer
├── urls.py
└── apps.py
```
**ViewSets to Create**:
1. **SiteBuilderViewSet** (`modules/site_builder/views.py`)
- CRUD for SiteBlueprint
- Actions:
- `generate_structure/` (POST) - Trigger AI structure generation
- `deploy/` (POST) - Deploy site to hosting
- `preview/` (GET) - Get preview JSON
2. **PageBlueprintViewSet** (`modules/site_builder/views.py`)
- CRUD for PageBlueprint
- Actions:
- `generate_content/` (POST) - Generate page content
- `regenerate/` (POST) - Regenerate page content
3. **FileUploadView** (`modules/site_builder/views.py`)
- `upload/` (POST) - Upload file to site assets
- `delete/` (DELETE) - Delete file
- `list/` (GET) - List files
#### File Storage Structure
```
/data/app/sites-data/
└── clients/
└── {site_id}/
└── v{version}/
├── site.json # Site definition
├── pages/ # Page definitions
│ ├── home.json
│ ├── about.json
│ └── ...
└── assets/ # User-managed files
├── images/
├── documents/
└── media/
```
### 3.2 Frontend Structure
#### Site Builder Container (`site-builder/`)
```
site-builder/
├── src/
│ ├── pages/
│ │ ├── wizard/
│ │ │ ├── Step1TypeSelection.tsx
│ │ │ ├── Step2BusinessBrief.tsx
│ │ │ ├── Step3Objectives.tsx
│ │ │ └── Step4Style.tsx
│ │ ├── preview/
│ │ │ └── PreviewCanvas.tsx
│ │ └── dashboard/
│ │ └── SiteList.tsx
│ ├── components/
│ │ ├── blocks/ # Block components (import from shared)
│ │ ├── forms/
│ │ ├── files/
│ │ │ └── FileBrowser.tsx
│ │ └── preview-canvas/
│ ├── state/
│ │ ├── builderStore.ts
│ │ └── siteDefinitionStore.ts
│ ├── api/
│ │ ├── builder.api.ts
│ │ └── sites.api.ts
│ └── main.tsx
├── package.json
├── vite.config.ts
└── Dockerfile
```
#### Shared Component Library (`frontend/src/components/shared/`)
```
frontend/src/components/shared/
├── blocks/
│ ├── Hero.tsx
│ ├── Features.tsx
│ ├── Services.tsx
│ ├── Products.tsx
│ ├── Testimonials.tsx
│ ├── ContactForm.tsx
│ └── ...
├── layouts/
│ ├── DefaultLayout.tsx
│ ├── MinimalLayout.tsx
│ ├── MagazineLayout.tsx
│ ├── EcommerceLayout.tsx
│ ├── PortfolioLayout.tsx
│ ├── BlogLayout.tsx
│ └── CorporateLayout.tsx
└── templates/
├── BlogTemplate.tsx
├── BusinessTemplate.tsx
└── PortfolioTemplate.tsx
```
### 3.3 Implementation Tasks
#### Backend Tasks (Priority Order)
1. **Create Business Models**
- [ ] Create `business/site_building/` folder
- [ ] Create `SiteBlueprint` model
- [ ] Create `PageBlueprint` model
- [ ] Create migrations
2. **Create Services**
- [ ] Create `FileManagementService`
- [ ] Create `StructureGenerationService`
- [ ] Create `PageGenerationService`
- [ ] Integrate with `CreditService`
3. **Create AI Function**
- [ ] Create `GenerateSiteStructureFunction`
- [ ] Add prompts for site structure generation
- [ ] Test AI function
4. **Create API Layer**
- [ ] Create `modules/site_builder/` folder
- [ ] Create `SiteBuilderViewSet`
- [ ] Create `PageBlueprintViewSet`
- [ ] Create `FileUploadView`
- [ ] Create serializers
- [ ] Register URLs
#### Frontend Tasks (Priority Order)
1. **Create Site Builder Container**
- [ ] Create `site-builder/` folder structure
- [ ] Set up Vite + React + TypeScript
- [ ] Configure Docker container
- [ ] Set up routing
2. **Create Wizard**
- [ ] Step 1: Type Selection
- [ ] Step 2: Business Brief
- [ ] Step 3: Objectives
- [ ] Step 4: Style Preferences
- [ ] Wizard state management
3. **Create Preview Canvas**
- [ ] Preview renderer
- [ ] Block rendering
- [ ] Layout rendering
4. **Create Shared Components**
- [ ] Block components
- [ ] Layout components
- [ ] Template components
---
## PHASE 4: LINKER & OPTIMIZER IMPLEMENTATION PLAN
### 4.1 Backend Structure
#### Business Layer
```
business/
├── linking/
│ ├── __init__.py
│ ├── models.py # InternalLink (optional)
│ └── services/
│ ├── __init__.py
│ ├── linker_service.py
│ ├── candidate_engine.py
│ └── injection_engine.py
├── optimization/
│ ├── __init__.py
│ ├── models.py # OptimizationTask, OptimizationScores
│ └── services/
│ ├── __init__.py
│ ├── optimizer_service.py
│ └── analyzer.py
└── content/
└── services/
└── content_pipeline_service.py # NEW: Orchestrates pipeline
```
#### Content Model Extensions
**Extend `business/content/models.py`**:
```python
class Content(SiteSectorBaseModel):
# Existing fields...
# NEW: Source tracking (Phase 4)
source = models.CharField(
max_length=50,
choices=[
('igny8', 'IGNY8 Generated'),
('wordpress', 'WordPress Synced'),
('shopify', 'Shopify Synced'),
('custom', 'Custom API Synced'),
],
default='igny8'
)
sync_status = models.CharField(
max_length=50,
choices=[
('native', 'Native IGNY8 Content'),
('imported', 'Imported from External'),
('synced', 'Synced from External'),
],
default='native'
)
external_id = models.CharField(max_length=255, blank=True, null=True)
external_url = models.URLField(blank=True, null=True)
sync_metadata = models.JSONField(default=dict)
# NEW: Linking fields
internal_links = models.JSONField(default=list)
linker_version = models.IntegerField(default=0)
# NEW: Optimization fields
optimizer_version = models.IntegerField(default=0)
optimization_scores = models.JSONField(default=dict)
```
#### Models to Create
1. **OptimizationTask** (`business/optimization/models.py`)
- Fields:
- `content` (ForeignKey to Content)
- `scores_before`, `scores_after` (JSON)
- `html_before`, `html_after` (Text)
- `status` (pending, completed, failed)
- `credits_used`
- Inherits from `AccountBaseModel`
2. **OptimizationScores** (`business/optimization/models.py`) - Optional
- Store detailed scoring metrics
#### Services to Create
1. **LinkerService** (`business/linking/services/linker_service.py`)
```python
class LinkerService:
def __init__(self):
self.candidate_engine = CandidateEngine()
self.injection_engine = InjectionEngine()
self.credit_service = CreditService()
def process(self, content_id)
def batch_process(self, content_ids)
```
2. **CandidateEngine** (`business/linking/services/candidate_engine.py`)
```python
class CandidateEngine:
def find_candidates(self, content)
def _find_relevant_content(self, content)
def _score_candidates(self, content, candidates)
```
3. **InjectionEngine** (`business/linking/services/injection_engine.py`)
```python
class InjectionEngine:
def inject_links(self, content, candidates)
def _inject_link_into_html(self, html, link_data)
```
4. **OptimizerService** (`business/optimization/services/optimizer_service.py`)
```python
class OptimizerService:
def __init__(self):
self.analyzer = ContentAnalyzer()
self.ai_function = OptimizeContentFunction()
self.credit_service = CreditService()
# Multiple entry points
def optimize_from_writer(self, content_id)
def optimize_from_wordpress_sync(self, content_id)
def optimize_from_external_sync(self, content_id)
def optimize_manual(self, content_id)
# Unified optimization logic
def optimize(self, content)
```
5. **ContentAnalyzer** (`business/optimization/services/analyzer.py`)
```python
class ContentAnalyzer:
def analyze(self, content)
def _calculate_seo_score(self, content)
def _calculate_readability_score(self, content)
def _calculate_engagement_score(self, content)
```
6. **ContentPipelineService** (`business/content/services/content_pipeline_service.py`)
```python
class ContentPipelineService:
def __init__(self):
self.linker_service = LinkerService()
self.optimizer_service = OptimizerService()
def process_writer_content(self, content_id, stages=['linking', 'optimization'])
def process_synced_content(self, content_id, stages=['optimization'])
```
#### AI Functions
1. **OptimizeContentFunction** (`infrastructure/ai/functions/optimize_content.py`)
- Operation type: `optimization`
- Credit cost: 1 credit per 200 words
- Optimizes content for SEO, readability, engagement
#### API Layer (`modules/linker/` and `modules/optimizer/`)
```
modules/
├── linker/
│ ├── __init__.py
│ ├── views.py # LinkerViewSet
│ ├── serializers.py
│ ├── urls.py
│ └── apps.py
└── optimizer/
├── __init__.py
├── views.py # OptimizerViewSet
├── serializers.py
├── urls.py
└── apps.py
```
**ViewSets to Create**:
1. **LinkerViewSet** (`modules/linker/views.py`)
- Actions:
- `process/` (POST) - Process content for linking
- `batch_process/` (POST) - Process multiple content items
2. **OptimizerViewSet** (`modules/optimizer/views.py`)
- Actions:
- `optimize/` (POST) - Optimize content (auto-detects source)
- `optimize_from_writer/` (POST) - Entry point 1
- `optimize_from_sync/` (POST) - Entry point 2 & 3
- `optimize_manual/` (POST) - Entry point 4
- `analyze/` (GET) - Analyze content without optimizing
### 4.2 Frontend Structure
#### Linker UI (`frontend/src/pages/Linker/`)
```
frontend/src/pages/Linker/
├── Dashboard.tsx
├── ContentList.tsx
└── LinkResults.tsx
```
#### Optimizer UI (`frontend/src/pages/Optimizer/`)
```
frontend/src/pages/Optimizer/
├── Dashboard.tsx
├── ContentSelector.tsx
├── OptimizationResults.tsx
└── ScoreComparison.tsx
```
#### Shared Components
```
frontend/src/components/
├── content/
│ ├── SourceBadge.tsx # Show content source (IGNY8, WordPress, etc.)
│ ├── SyncStatusBadge.tsx # Show sync status
│ ├── ContentFilter.tsx # Filter by source, sync_status
│ └── SourceFilter.tsx
```
### 4.3 Implementation Tasks
#### Backend Tasks (Priority Order)
1. **Extend Content Model**
- [ ] Add `source` field
- [ ] Add `sync_status` field
- [ ] Add `external_id`, `external_url`, `sync_metadata`
- [ ] Add `internal_links`, `linker_version`
- [ ] Add `optimizer_version`, `optimization_scores`
- [ ] Create migration
2. **Create Linking Services**
- [ ] Create `business/linking/` folder
- [ ] Create `LinkerService`
- [ ] Create `CandidateEngine`
- [ ] Create `InjectionEngine`
- [ ] Integrate with `CreditService`
3. **Create Optimization Services**
- [ ] Create `business/optimization/` folder
- [ ] Create `OptimizationTask` model
- [ ] Create `OptimizerService` (with multiple entry points)
- [ ] Create `ContentAnalyzer`
- [ ] Integrate with `CreditService`
4. **Create AI Function**
- [ ] Create `OptimizeContentFunction`
- [ ] Add optimization prompts
- [ ] Test AI function
5. **Create Pipeline Service**
- [ ] Create `ContentPipelineService`
- [ ] Integrate Linker and Optimizer
6. **Create API Layer**
- [ ] Create `modules/linker/` folder
- [ ] Create `LinkerViewSet`
- [ ] Create `modules/optimizer/` folder
- [ ] Create `OptimizerViewSet`
- [ ] Create serializers
- [ ] Register URLs
#### Frontend Tasks (Priority Order)
1. **Create Linker UI**
- [ ] Linker Dashboard
- [ ] Content List
- [ ] Link Results display
2. **Create Optimizer UI**
- [ ] Optimizer Dashboard
- [ ] Content Selector (with source filters)
- [ ] Optimization Results
- [ ] Score Comparison
3. **Create Shared Components**
- [ ] SourceBadge component
- [ ] SyncStatusBadge component
- [ ] ContentFilter component
- [ ] SourceFilter component
4. **Update Content List**
- [ ] Add source badges
- [ ] Add sync status badges
- [ ] Add filters (by source, sync_status)
- [ ] Add "Send to Optimizer" button
---
## INTEGRATION POINTS
### Phase 3 Integration
1. **With Phase 1 Services**
- `StructureGenerationService` uses `CreditService`
- `PageGenerationService` uses `ContentGenerationService`
- All operations check credits before execution
2. **With Phase 2 Automation**
- Automation rules can trigger site structure generation
- Automation can deploy sites automatically
3. **With Content Service**
- Page generation reuses `ContentGenerationService`
- Site pages stored as `Content` records
### Phase 4 Integration
1. **With Phase 1 Services**
- `LinkerService` uses `CreditService`
- `OptimizerService` uses `CreditService`
- `ContentPipelineService` orchestrates services
2. **With Writer Module**
- Writer → Linker → Optimizer pipeline
- Content generated in Writer flows to Linker/Optimizer
3. **With WordPress Sync** (Phase 6)
- WordPress content synced with `source='wordpress'`
- Optimizer works on synced content
4. **With 3rd Party Sync** (Phase 6)
- External content synced with `source='shopify'` or `source='custom'`
- Optimizer works on all sources
### Cross-Phase Integration
1. **Site Builder → Linker/Optimizer**
- Site pages can be optimized
- Site content can be linked internally
2. **Content Pipeline**
- Unified pipeline: Writer → Linker → Optimizer → Publish
- Works for all content sources
---
## FILE STRUCTURE
### Complete Backend Structure
```
backend/igny8_core/
├── business/
│ ├── automation/ # Phase 2 ✅
│ ├── billing/ # Phase 0, 1 ✅
│ ├── content/ # Phase 1 ✅
│ │ └── services/
│ │ └── content_pipeline_service.py # Phase 4 NEW
│ ├── linking/ # Phase 4 NEW
│ │ ├── models.py
│ │ └── services/
│ │ ├── linker_service.py
│ │ ├── candidate_engine.py
│ │ └── injection_engine.py
│ ├── optimization/ # Phase 4 NEW
│ │ ├── models.py
│ │ └── services/
│ │ ├── optimizer_service.py
│ │ └── analyzer.py
│ ├── planning/ # Phase 1 ✅
│ └── site_building/ # Phase 3 NEW
│ ├── models.py
│ └── services/
│ ├── file_management_service.py
│ ├── structure_generation_service.py
│ └── page_generation_service.py
├── modules/
│ ├── automation/ # Phase 2 ✅
│ ├── billing/ # Phase 0, 1 ✅
│ ├── linker/ # Phase 4 NEW
│ ├── optimizer/ # Phase 4 NEW
│ ├── planner/ # Phase 1 ✅
│ ├── site_builder/ # Phase 3 NEW
│ └── writer/ # Phase 1 ✅
└── infrastructure/
└── ai/
└── functions/
├── generate_site_structure.py # Phase 3 NEW
└── optimize_content.py # Phase 4 NEW
```
### Complete Frontend Structure
```
frontend/src/
├── components/
│ ├── shared/ # Phase 3 NEW
│ │ ├── blocks/
│ │ ├── layouts/
│ │ └── templates/
│ └── content/ # Phase 4 NEW
│ ├── SourceBadge.tsx
│ ├── SyncStatusBadge.tsx
│ └── ContentFilter.tsx
└── pages/
├── Linker/ # Phase 4 NEW
│ ├── Dashboard.tsx
│ └── ContentList.tsx
├── Optimizer/ # Phase 4 NEW
│ ├── Dashboard.tsx
│ └── ContentSelector.tsx
└── Writer/ # Phase 1 ✅
└── ...
site-builder/src/ # Phase 3 NEW
├── pages/
│ ├── wizard/
│ ├── preview/
│ └── dashboard/
└── components/
```
---
## DEPENDENCIES & ORDER
### Phase 3 Dependencies
1. **Required (Already Complete)**
- ✅ Phase 0: Credit system
- ✅ Phase 1: Service layer (ContentGenerationService, CreditService)
- ✅ Phase 2: Automation system (optional integration)
2. **Phase 3 Implementation Order**
- Step 1: Create models (SiteBlueprint, PageBlueprint)
- Step 2: Create FileManagementService
- Step 3: Create StructureGenerationService + AI function
- Step 4: Create PageGenerationService
- Step 5: Create API layer (ViewSets)
- Step 6: Create frontend container structure
- Step 7: Create wizard UI
- Step 8: Create preview canvas
- Step 9: Create shared component library
### Phase 4 Dependencies
1. **Required (Already Complete)**
- ✅ Phase 0: Credit system
- ✅ Phase 1: Service layer (ContentGenerationService, CreditService)
- ✅ Content model (needs extension)
2. **Phase 4 Implementation Order**
- Step 1: Extend Content model (add source, sync_status, linking, optimization fields)
- Step 2: Create linking services (LinkerService, CandidateEngine, InjectionEngine)
- Step 3: Create optimization services (OptimizerService, ContentAnalyzer)
- Step 4: Create optimization AI function
- Step 5: Create ContentPipelineService
- Step 6: Create API layer (LinkerViewSet, OptimizerViewSet)
- Step 7: Create frontend UI (Linker Dashboard, Optimizer Dashboard)
- Step 8: Create shared components (SourceBadge, ContentFilter)
### Parallel Implementation
- **Phase 3 and Phase 4 can be implemented in parallel** after:
- Content model extensions (Phase 4 Step 1) are complete
- Both phases use Phase 1 services independently
---
## TESTING STRATEGY
### Phase 3 Testing
1. **Backend Tests**
- Test SiteBlueprint CRUD
- Test PageBlueprint CRUD
- Test structure generation (AI function)
- Test file upload/delete/access
- Test credit deduction
2. **Frontend Tests**
- Test wizard flow
- Test preview rendering
- Test file browser
- Test component library
### Phase 4 Testing
1. **Backend Tests**
- Test Content model extensions
- Test LinkerService (find candidates, inject links)
- Test OptimizerService (all entry points)
- Test ContentPipelineService
- Test credit deduction
2. **Frontend Tests**
- Test Linker UI
- Test Optimizer UI
- Test source filtering
- Test content selection
### Integration Tests
1. **Writer → Linker → Optimizer Pipeline**
- Test full pipeline flow
- Test credit deduction at each stage
2. **WordPress Sync → Optimizer** (Phase 6)
- Test synced content optimization
- Test source tracking
---
## CREDIT COSTS
### Phase 3 Credit Costs
- `site_structure_generation`: 50 credits (per site blueprint)
- `site_page_generation`: 20 credits (per page)
- File storage: No credits (storage quota based)
### Phase 4 Credit Costs
- `linking`: 8 credits (per content piece)
- `optimization`: 1 credit per 200 words
---
## SUCCESS CRITERIA
### Phase 3 Success Criteria
- ✅ Site Builder wizard works end-to-end
- ✅ AI structure generation creates valid blueprints
- ✅ Preview renders correctly
- ✅ File management works
- ✅ Shared components work across apps
- ✅ Page generation reuses ContentGenerationService
### Phase 4 Success Criteria
- ✅ Linker finds appropriate link candidates
- ✅ Links inject correctly into content
- ✅ Optimizer works from all entry points (Writer, WordPress, 3rd party, Manual)
- ✅ Content source tracking works
- ✅ Pipeline orchestrates correctly
- ✅ UI shows content sources and filters
---
## RISK MITIGATION
### Phase 3 Risks
1. **AI Structure Generation Quality**
- Mitigation: Prompt engineering, validation, user feedback loop
2. **Component Compatibility**
- Mitigation: Shared component library, comprehensive testing
3. **File Management Security**
- Mitigation: Access control, validation, quota checks
### Phase 4 Risks
1. **Link Quality**
- Mitigation: Candidate scoring algorithm, relevance checks
2. **Optimization Quality**
- Mitigation: Content analysis, before/after comparison, user review
3. **Multiple Entry Points Complexity**
- Mitigation: Unified optimization logic, clear entry point methods
---
**END OF IMPLEMENTATION PLAN**