igny8/docs/planning/phases/PHASE-9-AI-FRAMEWORK-SITE-BUILDER-INTEGRATION.md

PHASE 9: AI FRAMEWORK & SITE BUILDER INTEGRATION

Detailed Implementation Plan

Goal: Complete AI framework integration for Site Builder, add prompt management UI, and implement blueprint-to-writer task queuing.

Timeline: 2-3 weeks
Priority: MEDIUM
Dependencies: Phase 3

---

TABLE OF CONTENTS

1. Overview
2. AI Framework Integration
3. Prompt Management UI
4. Response Handling & Structure Generation
5. Blueprint-to-Writer Task Queuing
6. Testing & Validation
7. Implementation Checklist

---

OVERVIEW

Objectives

• ✅ Add site_structure_generation prompt to Thinker /prompts UI
• ✅ Document AI framework integration for Site Builder
• ✅ Implement response handling for structure generation
• ✅ Enable blueprint pages to queue as Writer tasks (similar to ideas)
• ✅ Add bulk page content generation from blueprint

Key Principles

• Unified Prompt Management: Site Builder prompts managed in same UI as other prompts
• Reuse Existing Patterns: Follow same queuing pattern as ideas → writer tasks
• AI Framework Compliance: Use existing BaseAIFunction and AIEngine patterns
• Response Validation: Robust parsing and validation of AI responses

---

AI FRAMEWORK INTEGRATION

9.1 AI Framework Integration

Purpose: Document and ensure proper integration with existing AI framework.

Current AI Framework Architecture

The Site Builder uses the existing AI framework:

AIEngine
  ├─ BaseAIFunction (interface)
  │   ├─ validate()
  │   ├─ prepare()
  │   ├─ build_prompt()
  │   ├─ parse_response()
  │   └─ save_output()
  │
  └─ GenerateSiteStructureFunction
      ├─ Uses PromptRegistry for prompts
      ├─ Integrates with CreditService
      └─ Saves to SiteBlueprint + PageBlueprint models

AI Function Lifecycle

Task	File	Dependencies	Implementation
AI Function Registration	ai/registry.py	Existing	Register generate_site_structure
Prompt Integration	ai/prompts.py	Existing	Add site_structure_generation to DEFAULT_PROMPTS
Credit Cost Configuration	business/billing/services/credit_service.py	Phase 0	Add site_structure_generation operation type

AI Function Flow:

# infrastructure/ai/functions/generate_site_structure.py
class GenerateSiteStructureFunction(BaseAIFunction):
    def get_name(self) -> str:
        return 'generate_site_structure'
    
    def validate(self, payload, account=None):
        # Validates blueprint ID exists
        # Returns {'valid': True/False, 'error': '...'}
    
    def prepare(self, payload, account=None):
        # Loads SiteBlueprint
        # Extracts business_brief, objectives, style
        # Returns context dict
    
    def build_prompt(self, data, account=None):
        # Uses PromptRegistry.get_prompt('site_structure_generation')
        # Injects: BUSINESS_BRIEF, OBJECTIVES, STYLE, SITE_INFO
        # Returns formatted prompt string
    
    def parse_response(self, response, step_tracker=None):
        # Parses JSON from AI response
        # Handles noisy responses (extracts JSON from text)
        # Validates structure is dict
        # Returns parsed structure dict
    
    def save_output(self, parsed, original_data, account=None):
        # Saves structure_json to SiteBlueprint
        # Syncs PageBlueprint records (create/update/delete)
        # Updates blueprint status to 'ready'
        # Returns summary dict

Prompt Variable Injection

The prompt system supports variable injection:

Available Variables:

• [IGNY8_BUSINESS_BRIEF] - Business description
• [IGNY8_OBJECTIVES] - List of site objectives
• [IGNY8_STYLE] - Style preferences JSON
• [IGNY8_SITE_INFO] - Site metadata JSON

Prompt Template (from ai/prompts.py):

'site_structure_generation': """You are a senior UX architect...
BUSINESS BRIEF:
[IGNY8_BUSINESS_BRIEF]

PRIMARY OBJECTIVES:
[IGNY8_OBJECTIVES]

STYLE & BRAND NOTES:
[IGNY8_STYLE]

SITE INFO:
[IGNY8_SITE_INFO]
...
"""

---

PROMPT MANAGEMENT UI

9.2 Prompt Management UI

Purpose: Add Site Builder prompt to Thinker /prompts interface.

Add Prompt Type to Backend

Task	File	Dependencies	Implementation
Add prompt type choice	modules/system/models.py	Existing	Add 'site_structure_generation' to PROMPT_TYPE_CHOICES

AIPrompt Model Update:

# modules/system/models.py
class AIPrompt(AccountBaseModel):
    PROMPT_TYPE_CHOICES = [
        ('clustering', 'Clustering'),
        ('ideas', 'Ideas Generation'),
        ('content_generation', 'Content Generation'),
        ('image_prompt_extraction', 'Image Prompt Extraction'),
        ('image_prompt_template', 'Image Prompt Template'),
        ('negative_prompt', 'Negative Prompt'),
        ('site_structure_generation', 'Site Structure Generation'),  # NEW
    ]

Add Prompt Type to Frontend

Task	File	Dependencies	Implementation
Add prompt type config	frontend/src/pages/Thinker/Prompts.tsx	Existing	Add site_structure_generation to PROMPT_TYPES array

Frontend Prompt Type Config:

// frontend/src/pages/Thinker/Prompts.tsx
const PROMPT_TYPES = [
  // ... existing prompts
  {
    key: 'site_structure_generation',
    label: 'Site Structure Generation',
    description: 'Generate site structure from business brief. Use [IGNY8_BUSINESS_BRIEF], [IGNY8_OBJECTIVES], [IGNY8_STYLE], and [IGNY8_SITE_INFO] to inject data.',
    icon: '🏗️',
    color: 'teal',
  },
];

Prompt UI Section

The prompt will appear in a new "Site Builder" section in the Thinker Prompts page:

{/* Site Builder Prompts Section */}
<div className="mb-8">
  <div className="mb-4">
    <h2 className="text-xl font-semibold text-gray-800 dark:text-white">
      Site Builder
    </h2>
    <p className="text-sm text-gray-600 dark:text-gray-400">
      Configure prompts for AI-powered site structure generation
    </p>
  </div>
  
  {/* Site Structure Generation Prompt */}
  <PromptEditor
    promptType="site_structure_generation"
    label="Site Structure Generation Prompt"
    description="Generate complete site structure (pages, blocks, navigation) from business brief"
    variables={[
      '[IGNY8_BUSINESS_BRIEF]',
      '[IGNY8_OBJECTIVES]',
      '[IGNY8_STYLE]',
      '[IGNY8_SITE_INFO]'
    ]}
  />
</div>

Migration for Prompt Type

Task	File	Dependencies	Implementation
Create migration	modules/system/migrations/XXXX_add_site_structure_prompt_type.py	Existing	Add new choice to PROMPT_TYPE_CHOICES

---

RESPONSE HANDLING & STRUCTURE GENERATION

9.3 Response Handling & Structure Generation

Purpose: Document and enhance AI response parsing and structure generation.

Response Format

The AI returns a JSON structure:

{
  "site": {
    "name": "Acme Robotics",
    "primary_navigation": ["home", "services", "about", "contact"],
    "secondary_navigation": ["blog", "faq"],
    "hero_message": "Transform your warehouse operations",
    "tone": "Confident, professional, innovative"
  },
  "pages": [
    {
      "slug": "home",
      "title": "Home",
      "type": "home",
      "status": "draft",
      "objective": "Convert visitors to demo requests",
      "primary_cta": "Book a demo",
      "blocks": [
        {
          "type": "hero",
          "heading": "Warehouse Automation That Scales",
          "subheading": "AI-powered robotics for modern fulfillment",
          "layout": "full-width",
          "content": ["Hero section with CTA"]
        },
        {
          "type": "features",
          "heading": "Why Choose Us",
          "subheading": "Industry-leading solutions",
          "layout": "three-column",
          "content": ["Feature 1", "Feature 2", "Feature 3"]
        }
      ]
    }
  ]
}

Response Parsing Flow

Task	File	Dependencies	Implementation
Parse Response	ai/functions/generate_site_structure.py	Existing	parse_response() method
Extract JSON	ai/functions/generate_site_structure.py	Existing	_extract_json_object() helper
Validate Structure	ai/functions/generate_site_structure.py	Existing	_ensure_dict() validation

Parsing Logic:

def parse_response(self, response: str, step_tracker=None) -> Dict[str, Any]:
    """
    Parse AI response into structure dict.
    
    Handles:
    1. Direct JSON response
    2. JSON wrapped in text/markdown
    3. Noisy responses with commentary
    """
    response = response.strip()
    
    # Try direct JSON parse
    try:
        return self._ensure_dict(json.loads(response))
    except json.JSONDecodeError:
        # Extract JSON from text
        cleaned = self._extract_json_object(response)
        if cleaned:
            return self._ensure_dict(json.loads(cleaned))
        raise ValueError("Unable to parse AI response into JSON")

Structure Persistence

Task	File	Dependencies	Implementation
Save Structure	ai/functions/generate_site_structure.py	Existing	save_output() method
Sync Page Blueprints	ai/functions/generate_site_structure.py	Existing	_sync_page_blueprints() method

Persistence Flow:

def save_output(self, parsed, original_data, account=None):
    """
    Save structure and sync page blueprints.
    
    1. Save structure_json to SiteBlueprint
    2. Create/update PageBlueprint records
    3. Delete pages not in new structure
    4. Update blueprint status to 'ready'
    """
    blueprint = original_data['blueprint']
    structure = self._ensure_dict(parsed)
    pages = structure.get('pages', [])
    
    # Save structure
    blueprint.structure_json = structure
    blueprint.status = 'ready'
    blueprint.save()
    
    # Sync pages (create/update/delete)
    created, updated, deleted = self._sync_page_blueprints(blueprint, pages)
    
    return {
        'success': True,
        'pages_created': created,
        'pages_updated': updated,
        'pages_deleted': deleted,
    }

Error Handling

Error Scenarios:

1. Invalid JSON: Extract JSON from text, fallback to error
2. Missing Required Fields: Validate structure has site and pages
3. Invalid Page Types: Map to allowed types, default to 'custom'
4. Duplicate Slugs: Use update_or_create to handle conflicts

Error Response Format:

{
    'success': False,
    'error': 'Error message',
    'error_type': 'ParseError' | 'ValidationError' | 'SaveError'
}

---

BLUEPRINT-TO-WRITER TASK QUEUING

9.4 Blueprint-to-Writer Task Queuing

Purpose: Enable blueprint pages to be queued as Writer tasks, similar to how ideas are queued.

Current Pattern (Ideas → Writer)

The existing pattern for ideas:

ContentIdeas → Writer Tasks → Content Generation

Flow:

1. User selects ideas in Planner
2. Click "Generate Content"
3. Creates Tasks records for each idea
4. Queues generate_content AI task
5. Generates HTML content

New Pattern (Blueprint Pages → Writer)

The new pattern for blueprint pages:

PageBlueprint → Writer Tasks → Content Generation

Flow:

1. User has generated SiteBlueprint with PageBlueprints
2. User selects pages (or all pages)
3. Click "Generate Content for Pages"
4. Creates Tasks records for each page
5. Queues generate_content AI task
6. Generates HTML content

Bulk Page Content Generation

Task	File	Dependencies	Implementation
Bulk Generate Action	modules/site_builder/views.py	PageGenerationService	Add bulk_generate_content action
Task Creation Service	business/site_building/services/page_generation_service.py	Existing	Add create_tasks_for_pages() method
Bulk Queue Service	business/site_building/services/page_generation_service.py	ContentGenerationService	Add bulk_generate_pages() method

Bulk Generation API:

# modules/site_builder/views.py
class SiteBlueprintViewSet(SiteSectorModelViewSet):
    @action(detail=True, methods=['post'])
    def generate_all_pages(self, request, pk=None):
        """
        Generate content for all pages in blueprint.
        
        Request body:
        {
            "page_ids": [1, 2, 3],  # Optional: specific pages, or all if omitted
            "force": false  # Optional: force regenerate existing content
        }
        """
        blueprint = self.get_object()
        page_ids = request.data.get('page_ids')
        force = request.data.get('force', False)
        
        service = PageGenerationService()
        result = service.bulk_generate_pages(
            blueprint, 
            page_ids=page_ids,
            force_regenerate=force
        )
        return success_response(result, request=request)

Bulk Generation Service:

# business/site_building/services/page_generation_service.py
class PageGenerationService:
    def bulk_generate_pages(
        self, 
        site_blueprint: SiteBlueprint,
        page_ids: Optional[List[int]] = None,
        force_regenerate: bool = False
    ) -> dict:
        """
        Generate content for multiple pages in a blueprint.
        
        Similar to how ideas are queued to writer:
        1. Get pages (filtered by page_ids if provided)
        2. Create/update Writer Tasks for each page
        3. Queue content generation for all tasks
        4. Return task IDs for progress tracking
        """
        # Get pages
        pages = site_blueprint.pages.all()
        if page_ids:
            pages = pages.filter(id__in=page_ids)
        
        # Create tasks for all pages
        task_ids = []
        for page in pages:
            task = self._ensure_task(page, force_regenerate=force_regenerate)
            task_ids.append(task.id)
            page.status = 'generating'
            page.save(update_fields=['status', 'updated_at'])
        
        # Queue content generation (same as ideas → writer)
        account = site_blueprint.account
        result = self.content_service.generate_content(task_ids, account)
        
        return {
            'success': True,
            'pages_queued': len(task_ids),
            'task_ids': task_ids,
            'celery_task_id': result.get('task_id'),
        }
    
    def create_tasks_for_pages(
        self,
        site_blueprint: SiteBlueprint,
        page_ids: Optional[List[int]] = None,
        force_regenerate: bool = False
    ) -> List[Tasks]:
        """
        Create Writer Tasks for blueprint pages without generating content.
        
        Useful for:
        - Previewing what tasks will be created
        - Manual task management
        - Integration with existing Writer UI
        """
        pages = site_blueprint.pages.all()
        if page_ids:
            pages = pages.filter(id__in=page_ids)
        
        tasks = []
        for page in pages:
            task = self._ensure_task(page, force_regenerate=force_regenerate)
            tasks.append(task)
        
        return tasks

Frontend Integration

Task	File	Dependencies	Implementation
Bulk Generate UI	site-builder/src/pages/dashboard/SiteDashboard.tsx	API client	Add "Generate All Pages" button
Page Selection UI	site-builder/src/pages/preview/PreviewCanvas.tsx	State store	Add checkbox selection for pages
Progress Tracking	site-builder/src/components/common/ProgressModal.tsx	Existing	Track bulk generation progress

Frontend API Client:

// site-builder/src/api/builder.api.ts
export const builderApi = {
  // ... existing methods
  
  async generateAllPages(
    blueprintId: number,
    options?: { pageIds?: number[]; force?: boolean }
  ): Promise<{ task_id: string; pages_queued: number }> {
    const res = await client.post(
      `/blueprints/${blueprintId}/generate_all_pages/`,
      options || {}
    );
    return res.data;
  },
  
  async createTasksForPages(
    blueprintId: number,
    pageIds?: number[]
  ): Promise<{ tasks: Task[] }> {
    const res = await client.post(
      `/blueprints/${blueprintId}/create_tasks/`,
      { page_ids: pageIds }
    );
    return res.data;
  },
};

Task Mapping

PageBlueprint → Task Mapping:

PageBlueprint Field	Task Field	Notes
title	title	Prefixed with "[Site Builder]"
slug	keywords	Used as keyword hint
type	content_structure	Mapped (home → landing_page, etc.)
blocks_json	description	Extracts headings for context
site_blueprint.name	description	Added to description
account, site, sector	Same	Inherited from blueprint

Content Structure Mapping:

PAGE_TYPE_TO_STRUCTURE = {
    'home': 'landing_page',
    'about': 'supporting_page',
    'services': 'pillar_page',
    'products': 'pillar_page',
    'blog': 'cluster_hub',
    'contact': 'supporting_page',
    'custom': 'landing_page',
}

Integration with Writer UI

Task	File	Dependencies	Implementation
Show Site Builder Tasks	frontend/src/pages/Writer/Content.tsx	Existing	Filter tasks with "[Site Builder]" prefix
Link to Blueprint	frontend/src/pages/Writer/Content.tsx	API	Add link to source blueprint
Bulk Actions	frontend/src/pages/Writer/Content.tsx	Existing	Include Site Builder tasks in bulk actions

Task Identification:

• Tasks created from Site Builder have title prefix: "[Site Builder] {page_title}"
• Can filter Writer tasks by this prefix
• Can link back to source PageBlueprint via task description

---

TESTING & VALIDATION

9.5 Testing

Test Cases:

1. Prompt Management:

   • ✅ Site structure prompt appears in Thinker UI
   • ✅ Prompt can be edited and saved
   • ✅ Prompt reset works correctly
   • ✅ Custom prompt is used in structure generation

2. Response Handling:

   • ✅ Valid JSON response is parsed correctly
   • ✅ JSON wrapped in text is extracted
   • ✅ Invalid responses show proper errors
   • ✅ Structure validation works

3. Bulk Page Generation:

   • ✅ All pages can be queued to writer
   • ✅ Specific pages can be selected
   • ✅ Tasks are created with correct mapping
   • ✅ Content generation is queued correctly
   • ✅ Progress tracking works

4. Task Integration:

   • ✅ Site Builder tasks appear in Writer UI
   • ✅ Tasks link back to source blueprint
   • ✅ Bulk actions work with Site Builder tasks

---

IMPLEMENTATION CHECKLIST

Backend Tasks

• Add site_structure_generation to AIPrompt.PROMPT_TYPE_CHOICES
• Create migration for new prompt type
• Add bulk_generate_pages() to PageGenerationService
• Add create_tasks_for_pages() to PageGenerationService
• Add generate_all_pages action to SiteBlueprintViewSet
• Add create_tasks action to SiteBlueprintViewSet
• Document response handling in code comments
• Add error handling for bulk operations

Frontend Tasks

• Add site_structure_generation to PROMPT_TYPES in Prompts.tsx
• Add "Site Builder" section to Prompts page
• Add prompt editor for site structure generation
• Add "Generate All Pages" button to Site Dashboard
• Add page selection UI to Preview Canvas
• Add bulk generation API methods to builder.api.ts
• Add progress tracking for bulk generation
• Update Writer UI to show Site Builder tasks
• Add link from Writer tasks to source blueprint

Testing Tasks

• Test prompt management UI
• Test response parsing with various formats
• Test bulk page generation
• Test task creation and mapping
• Test integration with Writer UI
• Test error handling

---

RISK ASSESSMENT

Risk	Level	Mitigation
Prompt changes break existing blueprints	LOW	Version prompts, validate before save
Bulk generation overloads system	MEDIUM	Rate limiting, queue management
Task mapping inconsistencies	LOW	Comprehensive tests, validation
Response parsing failures	MEDIUM	Robust error handling, fallbacks

---

SUCCESS CRITERIA

• ✅ Site structure prompt is manageable in Thinker UI
• ✅ Response handling is robust and handles edge cases
• ✅ Blueprint pages can be queued to writer like ideas
• ✅ Bulk generation works for all pages
• ✅ Tasks appear correctly in Writer UI
• ✅ Integration is seamless and follows existing patterns

---

RELATED PHASES

• Phase 3: Site Builder foundation
• Phase 1: Service layer (ContentGenerationService)
• Phase 0: Credit system (for AI operations)

---

END OF PHASE 9 DOCUMENT