# PHASE 9: AI FRAMEWORK & SITE BUILDER INTEGRATION
**Detailed Implementation Plan**

**Goal**: Complete AI framework integration for Site Builder, add prompt management UI, and implement blueprint-to-writer task queuing.

**Timeline**: 2-3 weeks  
**Priority**: MEDIUM  
**Dependencies**: Phase 3

---

## TABLE OF CONTENTS

1. [Overview](#overview)
2. [AI Framework Integration](#ai-framework-integration)
3. [Prompt Management UI](#prompt-management-ui)
4. [Response Handling & Structure Generation](#response-handling--structure-generation)
5. [Blueprint-to-Writer Task Queuing](#blueprint-to-writer-task-queuing)
6. [Testing & Validation](#testing--validation)
7. [Implementation Checklist](#implementation-checklist)

---

## OVERVIEW

### Objectives
- ✅ Add `site_structure_generation` prompt to Thinker `/prompts` UI
- ✅ Document AI framework integration for Site Builder
- ✅ Implement response handling for structure generation
- ✅ Enable blueprint pages to queue as Writer tasks (similar to ideas)
- ✅ Add bulk page content generation from blueprint

### Key Principles
- **Unified Prompt Management**: Site Builder prompts managed in same UI as other prompts
- **Reuse Existing Patterns**: Follow same queuing pattern as ideas → writer tasks
- **AI Framework Compliance**: Use existing `BaseAIFunction` and `AIEngine` patterns
- **Response Validation**: Robust parsing and validation of AI responses

---

## AI FRAMEWORK INTEGRATION

### 9.1 AI Framework Integration

**Purpose**: Document and ensure proper integration with existing AI framework.

#### Current AI Framework Architecture

The Site Builder uses the existing AI framework:

```
AIEngine
  ├─ BaseAIFunction (interface)
  │   ├─ validate()
  │   ├─ prepare()
  │   ├─ build_prompt()
  │   ├─ parse_response()
  │   └─ save_output()
  │
  └─ GenerateSiteStructureFunction
      ├─ Uses PromptRegistry for prompts
      ├─ Integrates with CreditService
      └─ Saves to SiteBlueprint + PageBlueprint models
```

#### AI Function Lifecycle

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **AI Function Registration** | `ai/registry.py` | Existing | Register `generate_site_structure` |
| **Prompt Integration** | `ai/prompts.py` | Existing | Add `site_structure_generation` to DEFAULT_PROMPTS |
| **Credit Cost Configuration** | `business/billing/services/credit_service.py` | Phase 0 | Add `site_structure_generation` operation type |

**AI Function Flow**:
```python
# infrastructure/ai/functions/generate_site_structure.py
class GenerateSiteStructureFunction(BaseAIFunction):
    def get_name(self) -> str:
        return 'generate_site_structure'
    
    def validate(self, payload, account=None):
        # Validates blueprint ID exists
        # Returns {'valid': True/False, 'error': '...'}
    
    def prepare(self, payload, account=None):
        # Loads SiteBlueprint
        # Extracts business_brief, objectives, style
        # Returns context dict
    
    def build_prompt(self, data, account=None):
        # Uses PromptRegistry.get_prompt('site_structure_generation')
        # Injects: BUSINESS_BRIEF, OBJECTIVES, STYLE, SITE_INFO
        # Returns formatted prompt string
    
    def parse_response(self, response, step_tracker=None):
        # Parses JSON from AI response
        # Handles noisy responses (extracts JSON from text)
        # Validates structure is dict
        # Returns parsed structure dict
    
    def save_output(self, parsed, original_data, account=None):
        # Saves structure_json to SiteBlueprint
        # Syncs PageBlueprint records (create/update/delete)
        # Updates blueprint status to 'ready'
        # Returns summary dict
```

#### Prompt Variable Injection

The prompt system supports variable injection:

**Available Variables**:
- `[IGNY8_BUSINESS_BRIEF]` - Business description
- `[IGNY8_OBJECTIVES]` - List of site objectives
- `[IGNY8_STYLE]` - Style preferences JSON
- `[IGNY8_SITE_INFO]` - Site metadata JSON

**Prompt Template** (from `ai/prompts.py`):
```python
'site_structure_generation': """You are a senior UX architect...
BUSINESS BRIEF:
[IGNY8_BUSINESS_BRIEF]

PRIMARY OBJECTIVES:
[IGNY8_OBJECTIVES]

STYLE & BRAND NOTES:
[IGNY8_STYLE]

SITE INFO:
[IGNY8_SITE_INFO]
...
"""
```

---

## PROMPT MANAGEMENT UI

### 9.2 Prompt Management UI

**Purpose**: Add Site Builder prompt to Thinker `/prompts` interface.

#### Add Prompt Type to Backend

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Add prompt type choice** | `modules/system/models.py` | Existing | Add `'site_structure_generation'` to PROMPT_TYPE_CHOICES |

**AIPrompt Model Update**:
```python
# modules/system/models.py
class AIPrompt(AccountBaseModel):
    PROMPT_TYPE_CHOICES = [
        ('clustering', 'Clustering'),
        ('ideas', 'Ideas Generation'),
        ('content_generation', 'Content Generation'),
        ('image_prompt_extraction', 'Image Prompt Extraction'),
        ('image_prompt_template', 'Image Prompt Template'),
        ('negative_prompt', 'Negative Prompt'),
        ('site_structure_generation', 'Site Structure Generation'),  # NEW
    ]
```

#### Add Prompt Type to Frontend

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Add prompt type config** | `frontend/src/pages/Thinker/Prompts.tsx` | Existing | Add site_structure_generation to PROMPT_TYPES array |

**Frontend Prompt Type Config**:
```typescript
// frontend/src/pages/Thinker/Prompts.tsx
const PROMPT_TYPES = [
  // ... existing prompts
  {
    key: 'site_structure_generation',
    label: 'Site Structure Generation',
    description: 'Generate site structure from business brief. Use [IGNY8_BUSINESS_BRIEF], [IGNY8_OBJECTIVES], [IGNY8_STYLE], and [IGNY8_SITE_INFO] to inject data.',
    icon: '🏗️',
    color: 'teal',
  },
];
```

#### Prompt UI Section

The prompt will appear in a new "Site Builder" section in the Thinker Prompts page:

```typescript
{/* Site Builder Prompts Section */}
<div className="mb-8">
  <div className="mb-4">
    <h2 className="text-xl font-semibold text-gray-800 dark:text-white">
      Site Builder
    </h2>
    <p className="text-sm text-gray-600 dark:text-gray-400">
      Configure prompts for AI-powered site structure generation
    </p>
  </div>
  
  {/* Site Structure Generation Prompt */}
  <PromptEditor
    promptType="site_structure_generation"
    label="Site Structure Generation Prompt"
    description="Generate complete site structure (pages, blocks, navigation) from business brief"
    variables={[
      '[IGNY8_BUSINESS_BRIEF]',
      '[IGNY8_OBJECTIVES]',
      '[IGNY8_STYLE]',
      '[IGNY8_SITE_INFO]'
    ]}
  />
</div>
```

#### Migration for Prompt Type

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Create migration** | `modules/system/migrations/XXXX_add_site_structure_prompt_type.py` | Existing | Add new choice to PROMPT_TYPE_CHOICES |

---

## RESPONSE HANDLING & STRUCTURE GENERATION

### 9.3 Response Handling & Structure Generation

**Purpose**: Document and enhance AI response parsing and structure generation.

#### Response Format

The AI returns a JSON structure:

```json
{
  "site": {
    "name": "Acme Robotics",
    "primary_navigation": ["home", "services", "about", "contact"],
    "secondary_navigation": ["blog", "faq"],
    "hero_message": "Transform your warehouse operations",
    "tone": "Confident, professional, innovative"
  },
  "pages": [
    {
      "slug": "home",
      "title": "Home",
      "type": "home",
      "status": "draft",
      "objective": "Convert visitors to demo requests",
      "primary_cta": "Book a demo",
      "blocks": [
        {
          "type": "hero",
          "heading": "Warehouse Automation That Scales",
          "subheading": "AI-powered robotics for modern fulfillment",
          "layout": "full-width",
          "content": ["Hero section with CTA"]
        },
        {
          "type": "features",
          "heading": "Why Choose Us",
          "subheading": "Industry-leading solutions",
          "layout": "three-column",
          "content": ["Feature 1", "Feature 2", "Feature 3"]
        }
      ]
    }
  ]
}
```

#### Response Parsing Flow

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Parse Response** | `ai/functions/generate_site_structure.py` | Existing | `parse_response()` method |
| **Extract JSON** | `ai/functions/generate_site_structure.py` | Existing | `_extract_json_object()` helper |
| **Validate Structure** | `ai/functions/generate_site_structure.py` | Existing | `_ensure_dict()` validation |

**Parsing Logic**:
```python
def parse_response(self, response: str, step_tracker=None) -> Dict[str, Any]:
    """
    Parse AI response into structure dict.
    
    Handles:
    1. Direct JSON response
    2. JSON wrapped in text/markdown
    3. Noisy responses with commentary
    """
    response = response.strip()
    
    # Try direct JSON parse
    try:
        return self._ensure_dict(json.loads(response))
    except json.JSONDecodeError:
        # Extract JSON from text
        cleaned = self._extract_json_object(response)
        if cleaned:
            return self._ensure_dict(json.loads(cleaned))
        raise ValueError("Unable to parse AI response into JSON")
```

#### Structure Persistence

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Save Structure** | `ai/functions/generate_site_structure.py` | Existing | `save_output()` method |
| **Sync Page Blueprints** | `ai/functions/generate_site_structure.py` | Existing | `_sync_page_blueprints()` method |

**Persistence Flow**:
```python
def save_output(self, parsed, original_data, account=None):
    """
    Save structure and sync page blueprints.
    
    1. Save structure_json to SiteBlueprint
    2. Create/update PageBlueprint records
    3. Delete pages not in new structure
    4. Update blueprint status to 'ready'
    """
    blueprint = original_data['blueprint']
    structure = self._ensure_dict(parsed)
    pages = structure.get('pages', [])
    
    # Save structure
    blueprint.structure_json = structure
    blueprint.status = 'ready'
    blueprint.save()
    
    # Sync pages (create/update/delete)
    created, updated, deleted = self._sync_page_blueprints(blueprint, pages)
    
    return {
        'success': True,
        'pages_created': created,
        'pages_updated': updated,
        'pages_deleted': deleted,
    }
```

#### Error Handling

**Error Scenarios**:
1. **Invalid JSON**: Extract JSON from text, fallback to error
2. **Missing Required Fields**: Validate structure has `site` and `pages`
3. **Invalid Page Types**: Map to allowed types, default to 'custom'
4. **Duplicate Slugs**: Use `update_or_create` to handle conflicts

**Error Response Format**:
```python
{
    'success': False,
    'error': 'Error message',
    'error_type': 'ParseError' | 'ValidationError' | 'SaveError'
}
```

---

## BLUEPRINT-TO-WRITER TASK QUEUING

### 9.4 Blueprint-to-Writer Task Queuing

**Purpose**: Enable blueprint pages to be queued as Writer tasks, similar to how ideas are queued.

#### Current Pattern (Ideas → Writer)

The existing pattern for ideas:

```
ContentIdeas → Writer Tasks → Content Generation
```

**Flow**:
1. User selects ideas in Planner
2. Click "Generate Content"
3. Creates `Tasks` records for each idea
4. Queues `generate_content` AI task
5. Generates HTML content

#### New Pattern (Blueprint Pages → Writer)

The new pattern for blueprint pages:

```
PageBlueprint → Writer Tasks → Content Generation
```

**Flow**:
1. User has generated SiteBlueprint with PageBlueprints
2. User selects pages (or all pages)
3. Click "Generate Content for Pages"
4. Creates `Tasks` records for each page
5. Queues `generate_content` AI task
6. Generates HTML content

#### Bulk Page Content Generation

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Bulk Generate Action** | `modules/site_builder/views.py` | PageGenerationService | Add `bulk_generate_content` action |
| **Task Creation Service** | `business/site_building/services/page_generation_service.py` | Existing | Add `create_tasks_for_pages()` method |
| **Bulk Queue Service** | `business/site_building/services/page_generation_service.py` | ContentGenerationService | Add `bulk_generate_pages()` method |

**Bulk Generation API**:
```python
# modules/site_builder/views.py
class SiteBlueprintViewSet(SiteSectorModelViewSet):
    @action(detail=True, methods=['post'])
    def generate_all_pages(self, request, pk=None):
        """
        Generate content for all pages in blueprint.
        
        Request body:
        {
            "page_ids": [1, 2, 3],  # Optional: specific pages, or all if omitted
            "force": false  # Optional: force regenerate existing content
        }
        """
        blueprint = self.get_object()
        page_ids = request.data.get('page_ids')
        force = request.data.get('force', False)
        
        service = PageGenerationService()
        result = service.bulk_generate_pages(
            blueprint, 
            page_ids=page_ids,
            force_regenerate=force
        )
        return success_response(result, request=request)
```

**Bulk Generation Service**:
```python
# business/site_building/services/page_generation_service.py
class PageGenerationService:
    def bulk_generate_pages(
        self, 
        site_blueprint: SiteBlueprint,
        page_ids: Optional[List[int]] = None,
        force_regenerate: bool = False
    ) -> dict:
        """
        Generate content for multiple pages in a blueprint.
        
        Similar to how ideas are queued to writer:
        1. Get pages (filtered by page_ids if provided)
        2. Create/update Writer Tasks for each page
        3. Queue content generation for all tasks
        4. Return task IDs for progress tracking
        """
        # Get pages
        pages = site_blueprint.pages.all()
        if page_ids:
            pages = pages.filter(id__in=page_ids)
        
        # Create tasks for all pages
        task_ids = []
        for page in pages:
            task = self._ensure_task(page, force_regenerate=force_regenerate)
            task_ids.append(task.id)
            page.status = 'generating'
            page.save(update_fields=['status', 'updated_at'])
        
        # Queue content generation (same as ideas → writer)
        account = site_blueprint.account
        result = self.content_service.generate_content(task_ids, account)
        
        return {
            'success': True,
            'pages_queued': len(task_ids),
            'task_ids': task_ids,
            'celery_task_id': result.get('task_id'),
        }
    
    def create_tasks_for_pages(
        self,
        site_blueprint: SiteBlueprint,
        page_ids: Optional[List[int]] = None,
        force_regenerate: bool = False
    ) -> List[Tasks]:
        """
        Create Writer Tasks for blueprint pages without generating content.
        
        Useful for:
        - Previewing what tasks will be created
        - Manual task management
        - Integration with existing Writer UI
        """
        pages = site_blueprint.pages.all()
        if page_ids:
            pages = pages.filter(id__in=page_ids)
        
        tasks = []
        for page in pages:
            task = self._ensure_task(page, force_regenerate=force_regenerate)
            tasks.append(task)
        
        return tasks
```

#### Frontend Integration

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Bulk Generate UI** | `site-builder/src/pages/dashboard/SiteDashboard.tsx` | API client | Add "Generate All Pages" button |
| **Page Selection UI** | `site-builder/src/pages/preview/PreviewCanvas.tsx` | State store | Add checkbox selection for pages |
| **Progress Tracking** | `site-builder/src/components/common/ProgressModal.tsx` | Existing | Track bulk generation progress |

**Frontend API Client**:
```typescript
// site-builder/src/api/builder.api.ts
export const builderApi = {
  // ... existing methods
  
  async generateAllPages(
    blueprintId: number,
    options?: { pageIds?: number[]; force?: boolean }
  ): Promise<{ task_id: string; pages_queued: number }> {
    const res = await client.post(
      `/blueprints/${blueprintId}/generate_all_pages/`,
      options || {}
    );
    return res.data;
  },
  
  async createTasksForPages(
    blueprintId: number,
    pageIds?: number[]
  ): Promise<{ tasks: Task[] }> {
    const res = await client.post(
      `/blueprints/${blueprintId}/create_tasks/`,
      { page_ids: pageIds }
    );
    return res.data;
  },
};
```

#### Task Mapping

**PageBlueprint → Task Mapping**:

| PageBlueprint Field | Task Field | Notes |
|---------------------|-----------|-------|
| `title` | `title` | Prefixed with "[Site Builder]" |
| `slug` | `keywords` | Used as keyword hint |
| `type` | `content_structure` | Mapped (home → landing_page, etc.) |
| `blocks_json` | `description` | Extracts headings for context |
| `site_blueprint.name` | `description` | Added to description |
| `account`, `site`, `sector` | Same | Inherited from blueprint |

**Content Structure Mapping**:
```python
PAGE_TYPE_TO_STRUCTURE = {
    'home': 'landing_page',
    'about': 'supporting_page',
    'services': 'pillar_page',
    'products': 'pillar_page',
    'blog': 'cluster_hub',
    'contact': 'supporting_page',
    'custom': 'landing_page',
}
```

#### Integration with Writer UI

| Task | File | Dependencies | Implementation |
|------|------|--------------|----------------|
| **Show Site Builder Tasks** | `frontend/src/pages/Writer/Content.tsx` | Existing | Filter tasks with "[Site Builder]" prefix |
| **Link to Blueprint** | `frontend/src/pages/Writer/Content.tsx` | API | Add link to source blueprint |
| **Bulk Actions** | `frontend/src/pages/Writer/Content.tsx` | Existing | Include Site Builder tasks in bulk actions |

**Task Identification**:
- Tasks created from Site Builder have title prefix: `"[Site Builder] {page_title}"`
- Can filter Writer tasks by this prefix
- Can link back to source PageBlueprint via task description

---

## TESTING & VALIDATION

### 9.5 Testing

**Test Cases**:

1. **Prompt Management**:
   - ✅ Site structure prompt appears in Thinker UI
   - ✅ Prompt can be edited and saved
   - ✅ Prompt reset works correctly
   - ✅ Custom prompt is used in structure generation

2. **Response Handling**:
   - ✅ Valid JSON response is parsed correctly
   - ✅ JSON wrapped in text is extracted
   - ✅ Invalid responses show proper errors
   - ✅ Structure validation works

3. **Bulk Page Generation**:
   - ✅ All pages can be queued to writer
   - ✅ Specific pages can be selected
   - ✅ Tasks are created with correct mapping
   - ✅ Content generation is queued correctly
   - ✅ Progress tracking works

4. **Task Integration**:
   - ✅ Site Builder tasks appear in Writer UI
   - ✅ Tasks link back to source blueprint
   - ✅ Bulk actions work with Site Builder tasks

---

## IMPLEMENTATION CHECKLIST

### Backend Tasks

- [ ] Add `site_structure_generation` to `AIPrompt.PROMPT_TYPE_CHOICES`
- [ ] Create migration for new prompt type
- [ ] Add `bulk_generate_pages()` to `PageGenerationService`
- [ ] Add `create_tasks_for_pages()` to `PageGenerationService`
- [ ] Add `generate_all_pages` action to `SiteBlueprintViewSet`
- [ ] Add `create_tasks` action to `SiteBlueprintViewSet`
- [ ] Document response handling in code comments
- [ ] Add error handling for bulk operations

### Frontend Tasks

- [ ] Add `site_structure_generation` to `PROMPT_TYPES` in `Prompts.tsx`
- [ ] Add "Site Builder" section to Prompts page
- [ ] Add prompt editor for site structure generation
- [ ] Add "Generate All Pages" button to Site Dashboard
- [ ] Add page selection UI to Preview Canvas
- [ ] Add bulk generation API methods to `builder.api.ts`
- [ ] Add progress tracking for bulk generation
- [ ] Update Writer UI to show Site Builder tasks
- [ ] Add link from Writer tasks to source blueprint

### Testing Tasks

- [ ] Test prompt management UI
- [ ] Test response parsing with various formats
- [ ] Test bulk page generation
- [ ] Test task creation and mapping
- [ ] Test integration with Writer UI
- [ ] Test error handling

---

## RISK ASSESSMENT

| Risk | Level | Mitigation |
|------|-------|------------|
| **Prompt changes break existing blueprints** | LOW | Version prompts, validate before save |
| **Bulk generation overloads system** | MEDIUM | Rate limiting, queue management |
| **Task mapping inconsistencies** | LOW | Comprehensive tests, validation |
| **Response parsing failures** | MEDIUM | Robust error handling, fallbacks |

---

## SUCCESS CRITERIA

- ✅ Site structure prompt is manageable in Thinker UI
- ✅ Response handling is robust and handles edge cases
- ✅ Blueprint pages can be queued to writer like ideas
- ✅ Bulk generation works for all pages
- ✅ Tasks appear correctly in Writer UI
- ✅ Integration is seamless and follows existing patterns

---

## RELATED PHASES

- **Phase 3**: Site Builder foundation
- **Phase 1**: Service layer (ContentGenerationService)
- **Phase 0**: Credit system (for AI operations)

---

**END OF PHASE 9 DOCUMENT**