igny8/refactor-plan/UNIFIED_SITE_BUILDER_IMPLEMENTATION_PLAN.md

# IGNY8 Refactor Plan – Stage Implementation Guide

This document complements `SITE_BUILDER_PLANNER_ARCHITECTURE_PLAN.md` by outlining how to deliver the refactor in four controlled stages. Each stage lists scope, tasks, exits, and testing so workstreams can move independently without breaking user flows.

---

## Stage 1 – Data & Services Foundation
**Objective:** Build all backend scaffolding (schema + services) required for the guided site builder, richer planner, and downstream metadata.

### 1. Sequenced Plan of Work
1. **Preparation**
   - Freeze any pending migrations; branch off `refactor-stage1`.
   - Create feature flags (`USE_SITE_BUILDER_REFACTOR`) in settings/env files.
   - Draft ERD updates and share with backend team for approval.
2. **Schema Implementation**
   - **Site Builder tables**
     - `SiteBlueprintCluster`: fields for `cluster_id`, `role`, `coverage_status`, `metadata`.
     - `SiteBlueprintTaxonomy`: taxonomy type enum, `cluster_id`, optional external reference.
     - `WorkflowState`: blueprint FK, `current_step`, `step_status`, `blocking_reason`, timestamps.
   - **Writer tables**
     - `ContentClusterMap`, `ContentTaxonomyMap`, `ContentAttributeMap` with FK constraints + indexes for performant lookups.
   - Run migrations locally → staging; capture SQL diff in change log.
3. **Service Layer**
   - Implement `WorkflowStateService` with methods:
     - `initialize(site_blueprint)` – create default states.
     - `update_step(step, status, meta)` – persists transitions and emits audit logs.
     - `validate_step(step)` – raises descriptive errors for UI.
   - Implement `TaxonomyService`:
     - CRUD handlers.
     - Import adapters (WordPress categories/tags, WooCommerce product attributes).
     - Helper to map taxonomy ↔ cluster(s).
   - Update `ClusteringService` prompts to emit:
     - Topic cluster suggestions.
     - Attribute/feature recommendations per keyword set.
     - Taxonomy hints (category/tag names).
4. **API/Serializer Updates**
   - Planner serializers return `context_type`, `dimension_meta`.
   - Writer serializers include `entity_type`, taxonomy associations, validation errors.
   - Site builder endpoints expose workflow state + gating booleans.
5. **Validation Hooks**
   - Add reusable validators (e.g., `ensure_clusters_attached(site_blueprint)`).
   - Integrate validators into existing actions (`generate_structure`, `create_tasks_from_blueprint`).
6. **Internal Tooling & Docs**
   - Update Postman collection / API reference for new endpoints + payloads.
   - Write developer doc describing new models + services.
   - Set up monitoring dashboards (Grafana/NewRelic) for:
     - Migration duration.
     - Celery task errors referencing new services.

### 2. Owners & Dependencies
- **Backend Lead:** owns migrations, services, validator integration.
- **Data/DBA:** reviews migration plan, handles staging rollout.
- **QA:** prepares regression scripts for planner/writer.
- **Docs:** updates `master-docs` (backend + AI framework) with new tables/services.
- Dependencies: PostgreSQL 15 features already available; no external API changes needed.

### 3. Deliverables & Exit Criteria
- ✅ Migrations applied to staging + rollback verified.
- ✅ `WorkflowStateService`, `TaxonomyService`, `ClusteringService` updates merged with unit tests.
- ✅ Planner/Writer legacy flows pass smoke tests (imports, clustering, idea creation, content generation).
- ✅ API reference + architecture docs updated.
- ✅ Feature flag defaults to OFF (no user-facing change yet).

### 4. Testing Matrix
| Area | Automated | Manual |
| --- | --- | --- |
| Migrations | `pytest` migration test + Django fake migrations | Dry run on staging, rollback rehearsal |
| Services | Unit tests for new service methods | Postman calls to workflow/taxonomy endpoints |
| Validators | Serializer/endpoint tests expecting 4xx when requirements unmet | Trigger via API (e.g., attempt sitemap gen without clusters) |
| Regression | Existing planner/writer test suite | Smoke test: keyword import → cluster → idea → task → content |

### 5. Rollout Checklist
1. Merge migrations + services behind feature flag.
2. Deploy to staging, run migration + regression script.
3. Monitor Celery/DB logs for anomalies.
4. After 24h stability, promote to production (still gated by flag).
5. Announce availability to frontend team to begin Stage 2 work.

---

## Stage 2 – Planner + Wizard UX
**Objective:** Ship the state-aware, self-guided planner + site builder experience on top of Stage 1 services.

### 1. Sequenced Plan of Work
1. **Planning & Design**
   - UX team finalizes wizard wireframes + microcopy for each step, including helper text/tooltips.
   - Define telemetry schema (events: `wizard_step_started`, `wizard_step_completed`, `wizard_blocking_issue`).
   - Review backend responses needed per step (cluster metrics, taxonomy suggestions, validation errors).
2. **Backend Enhancements**
   - Extend planner endpoints to return:
     - Cluster stats (keyword count, projected hubs/support pages).
     - Suggested taxonomies/attributes from Stage 1 `dimension_meta`.
     - Validation error payloads structured for UI (“missing clusters”, “taxonomy incomplete”).
   - Add audit logging to `WorkflowStateService.update_step` for troubleshooting.
3. **Frontend Infrastructure**
   - Build `builderWorkflowStore` (Zustand):
     - State: `currentStep`, `completedSteps`, `blockingIssues`, `workflowState`, `loading`.
     - Actions: `goToStep`, `completeStep`, `refreshState`, `setBlockingIssue`.
     - Persistence: sessionStorage/localStorage to resume across refreshes.
   - Hook store into new `/site-builder/workflow/{blueprintId}` endpoints.
4. **Wizard Step Implementation**
   - **Step 1 – Business Details**
     - Components for site type selection, hosting detection (show WP integration status), brand inputs.
     - Save actions call blueprint PATCH + workflow state update.
   - **Step 2 – Cluster Assignment**
     - Table/list of clusters with filters (intent, volume, context type).
     - Coverage meter (keywords covered vs. target).
     - “Add cluster” action hitting `/blueprints/{id}/clusters/attach`.
     - Blocking logic: enforce minimum cluster count/keyword count per site type.
   - **Step 3 – Taxonomy Builder**
     - Tree/table for categories, tags, product attributes, service groups.
     - Import buttons (WordPress/WooCommerce) calling `TaxonomyService` endpoints.
     - Mapping UI linking taxonomy items to clusters (drag/drop or select dropdown).
   - **Step 4 – AI Sitemap Review**
     - Grid grouped by cluster/taxonomy.
     - Visual checklist (Hub page, Supporting pages count).
     - Ability to edit page titles/types, reorder, remove.
     - “Regenerate” button to re-run AI structure (with warning on credit usage).
   - **Step 5 – Coverage Validation**
     - Summary cards showing clusters covered/uncovered, taxonomy gaps, attribute coverage.
     - CTA to jump back to specific step if gaps remain.
   - **Step 6 – Ideas Hand-off**
     - Select pages to push to Planner Ideas.
     - Secondary prompt textarea; show preview of what data AI receives.
     - Confirmation modal summarizing cluster/page counts.
5. **Planner Module Enhancements**
   - Cluster matrix view showing clusters vs. taxonomy/attribute coverage, clickable to drill into wizard step.
   - Taxonomy management table (search, filters, bulk edits).
   - Inline warnings (e.g., banner on Planner home when blueprint still missing requirements).
6. **UX Guardrails**
   - Breadcrumb/progress component with checkmarks.
   - Disabled Next button shows tooltip listing unmet requirements.
   - Contextual helper drawer per step (links to docs, best practices).

### 2. Owners & Dependencies
- **Frontend Lead:** wizard implementation + planner UI updates.
- **Backend Support:** ensure required data returned, logging in place.
- **Design/UX:** final visuals, helper content, accessibility review.
- **Docs/Enablement:** in-app help, video walkthroughs.
- Dependency: Stage 1 feature flag must be AVAILABLE with APIs stable.

### 3. Deliverables & Exit Criteria
- ✅ Wizard completes end-to-end (feature-flagged) for both IGNY8-hosted and WP-linked sites.
- ✅ Telemetry events firing and visible in analytics dashboard.
- ✅ QA scripts covering success/failure states executed, accessibility issues resolved.
- ✅ Documentation (knowledge base, tooltips) updated.
- ✅ Rollout plan ready (progressive enablement per account/site type).

### 4. Testing Matrix
| Area | Automated | Manual |
| --- | --- | --- |
| Wizard Flow | Cypress tests for each step, gating logic, resume | Scenario walkthroughs (new site, WP import), cross-browser testing |
| Planner Views | Jest/unit tests for new components, store logic | Performance check on cluster matrix, UX review |
| Telemetry | Unit tests ensuring events dispatch | Verify dashboards receiving events |
| Accessibility | Lint checks (aria labels, keyboard nav) | Screen reader + keyboard-only validation |

### 5. Rollout Checklist
1. Feature flag ON for internal testers → gather feedback.
2. Fix critical issues, re-run automated suite.
3. Enable for pilot accounts (one IGNY8 host, one WP host).
4. Monitor telemetry/feedback; ensure support team ready.
5. Gradually roll out to all accounts; announce in release notes + in-app banner.

---

## Stage 3 – Writer / Linker / Optimizer Enhancements
**Objective:** Propagate the new metadata through content creation and optimization, ensuring validation before publish.

### 1. Sequenced Plan of Work
1. **Metadata Audit & Defaults**
   - Verify Stage 1 migrations populated defaults for legacy content/tasks.
   - Script to backfill `entity_type`, `taxonomy_id`, `cluster_role` for existing records (with audit log).
2. **Backend Pipeline Updates**
   - **Ideas → Tasks**: update services so every task inherits cluster/taxonomy/attribute metadata.
   - **Tasks → Content**: adjust `ContentPipelineService` to store mappings in `ContentClusterMap`, etc.
   - **Prompt Updates**: AI prompts include cluster role, taxonomy context, product attributes to produce richer output.
   - **Validation Services**: create reusable validators (e.g., `ensure_required_attributes(task)`) returning human-friendly errors.
3. **Linker & Optimizer Enhancements**
   - Linker service leverages mapping tables to suggest hub/supporting/attribute links; adds priority scoring.
   - Optimizer scoring factors:
     - Cluster coverage (are all hubs/supporting pages present?).
     - Taxonomy alignment (page categorized correctly).
     - Attribute completeness (product/service specs filled).
   - New APIs:
     - `GET /sites/{id}/progress` (cluster-level completion + validation status).
     - `GET /writer/content/{id}/validation` (aggregated checklist for UI).
4. **Frontend Enhancements**
   - **Planner Ideas / Writer Tasks lists**
     - Columns/chips for cluster, taxonomy, attributes.
     - Filters for entity type & validation status.
   - **Writer Editor**
     - Sidebar module with cluster summary, taxonomy tree, attribute form.
     - Validation panel showing remaining requirements; publish button disabled until cleared.
   - **Linker UI**
     - Group internal link suggestions by cluster role; show context snippet, CTA to insert.
   - **Optimizer Dashboard**
     - Scorecards per cluster dimension with color coding + “next action” cards.
   - **Site Progress Widgets**
     - On site overview, show bars for hubs/supporting/attribute completion; link to problematic clusters.
5. **Notifications & Workflow**
   - Inline toast + optional email when validation fails or content ready for review.
   - Credit reminder banner when user tries to regenerate content/optimization tasks.

### 2. Owners & Dependencies
- **Backend Lead:** pipeline + services, validation APIs.
- **Frontend Lead:** writer/linker/optimizer UI, progress widgets.
- **AI/Prompt Specialist:** adjust prompts/templates and verify outputs.
- **QA:** regression scripts for writer workflow, linker, optimizer.
- Dependencies: Stage 2 wizard must deliver cluster/taxonomy data; Stage 1 mapping tables in place.

### 3. Deliverables & Exit Criteria
- ✅ All new tasks/content carry entity/taxonomy metadata; legacy content backfilled.
- ✅ Publish action blocked when required metadata missing; errors surfaced clearly.
- ✅ Linker/optimizer screens powered by new data with acceptable performance.
- ✅ Site progress dashboards live for pilot sites with feedback loop established.

### 4. Testing Matrix
| Area | Automated | Manual |
| --- | --- | --- |
| Pipeline | Unit tests on service functions, serializer coverage | End-to-end run: blueprint → ideas → tasks → content |
| Validation | Tests ensuring validators trigger proper errors | Attempt publish without taxonomy/attributes; confirm UI flow |
| Linker/Optimizer | Service tests for scoring & suggestions | Performance profiling on large datasets; UX review |
| Progress Widgets | Component/unit tests | Compare counts vs. database for pilot site |

### 5. Rollout Checklist
1. Deploy to staging with feature flag ON; run regression suite.
2. Backfill metadata for existing content in staging; verify no issues.
3. Pilot with internal content team; gather feedback on validation friction.
4. Optimize slow queries (add indexes/caching) before production rollout.
5. Update training docs/videos for writers; enable flag gradually across accounts.

---

## Stage 4 – Publishing & Sync Integration
**Objective:** Achieve feature parity between IGNY8-hosted deployments and WordPress sites, using the shared metadata model.

### 1. Sequenced Plan of Work
1. **Sync Architecture Review**
   - Audit existing WordPress adapter + sync service.
   - Confirm required endpoints (categories, tags, WooCommerce attributes/products) and authentication flows.
   - Document data mapping between WP taxonomies and new IGNY8 taxonomy tables.
2. **Backend Enhancements**
   - **Import**
     - Extend `ContentSyncService` to fetch WP taxonomies, products, custom post types.
     - Map external IDs to `SiteBlueprintTaxonomy`, `ContentTaxonomyMap`, storing `external_reference`.
     - Auto-create missing clusters/taxonomies with flag indicating “imported”.
   - **Export**
     - Update `WordPressAdapter` to create/update taxonomies before publishing posts/products.
     - Ensure product attributes/tags pushed before product content.
   - **Sync Health**
     - New endpoints: `/sites/{id}/sync/status`, `/sites/{id}/sync/run`.
     - Track last sync time, mismatch counts, error logs.
   - **Deployment**
     - `SitesRendererAdapter` consumes cluster/taxonomy metadata to build navigation, breadcrumbs, internal links.
     - Deployment readiness check service summarizing cluster coverage, content publish status, validation flags.
   - **Logging/Monitoring**
     - Structured logs for sync runs (duration, items processed, failures).
     - Alerts for repeated sync failures or deployment errors.
3. **Frontend Enhancements**
   - **Sync Dashboard**
     - Parity indicators (taxonomies, products, posts) with status icons.
     - Buttons for manual sync, retry failed items, view logs.
     - Detail drawer showing mismatched items and suggested fixes.
   - **Deployment Panel**
     - Readiness checklist (clusters covered, content statuses, validation passes).
     - Deploy + rollback buttons with confirmation modals.
     - Toast/notification system for deploy/sync success/failure.
   - **WordPress Connection UI**
     - Surface integration status, credentials check, last sync info.
4. **Operational Runbooks**
   - Document sync troubleshooting steps, rollback procedures, and escalation path.
   - Provide support scripts for forcing sync, clearing queue, recovering failed deployment.

### 2. Owners & Dependencies
- **Backend Lead:** sync/import/export logic, deployment readiness service.
- **Frontend Lead:** sync dashboard, deployment UI.
- **Infra/DevOps:** monitoring, alerting, runbooks.
- **QA:** end-to-end publish tests IGNY8↔WP.
- Dependencies: Stage 3 metadata stable; WordPress credentials for pilot sites available.

### 3. Deliverables & Exit Criteria
- ✅ End-to-end publishing works for blog/ecommerce/company templates on IGNY8 hosting and WordPress (posts/products/services).
- ✅ Sync dashboard shows green status for pilot accounts; manual retry resolves mismatches.
- ✅ Deployment readiness checks enforced; logs + rollback procedures documented.
- ✅ Release notes + training material published.

### 4. Testing Matrix
| Area | Automated | Manual |
| --- | --- | --- |
| Sync Logic | Integration tests hitting mocked WP APIs | Staging sync from live WP instance; verify taxonomy parity |
| Deployment | Renderer tests verifying metadata usage | Deploy IGNY8 site, inspect navigation/internal links |
| Dashboards | Component/unit tests | Pilot user testing of sync dashboard + deployment flow |
| Runbooks | N/A | Tabletop exercises for failure scenarios (sync fails, deploy rollback) |

### 5. Rollout Checklist
1. Enable Stage 4 flag in staging; run full sync/import tests.
2. Pilot with one IGNY8-hosted and one WP-hosted site; gather feedback.
3. Train support team on new dashboards/runbooks.
4. Announce availability; roll out gradually to accounts with WordPress integrations.
5. Monitor logs/alerts closely during first production syncs; iterate on tooling as needed.

---

## Stage Coordination Notes
- **Feature flags**: introduce per-stage flags to roll out gradually.
- **Monitoring**: add dashboards for workflow step failures, sync errors, credit usage spikes.
- **Documentation cadence**: update `master-docs` + `part2-dev/planning` after each stage completes.
- **Entry/exit reviews**: hold short readiness reviews before moving to the next stage to ensure dependencies satisfied.

This plan ensures each layer stabilizes before the next, keeping users on a guided, predictable experience throughout the refactor.