diff --git a/backend/celerybeat-schedule b/backend/celerybeat-schedule index 2f8bbd79..cbf428af 100644 Binary files a/backend/celerybeat-schedule and b/backend/celerybeat-schedule differ diff --git a/part2-dev/planning/04-roadmap.md b/part2-dev/planning/04-roadmap.md index 308f4b8e..8e7f0393 100644 --- a/part2-dev/planning/04-roadmap.md +++ b/part2-dev/planning/04-roadmap.md @@ -85,4 +85,100 @@ Mitigations described in `02-workflows.md` (UX guardrails) and `03-phase-reports - Detailed workflows: `02-workflows.md` - Historical reports: `03-phase-reports.md` +--- +## 6. Stage Implementation Details + +### Stage 1 – Data & Services Foundation +**Goal:** Ship all schema + backend building blocks so later stages can focus on UX. + +**Backend** +- Migrations for `SiteBlueprintCluster`, `SiteBlueprintTaxonomy`, `WorkflowState`, writer mapping tables. +- `WorkflowStateService` with REST endpoints for fetching/updating step status. +- `TaxonomyService` (CRUD/import/export) and `ClusteringService` prompt upgrade for multi-dimensional clusters. +- API validation hooks (`validate_clusters_attached`, `validate_taxonomies_ready`, etc.) consumed by wizard gating. +- Serializer updates exposing entity metadata to frontend. + +**Frontend** +- Minimal work (feature flags + API clients) to prepare for new endpoints; no user-facing change yet. + +**Testing/Acceptance** +- Migration dry run, rollback verified. +- Unit/integration tests for services and validators. +- API documentation updated; Postman suite green. + +### Stage 2 – Planner + Wizard UX +**Goal:** Deliver the guided, state-aware planning experience. + +**Backend** +- Ensure planner endpoints return cluster metrics, taxonomy suggestions, validation errors. +- Audit logging for workflow transitions. + +**Frontend** +- Implement `builderWorkflowStore` (Zustand) with resume capability, blocking logic, telemetry. +- Build wizard steps: + 1. Business details + hosting detection. + 2. Cluster assignment (coverage metrics, filters). + 3. Taxonomy builder (create/import/match clusters). + 4. AI sitemap review with checklist + edits. + 5. Coverage validation summary. + 6. Ideas hand-off with secondary prompt. +- Update planner cluster/taxonomy management screens (matrix view, inline warnings). + +**UX Guardrails** +- Progress indicator with completion badges. +- Disabled buttons explain “what’s missing”. +- CTAs back to Planner when prerequisites unmet. + +**Testing/Acceptance** +- Manual script for full wizard flow (new IGNY8 site + WP-linked site). +- Cypress/e2e coverage for gating + resume behavior. +- Telemetry events for step completion firing. + +### Stage 3 – Writer / Linker / Optimizer Enhancements +**Goal:** Propagate metadata end-to-end and enforce validation. + +**Backend** +- Ensure ideas/tasks/content writer pipeline populates `entity_type`, `taxonomy_id`, `cluster_role`, `product_data`. +- Linker/optimizer services consume mapping tables and expose coverage metrics via APIs. +- Progress dashboard endpoints (per site/cluster) for frontend widgets. + +**Frontend** +- Planner Ideas + Writer Tasks lists show entity/taxonomy chips and warnings. +- Writer editor sidebar summarizing target cluster, taxonomy, attribute requirements; validation prompts before publish. +- Linker UI grouped by cluster role; Optimizer dashboards show dimension scorecards and “next actions”. +- Sites module progress bars summarizing blueprint completion. + +**Testing/Acceptance** +- Regression on task/content creation flows. +- QA verifies publish blocked when metadata missing. +- Performance profiling on linker/optimizer queries with new joins. + +### Stage 4 – Publishing & Sync Integration +**Goal:** Achieve parity between IGNY8-hosted and WordPress-hosted sites. + +**Backend** +- Complete WordPress taxonomy/product attribute import/export in `ContentSyncService`. +- Enhance `SitesRendererAdapter` to include cluster/taxonomy metadata for navigation + internal linking. +- Sync health endpoints (last sync, mismatched taxonomies, errors) and deployment logs/rollback hooks. + +**Frontend** +- Sync dashboard showing taxonomy parity, cluster coverage, last sync status, manual retry/reconcile actions. +- Deployment panel summarizing readiness (clusters covered, content publish state) before enabling deploy. +- Notifications/toasts for sync success/failure and deployments. + +**Testing/Acceptance** +- End-to-end tests: IGNY8 deployment, WP publish for posts/products/services (blog/ecom/company templates). +- Fallback behavior validated (auto-creating missing taxonomies). +- Final QA checklist signed off; release notes + training updates distributed. + +--- + +## 7. Stage Exit Criteria + +| Stage | Exit Criteria | +| --- | --- | +| 1 | Migrations live, services stable, existing Planner/Writer flows unaffected, docs updated. | +| 2 | Wizard fully usable with telemetry + QA sign-off; planner UIs reflect new taxonomy data. | +| 3 | Writer/linker/optimizer using metadata, publishing blocked without required info, dashboards live. | +| 4 | WordPress parity achieved, deployments verified, sync dashboards green, release announced. | diff --git a/SITE_BUILDER_PLANNER_ARCHITECTURE_PLAN.md b/refactor-plan/UNIFIED_SITE_BUILDER_ARCHITECTURE.md similarity index 100% rename from SITE_BUILDER_PLANNER_ARCHITECTURE_PLAN.md rename to refactor-plan/UNIFIED_SITE_BUILDER_ARCHITECTURE.md diff --git a/refactor-plan/UNIFIED_SITE_BUILDER_IMPLEMENTATION_PLAN.md b/refactor-plan/UNIFIED_SITE_BUILDER_IMPLEMENTATION_PLAN.md new file mode 100644 index 00000000..1862cdb0 --- /dev/null +++ b/refactor-plan/UNIFIED_SITE_BUILDER_IMPLEMENTATION_PLAN.md @@ -0,0 +1,311 @@ +# IGNY8 Refactor Plan – Stage Implementation Guide + +This document complements `SITE_BUILDER_PLANNER_ARCHITECTURE_PLAN.md` by outlining how to deliver the refactor in four controlled stages. Each stage lists scope, tasks, exits, and testing so workstreams can move independently without breaking user flows. + +--- + +## Stage 1 – Data & Services Foundation +**Objective:** Build all backend scaffolding (schema + services) required for the guided site builder, richer planner, and downstream metadata. + +### 1. Sequenced Plan of Work +1. **Preparation** + - Freeze any pending migrations; branch off `refactor-stage1`. + - Create feature flags (`USE_SITE_BUILDER_REFACTOR`) in settings/env files. + - Draft ERD updates and share with backend team for approval. +2. **Schema Implementation** + - **Site Builder tables** + - `SiteBlueprintCluster`: fields for `cluster_id`, `role`, `coverage_status`, `metadata`. + - `SiteBlueprintTaxonomy`: taxonomy type enum, `cluster_id`, optional external reference. + - `WorkflowState`: blueprint FK, `current_step`, `step_status`, `blocking_reason`, timestamps. + - **Writer tables** + - `ContentClusterMap`, `ContentTaxonomyMap`, `ContentAttributeMap` with FK constraints + indexes for performant lookups. + - Run migrations locally → staging; capture SQL diff in change log. +3. **Service Layer** + - Implement `WorkflowStateService` with methods: + - `initialize(site_blueprint)` – create default states. + - `update_step(step, status, meta)` – persists transitions and emits audit logs. + - `validate_step(step)` – raises descriptive errors for UI. + - Implement `TaxonomyService`: + - CRUD handlers. + - Import adapters (WordPress categories/tags, WooCommerce product attributes). + - Helper to map taxonomy ↔ cluster(s). + - Update `ClusteringService` prompts to emit: + - Topic cluster suggestions. + - Attribute/feature recommendations per keyword set. + - Taxonomy hints (category/tag names). +4. **API/Serializer Updates** + - Planner serializers return `context_type`, `dimension_meta`. + - Writer serializers include `entity_type`, taxonomy associations, validation errors. + - Site builder endpoints expose workflow state + gating booleans. +5. **Validation Hooks** + - Add reusable validators (e.g., `ensure_clusters_attached(site_blueprint)`). + - Integrate validators into existing actions (`generate_structure`, `create_tasks_from_blueprint`). +6. **Internal Tooling & Docs** + - Update Postman collection / API reference for new endpoints + payloads. + - Write developer doc describing new models + services. + - Set up monitoring dashboards (Grafana/NewRelic) for: + - Migration duration. + - Celery task errors referencing new services. + +### 2. Owners & Dependencies +- **Backend Lead:** owns migrations, services, validator integration. +- **Data/DBA:** reviews migration plan, handles staging rollout. +- **QA:** prepares regression scripts for planner/writer. +- **Docs:** updates `master-docs` (backend + AI framework) with new tables/services. +- Dependencies: PostgreSQL 15 features already available; no external API changes needed. + +### 3. Deliverables & Exit Criteria +- ✅ Migrations applied to staging + rollback verified. +- ✅ `WorkflowStateService`, `TaxonomyService`, `ClusteringService` updates merged with unit tests. +- ✅ Planner/Writer legacy flows pass smoke tests (imports, clustering, idea creation, content generation). +- ✅ API reference + architecture docs updated. +- ✅ Feature flag defaults to OFF (no user-facing change yet). + +### 4. Testing Matrix +| Area | Automated | Manual | +| --- | --- | --- | +| Migrations | `pytest` migration test + Django fake migrations | Dry run on staging, rollback rehearsal | +| Services | Unit tests for new service methods | Postman calls to workflow/taxonomy endpoints | +| Validators | Serializer/endpoint tests expecting 4xx when requirements unmet | Trigger via API (e.g., attempt sitemap gen without clusters) | +| Regression | Existing planner/writer test suite | Smoke test: keyword import → cluster → idea → task → content | + +### 5. Rollout Checklist +1. Merge migrations + services behind feature flag. +2. Deploy to staging, run migration + regression script. +3. Monitor Celery/DB logs for anomalies. +4. After 24h stability, promote to production (still gated by flag). +5. Announce availability to frontend team to begin Stage 2 work. + +--- + +## Stage 2 – Planner + Wizard UX +**Objective:** Ship the state-aware, self-guided planner + site builder experience on top of Stage 1 services. + +### 1. Sequenced Plan of Work +1. **Planning & Design** + - UX team finalizes wizard wireframes + microcopy for each step, including helper text/tooltips. + - Define telemetry schema (events: `wizard_step_started`, `wizard_step_completed`, `wizard_blocking_issue`). + - Review backend responses needed per step (cluster metrics, taxonomy suggestions, validation errors). +2. **Backend Enhancements** + - Extend planner endpoints to return: + - Cluster stats (keyword count, projected hubs/support pages). + - Suggested taxonomies/attributes from Stage 1 `dimension_meta`. + - Validation error payloads structured for UI (“missing clusters”, “taxonomy incomplete”). + - Add audit logging to `WorkflowStateService.update_step` for troubleshooting. +3. **Frontend Infrastructure** + - Build `builderWorkflowStore` (Zustand): + - State: `currentStep`, `completedSteps`, `blockingIssues`, `workflowState`, `loading`. + - Actions: `goToStep`, `completeStep`, `refreshState`, `setBlockingIssue`. + - Persistence: sessionStorage/localStorage to resume across refreshes. + - Hook store into new `/site-builder/workflow/{blueprintId}` endpoints. +4. **Wizard Step Implementation** + - **Step 1 – Business Details** + - Components for site type selection, hosting detection (show WP integration status), brand inputs. + - Save actions call blueprint PATCH + workflow state update. + - **Step 2 – Cluster Assignment** + - Table/list of clusters with filters (intent, volume, context type). + - Coverage meter (keywords covered vs. target). + - “Add cluster” action hitting `/blueprints/{id}/clusters/attach`. + - Blocking logic: enforce minimum cluster count/keyword count per site type. + - **Step 3 – Taxonomy Builder** + - Tree/table for categories, tags, product attributes, service groups. + - Import buttons (WordPress/WooCommerce) calling `TaxonomyService` endpoints. + - Mapping UI linking taxonomy items to clusters (drag/drop or select dropdown). + - **Step 4 – AI Sitemap Review** + - Grid grouped by cluster/taxonomy. + - Visual checklist (Hub page, Supporting pages count). + - Ability to edit page titles/types, reorder, remove. + - “Regenerate” button to re-run AI structure (with warning on credit usage). + - **Step 5 – Coverage Validation** + - Summary cards showing clusters covered/uncovered, taxonomy gaps, attribute coverage. + - CTA to jump back to specific step if gaps remain. + - **Step 6 – Ideas Hand-off** + - Select pages to push to Planner Ideas. + - Secondary prompt textarea; show preview of what data AI receives. + - Confirmation modal summarizing cluster/page counts. +5. **Planner Module Enhancements** + - Cluster matrix view showing clusters vs. taxonomy/attribute coverage, clickable to drill into wizard step. + - Taxonomy management table (search, filters, bulk edits). + - Inline warnings (e.g., banner on Planner home when blueprint still missing requirements). +6. **UX Guardrails** + - Breadcrumb/progress component with checkmarks. + - Disabled Next button shows tooltip listing unmet requirements. + - Contextual helper drawer per step (links to docs, best practices). + +### 2. Owners & Dependencies +- **Frontend Lead:** wizard implementation + planner UI updates. +- **Backend Support:** ensure required data returned, logging in place. +- **Design/UX:** final visuals, helper content, accessibility review. +- **Docs/Enablement:** in-app help, video walkthroughs. +- Dependency: Stage 1 feature flag must be AVAILABLE with APIs stable. + +### 3. Deliverables & Exit Criteria +- ✅ Wizard completes end-to-end (feature-flagged) for both IGNY8-hosted and WP-linked sites. +- ✅ Telemetry events firing and visible in analytics dashboard. +- ✅ QA scripts covering success/failure states executed, accessibility issues resolved. +- ✅ Documentation (knowledge base, tooltips) updated. +- ✅ Rollout plan ready (progressive enablement per account/site type). + +### 4. Testing Matrix +| Area | Automated | Manual | +| --- | --- | --- | +| Wizard Flow | Cypress tests for each step, gating logic, resume | Scenario walkthroughs (new site, WP import), cross-browser testing | +| Planner Views | Jest/unit tests for new components, store logic | Performance check on cluster matrix, UX review | +| Telemetry | Unit tests ensuring events dispatch | Verify dashboards receiving events | +| Accessibility | Lint checks (aria labels, keyboard nav) | Screen reader + keyboard-only validation | + +### 5. Rollout Checklist +1. Feature flag ON for internal testers → gather feedback. +2. Fix critical issues, re-run automated suite. +3. Enable for pilot accounts (one IGNY8 host, one WP host). +4. Monitor telemetry/feedback; ensure support team ready. +5. Gradually roll out to all accounts; announce in release notes + in-app banner. + +--- + +## Stage 3 – Writer / Linker / Optimizer Enhancements +**Objective:** Propagate the new metadata through content creation and optimization, ensuring validation before publish. + +### 1. Sequenced Plan of Work +1. **Metadata Audit & Defaults** + - Verify Stage 1 migrations populated defaults for legacy content/tasks. + - Script to backfill `entity_type`, `taxonomy_id`, `cluster_role` for existing records (with audit log). +2. **Backend Pipeline Updates** + - **Ideas → Tasks**: update services so every task inherits cluster/taxonomy/attribute metadata. + - **Tasks → Content**: adjust `ContentPipelineService` to store mappings in `ContentClusterMap`, etc. + - **Prompt Updates**: AI prompts include cluster role, taxonomy context, product attributes to produce richer output. + - **Validation Services**: create reusable validators (e.g., `ensure_required_attributes(task)`) returning human-friendly errors. +3. **Linker & Optimizer Enhancements** + - Linker service leverages mapping tables to suggest hub/supporting/attribute links; adds priority scoring. + - Optimizer scoring factors: + - Cluster coverage (are all hubs/supporting pages present?). + - Taxonomy alignment (page categorized correctly). + - Attribute completeness (product/service specs filled). + - New APIs: + - `GET /sites/{id}/progress` (cluster-level completion + validation status). + - `GET /writer/content/{id}/validation` (aggregated checklist for UI). +4. **Frontend Enhancements** + - **Planner Ideas / Writer Tasks lists** + - Columns/chips for cluster, taxonomy, attributes. + - Filters for entity type & validation status. + - **Writer Editor** + - Sidebar module with cluster summary, taxonomy tree, attribute form. + - Validation panel showing remaining requirements; publish button disabled until cleared. + - **Linker UI** + - Group internal link suggestions by cluster role; show context snippet, CTA to insert. + - **Optimizer Dashboard** + - Scorecards per cluster dimension with color coding + “next action” cards. + - **Site Progress Widgets** + - On site overview, show bars for hubs/supporting/attribute completion; link to problematic clusters. +5. **Notifications & Workflow** + - Inline toast + optional email when validation fails or content ready for review. + - Credit reminder banner when user tries to regenerate content/optimization tasks. + +### 2. Owners & Dependencies +- **Backend Lead:** pipeline + services, validation APIs. +- **Frontend Lead:** writer/linker/optimizer UI, progress widgets. +- **AI/Prompt Specialist:** adjust prompts/templates and verify outputs. +- **QA:** regression scripts for writer workflow, linker, optimizer. +- Dependencies: Stage 2 wizard must deliver cluster/taxonomy data; Stage 1 mapping tables in place. + +### 3. Deliverables & Exit Criteria +- ✅ All new tasks/content carry entity/taxonomy metadata; legacy content backfilled. +- ✅ Publish action blocked when required metadata missing; errors surfaced clearly. +- ✅ Linker/optimizer screens powered by new data with acceptable performance. +- ✅ Site progress dashboards live for pilot sites with feedback loop established. + +### 4. Testing Matrix +| Area | Automated | Manual | +| --- | --- | --- | +| Pipeline | Unit tests on service functions, serializer coverage | End-to-end run: blueprint → ideas → tasks → content | +| Validation | Tests ensuring validators trigger proper errors | Attempt publish without taxonomy/attributes; confirm UI flow | +| Linker/Optimizer | Service tests for scoring & suggestions | Performance profiling on large datasets; UX review | +| Progress Widgets | Component/unit tests | Compare counts vs. database for pilot site | + +### 5. Rollout Checklist +1. Deploy to staging with feature flag ON; run regression suite. +2. Backfill metadata for existing content in staging; verify no issues. +3. Pilot with internal content team; gather feedback on validation friction. +4. Optimize slow queries (add indexes/caching) before production rollout. +5. Update training docs/videos for writers; enable flag gradually across accounts. + +--- + +## Stage 4 – Publishing & Sync Integration +**Objective:** Achieve feature parity between IGNY8-hosted deployments and WordPress sites, using the shared metadata model. + +### 1. Sequenced Plan of Work +1. **Sync Architecture Review** + - Audit existing WordPress adapter + sync service. + - Confirm required endpoints (categories, tags, WooCommerce attributes/products) and authentication flows. + - Document data mapping between WP taxonomies and new IGNY8 taxonomy tables. +2. **Backend Enhancements** + - **Import** + - Extend `ContentSyncService` to fetch WP taxonomies, products, custom post types. + - Map external IDs to `SiteBlueprintTaxonomy`, `ContentTaxonomyMap`, storing `external_reference`. + - Auto-create missing clusters/taxonomies with flag indicating “imported”. + - **Export** + - Update `WordPressAdapter` to create/update taxonomies before publishing posts/products. + - Ensure product attributes/tags pushed before product content. + - **Sync Health** + - New endpoints: `/sites/{id}/sync/status`, `/sites/{id}/sync/run`. + - Track last sync time, mismatch counts, error logs. + - **Deployment** + - `SitesRendererAdapter` consumes cluster/taxonomy metadata to build navigation, breadcrumbs, internal links. + - Deployment readiness check service summarizing cluster coverage, content publish status, validation flags. + - **Logging/Monitoring** + - Structured logs for sync runs (duration, items processed, failures). + - Alerts for repeated sync failures or deployment errors. +3. **Frontend Enhancements** + - **Sync Dashboard** + - Parity indicators (taxonomies, products, posts) with status icons. + - Buttons for manual sync, retry failed items, view logs. + - Detail drawer showing mismatched items and suggested fixes. + - **Deployment Panel** + - Readiness checklist (clusters covered, content statuses, validation passes). + - Deploy + rollback buttons with confirmation modals. + - Toast/notification system for deploy/sync success/failure. + - **WordPress Connection UI** + - Surface integration status, credentials check, last sync info. +4. **Operational Runbooks** + - Document sync troubleshooting steps, rollback procedures, and escalation path. + - Provide support scripts for forcing sync, clearing queue, recovering failed deployment. + +### 2. Owners & Dependencies +- **Backend Lead:** sync/import/export logic, deployment readiness service. +- **Frontend Lead:** sync dashboard, deployment UI. +- **Infra/DevOps:** monitoring, alerting, runbooks. +- **QA:** end-to-end publish tests IGNY8↔WP. +- Dependencies: Stage 3 metadata stable; WordPress credentials for pilot sites available. + +### 3. Deliverables & Exit Criteria +- ✅ End-to-end publishing works for blog/ecommerce/company templates on IGNY8 hosting and WordPress (posts/products/services). +- ✅ Sync dashboard shows green status for pilot accounts; manual retry resolves mismatches. +- ✅ Deployment readiness checks enforced; logs + rollback procedures documented. +- ✅ Release notes + training material published. + +### 4. Testing Matrix +| Area | Automated | Manual | +| --- | --- | --- | +| Sync Logic | Integration tests hitting mocked WP APIs | Staging sync from live WP instance; verify taxonomy parity | +| Deployment | Renderer tests verifying metadata usage | Deploy IGNY8 site, inspect navigation/internal links | +| Dashboards | Component/unit tests | Pilot user testing of sync dashboard + deployment flow | +| Runbooks | N/A | Tabletop exercises for failure scenarios (sync fails, deploy rollback) | + +### 5. Rollout Checklist +1. Enable Stage 4 flag in staging; run full sync/import tests. +2. Pilot with one IGNY8-hosted and one WP-hosted site; gather feedback. +3. Train support team on new dashboards/runbooks. +4. Announce availability; roll out gradually to accounts with WordPress integrations. +5. Monitor logs/alerts closely during first production syncs; iterate on tooling as needed. + +--- + +## Stage Coordination Notes +- **Feature flags**: introduce per-stage flags to roll out gradually. +- **Monitoring**: add dashboards for workflow step failures, sync errors, credit usage spikes. +- **Documentation cadence**: update `master-docs` + `part2-dev/planning` after each stage completes. +- **Entry/exit reviews**: hold short readiness reviews before moving to the next stage to ensure dependencies satisfied. + +This plan ensures each layer stabilizes before the next, keeping users on a guided, predictable experience throughout the refactor. +