reference plugin and image gen analysis

igny8-ai-seo-wp-plugin/docs/MIGRATION_PLAN_EXTRACTED.txt

@@ -0,0 +1,524 @@
+IGNY8 WP PLUGIN TO IGNY8 APP MIGRATION PLAN
+============================================
+
+SECTION 1 - CODEBASE & DIRECTORY STRUCTURE (IGNY8 APP)
+Goal: Design the top-level and module-level folder hierarchy for the new Django + React (Vite + Tailwind) full-stack IGNY8 App, preserving 1:1 parity with the current WordPress plugin modules while introducing clear API, automation, and AI boundaries.
+
+1.1 Top-Level Repository Layout
+igny8-app/
+  backend/                    # Django backend (core API, models, automation)
+    igny8_core/              # Main Django app (replaces plugin core)
+      __init__.py
+      settings.py            # Env-aware settings (local / staging / prod)
+      urls.py
+      wsgi.py
+      asgi.py
+      modules/               # 1:1 migration of plugin modules
+        planner/
+          models.py
+          views.py
+          serializers.py
+          tasks.py           # CRON/async/queue handlers
+          urls.py
+        writer/
+        thinker/
+        ... (future: linker, optimizer, etc.)
+      api/                   # Unified REST entrypoints
+        routes/
+          ai/
+            content.py
+            images.py
+            publish.py
+            reparse.py
+          system/
+        middleware/
+      auth/                  # Multi-tenant + RBAC + Stripe linkage
+        models.py
+        roles.py
+        views.py
+        signals.py
+        serializers.py
+      utils/                 # Shared helpers (AI, logs, etc.)
+    cron/                    # Timed jobs (mapped from plugin CRON)
+    migrations/
+    manage.py
+  frontend/                  # React + Vite + Tailwind UI
+    src/
+      app/
+        globals.css
+        layout.tsx
+        pages/               # Page-per-module parity with WP admin
+          PlannerPage.tsx
+          WriterPage.tsx
+          ThinkerPage.tsx
+          DashboardPage.tsx
+        components/
+          ui/                # Local UI library (Card, Button, etc.)
+          layout/
+          charts/
+        hooks/
+        services/            # API clients (axios/fetch for each module)
+        store/               # Zustand/Redux state
+      vite.config.js
+      package.json
+      tsconfig.json
+  infra/                     # Docker, Portainer, Caddy, Postgres, Redis
+    docker-compose.yml
+    caddy/
+    scripts/
+    dev-config.yml
+    README.md
+  docs/
+    MIGRATION_PLAN.md
+    API_REFERENCE.md
+    DATABASE_MAP.md
+  .env / .env.dev / .env.prod  # environment-specific configs
+
+1.2 Key Design Rules
+- Modules: Each former WordPress module = isolated Django app inside /backend/modules/
+- API Layer: Use /backend/api/routes/{domain}/ for all external triggers
+- Frontend: Page = Module parity (PlannerPage.tsx, etc.)
+- UI Library: /frontend/src/components/ui/ replaces @/components/ui/ from TailAdmin
+- DevOps: Single infra/docker-compose.yml defines the full stack
+- Roles/Billing: /backend/auth/roles.py + Stripe integration
+- Docs: /docs folder in-repo
+
+1.3 Role-Based Access (RBAC) Foundation
+Roles:
+- Owner: Full access to all tenants, billing, and automation
+- Admin: Manage content modules, view billing but not edit
+- Editor: Generate AI content, manage clusters/tasks
+- Viewer: Read-only dashboards
+- SystemBot: Automation + CRON actions (No UI, API-only)
+
+1.4 AI-Assisted Dev Environment (Cursor / Local PC)
+- Expose selective sub-repos /backend/modules/, /frontend/src/, /docs/ to GitHub/GitLab
+- Keep /infra & .env excluded (.gitignore) for security
+- AI Context Indexing: Each module includes README.md describing schema, routes, and functions
+- Dockerized Dev Setup: Cursor runs Node + Python containers via docker-compose
+- Source Sync Hooks: Git hooks ensure migrations and schema docs stay synchronized
+
+
+SECTION 2 - DATABASE & SCHEMA MIGRATION PLAN
+Goal: Migrate all IGNY8 custom database tables and essential data from WordPress to a Django + PostgreSQL schema.
+
+2.1 IGNY8 Core Tables to Migrate 1:1
+WP Table → Django Model:
+- igny8_keywords → Keywords
+- igny8_clusters → Clusters
+- igny8_content_ideas → ContentIdeas
+- igny8_tasks → Tasks
+- igny8_variations → Variations
+- igny8_prompts → Prompts
+- igny8_logs → Logs
+- igny8_images → Images
+- igny8_schedules → Schedules
+- igny8_settings → Settings (Split into SystemSettings and UserSettings)
+- igny8_accounts → Accounts
+- igny8_links → Links (Future)
+- igny8_metrics → Metrics
+
+2.2 WordPress Default Tables (Partial Extraction)
+- wp_posts → WPPosts (ID, post_title, post_content, post_status, post_type, post_date)
+- wp_postmeta → WPPostMeta (post_id, meta_key, meta_value - only IGNY8 keys)
+- wp_users → WPUserMap (ID, user_email, display_name)
+- wp_options → WPOptions (option_name, option_value - plugin config only)
+
+2.3 Cross-System Integration Map
+- Keywords/Clusters: App → WP via WP REST API
+- Tasks/Content: App → WP via WP /wp-json/igny8/v1/import-content
+- Images: App → WP via WP Media REST endpoint
+- Settings: Bidirectional sync
+- Logs/Metrics: App-only
+
+2.4 Schema Example: Tasks Model (Django ORM)
+Includes: id, cluster, idea, task_type, ai_model, prompt, raw_ai_response, status, result, timestamps, user, tenant
+
+2.5 Database Migration & Sync Process
+Steps: Export → Create Django models → Import data → Verify → Migrate WP data → Run integrity tests → Enable periodic sync
+
+2.6 Tenant-Aware Model Design
+Every model inherits from TenantBaseModel with tenant, created_at, updated_at fields.
+
+
+SECTION 3 - COMPONENT & UI MAPPING PLAN
+Goal: Rebuild the WordPress plugin's admin interface as React/Vite pages.
+
+3.1 One-to-One Module → Page Mapping
+- Planner Dashboard → PlannerPage.tsx (/planner)
+- Writer Dashboard → WriterPage.tsx (/writer)
+- Thinker → ThinkerPage.tsx (/thinker)
+- Optimizer (Phase 2) → OptimizerPage.tsx (/optimizer)
+- Linker (Phase 2) → LinkerPage.tsx (/linker)
+- Settings → SettingsPage.tsx (/settings)
+- Dashboard → DashboardPage.tsx (/dashboard)
+
+3.2 Shared Frontend Component Map
+- DataTable, FilterPanel, FormModal, ActionButtons, StatsCards, ToastNotifications, AIStatusBar, ConfirmDialog, TenantSwitcher
+
+3.3 UI Data Flow Diagram (Per Module)
+User → React Page → Zustand Store → API Service → Django API → AI Processing → Database → React Refresh
+
+3.4 State Management Rules (Zustand)
+Each module defines its store following the same interface pattern.
+
+
+SECTION 4 - API ROUTES & AUTOMATION TRIGGERS
+Goal: Define all Django REST API endpoints and automation triggers.
+
+4.1 Unified API Structure
+- /api/planner/ - Keyword → Cluster → Idea generation
+- /api/writer/ - Content generation, reparsing, image injection
+- /api/thinker/ - Prompt testing, image model handling
+- /api/settings/ - Model rates, limits, keys, Stripe data
+- /api/system/ - Metrics, logs, and CRON control
+- /api/auth/ - Tenant registration, login, roles
+- /api/publish/ - Push content/images to WP site
+
+4.2 Example Endpoint Tree
+Detailed endpoint listing for each module with GET/POST/DELETE methods
+
+4.3 API → AI Execution Chain
+Common AI service layer (/backend/utils/ai.py) handles all AI requests
+
+4.4 Automation Triggers (Replacing WordPress CRON)
+- auto_cluster_keywords: every 6h
+- auto_generate_ideas: daily
+- auto_generate_content: hourly
+- auto_generate_images: hourly
+- auto_publish_posts: daily
+- cleanup_logs: weekly
+
+4.5 Queue Control Logic
+Shared queue core (/backend/utils/queue_manager.py) for automation and manual triggers
+
+4.6 Stripe Integration (API)
+Endpoints for billing, subscription, webhooks
+
+4.7 AI Key & Model Management
+Configured through /api/settings/ai-models
+
+4.8 Example: /api/writer/generate/<id> Workflow
+8-step process from React frontend to WordPress publish
+
+4.9 Authentication & Role Guard Middleware
+All endpoints protected by RoleRequired decorator
+
+4.10 CRON & API Coherence
+CRON functions reuse the same endpoints as manual buttons
+
+
+SECTION 5 - WORDPRESS INTEGRATION & SYNC LAYER
+Goal: Design a clean, secure bridge between Django backend and WordPress sites.
+
+5.1 Integration Overview
+- AI Content Publish: App → WP
+- Image Upload: App → WP
+- Settings Sync: Bidirectional
+- Post Status Check: WP → App
+- Keyword/Cluster Sync: App → WP
+- Auth Link: Manual JWT-based connection
+
+5.2 Connection Setup (One-Time Auth Handshake)
+WPIntegration model stores site_url, api_key, tenant, status, last_sync
+
+5.3 WordPress REST Endpoints (Added to Plugin)
+- /wp-json/igny8/v1/import-content
+- /wp-json/igny8/v1/upload-image
+- /wp-json/igny8/v1/sync-settings
+- /wp-json/igny8/v1/status
+- /wp-json/igny8/v1/pull-updates
+
+5.4 Django → WordPress Communication Flow
+Example publishing workflow with request/response structure
+
+5.5 WordPress Plugin: Receiving Side Implementation
+Minimalistic handler example code
+
+5.6 Bidirectional Settings Sync
+Field mapping between App and WP
+
+5.7 Multi-Tenant Integration Mapping
+Each tenant can connect multiple WP sites
+
+5.8 Sync Safety & Rate Control Layer
+Queue throttling, error handling, fallback mode, audit logs
+
+5.9 Security Model
+JWT with 12h expiry, HMAC signatures, CORS whitelist
+
+5.10 Example: Full Publish & Feedback Cycle
+Complete workflow from user action to periodic sync
+
+5.11 Future Extension (Phase-2+)
+Planned integrations for metrics, links, user data, schema
+
+
+SECTION 6 - DEPLOYMENT, ENVIRONMENT & LOCAL DEV PLAN
+Goal: Define complete development, deployment, and synchronization environment.
+
+6.1 Full Stack Architecture Summary
+- Reverse Proxy: Caddy (80/443)
+- Frontend: React (Vite) + Node 20 (8020→8021)
+- Backend: Django + Gunicorn (8010→8011)
+- Database: PostgreSQL 15 (5432)
+- Cache/Queue: Redis 7 (6379)
+- Admin DB UI: pgAdmin4 (5050)
+- File Manager: Filebrowser (8080)
+- Docker Manager: Portainer CE (9443/8000)
+
+6.2 Environment File (.env)
+Variables for DB, Redis, AI keys, Stripe, domains
+
+6.3 Docker Compose Structure
+Service definitions with volumes, networks, environment variables
+
+6.4 Local Development Setup (Cursor AI)
+Steps for local development with live mounts
+
+6.5 Portainer Stack Deployment
+Production deployment via Portainer stacks
+
+6.6 Environment-Specific Configs
+Separate configs for local, staging, production
+
+6.7 Backup & Recovery Procedures
+Automated backups for database, media, configs
+
+
+SECTION 7 - DATA MIGRATION & SYNC STRATEGY
+Goal: Extract, transform, and import all IGNY8 plugin data from WordPress MySQL to PostgreSQL.
+
+7.1 Migration Overview
+Extract from WP → Transform schema → Import to Django → Validate → Sync
+
+7.2 Table Mapping Details
+Complete mapping of all WP tables to Django models
+
+7.3 Migration Phases
+1. Extraction (Dump plugin tables)
+2. Transformation (Convert to Postgres schema)
+3. Import (Bulk insert via Django)
+4. Validation (Compare counts, hashes)
+5. Sync (Enable real-time sync to WP)
+
+7.4 Extraction Script Example
+Python script using mysql.connector
+
+7.5 Transformation Example
+Data transformation script
+
+7.6 Import to Django
+Django management command for bulk import
+
+7.7 Verification Step
+Comparison script for validation
+
+7.8 Syncable Tables (Remain Linked to WP)
+Tables that maintain bidirectional sync
+
+7.9 Migration Validation Dashboard
+UI section showing migration status
+
+7.10 Rollback Strategy
+Procedure for handling migration failures
+
+7.11 Final Verification Checklist
+Checkpoints for successful migration
+
+7.12 Post-Migration Tasks
+Deactivate old plugin CRON, update plugin config to Remote Mode
+
+
+SECTION 8 - TENANCY, BILLING & USER ACCESS MODEL
+Goal: Define multi-tenant access, workspace isolation, role permissions, and subscription billing.
+
+8.1 Core Concepts
+- Tenant: Logical workspace
+- User: Authenticated account inside tenant
+- Subscription: Stripe-linked billing plan
+- Workspace: UI grouping under tenant
+- Site Integration: Connected WordPress instances
+
+8.2 Data Model Overview
+Django Models: Tenant, User, Plan, Subscription
+
+8.3 Stripe Integration Workflow
+6-step process from plan selection to webhook handling
+
+8.4 Credit System Logic
+Credit costs per action:
+- Keyword clustering: 1 credit / 30 keywords
+- Content idea generation: 1 credit / idea
+- Full blog content: 3 credits
+- AI image generation: 1 credit / image
+- Reparse content: 1 credit
+- Auto publish: Free if already generated
+
+8.5 Roles & Access Permissions
+Owner, Admin, Editor, Viewer roles with specific permissions
+
+8.6 Tenant Isolation Enforcement
+Database-level, file-level, API-level, worker-level isolation
+
+8.7 Tenant Dashboard Layout (Frontend)
+React components for tenant management
+
+8.8 Billing Plans (Finalized Baseline)
+- Free/Trial: $1, 25 credits, 1 user, 1 site
+- Starter: $39, 200 credits, 3 users, 3 sites
+- Pro: $89, 600 credits, 5 users, 5 sites
+- Agency: $199, 2000 credits, 15 users, 10 sites
+- Custom/Enterprise: Variable
+
+8.9 Webhooks for Sync
+Stripe and WordPress webhook handlers
+
+8.10 Suspended Tenant Behavior
+Handling payment failures
+
+8.11 Multi-Tenant CRON Scheduler
+Per-tenant queue system
+
+8.12 Cross-Module Tenant Integration
+How each module enforces tenant boundaries
+
+8.13 Stripe Integration Files
+Backend implementation structure
+
+8.14 Security Enforcement
+Authentication, authorization, webhook validation, file access, admin audit
+
+8.15 Example Flow - New Tenant Signup
+Complete signup to activation workflow
+
+
+SECTION 9 - AI AUTOMATION PIPELINE & TASK ENGINE
+Goal: Establish unified, tenant-aware automation system for all AI-based tasks.
+
+9.1 Core Design Principles
+- Single-Item Execution
+- Tenant Isolation
+- Unified Scheduler
+- Configurable Limits
+- Recoverable Jobs
+
+9.2 AI Task Types & Flow
+- Planner: Keyword Clustering → Idea Generation
+- Writer: AI Draft Generation → Reparse → Publish
+- Thinker: Prompt Creation / Persona / Strategy
+- Image Generator: DALL-E / Runware image tasks
+- Linker (Phase-2): Backlink planner
+
+9.3 Queue Architecture (Redis-Backed)
+Redis 7.x with Celery or django-q, tenant-based queue naming
+
+9.4 AI Execution Workflow
+Scheduler → Redis Queue → Django Worker → AI Engine API → Parser Layer → DB + Logs
+
+9.5 Execution Limits & Global Settings
+Parameters for AI_MAX_ITEMS_PER_REQUEST, batch sizes, timeouts, retries, cost caps
+
+9.6 Task Lifecycle (Writer Example)
+Queued → Dispatch → AI Request → Response Handling → Validation → Storage → Optional Actions
+
+9.7 CRON Automation Flows
+- cron_auto_cluster(): 6h
+- cron_auto_ideas(): 6h
+- cron_auto_writer(): 3h
+- cron_auto_image(): 3h
+- cron_auto_publish(): 12h
+- cron_cleanup_logs(): 24h
+
+9.8 AI Model Routing Logic
+Model selector with auto-balancing and fallback
+
+9.9 AI Pipeline Directory Map
+Backend module structure for AI pipeline
+
+9.10 Credit Deduction Example
+Credit deduction logic
+
+9.11 Error Recovery Flow
+Handling timeouts, invalid JSON, parser failures, sync failures, credit errors
+
+9.12 Frontend Task Control (React Components)
+TaskQueueTable, TaskControlPanel, AutomationConfigForm with WebSocket feed
+
+9.13 Monitoring & Telemetry
+Structured logging with tenant, task_type, status, duration, model, credits_spent, output_size
+
+9.14 Local AI Dev Mode (Cursor-Ready)
+Development configuration for local testing
+
+9.15 Verification Checklist
+Checkpoints for queue, worker, task enqueue, credit deduction, AI response, CRON logs
+
+9.16 Future Enhancements
+Parallel pipelines, semantic caching, rate shaping, cost anomaly alerts, tracing integration
+
+
+SECTION 10 - SECURITY, LOGGING & MONITORING FRAMEWORK
+Goal: Define full-stack security, audit, and observability framework.
+
+10.1 Security Architecture Overview
+Layers: Frontend, Backend, Database, AI Layer, Infrastructure, Integrations
+
+10.2 Authentication & Token System
+JWT authentication with tenant context injection, role verification, session expiry
+
+10.3 Authorization & Role-Based Rules
+Role access scope definitions
+
+10.4 Secure API Architecture
+Endpoint pattern, JWT verification, tenant match, role access, input validation
+
+10.5 Stripe & Webhook Security
+HMAC validation for webhooks
+
+10.6 Data Encryption & Storage Policy
+Encryption for credentials/keys, storage policies for AI responses, files, backups, logs
+
+10.7 API Rate Limiting
+Per tenant (30 req/min), per IP (100 req/min), per worker (1 AI request/iteration), per model (60 req/min)
+
+10.8 Logging System Overview
+Centralized logging: App → Django Logger → Postgres → Loki Stream → Grafana
+
+10.9 Monitoring & Alerts Stack
+Tools: Portainer, Loki, Grafana, Alertmanager, Redis Inspector
+Custom alerts for AI timeout, failed tasks, Stripe failures, CRON lag
+
+10.10 Data Backup & Recovery
+Backup frequencies and retention:
+- Postgres DB: Every 6 hours, 7 days retention
+- FileBrowser/Media: Daily, 14 days
+- Caddy/Config: Daily, 7 days
+- Logs (Loki): Rolling window, 30 days
+
+10.11 Error & Exception Handling
+Handling for AI API errors, DB write errors, worker crashes, CRON failures, user API errors
+
+10.12 Developer Audit Trail
+Critical system events logged in igny8_audit with 1 year retention
+
+10.13 Local Dev & Security Mirrors
+Development configuration with AI_DEV_MODE, CORS settings, mock data
+
+10.14 Security Verification Checklist
+Checkpoints for HTTPS, JWT validation, tenant isolation, webhook verification, file access, backups, Redis security, network isolation, rate limiting
+
+10.15 Future Enhancements
+OAuth 2.0 login, tenant-level dashboards, ElasticSearch sink, offsite sync, smart anomaly detection
+
+10.16 Final Summary
+Security model fully container-aware and tenant-isolated
+Logging & metrics unified under Loki + Grafana
+Stripe billing, AI cost tracking, and access control audited
+Developer-friendly dev mode supported
+Production deployment validated under current Docker infrastructure
+
+
+END OF DOCUMENT
+
+