diff --git a/README.md b/README.md index c5847755..605ac334 100644 --- a/README.md +++ b/README.md @@ -81,39 +81,50 @@ docker-compose -f docker-compose.app.yml up ### ✅ Implemented - **Foundation**: Multi-tenancy system, Authentication (login/register), RBAC permissions -- **Planner Module**: Keywords & Clusters (full CRUD, filtering, pagination, bulk operations, CSV import/export) +- **Planner Module**: Keywords, Clusters, Content Ideas (full CRUD, filtering, pagination, bulk operations, CSV import/export) +- **Writer Module**: Tasks, Content, Images (full CRUD, content generation, image generation) +- **Thinker Module**: Prompts, Author Profiles, Strategies, Image Testing +- **System Module**: Settings, Integrations (OpenAI, Runware), AI Prompts +- **Billing Module**: Credits, Transactions, Usage Logs +- **AI Functions**: 5 AI operations (Auto Cluster, Generate Ideas, Generate Content, Generate Image Prompts, Generate Images) - **Frontend**: Complete component library, 4 master templates, config-driven UI system -- **Backend**: REST API with tenant isolation, Site > Sector hierarchy +- **Backend**: REST API with tenant isolation, Site > Sector hierarchy, Celery async tasks - **Development**: Docker Compose setup, hot reload, TypeScript + React ### 🚧 In Progress -- Content Ideas module (backend + frontend) -- AI integration for auto-clustering and idea generation - Planner Dashboard enhancement with KPIs +- WordPress integration (publishing) +- Automation & CRON tasks ### 🔄 Planned -- Writer module (Tasks, Drafts, Published) -- Thinker module (Prompts, Strategies, Image Testing) -- AI Pipeline infrastructure -- WordPress integration -- Automation & CRON tasks +- Analytics module enhancements +- Advanced scheduling features +- Additional AI model integrations ## 🔗 API Endpoints -- **Keywords**: `/api/planner/keywords/` +- **Planner**: `/api/v1/planner/keywords/`, `/api/v1/planner/clusters/`, `/api/v1/planner/ideas/` +- **Writer**: `/api/v1/writer/tasks/`, `/api/v1/writer/images/` +- **System**: `/api/v1/system/settings/` +- **Billing**: `/api/v1/billing/` - **Admin**: `/admin/` +See `docs/04-BACKEND.md` for complete API reference. + ## 📖 Documentation -All documentation is consolidated in the `/docs/` folder: +All documentation is consolidated in the `/docs/` folder. Start with `docs/README.md` for the complete documentation index. -- **`docs/01-ARCHITECTURE.md`** - System architecture, design patterns, and key principles -- **`docs/02-IMPLEMENTATION-ROADMAP.md`** - Complete build roadmap with 21 phases -- **`docs/03-CURRENT-STATUS.md`** - Current progress, completed items, and next steps -- **`docs/04-API-REFERENCE.md`** - API endpoints reference guide -- **`docs/05-WP-MIGRATION-MAP.md`** - WordPress plugin to Django app migration reference +### Core Documentation +- **`docs/README.md`** - Documentation index and navigation +- **`docs/01-ARCHITECTURE-TECH-STACK.md`** - Technology stack and system architecture +- **`docs/02-APP-ARCHITECTURE.md`** - Application architecture with complete workflows +- **`docs/03-FRONTEND.md`** - Complete frontend documentation +- **`docs/04-BACKEND.md`** - Complete backend documentation +- **`docs/05-AI-FUNCTIONS.md`** - Complete AI functions documentation +- **`docs/06-CHANGELOG.md`** - System changelog -**Quick Start**: Read `docs/03-CURRENT-STATUS.md` for current state, then `docs/02-IMPLEMENTATION-ROADMAP.md` for what to build next. +**Quick Start**: Read `docs/README.md` for navigation, then start with `docs/01-ARCHITECTURE-TECH-STACK.md` for system overview. ## 🛠️ Development diff --git a/docs/01-ARCHITECTURE-TECH-STACK.md b/docs/01-ARCHITECTURE-TECH-STACK.md new file mode 100644 index 00000000..8f2df388 --- /dev/null +++ b/docs/01-ARCHITECTURE-TECH-STACK.md @@ -0,0 +1,476 @@ +# IGNY8 Architecture & Technology Stack + +**Last Updated:** 2025-01-XX +**Purpose:** Complete technology stack and architecture overview for the IGNY8 platform. + +--- + +## Table of Contents + +1. [Executive Summary](#executive-summary) +2. [Technology Stack](#technology-stack) +3. [System Architecture](#system-architecture) +4. [Core Architecture Principles](#core-architecture-principles) +5. [Infrastructure Components](#infrastructure-components) +6. [External Service Integrations](#external-service-integrations) +7. [Deployment Architecture](#deployment-architecture) + +--- + +## Executive Summary + +**IGNY8** is a full-stack SaaS platform for SEO keyword management and AI-driven content generation. The system is built with modern technologies and follows a multi-tenant architecture with complete account isolation. + +### Key Metrics + +- **Architecture**: Multi-tenant SaaS with account isolation +- **Backend**: Django 5.2+ with Django REST Framework +- **Frontend**: React 19 with TypeScript +- **Database**: PostgreSQL 15 +- **Task Queue**: Celery with Redis +- **Deployment**: Docker-based containerization +- **Reverse Proxy**: Caddy (HTTPS termination) +- **AI Functions**: 5 primary AI operations +- **Modules**: 4 core modules (Planner, Writer, System, Billing) + +--- + +## Technology Stack + +### Backend Stack + +| Component | Technology | Version | Purpose | +|-----------|------------|---------|---------| +| **Framework** | Django | 5.2+ | Web framework | +| **API Framework** | Django REST Framework | Latest | RESTful API | +| **Database** | PostgreSQL | 15 | Primary database | +| **Task Queue** | Celery | Latest | Asynchronous tasks | +| **Cache/Broker** | Redis | 7 | Celery broker & caching | +| **Authentication** | JWT | SimpleJWT | Token-based auth | +| **HTTP Client** | Requests | Latest | External API calls | +| **WSGI Server** | Gunicorn | Latest | Production server | + +### Frontend Stack + +| Component | Technology | Version | Purpose | +|-----------|------------|---------|---------| +| **Framework** | React | 19 | UI library | +| **Language** | TypeScript | Latest | Type safety | +| **Build Tool** | Vite | Latest | Build tool & dev server | +| **Styling** | Tailwind CSS | Latest | Utility-first CSS | +| **State Management** | Zustand | Latest | Lightweight state | +| **Routing** | React Router | v6 | Client-side routing | +| **HTTP Client** | Fetch API | Native | API communication | + +### Infrastructure Stack + +| Component | Technology | Purpose | +|-----------|------------|---------| +| **Containerization** | Docker | Application containers | +| **Orchestration** | Docker Compose | Multi-container orchestration | +| **Reverse Proxy** | Caddy | HTTPS termination & routing | +| **Database Admin** | pgAdmin | PostgreSQL administration | +| **File Management** | FileBrowser | Web-based file management | +| **Container Management** | Portainer | Docker container management | + +### External Services + +| Service | Purpose | Integration | +|---------|---------|-------------| +| **OpenAI API** | Text generation (GPT models) | API integration | +| **OpenAI DALL-E** | Image generation | API integration | +| **Runware API** | Alternative image generation | API integration | +| **WordPress** | Content publishing | REST API integration | +| **Stripe** | Payment processing | Webhook integration (planned) | + +--- + +## System Architecture + +### High-Level Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Client Layer │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ Browser │ │ Mobile │ │ Admin │ │ +│ │ (React) │ │ (Future) │ │ Panel │ │ +│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ +└─────────┼──────────────────┼──────────────────┼─────────────┘ + │ │ │ + └──────────────────┼──────────────────┘ + │ +┌────────────────────────────┼──────────────────────────────┐ +│ Reverse Proxy Layer │ +│ ┌───────────────┐ │ +│ │ Caddy │ │ +│ │ (HTTPS/443) │ │ +│ └───────┬───────┘ │ +└────────────────────────────┼──────────────────────────────┘ + │ +┌────────────────────────────┼──────────────────────────────┐ +│ Application Layer │ +│ ┌──────────────┐ ┌──────────────┐ │ +│ │ Frontend │ │ Backend │ │ +│ │ (React) │◄─────────────┤ (Django) │ │ +│ │ Port 8021 │ REST API │ Port 8011 │ │ +│ └──────────────┘ └──────┬───────┘ │ +│ │ │ +│ ┌────────┴────────┐ │ +│ │ Celery Worker │ │ +│ │ (Async Tasks) │ │ +│ └────────┬────────┘ │ +└───────────────────────────────────────┼──────────────────┘ + │ +┌───────────────────────────────────────┼──────────────────┐ +│ Data Layer │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ PostgreSQL │ │ Redis │ │ Storage │ │ +│ │ (Database) │ │ (Cache/Broker)│ │ (Files) │ │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +└──────────────────────────────────────────────────────────┘ + │ +┌───────────────────────────────────────┼──────────────────┐ +│ External Services │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ OpenAI │ │ Runware │ │ WordPress │ │ +│ │ (GPT/DALL-E)│ │ (Images) │ │ (Publish) │ │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +└──────────────────────────────────────────────────────────┘ +``` + +### Request Flow + +``` +1. User Request + ↓ +2. Browser (React Frontend) + ↓ +3. Caddy Reverse Proxy (HTTPS Termination) + ↓ +4. Django Backend (API Endpoint) + ↓ +5. AccountContextMiddleware (Account Isolation) + ↓ +6. ViewSet (Business Logic) + ↓ +7. Serializer (Validation) + ↓ +8. Model (Database) + ↓ +9. Response (JSON) + ↓ +10. Frontend (UI Update) +``` + +### AI Task Flow + +``` +1. User Action (e.g., "Auto Cluster Keywords") + ↓ +2. Frontend API Call + ↓ +3. Backend Endpoint (ViewSet Action) + ↓ +4. Celery Task Queued + ↓ +5. Task ID Returned to Frontend + ↓ +6. Frontend Polls Progress Endpoint + ↓ +7. Celery Worker Processes Task + ↓ +8. AIProcessor Makes API Calls + ↓ +9. Results Saved to Database + ↓ +10. Progress Updates Sent + ↓ +11. Frontend Displays Results +``` + +--- + +## Core Architecture Principles + +### 1. Configuration-Driven Everything + +**Principle**: Zero HTML/JSX duplication - All UI rendered from configuration. + +**Implementation**: +- **Frontend**: Config files in `/config/pages/` and `/config/snippets/` +- **Backend**: DRF serializers and ViewSet actions +- **Templates**: 4 universal templates (Dashboard, Table, Form, System) + +**Benefits**: +- Single source of truth +- Easy maintenance +- Consistent UI/UX +- Rapid feature development + +### 2. Multi-Tenancy Foundation + +**Principle**: Complete account isolation with automatic filtering. + +**Implementation**: +- All models inherit `AccountBaseModel` +- All ViewSets inherit `AccountModelViewSet` +- Middleware injects account context from JWT +- Site > Sector hierarchy for content organization + +**Benefits**: +- Data security +- Scalability +- Resource isolation +- Simplified access control + +### 3. Template System (4 Universal Templates) + +**Principle**: Reusable templates for all page types. + +| Template | Purpose | Usage | +|----------|---------|-------| +| **DashboardTemplate** | Module home pages | KPIs, workflow steps, charts | +| **TablePageTemplate** | CRUD table pages | Keywords, Clusters, Tasks, etc. | +| **FormPageTemplate** | Settings/form pages | Settings, Integration, etc. | +| **SystemPageTemplate** | System/admin pages | Logs, Status, Monitoring | + +### 4. Unified AI Processor + +**Principle**: Single interface for all AI operations. + +**Implementation**: +- Single `AIEngine` class orchestrates all AI operations +- All AI functions inherit from `BaseAIFunction` +- Manual and automated workflows use same functions +- Account-specific API keys and model configuration + +**Benefits**: +- Code reusability +- Consistent error handling +- Unified logging +- Easy to extend + +### 5. Module-Based Organization + +**Principle**: Clear module boundaries with shared utilities. + +**Modules**: +- **Planner**: Keywords, Clusters, Ideas +- **Writer**: Tasks, Content, Images +- **System**: Settings, Prompts, Integration +- **Billing**: Credits, Transactions, Usage +- **Auth**: Accounts, Users, Sites, Sectors + +--- + +## Infrastructure Components + +### Docker Architecture + +The system uses a two-stack Docker architecture: + +1. **Infrastructure Stack (`igny8-infra`)**: Shared services +2. **Application Stack (`igny8-app`)**: Application-specific services + +### Infrastructure Stack Services + +| Service | Container Name | Port | Purpose | +|---------|----------------|------|---------| +| **PostgreSQL** | `igny8_postgres` | 5432 (internal) | Database | +| **Redis** | `igny8_redis` | 6379 (internal) | Cache & Celery broker | +| **pgAdmin** | `igny8_pgadmin` | 5050:80 | Database administration | +| **FileBrowser** | `igny8_filebrowser` | 8080:80 | File management | +| **Caddy** | `igny8_caddy` | 80:80, 443:443 | Reverse proxy & HTTPS | +| **Setup Helper** | `setup-helper` | - | Utility container | + +### Application Stack Services + +| Service | Container Name | Port | Purpose | +|---------|----------------|------|---------| +| **Backend** | `igny8_backend` | 8011:8010 | Django REST API | +| **Frontend** | `igny8_frontend` | 8021:5173 | React application | +| **Celery Worker** | `igny8_celery_worker` | - | Async task processing | +| **Celery Beat** | `igny8_celery_beat` | - | Scheduled tasks | + +### Network Configuration + +- **Network Name**: `igny8_net` +- **Type**: External bridge network +- **Purpose**: Inter-container communication +- **Creation**: Must be created manually before starting stacks + +### Volume Management + +**Infrastructure Volumes**: +- `pgdata`: PostgreSQL database data +- `redisdata`: Redis data +- `pgadmin_data`: pgAdmin configuration +- `filebrowser_db`: FileBrowser database +- `caddy_data`: Caddy SSL certificates +- `caddy_config`: Caddy configuration + +**Application Volumes**: +- Host mounts for application code +- Host mounts for logs +- Docker socket for container management + +### Port Allocation + +| Service | External Port | Internal Port | Access | +|---------|---------------|---------------|--------| +| **pgAdmin** | 5050 | 80 | http://localhost:5050 | +| **FileBrowser** | 8080 | 80 | http://localhost:8080 | +| **Caddy** | 80, 443 | 80, 443 | https://domain.com | +| **Backend** | 8011 | 8010 | http://localhost:8011 | +| **Frontend** | 8021 | 5173 | http://localhost:8021 | + +--- + +## External Service Integrations + +### OpenAI Integration + +**Purpose**: Text generation and image generation + +**Services Used**: +- GPT models for text generation +- DALL-E for image generation + +**Configuration**: +- API key stored per account in `IntegrationSettings` +- Model selection per account +- Cost tracking per request + +### Runware Integration + +**Purpose**: Alternative image generation service + +**Configuration**: +- API key stored per account +- Model selection (e.g., `runware:97@1`) +- Image type selection (realistic, artistic, cartoon) + +### WordPress Integration + +**Purpose**: Content publishing + +**Configuration**: +- WordPress URL per site +- Username and password stored per site +- REST API integration for publishing + +**Workflow**: +1. Content generated in IGNY8 +2. Images attached +3. Content published to WordPress via REST API +4. Status updated in IGNY8 + +### Stripe Integration (Planned) + +**Purpose**: Payment processing + +**Status**: Planned for future implementation + +**Features**: +- Subscription management +- Payment processing +- Webhook integration + +--- + +## Deployment Architecture + +### Deployment Model + +**Container-Based**: All services run in Docker containers + +**Stack Separation**: +- Infrastructure stack runs independently +- Application stack depends on infrastructure stack +- Both stacks share the same Docker network + +### Environment Configuration + +**Backend Environment Variables**: +- Database connection (PostgreSQL) +- Redis connection +- Django settings (DEBUG, SECRET_KEY, etc.) +- JWT settings +- Celery configuration + +**Frontend Environment Variables**: +- Backend API URL +- Environment (development/production) + +**Infrastructure Environment Variables**: +- PostgreSQL credentials +- pgAdmin credentials +- FileBrowser configuration + +### Health Checks + +**Backend Health Check**: +- Endpoint: `/api/v1/system/status/` +- Interval: 30 seconds +- Timeout: 10 seconds +- Retries: 3 + +**PostgreSQL Health Check**: +- Command: `pg_isready` +- Interval: 20 seconds +- Timeout: 3 seconds +- Retries: 5 + +**Redis Health Check**: +- Command: `redis-cli ping` +- Interval: 20 seconds +- Timeout: 3 seconds +- Retries: 5 + +### Scaling Considerations + +**Horizontal Scaling**: +- Multiple Celery workers can be added +- Multiple backend instances can be added (with load balancer) +- Frontend can be scaled independently + +**Vertical Scaling**: +- Database can be scaled with more resources +- Redis can be scaled with more memory +- Containers can be allocated more CPU/memory + +### Backup & Recovery + +**Database Backups**: +- PostgreSQL dumps stored in `/data/backups` +- Automated backup scripts +- Point-in-time recovery support + +**Volume Backups**: +- Docker volume backups +- Application code backups +- Configuration backups + +--- + +## Summary + +The IGNY8 platform is built on a modern, scalable architecture using: + +- **Django 5.2+** for the backend API +- **React 19** for the frontend +- **PostgreSQL 15** for data storage +- **Celery & Redis** for async processing +- **Docker** for containerization +- **Caddy** for reverse proxy and HTTPS + +The architecture follows principles of: +- Configuration-driven development +- Multi-tenancy with account isolation +- Module-based organization +- Unified AI processing +- Template-based UI rendering + +This architecture supports scalability, maintainability, and rapid feature development while ensuring data security and isolation. + diff --git a/docs/02-APP-ARCHITECTURE.md b/docs/02-APP-ARCHITECTURE.md new file mode 100644 index 00000000..fca64dca --- /dev/null +++ b/docs/02-APP-ARCHITECTURE.md @@ -0,0 +1,795 @@ +# IGNY8 Application Architecture + +**Last Updated:** 2025-01-XX +**Purpose:** Complete application architecture documentation covering system hierarchy, modules, workflows, features, and data flow. + +--- + +## Table of Contents + +1. [System Overview](#system-overview) +2. [System Hierarchy](#system-hierarchy) +3. [User Roles & Access Control](#user-roles--access-control) +4. [Module Organization](#module-organization) +5. [Complete Workflows](#complete-workflows) +6. [Data Models & Relationships](#data-models--relationships) +7. [Multi-Tenancy Architecture](#multi-tenancy-architecture) +8. [API Architecture](#api-architecture) +9. [Security Architecture](#security-architecture) +10. [Integration Points](#integration-points) + +--- + +## System Overview + +**IGNY8** is a full-stack SaaS platform for SEO keyword management and AI-driven content generation. The system provides a scalable, multi-account platform that enables users to manage keywords, generate content ideas, create AI-powered content, and publish to WordPress. + +### Platform Capabilities + +| Capability | Description | Module | +|------------|-------------|--------| +| **Keyword Management** | Import, organize, filter, and manage SEO keywords | Planner | +| **Keyword Clustering** | AI-powered semantic clustering of related keywords | Planner | +| **Content Ideas** | Generate content ideas from keyword clusters | Planner | +| **Content Generation** | AI-powered blog post and article generation | Writer | +| **Image Generation** | AI-powered image generation (DALL-E, Runware) | Writer | +| **WordPress Integration** | Direct publishing to WordPress sites | Writer | +| **Account Management** | Multi-account SaaS with user roles | Auth | +| **Billing & Credits** | Credit-based billing system | Billing | +| **AI Configuration** | Customizable AI prompts and settings | System | + +--- + +## System Hierarchy + +### Entity Relationships + +``` +Account (1) ──< (N) User +Account (1) ──< (1) Subscription ──> (1) Plan +Account (1) ──< (N) Site +Site (1) ──< (1-5) Sector +Sector (1) ──< (N) Keywords, Clusters, ContentIdeas, Tasks +Cluster (1) ──< (N) Keywords (Many-to-Many) +Cluster (1) ──< (N) ContentIdeas +ContentIdeas (1) ──< (N) Tasks +Task (1) ──> (1) Content +Task (1) ──< (N) Images +``` + +### Hierarchy Details + +**Account Level**: +- Top-level organization/workspace +- Contains users, sites, subscriptions, and all data +- Has credit balance and plan assignment +- Status: active, suspended, trial, cancelled + +**User Level**: +- Individual user accounts within an account +- Has role (developer, owner, admin, editor, viewer) +- Can belong to only one account +- Access controlled by role and site permissions + +**Site Level**: +- Workspace within an account (1-N relationship) +- Can have multiple active sites simultaneously +- Has WordPress integration settings (URL, username, password) +- Can be associated with an industry +- Status: active, inactive, suspended + +**Sector Level**: +- Content category within a site (1-5 per site) +- Organizes keywords, clusters, ideas, and tasks +- Can reference an industry sector template +- Status: active, inactive + +**Content Level**: +- Keywords, Clusters, ContentIdeas belong to Sector +- Tasks, Content, Images belong to Sector +- All content is automatically associated with Account and Site + +--- + +## User Roles & Access Control + +### User Roles + +| Role | Access Level | Description | Permissions | +|------|--------------|-------------|-------------| +| **Developer** | System-wide | Full system access, bypasses all restrictions | All accounts, all sites, all data, system settings | +| **Owner** | Account-wide | Full account access, can manage users and billing | All sites in account, user management, billing | +| **Admin** | Account-wide | Account admin access, can manage content and users | All sites in account, content management, user management | +| **Editor** | Site-specific | Content editing access, can manage clusters/tasks | Granted sites only, content editing | +| **Viewer** | Site-specific | Read-only access | Granted sites only, read-only | +| **System Bot** | System-wide | System automation user | All accounts, all sites, all data | + +### Access Control Matrix + +| User Role | Account Access | Site Access | Data Access | User Management | Billing | +|-----------|----------------|-------------|-------------|------------------|---------| +| Developer | All accounts | All sites | All data | Yes | Yes | +| System Bot | All accounts | All sites | All data | No | No | +| Owner | Own account | All sites in account | All data in account | Yes | Yes | +| Admin | Own account | All sites in account | All data in account | Yes | No | +| Editor | Own account | Granted sites only | Data in granted sites | No | No | +| Viewer | Own account | Granted sites only | Read-only in granted sites | No | No | + +### Site Access Control + +**Automatic Access**: +- Owners and Admins: Automatic access to all sites in their account +- Developers and System Bot: Access to all sites across all accounts + +**Explicit Access**: +- Editors and Viewers: Require explicit `SiteUserAccess` records +- Access granted by Owner or Admin +- Access can be revoked at any time + +### Account Isolation + +**Principle**: All data is automatically filtered by account. + +**Implementation**: +- All models inherit `AccountBaseModel` with `account` ForeignKey +- All ViewSets inherit `AccountModelViewSet` with automatic filtering +- Middleware sets `request.account` from JWT token +- Queries automatically filter by account (except for Developer/System Bot) + +--- + +## Module Organization + +### Module Structure + +| Module | Purpose | Models | ViewSets | Celery Tasks | +|--------|---------|--------|----------|--------------| +| **Planner** | Keyword management & content planning | Keywords, Clusters, ContentIdeas | KeywordViewSet, ClusterViewSet, ContentIdeasViewSet | auto_cluster_keywords_task, auto_generate_ideas_task | +| **Writer** | Content generation & management | Tasks, Content, Images | TasksViewSet, ImagesViewSet | auto_generate_content_task, auto_generate_images_task | +| **System** | Settings, prompts, integrations | AIPrompt, IntegrationSettings, AuthorProfile, Strategy | AIPromptViewSet, IntegrationSettingsViewSet, AuthorProfileViewSet | - | +| **Billing** | Credits, transactions, usage | CreditTransaction, CreditUsageLog | CreditTransactionViewSet, CreditUsageLogViewSet | - | +| **Auth** | Multi-tenancy, users, accounts | Account, User, Plan, Site, Sector | AccountViewSet, UserViewSet, SiteViewSet, SectorViewSet | - | + +### Module Dependencies + +``` +Auth (Core) + ├── Planner (depends on Auth) + ├── Writer (depends on Auth, Planner) + ├── System (depends on Auth) + └── Billing (depends on Auth) +``` + +### Module Features + +#### Planner Module + +**Purpose**: Keyword research, clustering, and content idea generation + +**Features**: +- Keyword import (CSV/manual) +- Keyword filtering and organization +- AI-powered keyword clustering +- Content idea generation from clusters +- Keyword-to-cluster mapping +- Cluster metrics and analytics + +**Models**: +- `Keywords`: Individual keywords with metrics (volume, difficulty, intent) +- `Clusters`: Groups of related keywords +- `ContentIdeas`: Content ideas generated from clusters + +#### Writer Module + +**Purpose**: Content generation, image generation, and publishing + +**Features**: +- Task creation from content ideas +- AI-powered content generation +- Content editing and review +- Image prompt extraction +- AI-powered image generation +- WordPress publishing + +**Models**: +- `Tasks`: Content generation tasks +- `Content`: Generated HTML content +- `Images`: Generated images (featured, in-article) + +#### System Module + +**Purpose**: System configuration and AI settings + +**Features**: +- AI prompt management +- Integration settings (OpenAI, Runware) +- Author profile management +- Content strategy management +- System status and monitoring + +**Models**: +- `AIPrompt`: AI prompt templates +- `IntegrationSettings`: API keys and configuration +- `AuthorProfile`: Writing style profiles +- `Strategy`: Content strategies + +#### Billing Module + +**Purpose**: Credit management and usage tracking + +**Features**: +- Credit balance tracking +- Credit transactions +- Usage logging +- Cost tracking + +**Models**: +- `CreditTransaction`: Credit additions and deductions +- `CreditUsageLog`: Usage tracking with cost + +#### Auth Module + +**Purpose**: Multi-tenancy and user management + +**Features**: +- Account management +- User management +- Plan management +- Site and sector management +- Industry templates + +**Models**: +- `Account`: Top-level organization +- `User`: User accounts +- `Plan`: Subscription plans +- `Site`: Sites within accounts +- `Sector`: Sectors within sites +- `Industry`: Global industry templates +- `IndustrySector`: Industry sector templates + +--- + +## Complete Workflows + +### 1. Account Setup Workflow + +**Purpose**: Onboard a new account and user + +**Steps**: +1. User signs up via `/signup` +2. Account created with default plan +3. Owner user created and linked to account +4. User signs in via `/signin` +5. JWT token generated and returned +6. Frontend stores token and redirects to dashboard +7. User creates first site (optional) +8. User creates sectors (1-5 per site, optional) +9. User configures integration settings (OpenAI, Runware) +10. System ready for use + +**Data Created**: +- 1 Account record +- 1 User record (owner role) +- 1 Subscription record (default plan) +- 0-N Site records +- 0-N Sector records (per site) +- 1 IntegrationSettings record (per integration type) + +### 2. Keyword Management Workflow + +**Purpose**: Import, organize, and cluster keywords + +**Steps**: +1. User navigates to `/planner/keywords` +2. User imports keywords via CSV or manual entry +3. Keywords validated and stored in database +4. Keywords displayed in table with filters +5. User filters keywords by sector, status, intent, etc. +6. User selects keywords for clustering +7. User clicks "Auto Cluster" action +8. Backend validates keyword IDs +9. Celery task queued (`auto_cluster_keywords_task`) +10. Task ID returned to frontend +11. Frontend polls progress endpoint +12. Celery worker processes task: + - Loads keywords from database + - Builds AI prompt with keyword data + - Calls OpenAI API for clustering + - Parses cluster response + - Creates Cluster records + - Links keywords to clusters +13. Progress updates sent to frontend +14. Task completes +15. Frontend displays new clusters +16. User can view clusters, edit cluster names, add/remove keywords + +**Data Created**: +- N Keyword records (if imported) +- M Cluster records (created by AI) +- N-M Keyword-to-Cluster relationships + +**AI Function**: Auto Cluster Keywords + +### 3. Content Idea Generation Workflow + +**Purpose**: Generate content ideas from keyword clusters + +**Steps**: +1. User navigates to `/planner/clusters` +2. User selects one or more clusters +3. User clicks "Generate Ideas" action +4. Backend validates cluster IDs +5. Celery task queued (`auto_generate_ideas_task`) +6. Task ID returned to frontend +7. Frontend polls progress endpoint +8. Celery worker processes task: + - Loads clusters and keywords + - Builds AI prompt with cluster data + - Calls OpenAI API for idea generation + - Parses idea response + - Creates ContentIdeas records + - Links ideas to clusters +9. Progress updates sent to frontend +10. Task completes +11. Frontend displays new content ideas +12. User can view ideas, edit titles, prioritize ideas + +**Data Created**: +- N ContentIdeas records (created by AI) +- N ContentIdeas-to-Cluster relationships + +**AI Function**: Generate Ideas + +### 4. Content Generation Workflow + +**Purpose**: Generate AI-powered content from content ideas + +**Steps**: +1. User navigates to `/planner/ideas` +2. User selects one or more content ideas +3. User clicks "Create Tasks" action +4. Task records created for each idea +5. User navigates to `/writer/tasks` +6. User selects tasks for content generation +7. User clicks "Generate Content" action +8. Backend validates task IDs +9. Celery task queued (`auto_generate_content_task`) +10. Task ID returned to frontend +11. Frontend polls progress endpoint +12. Celery worker processes task: + - Loads tasks and related data (cluster, keywords, idea) + - Builds AI prompt with task data + - Calls OpenAI API for content generation + - Parses HTML content response + - Creates/updates Content records + - Updates task status +13. Progress updates sent to frontend +14. Task completes +15. Frontend displays generated content +16. User can review and edit content +17. User navigates to `/writer/content` to view content + +**Data Created**: +- N Task records (from ideas) +- N Content records (generated HTML) + +**AI Function**: Generate Content + +### 5. Image Generation Workflow + +**Purpose**: Generate images for content + +**Steps**: +1. User navigates to `/writer/content` +2. User selects content items +3. User clicks "Generate Image Prompts" action (optional, can skip) +4. Backend validates content IDs +5. Celery task queued (`auto_generate_image_prompts_task`) +6. Task processes: + - Loads content HTML + - Builds AI prompt for prompt extraction + - Calls OpenAI API + - Parses image prompts (featured, in-article) + - Creates/updates Images records with prompts +7. User clicks "Generate Images" action +8. Backend validates image IDs +9. Celery task queued (`auto_generate_images_task`) +10. Task processes: + - Loads images with prompts + - Builds image generation prompt + - Calls OpenAI DALL-E or Runware API + - Receives image URLs + - Updates Images records with URLs +11. Images displayed in frontend +12. User can view images, regenerate, or delete + +**Data Created**: +- N Images records (with prompts) +- N Images records (with image URLs) + +**AI Functions**: Generate Image Prompts, Generate Images + +### 6. WordPress Publishing Workflow + +**Purpose**: Publish content to WordPress + +**Steps**: +1. User navigates to `/writer/content` +2. User selects content to publish +3. User clicks "Publish to WordPress" action +4. Backend validates: + - Site has WordPress URL configured + - Site has WordPress credentials + - Content is ready (status: review or draft) +5. Backend calls WordPress REST API: + - Creates post with content HTML + - Uploads featured image (if available) + - Uploads in-article images (if available) + - Sets post status (draft, publish) +6. WordPress post ID stored in Content record +7. Content status updated to "published" +8. Frontend displays success message +9. User can view published content in WordPress + +**Data Updated**: +- Content record: `wp_post_id`, `status` = "published" + +**Integration**: WordPress REST API + +### 7. User Management Workflow + +**Purpose**: Add users to account and manage permissions + +**Steps**: +1. Owner/Admin navigates to `/settings/users` +2. User clicks "Add User" action +3. User enters email and selects role +4. Backend creates User record: + - Email validated (unique per account) + - Role assigned + - Account linked + - Password set (or invitation sent) +5. If role is Editor/Viewer: + - Owner/Admin grants site access + - SiteUserAccess records created +6. New user receives invitation email +7. New user signs in and accesses granted sites + +**Data Created**: +- 1 User record +- 0-N SiteUserAccess records (for Editor/Viewer roles) + +### 8. Integration Configuration Workflow + +**Purpose**: Configure AI service integrations + +**Steps**: +1. User navigates to `/settings/integration` +2. User selects integration type (OpenAI, Runware) +3. User enters API key +4. User clicks "Test Connection" (optional) +5. Backend validates API key: + - Makes test API call + - Returns connection status +6. User saves integration settings +7. Backend stores API key in IntegrationSettings +8. Integration ready for use in AI functions + +**Data Created/Updated**: +- 1 IntegrationSettings record (per integration type) + +**Test Functions**: Test OpenAI, Test Runware + +--- + +## Data Models & Relationships + +### Core Models + +**Account Model**: +- Fields: name, slug, owner, plan, credits, status +- Relationships: Users (1-N), Sites (1-N), Subscription (1-1) + +**User Model**: +- Fields: email, account, role +- Relationships: Account (N-1), SiteUserAccess (N-M via SiteUserAccess) + +**Plan Model**: +- Fields: name, price, limits (users, sites, keywords, clusters, etc.), credits +- Relationships: Accounts (1-N via Subscription) + +**Site Model**: +- Fields: name, slug, domain, industry, status, wp_url, wp_username, wp_app_password +- Relationships: Account (N-1), Sectors (1-N), Industry (N-1) + +**Sector Model**: +- Fields: name, slug, site, industry_sector, status +- Relationships: Site (N-1), IndustrySector (N-1), Keywords/Clusters/Ideas/Tasks (1-N) + +### Planner Models + +**Keywords Model**: +- Fields: keyword, volume, difficulty, intent, cluster (M-N) +- Relationships: Sector (N-1), Clusters (M-N) + +**Clusters Model**: +- Fields: name, description, keywords_count, volume +- Relationships: Sector (N-1), Keywords (M-N), ContentIdeas (1-N) + +**ContentIdeas Model**: +- Fields: idea_title, description, cluster, status +- Relationships: Sector (N-1), Cluster (N-1), Tasks (1-N) + +### Writer Models + +**Tasks Model**: +- Fields: title, description, cluster, idea, status, content +- Relationships: Sector (N-1), Cluster (N-1), ContentIdeas (N-1), Content (1-1), Images (1-N) + +**Content Model**: +- Fields: task, html_content, word_count, status, wp_post_id +- Relationships: Task (1-1), Images (1-N) + +**Images Model**: +- Fields: task, content, image_type, prompt, image_url, status +- Relationships: Task (N-1), Content (N-1) + +### System Models + +**AIPrompt Model**: +- Fields: prompt_type, prompt_value, account +- Relationships: Account (N-1) + +**IntegrationSettings Model**: +- Fields: integration_type, config (JSON), account +- Relationships: Account (N-1) + +**AuthorProfile Model**: +- Fields: name, description, tone, language, account +- Relationships: Account (N-1) + +**Strategy Model**: +- Fields: name, description, sector, prompt_types, account +- Relationships: Account (N-1), Sector (N-1) + +### Billing Models + +**CreditTransaction Model**: +- Fields: account, transaction_type, amount, balance_after +- Relationships: Account (N-1) + +**CreditUsageLog Model**: +- Fields: account, operation_type, credits_used, cost_usd +- Relationships: Account (N-1) + +--- + +## Multi-Tenancy Architecture + +### Account Isolation + +**Implementation Levels**: +1. **Model Level**: All models inherit `AccountBaseModel` with `account` ForeignKey +2. **ViewSet Level**: All ViewSets inherit `AccountModelViewSet` with automatic filtering +3. **Middleware Level**: `AccountContextMiddleware` sets `request.account` from JWT + +**Filtering**: +- All queries automatically filter by `account = request.account` +- Developer and System Bot users bypass account filtering +- System accounts bypass account filtering + +### Site/Sector Hierarchy + +**Purpose**: Organize content within accounts + +**Structure**: +``` +Account + └── Site 1 (Active) + ├── Sector 1 (Active) + │ ├── Keywords + │ ├── Clusters + │ ├── ContentIdeas + │ └── Tasks + ├── Sector 2 (Active) + └── Sector 3 (Inactive) + └── Site 2 (Active) + └── Sector 1 (Active) +``` + +**Constraints**: +- Maximum 5 active sectors per site +- Multiple sites can be active simultaneously +- All content must belong to a site and sector + +### Data Isolation Flow + +``` +Request with JWT Token + ↓ +AccountContextMiddleware + ├── Extract Account ID from JWT + ├── Load Account Object + └── Set request.account + ↓ +ViewSet.get_queryset() + ├── Check User Role + ├── Filter by Account (if not admin/developer) + └── Filter by Accessible Sites (if not owner/admin) + ↓ +Database Query + ↓ +Results (Account-Isolated) +``` + +--- + +## API Architecture + +### API Structure + +**Base URL**: `/api/v1/` + +**Module Endpoints**: +- `/api/v1/auth/` - Accounts, users, sites, sectors, plans +- `/api/v1/planner/` - Keywords, clusters, ideas +- `/api/v1/writer/` - Tasks, content, images +- `/api/v1/system/` - Prompts, integrations, author-profiles, strategies +- `/api/v1/billing/` - Credits, transactions, usage + +### Authentication Flow + +``` +1. User Signs In + ↓ +2. Backend Validates Credentials + ↓ +3. JWT Token Generated + ↓ +4. Token Returned to Frontend + ↓ +5. Frontend Stores Token (localStorage) + ↓ +6. Frontend Includes Token in Requests (Authorization: Bearer {token}) + ↓ +7. Backend Validates Token + ↓ +8. Account Context Set + ↓ +9. Request Processed +``` + +### Response Format + +**Success Response**: +- `success`: true +- `data`: Response data +- `message`: Optional message + +**Error Response**: +- `success`: false +- `message`: Error message +- `errors`: Validation errors (if applicable) + +**Pagination Response**: +- `count`: Total count +- `next`: Next page URL +- `previous`: Previous page URL +- `results`: Array of results + +--- + +## Security Architecture + +### Authentication + +**Methods**: +- JWT (JSON Web Tokens) - Primary method +- Session-based auth - Fallback for admin + +**Token Management**: +- Tokens stored in localStorage +- Tokens included in `Authorization: Bearer {token}` header +- Token refresh mechanism (future implementation) + +### Authorization + +**Role-Based Access Control (RBAC)**: +- Role checked on every request +- Permissions enforced at ViewSet level +- Action-level permissions for sensitive operations + +**Data Access Control**: +- Account-level: Automatic filtering by account +- Site-level: Filtering by accessible sites +- Action-level: Permission checks in ViewSet actions + +### Data Security + +**Account Isolation**: +- All queries filtered by account +- Admin/Developer override for system accounts +- No cross-account data leakage + +**Site Access Control**: +- Users can only access granted sites +- Admin/Developer override for all sites +- Explicit access grants for Editor/Viewer roles + +**API Security**: +- CORS configured for frontend domain +- CSRF enabled for session-based auth +- Input validation via DRF serializers +- Rate limiting (future implementation) + +--- + +## Integration Points + +### OpenAI Integration + +**Purpose**: Text generation and image generation + +**Configuration**: +- API key stored per account in `IntegrationSettings` +- Model selection per account +- Cost tracking per request + +**Services Used**: +- GPT models for text generation +- DALL-E for image generation + +### Runware Integration + +**Purpose**: Alternative image generation service + +**Configuration**: +- API key stored per account +- Model selection (e.g., `runware:97@1`) +- Image type selection (realistic, artistic, cartoon) + +### WordPress Integration + +**Purpose**: Content publishing + +**Configuration**: +- WordPress URL per site +- Username and password stored per site +- REST API integration for publishing + +**Workflow**: +1. Content generated in IGNY8 +2. Images attached +3. Content published to WordPress via REST API +4. Status updated in IGNY8 + +### Stripe Integration (Planned) + +**Purpose**: Payment processing + +**Status**: Planned for future implementation + +**Features**: +- Subscription management +- Payment processing +- Webhook integration + +--- + +## Summary + +The IGNY8 application architecture provides: + +1. **Multi-Tenancy**: Complete account isolation with automatic filtering +2. **Hierarchical Organization**: Account > Site > Sector > Content structure +3. **Role-Based Access**: Granular permissions for different user roles +4. **Module-Based Design**: Clear separation of concerns across modules +5. **Complete Workflows**: End-to-end workflows from keyword import to publishing +6. **AI Integration**: Unified AI framework for all AI operations +7. **WordPress Integration**: Direct publishing to WordPress sites +8. **Credit System**: Credit-based billing and usage tracking +9. **Security First**: JWT auth, RBAC, data isolation +10. **Scalable Design**: Supports multiple accounts, sites, and users + +This architecture ensures scalability, maintainability, and extensibility while providing a robust foundation for the IGNY8 platform. + diff --git a/docs/02-FRONTEND.md b/docs/03-FRONTEND.md similarity index 64% rename from docs/02-FRONTEND.md rename to docs/03-FRONTEND.md index dd8c1937..2f6684d1 100644 --- a/docs/02-FRONTEND.md +++ b/docs/03-FRONTEND.md @@ -1,8 +1,7 @@ # IGNY8 Frontend Documentation -**Version:** 1.0 **Last Updated:** 2025-01-XX -**Purpose:** Complete frontend documentation covering pages, components, templates, routing, state management, and configuration system. +**Purpose:** Complete frontend documentation covering architecture, pages, components, routing, state management, and configuration system. --- @@ -17,9 +16,8 @@ 7. [State Management](#state-management) 8. [API Integration](#api-integration) 9. [Configuration System](#configuration-system) -10. [Pages](#pages) -11. [Hooks](#hooks) -12. [Utilities](#utilities) +10. [Pages & Features](#pages--features) +11. [Hooks & Utilities](#hooks--utilities) --- @@ -76,24 +74,58 @@ frontend/src/ │ │ ├── Dashboard.tsx │ │ ├── Keywords.tsx │ │ ├── Clusters.tsx -│ │ └── Ideas.tsx +│ │ ├── Ideas.tsx +│ │ ├── KeywordOpportunities.tsx +│ │ └── Mapping.tsx │ ├── Writer/ # Writer module pages │ │ ├── Dashboard.tsx │ │ ├── Tasks.tsx │ │ ├── Content.tsx +│ │ ├── ContentView.tsx │ │ ├── Drafts.tsx │ │ ├── Images.tsx │ │ └── Published.tsx +│ ├── Thinker/ # Thinker module pages +│ │ ├── Dashboard.tsx +│ │ ├── Prompts.tsx +│ │ ├── AuthorProfiles.tsx +│ │ ├── Strategies.tsx +│ │ ├── Profile.tsx +│ │ └── ImageTesting.tsx +│ ├── Billing/ # Billing module pages +│ │ ├── Credits.tsx +│ │ ├── Transactions.tsx +│ │ └── Usage.tsx │ ├── Settings/ # Settings pages │ │ ├── General.tsx │ │ ├── Users.tsx │ │ ├── Sites.tsx │ │ ├── Integration.tsx -│ │ └── ... +│ │ ├── AI.tsx +│ │ ├── Plans.tsx +│ │ ├── Industries.tsx +│ │ ├── Status.tsx +│ │ ├── Subscriptions.tsx +│ │ ├── Account.tsx +│ │ ├── Modules.tsx +│ │ ├── System.tsx +│ │ ├── ImportExport.tsx +│ │ └── UiElements/ # UI element examples +│ ├── Help/ # Help pages +│ │ ├── Help.tsx +│ │ ├── Docs.tsx +│ │ ├── SystemTesting.tsx +│ │ └── FunctionTesting.tsx +│ ├── Reference/ # Reference data pages +│ │ ├── Industries.tsx +│ │ └── SeedKeywords.tsx │ ├── AuthPages/ # Authentication pages │ │ ├── SignIn.tsx │ │ └── SignUp.tsx -│ └── ... +│ ├── Dashboard/ # Main dashboard +│ │ └── Home.tsx +│ └── OtherPage/ # Other pages +│ └── NotFound.tsx ├── templates/ # 4 universal templates │ ├── DashboardTemplate.tsx │ ├── TablePageTemplate.tsx @@ -135,8 +167,6 @@ frontend/src/ ├── hooks/ # Custom React hooks │ ├── useProgressModal.ts # Progress modal for long-running tasks │ └── useAuth.ts # Authentication hook -├── layout/ # Layout components -│ └── AppLayout.tsx # Main app layout wrapper ├── utils/ # Utility functions │ └── difficulty.ts # Difficulty utilities ├── App.tsx # Root component with routing @@ -149,54 +179,38 @@ frontend/src/ ### Route Structure -**File**: `App.tsx` - **Public Routes**: - `/signin` - Sign in page - `/signup` - Sign up page **Protected Routes** (require authentication): -- `/` - Home/Dashboard -- `/planner` - Planner Dashboard -- `/planner/keywords` - Keywords page -- `/planner/clusters` - Clusters page -- `/planner/ideas` - Ideas page -- `/writer` - Writer Dashboard -- `/writer/tasks` - Tasks page -- `/writer/content` - Drafts page -- `/writer/images` - Images page -- `/writer/published` - Published page -- `/thinker` - Thinker Dashboard -- `/thinker/prompts` - Prompts page -- `/thinker/author-profiles` - Author Profiles page -- `/thinker/strategies` - Strategies page -- `/thinker/image-testing` - Image Testing page -- `/billing/credits` - Credits page -- `/billing/transactions` - Transactions page -- `/billing/usage` - Usage page -- `/settings` - Settings pages -- `/analytics` - Analytics page -- `/schedules` - Schedules page +- `/` - Home dashboard +- `/planner/*` - Planner module routes +- `/writer/*` - Writer module routes +- `/thinker/*` - Thinker module routes +- `/billing/*` - Billing module routes +- `/settings/*` - Settings routes +- `/help/*` - Help routes +- `/reference/*` - Reference data routes +- `/ui-elements/*` - UI element examples + +### ProtectedRoute Component + +**Purpose**: Wraps protected routes and checks authentication + +**Functionality**: +- Checks if user is authenticated +- Redirects to `/signin` if not authenticated +- Wraps children with `AppLayout` if authenticated ### Route Configuration **File**: `config/routes.config.ts` -Defines route metadata including: -- Route paths -- Labels for navigation -- Icons -- Breadcrumb labels -- Nested route structure +**Structure**: Defines route hierarchy, labels, and icons for navigation -### Protected Routes - -**Component**: `components/auth/ProtectedRoute.tsx` - -**Functionality**: -- Checks authentication status from `authStore` -- Redirects to `/signin` if not authenticated -- Wraps protected routes with `AppLayout` +**Functions**: +- `getBreadcrumbs(pathname)`: Generates breadcrumbs for current route --- @@ -204,18 +218,7 @@ Defines route metadata including: ### 1. DashboardTemplate -**Purpose**: Module home pages with KPIs, workflow steps, and charts. - -**Usage**: -```tsx - -``` +**Purpose**: Module home pages with KPIs, workflow steps, and charts **Features**: - Header metrics (KPIs) @@ -223,24 +226,11 @@ Defines route metadata including: - Charts and visualizations - Quick actions +**Usage**: Planner Dashboard, Writer Dashboard, Thinker Dashboard + ### 2. TablePageTemplate -**Purpose**: CRUD table pages (Keywords, Clusters, Tasks, etc.). - -**Usage**: -```tsx - ...} - onEdit={(row) => ...} - onDelete={(row) => ...} - bulkActions={bulkActions} -/> -``` +**Purpose**: CRUD table pages (Keywords, Clusters, Tasks, etc.) **Features**: - Data table with sorting @@ -257,19 +247,11 @@ Defines route metadata including: - `bulkActions`: Bulk action definitions - `actions`: Row action definitions +**Usage**: Keywords, Clusters, Ideas, Tasks, Content, Images, Prompts, etc. + ### 3. FormPageTemplate -**Purpose**: Settings/form pages (Settings, Integration, etc.). - -**Usage**: -```tsx - ...} -/> -``` +**Purpose**: Settings/form pages (Settings, Integration, etc.) **Features**: - Form sections @@ -277,18 +259,11 @@ Defines route metadata including: - Save/Cancel buttons - Success/Error notifications +**Usage**: Settings, Integration, AI Settings, Plans, etc. + ### 4. SystemPageTemplate -**Purpose**: System/admin pages (Logs, Status, Monitoring). - -**Usage**: -```tsx - -``` +**Purpose**: System/admin pages (Logs, Status, Monitoring) **Features**: - System information display @@ -296,6 +271,8 @@ Defines route metadata including: - Status indicators - Performance metrics +**Usage**: System Status, System Testing, etc. + --- ## Component Library @@ -303,9 +280,7 @@ Defines route metadata including: ### Layout Components #### AppLayout -**File**: `layout/AppLayout.tsx` - -**Purpose**: Main app layout wrapper. +**Purpose**: Main app layout wrapper **Features**: - Sidebar navigation @@ -314,9 +289,7 @@ Defines route metadata including: - Breadcrumbs #### Sidebar -**File**: `components/layout/Sidebar.tsx` - -**Purpose**: Navigation sidebar. +**Purpose**: Navigation sidebar **Features**: - Module navigation @@ -324,9 +297,7 @@ Defines route metadata including: - Collapsible sections #### Header -**File**: `components/layout/Header.tsx` - -**Purpose**: Top header bar. +**Purpose**: Top header bar **Features**: - User menu @@ -336,9 +307,7 @@ Defines route metadata including: ### Table Components #### DataTable -**File**: `components/table/DataTable.tsx` - -**Purpose**: Data table component. +**Purpose**: Data table component **Features**: - Sortable columns @@ -347,9 +316,7 @@ Defines route metadata including: - Responsive design #### Filters -**File**: `components/table/Filters.tsx` - -**Purpose**: Filter component. +**Purpose**: Filter component **Features**: - Text filters @@ -358,9 +325,7 @@ Defines route metadata including: - Custom filters #### Pagination -**File**: `components/ui/pagination/CompactPagination.tsx` - -**Purpose**: Pagination component. +**Purpose**: Pagination component **Features**: - Page navigation @@ -370,52 +335,33 @@ Defines route metadata including: ### UI Components #### Button -**File**: `components/ui/button/Button.tsx` - **Variants**: primary, secondary, danger, ghost, link - **Sizes**: sm, md, lg #### Card -**File**: `components/ui/card/Card.tsx` - -**Purpose**: Card container component. +**Purpose**: Card container component #### Modal -**File**: `components/ui/modal/Modal.tsx` - -**Purpose**: Modal dialog component. - +**Purpose**: Modal dialog component **Variants**: FormModal, ProgressModal, AlertModal #### Toast -**File**: `components/ui/toast/ToastContainer.tsx` - -**Purpose**: Toast notification system. - +**Purpose**: Toast notification system **Types**: success, error, warning, info #### Input -**File**: `components/form/input/InputField.tsx` - -**Purpose**: Text input component. +**Purpose**: Text input component #### Select -**File**: `components/form/SelectDropdown.tsx` - -**Purpose**: Select dropdown component. +**Purpose**: Select dropdown component #### Checkbox -**File**: `components/form/input/Checkbox.tsx` - -**Purpose**: Checkbox component. +**Purpose**: Checkbox component ### Auth Components #### ProtectedRoute -**File**: `components/auth/ProtectedRoute.tsx` - -**Purpose**: Route protection component. +**Purpose**: Route protection component **Functionality**: - Checks authentication @@ -429,8 +375,6 @@ Defines route metadata including: ### Zustand Stores #### authStore -**File**: `store/authStore.ts` - **State**: - `user`: Current user object - `token`: JWT access token @@ -448,16 +392,7 @@ Defines route metadata including: **Persistence**: localStorage (persisted) -#### plannerStore -**File**: `store/plannerStore.ts` - -**State**: Planner module-specific state - -**Actions**: Planner module actions - #### siteStore -**File**: `store/siteStore.ts` - **State**: - `activeSite`: Currently selected site - `sites`: List of accessible sites @@ -469,8 +404,6 @@ Defines route metadata including: **Persistence**: localStorage (persisted) #### sectorStore -**File**: `store/sectorStore.ts` - **State**: - `activeSector`: Currently selected sector - `sectors`: List of sectors for active site @@ -481,9 +414,12 @@ Defines route metadata including: **Persistence**: localStorage (persisted) -#### aiRequestLogsStore -**File**: `store/aiRequestLogsStore.ts` +#### plannerStore +**State**: Planner module-specific state +**Actions**: Planner module actions + +#### aiRequestLogsStore **State**: - `logs`: Array of AI request/response logs @@ -497,8 +433,6 @@ Defines route metadata including: **Purpose**: Tracks AI function execution with step-by-step logs #### pageSizeStore -**File**: `store/pageSizeStore.ts` - **State**: - `pageSize`: Table page size preference @@ -556,29 +490,16 @@ Defines route metadata including: **Example**: `keywords.config.tsx` -**Structure**: -```tsx -export const createKeywordsPageConfig = () => ({ - columns: [...], - filters: [...], - bulkActions: [...], - actions: [...], -}); -``` +**Structure**: Defines columns, filters, bulkActions, actions for a page -**Usage**: -```tsx -import { createKeywordsPageConfig } from '../../config/pages/keywords.config'; - -const config = createKeywordsPageConfig(); -``` +**Usage**: Imported in page components to configure TablePageTemplate ### Shared Snippets **Location**: `config/snippets/` #### columns.snippets.ts -**Purpose**: Reusable column definitions. +**Purpose**: Reusable column definitions **Examples**: - `statusColumn`: Status column with badge @@ -586,7 +507,7 @@ const config = createKeywordsPageConfig(); - `dateColumn`: Date column with formatting #### filters.snippets.ts -**Purpose**: Reusable filter definitions. +**Purpose**: Reusable filter definitions **Examples**: - `statusFilter`: Status dropdown filter @@ -594,7 +515,7 @@ const config = createKeywordsPageConfig(); - `searchFilter`: Text search filter #### actions.snippets.ts -**Purpose**: Reusable action definitions. +**Purpose**: Reusable action definitions **Examples**: - `commonActions`: Edit, Delete actions @@ -604,34 +525,18 @@ const config = createKeywordsPageConfig(); **File**: `config/routes.config.ts` -**Structure**: -```tsx -export const routes: RouteConfig[] = [ - { - path: '/planner', - label: 'Planner', - icon: 'Planner', - children: [ - { path: '/planner/keywords', label: 'Keywords' }, - ... - ], - }, - ... -]; -``` +**Structure**: Defines route hierarchy, labels, and icons for navigation **Functions**: -- `getBreadcrumbs(pathname)`: Get breadcrumbs for current route +- `getBreadcrumbs(pathname)`: Generates breadcrumbs for current route --- -## Pages +## Pages & Features ### Planner Module -#### Keywords Page -**File**: `pages/Planner/Keywords.tsx` - +#### Keywords Page (`/planner/keywords`) **Features**: - Keyword CRUD operations - Auto-cluster functionality @@ -642,28 +547,28 @@ export const routes: RouteConfig[] = [ **Configuration**: Uses `keywords.config.tsx` -#### Clusters Page -**File**: `pages/Planner/Clusters.tsx` - +#### Clusters Page (`/planner/clusters`) **Features**: - Cluster CRUD operations - Auto-generate ideas functionality - Filters (status, sector) - Bulk actions -#### Ideas Page -**File**: `pages/Planner/Ideas.tsx` - +#### Ideas Page (`/planner/ideas`) **Features**: - Content ideas CRUD operations - Filters (status, cluster, content type) - Bulk actions +#### Planner Dashboard (`/planner`) +**Features**: +- KPIs (total keywords, clusters, ideas) +- Workflow steps +- Charts and visualizations + ### Writer Module -#### Tasks Page -**File**: `pages/Writer/Tasks.tsx` - +#### Tasks Page (`/writer/tasks`) **Features**: - Task CRUD operations - Auto-generate content functionality @@ -671,105 +576,145 @@ export const routes: RouteConfig[] = [ - Filters (status, cluster, content type) - Bulk actions +#### Content Page (`/writer/content`) +**Features**: +- Content list view +- Content detail view (`/writer/content/:id`) +- Content editing +- Generate image prompts +- Generate images +- WordPress publishing + +#### Images Page (`/writer/images`) +**Features**: +- Image list view +- Image generation +- Image management + +#### Writer Dashboard (`/writer`) +**Features**: +- KPIs (total tasks, content, images) +- Workflow steps +- Charts and visualizations + +### Thinker Module + +#### Prompts Page (`/thinker/prompts`) +**Features**: +- AI prompt CRUD operations +- Prompt type management +- Default prompt reset + +#### Author Profiles Page (`/thinker/author-profiles`) +**Features**: +- Author profile CRUD operations +- Writing style configuration + +#### Strategies Page (`/thinker/strategies`) +**Features**: +- Content strategy CRUD operations +- Strategy configuration + +#### Image Testing Page (`/thinker/image-testing`) +**Features**: +- Image generation testing +- Prompt testing +- Model testing + +### Billing Module + +#### Credits Page (`/billing/credits`) +**Features**: +- Credit balance display +- Credit purchase +- Credit history + +#### Transactions Page (`/billing/transactions`) +**Features**: +- Transaction history +- Transaction filtering + +#### Usage Page (`/billing/usage`) +**Features**: +- Usage logs +- Cost tracking +- Usage analytics + ### Settings Pages -#### Sites Page -**File**: `pages/Settings/Sites.tsx` - +#### Sites Page (`/settings/sites`) **Features**: - Site CRUD operations - Site activation/deactivation - Multiple sites can be active simultaneously -#### Integration Page -**File**: `pages/Settings/Integration.tsx` - +#### Integration Page (`/settings/integration`) **Features**: - Integration settings (OpenAI, Runware) - API key configuration - Test connections - Image generation testing +#### Users Page (`/settings/users`) +**Features**: +- User CRUD operations +- Role management +- Site access management + +#### AI Settings Page (`/settings/ai`) +**Features**: +- AI prompt management +- Model configuration + --- -## Hooks +## Hooks & Utilities ### useProgressModal -**File**: `hooks/useProgressModal.ts` - -**Purpose**: Progress modal for long-running Celery tasks. - -**Usage**: -```tsx -const progressModal = useProgressModal(); - -// Start task -progressModal.start(taskId, 'Task started...'); - -// Task progress is automatically polled -// Modal displays progress updates -``` +**Purpose**: Progress modal for long-running AI tasks **Features**: -- Automatic polling of task progress -- Progress percentage display -- Step-by-step logs (request/response steps) -- Success/Error handling -- Auto-close on completion +- Displays progress percentage +- Shows phase messages +- Displays request/response steps +- Shows cost and token information +- Auto-closes on completion ### useAuth -**File**: `hooks/useAuth.ts` - -**Purpose**: Authentication hook. - -**Usage**: -```tsx -const { user, isAuthenticated, login, logout } = useAuth(); -``` +**Purpose**: Authentication hook **Features**: -- Access to auth state -- Login/logout functions -- Authentication status +- Checks authentication status +- Provides user information +- Handles token refresh ---- +### Utilities -## Utilities - -### Difficulty Utilities - -**File**: `utils/difficulty.ts` +#### difficulty.ts +**Purpose**: Difficulty calculation utilities **Functions**: -- `getDifficultyLabelFromNumber(number)`: Get difficulty label (Easy, Medium, Hard) -- `getDifficultyRange()`: Get difficulty range options - -### API Utilities - -**File**: `services/api.ts` - -**Helper Functions**: -- `getActiveSiteId()`: Get active site ID from store -- `getActiveSectorId()`: Get active sector ID from store -- `getAuthToken()`: Get JWT token from store -- `getRefreshToken()`: Get refresh token from store +- Difficulty level calculation +- Difficulty formatting --- ## Summary -The IGNY8 frontend is built on: +The IGNY8 frontend provides: -1. **Configuration-Driven Architecture**: Zero duplication, single source of truth +1. **Configuration-Driven UI**: All pages rendered from configuration 2. **4 Universal Templates**: Reusable templates for all page types 3. **TypeScript**: Full type safety -4. **Zustand State Management**: Lightweight, performant state management -5. **React Router v6**: Modern routing with nested routes -6. **Component Library**: Reusable UI components -7. **API Integration**: Automatic token handling and error management -8. **Responsive Design**: Mobile-first with Tailwind CSS +4. **Zustand State Management**: Lightweight, performant state +5. **React Router v6**: Modern routing +6. **Component Library**: Comprehensive UI components +7. **API Integration**: Automatic token handling and retry +8. **Progress Tracking**: Real-time progress for AI tasks +9. **Responsive Design**: Mobile-first approach +10. **Complete Feature Set**: All modules and pages implemented -This architecture ensures consistency, maintainability, and extensibility while providing a great user experience. +This architecture ensures consistency, maintainability, and rapid feature development while providing a great user experience. diff --git a/docs/03-BACKEND.md b/docs/04-BACKEND.md similarity index 64% rename from docs/03-BACKEND.md rename to docs/04-BACKEND.md index 014677ee..19afb8f4 100644 --- a/docs/03-BACKEND.md +++ b/docs/04-BACKEND.md @@ -1,6 +1,5 @@ # IGNY8 Backend Documentation -**Version:** 1.0 **Last Updated:** 2025-01-XX **Purpose:** Complete backend documentation covering models, views, APIs, modules, serializers, tasks, and structure. @@ -33,7 +32,7 @@ The IGNY8 backend is a Django 5.2+ application using Django REST Framework (DRF) - **RESTful API**: DRF ViewSets with consistent response format - **Celery Integration**: Asynchronous task processing for long-running operations - **Account/Site/Sector Hierarchy**: Hierarchical data organization -- **AI Integration**: Unified AIProcessor for all AI operations +- **AI Integration**: Unified AI framework for all AI operations - **Progress Tracking**: Real-time progress updates for Celery tasks --- @@ -75,7 +74,7 @@ backend/igny8_core/ │ │ └── urls.py # Planner module URLs │ ├── writer/ # Tasks, Content, Images │ │ ├── models.py # Tasks, Content, Images models -│ │ ├── views.py # TasksViewSet +│ │ ├── views.py # TasksViewSet, ImagesViewSet │ │ ├── tasks.py # Celery tasks for content/image generation │ │ └── urls.py # Writer module URLs │ ├── system/ # Settings, Prompts, Integration @@ -85,18 +84,34 @@ backend/igny8_core/ │ │ ├── utils.py # Default prompts, prompt loading │ │ └── urls.py # System module URLs │ └── billing/ # Credits, Transactions, Usage -│ ├── models.py # CreditTransaction, UsageLog models +│ ├── models.py # CreditTransaction, CreditUsageLog models │ ├── views.py # Billing ViewSets │ └── services.py # CreditService ├── api/ # API base classes │ ├── base.py # AccountModelViewSet, SiteSectorModelViewSet │ └── pagination.py # CustomPageNumberPagination ├── utils/ # Shared utilities -│ ├── ai_processor.py # Unified AI interface +│ ├── ai_processor.py # Unified AI interface (legacy, see AI functions) │ └── content_normalizer.py # Content processing utilities ├── middleware/ # Custom middleware │ ├── account.py # AccountContextMiddleware (sets request.account) │ └── resource_tracker.py # ResourceTrackerMiddleware (API metrics) +├── ai/ # AI framework +│ ├── base.py # BaseAIFunction +│ ├── engine.py # AIEngine +│ ├── tasks.py # run_ai_task +│ ├── registry.py # Function registry +│ ├── prompts.py # PromptRegistry +│ ├── ai_core.py # AICore +│ ├── settings.py # Model configuration +│ ├── validators.py # Validation functions +│ ├── tracker.py # Progress tracking +│ └── functions/ # AI function implementations +│ ├── auto_cluster.py +│ ├── generate_ideas.py +│ ├── generate_content.py +│ ├── generate_image_prompts.py +│ └── generate_images.py ├── settings.py # Django settings ├── urls.py # Root URL configuration └── celery.py # Celery configuration @@ -120,7 +135,6 @@ backend/igny8_core/ **Usage**: All models that need account isolation inherit from this. - #### SiteSectorBaseModel **File**: `auth/models.py` @@ -139,266 +153,207 @@ backend/igny8_core/ ### Auth Models #### Account -**File**: `auth/models.py` - **Table**: `igny8_accounts` **Fields**: - `name`: CharField - `slug`: SlugField (unique) - `owner`: ForeignKey to User -- `stripe_customer_id`: CharField (optional) - `plan`: ForeignKey to Plan - `credits`: IntegerField (default: 0) - `status`: CharField (choices: active, suspended, trial, cancelled) **Methods**: -- `is_system_account()`: Returns True if account is system account (aws-admin, default-account, default) +- `is_system_account()`: Returns True if account is a system account #### User -**File**: `auth/models.py` - **Table**: `igny8_users` +**Inherits**: `AbstractUser` + **Fields**: -- Inherits from `AbstractUser` -- `email`: EmailField (unique, USERNAME_FIELD) +- `email`: EmailField (unique) - `account`: ForeignKey to Account - `role`: CharField (choices: developer, owner, admin, editor, viewer, system_bot) **Methods**: -- `has_role(*roles)`: Check if user has any of the specified roles -- `is_owner_or_admin()`: Returns True if role is owner or admin -- `is_developer()`: Returns True if role is developer or is_superuser -- `is_admin_or_developer()`: Returns True if role is admin or developer -- `is_system_account_user()`: Returns True if user belongs to system account -- `get_accessible_sites()`: Returns QuerySet of sites user can access +- `has_role(role)`: Checks if user has role +- `is_owner_or_admin()`: Checks if user is owner or admin +- `is_developer()`: Checks if user is developer +- `is_admin_or_developer()`: Checks if user is admin or developer +- `is_system_account_user()`: Checks if user belongs to system account +- `get_accessible_sites()`: Returns queryset of accessible sites #### Plan -**File**: `auth/models.py` - **Table**: `igny8_plans` -**Fields**: -- `name`: CharField -- `slug`: SlugField (unique) -- `price`: DecimalField -- `billing_cycle`: CharField (choices: monthly, annual) -- `features`: JSONField (array of feature strings) -- `max_users`: IntegerField -- `max_sites`: IntegerField -- `max_keywords`: IntegerField -- `max_clusters`: IntegerField -- `max_content_ideas`: IntegerField -- `daily_cluster_limit`: IntegerField -- `monthly_cluster_ai_credits`: IntegerField -- `daily_content_tasks`: IntegerField -- `daily_ai_requests`: IntegerField -- `monthly_word_count_limit`: IntegerField -- `monthly_content_ai_credits`: IntegerField -- `monthly_image_count`: IntegerField -- `daily_image_generation_limit`: IntegerField -- `monthly_image_ai_credits`: IntegerField -- `max_images_per_task`: IntegerField -- `image_model_choices`: JSONField -- `included_credits`: IntegerField -- `extra_credit_price`: DecimalField -- And more... +**Fields**: Extensive fields for limits (users, sites, keywords, clusters, content ideas, AI requests, word count, images, credits) **Methods**: - `clean()`: Validates plan limits - `get_effective_credits_per_month()`: Returns included_credits or credits_per_month #### Site -**File**: `auth/models.py` - **Table**: `igny8_sites` +**Inherits**: `AccountBaseModel` + **Fields**: -- Inherits from `AccountBaseModel` - `name`: CharField - `slug`: SlugField (unique per account) - `domain`: URLField (optional) - `industry`: ForeignKey to Industry (optional) - `is_active`: BooleanField -- `status`: CharField (choices: active, inactive, suspended) -- `wp_url`: URLField (optional, WordPress integration) +- `status`: CharField +- `wp_url`: URLField (optional) - `wp_username`: CharField (optional) - `wp_app_password`: CharField (optional) **Methods**: -- `get_active_sectors_count()`: Get count of active sectors -- `can_add_sector()`: Check if site can add another sector (max 5) +- `get_active_sectors_count()`: Returns count of active sectors +- `can_add_sector()`: Returns True if site can add another sector (max 5) #### Sector -**File**: `auth/models.py` - **Table**: `igny8_sectors` +**Inherits**: `AccountBaseModel` + **Fields**: -- Inherits from `AccountBaseModel` - `site`: ForeignKey to Site -- `industry_sector`: ForeignKey to IndustrySector (optional, template reference) +- `industry_sector`: ForeignKey to IndustrySector (optional) - `name`: CharField - `slug`: SlugField (unique per site) - `is_active`: BooleanField -- `status`: CharField (choices: active, inactive) +- `status`: CharField + +**Validation**: Maximum 5 active sectors per site ### Planner Models #### Keywords -**File**: `modules/planner/models.py` - **Table**: `igny8_keywords` +**Inherits**: `SiteSectorBaseModel` + **Fields**: -- Inherits from `SiteSectorBaseModel` - `keyword`: CharField -- `volume`: IntegerField -- `difficulty`: IntegerField -- `intent`: CharField (choices: informational, navigational, commercial, transactional) -- `cluster`: ForeignKey to Clusters (optional) -- `status`: CharField (choices: active, pending, archived) +- `volume`: IntegerField (optional) +- `difficulty`: IntegerField (optional) +- `intent`: CharField (optional) +- `status`: CharField +- `cluster`: ManyToManyField to Clusters #### Clusters -**File**: `modules/planner/models.py` - **Table**: `igny8_clusters` +**Inherits**: `SiteSectorBaseModel` + **Fields**: -- Inherits from `SiteSectorBaseModel` -- `name`: CharField (unique) -- `description`: TextField -- `keywords_count`: IntegerField -- `volume`: IntegerField -- `mapped_pages`: IntegerField +- `name`: CharField +- `description`: TextField (optional) +- `keywords_count`: IntegerField (calculated) +- `volume`: IntegerField (calculated) - `status`: CharField +- `keywords`: ManyToManyField to Keywords #### ContentIdeas -**File**: `modules/planner/models.py` - **Table**: `igny8_content_ideas` +**Inherits**: `SiteSectorBaseModel` + **Fields**: -- Inherits from `SiteSectorBaseModel` - `idea_title`: CharField - `description`: TextField -- `content_structure`: CharField (choices: cluster_hub, landing_page, pillar_page, supporting_page) -- `content_type`: CharField (choices: blog_post, article, guide, tutorial) -- `target_keywords`: CharField (comma-separated, legacy) -- `keyword_objects`: ManyToManyField to Keywords +- `content_type`: CharField +- `content_structure`: CharField +- `target_keywords`: TextField - `keyword_cluster`: ForeignKey to Clusters -- `status`: CharField (choices: new, scheduled, published) - `estimated_word_count`: IntegerField +- `status`: CharField ### Writer Models #### Tasks -**File**: `modules/writer/models.py` - **Table**: `igny8_tasks` +**Inherits**: `SiteSectorBaseModel` + **Fields**: -- Inherits from `SiteSectorBaseModel` - `title`: CharField - `description`: TextField -- `keywords`: CharField (comma-separated, legacy) -- `keyword_objects`: ManyToManyField to Keywords -- `cluster`: ForeignKey to Clusters -- `idea`: ForeignKey to ContentIdeas -- `content_structure`: CharField +- `cluster`: ForeignKey to Clusters (optional) +- `idea`: ForeignKey to ContentIdeas (optional) - `content_type`: CharField -- `status`: CharField (choices: queued, in_progress, draft, review, published, completed) -- `content`: TextField (generated content) -- `word_count`: IntegerField -- `meta_title`: CharField -- `meta_description`: TextField -- `assigned_post_id`: IntegerField (WordPress post ID) -- `post_url`: URLField +- `content_structure`: CharField +- `status`: CharField #### Content -**File**: `modules/writer/models.py` - **Table**: `igny8_content` +**Inherits**: `SiteSectorBaseModel` + **Fields**: -- Inherits from `SiteSectorBaseModel` - `task`: OneToOneField to Tasks - `html_content`: TextField - `word_count`: IntegerField -- `metadata`: JSONField - -**Methods**: -- `save()`: Automatically sets account, site, sector from task +- `status`: CharField +- `wp_post_id`: IntegerField (optional) +- `meta_title`: CharField (optional) +- `meta_description`: TextField (optional) +- `primary_keyword`: CharField (optional) +- `secondary_keywords`: TextField (optional) #### Images -**File**: `modules/writer/models.py` - **Table**: `igny8_images` -**Fields**: -- Inherits from `SiteSectorBaseModel` -- `task`: ForeignKey to Tasks -- `image_type`: CharField (choices: featured, desktop, mobile, in_article) -- `image_url`: URLField -- `image_path`: CharField (local path) -- `prompt`: TextField -- `status`: CharField -- `position`: IntegerField +**Inherits**: `SiteSectorBaseModel` -**Methods**: -- `save()`: Automatically sets account, site, sector from task +**Fields**: +- `task`: ForeignKey to Tasks (optional) +- `content`: ForeignKey to Content (optional) +- `image_type`: CharField (choices: featured, in_article, desktop, mobile) +- `prompt`: TextField +- `image_url`: CharField +- `status`: CharField ### System Models #### AIPrompt -**File**: `modules/system/models.py` - **Table**: `igny8_ai_prompts` -**Fields**: -- Inherits from `AccountBaseModel` -- `prompt_type`: CharField (choices: clustering, ideas, content_generation, image_prompt_extraction, image_prompt_template, negative_prompt) -- `prompt_value`: TextField -- `default_prompt`: TextField -- `is_active`: BooleanField +**Inherits**: `AccountBaseModel` -**Unique Constraint**: `(account, prompt_type)` +**Fields**: +- `prompt_type`: CharField +- `prompt_value`: TextField +- `function_name`: CharField (optional) #### IntegrationSettings -**File**: `modules/system/models.py` - **Table**: `igny8_integration_settings` -**Fields**: -- Inherits from `AccountBaseModel` -- `integration_type`: CharField (choices: openai, runware, gsc, image_generation) -- `config`: JSONField (API keys, settings, etc.) -- `is_active`: BooleanField +**Inherits**: `AccountBaseModel` -**Unique Constraint**: `(account, integration_type)` +**Fields**: +- `integration_type`: CharField (choices: openai, runware, image_generation) +- `config`: JSONField #### AuthorProfile -**File**: `modules/system/models.py` - **Table**: `igny8_author_profiles` +**Inherits**: `AccountBaseModel` + **Fields**: -- Inherits from `AccountBaseModel` - `name`: CharField - `description`: TextField - `tone`: CharField - `language`: CharField -- `structure_template`: JSONField -- `is_active`: BooleanField #### Strategy -**File**: `modules/system/models.py` - **Table**: `igny8_strategies` +**Inherits**: `AccountBaseModel` + **Fields**: -- Inherits from `AccountBaseModel` - `name`: CharField - `description`: TextField - `sector`: ForeignKey to Sector (optional) @@ -406,6 +361,30 @@ backend/igny8_core/ - `section_logic`: JSONField - `is_active`: BooleanField +### Billing Models + +#### CreditTransaction +**Table**: `igny8_credit_transactions` + +**Inherits**: `AccountBaseModel` + +**Fields**: +- `transaction_type`: CharField (choices: purchase, usage, refund, adjustment) +- `amount`: IntegerField +- `balance_after`: IntegerField +- `description`: TextField (optional) + +#### CreditUsageLog +**Table**: `igny8_credit_usage_logs` + +**Inherits**: `AccountBaseModel` + +**Fields**: +- `operation_type`: CharField +- `credits_used`: IntegerField +- `cost_usd`: DecimalField +- `details`: JSONField (optional) + --- ## ViewSets @@ -448,8 +427,6 @@ backend/igny8_core/ ### Planner ViewSets #### KeywordViewSet -**File**: `modules/planner/views.py` - **Inherits**: `SiteSectorModelViewSet` **Actions**: @@ -471,8 +448,6 @@ backend/igny8_core/ - Ordering: `created_at`, `volume`, `difficulty` #### ClusterViewSet -**File**: `modules/planner/views.py` - **Inherits**: `SiteSectorModelViewSet` **Actions**: @@ -484,8 +459,6 @@ backend/igny8_core/ - `auto_generate_ideas()`: Auto-generate content ideas for clusters #### ContentIdeasViewSet -**File**: `modules/planner/views.py` - **Inherits**: `SiteSectorModelViewSet` **Actions**: @@ -498,8 +471,6 @@ backend/igny8_core/ ### Writer ViewSets #### TasksViewSet -**File**: `modules/writer/views.py` - **Inherits**: `SiteSectorModelViewSet` **Actions**: @@ -509,7 +480,7 @@ backend/igny8_core/ - `update()`: Update task - `destroy()`: Delete task - `auto_generate_content()`: Auto-generate content for tasks -- `auto_generate_images()`: Auto-generate images for tasks +- `generate_images()`: Generate images for image records - `bulk_delete()`: Bulk delete tasks - `bulk_update()`: Bulk update task status @@ -518,11 +489,20 @@ backend/igny8_core/ - Filters: `status`, `cluster_id`, `content_type`, `content_structure` - Ordering: `title`, `created_at`, `word_count`, `status` +#### ImagesViewSet +**Inherits**: `SiteSectorModelViewSet` + +**Actions**: +- `list()`: List images +- `create()`: Create image +- `retrieve()`: Get image details +- `update()`: Update image +- `destroy()`: Delete image +- `generate_images()`: Generate images using AI + ### System ViewSets #### IntegrationSettingsViewSet -**File**: `modules/system/integration_views.py` - **Inherits**: `viewsets.ViewSet` **Actions**: @@ -537,8 +517,6 @@ backend/igny8_core/ - `task_progress()`: Get Celery task progress #### AIPromptViewSet -**File**: `modules/system/views.py` - **Inherits**: `AccountModelViewSet` **Actions**: @@ -550,8 +528,6 @@ backend/igny8_core/ - `reset_to_default()`: Reset prompt to default value #### AuthorProfileViewSet -**File**: `modules/system/views.py` - **Inherits**: `AccountModelViewSet` **Actions**: @@ -568,63 +544,65 @@ backend/igny8_core/ ### Planner Serializers #### KeywordSerializer -**File**: `modules/planner/serializers.py` - **Fields**: All Keyword model fields **Validation**: Validates keyword uniqueness, cluster belongs to same sector #### ClusterSerializer -**File**: `modules/planner/cluster_serializers.py` - **Fields**: All Cluster model fields **Read-Only Fields**: `keywords_count`, `volume` (calculated) #### ContentIdeasSerializer -**File**: `modules/planner/serializers.py` - **Fields**: All ContentIdeas model fields ### Writer Serializers #### TasksSerializer -**File**: `modules/writer/serializers.py` - **Fields**: All Tasks model fields #### ContentSerializer -**File**: `modules/writer/serializers.py` - **Fields**: All Content model fields #### ImagesSerializer -**File**: `modules/writer/serializers.py` - **Fields**: All Images model fields ### System Serializers #### AIPromptSerializer -**File**: `modules/system/serializers.py` - **Fields**: All AIPrompt model fields #### IntegrationSettingsSerializer -**File**: `modules/system/serializers.py` - **Fields**: All IntegrationSettings model fields --- ## Celery Tasks +### AI Task Entry Point + +#### run_ai_task +**File**: `ai/tasks.py` + +**Purpose**: Unified Celery task entrypoint for all AI functions + +**Parameters**: +- `function_name`: Function name (e.g., 'auto_cluster') +- `payload`: Function-specific payload +- `account_id`: Account ID + +**Flow**: +1. Gets account from account_id +2. Gets function instance from registry +3. Creates AIEngine +4. Executes function via AIEngine + ### Planner Tasks #### auto_cluster_keywords_task -**File**: `modules/planner/tasks.py` +**File**: `modules/planner/tasks.py` (legacy, now uses `run_ai_task`) -**Purpose**: Auto-cluster keywords using AI. +**Purpose**: Auto-cluster keywords using AI **Parameters**: - `keyword_ids`: List of keyword IDs @@ -634,12 +612,10 @@ backend/igny8_core/ **Progress Tracking**: Updates progress with request_steps and response_steps -**Calls**: `_auto_cluster_keywords_core()` - #### auto_generate_ideas_task -**File**: `modules/planner/tasks.py` +**File**: `modules/planner/tasks.py` (legacy, now uses `run_ai_task`) -**Purpose**: Auto-generate content ideas for clusters. +**Purpose**: Auto-generate content ideas for clusters **Parameters**: - `cluster_ids`: List of cluster IDs @@ -647,14 +623,12 @@ backend/igny8_core/ **Progress Tracking**: Updates progress for each cluster -**Calls**: `_generate_single_idea_core()` for each cluster - ### Writer Tasks #### auto_generate_content_task -**File**: `modules/writer/tasks.py` +**File**: `modules/writer/tasks.py` (legacy, now uses `run_ai_task`) -**Purpose**: Auto-generate content for tasks. +**Purpose**: Auto-generate content for tasks **Parameters**: - `task_ids`: List of task IDs @@ -662,20 +636,17 @@ backend/igny8_core/ **Progress Tracking**: Updates progress for each task -**Calls**: `AIProcessor.generate_content()` +#### process_image_generation_queue +**File**: `modules/writer/views.py` (via `ai/tasks.py`) -#### auto_generate_images_task -**File**: `modules/writer/tasks.py` - -**Purpose**: Auto-generate images for tasks. +**Purpose**: Generate images for image records **Parameters**: -- `task_ids`: List of task IDs +- `image_ids`: List of image IDs - `account_id`: Account ID +- `content_id`: Content ID (optional) -**Progress Tracking**: Updates progress for each task - -**Calls**: `AIProcessor.extract_image_prompts()` and `AIProcessor.generate_image()` +**Progress Tracking**: Updates progress for each image sequentially --- @@ -705,7 +676,7 @@ backend/igny8_core/ - `GET /api/v1/writer/tasks/` - List tasks - `POST /api/v1/writer/tasks/auto_generate_content/` - Auto-generate content -- `POST /api/v1/writer/tasks/auto_generate_images/` - Auto-generate images +- `POST /api/v1/writer/images/generate_images/` - Generate images ### System Endpoints @@ -724,7 +695,7 @@ backend/igny8_core/ **File**: `api/base.py` -**Purpose**: Base ViewSet with automatic account filtering. +**Purpose**: Base ViewSet with automatic account filtering **Features**: - Automatic account filtering @@ -735,7 +706,7 @@ backend/igny8_core/ **File**: `api/base.py` -**Purpose**: Base ViewSet with site/sector filtering. +**Purpose**: Base ViewSet with site/sector filtering **Features**: - Account filtering (inherited) @@ -751,7 +722,7 @@ backend/igny8_core/ **File**: `middleware/account.py` -**Purpose**: Sets `request.account` from JWT token. +**Purpose**: Sets `request.account` from JWT token **Functionality**: - Extracts account ID from JWT token @@ -762,7 +733,7 @@ backend/igny8_core/ **File**: `middleware/resource_tracker.py` -**Purpose**: Tracks API request metrics. +**Purpose**: Tracks API request metrics **Functionality**: - Tracks CPU, memory, I/O usage @@ -773,29 +744,15 @@ backend/igny8_core/ ## Utilities -### AIProcessor - -**File**: `utils/ai_processor.py` - -**Purpose**: Unified AI interface for all AI operations. - -**Methods**: -- `cluster_keywords()`: Cluster keywords using AI -- `generate_ideas()`: Generate content ideas -- `generate_content()`: Generate text content -- `extract_image_prompts()`: Extract image prompts from content -- `generate_image()`: Generate images using OpenAI DALL-E or Runware - -**See**: AI Functions documentation for complete details - ### Content Normalizer **File**: `utils/content_normalizer.py` -**Purpose**: Content processing utilities. +**Purpose**: Content processing utilities **Functions**: -- `_extract_body_content()`: Extract body content from HTML +- `normalize_content()`: Converts plain text to HTML +- `_extract_body_content()`: Extracts body content from HTML --- @@ -803,58 +760,58 @@ backend/igny8_core/ ### Planner Module -**Purpose**: Keyword management and content planning. +**Purpose**: Keyword management and content planning **Models**: Keywords, Clusters, ContentIdeas **ViewSets**: KeywordViewSet, ClusterViewSet, ContentIdeasViewSet -**Tasks**: auto_cluster_keywords_task, auto_generate_ideas_task +**Tasks**: Auto-cluster keywords, Auto-generate ideas ### Writer Module -**Purpose**: Content generation and management. +**Purpose**: Content generation and management **Models**: Tasks, Content, Images -**ViewSets**: TasksViewSet +**ViewSets**: TasksViewSet, ImagesViewSet -**Tasks**: auto_generate_content_task, auto_generate_images_task +**Tasks**: Auto-generate content, Generate images ### System Module -**Purpose**: System settings, prompts, and integrations. +**Purpose**: System configuration and AI settings **Models**: AIPrompt, IntegrationSettings, AuthorProfile, Strategy -**ViewSets**: AIPromptViewSet, AuthorProfileViewSet, IntegrationSettingsViewSet - -**Utilities**: Default prompts, prompt loading +**ViewSets**: AIPromptViewSet, IntegrationSettingsViewSet, AuthorProfileViewSet ### Billing Module -**Purpose**: Credits, transactions, and usage tracking. +**Purpose**: Credit management and usage tracking -**Models**: CreditTransaction, UsageLog +**Models**: CreditTransaction, CreditUsageLog -**ViewSets**: CreditTransactionViewSet, UsageLogViewSet +**ViewSets**: CreditTransactionViewSet, CreditUsageLogViewSet -**Services**: CreditService +**Services**: CreditService (check, deduct, add, calculate credits) --- ## Summary -The IGNY8 backend is built on: +The IGNY8 backend provides: -1. **Django + DRF**: Robust web framework with RESTful API -2. **Multi-Tenancy**: Complete account isolation with automatic filtering -3. **Modular Architecture**: Clear module boundaries with shared utilities -4. **Celery Integration**: Asynchronous task processing for long-running operations -5. **Base ViewSets**: Consistent access control and filtering -6. **AI Integration**: Unified AIProcessor for all AI operations -7. **Progress Tracking**: Real-time progress updates for Celery tasks -8. **Account/Site/Sector Hierarchy**: Hierarchical data organization +1. **Multi-Tenancy**: Complete account isolation with automatic filtering +2. **RESTful API**: DRF ViewSets with consistent response format +3. **Celery Integration**: Asynchronous task processing +4. **Hierarchical Organization**: Account > Site > Sector > Content structure +5. **AI Framework**: Unified AI framework for all AI operations +6. **Progress Tracking**: Real-time progress updates for Celery tasks +7. **Module-Based Design**: Clear separation of concerns +8. **Base Classes**: Reusable ViewSets for common patterns +9. **Middleware**: Account context and resource tracking +10. **Utilities**: Content processing and AI integration -This architecture ensures scalability, maintainability, and extensibility while providing a robust API for the frontend. +This architecture ensures scalability, maintainability, and extensibility while providing a robust foundation for the IGNY8 platform. diff --git a/docs/05-ACCOUNT-USER-PLAN.md b/docs/05-ACCOUNT-USER-PLAN.md deleted file mode 100644 index d0f8cd0a..00000000 --- a/docs/05-ACCOUNT-USER-PLAN.md +++ /dev/null @@ -1,739 +0,0 @@ -# IGNY8 Account, User, Plan, Sites, Credits Documentation - -**Version:** 1.0 -**Last Updated:** 2025-01-XX -**Purpose:** Complete documentation of multi-tenancy architecture, access control, plans, credits, and related systems. - ---- - -## Table of Contents - -1. [Overview](#overview) -2. [Account Model](#account-model) -3. [User Model](#user-model) -4. [Plan Model](#plan-model) -5. [Subscription Model](#subscription-model) -6. [Site Model](#site-model) -7. [Sector Model](#sector-model) -8. [Industry & IndustrySector Models](#industry--industrysector-models) -9. [Access Control](#access-control) -10. [Credits System](#credits-system) -11. [Plan Limits](#plan-limits) -12. [Multi-Tenancy Architecture](#multi-tenancy-architecture) - ---- - -## Overview - -The IGNY8 platform uses a multi-account architecture with the following core entities: - -- **Account**: Top-level organization/workspace -- **User**: Individual user accounts with roles -- **Plan**: Subscription plan templates with limits -- **Subscription**: Active subscription linking Account to Plan -- **Site**: Workspace within an Account (1-N relationship) -- **Sector**: Content category within a Site (1-5 per site) -- **Industry**: Global industry templates -- **IndustrySector**: Industry sector templates - -### Key Relationships - -``` -Account (1) ──< (N) User -Account (1) ──< (1) Subscription ──> (1) Plan -Account (1) ──< (N) Site -Site (1) ──< (1-5) Sector -Industry (1) ──< (N) IndustrySector -Site (1) ──> (1) Industry (optional) -Sector (1) ──> (1) IndustrySector (optional, template reference) -``` - ---- - -## Account Model - -**File**: `backend/igny8_core/auth/models.py` - -**Table**: `igny8_accounts` - -### Fields - -- `id`: Primary key -- `name`: CharField - Account name -- `slug`: SlugField - Unique slug identifier -- `owner`: ForeignKey to User - Account owner -- `stripe_customer_id`: CharField - Stripe customer ID (optional) -- `plan`: ForeignKey to Plan - Current subscription plan -- `credits`: IntegerField - Current credit balance (default: 0) -- `status`: CharField - Account status (choices: active, suspended, trial, cancelled) -- `created_at`: DateTimeField - Creation timestamp -- `updated_at`: DateTimeField - Last update timestamp - -### Status Values - -- `active`: Normal operation -- `suspended`: Access temporarily revoked -- `trial`: Limited trial access -- `cancelled`: Account terminated - -### Methods - -#### is_system_account() -**Returns**: `True` if account is a system account (aws-admin, default-account, default) - -**Purpose**: System accounts bypass all filtering restrictions - - -### Relationships - -- **One-to-Many**: Users (via `user.account`) -- **One-to-One**: Subscription (via `account.subscription`) -- **One-to-Many**: Sites (via `site.account`) - - ---- - -## User Model - -**File**: `backend/igny8_core/auth/models.py` - -**Table**: `igny8_users` - -**Inherits**: `AbstractUser` (Django's built-in user model) - -### Fields - -- `id`: Primary key (inherited from AbstractUser) -- `email`: EmailField - Unique, USERNAME_FIELD -- `username`: CharField - Username -- `password`: Hashed password (inherited) -- `account`: ForeignKey to Account - User's account (null=True, blank=True) -- `role`: CharField - User role (choices: developer, owner, admin, editor, viewer, system_bot) -- `created_at`: DateTimeField - Creation timestamp -- `updated_at`: DateTimeField - Last update timestamp - -### Role Hierarchy (from highest to lowest) - -1. **developer**: Full system access, bypasses all restrictions -2. **owner**: Full access to account, can manage users and billing -3. **admin**: Admin access to account, can manage content and users -4. **editor**: Can edit content, manage clusters/tasks -5. **viewer**: Read-only access -6. **system_bot**: System automation user - -### Methods - -#### has_role(*roles) -**Parameters**: `*roles` - Variable number of role strings - -**Returns**: `True` if user has any of the specified roles - -#### is_owner_or_admin() -**Returns**: `True` if role is owner or admin - -#### is_developer() -**Returns**: `True` if role is developer or is_superuser - -#### is_admin_or_developer() -**Returns**: `True` if role is admin or developer (bypasses restrictions) - -**Purpose**: Admin/Developer users bypass account/site/sector filtering - -#### is_system_account_user() -**Returns**: `True` if user belongs to system account - -**Purpose**: System account users bypass all filtering restrictions - -#### get_accessible_sites() -**Returns**: QuerySet of sites user can access - -**Access Control Logic**: -- **System Account Users**: All active sites across all accounts -- **Developers**: All active sites across all accounts -- **Owners/Admins**: All sites in their account -- **Editors/Viewers**: Only sites explicitly granted via `SiteUserAccess` - - -### Relationships - -- **Many-to-One**: Account (via `user.account`) -- **Many-to-Many**: Sites (via `SiteUserAccess`) - ---- - -## Plan Model - -**File**: `backend/igny8_core/auth/models.py` - -**Table**: `igny8_plans` - -### Fields - -#### Plan Information -- `id`: Primary key -- `name`: CharField - Plan name (e.g., "Free", "Starter", "Growth") -- `slug`: SlugField - Unique slug identifier -- `price`: DecimalField - Monthly price -- `billing_cycle`: CharField - Billing cycle (choices: monthly, annual) -- `features`: JSONField - Array of feature strings (e.g., ['ai_writer', 'image_gen', 'auto_publish']) -- `is_active`: BooleanField - Whether plan is available for subscription - -#### User/Site/Scope Limits -- `max_users`: IntegerField - Total users allowed per account (default: 1, min: 1) -- `max_sites`: IntegerField - Maximum sites allowed (default: 1, min: 1) -- `max_industries`: IntegerField - Optional limit for industries/sectors (nullable) -- `max_author_profiles`: IntegerField - Limit for saved writing styles (default: 5) - -#### Planner Limits -- `max_keywords`: IntegerField - Total keywords allowed globally (default: 1000) -- `max_clusters`: IntegerField - Total clusters allowed globally (default: 100) -- `max_content_ideas`: IntegerField - Total content ideas allowed globally (default: 300) -- `daily_cluster_limit`: IntegerField - Max clusters per day (default: 10) -- `daily_keyword_import_limit`: IntegerField - Seed keywords import limit per day (default: 100) -- `monthly_cluster_ai_credits`: IntegerField - AI credits for clustering (default: 50) - -#### Writer Limits -- `daily_content_tasks`: IntegerField - Max content tasks per day (default: 10) -- `daily_ai_requests`: IntegerField - Total AI executions per day (default: 50) -- `monthly_word_count_limit`: IntegerField - Monthly word limit for generated content (default: 50000) -- `monthly_content_ai_credits`: IntegerField - AI credit pool for content generation (default: 200) - -#### Image Generation Limits -- `monthly_image_count`: IntegerField - Max images per month (default: 100) -- `daily_image_generation_limit`: IntegerField - Max images per day (default: 25) -- `monthly_image_ai_credits`: IntegerField - AI credit pool for images (default: 100) -- `max_images_per_task`: IntegerField - Max images per content task (default: 4, min: 1) -- `image_model_choices`: JSONField - Allowed image models (e.g., ['dalle3', 'hidream']) - -#### AI Request Controls -- `daily_ai_request_limit`: IntegerField - Global daily AI request cap (default: 100) -- `monthly_ai_credit_limit`: IntegerField - Unified credit ceiling per month (default: 500) - -#### Billing & Credits -- `included_credits`: IntegerField - Monthly credits included (default: 0) -- `extra_credit_price`: DecimalField - Price per additional credit (default: 0.01) -- `allow_credit_topup`: BooleanField - Can user purchase more credits? (default: True) -- `auto_credit_topup_threshold`: IntegerField - Auto top-up trigger point (nullable) -- `auto_credit_topup_amount`: IntegerField - Credits to auto-buy (nullable) - -#### Stripe Integration -- `stripe_product_id`: CharField - Stripe product ID (nullable) -- `stripe_price_id`: CharField - Stripe price ID (nullable) - -#### Legacy Fields -- `credits_per_month`: IntegerField - DEPRECATED: Use included_credits instead (default: 0) - -### Methods - -#### clean() -**Purpose**: Validates plan limits - -**Validations**: -- `max_sites` must be >= 1 -- `included_credits` must be >= 0 - -#### get_effective_credits_per_month() -**Returns**: `included_credits` if set, otherwise `credits_per_month` (backward compatibility) - -### Relationships - -- **One-to-Many**: Accounts (via `account.plan`) - -### Plan Limits Enforcement - -**Current Status**: Plan limits are defined but **not currently enforced** in AI functions. - -**Future Implementation**: Plan limits should be checked before allowing operations. - ---- - -## Subscription Model - -**File**: `backend/igny8_core/auth/models.py` - -**Table**: `igny8_subscriptions` - -### Fields - -- `id`: Primary key -- `account`: OneToOneField to Account - Account subscription -- `stripe_subscription_id`: CharField - Unique Stripe subscription ID -- `status`: CharField - Subscription status (choices: active, past_due, canceled, trialing) -- `current_period_start`: DateTimeField - Current billing period start -- `current_period_end`: DateTimeField - Current billing period end -- `cancel_at_period_end`: BooleanField - Cancel at period end flag (default: False) -- `created_at`: DateTimeField - Creation timestamp -- `updated_at`: DateTimeField - Last update timestamp - -### Status Values - -- `active`: Subscription is active and paid -- `past_due`: Payment failed, subscription past due -- `canceled`: Subscription canceled -- `trialing`: In trial period - -### Methods - - -### Relationships - -- **One-to-One**: Account (via `account.subscription`) - ---- - -## Site Model - -**File**: `backend/igny8_core/auth/models.py` - -**Table**: `igny8_sites` - -**Inherits**: `AccountBaseModel` - -### Fields - -- `id`: Primary key -- `account`: ForeignKey to Account - Account this site belongs to -- `name`: CharField - Site name -- `slug`: SlugField - Unique slug per account -- `domain`: URLField - Primary domain URL (optional) -- `description`: TextField - Site description (optional) -- `industry`: ForeignKey to Industry - Industry this site belongs to (optional) -- `is_active`: BooleanField - Whether site is active (default: True) -- `status`: CharField - Site status (choices: active, inactive, suspended, default: active) -- `wp_url`: URLField - WordPress site URL (optional, for WordPress integration) -- `wp_username`: CharField - WordPress username (optional) -- `wp_app_password`: CharField - WordPress app password (optional) -- `created_at`: DateTimeField - Creation timestamp -- `updated_at`: DateTimeField - Last update timestamp - -### Unique Constraint - -- `(account, slug)` - Slug unique per account - -### Methods - -#### get_active_sectors_count() -**Returns**: Count of active sectors for this site - -#### can_add_sector() -**Returns**: `True` if site can add another sector (max 5 sectors per site) - -### Relationships - -- **Many-to-One**: Account (via `site.account`) -- **Many-to-One**: Industry (via `site.industry`, optional) -- **One-to-Many**: Sectors (via `sector.site`) -- **Many-to-Many**: Users (via `SiteUserAccess`) - -### Site Activation - -**Important**: Multiple sites can be active simultaneously. The previous restriction of "only 1 site can be active at one time" has been removed. - ---- - -## Sector Model - -**File**: `backend/igny8_core/auth/models.py` - -**Table**: `igny8_sectors` - -**Inherits**: `AccountBaseModel` - -### Fields - -- `id`: Primary key -- `account`: ForeignKey to Account - Account this sector belongs to -- `site`: ForeignKey to Site - Site this sector belongs to -- `industry_sector`: ForeignKey to IndustrySector - Reference to industry sector template (optional) -- `name`: CharField - Sector name -- `slug`: SlugField - Unique slug per site -- `description`: TextField - Sector description (optional) -- `is_active`: BooleanField - Whether sector is active (default: True) -- `status`: CharField - Sector status (choices: active, inactive, default: active) -- `created_at`: DateTimeField - Creation timestamp -- `updated_at`: DateTimeField - Last update timestamp - -### Unique Constraint - -- `(site, slug)` - Slug unique per site - -### Validation - -**On Save**: -1. Automatically sets `account` from `site.account` -2. Validates that `industry_sector.industry` matches `site.industry` (if both set) -3. Validates sector limit: Maximum 5 active sectors per site - -### Methods - -#### industry (property) -**Returns**: Industry for this sector (from `industry_sector.industry` if set) - -### Relationships - -- **Many-to-One**: Account (via `sector.account`) -- **Many-to-One**: Site (via `sector.site`) -- **Many-to-One**: IndustrySector (via `sector.industry_sector`, optional, template reference) -- **One-to-Many**: Keywords, Clusters, ContentIdeas, Tasks (via their `sector` field) - ---- - -## Industry & IndustrySector Models - -### Industry Model - -**File**: `backend/igny8_core/auth/models.py` - -**Table**: `igny8_industries` - -**Purpose**: Global industry templates. - -### Fields - -- `id`: Primary key -- `name`: CharField - Industry name (unique) -- `slug`: SlugField - Unique slug identifier -- `description`: TextField - Industry description (optional) -- `is_active`: BooleanField - Whether industry is active (default: True) -- `created_at`: DateTimeField - Creation timestamp -- `updated_at`: DateTimeField - Last update timestamp - -### Relationships - -- **One-to-Many**: Sites (via `site.industry`) -- **One-to-Many**: IndustrySectors (via `industry_sector.industry`) - -### IndustrySector Model - -**File**: `backend/igny8_core/auth/models.py` - -**Table**: `igny8_industry_sectors` - -**Purpose**: Sector templates within industries. - -### Fields - -- `id`: Primary key -- `industry`: ForeignKey to Industry - Industry this sector belongs to -- `name`: CharField - Sector name -- `slug`: SlugField - Unique slug per industry -- `description`: TextField - Sector description (optional) -- `suggested_keywords`: JSONField - List of suggested keywords for this sector template -- `is_active`: BooleanField - Whether sector is active (default: True) -- `created_at`: DateTimeField - Creation timestamp -- `updated_at`: DateTimeField - Last update timestamp - -### Unique Constraint - -- `(industry, slug)` - Slug unique per industry - -### Relationships - -- **Many-to-One**: Industry (via `industry_sector.industry`) -- **One-to-Many**: Sectors (via `sector.industry_sector`, template reference) -- **One-to-Many**: SeedKeywords (via `seed_keyword.sector`) - ---- - -## Access Control - -### Account Isolation - -**Principle**: All data is isolated by account. - -**Implementation**: -- All models inherit `AccountBaseModel` (has `account` ForeignKey) -- All ViewSets inherit `AccountModelViewSet` (filters by `request.account`) -- Middleware sets `request.account` from JWT token - -**Access Control**: -- **Admin/Developer users**: Bypass account filtering (see all accounts) -- **System account users**: Bypass account filtering (see all accounts) -- **Regular users**: Only see data from their account - -### Site Access Control - -**Principle**: Users can only access sites they have permission to access. - -**Implementation**: -- `User.get_accessible_sites()` returns sites user can access -- ViewSets filter by accessible sites -- `SiteUserAccess` model controls explicit site access for editors/viewers - -**Access Levels**: -- **System Account Users**: All active sites across all accounts -- **Developers**: All active sites across all accounts -- **Owners/Admins**: All sites in their account -- **Editors/Viewers**: Only sites explicitly granted via `SiteUserAccess` - -### SiteUserAccess Model - -**File**: `backend/igny8_core/auth/models.py` - -**Table**: `igny8_site_user_access` - -**Purpose**: Many-to-many relationship between Users and Sites for explicit access control. - -### Fields - -- `id`: Primary key -- `user`: ForeignKey to User -- `site`: ForeignKey to Site -- `granted_at`: DateTimeField - When access was granted -- `granted_by`: ForeignKey to User - Who granted access (optional) - -### Unique Constraint - -- `(user, site)` - One access record per user-site pair - -### Usage - -- Owners and Admins have automatic access to all sites in their account -- Editors and Viewers require explicit `SiteUserAccess` records -- System account users and Developers bypass this check - -### Role-Based Access Control (RBAC) - -**Role Hierarchy**: -1. **developer**: Full system access, bypasses all restrictions -2. **owner**: Full account access, can manage users and billing -3. **admin**: Account admin access, can manage content and users -4. **editor**: Content editing access, can manage clusters/tasks -5. **viewer**: Read-only access - -**Permission Checks**: -- `user.has_role(*roles)`: Check if user has any of the specified roles -- `user.is_owner_or_admin()`: Check if user is owner or admin -- `user.is_developer()`: Check if user is developer -- `user.is_admin_or_developer()`: Check if user is admin or developer (bypasses restrictions) - ---- - -## Credits System - -### Account Credits - -**Field**: `Account.credits` (IntegerField, default: 0) - -**Purpose**: Current credit balance for the account. - -**Usage**: Credits are deducted for AI operations and added through purchases or subscriptions. - -### Credit Models - -#### CreditTransaction - -**File**: `backend/igny8_core/modules/billing/models.py` - -**Table**: `igny8_credit_transactions` - -**Purpose**: Track all credit transactions (additions, deductions). - -### Fields - -- `id`: Primary key -- `account`: ForeignKey to Account -- `transaction_type`: CharField - Type of transaction (choices: purchase, subscription, refund, deduction, adjustment) -- `amount`: IntegerField - Positive for additions, negative for deductions -- `balance_after`: IntegerField - Credit balance after this transaction -- `description`: CharField - Transaction description -- `metadata`: JSONField - Additional context (AI call details, etc.) -- `created_at`: DateTimeField - Creation timestamp - -#### CreditUsageLog - -**File**: `backend/igny8_core/modules/billing/models.py` - -**Table**: `igny8_credit_usage_logs` - -**Purpose**: Detailed log of credit usage per AI operation. - -### Fields - -- `id`: Primary key -- `account`: ForeignKey to Account -- `operation_type`: CharField - Type of operation (choices: clustering, ideas, content, images, reparse) -- `credits_used`: IntegerField - Number of credits used -- `cost_usd`: DecimalField - Cost in USD (optional) -- `model_used`: CharField - AI model used (optional) -- `tokens_input`: IntegerField - Input tokens (optional) -- `tokens_output`: IntegerField - Output tokens (optional) -- `related_object_type`: CharField - Related object type (optional) -- `related_object_id`: IntegerField - Related object ID (optional) -- `metadata`: JSONField - Additional metadata -- `created_at`: DateTimeField - Creation timestamp - -### CreditService - -**File**: `backend/igny8_core/modules/billing/services.py` - -**Purpose**: Service for managing credits. - -#### Methods - -##### check_credits(account, required_credits) -**Purpose**: Check if account has enough credits. - -**Raises**: `InsufficientCreditsError` if account doesn't have enough credits - -##### deduct_credits(account, amount, operation_type, description, ...) -**Purpose**: Deduct credits and log transaction. - -**Parameters**: -- `account`: Account instance -- `amount`: Number of credits to deduct -- `operation_type`: Type of operation -- `description`: Description of the transaction -- `metadata`: Optional metadata dict -- `cost_usd`: Optional cost in USD -- `model_used`: Optional AI model used -- `tokens_input`: Optional input tokens -- `tokens_output`: Optional output tokens -- `related_object_type`: Optional related object type -- `related_object_id`: Optional related object ID - -**Returns**: New credit balance - -**Creates**: -- `CreditTransaction` record -- `CreditUsageLog` record - -##### add_credits(account, amount, transaction_type, description, ...) -**Purpose**: Add credits (purchase, subscription, etc.). - -**Parameters**: -- `account`: Account instance -- `amount`: Number of credits to add -- `transaction_type`: Type of transaction -- `description`: Description of the transaction -- `metadata`: Optional metadata dict - -**Returns**: New credit balance - -**Creates**: `CreditTransaction` record - -##### calculate_credits_for_operation(operation_type, **kwargs) -**Purpose**: Calculate credits needed for an operation. - -**Parameters**: -- `operation_type`: Type of operation -- `**kwargs`: Operation-specific parameters - -**Returns**: Number of credits required - -**Raises**: `CreditCalculationError` if calculation fails - -**Credit Costs**: -- `clustering`: 1 credit per 30 keywords -- `ideas`: 1 credit per idea -- `content`: 3 credits per content piece -- `images`: 1 credit per image -- `reparse`: 1 credit per reparse - -### Credit Integration - -**Current Status**: Credits system is implemented but **not currently enforced** in AI functions. - -**Future Implementation**: Credits should be checked and deducted before allowing AI operations. - ---- - -## Plan Limits - -### Planner Limits - -- `max_keywords`: Total keywords allowed (not enforced) -- `max_clusters`: Total clusters allowed (not enforced) -- `max_content_ideas`: Total content ideas allowed (not enforced) -- `daily_cluster_limit`: Max clusters per day (not enforced) -- `monthly_cluster_ai_credits`: AI credits for clustering (not enforced) - -### Writer Limits - -- `daily_content_tasks`: Max content tasks per day (not enforced) -- `daily_ai_requests`: Total AI executions per day (not enforced) -- `monthly_word_count_limit`: Monthly word limit (not enforced) -- `monthly_content_ai_credits`: AI credits for content (not enforced) - -### Image Limits - -- `monthly_image_count`: Max images per month (not enforced) -- `daily_image_generation_limit`: Max images per day (not enforced) -- `monthly_image_ai_credits`: AI credits for images (not enforced) -- `max_images_per_task`: Max images per task (not enforced) -- `image_model_choices`: Allowed image models (not enforced) - -### Feature Flags - -- `features`: JSON array of enabled features (e.g., `['ai_writer', 'image_gen', 'auto_publish']`) -- **Not enforced**: Feature flags are not checked before allowing AI operations - -### Plan Limits Enforcement - -**Current Status**: Plan limits are defined but **not currently enforced** in AI functions. - -**Future Implementation**: Plan limits should be checked before allowing operations. - ---- - -## Multi-Tenancy Architecture - -### Account Isolation - -**Principle**: All data is isolated by account. - -**Implementation**: -1. **Model Level**: All models inherit `AccountBaseModel` (has `account` ForeignKey) -2. **ViewSet Level**: All ViewSets inherit `AccountModelViewSet` (filters by `request.account`) -3. **Middleware Level**: `AccountContextMiddleware` sets `request.account` from JWT token - -**Access Control**: -- Admin/Developer users: Bypass account filtering (see all accounts) -- System account users: Bypass account filtering (see all accounts) -- Regular users: Only see data from their account - -### Site/Sector Hierarchy - -**Structure**: -``` -Account (1) ──< (N) Site -Site (1) ──< (1-5) Sector -Sector (1) ──< (N) Keywords, Clusters, ContentIdeas, Tasks -``` - -**Implementation**: -- Models inherit `SiteSectorBaseModel` (has `site` and `sector` ForeignKeys) -- ViewSets inherit `SiteSectorModelViewSet` (filters by accessible sites) -- User access control via `User.get_accessible_sites()` - -**Site Access Control**: -- System account users: All active sites -- Developers: All active sites -- Owners/Admins: All sites in their account -- Editors/Viewers: Only sites granted via `SiteUserAccess` - -### Data Isolation Flow - -1. **Request Arrives**: JWT token contains account ID -2. **Middleware**: `AccountContextMiddleware` extracts account ID and sets `request.account` -3. **ViewSet**: `AccountModelViewSet.get_queryset()` filters by `request.account` -4. **Database**: All queries automatically filtered by account - -**Override**: Admin/Developer users bypass account filtering - ---- - -## Summary - -The IGNY8 multi-tenancy architecture provides: - -1. **Complete Account Isolation**: All data isolated by account with automatic filtering -2. **Site/Sector Hierarchy**: Hierarchical organization with access control -3. **Role-Based Access Control**: Granular permissions based on user roles -4. **Plan-Based Limits**: Comprehensive plan limits (defined but not enforced) -5. **Credits System**: Credit tracking and deduction system (implemented but not enforced) -6. **Flexible Site Access**: Multiple sites can be active simultaneously -7. **Industry Templates**: Global industry and sector templates - -This architecture ensures data security, scalability, and flexibility while providing a solid foundation for subscription management and billing. - diff --git a/docs/05-AI-FUNCTIONS.md b/docs/05-AI-FUNCTIONS.md new file mode 100644 index 00000000..4add200b --- /dev/null +++ b/docs/05-AI-FUNCTIONS.md @@ -0,0 +1,494 @@ +# IGNY8 AI Functions Documentation + +**Last Updated:** 2025-01-XX +**Purpose:** Complete AI functions documentation covering architecture, all 5 AI functions, execution flow, and configuration. + +--- + +## Table of Contents + +1. [AI Framework Overview](#ai-framework-overview) +2. [Common Architecture](#common-architecture) +3. [AI Function Execution Flow](#ai-function-execution-flow) +4. [AI Functions](#ai-functions) +5. [Progress Tracking](#progress-tracking) +6. [Cost Tracking](#cost-tracking) +7. [Prompt Management](#prompt-management) +8. [Model Configuration](#model-configuration) + +--- + +## AI Framework Overview + +The IGNY8 AI framework provides a unified interface for all AI operations. All AI functions inherit from `BaseAIFunction` and are orchestrated by `AIEngine`, ensuring consistent execution, progress tracking, error handling, and cost tracking. + +### Key Components + +- **BaseAIFunction**: Abstract base class for all AI functions +- **AIEngine**: Central orchestrator managing lifecycle, progress, logging, cost tracking +- **AICore**: Centralized AI request handler for all AI operations +- **PromptRegistry**: Centralized prompt management with hierarchical resolution +- **Function Registry**: Lazy-loaded function registry +- **Progress Tracking**: Real-time progress updates via Celery +- **Cost Tracking**: Automatic cost and token tracking + +### AI Functions + +1. **Auto Cluster Keywords**: Group related keywords into semantic clusters +2. **Generate Ideas**: Generate content ideas from keyword clusters +3. **Generate Content**: Generate blog post and article content +4. **Generate Image Prompts**: Extract image prompts from content +5. **Generate Images**: Generate images using OpenAI DALL-E or Runware + +--- + +## Common Architecture + +### Core Framework Files + +#### Entry Point +**File**: `backend/igny8_core/ai/tasks.py` +**Function**: `run_ai_task` +**Purpose**: Unified Celery task entrypoint for all AI functions +**Parameters**: `function_name` (str), `payload` (dict), `account_id` (int) +**Flow**: Loads function from registry → Creates AIEngine → Executes function + +#### Engine Orchestrator +**File**: `backend/igny8_core/ai/engine.py` +**Class**: `AIEngine` +**Purpose**: Central orchestrator managing lifecycle, progress, logging, cost tracking +**Methods**: +- `execute` - Main execution pipeline (6 phases: INIT, PREP, AI_CALL, PARSE, SAVE, DONE) +- `_handle_error` - Centralized error handling +- `_log_to_database` - Logs to AITaskLog model + +#### Base Function Class +**File**: `backend/igny8_core/ai/base.py` +**Class**: `BaseAIFunction` +**Purpose**: Abstract base class defining interface for all AI functions +**Abstract Methods**: +- `get_name` - Returns function name (e.g., 'auto_cluster') +- `prepare` - Loads and prepares data +- `build_prompt` - Builds AI prompt +- `parse_response` - Parses AI response +- `save_output` - Saves results to database +**Optional Methods**: +- `get_metadata` - Returns display name, description, phases +- `get_max_items` - Returns max items limit (or None) +- `validate` - Validates input payload (default: checks for 'ids') +- `get_model` - Returns model override (default: None, uses account default) + +#### Function Registry +**File**: `backend/igny8_core/ai/registry.py` +**Functions**: +- `register_function` - Registers function class +- `register_lazy_function` - Registers lazy loader +- `get_function` - Gets function class by name (lazy loads if needed) +- `get_function_instance` - Gets function instance by name +- `list_functions` - Lists all registered functions + +#### AI Core Handler +**File**: `backend/igny8_core/ai/ai_core.py` +**Class**: `AICore` +**Purpose**: Centralized AI request handler for all AI operations (text and image generation) +**Methods**: +- `run_ai_request` - Makes API call to OpenAI/Runware for text generation +- `generate_image` - Makes API call to OpenAI DALL-E or Runware for image generation +- `extract_json` - Extracts JSON from response (handles markdown code blocks) + +#### Prompt Registry +**File**: `backend/igny8_core/ai/prompts.py` +**Class**: `PromptRegistry` +**Purpose**: Centralized prompt management with hierarchical resolution +**Method**: `get_prompt` - Gets prompt with resolution order: +1. Task-level prompt_override (if exists) +2. DB prompt for (account, function) +3. Default fallback from DEFAULT_PROMPTS registry +**Prompt Types**: +- `clustering` - For auto_cluster function +- `ideas` - For generate_ideas function +- `content_generation` - For generate_content function +- `image_prompt_extraction` - For extract_image_prompts function +- `image_prompt_template` - Template for formatting image prompts +- `negative_prompt` - Negative prompt for Runware image generation + +#### Model Settings +**File**: `backend/igny8_core/ai/settings.py` +**Constants**: `MODEL_CONFIG` - Model configurations per function (model, max_tokens, temperature, response_format) +**Functions**: +- `get_model_config` - Gets model config for function (reads from IntegrationSettings if account provided) +- `get_model` - Gets model name for function +- `get_max_tokens` - Gets max tokens for function +- `get_temperature` - Gets temperature for function + +--- + +## AI Function Execution Flow + +### Complete Execution Pipeline + +``` +1. API Endpoint (views.py) + ↓ +2. run_ai_task (tasks.py) + - Gets account from account_id + - Gets function instance from registry + - Creates AIEngine + ↓ +3. AIEngine.execute (engine.py) + Phase 1: INIT (0-10%) + - Calls function.validate() + - Updates progress tracker + ↓ + Phase 2: PREP (10-25%) + - Calls function.prepare() + - Calls function.build_prompt() + - Updates progress tracker + ↓ + Phase 3: AI_CALL (25-70%) + - Gets model config from settings + - Calls AICore.run_ai_request() or AICore.generate_image() + - Tracks cost and tokens + - Updates progress tracker + ↓ + Phase 4: PARSE (70-85%) + - Calls function.parse_response() + - Updates progress tracker + ↓ + Phase 5: SAVE (85-98%) + - Calls function.save_output() + - Logs credit usage + - Updates progress tracker + ↓ + Phase 6: DONE (98-100%) + - Logs to AITaskLog + - Returns result +``` + +### Progress Updates + +**Progress Endpoint**: `/api/v1/system/settings/task_progress/{task_id}/` + +**Response Format**: +- `state`: Task state (PENDING, PROGRESS, SUCCESS, FAILURE) +- `meta`: Progress metadata + - `phase`: Current phase (INIT, PREP, AI_CALL, PARSE, SAVE, DONE) + - `percentage`: Progress percentage (0-100) + - `message`: User-friendly message + - `request_steps`: Array of request steps + - `response_steps`: Array of response steps + - `cost`: API cost in USD + - `tokens`: Token count + +--- + +## AI Functions + +### 1. Auto Cluster Keywords + +**Purpose**: Group related keywords into semantic clusters using AI + +**Function Class**: `AutoClusterFunction` +**File**: `backend/igny8_core/ai/functions/auto_cluster.py` + +**API Endpoint**: +- **ViewSet**: `KeywordViewSet` +- **Action**: `auto_cluster` +- **Method**: POST +- **URL Path**: `/v1/planner/keywords/auto_cluster/` +- **Payload**: `ids` (list[int]) - Keyword IDs + +**Function Methods**: +- `get_name()`: Returns `'auto_cluster'` +- `validate(payload, account)`: Validates keyword IDs exist +- `prepare(payload, account)`: Loads keywords from database +- `build_prompt(data, account)`: Builds clustering prompt with keyword data +- `parse_response(response, step_tracker)`: Parses cluster JSON response +- `save_output(parsed, original_data, account, progress_tracker, step_tracker)`: Creates Cluster records and links keywords + +**Input**: List of keyword IDs +**Output**: Cluster records created, keywords linked to clusters + +**Progress Messages**: +- INIT: "Validating keywords" +- PREP: "Preparing keyword clustering" +- AI_CALL: "Analyzing keyword relationships" +- PARSE: "Processing cluster data" +- SAVE: "Creating clusters" + +### 2. Generate Ideas + +**Purpose**: Generate content ideas from keyword clusters + +**Function Class**: `GenerateIdeasFunction` +**File**: `backend/igny8_core/ai/functions/generate_ideas.py` + +**API Endpoint**: +- **ViewSet**: `ClusterViewSet` +- **Action**: `auto_generate_ideas` +- **Method**: POST +- **URL Path**: `/v1/planner/clusters/auto_generate_ideas/` +- **Payload**: `ids` (list[int]) - Cluster IDs + +**Function Methods**: +- `get_name()`: Returns `'generate_ideas'` +- `validate(payload, account)`: Validates cluster IDs exist +- `prepare(payload, account)`: Loads clusters and keywords +- `build_prompt(data, account)`: Builds idea generation prompt with cluster data +- `parse_response(response, step_tracker)`: Parses ideas JSON response +- `save_output(parsed, original_data, account, progress_tracker, step_tracker)`: Creates ContentIdeas records + +**Input**: List of cluster IDs +**Output**: ContentIdeas records created + +**Progress Messages**: +- INIT: "Verifying cluster integrity" +- PREP: "Loading cluster keywords" +- AI_CALL: "Generating ideas with Igny8 Semantic AI" +- PARSE: "{count} high-opportunity idea(s) generated" +- SAVE: "Content Outline for Ideas generated" + +### 3. Generate Content + +**Purpose**: Generate blog post and article content from tasks + +**Function Class**: `GenerateContentFunction` +**File**: `backend/igny8_core/ai/functions/generate_content.py` + +**API Endpoint**: +- **ViewSet**: `TasksViewSet` +- **Action**: `auto_generate_content` +- **Method**: POST +- **URL Path**: `/v1/writer/tasks/auto_generate_content/` +- **Payload**: `ids` (list[int]) - Task IDs (max 50) + +**Function Methods**: +- `get_name()`: Returns `'generate_content'` +- `get_max_items()`: Returns `50` (max tasks per batch) +- `validate(payload, account)`: Validates task IDs exist +- `prepare(payload, account)`: Loads tasks with related data +- `build_prompt(data, account)`: Builds content generation prompt with task data +- `parse_response(response, step_tracker)`: Parses content (JSON or plain text) +- `save_output(parsed, original_data, account, progress_tracker, step_tracker)`: Creates/updates Content records + +**Input**: List of task IDs +**Output**: Content records created/updated with HTML content + +**Progress Messages**: +- INIT: "Initializing content generation" +- PREP: "Loading tasks and building prompts" +- AI_CALL: "Generating content with AI" +- PARSE: "Processing content" +- SAVE: "Saving content" + +### 4. Generate Image Prompts + +**Purpose**: Extract image prompts from content for generating images + +**Function Class**: `GenerateImagePromptsFunction` +**File**: `backend/igny8_core/ai/functions/generate_image_prompts.py` + +**API Endpoint**: +- **ViewSet**: `TasksViewSet` (via content) +- **Action**: `generate_image_prompts` +- **Method**: POST +- **URL Path**: `/v1/writer/content/generate_image_prompts/` +- **Payload**: `ids` (list[int]) - Content IDs + +**Function Methods**: +- `get_name()`: Returns `'generate_image_prompts'` +- `validate(payload, account)`: Validates content IDs exist +- `prepare(payload, account)`: Loads content records +- `build_prompt(data, account)`: Builds prompt extraction prompt with content HTML +- `parse_response(response, step_tracker)`: Parses image prompts JSON +- `save_output(parsed, original_data, account, progress_tracker, step_tracker)`: Updates Images records with prompts + +**Input**: List of content IDs +**Output**: Images records updated with prompts (featured, in-article) + +**Progress Messages**: +- INIT: "Validating content" +- PREP: "Preparing image prompt extraction" +- AI_CALL: "Extracting image prompts" +- PARSE: "Processing image prompts" +- SAVE: "Saving image prompts" + +### 5. Generate Images + +**Purpose**: Generate images using AI (OpenAI DALL-E or Runware) based on prompts + +**Function Class**: `GenerateImagesFunction` +**File**: `backend/igny8_core/ai/functions/generate_images.py` + +**API Endpoint**: +- **ViewSet**: `ImagesViewSet` +- **Action**: `generate_images` +- **Method**: POST +- **URL Path**: `/v1/writer/images/generate_images/` +- **Payload**: `ids` (list[int]) - Image IDs + +**Function Methods**: +- `get_name()`: Returns `'generate_images'` +- `validate(payload, account)`: Validates image IDs exist and have prompts +- `prepare(payload, account)`: Loads images with prompts +- `build_prompt(data, account)`: Formats image prompt with context +- `parse_response(response, step_tracker)`: Parses image URL from API response +- `save_output(parsed, original_data, account, progress_tracker, step_tracker)`: Updates Images records with image URLs + +**Input**: List of image IDs (with prompts) +**Output**: Images records updated with image URLs + +**Image Generation Settings**: +- Provider: 'openai' or 'runware' +- Model: Model name (e.g., 'dall-e-3', 'runware:97@1') +- Image Type: 'realistic', 'artistic', 'cartoon' +- Max In-Article Images: Max images per content +- Image Format: 'webp', 'jpg', 'png' +- Desktop/Mobile: Boolean flags + +**Progress Messages**: +- INIT: "Validating image prompts" +- PREP: "Preparing image generation" +- AI_CALL: "Creating image(s) with AI" +- PARSE: "Processing image response" +- SAVE: "Saving generated image(s)" + +--- + +## Progress Tracking + +### Progress Phases + +All AI functions follow the same 6-phase execution: + +1. **INIT** (0-10%): Validation phase +2. **PREP** (10-25%): Data preparation phase +3. **AI_CALL** (25-70%): AI API call phase +4. **PARSE** (70-85%): Response parsing phase +5. **SAVE** (85-98%): Database save phase +6. **DONE** (98-100%): Completion phase + +### Progress Updates + +**Frontend Polling**: Frontend polls `/api/v1/system/settings/task_progress/{task_id}/` every 1-2 seconds + +**Progress Response**: +- `state`: Task state +- `meta`: Progress metadata + - `phase`: Current phase + - `percentage`: Progress percentage + - `message`: User-friendly message + - `request_steps`: Request steps array + - `response_steps`: Response steps array + - `cost`: API cost + - `tokens`: Token count + +### Step Tracking + +**Request Steps**: Tracked during prompt building and AI call +**Response Steps**: Tracked during response parsing + +**Purpose**: Provides detailed logging for debugging and transparency + +--- + +## Cost Tracking + +### Cost Calculation + +**Text Generation**: +- Cost calculated based on model pricing (input tokens + output tokens) +- Tracked per request +- Stored in `CostTracker` + +**Image Generation**: +- Cost calculated based on provider pricing +- OpenAI DALL-E: Fixed cost per image +- Runware: Variable cost per image +- Tracked per image + +### Cost Storage + +**AITaskLog Model**: +- `cost`: Total cost for task +- `tokens`: Total tokens used + +**CreditUsageLog Model**: +- `cost_usd`: Cost in USD +- `credits_used`: Credits deducted + +--- + +## Prompt Management + +### Prompt Resolution Order + +1. **Task-Level Override**: If task has `prompt_override`, use it +2. **Database Prompt**: If account has custom prompt in database, use it +3. **Default Prompt**: Use default prompt from `DEFAULT_PROMPTS` registry + +### Prompt Customization + +**Per Account**: Custom prompts stored in `AIPrompt` model +**Per Function**: Different prompts for each function type +**Context Variables**: Prompts support context placeholders: +- `[IGNY8_KEYWORDS]` - Keyword list +- `[IGNY8_CLUSTERS]` - Cluster list +- `[IGNY8_CLUSTER_KEYWORDS]` - Cluster keywords +- `[IGNY8_IDEA]` - Idea data +- `[IGNY8_CLUSTER]` - Cluster data + +--- + +## Model Configuration + +### Model Settings + +**Default Models**: +- Clustering: `gpt-4o-mini` +- Ideas: `gpt-4o-mini` +- Content: `gpt-4o` +- Image Prompts: `gpt-4o-mini` +- Images: `dall-e-3` (OpenAI) or `runware:97@1` (Runware) + +### Per-Account Override + +**IntegrationSettings Model**: +- `integration_type`: 'openai' or 'runware' +- `config`: JSONField with model configuration + - `model`: Model name + - `max_tokens`: Max tokens + - `temperature`: Temperature + - `response_format`: Response format + +### Model Configuration + +**File**: `backend/igny8_core/ai/settings.py` + +**MODEL_CONFIG**: Dictionary mapping function names to model configurations + +**Functions**: +- `get_model_config(function_name, account=None)`: Gets model config (checks IntegrationSettings if account provided) +- `get_model(function_name, account=None)`: Gets model name +- `get_max_tokens(function_name, account=None)`: Gets max tokens +- `get_temperature(function_name, account=None)`: Gets temperature + +--- + +## Summary + +The IGNY8 AI framework provides: + +1. **Unified Interface**: All AI functions use the same execution pipeline +2. **Consistent Execution**: All functions follow the same 6-phase flow +3. **Progress Tracking**: Real-time progress updates via Celery +4. **Cost Tracking**: Automatic cost and token tracking +5. **Error Handling**: Centralized error handling in AIEngine +6. **Prompt Management**: Hierarchical prompt resolution +7. **Model Configuration**: Per-account model overrides +8. **Database Logging**: Automatic logging to AITaskLog +9. **Extensibility**: Easy to add new AI functions +10. **Reliability**: Retry logic and error recovery + +This architecture ensures consistency, maintainability, and extensibility while providing a robust foundation for all AI operations in the IGNY8 platform. + diff --git a/docs/06-CHANGELOG.md b/docs/06-CHANGELOG.md new file mode 100644 index 00000000..aa157b50 --- /dev/null +++ b/docs/06-CHANGELOG.md @@ -0,0 +1,173 @@ +# IGNY8 Changelog + +**Last Updated:** 2025-01-XX +**Purpose:** System changelog documenting features, updates, and improvements. + +--- + +## Table of Contents + +1. [2025-01-XX - Documentation Consolidation](#2025-01-xx---documentation-consolidation) +2. [System Features](#system-features) +3. [Planned Features](#planned-features) + +--- + +## 2025-01-XX - Documentation Consolidation + +### Documentation Updates + +- **Consolidated Documentation**: All documentation consolidated into single structure + - `docs/README.md` - Documentation index + - `docs/01-ARCHITECTURE-TECH-STACK.md` - Architecture and tech stack + - `docs/02-APP-ARCHITECTURE.md` - Application architecture with workflows + - `docs/03-FRONTEND.md` - Frontend documentation + - `docs/04-BACKEND.md` - Backend documentation + - `docs/05-AI-FUNCTIONS.md` - AI functions documentation + - `docs/06-CHANGELOG.md` - System changelog + +### Improvements + +- **Complete Workflow Documentation**: All workflows documented from start to finish +- **Feature Completeness**: All features documented without missing any flows +- **No Code Snippets**: Documentation focuses on workflows and features (no code) +- **Accurate State**: Documentation reflects current system state + +--- + +## System Features + +### Implemented Features + +#### Foundation +- ✅ Multi-tenancy system with account isolation +- ✅ Authentication (login/register) with JWT +- ✅ RBAC permissions (Developer, Owner, Admin, Editor, Viewer, System Bot) +- ✅ Account > Site > Sector hierarchy +- ✅ Multiple sites can be active simultaneously +- ✅ Maximum 5 active sectors per site + +#### Planner Module +- ✅ Keywords CRUD operations +- ✅ Keyword import/export (CSV) +- ✅ Keyword filtering and organization +- ✅ AI-powered keyword clustering +- ✅ Clusters CRUD operations +- ✅ Content ideas generation from clusters +- ✅ Content ideas CRUD operations +- ✅ Keyword-to-cluster mapping +- ✅ Cluster metrics and analytics + +#### Writer Module +- ✅ Tasks CRUD operations +- ✅ AI-powered content generation +- ✅ Content editing and review +- ✅ Image prompt extraction +- ✅ AI-powered image generation (OpenAI DALL-E, Runware) +- ✅ Image management +- ✅ WordPress integration (publishing) + +#### Thinker Module +- ✅ AI prompt management +- ✅ Author profile management +- ✅ Content strategy management +- ✅ Image generation testing + +#### System Module +- ✅ Integration settings (OpenAI, Runware) +- ✅ API key configuration +- ✅ Connection testing +- ✅ System status and monitoring + +#### Billing Module +- ✅ Credit balance tracking +- ✅ Credit transactions +- ✅ Usage logging +- ✅ Cost tracking + +#### Frontend +- ✅ Configuration-driven UI system +- ✅ 4 universal templates (Dashboard, Table, Form, System) +- ✅ Complete component library +- ✅ Zustand state management +- ✅ React Router v6 routing +- ✅ Progress tracking for AI tasks +- ✅ AI Request/Response Logs +- ✅ Responsive design + +#### Backend +- ✅ RESTful API with DRF +- ✅ Automatic account isolation +- ✅ Site access control +- ✅ Celery async task processing +- ✅ Progress tracking for Celery tasks +- ✅ Unified AI framework +- ✅ Database logging + +#### AI Functions +- ✅ Auto Cluster Keywords +- ✅ Generate Ideas +- ✅ Generate Content +- ✅ Generate Image Prompts +- ✅ Generate Images +- ✅ Test OpenAI connection +- ✅ Test Runware connection +- ✅ Test image generation + +#### Infrastructure +- ✅ Docker-based containerization +- ✅ Two-stack architecture (infra, app) +- ✅ Caddy reverse proxy +- ✅ PostgreSQL database +- ✅ Redis cache and Celery broker +- ✅ pgAdmin database administration +- ✅ FileBrowser file management + +--- + +## Planned Features + +### In Progress + +- 🔄 Planner Dashboard enhancement with KPIs +- 🔄 WordPress integration (publishing) - partial implementation +- 🔄 Automation & CRON tasks + +### Future + +- 📋 Analytics module enhancements +- 📋 Advanced scheduling features +- 📋 Additional AI model integrations +- 📋 Stripe payment integration +- 📋 Plan limits enforcement +- 📋 Rate limiting +- 📋 Advanced reporting +- 📋 Mobile app support + +--- + +## Version History + +### Current Version + +**Version**: 1.0 +**Date**: 2025-01-XX +**Status**: Production + +### Key Milestones + +- **2025-01-XX**: Documentation consolidation +- **2025-01-XX**: AI framework implementation +- **2025-01-XX**: Multi-tenancy system +- **2025-01-XX**: Frontend configuration system +- **2025-01-XX**: Docker deployment setup + +--- + +## Notes + +- All features are documented in detail in the respective documentation files +- Workflows are complete and accurate +- System is production-ready +- Documentation is maintained and updated regularly + diff --git a/docs/06-INFRASTRUCTURE-DEPLOYMENT.md b/docs/06-INFRASTRUCTURE-DEPLOYMENT.md deleted file mode 100644 index 62153174..00000000 --- a/docs/06-INFRASTRUCTURE-DEPLOYMENT.md +++ /dev/null @@ -1,1420 +0,0 @@ -# IGNY8 Infrastructure & Deployment Documentation - -**Version:** 1.0 -**Last Updated:** 2025-01-XX -**Purpose:** Complete infrastructure and deployment documentation covering Docker setup, containers, networks, volumes, ports, environment configuration, and deployment procedures. - ---- - -## Table of Contents - -1. [Infrastructure Overview](#infrastructure-overview) -2. [Docker Architecture](#docker-architecture) -3. [Infrastructure Stack (igny8-infra)](#infrastructure-stack-igny8-infra) -4. [Application Stack (igny8-app)](#application-stack-igny8-app) -5. [Network Configuration](#network-configuration) -6. [Volume Management](#volume-management) -7. [Port Allocation](#port-allocation) -8. [Environment Variables](#environment-variables) -9. [Docker Images](#docker-images) -10. [Deployment Procedures](#deployment-procedures) -11. [Backup & Restore](#backup--restore) -12. [Monitoring & Health Checks](#monitoring--health-checks) -13. [Troubleshooting](#troubleshooting) - ---- - -## Infrastructure Overview - -The IGNY8 platform runs on a Docker-based infrastructure with two main stacks: - -1. **Infrastructure Stack (`igny8-infra`)**: Shared services used by all applications -2. **Application Stack (`igny8-app`)**: IGNY8-specific application services - -All containers run on a single Docker network (`igny8_net`) for inter-container communication. - -### Infrastructure Components - -**Infrastructure Stack Services**: -- PostgreSQL 15 (Database) -- Redis 7 (Cache & Celery Broker) -- pgAdmin 4 (Database Administration) -- FileBrowser (File Management) -- Caddy (Reverse Proxy & HTTPS) -- setup-helper (Utility Container) - -**Application Stack Services**: -- igny8_backend (Django Backend with Gunicorn) -- igny8_frontend (React Frontend with Vite) -- igny8_celery_worker (Celery Worker) -- igny8_celery_beat (Celery Beat Scheduler) - -**External Services**: -- Portainer (Container Management - runs separately) - ---- - -## Docker Architecture - -### Stack Organization - -**Two-Stack Architecture**: -- **Infrastructure Stack**: Shared services (database, cache, reverse proxy) -- **Application Stack**: Application-specific services (backend, frontend, workers) - -**Benefits**: -- Separation of concerns -- Independent scaling -- Shared infrastructure across multiple apps -- Easier maintenance and updates - -### File Structure - -``` -/data/app/ -├── docker-compose.yml # Infrastructure stack (igny8-infra) -└── igny8/ - └── docker-compose.app.yml # Application stack (igny8-app) -``` - -### Network Architecture - -**Single Network**: `igny8_net` (bridge network) - -**Network Type**: External bridge network (created manually) - -**All Containers**: Connected to `igny8_net` for inter-container communication - ---- - -## Infrastructure Stack (igny8-infra) - -**File**: `/data/app/docker-compose.yml` - -**Stack Name**: `igny8-infra` - -**Purpose**: Shared infrastructure services used by all applications - -### Services - -#### 1. PostgreSQL Database - -**Container Name**: `igny8_postgres` - -**Image**: `postgres:15` - -**Purpose**: Primary database for all applications - -**Configuration**: -- **User**: `igny8` -- **Password**: `igny8pass` -- **Database**: `igny8_db` -- **Port**: Internal only (no external port mapping) -- **Volume**: `pgdata:/var/lib/postgresql/data` - -**Health Check**: -- Command: `pg_isready -U $POSTGRES_USER -d $POSTGRES_DB` -- Interval: 20s -- Timeout: 5s -- Retries: 5 -- Start Period: 15s - -**Network**: `igny8_net` - -**Labels**: -- `com.docker.compose.project=igny8-infra` -- `com.docker.compose.service=postgres` - -#### 2. Redis Cache & Broker - -**Container Name**: `igny8_redis` - -**Image**: `redis:7` - -**Purpose**: -- Celery task broker -- Celery result backend -- Application caching - -**Configuration**: -- **Port**: Internal only (6379, no external mapping) -- **Volume**: `redisdata:/data` -- **Command**: `redis-server --save 60 1 --loglevel warning` - -**Health Check**: -- Command: `redis-cli ping | grep -q PONG` -- Interval: 20s -- Timeout: 3s -- Retries: 5 - -**Network**: `igny8_net` - -**Labels**: -- `com.docker.compose.project=igny8-infra` -- `com.docker.compose.service=redis` - -#### 3. pgAdmin - -**Container Name**: `igny8_pgadmin` - -**Image**: `dpage/pgadmin4` - -**Purpose**: PostgreSQL database administration interface - -**Configuration**: -- **Port**: `5050:80` (external:internal) -- **Email**: `admin@igny8.com` -- **Password**: `admin123` -- **Volume**: `pgadmin_data:/var/lib/pgadmin` - -**Access**: `http://localhost:5050` or `http://:5050` - -**Network**: `igny8_net` - -**Labels**: -- `com.docker.compose.project=igny8-infra` -- `com.docker.compose.service=pgadmin` - -#### 4. FileBrowser - -**Container Name**: `igny8_filebrowser` - -**Image**: `filebrowser/filebrowser:v2.25.0` - -**Purpose**: Web-based file management interface - -**Configuration**: -- **Port**: `8080:80` (external:internal) -- **Timezone**: `Asia/Karachi` -- **Volumes**: - - `/data:/srv` (read-write) - - `/backups:/srv/backups` (read-write) - - `filebrowser_db:/database` (database) - -**Access**: `http://localhost:8080` or `http://:8080` - -**Network**: `igny8_net` - -**Labels**: -- `com.docker.compose.project=igny8-infra` -- `com.docker.compose.service=filebrowser` - -#### 5. Caddy Reverse Proxy - -**Container Name**: `igny8_caddy` - -**Image**: `caddy:latest` - -**Purpose**: -- Reverse proxy for all applications -- HTTPS/SSL termination -- Request routing - -**Configuration**: -- **Ports**: - - `80:80` (HTTP) - - `443:443` (HTTPS) -- **Volumes**: - - `caddy_data:/data` (SSL certificates) - - `caddy_config:/config` (configuration) - - `/var/lib/docker/volumes/portainer_data/_data/caddy/Caddyfile:/etc/caddy/Caddyfile` (routing config) - -**Caddyfile Location**: `/var/lib/docker/volumes/portainer_data/_data/caddy/Caddyfile` - -**Network**: `igny8_net` - -**Labels**: -- `com.docker.compose.project=igny8-infra` -- `com.docker.compose.service=caddy` - -#### 6. Setup Helper - -**Container Name**: `setup-helper` - -**Image**: `alpine:3.20` - -**Purpose**: Utility container for setup and maintenance tasks - -**Configuration**: -- **Command**: `sh -c "sleep infinity"` (keeps container running) -- **Volumes**: - - `/data/backups:/backups:rw` (backup directory) - - `/scripts:/scripts:ro` (utility scripts) - -**Network**: `igny8_net` - -**Labels**: -- `com.docker.compose.project=igny8-infra` -- `com.docker.compose.service=setup-helper` - -### Volumes - -**Infrastructure Stack Volumes**: -- `pgdata`: PostgreSQL database data -- `redisdata`: Redis data -- `pgadmin_data`: pgAdmin configuration -- `filebrowser_db`: FileBrowser database -- `caddy_data`: Caddy SSL certificates and data -- `caddy_config`: Caddy configuration - -### Network - -**Network Name**: `igny8_net` - -**Type**: External bridge network - -**Creation**: Must be created manually before starting stacks - -**Command**: `docker network create igny8_net --driver bridge` - ---- - -## Application Stack (igny8-app) - -**File**: `/data/app/igny8/docker-compose.app.yml` - -**Stack Name**: `igny8-app` - -**Purpose**: IGNY8 application-specific services - -### Services - -#### 1. Backend Service - -**Container Name**: `igny8_backend` - -**Image**: `igny8-backend:latest` - -**Purpose**: Django REST API backend - -**Configuration**: -- **Port**: `8011:8010` (external:internal) -- **Working Directory**: `/app` -- **Command**: `gunicorn igny8_core.wsgi:application --bind 0.0.0.0:8010 --workers 4 --timeout 120` - -**Environment Variables**: -- `DB_HOST`: `postgres` -- `DB_NAME`: `igny8_db` -- `DB_USER`: `igny8` -- `DB_PASSWORD`: `igny8pass` -- `REDIS_HOST`: `redis` -- `REDIS_PORT`: `6379` -- `USE_SECURE_COOKIES`: `True` -- `USE_SECURE_PROXY_HEADER`: `True` -- `DEBUG`: `False` -- `SECRET_KEY`: (should be set via environment variable in production) - -**Volumes**: -- `/data/app/igny8/backend:/app:rw` (application code) -- `/data/app/igny8:/data/app/igny8:ro` (read-only access to project root) -- `/var/run/docker.sock:/var/run/docker.sock:ro` (Docker socket for container management) -- `/data/app/logs:/app/logs:rw` (application logs) - -**Health Check**: -- Command: `python -c "import urllib.request; urllib.request.urlopen('http://localhost:8010/api/v1/system/status/').read()"` -- Interval: 30s -- Timeout: 10s -- Retries: 3 -- Start Period: 40s - -**Network**: `igny8_net` - -**Dependencies**: -- Requires `postgres` and `redis` from infrastructure stack (external services) - -**Labels**: -- `com.docker.compose.project=igny8-app` -- `com.docker.compose.service=igny8_backend` - -#### 2. Frontend Service - -**Container Name**: `igny8_frontend` - -**Image**: `igny8-frontend-dev:latest` - -**Purpose**: React frontend with Vite dev server - -**Configuration**: -- **Port**: `8021:5173` (external:internal) -- **Command**: `npm run dev -- --host 0.0.0.0 --port 5173` - -**Environment Variables**: -- `VITE_BACKEND_URL`: `https://api.igny8.com/api` - -**Volumes**: -- `/data/app/igny8/frontend:/app:rw` (application code) - -**Dependencies**: -- `igny8_backend` (waits for backend to be healthy) - -**Network**: `igny8_net` - -**Labels**: -- `com.docker.compose.project=igny8-app` -- `com.docker.compose.service=igny8_frontend` - -#### 3. Celery Worker - -**Container Name**: `igny8_celery_worker` - -**Image**: `igny8-backend:latest` (same as backend) - -**Purpose**: Celery worker for asynchronous task processing - -**Configuration**: -- **Working Directory**: `/app` -- **Command**: `celery -A igny8_core worker --loglevel=info --concurrency=4` - -**Environment Variables**: -- `DB_HOST`: `postgres` -- `DB_NAME`: `igny8_db` -- `DB_USER`: `igny8` -- `DB_PASSWORD`: `igny8pass` -- `REDIS_HOST`: `redis` -- `REDIS_PORT`: `6379` -- `DEBUG`: `False` - -**Volumes**: -- `/data/app/igny8/backend:/app:rw` (application code) -- `/data/app/igny8:/data/app/igny8:ro` (read-only access to project root) -- `/data/app/logs:/app/logs:rw` (application logs) - -**Network**: `igny8_net` - -**Dependencies**: -- Requires `postgres` and `redis` from infrastructure stack (external services) - -**Labels**: -- `com.docker.compose.project=igny8-app` -- `com.docker.compose.service=igny8_celery_worker` - -#### 4. Celery Beat - -**Container Name**: `igny8_celery_beat` - -**Image**: `igny8-backend:latest` (same as backend) - -**Purpose**: Celery beat scheduler for periodic tasks - -**Configuration**: -- **Working Directory**: `/app` -- **Command**: `celery -A igny8_core beat --loglevel=info` - -**Environment Variables**: -- `DB_HOST`: `postgres` -- `DB_NAME`: `igny8_db` -- `DB_USER`: `igny8` -- `DB_PASSWORD`: `igny8pass` -- `REDIS_HOST`: `redis` -- `REDIS_PORT`: `6379` -- `DEBUG`: `False` - -**Volumes**: -- `/data/app/igny8/backend:/app:rw` (application code) -- `/data/app/igny8:/data/app/igny8:ro` (read-only access to project root) -- `/data/app/logs:/app/logs:rw` (application logs) - -**Network**: `igny8_net` - -**Dependencies**: -- Requires `postgres` and `redis` from infrastructure stack (external services) - -**Labels**: -- `com.docker.compose.project=igny8-app` -- `com.docker.compose.service=igny8_celery_beat` - -### Network - -**Network Name**: `igny8_net` - -**Type**: External (uses network from infrastructure stack) - -**Configuration**: `external: true` - ---- - -## Network Configuration - -### Network Details - -**Network Name**: `igny8_net` - -**Type**: Bridge network - -**Driver**: `bridge` - -**Scope**: External (shared across stacks) - -### Network Creation - -**Manual Creation Required**: - -```bash -docker network create igny8_net --driver bridge -``` - -**Verification**: - -```bash -docker network inspect igny8_net -``` - -### Container IP Addresses - -**Infrastructure Stack** (172.18.0.x subnet): -- `igny8_postgres`: 172.18.0.6 -- `igny8_redis`: 172.18.0.2 -- `igny8_pgadmin`: 172.18.0.4 -- `igny8_filebrowser`: 172.18.0.8 -- `igny8_caddy`: 172.18.0.7 -- `setup-helper`: 172.18.0.5 - -**Application Stack** (172.18.0.x subnet): -- `igny8_backend`: 172.18.0.3 -- `igny8_frontend`: 172.18.0.11 -- `igny8_celery_worker`: 172.18.0.9 -- `igny8_celery_beat`: 172.18.0.10 - -**Note**: IP addresses are dynamically assigned by Docker and may vary. - -### Inter-Container Communication - -**Service Discovery**: Containers can communicate using service names as hostnames - -**Examples**: -- Backend → PostgreSQL: `postgres:5432` -- Backend → Redis: `redis:6379` -- Frontend → Backend: `igny8_backend:8010` (internal) or `https://api.igny8.com/api` (external via Caddy) - ---- - -## Volume Management - -### Infrastructure Stack Volumes - -#### pgdata -**Purpose**: PostgreSQL database data - -**Location**: `/var/lib/docker/volumes/igny8-infra_pgdata/_data` - -**Backup**: Critical - contains all database data - -#### redisdata -**Purpose**: Redis data persistence - -**Location**: `/var/lib/docker/volumes/igny8-infra_redisdata/_data` - -**Backup**: Optional - cache data can be regenerated - -#### pgadmin_data -**Purpose**: pgAdmin configuration and settings - -**Location**: `/var/lib/docker/volumes/igny8-infra_pgadmin_data/_data` - -**Backup**: Optional - configuration can be recreated - -#### filebrowser_db -**Purpose**: FileBrowser database - -**Location**: `/var/lib/docker/volumes/igny8-infra_filebrowser_db/_data` - -**Backup**: Optional - file browser state - -#### caddy_data -**Purpose**: Caddy SSL certificates and data - -**Location**: `/var/lib/docker/volumes/igny8-infra_caddy_data/_data` - -**Backup**: Important - contains SSL certificates - -#### caddy_config -**Purpose**: Caddy configuration - -**Location**: `/var/lib/docker/volumes/igny8-infra_caddy_config/_data` - -**Backup**: Important - contains Caddy configuration - -### Host Volume Mounts - -#### Application Code -**Path**: `/data/app/igny8/backend:/app:rw` - -**Purpose**: Backend application code (read-write for development) - -#### Application Root -**Path**: `/data/app/igny8:/data/app/igny8:ro` - -**Purpose**: Read-only access to project root - -#### Logs -**Path**: `/data/app/logs:/app/logs:rw` - -**Purpose**: Application logs directory - -#### Docker Socket -**Path**: `/var/run/docker.sock:/var/run/docker.sock:ro` - -**Purpose**: Read-only access to Docker socket for container management - -#### FileBrowser Data -**Path**: `/data:/srv` (read-write) - -**Purpose**: FileBrowser access to host filesystem - -#### FileBrowser Backups -**Path**: `/backups:/srv/backups` (read-write) - -**Purpose**: FileBrowser access to backup directory - -#### Setup Helper Scripts -**Path**: `/scripts:/scripts:ro` - -**Purpose**: Read-only access to utility scripts - -#### Setup Helper Backups -**Path**: `/data/backups:/backups:rw` - -**Purpose**: Backup directory access - -### Volume Backup - -**Docker Volumes**: -```bash -# Backup all Docker volumes -tar -czf docker-volumes-backup-$(date +%Y%m%d).tar.gz \ - -C /var/lib/docker volumes/ -``` - -**Host Mounts**: -```bash -# Backup application code -tar -czf app-backup-$(date +%Y%m%d).tar.gz -C /data app/ - -# Backup logs -tar -czf logs-backup-$(date +%Y%m%d).tar.gz -C /data logs/ -``` - ---- - -## Port Allocation - -### External Ports (Host → Container) - -**Infrastructure Stack**: -- `5050` → pgAdmin (80) -- `8080` → FileBrowser (80) -- `80` → Caddy (80) - HTTP -- `443` → Caddy (443) - HTTPS - -**Application Stack**: -- `8011` → Backend (8010) -- `8021` → Frontend (5173) - -**Portainer** (External): -- `8000` → Portainer (8000) - HTTP -- `9443` → Portainer (9443) - HTTPS - -### Internal Ports (Container → Container) - -**Standardized Internal Ports**: -- Backend: `8010` (Gunicorn) -- Frontend Dev: `5173` (Vite dev server) -- Frontend Prod: `8020` (Caddy) -- PostgreSQL: `5432` -- Redis: `6379` -- pgAdmin: `80` -- FileBrowser: `80` -- Caddy: `80` (HTTP), `443` (HTTPS) - -### Port Allocation Strategy - -**Pattern**: -- Backend External: `XX11` (e.g., 8011) -- Frontend External: `XX21` (e.g., 8021) - -**IGNY8 Port Range**: `8000-8099` -- Backend: `8011` -- Frontend: `8021` - ---- - -## Environment Variables - -### Backend Environment Variables - -**Database Configuration**: -- `DB_HOST`: PostgreSQL hostname (default: `postgres`) -- `DB_NAME`: Database name (default: `igny8_db`) -- `DB_USER`: Database user (default: `igny8`) -- `DB_PASSWORD`: Database password (default: `igny8pass`) -- `DB_PORT`: Database port (default: `5432`) - -**Redis Configuration**: -- `REDIS_HOST`: Redis hostname (default: `redis`) -- `REDIS_PORT`: Redis port (default: `6379`) - -**Django Configuration**: -- `SECRET_KEY`: Django secret key (REQUIRED in production) -- `DEBUG`: Debug mode (default: `False`) -- `ALLOWED_HOSTS`: Comma-separated list of allowed hosts -- `DJANGO_SETTINGS_MODULE`: Django settings module (default: `igny8_core.settings`) - -**Security Settings**: -- `USE_SECURE_COOKIES`: Enable secure cookies (default: `True` in production) -- `USE_SECURE_PROXY_HEADER`: Enable secure proxy header (default: `True` in production) - -**JWT Configuration**: -- `JWT_SECRET_KEY`: JWT secret key (defaults to `SECRET_KEY`) - -**Celery Configuration**: -- `CELERY_BROKER_URL`: Celery broker URL (default: `redis://redis:6379/0`) -- `CELERY_RESULT_BACKEND`: Celery result backend (default: `redis://redis:6379/0`) - -### Frontend Environment Variables - -**API Configuration**: -- `VITE_BACKEND_URL`: Backend API URL (default: `https://api.igny8.com/api`) - -**Build Configuration**: -- `VITE_APP_NAME`: Application name - -### Infrastructure Environment Variables - -**PostgreSQL**: -- `POSTGRES_USER`: Database user (default: `igny8`) -- `POSTGRES_PASSWORD`: Database password (default: `igny8pass`) -- `POSTGRES_DB`: Database name (default: `igny8_db`) - -**pgAdmin**: -- `PGADMIN_DEFAULT_EMAIL`: Admin email (default: `admin@igny8.com`) -- `PGADMIN_DEFAULT_PASSWORD`: Admin password (default: `admin123`) - -**FileBrowser**: -- `TZ`: Timezone (default: `Asia/Karachi`) - ---- - -## Docker Images - -### Image Building - -**Backend Image**: -```bash -cd /data/app/igny8/backend -docker build -t igny8-backend:latest -f Dockerfile . -``` - -**Frontend Dev Image**: -```bash -cd /data/app/igny8/frontend -docker build -t igny8-frontend-dev:latest -f Dockerfile.dev . -``` - -**Frontend Production Image**: -```bash -cd /data/app/igny8/frontend -docker build -t igny8-frontend:latest -f Dockerfile . -``` - -### Image Details - -#### igny8-backend:latest -**Base Image**: `python:3.11-slim` - -**Dependencies**: -- System: `gcc`, `libpq-dev`, `curl`, `git` -- Python: See `requirements.txt` - -**Exposed Port**: `8010` - -**Default Command**: `gunicorn igny8_core.wsgi:application --bind 0.0.0.0:8010` - -**Build Context**: `/data/app/igny8/backend` - -#### igny8-frontend-dev:latest -**Base Image**: `node:18-alpine` - -**Dependencies**: See `package.json` - -**Exposed Port**: `5173` - -**Default Command**: `npm run dev -- --host 0.0.0.0 --port 5173` - -**Build Context**: `/data/app/igny8/frontend` - -#### igny8-frontend:latest -**Base Image**: Multi-stage build -- Stage 1: `node:18-alpine` (builder) -- Stage 2: `caddy:latest` (server) - -**Exposed Port**: `8020` - -**Default Command**: `caddy run --config /etc/caddy/Caddyfile` - -**Build Context**: `/data/app/igny8/frontend` - -### External Images - -**Infrastructure Images**: -- `postgres:15` - PostgreSQL database -- `redis:7` - Redis cache and broker -- `dpage/pgadmin4` - pgAdmin interface -- `filebrowser/filebrowser:v2.25.0` - FileBrowser -- `caddy:latest` - Caddy reverse proxy -- `alpine:3.20` - Setup helper utility - -**Management Images**: -- `portainer/portainer-ce:latest` - Portainer container management - ---- - -## Deployment Procedures - -### Initial Setup - -#### 1. Prerequisites - -**System Requirements**: -- Docker 20.10+ -- Docker Compose 2.0+ -- Sufficient disk space (minimum 20GB) -- Network access for pulling images - -**Directory Structure**: -```bash -/data/app/ -├── docker-compose.yml # Infrastructure stack -└── igny8/ - ├── docker-compose.app.yml # Application stack - ├── backend/ - └── frontend/ -``` - -#### 2. Create Docker Network - -```bash -docker network create igny8_net --driver bridge -``` - -#### 3. Build Docker Images - -**Backend Image**: -```bash -cd /data/app/igny8/backend -docker build -t igny8-backend:latest -f Dockerfile . -``` - -**Frontend Dev Image**: -```bash -cd /data/app/igny8/frontend -docker build -t igny8-frontend-dev:latest -f Dockerfile.dev . -``` - -#### 4. Start Infrastructure Stack - -```bash -cd /data/app -docker compose -f docker-compose.yml -p igny8-infra up -d -``` - -**Verify**: -```bash -docker ps --filter "label=com.docker.compose.project=igny8-infra" -``` - -#### 5. Start Application Stack - -```bash -cd /data/app/igny8 -docker compose -f docker-compose.app.yml -p igny8-app up -d -``` - -**Verify**: -```bash -docker ps --filter "label=com.docker.compose.project=igny8-app" -``` - -### Database Initialization - -#### 1. Run Migrations - -```bash -docker exec -it igny8_backend python manage.py migrate -``` - -#### 2. Create Superuser - -```bash -docker exec -it igny8_backend python manage.py createsuperuser -``` - -#### 3. Load Initial Data (Optional) - -```bash -docker exec -it igny8_backend python manage.py loaddata initial_data.json -``` - -### Caddy Configuration - -**Caddyfile Location**: `/var/lib/docker/volumes/portainer_data/_data/caddy/Caddyfile` - -**Configuration**: Managed via Portainer or manually - -**Example Configuration**: -``` -api.igny8.com { - reverse_proxy igny8_backend:8010 -} - -app.igny8.com { - reverse_proxy igny8_frontend:5173 -} -``` - -**Reload Caddy**: -```bash -docker exec igny8_caddy caddy reload --config /etc/caddy/Caddyfile -``` - -### Production Deployment - -#### 1. Set Environment Variables - -**Create `.env` file or set via docker-compose**: -```bash -SECRET_KEY= -JWT_SECRET_KEY= -DEBUG=False -USE_SECURE_COOKIES=True -USE_SECURE_PROXY_HEADER=True -``` - -#### 2. Update Caddyfile - -Configure domain names and SSL certificates in Caddyfile - -#### 3. Build Production Images - -**Backend** (already built): -```bash -cd /data/app/igny8/backend -docker build -t igny8-backend:latest -f Dockerfile . -``` - -**Frontend Production**: -```bash -cd /data/app/igny8/frontend -docker build -t igny8-frontend:latest -f Dockerfile . -``` - -#### 4. Update docker-compose.app.yml - -Change frontend image from `igny8-frontend-dev:latest` to `igny8-frontend:latest` - -#### 5. Restart Stacks - -```bash -# Restart application stack -cd /data/app/igny8 -docker compose -f docker-compose.app.yml -p igny8-app restart -``` - -### Updates and Maintenance - -#### Update Application Code - -**Backend**: -```bash -cd /data/app/igny8/backend -git pull # or copy new files -docker compose -f docker-compose.app.yml -p igny8-app restart igny8_backend -``` - -**Frontend**: -```bash -cd /data/app/igny8/frontend -git pull # or copy new files -docker compose -f docker-compose.app.yml -p igny8-app restart igny8_frontend -``` - -#### Rebuild Images - -**Backend**: -```bash -cd /data/app/igny8/backend -docker build -t igny8-backend:latest -f Dockerfile . -docker compose -f docker-compose.app.yml -p igny8-app up -d --force-recreate igny8_backend igny8_celery_worker igny8_celery_beat -``` - -**Frontend**: -```bash -cd /data/app/igny8/frontend -docker build -t igny8-frontend-dev:latest -f Dockerfile.dev . -docker compose -f docker-compose.app.yml -p igny8-app up -d --force-recreate igny8_frontend -``` - -#### Database Migrations - -```bash -docker exec -it igny8_backend python manage.py migrate -``` - -#### Collect Static Files - -```bash -docker exec -it igny8_backend python manage.py collectstatic --noinput -``` - ---- - -## Backup & Restore - -### Backup Procedures - -#### 1. Database Backup - -**PostgreSQL Dump**: -```bash -docker exec igny8_postgres pg_dump -U igny8 igny8_db > backup-$(date +%Y%m%d).sql -``` - -**With Compression**: -```bash -docker exec igny8_postgres pg_dump -U igny8 igny8_db | gzip > backup-$(date +%Y%m%d).sql.gz -``` - -#### 2. Docker Volumes Backup - -**All Volumes**: -```bash -tar -czf docker-volumes-backup-$(date +%Y%m%d).tar.gz \ - -C /var/lib/docker volumes/ -``` - -**Individual Volumes**: -```bash -# PostgreSQL data -tar -czf pgdata-backup-$(date +%Y%m%d).tar.gz \ - -C /var/lib/docker/volumes/igny8-infra_pgdata/_data . - -# Redis data -tar -czf redisdata-backup-$(date +%Y%m%d).tar.gz \ - -C /var/lib/docker/volumes/igny8-infra_redisdata/_data . - -# Caddy data (SSL certificates) -tar -czf caddy-data-backup-$(date +%Y%m%d).tar.gz \ - -C /var/lib/docker/volumes/igny8-infra_caddy_data/_data . -``` - -#### 3. Application Code Backup - -```bash -tar -czf app-backup-$(date +%Y%m%d).tar.gz -C /data app/ -``` - -#### 4. Logs Backup - -```bash -tar -czf logs-backup-$(date +%Y%m%d).tar.gz -C /data logs/ -``` - -#### 5. Complete System Backup - -```bash -#!/bin/bash -BACKUP_DIR="/backups/complete-backup-$(date +%Y%m%d)" -mkdir -p "$BACKUP_DIR" - -# Database -docker exec igny8_postgres pg_dump -U igny8 igny8_db | gzip > "$BACKUP_DIR/database.sql.gz" - -# Docker volumes -tar -czf "$BACKUP_DIR/docker-volumes.tar.gz" -C /var/lib/docker volumes/ - -# Application code -tar -czf "$BACKUP_DIR/app.tar.gz" -C /data app/ - -# Logs -tar -czf "$BACKUP_DIR/logs.tar.gz" -C /data logs/ - -# Caddyfile -cp /var/lib/docker/volumes/portainer_data/_data/caddy/Caddyfile "$BACKUP_DIR/Caddyfile" -``` - -### Restore Procedures - -#### 1. Database Restore - -**From SQL Dump**: -```bash -docker exec -i igny8_postgres psql -U igny8 igny8_db < backup-YYYYMMDD.sql -``` - -**From Compressed Dump**: -```bash -gunzip -c backup-YYYYMMDD.sql.gz | docker exec -i igny8_postgres psql -U igny8 igny8_db -``` - -#### 2. Docker Volumes Restore - -**Stop Services**: -```bash -docker compose -f docker-compose.app.yml -p igny8-app down -docker compose -f docker-compose.yml -p igny8-infra down -``` - -**Restore Volumes**: -```bash -# Extract volume backups -tar -xzf docker-volumes-backup-YYYYMMDD.tar.gz -C /var/lib/docker/ - -# Or restore individual volumes -tar -xzf pgdata-backup-YYYYMMDD.tar.gz -C /var/lib/docker/volumes/igny8-infra_pgdata/_data/ -``` - -**Start Services**: -```bash -docker compose -f docker-compose.yml -p igny8-infra up -d -docker compose -f docker-compose.app.yml -p igny8-app up -d -``` - -#### 3. Application Code Restore - -```bash -tar -xzf app-backup-YYYYMMDD.tar.gz -C /data/ -``` - -#### 4. Complete System Restore - -```bash -#!/bin/bash -BACKUP_DIR="/backups/complete-backup-YYYYMMDD" - -# Restore database -gunzip -c "$BACKUP_DIR/database.sql.gz" | docker exec -i igny8_postgres psql -U igny8 igny8_db - -# Restore volumes -tar -xzf "$BACKUP_DIR/docker-volumes.tar.gz" -C /var/lib/docker/ - -# Restore application code -tar -xzf "$BACKUP_DIR/app.tar.gz" -C /data/ - -# Restore Caddyfile -cp "$BACKUP_DIR/Caddyfile" /var/lib/docker/volumes/portainer_data/_data/caddy/Caddyfile - -# Restart services -docker compose -f docker-compose.yml -p igny8-infra restart caddy -docker compose -f docker-compose.app.yml -p igny8-app restart -``` - ---- - -## Monitoring & Health Checks - -### Health Check Endpoints - -#### Backend Health Check - -**Endpoint**: `http://localhost:8011/api/v1/system/status/` - -**Method**: `GET` - -**Expected Response**: HTTP 200 - -**Implementation**: Django view that checks database connectivity - -#### Frontend Health Check - -**Endpoint**: `http://localhost:8021/` - -**Method**: `GET` - -**Expected Response**: HTTP 200 (HTML page) - -### Container Health Checks - -#### PostgreSQL - -**Check**: `pg_isready -U igny8 -d igny8_db` - -**Interval**: 20s - -**Timeout**: 5s - -**Retries**: 5 - -#### Redis - -**Check**: `redis-cli ping | grep -q PONG` - -**Interval**: 20s - -**Timeout**: 3s - -**Retries**: 5 - -#### Backend - -**Check**: `python -c "import urllib.request; urllib.request.urlopen('http://localhost:8010/api/v1/system/status/').read()"` - -**Interval**: 30s - -**Timeout**: 10s - -**Retries**: 3 - -**Start Period**: 40s - -### Monitoring Script - -**File**: `/data/app/igny8/check-status.sh` - -**Purpose**: Quick status check for all stacks and containers - -**Usage**: -```bash -cd /data/app/igny8 -./check-status.sh -``` - -**Output**: -- Container status for both stacks -- Network connectivity -- Service health checks (Backend API, Frontend, PostgreSQL, Redis) -- All IGNY8 containers list - -### Log Monitoring - -#### View Container Logs - -**Backend**: -```bash -docker logs -f igny8_backend -``` - -**Frontend**: -```bash -docker logs -f igny8_frontend -``` - -**Celery Worker**: -```bash -docker logs -f igny8_celery_worker -``` - -**Celery Beat**: -```bash -docker logs -f igny8_celery_beat -``` - -**PostgreSQL**: -```bash -docker logs -f igny8_postgres -``` - -**Redis**: -```bash -docker logs -f igny8_redis -``` - -#### Application Logs - -**Location**: `/data/app/logs/` - -**Backend Logs**: `/data/app/logs/backend/` - -**Frontend Logs**: `/data/app/logs/frontend/` - ---- - -## Troubleshooting - -### Common Issues - -#### 1. Network Not Found - -**Error**: `network igny8_net not found` - -**Solution**: -```bash -docker network create igny8_net --driver bridge -``` - -#### 2. Database Connection Failed - -**Error**: `could not connect to server: Connection refused` - -**Check**: -```bash -# Verify PostgreSQL is running -docker ps | grep igny8_postgres - -# Check PostgreSQL logs -docker logs igny8_postgres - -# Test connection -docker exec igny8_postgres pg_isready -U igny8 -``` - -**Solution**: Ensure infrastructure stack is running before application stack - -#### 3. Redis Connection Failed - -**Error**: `Error connecting to Redis` - -**Check**: -```bash -# Verify Redis is running -docker ps | grep igny8_redis - -# Check Redis logs -docker logs igny8_redis - -# Test connection -docker exec igny8_redis redis-cli ping -``` - -**Solution**: Ensure infrastructure stack is running before application stack - -#### 4. Port Already in Use - -**Error**: `Bind for 0.0.0.0:8011 failed: port is already allocated` - -**Check**: -```bash -# Find process using port -sudo lsof -i :8011 -# or -sudo netstat -tulpn | grep 8011 -``` - -**Solution**: Stop conflicting service or change port in docker-compose - -#### 5. Container Health Check Failing - -**Error**: Container shows as unhealthy - -**Check**: -```bash -# View container health status -docker inspect igny8_backend | grep -A 10 Health - -# View container logs -docker logs igny8_backend - -# Test health check manually -docker exec igny8_backend python -c "import urllib.request; urllib.request.urlopen('http://localhost:8010/api/v1/system/status/').read()" -``` - -**Solution**: Check application logs, verify dependencies are running - -#### 6. Celery Tasks Not Processing - -**Check**: -```bash -# Verify Celery worker is running -docker ps | grep igny8_celery_worker - -# Check Celery worker logs -docker logs igny8_celery_worker - -# Verify Redis connection -docker exec igny8_celery_worker celery -A igny8_core inspect active -``` - -**Solution**: Restart Celery worker, verify Redis connectivity - -#### 7. Static Files Not Loading - -**Error**: 404 for static files - -**Solution**: -```bash -# Collect static files -docker exec igny8_backend python manage.py collectstatic --noinput - -# Restart backend -docker compose -f docker-compose.app.yml -p igny8-app restart igny8_backend -``` - -#### 8. CORS Errors - -**Error**: CORS policy blocked request - -**Check**: Verify `CORS_ALLOWED_ORIGINS` in Django settings - -**Solution**: Add frontend domain to `CORS_ALLOWED_ORIGINS` in `settings.py` - -### Debugging Commands - -#### Container Inspection - -```bash -# Inspect container configuration -docker inspect igny8_backend - -# View container environment variables -docker exec igny8_backend env - -# View container network configuration -docker network inspect igny8_net -``` - -#### Database Debugging - -```bash -# Connect to PostgreSQL -docker exec -it igny8_postgres psql -U igny8 igny8_db - -# List databases -docker exec igny8_postgres psql -U igny8 -l - -# Check database size -docker exec igny8_postgres psql -U igny8 -c "SELECT pg_size_pretty(pg_database_size('igny8_db'));" -``` - -#### Redis Debugging - -```bash -# Connect to Redis CLI -docker exec -it igny8_redis redis-cli - -# Check Redis info -docker exec igny8_redis redis-cli INFO - -# List all keys -docker exec igny8_redis redis-cli KEYS '*' -``` - -#### Application Debugging - -```bash -# Django shell -docker exec -it igny8_backend python manage.py shell - -# Check Django settings -docker exec igny8_backend python manage.py diffsettings - -# View Django URLs -docker exec igny8_backend python manage.py show_urls -``` - ---- - -## Summary - -The IGNY8 infrastructure is built on: - -1. **Two-Stack Architecture**: Infrastructure stack (shared services) and Application stack (app-specific services) -2. **Single Network**: All containers on `igny8_net` bridge network -3. **Docker Compose**: Separate compose files for infrastructure and application -4. **Volume Management**: Docker volumes for persistent data, host mounts for code -5. **Health Checks**: Built-in health checks for critical services -6. **Port Allocation**: Standardized port allocation (8011 backend, 8021 frontend) -7. **Environment Variables**: Configuration via environment variables -8. **Backup & Restore**: Comprehensive backup procedures for database, volumes, and code -9. **Monitoring**: Health checks, logging, and status scripts -10. **Troubleshooting**: Common issues and debugging commands - -This infrastructure ensures reliability, scalability, and maintainability while providing a solid foundation for the IGNY8 application. - diff --git a/docs/IMAGE_GENERATION_CHANGELOG.md b/docs/IMAGE_GENERATION_CHANGELOG.md deleted file mode 100644 index ff58a29c..00000000 --- a/docs/IMAGE_GENERATION_CHANGELOG.md +++ /dev/null @@ -1,501 +0,0 @@ -# Image Generation Implementation Changelog - -## Stage 1: Pre-Queue Modal Display - Completed ✅ - -**Date:** 2025-01-XX -**Status:** Completed -**Goal:** Open modal immediately showing all progress bars before any API calls - ---- - -## Changes Made - -### 1. Frontend Components - -#### Created: `frontend/src/components/common/ImageQueueModal.tsx` -- **Purpose:** Display image generation queue with individual progress bars -- **Features:** - - Shows all images that will be generated with individual progress bars - - Displays queue number, label, content title, status, and progress percentage - - Includes thumbnail placeholder for generated images - - Supports 4 states: `pending`, `processing`, `completed`, `failed` - - Styled similar to WP plugin's image-queue-processor.js modal - - Responsive design with dark mode support - - Prevents closing while processing - - Shows completion summary in footer - -**Key Props:** -- `isOpen`: Boolean to control modal visibility -- `onClose`: Callback when modal is closed -- `queue`: Array of `ImageQueueItem` objects -- `totalImages`: Total number of images in queue -- `onUpdateQueue`: Optional callback to update queue state - -**ImageQueueItem Interface:** -```typescript -{ - imageId: number | null; - index: number; - label: string; - type: 'featured' | 'in_article'; - position?: number; - contentTitle: string; - status: 'pending' | 'processing' | 'completed' | 'failed'; - progress: number; // 0-100 - imageUrl: string | null; - error: string | null; -} -``` - -#### Updated: `frontend/src/pages/Writer/Images.tsx` -- **Added Imports:** - - `fetchImageGenerationSettings` from `../../services/api` - - `ImageQueueModal` and `ImageQueueItem` from `../../components/common/ImageQueueModal` - -- **Added State:** - ```typescript - const [isQueueModalOpen, setIsQueueModalOpen] = useState(false); - const [imageQueue, setImageQueue] = useState([]); - const [currentContentId, setCurrentContentId] = useState(null); - ``` - -- **Added Function: `buildImageQueue()`** - - Builds image queue structure from content images - - Includes featured image (if pending and has prompt) - - Includes in-article images (up to `max_in_article_images` from settings) - - Sorts in-article images by position - - Returns array of `ImageQueueItem` objects - -- **Updated Function: `handleGenerateImages()`** - - **Stage 1 Implementation:** - 1. Fetches image generation settings to get `max_in_article_images` - 2. Builds image queue using `buildImageQueue()` - 3. Opens modal immediately with all progress bars at 0% - 4. Collects image IDs for future API call (Stage 2) - 5. Logs Stage 1 completion - -- **Added Modal Component:** - - Renders `ImageQueueModal` at end of component - - Handles modal close with image reload - - Passes queue state and update callback - -### 2. API Services - -#### Updated: `frontend/src/services/api.ts` -- **Added Interface: `ImageGenerationSettings`** - ```typescript - { - success: boolean; - config: { - provider: string; - model: string; - image_type: string; - max_in_article_images: number; - image_format: string; - desktop_enabled: boolean; - mobile_enabled: boolean; - }; - } - ``` - -- **Added Function: `fetchImageGenerationSettings()`** - - Fetches image generation settings from backend - - Endpoint: `/v1/system/integrations/image_generation/` - - Returns settings including `max_in_article_images` - -### 3. Backend API - -#### Updated: `backend/igny8_core/modules/system/integration_views.py` -- **Added Method: `get_image_generation_settings()`** - - Action decorator: `@action(detail=False, methods=['get'], url_path='image_generation')` - - Gets account from request (with fallbacks) - - Retrieves `IntegrationSettings` for `image_generation` type - - Returns formatted config with defaults if not found - - Default values: - - `provider`: 'openai' - - `model`: 'dall-e-3' - - `image_type`: 'realistic' - - `max_in_article_images`: 2 - - `image_format`: 'webp' - - `desktop_enabled`: True - - `mobile_enabled`: True - -#### Updated: `backend/igny8_core/modules/system/urls.py` -- **Added URL Route:** - ```python - path('integrations/image_generation/', integration_image_gen_settings_viewset, name='integration-image-gen-settings') - ``` -- **Added ViewSet:** - ```python - integration_image_gen_settings_viewset = IntegrationSettingsViewSet.as_view({ - 'get': 'get_image_generation_settings', - }) - ``` - ---- - -## How It Works - -### User Flow: -1. User clicks "Generate Images" button on Images page -2. System fetches `max_in_article_images` from IntegrationSettings -3. System builds queue: - - 1 Featured Image (if pending and has prompt) - - N In-Article Images (up to `max_in_article_images`, if pending and have prompts) -4. **Modal opens immediately** showing all progress bars at 0% -5. Each progress bar displays: - - Queue number (1, 2, 3...) - - Label (Featured Image, In-Article Image 1, etc.) - - Content title - - Status: "⏳ Pending" - - Progress: 0% - - Thumbnail placeholder: "No image" - -### Queue Calculation: -``` -Total Images = 1 (featured) + min(pending_in_article_count, max_in_article_images) -``` - -Example: -- Settings: `max_in_article_images = 3` -- Content has: 1 featured (pending), 5 in-article (pending) -- Queue: 1 featured + 3 in-article = **4 total progress bars** - ---- - -## Files Modified - -### Frontend: -1. ✅ `frontend/src/components/common/ImageQueueModal.tsx` (NEW) -2. ✅ `frontend/src/pages/Writer/Images.tsx` (UPDATED) -3. ✅ `frontend/src/services/api.ts` (UPDATED) - -### Backend: -1. ✅ `backend/igny8_core/modules/system/integration_views.py` (UPDATED) -2. ✅ `backend/igny8_core/modules/system/urls.py` (UPDATED) - ---- - -## Testing Checklist - -- [x] Modal opens immediately when "Generate Images" button is clicked -- [x] Modal shows correct number of progress bars (1 featured + N in-article) -- [x] Progress bars display correct labels and content titles -- [x] All progress bars start at 0% with "Pending" status -- [x] Modal can be closed (when not processing) -- [x] API endpoint returns correct `max_in_article_images` value -- [x] Queue respects `max_in_article_images` setting -- [x] Only pending images with prompts are included in queue - ---- - -## Next Steps (Stage 2) - -### Planned Features: -1. **Start Actual Generation** - - Call `generateImages()` API with image IDs - - Handle async task response with `task_id` - -2. **Real-Time Progress Updates** - - Poll task progress or use WebSocket - - Update individual progress bars as images are generated - - Implement progressive loading (0-50% in 7s, 50-75% in 5s, etc.) - -3. **Sequential Processing** - - Process images one at a time (sequential) - - Update status: pending → processing → completed/failed - - Update progress percentage for each image - -4. **Error Handling** - - Mark failed images with error message - - Continue processing other images if one fails - - Display error in queue item - -5. **Completion Handling** - - Show generated image thumbnails - - Update all statuses to completed/failed - - Reload images list on modal close - ---- - -## Notes - -- Stage 1 focuses on **immediate visual feedback** - modal opens instantly -- No API calls are made in Stage 1 (only settings fetch) -- Queue structure is built client-side from existing image data -- Modal is ready to receive progress updates in Stage 2 -- Design matches WP plugin's image-queue-processor.js modal - ---- - -## Related Files - -- **Reference Implementation:** `igny8-ai-seo-wp-plugin/assets/js/image-queue-processor.js` -- **Implementation Plan:** `docs/IMAGE_GENERATION_IMPLEMENTATION_PLAN.md` -- **Backend Function:** `backend/igny8_core/ai/functions/generate_images_from_prompts.py` - ---- - ---- - -## Stage 1 Updates: Modal Styling Improvements - -**Date:** 2025-01-XX -**Status:** Completed - -### Changes Made: - -1. **Updated Modal Background** - - Changed from `bg-black bg-opacity-50` to standard modal backdrop - - Now uses `Modal` component with `bg-gray-400/50 backdrop-blur-[32px]` (standard glass effect) - - Matches all other modals in the system - -2. **Replaced Emojis with React Icons** - - Header: Replaced 🎨 emoji with `FileIcon` component - - Pending status: Replaced ⏳ emoji with `TimeIcon` component - - Processing status: Added spinning SVG icon - - Completed status: Uses `CheckCircleIcon` component - - Failed status: Uses `ErrorIcon` component - -3. **Standard Modal Width** - - Changed from custom width to `max-w-4xl` (standard modal width) - - Uses `Modal` component which provides consistent styling across system - -### Files Modified: -- ✅ `frontend/src/components/common/ImageQueueModal.tsx` (UPDATED) - -### Visual Improvements: -- Modal now has proper blur/glass backdrop effect -- Icons are consistent with system design -- Modal width matches other modals in the system -- Better visual hierarchy with icon in header - ---- - -**End of Stage 1 Changelog** - ---- - -## Stage 2 & 3: Image Generation Execution & Real-Time Progress - Completed ✅ - -**Date:** 2025-01-XX -**Status:** Completed -**Goal:** Execute image generation sequentially and update progress in real-time - ---- - -## Changes Made - -### 1. Backend API - -#### Updated: `backend/igny8_core/modules/writer/views.py` -- **Added Method: `generate_images()`** - - Action decorator: `@action(detail=False, methods=['post'], url_path='generate_images')` - - Accepts `ids` (image IDs array) and `content_id` (optional) - - Queues `process_image_generation_queue` Celery task - - Returns `task_id` for progress tracking - - Handles validation errors and execution errors - -### 2. Celery Task - -#### Updated: `backend/igny8_core/ai/tasks.py` -- **Added Task: `process_image_generation_queue()`** - - Decorator: `@shared_task(bind=True, name='igny8_core.ai.tasks.process_image_generation_queue')` - - Parameters: `image_ids`, `account_id`, `content_id` - - Loads account from `account_id` - - Gets image generation settings from `IntegrationSettings` (provider, model, image_type, etc.) - - Gets provider API key from `IntegrationSettings` (OpenAI or Runware) - - Gets prompt templates from `PromptRegistry` - - Processes images sequentially (one at a time) - - For each image: - - Updates task meta with current progress - - Loads image record from database - - Validates prompt exists - - Gets content for prompt formatting - - Formats prompt using template - - Calls `AICore.generate_image()` - - Updates `Images` model: `image_url`, `status` - - Handles errors per image (continues on failure) - - Returns final result with counts and per-image results - -### 3. Frontend API - -#### Updated: `frontend/src/services/api.ts` -- **Updated Function: `generateImages()`** - - Added `contentId` parameter (optional) - - Sends both `ids` and `content_id` to backend - - Returns task response with `task_id` - -### 4. Frontend Components - -#### Updated: `frontend/src/pages/Writer/Images.tsx` -- **Added State:** - - `taskId`: Stores Celery task ID for polling -- **Updated Function: `handleGenerateImages()`** - - Stage 2: Calls `generateImages()` API after opening modal - - Stores `task_id` in state - - Handles API errors -- **Updated Modal Props:** - - Passes `taskId` to `ImageQueueModal` - -#### Updated: `frontend/src/components/common/ImageQueueModal.tsx` -- **Added Prop: `taskId`** - - Optional string for Celery task ID -- **Added Polling Mechanism:** - - `useEffect` hook polls task status every 1 second - - Endpoint: `/api/v1/system/settings/task_progress/{taskId}/` - - Stops polling when task completes (SUCCESS/FAILURE) -- **Added Function: `updateQueueFromTaskMeta()`** - - Updates queue items from task meta data - - Maps `current_image`, `completed`, `failed`, `results` to queue items - - Updates status, progress, imageUrl, error for each item -- **Added Function: `updateQueueFromTaskResult()`** - - Updates queue items from final task result - - Sets final status and image URLs - - Handles completion state - -### 5. Backend Task Status Endpoint - -#### Existing: `backend/igny8_core/modules/system/integration_views.py` -- **Method: `task_progress()`** - - Already exists and handles Celery task status polling - - Returns task state (PENDING, PROGRESS, SUCCESS, FAILURE) - - Returns task meta for progress updates - - Returns task result on completion - ---- - -## How It Works - -### User Flow: -1. User clicks "Generate Images" button -2. **Stage 1:** Modal opens immediately with all progress bars -3. **Stage 2:** Frontend calls `generate_images()` API -4. Backend queues `process_image_generation_queue` Celery task -5. **Stage 3:** Frontend polls task status every 1 second -6. Celery task processes images sequentially: - - Image 1: Load → Format Prompt → Generate → Save - - Image 2: Load → Format Prompt → Generate → Save - - ... (continues for all images) -7. Task updates meta with progress after each image -8. Frontend receives updates and updates modal progress bars -9. On completion, modal shows final status for all images -10. User closes modal → Images list reloads - -### Data Flow: -``` -Frontend (Images.tsx) - ↓ handleGenerateImages() - ↓ generateImages() API call -Backend (ImagesViewSet.generate_images) - ↓ process_image_generation_queue.delay() -Celery Task (process_image_generation_queue) - ↓ For each image: - ↓ Load from DB - ↓ Get settings (IntegrationSettings) - ↓ Format prompt (PromptRegistry) - ↓ AICore.generate_image() - ↓ Update Images model (image_url, status) - ↓ Update task meta -Frontend (ImageQueueModal) - ↓ Poll task_progress endpoint - ↓ Update queue from meta/result - ↓ Display progress bars -``` - -### Progress Updates: -- **Task Meta Structure:** - - `current_image`: Index of image being processed - - `total_images`: Total images in queue - - `completed`: Number of completed images - - `failed`: Number of failed images - - `results`: Array of per-image results -- **Per-Image Result:** - - `image_id`: Image record ID - - `status`: 'completed' or 'failed' - - `image_url`: Generated image URL (if successful) - - `error`: Error message (if failed) - ---- - -## Files Modified - -### Backend: -1. ✅ `backend/igny8_core/modules/writer/views.py` (UPDATED) - - `generate_images()` method - -2. ✅ `backend/igny8_core/ai/tasks.py` (UPDATED) - - `process_image_generation_queue()` task - -### Frontend: -1. ✅ `frontend/src/services/api.ts` (UPDATED) - - `generateImages()` function - -2. ✅ `frontend/src/pages/Writer/Images.tsx` (UPDATED) - - `handleGenerateImages()` function - - `taskId` state - - Modal props - -3. ✅ `frontend/src/components/common/ImageQueueModal.tsx` (UPDATED) - - `taskId` prop - - Polling mechanism - - `updateQueueFromTaskMeta()` function - - `updateQueueFromTaskResult()` function - ---- - -## Integration Points - -### Settings Integration: -- **IntegrationSettings Model:** - - `image_generation` type: Provider, model, image_type, max_in_article_images, etc. - - `openai` type: API key, model - - `runware` type: API key, model - -### Prompt Templates: -- **PromptRegistry:** - - `get_image_prompt_template()`: Formats prompt with post_title, image_prompt, image_type - - `get_negative_prompt()`: Returns negative prompt for Runware - -### Image Storage: -- **Images Model:** - - `image_url`: Updated with generated image URL - - `status`: Updated to 'generated' or 'failed' - - `prompt`: Used for generation (already set) - ---- - -## Testing Checklist - -- [x] API endpoint accepts image IDs and content ID -- [x] Celery task queues successfully -- [x] Task processes images sequentially -- [x] Task reads settings from IntegrationSettings -- [x] Task formats prompts using PromptRegistry -- [x] Task calls AICore.generate_image() correctly -- [x] Task updates Images model with URLs and status -- [x] Task updates meta with progress -- [x] Frontend polls task status correctly -- [x] Modal updates progress bars in real-time -- [x] Modal shows generated image thumbnails -- [x] Modal handles errors per image -- [x] Modal shows completion summary -- [x] Images list reloads after modal close - ---- - -## Notes - -- Images are processed **sequentially** (one at a time) to respect API rate limits -- Each image failure is handled independently (doesn't stop other images) -- Progress updates are sent via Celery task meta (polled every 1 second) -- Task status endpoint already exists and is reused -- Integration settings are read from database (not hardcoded) -- Prompt templates support fallback if formatting fails -- Image URLs are saved directly to Images model -- Status is updated to 'generated' or 'failed' per image - ---- - -**End of Stage 2 & 3 Changelog** - diff --git a/docs/IMAGE_GENERATION_IMPLEMENTATION_PLAN.md b/docs/IMAGE_GENERATION_IMPLEMENTATION_PLAN.md deleted file mode 100644 index 9f907ac9..00000000 --- a/docs/IMAGE_GENERATION_IMPLEMENTATION_PLAN.md +++ /dev/null @@ -1,841 +0,0 @@ -# Image Generation Implementation Plan -## Complete Plan for Generating Images from Prompts - -**Date:** 2025-01-XX -**Scope:** Implement image generation AI function following existing AI framework patterns - ---- - -## Table of Contents - -1. [System Understanding](#1-system-understanding) -2. [Architecture Overview](#2-architecture-overview) -3. [Implementation Plan](#3-implementation-plan) -4. [Technical Details](#4-technical-details) -5. [Frontend Integration](#5-frontend-integration) -6. [Testing Strategy](#6-testing-strategy) - ---- - -## 1. System Understanding - -### 1.1 Current AI Framework Architecture - -The system uses a unified AI framework with the following components: - -**Core Flow:** -``` -Frontend API Call - ↓ -views.py (@action endpoint) - ↓ -run_ai_task (ai/tasks.py) - Unified Celery task entrypoint - ↓ -AIEngine (ai/engine.py) - Orchestrator (6 phases: INIT, PREP, AI_CALL, PARSE, SAVE, DONE) - ↓ -BaseAIFunction implementation - ↓ -AICore (ai/ai_core.py) - Centralized AI request handler - ↓ -AI Provider (OpenAI/Runware) -``` - -**Existing AI Functions:** -1. **AutoClusterFunction** (`auto_cluster.py`) - Groups keywords into clusters -2. **GenerateIdeasFunction** (`generate_ideas.py`) - Generates content ideas from clusters -3. **GenerateContentFunction** (`generate_content.py`) - Generates article content from ideas -4. **GenerateImagePromptsFunction** (`generate_image_prompts.py`) - Extracts image prompts from content - -**Key Components:** -- **BaseAIFunction** - Abstract base class with methods: `get_name()`, `validate()`, `prepare()`, `build_prompt()`, `parse_response()`, `save_output()` -- **AIEngine** - Manages lifecycle, progress tracking, cost tracking, error handling -- **PromptRegistry** - Centralized prompt management with hierarchy (task → DB → default) -- **AICore** - Handles API calls to OpenAI/Runware for both text and image generation -- **IntegrationSettings** - Stores account-specific configurations (models, API keys, image settings) - -### 1.2 Image Generation System (WordPress Plugin Reference) - -**Key Learnings from WP Plugin:** - -1. **Queue-Based Processing:** - - Images are processed sequentially in a queue - - Each image has its own progress bar (0-50% in 7s, 50-75% in 5s, 75-95% incrementally) - - Progress modal shows all images being processed with individual status - -2. **Image Types:** - - Featured image (1 per content) - - In-article images (configurable: 1-5 per content) - - Desktop images (if enabled) - - Mobile images (if enabled) - -3. **Settings from IntegrationSettings:** - - `provider`: 'openai' or 'runware' - - `model`: Model name (e.g., 'dall-e-3', 'runware:97@1') - - `image_type`: 'realistic', 'artistic', 'cartoon' - - `max_in_article_images`: 1-5 - - `image_format`: 'webp', 'jpg', 'png' - - `desktop_enabled`: boolean - - `mobile_enabled`: boolean - -4. **Prompt Templates:** - - `image_prompt_template`: Template for formatting prompts (uses {post_title}, {image_prompt}, {image_type}) - - `negative_prompt`: Negative prompt for Runware (OpenAI doesn't support) - -5. **Progress Tracking:** - - Real-time progress updates via Celery - - Individual image status tracking - - Success/failure per image - -### 1.3 Current Image Generation Function - -**Existing:** `GenerateImagesFunction` (`generate_images.py`) -- **Status:** Partially implemented, uses old pattern -- **Issues:** - - Still references `Tasks` instead of `Content` - - Doesn't follow the new unified framework pattern - - Uses legacy `generate_images_core()` wrapper - - Doesn't properly queue multiple images - -**What We Need:** -- New function: `GenerateImagesFromPromptsFunction` -- Should work with `Images` model (which now has `content` relationship) -- Should process images in queue (one at a time) -- Should use progress modal similar to other AI functions -- Should use prompt templates and negative prompts from Thinker/Prompts - ---- - -## 2. Architecture Overview - -### 2.1 New Function: `GenerateImagesFromPromptsFunction` - -**Purpose:** Generate actual images from existing image prompts stored in `Images` model - -**Input:** -- `ids`: List of Image IDs (or Content IDs) to generate images for -- Images must have `prompt` field populated (from `GenerateImagePromptsFunction`) - -**Output:** -- Updates `Images` records with: - - `image_url`: Generated image URL - - `status`: 'generated' (or 'failed' on error) - -**Flow:** -1. **INIT (0-10%)**: Validate image IDs, check prompts exist -2. **PREP (10-25%)**: Load images, get settings, prepare queue -3. **AI_CALL (25-70%)**: Generate images sequentially (one per AI_CALL phase) -4. **PARSE (70-85%)**: Parse image URLs from responses -5. **SAVE (85-98%)**: Update Images records with URLs -6. **DONE (98-100%)**: Complete - -### 2.2 Key Differences from Other Functions - -**Unlike text generation functions:** -- **Multiple AI calls**: One AI call per image (not one call for all) -- **Sequential processing**: Images must be generated one at a time (rate limits) -- **Progress per image**: Need to track progress for each individual image -- **Different API**: Uses `AICore.generate_image()` instead of `AICore.run_ai_request()` - -**Similarities:** -- Uses same `BaseAIFunction` pattern -- Uses same `AIEngine` orchestrator -- Uses same progress tracking system -- Uses same error handling - ---- - -## 3. Implementation Plan - -### Phase 1: Backend AI Function - -#### 3.1 Create `GenerateImagesFromPromptsFunction` - -**File:** `backend/igny8_core/ai/functions/generate_images_from_prompts.py` - -**Class Structure:** -```python -class GenerateImagesFromPromptsFunction(BaseAIFunction): - def get_name(self) -> str: - return 'generate_images_from_prompts' - - def get_metadata(self) -> Dict: - return { - 'display_name': 'Generate Images from Prompts', - 'description': 'Generate actual images from image prompts', - 'phases': { - 'INIT': 'Validating image prompts...', - 'PREP': 'Preparing image generation queue...', - 'AI_CALL': 'Generating images with AI...', - 'PARSE': 'Processing image URLs...', - 'SAVE': 'Saving image URLs...', - 'DONE': 'Images generated!' - } - } - - def validate(self, payload: dict, account=None) -> Dict: - """Validate image IDs and check prompts exist""" - # Check for 'ids' array - # Check images exist and have prompts - # Check images have status='pending' - # Check account matches - - def prepare(self, payload: dict, account=None) -> Dict: - """Load images and settings""" - # Load Images records by IDs - # Get IntegrationSettings for image_generation - # Extract: provider, model, image_type, image_format, etc. - # Get prompt templates from PromptRegistry - # Return: { - # 'images': [Image objects], - # 'settings': {...}, - # 'image_prompt_template': str, - # 'negative_prompt': str - # } - - def build_prompt(self, data: Dict, account=None) -> Dict: - """Format prompt using template""" - # For each image in queue: - # - Get content title (from image.content) - # - Format prompt using image_prompt_template - # - Return formatted prompt + image_type - # Note: This is called once per image (AIEngine handles iteration) - - def parse_response(self, response: Dict, step_tracker=None) -> Dict: - """Parse image URL from response""" - # Response from AICore.generate_image() has: - # - 'url': Image URL - # - 'revised_prompt': (optional) - # - 'cost': (optional) - # Return: {'url': str, 'revised_prompt': str, 'cost': float} - - def save_output(self, parsed: Dict, original_data: Dict, account=None, ...) -> Dict: - """Update Images record with URL""" - # Get image from original_data - # Update Images record: - # - image_url = parsed['url'] - # - status = 'generated' - # - updated_at = now() - # Return: {'count': 1, 'images_generated': 1} -``` - -**Key Implementation Details:** - -1. **Multiple AI Calls Handling:** - - `AIEngine` will call `build_prompt()` → `AI_CALL` → `parse_response()` → `SAVE` for each image - - Need to track which image is being processed - - Use `step_tracker` to log progress per image - -2. **Prompt Formatting:** - ```python - # Get template from PromptRegistry - template = PromptRegistry.get_image_prompt_template(account) - - # Format with content title and prompt - formatted = template.format( - post_title=image.content.title or image.content.meta_title, - image_prompt=image.prompt, - image_type=settings['image_type'] - ) - ``` - -3. **Image Generation:** - ```python - # Use AICore.generate_image() - result = ai_core.generate_image( - prompt=formatted_prompt, - provider=settings['provider'], - model=settings['model'], - size='1024x1024', # Default or from settings - negative_prompt=negative_prompt if provider == 'runware' else None, - function_name='generate_images_from_prompts' - ) - ``` - -4. **Progress Tracking:** - - Track total images: `len(images)` - - Track completed: Increment after each SAVE - - Update progress: `(completed / total) * 100` - -#### 3.2 Update AIEngine for Multiple AI Calls - -**File:** `backend/igny8_core/ai/engine.py` - -**Changes Needed:** -- Detect if function needs multiple AI calls (check function name or metadata) -- For `generate_images_from_prompts`: - - Loop through images in PREP data - - For each image: - - Call `build_prompt()` with single image - - Call `AI_CALL` phase (generate image) - - Call `parse_response()` - - Call `SAVE` phase - - Update progress: `(current_image / total_images) * 100` - - After all images: Call DONE phase - -**Alternative Approach (Simpler):** -- Process all images in `save_output()` method -- Make AI calls directly in `save_output()` (not through AIEngine phases) -- Update progress manually via `progress_tracker.update()` -- This is simpler but less consistent with framework - -**Recommended Approach:** -- Use AIEngine's phase system -- Add metadata flag: `requires_multiple_ai_calls: True` -- AIEngine detects this and loops through items - -#### 3.3 Register Function - -**File:** `backend/igny8_core/ai/registry.py` - -```python -def _load_generate_images_from_prompts(): - from igny8_core.ai.functions.generate_images_from_prompts import GenerateImagesFromPromptsFunction - return GenerateImagesFromPromptsFunction - -register_lazy_function('generate_images_from_prompts', _load_generate_images_from_prompts) -``` - -**File:** `backend/igny8_core/ai/functions/__init__.py` - -```python -from .generate_images_from_prompts import GenerateImagesFromPromptsFunction - -__all__ = [ - ... - 'GenerateImagesFromPromptsFunction', -] -``` - -#### 3.4 Add Model Configuration - -**File:** `backend/igny8_core/ai/settings.py` - -```python -MODEL_CONFIG = { - ... - 'generate_images_from_prompts': { - 'model': 'dall-e-3', # Default, overridden by IntegrationSettings - 'max_tokens': None, # Not used for images - 'temperature': None, # Not used for images - 'response_format': None, # Not used for images - }, -} - -FUNCTION_TO_PROMPT_TYPE = { - ... - 'generate_images_from_prompts': None, # Uses image_prompt_template, not text prompt -} -``` - -#### 3.5 Update Progress Messages - -**File:** `backend/igny8_core/ai/engine.py` - -```python -def _get_prep_message(self, function_name: str, count: int, data: Any) -> str: - ... - elif function_name == 'generate_images_from_prompts': - total_images = len(data.get('images', [])) - return f"Preparing to generate {total_images} image{'s' if total_images != 1 else ''}" - -def _get_ai_call_message(self, function_name: str, count: int) -> str: - ... - elif function_name == 'generate_images_from_prompts': - return f"Generating image {count} of {total} with AI" - -def _get_parse_message_with_count(self, function_name: str, count: int) -> str: - ... - elif function_name == 'generate_images_from_prompts': - return f"{count} image{'s' if count != 1 else ''} generated" - -def _get_save_message(self, function_name: str, count: int) -> str: - ... - elif function_name == 'generate_images_from_prompts': - return f"Saving {count} image{'s' if count != 1 else ''}" -``` - -### Phase 2: API Endpoint - -#### 3.6 Add API Endpoint - -**File:** `backend/igny8_core/modules/writer/views.py` - -**Add to `ImagesViewSet`:** - -```python -@action(detail=False, methods=['post'], url_path='generate_images', url_name='generate_images') -def generate_images(self, request): - """Generate images from prompts for image records""" - from igny8_core.ai.tasks import run_ai_task - - account = getattr(request, 'account', None) - ids = request.data.get('ids', []) - - if not ids: - return Response({ - 'error': 'No IDs provided', - 'type': 'ValidationError' - }, status=status.HTTP_400_BAD_REQUEST) - - account_id = account.id if account else None - - # Queue Celery task - try: - if hasattr(run_ai_task, 'delay'): - task = run_ai_task.delay( - function_name='generate_images_from_prompts', - payload={'ids': ids}, - account_id=account_id - ) - return Response({ - 'success': True, - 'task_id': str(task.id), - 'message': 'Image generation started' - }, status=status.HTTP_200_OK) - else: - # Fallback to synchronous execution - result = run_ai_task( - function_name='generate_images_from_prompts', - payload={'ids': ids}, - account_id=account_id - ) - if result.get('success'): - return Response({ - 'success': True, - 'images_generated': result.get('count', 0), - 'message': 'Images generated successfully' - }, status=status.HTTP_200_OK) - else: - return Response({ - 'error': result.get('error', 'Image generation failed'), - 'type': 'TaskExecutionError' - }, status=status.HTTP_500_INTERNAL_SERVER_ERROR) - except Exception as e: - return Response({ - 'error': str(e), - 'type': 'ExecutionError' - }, status=status.HTTP_500_INTERNAL_SERVER_ERROR) -``` - -### Phase 3: Frontend Integration - -#### 3.7 Add API Function - -**File:** `frontend/src/services/api.ts` - -```typescript -export async function generateImages(imageIds: number[]): Promise { - return fetchAPI('/v1/writer/images/generate_images/', { - method: 'POST', - body: JSON.stringify({ ids: imageIds }), - }); -} -``` - -#### 3.8 Add Generate Images Button - -**File:** `frontend/src/config/pages/images.config.tsx` - -**Add to row actions or status column:** -- Add "Generate Images" button in status column -- Only show if status is 'pending' and prompt exists -- Button should trigger generation for all images for that content - -**File:** `frontend/src/pages/Writer/Images.tsx` - -**Add handler:** -```typescript -const handleGenerateImages = useCallback(async (contentId: number) => { - try { - // Get all pending images for this content - const contentImages = images.find(g => g.content_id === contentId); - if (!contentImages) return; - - // Collect all image IDs with prompts - const imageIds: number[] = []; - if (contentImages.featured_image?.id && contentImages.featured_image.status === 'pending') { - imageIds.push(contentImages.featured_image.id); - } - contentImages.in_article_images.forEach(img => { - if (img.id && img.status === 'pending' && img.prompt) { - imageIds.push(img.id); - } - }); - - if (imageIds.length === 0) { - toast.info('No pending images with prompts found'); - return; - } - - const result = await generateImages(imageIds); - if (result.success) { - if (result.task_id) { - // Open progress modal - progressModal.openModal( - result.task_id, - 'Generate Images', - 'ai-generate-images-from-prompts-01-desktop' - ); - } else { - toast.success(`Images generated: ${result.images_generated || 0} image${(result.images_generated || 0) === 1 ? '' : 's'} created`); - loadImages(); - } - } else { - toast.error(result.error || 'Failed to generate images'); - } - } catch (error: any) { - toast.error(`Failed to generate images: ${error.message}`); - } -}, [toast, progressModal, loadImages, images]); -``` - -#### 3.9 Update Progress Modal - -**File:** `frontend/src/components/common/ProgressModal.tsx` - -**Add support for image generation:** -- Update step labels for `generate_images_from_prompts` -- Show progress per image -- Display generated images in modal (optional, like WP plugin) - -**Step Labels:** -```typescript -if (funcName.includes('generate_images_from_prompts')) { - return [ - { phase: 'INIT', label: 'Validating image prompts' }, - { phase: 'PREP', label: 'Preparing image generation queue' }, - { phase: 'AI_CALL', label: 'Generating images with AI' }, - { phase: 'PARSE', label: 'Processing image URLs' }, - { phase: 'SAVE', label: 'Saving image URLs' }, - ]; -} -``` - -**Success Message:** -```typescript -if (funcName.includes('generate_images_from_prompts')) { - const imageCount = extractCount(/(\d+)\s+image/i, stepLogs || []); - if (imageCount) { - return `${imageCount} image${imageCount !== '1' ? 's' : ''} generated successfully`; - } - return 'Images generated successfully'; -} -``` - ---- - -## 4. Technical Details - -### 4.1 Image Generation API - -**AICore.generate_image()** already exists and handles: -- OpenAI DALL-E (dall-e-2, dall-e-3) -- Runware API -- Negative prompts (Runware only) -- Cost tracking -- Error handling - -**Usage:** -```python -result = ai_core.generate_image( - prompt=formatted_prompt, - provider='openai', # or 'runware' - model='dall-e-3', # or 'runware:97@1' - size='1024x1024', - negative_prompt=negative_prompt, # Only for Runware - function_name='generate_images_from_prompts' -) -``` - -**Response:** -```python -{ - 'url': 'https://...', # Image URL - 'revised_prompt': '...', # OpenAI may revise prompt - 'cost': 0.04, # Cost in USD - 'error': None # Error message if failed -} -``` - -### 4.2 Settings Retrieval - -**From IntegrationSettings:** -```python -integration = IntegrationSettings.objects.get( - account=account, - integration_type='image_generation', - is_active=True -) -config = integration.config - -provider = config.get('provider') or config.get('service', 'openai') -if provider == 'runware': - model = config.get('model') or config.get('runwareModel', 'runware:97@1') -else: - model = config.get('model', 'dall-e-3') - -image_type = config.get('image_type', 'realistic') -image_format = config.get('image_format', 'webp') -``` - -### 4.3 Prompt Templates - -**From PromptRegistry:** -```python -image_prompt_template = PromptRegistry.get_image_prompt_template(account) -negative_prompt = PromptRegistry.get_negative_prompt(account) -``` - -**Formatting:** -```python -formatted = image_prompt_template.format( - post_title=content.title or content.meta_title, - image_prompt=image.prompt, - image_type=image_type # 'realistic', 'artistic', 'cartoon' -) -``` - -### 4.4 Error Handling - -**Per-Image Errors:** -- If one image fails, continue with others -- Mark failed image: `status='failed'` -- Log error in `Images` record or separate error field -- Return success with partial count: `{'success': True, 'images_generated': 3, 'images_failed': 1}` - -**Validation Errors:** -- No prompts: Skip image, log warning -- No settings: Return error, don't start generation -- Invalid provider/model: Return error - ---- - -## 5. Frontend Integration - -### 5.1 Images Page Updates - -**File:** `frontend/src/pages/Writer/Images.tsx` - -**Changes:** -1. Add "Generate Images" button in status column (or row actions) -2. Button only enabled if: - - Status is 'pending' - - Prompt exists - - Content has at least one pending image -3. On click: Collect all pending image IDs for that content -4. Call API: `generateImages(imageIds)` -5. Open progress modal if async -6. Reload images on completion - -### 5.2 Progress Modal Updates - -**File:** `frontend/src/components/common/ProgressModal.tsx` - -**Changes:** -1. Add step definitions for `generate_images_from_prompts` -2. Update progress messages -3. Show image count in messages -4. Optional: Display generated images in modal (like WP plugin) - -### 5.3 Table Actions Config - -**File:** `frontend/src/config/pages/table-actions.config.tsx` - -**Add row action (optional):** -```typescript -'/writer/images': { - rowActions: [ - { - key: 'generate_images', - label: 'Generate Images', - icon: , - variant: 'primary', - }, - ], -} -``` - ---- - -## 6. Testing Strategy - -### 6.1 Unit Tests - -**Test Function Methods:** -- `validate()`: Test with valid/invalid IDs, missing prompts, wrong status -- `prepare()`: Test settings retrieval, prompt template loading -- `build_prompt()`: Test prompt formatting -- `parse_response()`: Test URL extraction -- `save_output()`: Test Images record update - -### 6.2 Integration Tests - -**Test Full Flow:** -1. Create Images records with prompts -2. Call API endpoint -3. Verify Celery task created -4. Verify progress updates -5. Verify Images records updated with URLs -6. Verify status changed to 'generated' - -### 6.3 Error Scenarios - -**Test:** -- Missing IntegrationSettings -- Invalid provider/model -- API errors (rate limits, invalid API key) -- Partial failures (some images succeed, some fail) -- Missing prompts -- Invalid image IDs - ---- - -## 7. Implementation Checklist - -### Backend -- [ ] Create `GenerateImagesFromPromptsFunction` class -- [ ] Implement `validate()` method -- [ ] Implement `prepare()` method -- [ ] Implement `build_prompt()` method -- [ ] Implement `parse_response()` method -- [ ] Implement `save_output()` method -- [ ] Register function in `registry.py` -- [ ] Add to `__init__.py` exports -- [ ] Add model config in `settings.py` -- [ ] Update `AIEngine` progress messages -- [ ] Add API endpoint in `ImagesViewSet` -- [ ] Test with OpenAI provider -- [ ] Test with Runware provider -- [ ] Test error handling - -### Frontend -- [ ] Add `generateImages()` API function -- [ ] Add "Generate Images" button to Images page -- [ ] Add click handler -- [ ] Integrate progress modal -- [ ] Update progress modal step labels -- [ ] Update success messages -- [ ] Test UI flow -- [ ] Test error handling - -### Documentation -- [ ] Update AI_MASTER_ARCHITECTURE.md -- [ ] Add function to AI_FUNCTIONS_AUDIT_REPORT.md -- [ ] Document API endpoint -- [ ] Document settings requirements - ---- - -## 8. Key Considerations - -### 8.1 Rate Limiting - -**Issue:** Image generation APIs have rate limits -**Solution:** Process images sequentially (one at a time) -**Implementation:** AIEngine loops through images, waits for each to complete - -### 8.2 Cost Tracking - -**Issue:** Need to track costs per image -**Solution:** AICore already tracks costs, store in AITaskLog -**Implementation:** Cost is returned from `generate_image()`, log in step_tracker - -### 8.3 Progress Updates - -**Issue:** Need granular progress (per image) -**Solution:** Update progress after each image: `(completed / total) * 100` -**Implementation:** Track in `save_output()`, update via `progress_tracker.update()` - -### 8.4 Error Recovery - -**Issue:** If one image fails, should continue with others -**Solution:** Catch errors per image, mark as failed, continue -**Implementation:** Try-catch in `save_output()` per image - -### 8.5 Image Display - -**Issue:** Should show generated images in progress modal? -**Solution:** Optional enhancement, can add later -**Implementation:** Store image URLs in step logs, display in modal - ---- - -## 9. Alternative Approaches Considered - -### 9.1 Process All in save_output() - -**Pros:** -- Simpler implementation -- Direct control over loop - -**Cons:** -- Doesn't use AIEngine phases properly -- Harder to track progress per image -- Less consistent with framework - -**Decision:** Use AIEngine phases with loop detection - -### 9.2 Separate Function Per Image - -**Pros:** -- Each image is independent task -- Better error isolation - -**Cons:** -- Too many Celery tasks -- Harder to track overall progress -- More complex frontend - -**Decision:** Single function processes all images sequentially - ---- - -## 10. Success Criteria - -✅ Function follows `BaseAIFunction` pattern -✅ Uses `AIEngine` orchestrator -✅ Integrates with progress modal -✅ Uses prompt templates from Thinker/Prompts -✅ Uses settings from IntegrationSettings -✅ Handles errors gracefully -✅ Tracks progress per image -✅ Updates Images records correctly -✅ Works with both OpenAI and Runware -✅ Frontend button triggers generation -✅ Progress modal shows correct steps -✅ Success message shows image count - ---- - -## 11. Next Steps - -1. **Start with Backend Function** - - Create `GenerateImagesFromPromptsFunction` - - Implement all methods - - Test with single image - -2. **Add API Endpoint** - - Add to `ImagesViewSet` - - Test endpoint - -3. **Frontend Integration** - - Add button - - Add handler - - Test flow - -4. **Progress Modal** - - Update step labels - - Test progress updates - -5. **Error Handling** - - Test error scenarios - - Verify graceful failures - -6. **Documentation** - - Update architecture docs - - Add API docs - ---- - -**End of Plan** - diff --git a/docs/MASTER-ARCHITECTURE.md b/docs/MASTER-ARCHITECTURE.md deleted file mode 100644 index c61ae4dc..00000000 --- a/docs/MASTER-ARCHITECTURE.md +++ /dev/null @@ -1,1153 +0,0 @@ -# IGNY8 Master Architecture Document - -**Version:** 1.0 -**Last Updated:** 2025-01-XX -**Purpose:** Comprehensive master architecture document covering all aspects of the IGNY8 platform including workflows, functions, tech stack, infrastructure, and system design. - ---- - -## Table of Contents - -1. [Executive Summary](#executive-summary) -2. [System Overview](#system-overview) -3. [Technology Stack](#technology-stack) -4. [Core Architecture Principles](#core-architecture-principles) -5. [System Architecture](#system-architecture) -6. [Module Organization](#module-organization) -7. [Workflows & Functions](#workflows--functions) -8. [Data Flow & Processing](#data-flow--processing) -9. [Multi-Tenancy Architecture](#multi-tenancy-architecture) -10. [API Architecture](#api-architecture) -11. [Frontend Architecture](#frontend-architecture) -12. [Backend Architecture](#backend-architecture) -13. [AI Framework](#ai-framework) -14. [Infrastructure & Deployment](#infrastructure--deployment) -15. [Security Architecture](#security-architecture) -16. [Database Architecture](#database-architecture) -17. [Integration Points](#integration-points) -18. [System Diagrams](#system-diagrams) - ---- - -## Executive Summary - -**IGNY8** is a full-stack SaaS platform for SEO keyword management and AI-driven content generation. The system provides a scalable, multi-account platform that enables users to: - -- **Manage SEO Keywords**: Import, organize, and cluster keywords -- **Plan Content**: Generate content ideas from keyword clusters -- **Generate Content**: AI-powered blog post and article generation -- **Create Images**: AI-powered image generation for content -- **Publish Content**: Direct WordPress integration for publishing - -### Key Metrics - -- **Architecture**: Multi-tenant SaaS with account isolation -- **Backend**: Django 5.2+ with Django REST Framework -- **Frontend**: React 19 with TypeScript -- **Database**: PostgreSQL 15 -- **Task Queue**: Celery with Redis -- **Deployment**: Docker-based containerization -- **AI Functions**: 8 primary AI operations -- **Modules**: 4 core modules (Planner, Writer, System, Billing) - ---- - -## System Overview - -### Platform Capabilities - -| Capability | Description | Module | -|------------|-------------|--------| -| **Keyword Management** | Import, organize, filter, and manage SEO keywords | Planner | -| **Keyword Clustering** | AI-powered semantic clustering of related keywords | Planner | -| **Content Ideas** | Generate content ideas from keyword clusters | Planner | -| **Content Generation** | AI-powered blog post and article generation | Writer | -| **Image Generation** | AI-powered image generation (DALL-E, Runware) | Writer | -| **WordPress Integration** | Direct publishing to WordPress sites | Writer | -| **Account Management** | Multi-account SaaS with user roles | Auth | -| **Billing & Credits** | Credit-based billing system | Billing | -| **AI Configuration** | Customizable AI prompts and settings | System | - -### System Hierarchy - -``` -Account (1) ──< (N) User -Account (1) ──< (1) Subscription ──> (1) Plan -Account (1) ──< (N) Site -Site (1) ──< (1-5) Sector -Sector (1) ──< (N) Keywords, Clusters, ContentIdeas, Tasks -``` - -### User Roles - -| Role | Access Level | Description | -|------|--------------|-------------| -| **Developer** | System-wide | Full system access, bypasses all restrictions | -| **Owner** | Account-wide | Full account access, can manage users and billing | -| **Admin** | Account-wide | Account admin access, can manage content and users | -| **Editor** | Site-specific | Content editing access, can manage clusters/tasks | -| **Viewer** | Site-specific | Read-only access | -| **System Bot** | System-wide | System automation user | - ---- - -## Technology Stack - -### Backend Stack - -| Component | Technology | Version | Purpose | -|-----------|------------|---------|---------| -| **Framework** | Django | 5.2+ | Web framework | -| **API Framework** | Django REST Framework | Latest | RESTful API | -| **Database** | PostgreSQL | 15 | Primary database | -| **Task Queue** | Celery | Latest | Asynchronous tasks | -| **Cache/Broker** | Redis | 7 | Celery broker & caching | -| **Authentication** | JWT | SimpleJWT | Token-based auth | -| **HTTP Client** | Requests | Latest | External API calls | -| **WSGI Server** | Gunicorn | Latest | Production server | - -### Frontend Stack - -| Component | Technology | Version | Purpose | -|-----------|------------|---------|---------| -| **Framework** | React | 19 | UI library | -| **Language** | TypeScript | Latest | Type safety | -| **Build Tool** | Vite | Latest | Build tool & dev server | -| **Styling** | Tailwind CSS | Latest | Utility-first CSS | -| **State Management** | Zustand | Latest | Lightweight state | -| **Routing** | React Router | v6 | Client-side routing | -| **HTTP Client** | Fetch API | Native | API communication | - -### Infrastructure Stack - -| Component | Technology | Purpose | -|-----------|------------|---------| -| **Containerization** | Docker | Application containers | -| **Orchestration** | Docker Compose | Multi-container orchestration | -| **Reverse Proxy** | Caddy | HTTPS termination & routing | -| **Database Admin** | pgAdmin | PostgreSQL administration | -| **File Management** | FileBrowser | Web-based file management | -| **Container Management** | Portainer | Docker container management | - -### External Services - -| Service | Purpose | Integration | -|---------|---------|-------------| -| **OpenAI API** | Text generation (GPT models) | API integration | -| **OpenAI DALL-E** | Image generation | API integration | -| **Runware API** | Alternative image generation | API integration | -| **WordPress** | Content publishing | REST API integration | -| **Stripe** | Payment processing | Webhook integration (planned) | - ---- - -## Core Architecture Principles - -### 1. Configuration-Driven Everything - -**Principle**: Zero HTML/JSX duplication - All UI rendered from configuration. - -**Implementation**: -- **Frontend**: Config files in `/config/pages/` and `/config/snippets/` -- **Backend**: DRF serializers and ViewSet actions -- **Templates**: 4 universal templates (Dashboard, Table, Form, System) - -**Benefits**: -- Single source of truth -- Easy maintenance -- Consistent UI/UX -- Rapid feature development - -### 2. Multi-Tenancy Foundation - -**Principle**: Complete account isolation with automatic filtering. - -**Implementation**: -- All models inherit `AccountBaseModel` -- All ViewSets inherit `AccountModelViewSet` -- Middleware injects account context from JWT -- Site > Sector hierarchy for content organization - -**Benefits**: -- Data security -- Scalability -- Resource isolation -- Simplified access control - -### 3. Template System (4 Universal Templates) - -**Principle**: Reusable templates for all page types. - -| Template | Purpose | Usage | -|----------|---------|-------| -| **DashboardTemplate** | Module home pages | KPIs, workflow steps, charts | -| **TablePageTemplate** | CRUD table pages | Keywords, Clusters, Tasks, etc. | -| **FormPageTemplate** | Settings/form pages | Settings, Integration, etc. | -| **SystemPageTemplate** | System/admin pages | Logs, Status, Monitoring | - -### 4. Unified AI Processor - -**Principle**: Single interface for all AI operations. - -**Implementation**: -- Single `AIProcessor` class handles all AI operations -- Manual and automated workflows use same functions -- Action-based routing: 'clustering', 'ideas', 'content_generation', 'image_generation' -- Account-specific API keys and model configuration - -**Benefits**: -- Code reusability -- Consistent error handling -- Unified logging -- Easy to extend - -### 5. Module-Based Organization - -**Principle**: Clear module boundaries with shared utilities. - -**Modules**: -- **Planner**: Keywords, Clusters, Ideas -- **Writer**: Tasks, Content, Images -- **System**: Settings, Prompts, Integration -- **Billing**: Credits, Transactions, Usage - ---- - -## System Architecture - -### High-Level Architecture - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Client Layer │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ -│ │ Browser │ │ Mobile │ │ Admin │ │ -│ │ (React) │ │ (Future) │ │ Panel │ │ -│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ -└─────────┼──────────────────┼──────────────────┼─────────────┘ - │ │ │ - └──────────────────┼──────────────────┘ - │ -┌────────────────────────────┼──────────────────────────────┐ -│ Reverse Proxy Layer │ -│ ┌───────────────┐ │ -│ │ Caddy │ │ -│ │ (HTTPS/443) │ │ -│ └───────┬───────┘ │ -└────────────────────────────┼──────────────────────────────┘ - │ -┌────────────────────────────┼──────────────────────────────┐ -│ Application Layer │ -│ ┌──────────────┐ ┌──────────────┐ │ -│ │ Frontend │ │ Backend │ │ -│ │ (React) │◄─────────────┤ (Django) │ │ -│ │ Port 8021 │ REST API │ Port 8011 │ │ -│ └──────────────┘ └──────┬───────┘ │ -│ │ │ -│ ┌────────┴────────┐ │ -│ │ Celery Worker │ │ -│ │ (Async Tasks) │ │ -│ └────────┬────────┘ │ -└───────────────────────────────────────┼──────────────────┘ - │ -┌───────────────────────────────────────┼──────────────────┐ -│ Data Layer │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ -│ │ PostgreSQL │ │ Redis │ │ Storage │ │ -│ │ (Database) │ │ (Cache/Broker)│ │ (Files) │ │ -│ └──────────────┘ └──────────────┘ └──────────────┘ │ -└──────────────────────────────────────────────────────────┘ - │ -┌───────────────────────────────────────┼──────────────────┐ -│ External Services │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ -│ │ OpenAI │ │ Runware │ │ WordPress │ │ -│ │ (GPT/DALL-E)│ │ (Images) │ │ (Publish) │ │ -│ └──────────────┘ └──────────────┘ └──────────────┘ │ -└──────────────────────────────────────────────────────────┘ -``` - -### Request Flow - -``` -1. User Request - ↓ -2. Browser (React Frontend) - ↓ -3. Caddy Reverse Proxy (HTTPS Termination) - ↓ -4. Django Backend (API Endpoint) - ↓ -5. AccountContextMiddleware (Account Isolation) - ↓ -6. ViewSet (Business Logic) - ↓ -7. Serializer (Validation) - ↓ -8. Model (Database) - ↓ -9. Response (JSON) - ↓ -10. Frontend (UI Update) -``` - -### AI Task Flow - -``` -1. User Action (e.g., "Auto Cluster Keywords") - ↓ -2. Frontend API Call - ↓ -3. Backend Endpoint (ViewSet Action) - ↓ -4. Celery Task Queued - ↓ -5. Task ID Returned to Frontend - ↓ -6. Frontend Polls Progress Endpoint - ↓ -7. Celery Worker Processes Task - ↓ -8. AIProcessor Makes API Calls - ↓ -9. Results Saved to Database - ↓ -10. Progress Updates Sent - ↓ -11. Frontend Displays Results -``` - ---- - -## Module Organization - -### Module Structure - -| Module | Purpose | Models | ViewSets | Tasks | -|--------|---------|--------|----------|-------| -| **Planner** | Keyword management & content planning | Keywords, Clusters, ContentIdeas | KeywordViewSet, ClusterViewSet, ContentIdeasViewSet | auto_cluster_keywords_task, auto_generate_ideas_task | -| **Writer** | Content generation & management | Tasks, Content, Images | TasksViewSet | auto_generate_content_task, auto_generate_images_task | -| **System** | Settings, prompts, integrations | AIPrompt, IntegrationSettings, AuthorProfile, Strategy | AIPromptViewSet, IntegrationSettingsViewSet, AuthorProfileViewSet | - | -| **Billing** | Credits, transactions, usage | CreditTransaction, UsageLog | CreditTransactionViewSet, UsageLogViewSet | - | -| **Auth** | Multi-tenancy, users, accounts | Account, User, Plan, Site, Sector | AccountViewSet, UserViewSet, SiteViewSet, SectorViewSet | - | - -### Module Dependencies - -``` -Auth (Core) - ├── Planner (depends on Auth) - ├── Writer (depends on Auth, Planner) - ├── System (depends on Auth) - └── Billing (depends on Auth) -``` - ---- - -## Workflows & Functions - -### Primary Workflows - -#### 1. Keyword Management Workflow - -``` -1. Import Keywords (CSV/Manual) - ↓ -2. Keywords Stored in Database - ↓ -3. Filter & Organize Keywords - ↓ -4. Select Keywords for Clustering - ↓ -5. Auto Cluster (AI Function) - ↓ -6. Clusters Created - ↓ -7. Generate Ideas from Clusters - ↓ -8. Content Ideas Created -``` - -#### 2. Content Generation Workflow - -``` -1. Select Content Idea - ↓ -2. Create Task - ↓ -3. Auto Generate Content (AI Function) - ↓ -4. Content Generated & Saved - ↓ -5. Review & Edit Content - ↓ -6. Auto Generate Images (AI Function) - ↓ -7. Images Generated & Attached - ↓ -8. Publish to WordPress -``` - -#### 3. Account Setup Workflow - -``` -1. User Signs Up - ↓ -2. Account Created - ↓ -3. Default Plan Assigned - ↓ -4. Site Created - ↓ -5. Sectors Created (1-5 per site) - ↓ -6. Integration Settings Configured - ↓ -7. Ready to Use -``` - -### AI Functions - -| Function | Module | Purpose | Input | Output | -|----------|--------|---------|-------|--------| -| **Auto Cluster Keywords** | Planner | Group related keywords | Keyword IDs | Clusters | -| **Auto Generate Ideas** | Planner | Generate content ideas | Cluster IDs | Content Ideas | -| **Auto Generate Content** | Writer | Generate blog posts | Task IDs | HTML Content | -| **Auto Generate Images** | Writer | Generate images | Task IDs | Images | -| **Extract Image Prompts** | Writer | Extract prompts from content | Content Text | Image Prompts | -| **Test OpenAI** | System | Test OpenAI connection | API Key | Connection Status | -| **Test Runware** | System | Test Runware connection | API Key | Connection Status | -| **Generate Image (Test)** | System | Test image generation | Prompt | Image URL | - -### AI Function Execution Flow - -``` -1. User Initiates AI Function - ↓ -2. Frontend Calls API Endpoint - ↓ -3. Backend Validates Input - ↓ -4. Celery Task Queued - ↓ -5. Task ID Returned - ↓ -6. Frontend Polls Progress - ↓ -7. Celery Worker Executes: - a. Prepare Data - b. Build Prompt - c. Call AI API - d. Parse Response - e. Save Results - ↓ -8. Progress Updates Sent - ↓ -9. Task Completes - ↓ -10. Results Displayed -``` - ---- - -## Data Flow & Processing - -### Data Models Hierarchy - -``` -AccountBaseModel - ├── Account - ├── User - ├── Plan - ├── Site - ├── Sector - └── SiteSectorBaseModel - ├── Keywords - ├── Clusters - ├── ContentIdeas - ├── Tasks - ├── Content - └── Images -``` - -### Data Relationships - -| Parent | Child | Relationship | Constraints | -|--------|-------|--------------|-------------| -| Account | User | One-to-Many | N users per account | -| Account | Site | One-to-Many | N sites per account (plan limit) | -| Account | Subscription | One-to-One | 1 subscription per account | -| Subscription | Plan | Many-to-One | 1 plan per subscription | -| Site | Sector | One-to-Many | 1-5 sectors per site | -| Sector | Keywords | One-to-Many | N keywords per sector | -| Sector | Clusters | One-to-Many | N clusters per sector | -| Sector | ContentIdeas | One-to-Many | N ideas per sector | -| Sector | Tasks | One-to-Many | N tasks per sector | -| Cluster | Keywords | Many-to-Many | N keywords per cluster | -| Cluster | ContentIdeas | One-to-Many | N ideas per cluster | -| Task | Content | One-to-One | 1 content per task | -| Task | Images | One-to-Many | N images per task | -| ContentIdeas | Task | One-to-Many | N tasks per idea | - -### Data Processing Pipeline - -``` -Raw Data (CSV/Manual) - ↓ -Import & Validation - ↓ -Database Storage - ↓ -AI Processing (Optional) - ↓ -Enriched Data - ↓ -User Review & Edit - ↓ -Final Data - ↓ -Export/Publish -``` - ---- - -## Multi-Tenancy Architecture - -### Account Isolation - -**Implementation Levels**: - -1. **Model Level**: All models inherit `AccountBaseModel` with `account` ForeignKey -2. **ViewSet Level**: All ViewSets inherit `AccountModelViewSet` with automatic filtering -3. **Middleware Level**: `AccountContextMiddleware` sets `request.account` from JWT - -### Access Control Matrix - -| User Role | Account Access | Site Access | Data Access | -|-----------|----------------|-------------|-------------| -| Developer | All accounts | All sites | All data | -| System Bot | All accounts | All sites | All data | -| Owner | Own account | All sites in account | All data in account | -| Admin | Own account | All sites in account | All data in account | -| Editor | Own account | Granted sites only | Data in granted sites | -| Viewer | Own account | Granted sites only | Read-only in granted sites | - -### Site/Sector Hierarchy - -``` -Account - └── Site 1 (Active) - ├── Sector 1 (Active) - │ ├── Keywords - │ ├── Clusters - │ ├── ContentIdeas - │ └── Tasks - ├── Sector 2 (Active) - └── Sector 3 (Inactive) - └── Site 2 (Active) - └── Sector 1 (Active) -``` - ---- - -## API Architecture - -### API Structure - -**Base URL**: `/api/v1/` - -**Module Endpoints**: - -| Module | Base Path | Resources | -|--------|-----------|-----------| -| **Auth** | `/api/v1/auth/` | accounts, users, sites, sectors, plans | -| **Planner** | `/api/v1/planner/` | keywords, clusters, ideas | -| **Writer** | `/api/v1/writer/` | tasks, content, images | -| **System** | `/api/v1/system/` | prompts, integrations, author-profiles, strategies | -| **Billing** | `/api/v1/billing/` | credits, transactions, usage | - -### API Endpoints - -#### Planner Endpoints - -| Method | Endpoint | Purpose | -|--------|----------|---------| -| GET | `/api/v1/planner/keywords/` | List keywords | -| POST | `/api/v1/planner/keywords/` | Create keyword | -| GET | `/api/v1/planner/keywords/{id}/` | Get keyword | -| PUT | `/api/v1/planner/keywords/{id}/` | Update keyword | -| DELETE | `/api/v1/planner/keywords/{id}/` | Delete keyword | -| POST | `/api/v1/planner/keywords/auto_cluster/` | Auto-cluster keywords | -| POST | `/api/v1/planner/keywords/bulk_delete/` | Bulk delete | -| GET | `/api/v1/planner/keywords/export_csv/` | Export CSV | -| POST | `/api/v1/planner/keywords/import_csv/` | Import CSV | -| GET | `/api/v1/planner/clusters/` | List clusters | -| POST | `/api/v1/planner/clusters/auto_generate_ideas/` | Generate ideas | -| GET | `/api/v1/planner/ideas/` | List ideas | - -#### Writer Endpoints - -| Method | Endpoint | Purpose | -|--------|----------|---------| -| GET | `/api/v1/writer/tasks/` | List tasks | -| POST | `/api/v1/writer/tasks/auto_generate_content/` | Generate content | -| POST | `/api/v1/writer/tasks/auto_generate_images/` | Generate images | - -#### System Endpoints - -| Method | Endpoint | Purpose | -|--------|----------|---------| -| GET | `/api/v1/system/settings/integrations/{id}/` | Get integration | -| PUT | `/api/v1/system/settings/integrations/{id}/` | Save integration | -| POST | `/api/v1/system/settings/integrations/{id}/test_openai/` | Test OpenAI | -| POST | `/api/v1/system/settings/integrations/{id}/test_runware/` | Test Runware | -| GET | `/api/v1/system/settings/task_progress/{task_id}/` | Get task progress | - -### Authentication Flow - -``` -1. User Signs In - ↓ -2. Backend Validates Credentials - ↓ -3. JWT Token Generated - ↓ -4. Token Returned to Frontend - ↓ -5. Frontend Stores Token (localStorage) - ↓ -6. Frontend Includes Token in Requests - ↓ -7. Backend Validates Token - ↓ -8. Account Context Set - ↓ -9. Request Processed -``` - -### Response Format - -**Success Response**: -```json -{ - "success": true, - "data": { ... }, - "message": "Optional message" -} -``` - -**Error Response**: -```json -{ - "success": false, - "message": "Error message", - "errors": { ... } -} -``` - -**Pagination Response**: -```json -{ - "count": 100, - "next": "http://api.example.com/api/v1/resource/?page=2", - "previous": null, - "results": [ ... ] -} -``` - ---- - -## Frontend Architecture - -### Component Hierarchy - -``` -App -└── AppLayout - ├── Sidebar (Navigation) - ├── Header (User Menu, Notifications) - └── Main Content - └── Page Component - └── Template (DashboardTemplate, TablePageTemplate, etc.) - └── Components (DataTable, Filters, etc.) -``` - -### State Management - -| Store | Purpose | State | -|-------|---------|-------| -| **authStore** | Authentication | user, token, isAuthenticated | -| **siteStore** | Site selection | activeSite, sites | -| **sectorStore** | Sector selection | activeSector, sectors | -| **plannerStore** | Planner module | module-specific state | -| **aiRequestLogsStore** | AI logs | request/response logs | -| **pageSizeStore** | Table preferences | pageSize | - -### Routing Structure - -| Route | Component | Template | -|-------|-----------|----------| -| `/` | Home | DashboardTemplate | -| `/planner` | PlannerDashboard | DashboardTemplate | -| `/planner/keywords` | Keywords | TablePageTemplate | -| `/planner/clusters` | Clusters | TablePageTemplate | -| `/planner/ideas` | Ideas | TablePageTemplate | -| `/writer` | WriterDashboard | DashboardTemplate | -| `/writer/tasks` | Tasks | TablePageTemplate | -| `/thinker` | ThinkerDashboard | DashboardTemplate | -| `/thinker/prompts` | Prompts | TablePageTemplate | -| `/billing/credits` | Credits | FormPageTemplate | -| `/settings` | Settings | FormPageTemplate | - -### Configuration System - -**Page-Local Config**: `/config/pages/` -- Page-specific table, filter, and action configurations - -**Shared Snippets**: `/config/snippets/` -- Reusable column definitions -- Reusable filter definitions -- Reusable action definitions - ---- - -## Backend Architecture - -### Model Inheritance Hierarchy - -``` -models.Model -└── AccountBaseModel (adds account ForeignKey) - └── SiteSectorBaseModel (adds site, sector ForeignKeys) - └── Keywords, Clusters, ContentIdeas, Tasks, etc. -``` - -### ViewSet Inheritance Hierarchy - -``` -viewsets.ModelViewSet -└── AccountModelViewSet (adds account filtering) - └── SiteSectorModelViewSet (adds site/sector filtering) - └── KeywordViewSet, ClusterViewSet, TasksViewSet, etc. -``` - -### Middleware Stack - -1. **SecurityMiddleware**: Django security middleware -2. **SessionMiddleware**: Session management -3. **AuthenticationMiddleware**: User authentication -4. **AccountContextMiddleware**: Sets `request.account` from JWT -5. **ResourceTrackerMiddleware**: Tracks API request metrics - -### Celery Task Architecture - -**Broker**: Redis - -**Workers**: Separate Celery worker processes - -**Task Structure**: -```python -@shared_task(bind=True) -def my_task(self, ...): - # Update progress - self.update_state(state='PROGRESS', meta={...}) - # Do work - result = do_work() - # Return result - return result -``` - -**Progress Tracking**: -- Frontend polls `/api/v1/system/settings/task_progress/{task_id}/` -- Backend returns task state and meta information -- Progress displayed in modal - ---- - -## AI Framework - -### AI Framework Architecture - -``` -BaseAIFunction (Abstract) - ├── AutoClusterFunction - ├── AutoGenerateIdeasFunction - ├── AutoGenerateContentFunction - └── AutoGenerateImagesFunction - -AIEngine (Orchestrator) - ├── INIT Phase (0-10%) - ├── PREP Phase (10-25%) - ├── AI_CALL Phase (25-60%) - ├── PARSE Phase (60-80%) - ├── SAVE Phase (80-95%) - └── DONE Phase (95-100%) - -AIProcessor (Wrapper) - ├── cluster_keywords() - ├── generate_ideas() - ├── generate_content() - ├── extract_image_prompts() - └── generate_image() -``` - -### AI Function Execution - -``` -1. User Action - ↓ -2. API Endpoint - ↓ -3. run_ai_task.delay() - ↓ -4. AIEngine.execute() - ↓ -5. Function.prepare() - ↓ -6. Function.build_prompt() - ↓ -7. AIProcessor API Call - ↓ -8. Function.parse_response() - ↓ -9. Function.save_output() - ↓ -10. Task Complete -``` - -### Progress Tracking - -**Progress Endpoint**: `/api/v1/system/settings/task_progress/{task_id}/` - -**Response Format**: -```json -{ - "state": "PROGRESS", - "meta": { - "phase": "AI_CALL", - "percentage": 45, - "message": "Processing...", - "request_steps": [...], - "response_steps": [...], - "cost": 0.000123, - "tokens": 1500 - } -} -``` - ---- - -## Infrastructure & Deployment - -### Docker Architecture - -**Two-Stack Architecture**: - -1. **Infrastructure Stack** (`igny8-infra`): - - PostgreSQL - - Redis - - pgAdmin - - FileBrowser - - Caddy - -2. **Application Stack** (`igny8-app`): - - Backend (Django) - - Frontend (React) - - Celery Worker - - Celery Beat - -### Network Architecture - -**Single Network**: `igny8_net` (bridge network) - -**Container Communication**: Service names as hostnames - -### Port Allocation - -| Service | External Port | Internal Port | Purpose | -|---------|---------------|---------------|---------| -| Backend | 8011 | 8010 | Django API | -| Frontend | 8021 | 5173 | React Dev Server | -| PostgreSQL | - | 5432 | Database | -| Redis | - | 6379 | Cache/Broker | -| pgAdmin | 5050 | 80 | DB Admin | -| FileBrowser | 8080 | 80 | File Manager | -| Caddy | 80, 443 | 80, 443 | Reverse Proxy | - -### Volume Management - -| Volume | Purpose | Location | -|--------|---------|----------| -| `pgdata` | PostgreSQL data | `/var/lib/docker/volumes/igny8-infra_pgdata/` | -| `redisdata` | Redis data | `/var/lib/docker/volumes/igny8-infra_redisdata/` | -| `caddy_data` | SSL certificates | `/var/lib/docker/volumes/igny8-infra_caddy_data/` | -| Application Code | Source code | `/data/app/igny8/` (host mount) | - ---- - -## Security Architecture - -### Authentication - -**Methods**: -- JWT (JSON Web Tokens) - Primary method -- Session-based auth - Fallback for admin - -**Flow**: -1. User signs in → Backend returns JWT token -2. Frontend stores token in localStorage -3. Frontend includes token in `Authorization: Bearer {token}` header -4. Backend middleware validates token and sets `request.user` and `request.account` - -### Authorization - -**Role-Based Access Control (RBAC)**: -- `developer`: Full system access -- `owner`: Full account access -- `admin`: Account admin access -- `editor`: Content editing access -- `viewer`: Read-only access - -**Access Control**: -- Account-level: Automatic filtering by `request.account` -- Site-level: Filtering by `user.get_accessible_sites()` -- Action-level: Permission checks in ViewSet actions - -### Data Isolation - -**Account Isolation**: -- All queries filtered by account -- Admin/Developer override for system accounts - -**Site Access Control**: -- Users can only access granted sites -- Admin/Developer override for all sites - -### API Security - -- **CORS**: Configured for frontend domain -- **CSRF**: Enabled for session-based auth -- **Input Validation**: DRF serializers validate all input -- **Rate Limiting**: (Future implementation) - ---- - -## Database Architecture - -### Core Tables - -| Table | Purpose | Key Fields | -|-------|---------|------------| -| `igny8_accounts` | Account information | name, slug, owner, plan, credits, status | -| `igny8_users` | User accounts | email, account, role | -| `igny8_plans` | Subscription plans | name, price, limits, features | -| `igny8_subscriptions` | Active subscriptions | account, plan, status, period | -| `igny8_sites` | Sites within accounts | name, slug, account, industry, status | -| `igny8_sectors` | Sectors within sites | name, slug, site, status | -| `igny8_industries` | Global industry templates | name, slug, description | -| `igny8_industry_sectors` | Industry sector templates | name, slug, industry | - -### Planner Tables - -| Table | Purpose | Key Fields | -|-------|---------|------------| -| `igny8_keywords` | Keywords | keyword, volume, difficulty, intent, cluster | -| `igny8_clusters` | Keyword clusters | name, description, keywords_count, volume | -| `igny8_content_ideas` | Content ideas | idea_title, description, cluster, status | - -### Writer Tables - -| Table | Purpose | Key Fields | -|-------|---------|------------| -| `igny8_tasks` | Content generation tasks | title, description, cluster, idea, status, content | -| `igny8_content` | Generated content | task, html_content, word_count | -| `igny8_images` | Generated images | task, image_type, image_url, prompt | - -### System Tables - -| Table | Purpose | Key Fields | -|-------|---------|------------| -| `igny8_ai_prompts` | AI prompt templates | prompt_type, prompt_value, account | -| `igny8_integration_settings` | API keys and configuration | integration_type, config, account | -| `igny8_author_profiles` | Writing style profiles | name, description, tone, language | -| `igny8_strategies` | Content strategies | name, description, sector, prompt_types | - -### Billing Tables - -| Table | Purpose | Key Fields | -|-------|---------|------------| -| `igny8_credit_transactions` | Credit transactions | account, transaction_type, amount, balance_after | -| `igny8_credit_usage_logs` | Usage tracking | account, operation_type, credits_used, cost_usd | - -### Indexes - -**Account Isolation**: All tables have indexes on `account` - -**Site/Sector Filtering**: Tables with site/sector have composite indexes on `(account, site, sector)` - -**Performance**: Indexes on frequently queried fields (status, created_at, etc.) - ---- - -## Integration Points - -### External API Integrations - -| Service | Integration Type | Purpose | Configuration | -|---------|------------------|---------|---------------| -| **OpenAI** | REST API | Text generation (GPT) | API key in IntegrationSettings | -| **OpenAI DALL-E** | REST API | Image generation | API key in IntegrationSettings | -| **Runware** | REST API | Alternative image generation | API key in IntegrationSettings | -| **WordPress** | REST API | Content publishing | wp_url, wp_username, wp_app_password in Site model | - -### Integration Flow - -``` -1. User Configures Integration - ↓ -2. API Key Stored in IntegrationSettings - ↓ -3. Test Connection (Optional) - ↓ -4. Integration Active - ↓ -5. AI Functions Use Integration - ↓ -6. API Calls Made - ↓ -7. Results Returned -``` - ---- - -## System Diagrams - -### System Component Diagram - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Frontend │ -│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ -│ │ React │ │ Zustand │ │ Vite │ │ -│ │ (UI) │ │ (State) │ │ (Build) │ │ -│ └────────────┘ └────────────┘ └────────────┘ │ -└───────────────────────────┬─────────────────────────────────┘ - │ HTTP/REST -┌───────────────────────────┴─────────────────────────────────┐ -│ Backend │ -│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ -│ │ Django │ │ DRF │ │ Celery │ │ -│ │ (Framework)│ │ (API) │ │ (Tasks) │ │ -│ └────────────┘ └────────────┘ └────────────┘ │ -└───────────────────────────┬─────────────────────────────────┘ - │ -┌───────────────────────────┴─────────────────────────────────┐ -│ Database │ -│ ┌────────────┐ ┌────────────┐ │ -│ │ PostgreSQL │ │ Redis │ │ -│ │ (Data) │ │ (Cache) │ │ -│ └────────────┘ └────────────┘ │ -└─────────────────────────────────────────────────────────────┘ -``` - -### Data Flow Diagram - -``` -User Action - ↓ -Frontend (React) - ↓ -API Request (REST) - ↓ -Caddy (Reverse Proxy) - ↓ -Django Backend - ↓ -AccountContextMiddleware (Account Isolation) - ↓ -ViewSet (Business Logic) - ↓ -Serializer (Validation) - ↓ -Model (Database) - ↓ -PostgreSQL - ↓ -Response (JSON) - ↓ -Frontend (UI Update) -``` - -### AI Function Flow Diagram - -``` -User Initiates AI Function - ↓ -Frontend API Call - ↓ -Backend Endpoint - ↓ -Celery Task Queued - ↓ -Task ID Returned - ↓ -Frontend Polls Progress - ↓ -Celery Worker Executes - ├── Prepare Data - ├── Build Prompt - ├── Call AI API (OpenAI/Runware) - ├── Parse Response - └── Save Results - ↓ -Progress Updates - ↓ -Task Complete - ↓ -Results Displayed -``` - -### Multi-Tenancy Flow Diagram - -``` -Request with JWT Token - ↓ -AccountContextMiddleware - ├── Extract Account ID from JWT - ├── Load Account Object - └── Set request.account - ↓ -ViewSet.get_queryset() - ├── Check User Role - ├── Filter by Account (if not admin/developer) - └── Filter by Accessible Sites (if not owner/admin) - ↓ -Database Query - ↓ -Results (Account-Isolated) -``` - ---- - -## Summary - -The IGNY8 platform is built on a modern, scalable architecture that provides: - -1. **Multi-Tenancy**: Complete account isolation with automatic filtering -2. **Configuration-Driven UI**: Zero duplication, single source of truth -3. **Unified AI Framework**: Single interface for all AI operations -4. **Module-Based Organization**: Clear boundaries with shared utilities -5. **RESTful API**: Consistent API design with DRF -6. **Modern Frontend**: React + TypeScript with Zustand state management -7. **Scalable Backend**: Django + Celery for async processing -8. **Security First**: JWT auth, RBAC, data isolation -9. **Docker Deployment**: Containerized for easy deployment and scaling -10. **Comprehensive Logging**: Detailed tracking for all operations - -This architecture ensures scalability, maintainability, and extensibility while providing a robust foundation for the IGNY8 platform. - ---- - -**Document Version**: 1.0 -**Last Updated**: 2025-01-XX -**Maintained By**: IGNY8 Development Team - diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..4b3b06ea --- /dev/null +++ b/docs/README.md @@ -0,0 +1,146 @@ +# IGNY8 Documentation + +**Last Updated:** 2025-01-XX +**Purpose:** Complete documentation index for the IGNY8 platform. + +--- + +## 📚 Documentation Structure + +All documentation is organized in this single folder with the following structure: + +### Core Documentation + +1. **[01-ARCHITECTURE-TECH-STACK.md](./01-ARCHITECTURE-TECH-STACK.md)** + - Technology stack overview + - System architecture principles + - Infrastructure components + - External service integrations + +2. **[02-APP-ARCHITECTURE.md](./02-APP-ARCHITECTURE.md)** + - IGNY8 application architecture + - System hierarchy and relationships + - Module organization + - Complete workflows + - Data flow and processing + - Multi-tenancy architecture + - Security architecture + +3. **[03-FRONTEND.md](./03-FRONTEND.md)** + - Frontend architecture + - Project structure + - Routing system + - Template system + - Component library + - State management + - API integration + - Configuration system + - All pages and features + +4. **[04-BACKEND.md](./04-BACKEND.md)** + - Backend architecture + - Project structure + - Models and relationships + - ViewSets and API endpoints + - Serializers + - Celery tasks + - Middleware + - All modules (Planner, Writer, System, Billing, Auth) + +5. **[05-AI-FUNCTIONS.md](./05-AI-FUNCTIONS.md)** + - AI framework architecture + - All 5 AI functions (complete details) + - AI function execution flow + - Prompt management + - Model configuration + - Progress tracking + - Cost tracking + +6. **[06-CHANGELOG.md](./06-CHANGELOG.md)** + - System changelog + - Feature additions + - Updates and improvements + - Version history + +--- + +## 🚀 Quick Start + +1. **New to IGNY8?** Start with [01-ARCHITECTURE-TECH-STACK.md](./01-ARCHITECTURE-TECH-STACK.md) for technology overview +2. **Understanding the System?** Read [02-APP-ARCHITECTURE.md](./02-APP-ARCHITECTURE.md) for complete architecture +3. **Frontend Development?** See [03-FRONTEND.md](./03-FRONTEND.md) for all frontend details +4. **Backend Development?** See [04-BACKEND.md](./04-BACKEND.md) for all backend details +5. **Working with AI?** See [05-AI-FUNCTIONS.md](./05-AI-FUNCTIONS.md) for AI functions +6. **What's New?** Check [06-CHANGELOG.md](./06-CHANGELOG.md) for recent changes + +--- + +## 📋 Documentation Overview + +### System Capabilities + +- **Multi-Tenancy**: Complete account isolation with automatic filtering +- **Planner Module**: Keywords, Clusters, Content Ideas management +- **Writer Module**: Tasks, Content, Images generation and management +- **Thinker Module**: Prompts, Author Profiles, Strategies, Image Testing +- **System Module**: Settings, Integrations, AI Prompts +- **Billing Module**: Credits, Transactions, Usage Logs +- **AI Functions**: 5 AI operations (Auto Cluster, Generate Ideas, Generate Content, Generate Image Prompts, Generate Images) + +### Technology Stack + +- **Backend**: Django 5.2+ with Django REST Framework +- **Frontend**: React 19 with TypeScript and Vite +- **Database**: PostgreSQL 15 +- **Task Queue**: Celery with Redis +- **Deployment**: Docker-based containerization +- **Reverse Proxy**: Caddy (HTTPS) + +--- + +## 🔍 Finding Information + +### By Topic + +- **Architecture & Design**: [01-ARCHITECTURE-TECH-STACK.md](./01-ARCHITECTURE-TECH-STACK.md), [02-APP-ARCHITECTURE.md](./02-APP-ARCHITECTURE.md) +- **Frontend Development**: [03-FRONTEND.md](./03-FRONTEND.md) +- **Backend Development**: [04-BACKEND.md](./04-BACKEND.md) +- **AI Functions**: [05-AI-FUNCTIONS.md](./05-AI-FUNCTIONS.md) +- **Changes & Updates**: [06-CHANGELOG.md](./06-CHANGELOG.md) + +### By Module + +- **Planner**: See [02-APP-ARCHITECTURE.md](./02-APP-ARCHITECTURE.md) (Module Organization) and [04-BACKEND.md](./04-BACKEND.md) (Planner Module) +- **Writer**: See [02-APP-ARCHITECTURE.md](./02-APP-ARCHITECTURE.md) (Module Organization) and [04-BACKEND.md](./04-BACKEND.md) (Writer Module) +- **Thinker**: See [03-FRONTEND.md](./03-FRONTEND.md) (Thinker Pages) and [04-BACKEND.md](./04-BACKEND.md) (System Module) +- **System**: See [04-BACKEND.md](./04-BACKEND.md) (System Module) +- **Billing**: See [04-BACKEND.md](./04-BACKEND.md) (Billing Module) + +--- + +## 📝 Documentation Standards + +- **No Code**: Documentation focuses on workflows, features, and architecture (no code snippets) +- **Complete**: All workflows and features are documented +- **Accurate**: Documentation reflects current system state +- **Detailed**: Comprehensive coverage of all aspects + +--- + +## 🔄 Keeping Documentation Updated + +Documentation is updated when: +- New features are added +- Workflows change +- Architecture evolves +- Modules are modified + +**Last Review**: 2025-01-XX +**Next Review**: As system evolves + +--- + +## 📞 Support + +For questions or clarifications about the documentation, refer to the specific document or contact the development team. + diff --git a/docs/ai-docs/AI_FILES_ANALYSIS.md b/docs/ai-docs/AI_FILES_ANALYSIS.md deleted file mode 100644 index 93598b75..00000000 --- a/docs/ai-docs/AI_FILES_ANALYSIS.md +++ /dev/null @@ -1,137 +0,0 @@ -# AI Folder Files Analysis -## Which Files Are Actually Needed for 3 Active Functions - -### ✅ **REQUIRED FILES** (All needed for active functions) - -#### 1. **tasks.py** - ✅ REQUIRED -- **Purpose:** Unified Celery task entrypoint -- **Used by:** All views (planner/views.py, writer/views.py) -- **Dependencies:** engine.py, registry.py -- **Status:** KEEP - -#### 2. **engine.py** - ✅ REQUIRED -- **Purpose:** Central orchestrator for all AI functions -- **Used by:** tasks.py -- **Dependencies:** base.py, tracker.py, ai_core.py, settings.py, models.py -- **Status:** KEEP - -#### 3. **base.py** - ✅ REQUIRED -- **Purpose:** Abstract base class for all AI functions -- **Used by:** All function classes (AutoClusterFunction, GenerateIdeasFunction, GenerateContentFunction) -- **Dependencies:** None -- **Status:** KEEP - -#### 4. **registry.py** - ✅ REQUIRED -- **Purpose:** Function registry with lazy loading -- **Used by:** tasks.py -- **Dependencies:** base.py, function modules -- **Status:** KEEP - -#### 5. **ai_core.py** - ✅ REQUIRED -- **Purpose:** Centralized AI request handler -- **Used by:** engine.py, all function classes -- **Dependencies:** constants.py, tracker.py -- **Status:** KEEP - -#### 6. **prompts.py** - ✅ REQUIRED -- **Purpose:** Centralized prompt management -- **Used by:** All function classes (build_prompt methods) -- **Dependencies:** None -- **Status:** KEEP - -#### 7. **settings.py** - ✅ REQUIRED -- **Purpose:** Model configurations per function -- **Used by:** engine.py -- **Dependencies:** constants.py (via ai_processor import) -- **Status:** KEEP - -#### 8. **validators.py** - ✅ REQUIRED -- **Purpose:** Validation functions -- **Used by:** All function classes (validate methods) -- **Dependencies:** constants.py, models -- **Status:** KEEP - -#### 9. **tracker.py** - ✅ REQUIRED -- **Purpose:** Progress tracking utilities -- **Used by:** engine.py, ai_core.py -- **Dependencies:** types.py (imported but NOT USED), constants.py -- **Status:** KEEP (but can remove types.py import) - -#### 10. **constants.py** - ✅ REQUIRED -- **Purpose:** AI constants (model rates, valid models) -- **Used by:** ai_core.py, settings.py, validators.py -- **Dependencies:** None -- **Status:** KEEP - -#### 11. **models.py** - ✅ REQUIRED -- **Purpose:** AITaskLog database model -- **Used by:** engine.py (for logging), admin.py -- **Dependencies:** AccountBaseModel -- **Status:** KEEP - -#### 12. **apps.py** - ✅ REQUIRED -- **Purpose:** Django app configuration -- **Used by:** Django (app registration) -- **Dependencies:** admin.py -- **Status:** KEEP - -#### 13. **admin.py** - ✅ REQUIRED (Optional but recommended) -- **Purpose:** Django admin interface for AITaskLog -- **Used by:** apps.py (auto-imported) -- **Dependencies:** models.py -- **Status:** KEEP (useful for debugging/admin) - ---- - -### ❌ **UNUSED FILES** (Can be removed) - -#### 1. **types.py** - ❌ NOT USED -- **Purpose:** Type definitions (StepLog, ProgressState, AITaskResult dataclasses) -- **Used by:** tracker.py (imported but never instantiated) -- **Actual usage:** None - tracker.py uses plain Dict types instead -- **Status:** **SAFE TO REMOVE** -- **Action:** Remove file and remove import from tracker.py - ---- - -## Summary - -### Files to KEEP (13 files): -1. ✅ tasks.py -2. ✅ engine.py -3. ✅ base.py -4. ✅ registry.py -5. ✅ ai_core.py -6. ✅ prompts.py -7. ✅ settings.py -8. ✅ validators.py -9. ✅ tracker.py -10. ✅ constants.py -11. ✅ models.py -12. ✅ apps.py -13. ✅ admin.py - -### Files to REMOVE (1 file): -1. ❌ **types.py** - Imported but never used - ---- - -## Action Items - -1. **Remove types.py** - File is not used -2. **Remove import from tracker.py** - Remove `from igny8_core.ai.types import StepLog, ProgressState` (line 8) - ---- - -## Verification - -All other files are actively used in the execution flow: -- **tasks.py** → **registry.py** → **engine.py** → **base.py** (function classes) -- **engine.py** → **ai_core.py** → **constants.py** -- **engine.py** → **tracker.py** → **constants.py** -- **engine.py** → **settings.py** → **constants.py** -- **engine.py** → **models.py** (AITaskLog) -- **function classes** → **prompts.py**, **validators.py** -- **apps.py** → **admin.py** → **models.py** - -1 diff --git a/docs/ai-docs/AI_FUNCTIONS_AUDIT_REPORT.md b/docs/ai-docs/AI_FUNCTIONS_AUDIT_REPORT.md deleted file mode 100644 index 6bd16884..00000000 --- a/docs/ai-docs/AI_FUNCTIONS_AUDIT_REPORT.md +++ /dev/null @@ -1,277 +0,0 @@ -# AI Functions Deep Audit Report -## Clustering, Idea Generation, and Content Generation - -**Date:** 2025-01-XX -**Scope:** Complete audit of AI functions for clustering, idea generation, and content generation to identify unused code and files safe to remove. - ---- - -## Executive Summary - -This audit identifies **legacy code, deprecated functions, and unused implementations** that are safe to remove from the codebase. The current system uses a unified AI framework through `run_ai_task` → `AIEngine` → `BaseAIFunction` implementations. - ---- - -## 1. CURRENT ACTIVE ARCHITECTURE - -### 1.1 Active Flow (What's Actually Used) - -``` -Frontend API Call - ↓ -views.py (auto_cluster/auto_generate_ideas/auto_generate_content) - ↓ -run_ai_task (ai/tasks.py) - Unified Celery task entrypoint - ↓ -AIEngine (ai/engine.py) - Orchestrator - ↓ -BaseAIFunction implementations: - - AutoClusterFunction (ai/functions/auto_cluster.py) - - GenerateIdeasFunction (ai/functions/generate_ideas.py) - - GenerateContentFunction (ai/functions/generate_content.py) - ↓ -AICore (ai/ai_core.py) - Centralized AI request handler - ↓ -AIProvider (OpenAI/Runware) -``` - -### 1.2 Active Files (KEEP) - -#### Core Framework -- ✅ `backend/igny8_core/ai/tasks.py` - Unified Celery task (`run_ai_task`) -- ✅ `backend/igny8_core/ai/engine.py` - AIEngine orchestrator -- ✅ `backend/igny8_core/ai/base.py` - BaseAIFunction abstract class -- ✅ `backend/igny8_core/ai/ai_core.py` - AICore centralized handler -- ✅ `backend/igny8_core/ai/registry.py` - Function registry -- ✅ `backend/igny8_core/ai/prompts.py` - PromptRegistry -- ✅ `backend/igny8_core/ai/settings.py` - Model configurations -- ✅ `backend/igny8_core/ai/constants.py` - AI constants -- ✅ `backend/igny8_core/ai/tracker.py` - Progress tracking -- ✅ `backend/igny8_core/ai/validators.py` - Validation functions - -#### Function Implementations -- ✅ `backend/igny8_core/ai/functions/auto_cluster.py` - AutoClusterFunction -- ✅ `backend/igny8_core/ai/functions/generate_ideas.py` - GenerateIdeasFunction -- ✅ `backend/igny8_core/ai/functions/generate_content.py` - GenerateContentFunction -- ✅ `backend/igny8_core/ai/functions/__init__.py` - Exports - -#### API Endpoints -- ✅ `backend/igny8_core/modules/planner/views.py` - KeywordViewSet.auto_cluster(), ClusterViewSet.auto_generate_ideas() -- ✅ `backend/igny8_core/modules/writer/views.py` - TasksViewSet.auto_generate_content() - ---- - -## 2. DEPRECATED / UNUSED CODE (SAFE TO REMOVE) - -### 2.1 Legacy Wrapper Functions (NOT USED) - -#### ❌ `generate_ideas_core()` - **SAFE TO REMOVE** -- **Location:** `backend/igny8_core/ai/functions/generate_ideas.py:234` -- **Status:** Legacy wrapper function for backward compatibility -- **Usage:** ❌ **NOT CALLED ANYWHERE** (only in commented test code) -- **Purpose:** Was meant for direct calls without Celery, but all calls now go through `run_ai_task` -- **Action:** **REMOVE** - Function and export from `__init__.py` - -#### ❌ `generate_content_core()` - **SAFE TO REMOVE** -- **Location:** `backend/igny8_core/ai/functions/generate_content.py:303` -- **Status:** Legacy wrapper function for backward compatibility -- **Usage:** ❌ **NOT CALLED ANYWHERE** (only in commented test code) -- **Purpose:** Was meant for direct calls without Celery, but all calls now go through `run_ai_task` -- **Action:** **REMOVE** - Function and export from `__init__.py` - -### 2.2 AIProcessor Deprecated Methods (NOT USED) - -#### ❌ `AIProcessor.cluster_keywords()` - **SAFE TO REMOVE** -- **Location:** `backend/igny8_core/utils/ai_processor.py:1049-1282` -- **Status:** ⚠️ **DEPRECATED** (marked with deprecation warning) -- **Usage:** ❌ **NOT CALLED ANYWHERE** -- **Lines:** ~233 lines of code -- **Action:** **REMOVE** entire method - -#### ❌ `AIProcessor.generate_ideas()` - **SAFE TO REMOVE** -- **Location:** `backend/igny8_core/utils/ai_processor.py:1284-1363` -- **Status:** Legacy method -- **Usage:** ❌ **NOT CALLED ANYWHERE** -- **Lines:** ~80 lines of code -- **Action:** **REMOVE** entire method - -#### ❌ `AIProcessor.generate_content()` - **SAFE TO REMOVE** -- **Location:** `backend/igny8_core/utils/ai_processor.py:433-531` -- **Status:** Legacy method -- **Usage:** ❌ **NOT CALLED ANYWHERE** (only called internally by `extract_image_prompts`, which is also unused) -- **Lines:** ~98 lines of code -- **Action:** **REMOVE** entire method - -#### ❌ `AIProcessor.extract_image_prompts()` - **SAFE TO REMOVE** -- **Location:** `backend/igny8_core/utils/ai_processor.py:471-580` -- **Status:** Legacy method -- **Usage:** ❌ **NOT CALLED ANYWHERE** (new framework uses `GenerateImagesFunction` which uses `AICore.run_ai_request` directly) -- **Lines:** ~110 lines of code -- **Action:** **REMOVE** entire method - -**Note:** `AIProcessor.generate_image()` is **STILL USED** in `integration_views.py` for image generation, so keep the class and that method. - -### 2.3 Unused Exports - -#### ❌ `generate_ideas_core` export - **SAFE TO REMOVE** -- **Location:** `backend/igny8_core/ai/functions/__init__.py:5,12` -- **Action:** Remove from imports and `__all__` - -#### ❌ `generate_content_core` export - **SAFE TO REMOVE** -- **Location:** `backend/igny8_core/ai/functions/__init__.py:6,14` -- **Action:** Remove from imports and `__all__` - -### 2.4 Test File Cleanup - -#### ⚠️ `backend/igny8_core/ai/tests/test_run.py` -- **Status:** Contains commented-out code referencing removed functions -- **Action:** Clean up commented code (lines 16, 63, 75, 126-127) - ---- - -## 3. AIProcessor - PARTIAL CLEANUP - -### 3.1 What to KEEP in AIProcessor - -✅ **KEEP:** -- `AIProcessor.__init__()` - Initialization -- `AIProcessor._get_api_key()` - API key retrieval -- `AIProcessor._get_model()` - Model retrieval -- `AIProcessor._call_openai()` - OpenAI API calls (used by generate_image) -- `AIProcessor._extract_json_from_response()` - JSON extraction -- `AIProcessor.generate_image()` - **STILL USED** in `integration_views.py` -- `AIProcessor.get_prompt()` - May be used by generate_image -- Constants imports (MODEL_RATES, etc.) - Used by AICore - -### 3.2 What to REMOVE from AIProcessor - -❌ **REMOVE:** -- `AIProcessor.cluster_keywords()` - ~233 lines -- `AIProcessor.generate_ideas()` - ~80 lines -- `AIProcessor.generate_content()` - ~98 lines -- `AIProcessor.extract_image_prompts()` - ~110 lines -- **Total:** ~521 lines of unused code - ---- - -## 4. FILES ALREADY DELETED (Good!) - -✅ **Already Removed:** -- `backend/igny8_core/modules/planner/tasks.py` - ✅ Already deleted -- `backend/igny8_core/modules/writer/tasks.py` - ✅ Already deleted - ---- - -## 5. SUMMARY OF REMOVALS - -### 5.1 Functions to Remove - -| Function | Location | Lines | Status | -|----------|----------|-------|--------| -| `generate_ideas_core()` | `ai/functions/generate_ideas.py:234` | ~100 | ❌ Remove | -| `generate_content_core()` | `ai/functions/generate_content.py:303` | ~85 | ❌ Remove | -| `AIProcessor.cluster_keywords()` | `utils/ai_processor.py:1049` | ~233 | ❌ Remove | -| `AIProcessor.generate_ideas()` | `utils/ai_processor.py:1284` | ~80 | ❌ Remove | -| `AIProcessor.generate_content()` | `utils/ai_processor.py:433` | ~98 | ❌ Remove | -| `AIProcessor.extract_image_prompts()` | `utils/ai_processor.py:471` | ~110 | ❌ Remove | - -**Total Lines to Remove:** ~706 lines - -### 5.2 Exports to Remove - -| Export | Location | -|--------|----------| -| `generate_ideas_core` | `ai/functions/__init__.py:5,12` | -| `generate_content_core` | `ai/functions/__init__.py:6,14` | - -### 5.3 Test Cleanup - -| File | Action | -|------|--------| -| `ai/tests/test_run.py` | Remove commented code (lines 16, 63, 75, 126-127) | - ---- - -## 6. VERIFICATION CHECKLIST - -Before removing, verify: - -- [ ] ✅ No imports of `generate_ideas_core` or `generate_content_core` anywhere -- [ ] ✅ No calls to `AIProcessor.cluster_keywords()`, `generate_ideas()`, or `generate_content()` -- [ ] ✅ All active code paths use `run_ai_task` → `AIEngine` → `BaseAIFunction` -- [ ] ✅ `AIProcessor.generate_image()` is still used (verify in `integration_views.py`) -- [ ] ✅ Constants from `ai_processor.py` are still imported by `AICore` (verify) - ---- - -## 7. RECOMMENDED REMOVAL ORDER - -1. **Phase 1: Remove Legacy Wrapper Functions** - - Remove `generate_ideas_core()` from `generate_ideas.py` - - Remove `generate_content_core()` from `generate_content.py` - - Remove exports from `__init__.py` - -2. **Phase 2: Remove AIProcessor Deprecated Methods** - - Remove `AIProcessor.cluster_keywords()` - - Remove `AIProcessor.generate_ideas()` - - Remove `AIProcessor.generate_content()` - - Remove `AIProcessor.extract_image_prompts()` (depends on generate_content, so remove after) - -3. **Phase 3: Cleanup Tests** - - Clean up commented code in `test_run.py` - -4. **Phase 4: Verification** - - Run full test suite - - Verify all AI functions still work - - Check for any broken imports - ---- - -## 8. IMPACT ANALYSIS - -### 8.1 No Breaking Changes Expected - -- ✅ All active code paths use the new framework -- ✅ No external dependencies on removed functions -- ✅ Views only use `run_ai_task` (verified) - -### 8.2 Benefits - -- 🎯 **~706 lines of dead code removed** -- 🎯 **Clearer codebase** - Only active code remains -- 🎯 **Easier maintenance** - No confusion about which path to use -- 🎯 **Reduced technical debt** - ---- - -## 9. FILES TO MODIFY - -### 9.1 Files to Edit - -1. `backend/igny8_core/ai/functions/generate_ideas.py` - - Remove `generate_ideas_core()` function (lines 234-333) - -2. `backend/igny8_core/ai/functions/generate_content.py` - - Remove `generate_content_core()` function (lines 303-386) - -3. `backend/igny8_core/ai/functions/__init__.py` - - Remove `generate_ideas_core` import and export - - Remove `generate_content_core` import and export - -4. `backend/igny8_core/utils/ai_processor.py` - - Remove `cluster_keywords()` method (lines 1049-1282) - - Remove `generate_ideas()` method (lines 1284-1363) - - Remove `generate_content()` method (lines 433-531) - - Remove `extract_image_prompts()` method (lines 471-580) - -5. `backend/igny8_core/ai/tests/test_run.py` - - Remove commented code referencing removed functions - ---- - -## 10. CONCLUSION - -This audit identifies **~706 lines of unused legacy code** that can be safely removed without impacting functionality. All active code paths use the unified AI framework (`run_ai_task` → `AIEngine` → `BaseAIFunction`), making these legacy functions obsolete. - -**Recommendation:** Proceed with removal in phases as outlined above, with thorough testing after each phase. - diff --git a/docs/ai-docs/AI_MASTER_ARCHITECTURE.md b/docs/ai-docs/AI_MASTER_ARCHITECTURE.md deleted file mode 100644 index bfbeaff6..00000000 --- a/docs/ai-docs/AI_MASTER_ARCHITECTURE.md +++ /dev/null @@ -1,802 +0,0 @@ -# AI Master Architecture Document -## Clustering, Idea Generation, and Content Generation - -**Version:** 1.0 -**Date:** 2025-01-XX -**Scope:** Complete architecture for 3 verified AI functions (clustering, idea generation, content generation) - ---- - -## Table of Contents - -1. [Common Architecture](#1-common-architecture) -2. [Auto Cluster Keywords](#2-auto-cluster-keywords) -3. [Generate Ideas](#3-generate-ideas) -4. [Generate Content](#4-generate-content) - ---- - -## 1. Common Architecture - -### 1.1 Core Framework Files - -#### Entry Point -- **File:** `backend/igny8_core/ai/tasks.py` -- **Function:** `run_ai_task` -- **Purpose:** Unified Celery task entrypoint for all AI functions -- **Parameters:** `function_name` (str), `payload` (dict), `account_id` (int) -- **Flow:** Loads function from registry → Creates AIEngine → Executes function - -#### Engine Orchestrator -- **File:** `backend/igny8_core/ai/engine.py` -- **Class:** `AIEngine` -- **Purpose:** Central orchestrator managing lifecycle, progress, logging, cost tracking -- **Methods:** - - `execute` - Main execution pipeline (6 phases: INIT, PREP, AI_CALL, PARSE, SAVE, DONE) - - `_handle_error` - Centralized error handling - - `_log_to_database` - Logs to AITaskLog model - - Helper methods: `_get_input_description`, `_build_validation_message`, `_get_prep_message`, `_get_ai_call_message`, `_get_parse_message`, `_get_parse_message_with_count`, `_get_save_message`, `_calculate_credits_for_clustering` - -#### Base Function Class -- **File:** `backend/igny8_core/ai/base.py` -- **Class:** `BaseAIFunction` -- **Purpose:** Abstract base class defining interface for all AI functions -- **Abstract Methods:** - - `get_name` - Returns function name (e.g., 'auto_cluster') - - `prepare` - Loads and prepares data - - `build_prompt` - Builds AI prompt - - `parse_response` - Parses AI response - - `save_output` - Saves results to database -- **Optional Methods:** - - `get_metadata` - Returns display name, description, phases - - `get_max_items` - Returns max items limit (or None) - - `validate` - Validates input payload (default: checks for 'ids') - - `get_model` - Returns model override (default: None, uses account default) - -#### Function Registry -- **File:** `backend/igny8_core/ai/registry.py` -- **Functions:** - - `register_function` - Registers function class - - `register_lazy_function` - Registers lazy loader - - `get_function` - Gets function class by name (lazy loads if needed) - - `get_function_instance` - Gets function instance by name - - `list_functions` - Lists all registered functions -- **Lazy Loaders:** - - `_load_auto_cluster` - Loads AutoClusterFunction - - `_load_generate_ideas` - Loads GenerateIdeasFunction - - `_load_generate_content` - Loads GenerateContentFunction - -#### AI Core Handler -- **File:** `backend/igny8_core/ai/ai_core.py` -- **Class:** `AICore` -- **Purpose:** Centralized AI request handler for all text generation -- **Methods:** - - `run_ai_request` - Makes API call to OpenAI/Runware - - `extract_json` - Extracts JSON from response (handles markdown code blocks) - -#### Prompt Registry -- **File:** `backend/igny8_core/ai/prompts.py` -- **Class:** `PromptRegistry` -- **Purpose:** Centralized prompt management with hierarchical resolution -- **Method:** `get_prompt` - Gets prompt with resolution order: - 1. Task-level prompt_override (if exists) - 2. DB prompt for (account, function) - 3. Default fallback from DEFAULT_PROMPTS registry -- **Prompt Types:** - - `clustering` - For auto_cluster function - - `ideas` - For generate_ideas function - - `content_generation` - For generate_content function -- **Context Placeholders:** - - `[IGNY8_KEYWORDS]` - Replaced with keyword list - - `[IGNY8_CLUSTERS]` - Replaced with cluster list - - `[IGNY8_CLUSTER_KEYWORDS]` - Replaced with cluster keywords - - `[IGNY8_IDEA]` - Replaced with idea data - - `[IGNY8_CLUSTER]` - Replaced with cluster data - - `[IGNY8_KEYWORDS]` - Replaced with keywords (for content) - -#### Model Settings -- **File:** `backend/igny8_core/ai/settings.py` -- **Constants:** - - `MODEL_CONFIG` - Model configurations per function (model, max_tokens, temperature, response_format) - - `FUNCTION_ALIASES` - Legacy function name mappings -- **Functions:** - - `get_model_config` - Gets model config for function (reads from IntegrationSettings if account provided) - - `get_model` - Gets model name for function - - `get_max_tokens` - Gets max tokens for function - - `get_temperature` - Gets temperature for function - -#### Validators -- **File:** `backend/igny8_core/ai/validators.py` -- **Functions:** - - `validate_ids` - Validates 'ids' array in payload - - `validate_keywords_exist` - Validates keywords exist in database - - `validate_cluster_exists` - Validates cluster exists - - `validate_tasks_exist` - Validates tasks exist - - `validate_cluster_limits` - Validates plan limits (currently disabled - always returns valid) - - `validate_api_key` - Validates API key is configured - - `validate_model` - Validates model is in supported list - - `validate_image_size` - Validates image size for model - -#### Progress Tracking -- **File:** `backend/igny8_core/ai/tracker.py` -- **Classes:** - - `StepTracker` - Tracks request/response steps - - `ProgressTracker` - Tracks Celery progress updates - - `CostTracker` - Tracks API costs and tokens - - `ConsoleStepTracker` - Console-based step logging - -#### Database Logging -- **File:** `backend/igny8_core/ai/models.py` -- **Model:** `AITaskLog` -- **Fields:** `task_id`, `function_name`, `account`, `phase`, `message`, `status`, `duration`, `cost`, `tokens`, `request_steps`, `response_steps`, `error`, `payload`, `result` - -### 1.2 Execution Flow (All Functions) - -``` -1. API Endpoint (views.py) - ↓ -2. run_ai_task (tasks.py) - - Gets account from account_id - - Gets function instance from registry - - Creates AIEngine - ↓ -3. AIEngine.execute (engine.py) - Phase 1: INIT (0-10%) - - Calls function.validate() - - Updates progress tracker - ↓ - Phase 2: PREP (10-25%) - - Calls function.prepare() - - Calls function.build_prompt() - - Updates progress tracker - ↓ - Phase 3: AI_CALL (25-70%) - - Gets model config from settings - - Calls AICore.run_ai_request() - - Tracks cost and tokens - - Updates progress tracker - ↓ - Phase 4: PARSE (70-85%) - - Calls function.parse_response() - - Updates progress tracker - ↓ - Phase 5: SAVE (85-98%) - - Calls function.save_output() - - Logs credit usage - - Updates progress tracker - ↓ - Phase 6: DONE (98-100%) - - Logs to AITaskLog - - Returns result -``` - ---- - -## 2. Auto Cluster Keywords - -### 2.1 Function Implementation - -- **File:** `backend/igny8_core/ai/functions/auto_cluster.py` -- **Class:** `AutoClusterFunction` -- **Inherits:** `BaseAIFunction` - -### 2.2 API Endpoint - -- **File:** `backend/igny8_core/modules/planner/views.py` -- **ViewSet:** `KeywordViewSet` -- **Action:** `auto_cluster` -- **Method:** POST -- **URL Path:** `/v1/planner/keywords/auto_cluster/` -- **Payload:** - - `ids` (list[int]) - Keyword IDs to cluster - - `sector_id` (int, optional) - Sector ID for filtering -- **Response:** - - `success` (bool) - - `task_id` (str) - Celery task ID if async - - `clusters_created` (int) - Number of clusters created - - `keywords_updated` (int) - Number of keywords updated - - `message` (str) - -### 2.3 Function Methods - -#### `get_name()` -- **Returns:** `'auto_cluster'` - -#### `get_metadata()` -- **Returns:** Dict with `display_name`, `description`, `phases` (INIT, PREP, AI_CALL, PARSE, SAVE, DONE) - -#### `get_max_items()` -- **Returns:** `None` (no limit) - -#### `validate(payload, account)` -- **Validates:** - - Calls `validate_ids` to check for 'ids' array - - Calls `validate_keywords_exist` to verify keywords exist -- **Returns:** Dict with `valid` (bool) and optional `error` (str) - -#### `prepare(payload, account)` -- **Loads:** - - Keywords from database (filters by `ids`, `account`, optional `sector_id`) - - Uses `select_related` for: `account`, `site`, `site__account`, `sector`, `sector__site` -- **Returns:** Dict with: - - `keywords` (list[Keyword objects]) - - `keyword_data` (list[dict]) - Formatted data with: `id`, `keyword`, `volume`, `difficulty`, `intent` - - `sector_id` (int, optional) - -#### `build_prompt(data, account)` -- **Gets Prompt:** - - Calls `PromptRegistry.get_prompt(function_name='auto_cluster', account, context)` - - Context includes: `KEYWORDS` (formatted keyword list), optional `SECTOR` (sector name) -- **Formatting:** - - Formats keywords as: `"- {keyword} (Volume: {volume}, Difficulty: {difficulty}, Intent: {intent})"` - - Replaces `[IGNY8_KEYWORDS]` placeholder - - Adds JSON mode instruction if not present -- **Returns:** Prompt string - -#### `parse_response(response, step_tracker)` -- **Parsing:** - - Tries direct JSON parse first - - Falls back to `AICore.extract_json()` if needed (handles markdown code blocks) -- **Extraction:** - - Extracts `clusters` array from JSON - - Handles both dict with 'clusters' key and direct array -- **Returns:** List[Dict] with cluster data: - - `name` (str) - Cluster name - - `description` (str) - Cluster description - - `keywords` (list[str]) - List of keyword strings - -#### `save_output(parsed, original_data, account, progress_tracker, step_tracker)` -- **Input:** - - `parsed` - List of cluster dicts from parse_response - - `original_data` - Dict from prepare() with `keywords` and `sector_id` -- **Process:** - - Gets account, site, sector from first keyword - - For each cluster in parsed: - - Gets or creates `Clusters` record: - - Fields: `name`, `description`, `account`, `site`, `sector`, `status='active'` - - Uses `get_or_create` with name + account + site + sector - - Matches keywords (case-insensitive): - - Normalizes cluster keywords and available keywords to lowercase - - Updates matched `Keywords` records: - - Sets `cluster` foreign key - - Sets `status='mapped'` - - Recalculates cluster metrics: - - `keywords_count` - Count of keywords in cluster - - `volume` - Sum of keyword volumes (uses `volume_override` if available, else `seed_keyword__volume`) -- **Returns:** Dict with: - - `count` (int) - Clusters created - - `clusters_created` (int) - Clusters created - - `keywords_updated` (int) - Keywords updated - -### 2.4 Database Models - -#### Keywords Model -- **File:** `backend/igny8_core/modules/planner/models.py` -- **Model:** `Keywords` -- **Fields Used:** - - `id` - Keyword ID - - `seed_keyword` (ForeignKey) - Reference to SeedKeyword - - `keyword` (property) - Gets keyword text from seed_keyword - - `volume` (property) - Gets volume from volume_override or seed_keyword - - `difficulty` (property) - Gets difficulty from difficulty_override or seed_keyword - - `intent` (property) - Gets intent from seed_keyword - - `cluster` (ForeignKey) - Assigned cluster - - `status` - Status ('active', 'pending', 'mapped', 'archived') - - `account`, `site`, `sector` - From SiteSectorBaseModel - -#### Clusters Model -- **File:** `backend/igny8_core/modules/planner/models.py` -- **Model:** `Clusters` -- **Fields Used:** - - `name` - Cluster name (unique) - - `description` - Cluster description - - `keywords_count` - Count of keywords (recalculated) - - `volume` - Sum of keyword volumes (recalculated) - - `status` - Status ('active') - - `account`, `site`, `sector` - From SiteSectorBaseModel - -### 2.5 AI Response Format - -**Expected JSON:** -```json -{ - "clusters": [ - { - "name": "Cluster Name", - "description": "Cluster description", - "keywords": ["keyword1", "keyword2", "keyword3"] - } - ] -} -``` - -### 2.6 Progress Messages - -- **INIT:** "Validating {keyword1}, {keyword2}, {keyword3} and {X} more keywords" (shows first 3, then count) -- **PREP:** "Loading {count} keyword(s)" -- **AI_CALL:** "Generating clusters with Igny8 Semantic SEO Model" -- **PARSE:** "{count} cluster(s) created" -- **SAVE:** "Saving {count} cluster(s)" - ---- - -## 3. Generate Ideas - -### 3.1 Function Implementation - -- **File:** `backend/igny8_core/ai/functions/generate_ideas.py` -- **Class:** `GenerateIdeasFunction` -- **Inherits:** `BaseAIFunction` - -### 3.2 API Endpoint - -- **File:** `backend/igny8_core/modules/planner/views.py` -- **ViewSet:** `ClusterViewSet` -- **Action:** `auto_generate_ideas` -- **Method:** POST -- **URL Path:** `/v1/planner/clusters/auto_generate_ideas/` -- **Payload:** - - `ids` (list[int]) - Cluster IDs (max 10) -- **Response:** - - `success` (bool) - - `task_id` (str) - Celery task ID if async - - `ideas_created` (int) - Number of ideas created - - `message` (str) - -### 3.3 Function Methods - -#### `get_name()` -- **Returns:** `'generate_ideas'` - -#### `get_metadata()` -- **Returns:** Dict with `display_name`, `description`, `phases` (INIT, PREP, AI_CALL, PARSE, SAVE, DONE) - -#### `get_max_items()` -- **Returns:** `10` (max clusters per generation) - -#### `validate(payload, account)` -- **Validates:** - - Calls `super().validate()` to check for 'ids' array and max_items limit - - Calls `validate_cluster_exists` for first cluster ID - - Calls `validate_cluster_limits` for plan limits (currently disabled) -- **Returns:** Dict with `valid` (bool) and optional `error` (str) - -#### `prepare(payload, account)` -- **Loads:** - - Clusters from database (filters by `ids`, `account`) - - Uses `select_related` for: `sector`, `account`, `site`, `sector__site` - - Uses `prefetch_related` for: `keywords` -- **Gets Keywords:** - - For each cluster, loads `Keywords` with `select_related('seed_keyword')` - - Extracts keyword text from `seed_keyword.keyword` -- **Returns:** Dict with: - - `clusters` (list[Cluster objects]) - - `cluster_data` (list[dict]) - Formatted data with: `id`, `name`, `description`, `keywords` (list[str]) - - `account` (Account object) - -#### `build_prompt(data, account)` -- **Gets Prompt:** - - Calls `PromptRegistry.get_prompt(function_name='generate_ideas', account, context)` - - Context includes: - - `CLUSTERS` - Formatted cluster list: `"Cluster ID: {id} | Name: {name} | Description: {description}"` - - `CLUSTER_KEYWORDS` - Formatted cluster keywords: `"Cluster ID: {id} | Name: {name} | Keywords: {keyword1}, {keyword2}"` -- **Replaces Placeholders:** - - `[IGNY8_CLUSTERS]` → clusters_text - - `[IGNY8_CLUSTER_KEYWORDS]` → cluster_keywords_text -- **Returns:** Prompt string - -#### `parse_response(response, step_tracker)` -- **Parsing:** - - Calls `AICore.extract_json()` to extract JSON from response - - Validates 'ideas' key exists in JSON -- **Returns:** List[Dict] with idea data: - - `title` (str) - Idea title - - `description` (str or dict) - Idea description (can be JSON string) - - `content_type` (str) - Content type ('blog_post', 'article', etc.) - - `content_structure` (str) - Content structure ('cluster_hub', 'supporting_page', etc.) - - `cluster_id` (int, optional) - Cluster ID reference - - `cluster_name` (str, optional) - Cluster name reference - - `estimated_word_count` (int) - Estimated word count - - `covered_keywords` or `target_keywords` (str) - Target keywords - -#### `save_output(parsed, original_data, account, progress_tracker, step_tracker)` -- **Input:** - - `parsed` - List of idea dicts from parse_response - - `original_data` - Dict from prepare() with `clusters` and `cluster_data` -- **Process:** - - For each idea in parsed: - - Matches cluster: - - First tries by `cluster_id` from AI response - - Falls back to `cluster_name` matching - - Last resort: position-based matching (first idea → first cluster) - - Gets site from cluster (or cluster.sector.site) - - Handles description: - - If dict, converts to JSON string - - If not string, converts to string - - Creates `ContentIdeas` record: - - Fields: - - `idea_title` - From `title` - - `description` - Processed description - - `content_type` - From `content_type` (default: 'blog_post') - - `content_structure` - From `content_structure` (default: 'supporting_page') - - `target_keywords` - From `covered_keywords` or `target_keywords` - - `keyword_cluster` - Matched cluster - - `estimated_word_count` - From `estimated_word_count` (default: 1500) - - `status` - 'new' - - `account`, `site`, `sector` - From cluster -- **Returns:** Dict with: - - `count` (int) - Ideas created - - `ideas_created` (int) - Ideas created - -### 3.4 Database Models - -#### Clusters Model -- **File:** `backend/igny8_core/modules/planner/models.py` -- **Model:** `Clusters` -- **Fields Used:** - - `id` - Cluster ID - - `name` - Cluster name - - `description` - Cluster description - - `keywords` (related_name) - Related Keywords - - `account`, `site`, `sector` - From SiteSectorBaseModel - -#### ContentIdeas Model -- **File:** `backend/igny8_core/modules/planner/models.py` -- **Model:** `ContentIdeas` -- **Fields Used:** - - `idea_title` - Idea title - - `description` - Idea description (can be JSON string) - - `content_type` - Content type ('blog_post', 'article', 'guide', 'tutorial') - - `content_structure` - Content structure ('cluster_hub', 'landing_page', 'pillar_page', 'supporting_page') - - `target_keywords` - Target keywords string - - `keyword_cluster` (ForeignKey) - Related cluster - - `estimated_word_count` - Estimated word count - - `status` - Status ('new', 'scheduled', 'published') - - `account`, `site`, `sector` - From SiteSectorBaseModel - -### 3.5 AI Response Format - -**Expected JSON:** -```json -{ - "ideas": [ - { - "title": "Idea Title", - "description": "Idea description or JSON structure", - "content_type": "blog_post", - "content_structure": "supporting_page", - "cluster_id": 1, - "cluster_name": "Cluster Name", - "estimated_word_count": 1500, - "covered_keywords": "keyword1, keyword2" - } - ] -} -``` - -### 3.6 Progress Messages - -- **INIT:** "Verifying cluster integrity" -- **PREP:** "Loading cluster keywords" -- **AI_CALL:** "Generating ideas with Igny8 Semantic AI" -- **PARSE:** "{count} high-opportunity idea(s) generated" -- **SAVE:** "Content Outline for Ideas generated" - ---- - -## 4. Generate Content - -### 4.1 Function Implementation - -- **File:** `backend/igny8_core/ai/functions/generate_content.py` -- **Class:** `GenerateContentFunction` -- **Inherits:** `BaseAIFunction` - -### 4.2 API Endpoint - -- **File:** `backend/igny8_core/modules/writer/views.py` -- **ViewSet:** `TasksViewSet` -- **Action:** `auto_generate_content` -- **Method:** POST -- **URL Path:** `/v1/writer/tasks/auto_generate_content/` -- **Payload:** - - `ids` (list[int]) - Task IDs (max 10) -- **Response:** - - `success` (bool) - - `task_id` (str) - Celery task ID if async - - `tasks_updated` (int) - Number of tasks updated - - `message` (str) - -### 4.3 Function Methods - -#### `get_name()` -- **Returns:** `'generate_content'` - -#### `get_metadata()` -- **Returns:** Dict with `display_name`, `description`, `phases` (INIT, PREP, AI_CALL, PARSE, SAVE, DONE) - -#### `get_max_items()` -- **Returns:** `50` (max tasks per batch) - -#### `validate(payload, account)` -- **Validates:** - - Calls `super().validate()` to check for 'ids' array and max_items limit - - Calls `validate_tasks_exist` to verify tasks exist -- **Returns:** Dict with `valid` (bool) and optional `error` (str) - -#### `prepare(payload, account)` -- **Loads:** - - Tasks from database (filters by `ids`, `account`) - - Uses `select_related` for: `account`, `site`, `sector`, `cluster`, `idea` -- **Returns:** List[Task objects] - -#### `build_prompt(data, account)` -- **Input:** Can be single Task or list[Task] (handles first task if list) -- **Builds Idea Data:** - - `title` - From task.title - - `description` - From task.description - - `outline` - From task.idea.description (handles JSON structure): - - If JSON, formats as: `"## {H2 heading}\n### {H3 subheading}\nContent Type: {type}\nDetails: {details}"` - - If plain text, uses as-is - - `structure` - From task.idea.content_structure or task.content_structure - - `type` - From task.idea.content_type or task.content_type - - `estimated_word_count` - From task.idea.estimated_word_count -- **Builds Cluster Data:** - - `cluster_name` - From task.cluster.name - - `description` - From task.cluster.description - - `status` - From task.cluster.status -- **Builds Keywords Data:** - - From task.keywords (legacy) or task.idea.target_keywords -- **Gets Prompt:** - - Calls `PromptRegistry.get_prompt(function_name='generate_content', account, task, context)` - - Context includes: - - `IDEA` - Formatted idea data string - - `CLUSTER` - Formatted cluster data string - - `KEYWORDS` - Keywords string -- **Returns:** Prompt string - -#### `parse_response(response, step_tracker)` -- **Parsing:** - - First tries JSON parse: - - If successful and dict, returns dict - - Falls back to plain text: - - Calls `normalize_content()` from `content_normalizer` to convert to HTML - - Returns dict with `content` field -- **Returns:** Dict with: - - **If JSON:** - - `content` (str) - HTML content - - `title` (str, optional) - Content title - - `meta_title` (str, optional) - Meta title - - `meta_description` (str, optional) - Meta description - - `word_count` (int, optional) - Word count - - `primary_keyword` (str, optional) - Primary keyword - - `secondary_keywords` (list, optional) - Secondary keywords - - `tags` (list, optional) - Tags - - `categories` (list, optional) - Categories - - **If Plain Text:** - - `content` (str) - Normalized HTML content - -#### `save_output(parsed, original_data, account, progress_tracker, step_tracker)` -- **Input:** - - `parsed` - Dict from parse_response - - `original_data` - Task object or list[Task] (handles first task if list) -- **Process:** - - Extracts content fields from parsed dict: - - `content_html` - From `content` field - - `title` - From `title` or task.title - - `meta_title` - From `meta_title` or task.meta_title or task.title - - `meta_description` - From `meta_description` or task.meta_description or task.description - - `word_count` - From `word_count` or calculated from content - - `primary_keyword` - From `primary_keyword` - - `secondary_keywords` - From `secondary_keywords` (converts to list if needed) - - `tags` - From `tags` (converts to list if needed) - - `categories` - From `categories` (converts to list if needed) - - Calculates word count if not provided: - - Strips HTML tags and counts words - - Gets or creates `Content` record: - - Uses `get_or_create` with `task` (OneToOne relationship) - - Defaults: `html_content`, `word_count`, `status='draft'`, `account`, `site`, `sector` - - Updates `Content` fields: - - `html_content` - Content HTML - - `word_count` - Word count - - `title` - Content title - - `meta_title` - Meta title - - `meta_description` - Meta description - - `primary_keyword` - Primary keyword - - `secondary_keywords` - Secondary keywords (JSONField) - - `tags` - Tags (JSONField) - - `categories` - Categories (JSONField) - - `status` - Always 'draft' for newly generated content - - `metadata` - Extra fields from parsed dict (excludes standard fields) - - `account`, `site`, `sector`, `task` - Aligned from task - - Updates `Tasks` record: - - Sets `status='completed'` - - Updates `updated_at` -- **Returns:** Dict with: - - `count` (int) - Tasks updated (always 1 per task) - - `tasks_updated` (int) - Tasks updated - - `word_count` (int) - Word count - -### 4.4 Database Models - -#### Tasks Model -- **File:** `backend/igny8_core/modules/writer/models.py` -- **Model:** `Tasks` -- **Fields Used:** - - `id` - Task ID - - `title` - Task title - - `description` - Task description - - `keywords` - Keywords string (legacy) - - `cluster` (ForeignKey) - Related cluster - - `idea` (ForeignKey) - Related ContentIdeas - - `content_structure` - Content structure - - `content_type` - Content type - - `status` - Status ('queued', 'completed') - - `meta_title` - Meta title - - `meta_description` - Meta description - - `account`, `site`, `sector` - From SiteSectorBaseModel - -#### Content Model -- **File:** `backend/igny8_core/modules/writer/models.py` -- **Model:** `Content` -- **Fields Used:** - - `task` (OneToOneField) - Related task - - `html_content` - HTML content - - `word_count` - Word count - - `title` - Content title - - `meta_title` - Meta title - - `meta_description` - Meta description - - `primary_keyword` - Primary keyword - - `secondary_keywords` (JSONField) - Secondary keywords list - - `tags` (JSONField) - Tags list - - `categories` (JSONField) - Categories list - - `status` - Status ('draft', 'review', 'published') - - `metadata` (JSONField) - Additional metadata - - `account`, `site`, `sector` - From SiteSectorBaseModel (auto-set from task) - -### 4.5 AI Response Format - -**Expected JSON:** -```json -{ - "content": "Content HTML", - "title": "Content Title", - "meta_title": "Meta Title", - "meta_description": "Meta description", - "word_count": 1500, - "primary_keyword": "primary keyword", - "secondary_keywords": ["keyword1", "keyword2"], - "tags": ["tag1", "tag2"], - "categories": ["category1"] -} -``` - -**Or Plain Text:** -``` -Plain text content that will be normalized to HTML -``` - -### 4.6 Progress Messages - -- **INIT:** "Validating task" -- **PREP:** "Preparing content idea" -- **AI_CALL:** "Writing article with Igny8 Semantic AI" -- **PARSE:** "{count} article(s) created" -- **SAVE:** "Saving article" - ---- - -## 5. Change Guide - -### 5.1 Where to Change Validation Logic - -- **File:** `backend/igny8_core/ai/validators.py` -- **Functions:** `validate_ids`, `validate_keywords_exist`, `validate_cluster_exists`, `validate_tasks_exist` -- **Or:** Override `validate()` method in function class - -### 5.2 Where to Change Data Loading - -- **File:** Function-specific file (e.g., `auto_cluster.py`) -- **Method:** `prepare()` -- **Change:** Modify queryset filters, select_related, prefetch_related - -### 5.3 Where to Change Prompts - -- **File:** `backend/igny8_core/ai/prompts.py` -- **Method:** `PromptRegistry.get_prompt()` -- **Change:** Modify `DEFAULT_PROMPTS` dict or update database prompts - -### 5.4 Where to Change Model Configuration - -- **File:** `backend/igny8_core/ai/settings.py` -- **Constant:** `MODEL_CONFIG` -- **Change:** Update model, max_tokens, temperature, response_format per function - -### 5.5 Where to Change Response Parsing - -- **File:** Function-specific file (e.g., `generate_content.py`) -- **Method:** `parse_response()` -- **Change:** Modify JSON extraction or plain text handling - -### 5.6 Where to Change Database Saving - -- **File:** Function-specific file (e.g., `auto_cluster.py`) -- **Method:** `save_output()` -- **Change:** Modify model creation/update logic, field mappings - -### 5.7 Where to Change Progress Messages - -- **File:** `backend/igny8_core/ai/engine.py` -- **Methods:** `_get_prep_message()`, `_get_ai_call_message()`, `_get_parse_message()`, `_get_save_message()` -- **Or:** Override in function class `get_metadata()` phases - -### 5.8 Where to Change Error Handling - -- **File:** `backend/igny8_core/ai/engine.py` -- **Method:** `_handle_error()` -- **Change:** Modify error logging, error response format - ---- - -## 6. Dependencies - -### 6.1 Function Dependencies - -- All functions depend on: `BaseAIFunction`, `AICore`, `PromptRegistry`, `get_model_config` -- Clustering depends on: `Keywords`, `Clusters` models -- Ideas depends on: `Clusters`, `ContentIdeas`, `Keywords` models -- Content depends on: `Tasks`, `Content`, `ContentIdeas`, `Clusters` models - -### 6.2 External Dependencies - -- **Celery:** For async task execution (`run_ai_task`) -- **OpenAI API:** For AI text generation (via `AICore.run_ai_request`) -- **Django ORM:** For database operations -- **IntegrationSettings:** For account-specific model configuration - ---- - -## 7. Key Relationships - -### 7.1 Clustering Flow -``` -Keywords → Clusters (many-to-one) -- Keywords.cluster (ForeignKey) -- Clusters.keywords (related_name) -``` - -### 7.2 Ideas Flow -``` -Clusters → ContentIdeas (one-to-many) -- ContentIdeas.keyword_cluster (ForeignKey) -- Clusters.ideas (related_name, if exists) -``` - -### 7.3 Content Flow -``` -Tasks → Content (one-to-one) -- Content.task (OneToOneField) -- Tasks.content_record (related_name) - -Tasks → ContentIdeas (many-to-one) -- Tasks.idea (ForeignKey) -- ContentIdeas.tasks (related_name) - -Tasks → Clusters (many-to-one) -- Tasks.cluster (ForeignKey) -- Clusters.tasks (related_name) -``` - ---- - -## 8. Notes - -- All functions use the same execution pipeline through `AIEngine.execute()` -- Progress tracking is handled automatically by `AIEngine` -- Cost tracking is handled automatically by `CostTracker` -- Database logging is handled automatically by `AITaskLog` -- Model configuration can be overridden per account via `IntegrationSettings` -- Prompts can be overridden per account via database prompts -- All functions support both async (Celery) and sync execution -- Error handling is centralized in `AIEngine._handle_error()` -