salman/igny8
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

rearrange

Browse Source
This commit is contained in:
alorig
2025-11-18 04:16:37 +05:00
parent 8040d983de
commit 342d9eab17
17 changed files with 0 additions and 261 deletions
Download Patch File Download Diff File Expand all files Collapse all files

411
master-docs/00-DOCUMENTATION-MANAGEMENT.md Normal file
View File

@@ -0,0 +1,411 @@
# IGNY8 Documentation & Changelog Management System
**Last Updated:** 2025-01-XX
**Purpose:** Complete guide for managing documentation versioning, changelog updates, and DRY principles. This document must be read by all AI agents at the start of any session.
---
## Table of Contents
1. [Versioning System](#versioning-system)
2. [Changelog Management](#changelog-management)
3. [Documentation Update Process](#documentation-update-process)
4. [DRY Principles & Standards](#dry-principles--standards)
5. [AI Agent Instructions](#ai-agent-instructions)
---
## Versioning System
### Version Format
**Format:** `MAJOR.MINOR.PATCH` (Semantic Versioning)
- **MAJOR**: Breaking changes, major feature additions, architecture changes
- **MINOR**: New features, new modules, significant enhancements
- **PATCH**: Bug fixes, small improvements, documentation updates
**Current Version:** `1.0.0`
### Version Tracking
**Location:**
- Root `CHANGELOG.md` - Main version history
- Each documentation file header - Last updated date
**Version Update Rules:**
- **MAJOR**: Only updated when user confirms major release
- **MINOR**: Updated when user confirms new feature is complete
- **PATCH**: Updated when user confirms bug fix is complete
### Version Update Process
1. **Code Change Made**: Developer/AI makes code changes
2. **User Confirmation**: User confirms fix/feature is complete
3. **Version Update**: Update version in CHANGELOG.md
4. **Changelog Entry**: Add entry to CHANGELOG.md
5. **Documentation Update**: Update relevant documentation files if needed
**IMPORTANT**: Never update version or changelog without user confirmation that the change is complete and working.
---
## Changelog Management
### Changelog Location
**File:** `/CHANGELOG.md` (root directory)
### Changelog Structure
```markdown
## [Version] - YYYY-MM-DD
### Added
- New features, modules, or capabilities
### Changed
- Changes to existing features or behavior
### Fixed
- Bug fixes and corrections
### Deprecated
- Features that will be removed in future versions
### Removed
- Features that have been removed
### Security
- Security fixes and improvements
```
### Changelog Entry Format
Each entry must include:
- **Date**: YYYY-MM-DD format
- **Type**: Added, Changed, Fixed, Deprecated, Removed, Security
- **Description**: Clear, concise description of the change
- **Affected Areas**: Modules, components, or features affected
- **Documentation**: Reference to updated documentation files (if any)
### Example Changelog Entry
```markdown
## [1.0.1] - 2025-01-15
### Fixed
- Fixed keyword clustering issue where keywords were not properly linked to clusters
- **Affected**: Planner Module, Keyword Clustering
- **Documentation**: Updated 06-FUNCTIONAL-BUSINESS-LOGIC.md (Keyword Clustering section)
### Added
- Added bulk delete functionality for content ideas
- **Affected**: Planner Module, Content Ideas
- **Documentation**: Updated 06-FUNCTIONAL-BUSINESS-LOGIC.md (Content Ideas Management section)
```
### Changelog Update Rules
1. **Only Update After User Confirmation**: Never add changelog entries until user confirms the change is complete
2. **One Entry Per Change**: Each fix or feature gets its own entry
3. **Chronological Order**: Newest entries at the top
4. **Be Specific**: Include what was changed, why, and where
5. **Link Documentation**: Reference updated documentation files
6. **Version Bump**: Update version number when adding entries
### Changelog Categories
**Added**: New features, new modules, new endpoints, new functions
**Changed**: Modified existing features, updated workflows, refactored code
**Fixed**: Bug fixes, error corrections, issue resolutions
**Deprecated**: Features marked for removal (include migration path)
**Removed**: Features that have been completely removed
**Security**: Security patches, vulnerability fixes, access control updates
---
## Documentation Update Process
### When to Update Documentation
1. **New Feature Added**: Update relevant documentation files
2. **Feature Changed**: Update affected sections in documentation
3. **Bug Fixed**: Update documentation if behavior changed
4. **Workflow Modified**: Update workflow documentation
5. **API Changed**: Update API documentation
6. **Architecture Changed**: Update architecture documentation
### Documentation Files Structure
```
docs/
├── 00-DOCUMENTATION-MANAGEMENT.md # This file (management guide)
├── 01-TECH-STACK-AND-INFRASTRUCTURE.md
├── 02-APPLICATION-ARCHITECTURE.md
├── 03-FRONTEND-ARCHITECTURE.md
├── 04-BACKEND-IMPLEMENTATION.md
├── 05-AI-FRAMEWORK-IMPLEMENTATION.md
├── 06-FUNCTIONAL-BUSINESS-LOGIC.md
├── API-COMPLETE-REFERENCE.md
├── WORDPRESS-PLUGIN-INTEGRATION.md
├── planning/ # Architecture & implementation planning
│ ├── IGNY8-HOLISTIC-ARCHITECTURE-PLAN.md
│ ├── IGNY8-IMPLEMENTATION-PLAN.md
│ ├── Igny8-phase-2-plan.md
│ ├── CONTENT-WORKFLOW-DIAGRAM.md
│ ├── ARCHITECTURE_CONTEXT.md
│ └── sample-usage-limits-credit-system
└── refactor/ # Refactoring plans and documentation
├── README.md
├── routes/
├── folder-structure/
└── migrations/
```
### Documentation Update Checklist
- [ ] Identify which documentation file(s) need updating
- [ ] Update the relevant section(s)
- [ ] Update "Last Updated" date in file header
- [ ] Add changelog entry (after user confirmation)
- [ ] Verify all links still work
- [ ] Ensure consistency across all documentation
### Documentation Standards
1. **No Code Snippets**: Documentation focuses on workflows, features, and architecture
2. **Complete Coverage**: All features and workflows must be documented
3. **Accurate State**: Documentation must reflect current system state
4. **Clear Structure**: Use consistent headings and formatting
5. **Cross-References**: Link related sections and documents
---
## DRY Principles & Standards
### DRY (Don't Repeat Yourself) Philosophy
**Core Principle**: Use existing, predefined, standardized components, utilities, functions, and configurations instead of creating parallel systems or duplicating code.
### Frontend DRY Standards
#### Components
**MUST USE Existing Components:**
- **Templates**: Use 4 universal templates (DashboardTemplate, TablePageTemplate, FormPageTemplate, SystemPageTemplate)
- **UI Components**: Use components from `/frontend/src/components/`
- **Common Components**: Use ScrollToTop, GlobalErrorDisplay, LoadingStateMonitor
- **Form Components**: Use existing form components with props and configs
**DO NOT:**
- Create new templates when existing ones can be used
- Duplicate component logic
- Create parallel component systems
- Hardcode UI elements that can use config-driven approach
#### Configuration-Driven Development
**MUST USE Configuration Files:**
- **Page Configs**: `/frontend/src/config/pages/` - Define page structure
- **Snippet Configs**: `/frontend/src/config/snippets/` - Define reusable snippets
- **Route Configs**: `/frontend/src/config/routes.config.ts` - Define routes
- **API Configs**: Use existing API client patterns
**DO NOT:**
- Hardcode page structures
- Create pages without config files
- Duplicate configuration patterns
#### State Management
**MUST USE Existing Stores:**
- **Zustand Stores**: Use stores from `/frontend/src/stores/`
- Auth Store, Site Store, Sector Store
- Planner Store, Writer Store, Billing Store
- Settings Store, Page Size Store, Column Visibility Store
- **React Contexts**: Use contexts from `/frontend/src/contexts/`
- Theme Context, Sidebar Context, Header Metrics Context, Toast Context
**DO NOT:**
- Create new stores for existing functionality
- Duplicate state management logic
- Create parallel state systems
#### Utilities & Helpers
**MUST USE Existing Utilities:**
- **API Client**: Use `/frontend/src/services/api.ts` patterns
- **Hooks**: Use custom hooks from `/frontend/src/hooks/`
- **Utils**: Use utility functions from `/frontend/src/utils/`
- **Constants**: Use constants from `/frontend/src/constants/`
**DO NOT:**
- Create duplicate utility functions
- Implement API calls without using existing patterns
- Create new helper functions when existing ones work
#### CSS & Styling
**MUST USE:**
- **Tailwind CSS**: Use Tailwind utility classes
- **Existing Styles**: Use predefined styles and classes
- **Component Styles**: Use component-level styles from existing components
- **Theme System**: Use theme context for dark/light mode
**DO NOT:**
- Create custom CSS when Tailwind classes exist
- Duplicate styling patterns
- Create parallel style systems
- Hardcode colors or spacing values
### Backend DRY Standards
#### Base Classes
**MUST USE Existing Base Classes:**
- **AccountModelViewSet**: For account-isolated models
- **SiteSectorModelViewSet**: For site/sector-scoped models
- **AccountBaseModel**: For account-isolated models
- **SiteSectorBaseModel**: For site/sector-scoped models
**DO NOT:**
- Create new base classes when existing ones work
- Duplicate filtering logic
- Create parallel isolation systems
#### AI Framework
**MUST USE AI Framework:**
- **BaseAIFunction**: Inherit from this for all AI functions
- **AIEngine**: Use for executing AI functions
- **AICore**: Use for AI API calls
- **PromptRegistry**: Use for prompt management
- **run_ai_task**: Use this Celery task entry point
**DO NOT:**
- Create new AI function patterns
- Duplicate AI execution logic
- Create parallel AI systems
#### Utilities & Services
**MUST USE Existing Services:**
- **CreditService**: For credit management
- **Content Normalizer**: For content processing
- **AI Functions**: Use existing 5 AI functions
- **Middleware**: Use AccountContextMiddleware, ResourceTrackerMiddleware
**DO NOT:**
- Create duplicate service logic
- Implement credit management without CreditService
- Create parallel utility systems
### DRY Violation Detection
**Red Flags:**
- Creating new components when similar ones exist
- Duplicating API call patterns
- Creating new state management when stores exist
- Hardcoding values that should be config-driven
- Creating parallel systems for existing functionality
**Action Required:**
- Check existing components, utilities, and patterns first
- Refactor to use existing systems
- Update documentation if new patterns are truly needed
---
## AI Agent Instructions
### Mandatory Reading
**At the start of EVERY session, AI agents MUST:**
1. Read this file (`00-DOCUMENTATION-MANAGEMENT.md`)
2. Read root `README.md`
3. Read `CHANGELOG.md`
4. Understand versioning system
5. Understand changelog management
6. Understand DRY principles
### Versioning & Changelog Rules for AI Agents
1. **NEVER update version or changelog without user confirmation**
2. **ALWAYS ask user before adding changelog entries**
3. **ONLY update changelog after user confirms change is complete**
4. **ALWAYS follow changelog structure and format**
5. **ALWAYS reference updated documentation files in changelog**
### DRY Principles for AI Agents
1. **ALWAYS check for existing components/utilities before creating new ones**
2. **ALWAYS use configuration-driven approach when possible**
3. **ALWAYS use existing templates and base classes**
4. **NEVER create parallel systems**
5. **NEVER duplicate code that can be reused**
### Documentation Update Rules for AI Agents
1. **ALWAYS update documentation when making changes**
2. **ALWAYS update "Last Updated" date in file header**
3. **ALWAYS maintain consistency across documentation**
4. **ALWAYS verify links after updates**
5. **ALWAYS follow documentation standards**
### Workflow for AI Agents
**When Making Code Changes:**
1. Check existing components/utilities first (DRY)
2. Make code changes
3. Update relevant documentation
4. Wait for user confirmation
5. Add changelog entry (after confirmation)
6. Update version (if needed, after confirmation)
**When User Confirms Fix/Feature:**
1. Add changelog entry following structure
2. Update version if needed
3. Update documentation "Last Updated" dates
4. Verify all changes are documented
### Self-Explaining System
This documentation management system is designed to be self-explaining:
- **Clear Rules**: All rules are explicitly stated
- **Examples**: Examples provided for clarity
- **Structure**: Consistent structure across all documents
- **Cross-References**: Links between related documents
- **Standards**: Clear standards for all operations
**Any AI agent reading this file should understand:**
- How to manage versions
- How to update changelog
- How to follow DRY principles
- How to update documentation
- When to ask for user confirmation
---
## Summary
### Key Principles
1. **Versioning**: Semantic versioning, only update after user confirmation
2. **Changelog**: Structured entries, only after user confirmation
3. **Documentation**: Always update when making changes
4. **DRY**: Always use existing components, utilities, and patterns
5. **Confirmation**: Never update version/changelog without user confirmation
### Lock Status
**Documentation Management System**: ✅ **LOCKED**
This system is finalized and should not be changed without explicit user approval. All AI agents must follow these rules.
---
**Last Updated:** 2025-01-XX
**Version:** 1.0.0
**Status:** Locked

1153
master-docs/01-TECH-STACK-AND-INFRASTRUCTURE.md Normal file
View File

File diff suppressed because it is too large Load Diff

795
master-docs/02-APPLICATION-ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,795 @@
# IGNY8 Application Architecture
**Last Updated:** 2025-01-XX
**Purpose:** Complete application architecture documentation covering system hierarchy, user roles, access control, modules, workflows, data models, multi-tenancy, API architecture, and security.
---
## Table of Contents
1. [System Overview](#system-overview)
2. [System Hierarchy](#system-hierarchy)
3. [User Roles & Access Control](#user-roles--access-control)
4. [Module Organization](#module-organization)
5. [Complete Workflows](#complete-workflows)
6. [Data Models & Relationships](#data-models--relationships)
7. [Multi-Tenancy Architecture](#multi-tenancy-architecture)
8. [API Architecture](#api-architecture)
9. [Security Architecture](#security-architecture)
10. [Integration Points](#integration-points)
---
## System Overview
**IGNY8** is a full-stack SaaS platform for SEO keyword management and AI-driven content generation. The system provides a scalable, multi-account platform that enables users to manage keywords, generate content ideas, create AI-powered content, and publish to WordPress.
### Platform Capabilities
| Capability | Description | Module |
|------------|-------------|--------|
| **Keyword Management** | Import, organize, filter, and manage SEO keywords | Planner |
| **Keyword Clustering** | AI-powered semantic clustering of related keywords | Planner |
| **Content Ideas** | Generate content ideas from keyword clusters | Planner |
| **Content Generation** | AI-powered blog post and article generation | Writer |
| **Image Generation** | AI-powered image generation (DALL-E, Runware) | Writer |
| **WordPress Integration** | Direct publishing to WordPress sites | Writer |
| **Account Management** | Multi-account SaaS with user roles | Auth |
| **Billing & Credits** | Credit-based billing system | Billing |
| **AI Configuration** | Customizable AI prompts and settings | System |
---
## System Hierarchy
### Entity Relationships
```
Account (1) ──< (N) User
Account (1) ──< (1) Subscription ──> (1) Plan
Account (1) ──< (N) Site
Site (1) ──< (1-5) Sector
Sector (1) ──< (N) Keywords, Clusters, ContentIdeas, Tasks
Cluster (1) ──< (N) Keywords (Many-to-Many)
Cluster (1) ──< (N) ContentIdeas
ContentIdeas (1) ──< (N) Tasks
Task (1) ──> (1) Content
Task (1) ──< (N) Images
```
### Hierarchy Details
**Account Level**:
- Top-level organization/workspace
- Contains users, sites, subscriptions, and all data
- Has credit balance and plan assignment
- Status: active, suspended, trial, cancelled
**User Level**:
- Individual user accounts within an account
- Has role (developer, owner, admin, editor, viewer)
- Can belong to only one account
- Access controlled by role and site permissions
**Site Level**:
- Workspace within an account (1-N relationship)
- Can have multiple active sites simultaneously
- Has WordPress integration settings (URL, username, password)
- Can be associated with an industry
- Status: active, inactive, suspended
**Sector Level**:
- Content category within a site (1-5 per site)
- Organizes keywords, clusters, ideas, and tasks
- Can reference an industry sector template
- Status: active, inactive
**Content Level**:
- Keywords, Clusters, ContentIdeas belong to Sector
- Tasks, Content, Images belong to Sector
- All content is automatically associated with Account and Site
---
## User Roles & Access Control
### User Roles
| Role | Access Level | Description | Permissions |
|------|--------------|-------------|-------------|
| **Developer** | System-wide | Full system access, bypasses all restrictions | All accounts, all sites, all data, system settings |
| **Owner** | Account-wide | Full account access, can manage users and billing | All sites in account, user management, billing |
| **Admin** | Account-wide | Account admin access, can manage content and users | All sites in account, content management, user management |
| **Editor** | Site-specific | Content editing access, can manage clusters/tasks | Granted sites only, content editing |
| **Viewer** | Site-specific | Read-only access | Granted sites only, read-only |
| **System Bot** | System-wide | System automation user | All accounts, all sites, all data |
### Access Control Matrix
| User Role | Account Access | Site Access | Data Access | User Management | Billing |
|-----------|----------------|-------------|-------------|------------------|---------|
| Developer | All accounts | All sites | All data | Yes | Yes |
| System Bot | All accounts | All sites | All data | No | No |
| Owner | Own account | All sites in account | All data in account | Yes | Yes |
| Admin | Own account | All sites in account | All data in account | Yes | No |
| Editor | Own account | Granted sites only | Data in granted sites | No | No |
| Viewer | Own account | Granted sites only | Read-only in granted sites | No | No |
### Site Access Control
**Automatic Access**:
- Owners and Admins: Automatic access to all sites in their account
- Developers and System Bot: Access to all sites across all accounts
**Explicit Access**:
- Editors and Viewers: Require explicit `SiteUserAccess` records
- Access granted by Owner or Admin
- Access can be revoked at any time
### Account Isolation
**Principle**: All data is automatically filtered by account.
**Implementation**:
- All models inherit `AccountBaseModel` with `account` ForeignKey
- All ViewSets inherit `AccountModelViewSet` with automatic filtering
- Middleware sets `request.account` from JWT token
- Queries automatically filter by account (except for Developer/System Bot)
---
## Module Organization
### Module Structure
| Module | Purpose | Models | ViewSets | Celery Tasks |
|--------|---------|--------|----------|--------------|
| **Planner** | Keyword management & content planning | Keywords, Clusters, ContentIdeas | KeywordViewSet, ClusterViewSet, ContentIdeasViewSet | auto_cluster_keywords_task, auto_generate_ideas_task |
| **Writer** | Content generation & management | Tasks, Content, Images | TasksViewSet, ImagesViewSet | auto_generate_content_task, auto_generate_images_task |
| **System** | Settings, prompts, integrations | AIPrompt, IntegrationSettings, AuthorProfile, Strategy | AIPromptViewSet, IntegrationSettingsViewSet, AuthorProfileViewSet | - |
| **Billing** | Credits, transactions, usage | CreditTransaction, CreditUsageLog | CreditTransactionViewSet, CreditUsageLogViewSet | - |
| **Auth** | Multi-tenancy, users, accounts | Account, User, Plan, Site, Sector | AccountViewSet, UserViewSet, SiteViewSet, SectorViewSet | - |
### Module Dependencies
```
Auth (Core)
├── Planner (depends on Auth)
├── Writer (depends on Auth, Planner)
├── System (depends on Auth)
└── Billing (depends on Auth)
```
### Module Features
#### Planner Module
**Purpose**: Keyword research, clustering, and content idea generation
**Features**:
- Keyword import (CSV/manual)
- Keyword filtering and organization
- AI-powered keyword clustering
- Content idea generation from clusters
- Keyword-to-cluster mapping
- Cluster metrics and analytics
**Models**:
- `Keywords`: Individual keywords with metrics (volume, difficulty, intent)
- `Clusters`: Groups of related keywords
- `ContentIdeas`: Content ideas generated from clusters
#### Writer Module
**Purpose**: Content generation, image generation, and publishing
**Features**:
- Task creation from content ideas
- AI-powered content generation
- Content editing and review
- Image prompt extraction
- AI-powered image generation
- WordPress publishing
**Models**:
- `Tasks`: Content generation tasks
- `Content`: Generated HTML content
- `Images`: Generated images (featured, in-article)
#### System Module
**Purpose**: System configuration and AI settings
**Features**:
- AI prompt management
- Integration settings (OpenAI, Runware)
- Author profile management
- Content strategy management
- System status and monitoring
**Models**:
- `AIPrompt`: AI prompt templates
- `IntegrationSettings`: API keys and configuration
- `AuthorProfile`: Writing style profiles
- `Strategy`: Content strategies
#### Billing Module
**Purpose**: Credit management and usage tracking
**Features**:
- Credit balance tracking
- Credit transactions
- Usage logging
- Cost tracking
**Models**:
- `CreditTransaction`: Credit additions and deductions
- `CreditUsageLog`: Usage tracking with cost
#### Auth Module
**Purpose**: Multi-tenancy and user management
**Features**:
- Account management
- User management
- Plan management
- Site and sector management
- Industry templates
**Models**:
- `Account`: Top-level organization
- `User`: User accounts
- `Plan`: Subscription plans
- `Site`: Sites within accounts
- `Sector`: Sectors within sites
- `Industry`: Global industry templates
- `IndustrySector`: Industry sector templates
---
## Complete Workflows
### 1. Account Setup Workflow
**Purpose**: Onboard a new account and user
**Steps**:
1. User signs up via `/signup`
2. Account created with default plan
3. Owner user created and linked to account
4. User signs in via `/signin`
5. JWT token generated and returned
6. Frontend stores token and redirects to dashboard
7. User creates first site (optional)
8. User creates sectors (1-5 per site, optional)
9. User configures integration settings (OpenAI, Runware)
10. System ready for use
**Data Created**:
- 1 Account record
- 1 User record (owner role)
- 1 Subscription record (default plan)
- 0-N Site records
- 0-N Sector records (per site)
- 1 IntegrationSettings record (per integration type)
### 2. Keyword Management Workflow
**Purpose**: Import, organize, and cluster keywords
**Steps**:
1. User navigates to `/planner/keywords`
2. User imports keywords via CSV or manual entry
3. Keywords validated and stored in database
4. Keywords displayed in table with filters
5. User filters keywords by sector, status, intent, etc.
6. User selects keywords for clustering
7. User clicks "Auto Cluster" action
8. Backend validates keyword IDs
9. Celery task queued (`auto_cluster_keywords_task`)
10. Task ID returned to frontend
11. Frontend polls progress endpoint
12. Celery worker processes task:
- Loads keywords from database
- Builds AI prompt with keyword data
- Calls OpenAI API for clustering
- Parses cluster response
- Creates Cluster records
- Links keywords to clusters
13. Progress updates sent to frontend
14. Task completes
15. Frontend displays new clusters
16. User can view clusters, edit cluster names, add/remove keywords
**Data Created**:
- N Keyword records (if imported)
- M Cluster records (created by AI)
- N-M Keyword-to-Cluster relationships
**AI Function**: Auto Cluster Keywords
### 3. Content Idea Generation Workflow
**Purpose**: Generate content ideas from keyword clusters
**Steps**:
1. User navigates to `/planner/clusters`
2. User selects one or more clusters
3. User clicks "Generate Ideas" action
4. Backend validates cluster IDs
5. Celery task queued (`auto_generate_ideas_task`)
6. Task ID returned to frontend
7. Frontend polls progress endpoint
8. Celery worker processes task:
- Loads clusters and keywords
- Builds AI prompt with cluster data
- Calls OpenAI API for idea generation
- Parses idea response
- Creates ContentIdeas records
- Links ideas to clusters
9. Progress updates sent to frontend
10. Task completes
11. Frontend displays new content ideas
12. User can view ideas, edit titles, prioritize ideas
**Data Created**:
- N ContentIdeas records (created by AI)
- N ContentIdeas-to-Cluster relationships
**AI Function**: Generate Ideas
### 4. Content Generation Workflow
**Purpose**: Generate AI-powered content from content ideas
**Steps**:
1. User navigates to `/planner/ideas`
2. User selects one or more content ideas
3. User clicks "Create Tasks" action
4. Task records created for each idea
5. User navigates to `/writer/tasks`
6. User selects tasks for content generation
7. User clicks "Generate Content" action
8. Backend validates task IDs
9. Celery task queued (`auto_generate_content_task`)
10. Task ID returned to frontend
11. Frontend polls progress endpoint
12. Celery worker processes task:
- Loads tasks and related data (cluster, keywords, idea)
- Builds AI prompt with task data
- Calls OpenAI API for content generation
- Parses HTML content response
- Creates/updates Content records
- Updates task status
13. Progress updates sent to frontend
14. Task completes
15. Frontend displays generated content
16. User can review and edit content
17. User navigates to `/writer/content` to view content
**Data Created**:
- N Task records (from ideas)
- N Content records (generated HTML)
**AI Function**: Generate Content
### 5. Image Generation Workflow
**Purpose**: Generate images for content
**Steps**:
1. User navigates to `/writer/content`
2. User selects content items
3. User clicks "Generate Image Prompts" action (optional, can skip)
4. Backend validates content IDs
5. Celery task queued (`auto_generate_image_prompts_task`)
6. Task processes:
- Loads content HTML
- Builds AI prompt for prompt extraction
- Calls OpenAI API
- Parses image prompts (featured, in-article)
- Creates/updates Images records with prompts
7. User clicks "Generate Images" action
8. Backend validates image IDs
9. Celery task queued (`auto_generate_images_task`)
10. Task processes:
- Loads images with prompts
- Builds image generation prompt
- Calls OpenAI DALL-E or Runware API
- Receives image URLs
- Updates Images records with URLs
11. Images displayed in frontend
12. User can view images, regenerate, or delete
**Data Created**:
- N Images records (with prompts)
- N Images records (with image URLs)
**AI Functions**: Generate Image Prompts, Generate Images
### 6. WordPress Publishing Workflow
**Purpose**: Publish content to WordPress
**Steps**:
1. User navigates to `/writer/content`
2. User selects content to publish
3. User clicks "Publish to WordPress" action
4. Backend validates:
- Site has WordPress URL configured
- Site has WordPress credentials
- Content is ready (status: review or draft)
5. Backend calls WordPress REST API:
- Creates post with content HTML
- Uploads featured image (if available)
- Uploads in-article images (if available)
- Sets post status (draft, publish)
6. WordPress post ID stored in Content record
7. Content status updated to "published"
8. Frontend displays success message
9. User can view published content in WordPress
**Data Updated**:
- Content record: `wp_post_id`, `status` = "published"
**Integration**: WordPress REST API
### 7. User Management Workflow
**Purpose**: Add users to account and manage permissions
**Steps**:
1. Owner/Admin navigates to `/settings/users`
2. User clicks "Add User" action
3. User enters email and selects role
4. Backend creates User record:
- Email validated (unique per account)
- Role assigned
- Account linked
- Password set (or invitation sent)
5. If role is Editor/Viewer:
- Owner/Admin grants site access
- SiteUserAccess records created
6. New user receives invitation email
7. New user signs in and accesses granted sites
**Data Created**:
- 1 User record
- 0-N SiteUserAccess records (for Editor/Viewer roles)
### 8. Integration Configuration Workflow
**Purpose**: Configure AI service integrations
**Steps**:
1. User navigates to `/settings/integration`
2. User selects integration type (OpenAI, Runware)
3. User enters API key
4. User clicks "Test Connection" (optional)
5. Backend validates API key:
- Makes test API call
- Returns connection status
6. User saves integration settings
7. Backend stores API key in IntegrationSettings
8. Integration ready for use in AI functions
**Data Created/Updated**:
- 1 IntegrationSettings record (per integration type)
**Test Functions**: Test OpenAI, Test Runware
---
## Data Models & Relationships
### Core Models
**Account Model**:
- Fields: name, slug, owner, plan, credits, status
- Relationships: Users (1-N), Sites (1-N), Subscription (1-1)
**User Model**:
- Fields: email, account, role
- Relationships: Account (N-1), SiteUserAccess (N-M via SiteUserAccess)
**Plan Model**:
- Fields: name, price, limits (users, sites, keywords, clusters, etc.), credits
- Relationships: Accounts (1-N via Subscription)
**Site Model**:
- Fields: name, slug, domain, industry, status, wp_url, wp_username, wp_app_password
- Relationships: Account (N-1), Sectors (1-N), Industry (N-1)
**Sector Model**:
- Fields: name, slug, site, industry_sector, status
- Relationships: Site (N-1), IndustrySector (N-1), Keywords/Clusters/Ideas/Tasks (1-N)
### Planner Models
**Keywords Model**:
- Fields: keyword, volume, difficulty, intent, cluster (M-N)
- Relationships: Sector (N-1), Clusters (M-N)
**Clusters Model**:
- Fields: name, description, keywords_count, volume
- Relationships: Sector (N-1), Keywords (M-N), ContentIdeas (1-N)
**ContentIdeas Model**:
- Fields: idea_title, description, cluster, status
- Relationships: Sector (N-1), Cluster (N-1), Tasks (1-N)
### Writer Models
**Tasks Model**:
- Fields: title, description, cluster, idea, status, content
- Relationships: Sector (N-1), Cluster (N-1), ContentIdeas (N-1), Content (1-1), Images (1-N)
**Content Model**:
- Fields: task, html_content, word_count, status, wp_post_id
- Relationships: Task (1-1), Images (1-N)
**Images Model**:
- Fields: task, content, image_type, prompt, image_url, status
- Relationships: Task (N-1), Content (N-1)
### System Models
**AIPrompt Model**:
- Fields: prompt_type, prompt_value, account
- Relationships: Account (N-1)
**IntegrationSettings Model**:
- Fields: integration_type, config (JSON), account
- Relationships: Account (N-1)
**AuthorProfile Model**:
- Fields: name, description, tone, language, account
- Relationships: Account (N-1)
**Strategy Model**:
- Fields: name, description, sector, prompt_types, account
- Relationships: Account (N-1), Sector (N-1)
### Billing Models
**CreditTransaction Model**:
- Fields: account, transaction_type, amount, balance_after
- Relationships: Account (N-1)
**CreditUsageLog Model**:
- Fields: account, operation_type, credits_used, cost_usd
- Relationships: Account (N-1)
---
## Multi-Tenancy Architecture
### Account Isolation
**Implementation Levels**:
1. **Model Level**: All models inherit `AccountBaseModel` with `account` ForeignKey
2. **ViewSet Level**: All ViewSets inherit `AccountModelViewSet` with automatic filtering
3. **Middleware Level**: `AccountContextMiddleware` sets `request.account` from JWT
**Filtering**:
- All queries automatically filter by `account = request.account`
- Developer and System Bot users bypass account filtering
- System accounts bypass account filtering
### Site/Sector Hierarchy
**Purpose**: Organize content within accounts
**Structure**:
```
Account
└── Site 1 (Active)
├── Sector 1 (Active)
│ ├── Keywords
│ ├── Clusters
│ ├── ContentIdeas
│ └── Tasks
├── Sector 2 (Active)
└── Sector 3 (Inactive)
└── Site 2 (Active)
└── Sector 1 (Active)
```
**Constraints**:
- Maximum 5 active sectors per site
- Multiple sites can be active simultaneously
- All content must belong to a site and sector
### Data Isolation Flow
```
Request with JWT Token
↓
AccountContextMiddleware
├── Extract Account ID from JWT
├── Load Account Object
└── Set request.account
↓
ViewSet.get_queryset()
├── Check User Role
├── Filter by Account (if not admin/developer)
└── Filter by Accessible Sites (if not owner/admin)
↓
Database Query
↓
Results (Account-Isolated)
```
---
## API Architecture
### API Structure
**Base URL**: `/api/v1/`
**Module Endpoints**:
- `/api/v1/auth/` - Accounts, users, sites, sectors, plans
- `/api/v1/planner/` - Keywords, clusters, ideas
- `/api/v1/writer/` - Tasks, content, images
- `/api/v1/system/` - Prompts, integrations, author-profiles, strategies
- `/api/v1/billing/` - Credits, transactions, usage
### Authentication Flow
```
1. User Signs In
↓
2. Backend Validates Credentials
↓
3. JWT Token Generated
↓
4. Token Returned to Frontend
↓
5. Frontend Stores Token (localStorage)
↓
6. Frontend Includes Token in Requests (Authorization: Bearer {token})
↓
7. Backend Validates Token
↓
8. Account Context Set
↓
9. Request Processed
```
### Response Format
**Success Response**:
- `success`: true
- `data`: Response data
- `message`: Optional message
**Error Response**:
- `success`: false
- `message`: Error message
- `errors`: Validation errors (if applicable)
**Pagination Response**:
- `count`: Total count
- `next`: Next page URL
- `previous`: Previous page URL
- `results`: Array of results
---
## Security Architecture
### Authentication
**Methods**:
- JWT (JSON Web Tokens) - Primary method
- Session-based auth - Fallback for admin
**Token Management**:
- Tokens stored in localStorage
- Tokens included in `Authorization: Bearer {token}` header
- Token refresh mechanism (future implementation)
### Authorization
**Role-Based Access Control (RBAC)**:
- Role checked on every request
- Permissions enforced at ViewSet level
- Action-level permissions for sensitive operations
**Data Access Control**:
- Account-level: Automatic filtering by account
- Site-level: Filtering by accessible sites
- Action-level: Permission checks in ViewSet actions
### Data Security
**Account Isolation**:
- All queries filtered by account
- Admin/Developer override for system accounts
- No cross-account data leakage
**Site Access Control**:
- Users can only access granted sites
- Admin/Developer override for all sites
- Explicit access grants for Editor/Viewer roles
**API Security**:
- CORS configured for frontend domain
- CSRF enabled for session-based auth
- Input validation via DRF serializers
- Rate limiting (future implementation)
---
## Integration Points
### OpenAI Integration
**Purpose**: Text generation and image generation
**Configuration**:
- API key stored per account in `IntegrationSettings`
- Model selection per account
- Cost tracking per request
**Services Used**:
- GPT models for text generation
- DALL-E for image generation
### Runware Integration
**Purpose**: Alternative image generation service
**Configuration**:
- API key stored per account
- Model selection (e.g., `runware:97@1`)
- Image type selection (realistic, artistic, cartoon)
### WordPress Integration
**Purpose**: Content publishing
**Configuration**:
- WordPress URL per site
- Username and password stored per site
- REST API integration for publishing
**Workflow**:
1. Content generated in IGNY8
2. Images attached
3. Content published to WordPress via REST API
4. Status updated in IGNY8
### Stripe Integration (Planned)
**Purpose**: Payment processing
**Status**: Planned for future implementation
**Features**:
- Subscription management
- Payment processing
- Webhook integration
---
## Summary
The IGNY8 application architecture provides:
1. **Multi-Tenancy**: Complete account isolation with automatic filtering
2. **Hierarchical Organization**: Account > Site > Sector > Content structure
3. **Role-Based Access**: Granular permissions for different user roles
4. **Module-Based Design**: Clear separation of concerns across modules
5. **Complete Workflows**: End-to-end workflows from keyword import to publishing
6. **AI Integration**: Unified AI framework for all AI operations
7. **WordPress Integration**: Direct publishing to WordPress sites
8. **Credit System**: Credit-based billing and usage tracking
9. **Security First**: JWT auth, RBAC, data isolation
10. **Scalable Design**: Supports multiple accounts, sites, and users
This architecture ensures scalability, maintainability, and extensibility while providing a robust foundation for the IGNY8 platform.

1492
master-docs/03-FRONTEND-ARCHITECTURE.md Normal file
View File

File diff suppressed because it is too large Load Diff

817
master-docs/04-BACKEND-IMPLEMENTATION.md Normal file
View File

@@ -0,0 +1,817 @@
# IGNY8 Backend Implementation Reference
**Last Updated:** 2025-01-XX
**Purpose:** Complete backend implementation reference covering project structure, models, ViewSets, serializers, Celery tasks, API endpoints, base classes, middleware, and utilities.
---
## Table of Contents
1. [Backend Overview](#backend-overview)
2. [Tech Stack](#tech-stack)
3. [Project Structure](#project-structure)
4. [Models](#models)
5. [ViewSets](#viewsets)
6. [Serializers](#serializers)
7. [Celery Tasks](#celery-tasks)
8. [API Endpoints](#api-endpoints)
9. [Base Classes](#base-classes)
10. [Middleware](#middleware)
11. [Utilities](#utilities)
12. [Modules](#modules)
---
## Backend Overview
The IGNY8 backend is a Django 5.2+ application using Django REST Framework (DRF) for API endpoints. The backend follows a modular architecture with clear separation of concerns, automatic account isolation, and support for asynchronous task processing via Celery.
### Key Features
- **Multi-Tenancy**: Complete account isolation with automatic filtering
- **RESTful API**: DRF ViewSets with consistent response format
- **Celery Integration**: Asynchronous task processing for long-running operations
- **Account/Site/Sector Hierarchy**: Hierarchical data organization
- **AI Integration**: Unified AI framework for all AI operations
- **Progress Tracking**: Real-time progress updates for Celery tasks
---
## Tech Stack
### Core Technologies
- **Django 5.2+**: Web framework
- **Django REST Framework**: API framework
- **PostgreSQL**: Database
- **Celery**: Asynchronous task queue
- **Redis**: Celery broker and caching
### Key Libraries
- **django-filter**: Advanced filtering
- **djangorestframework-simplejwt**: JWT authentication
- **requests**: HTTP client for external APIs
- **python-dotenv**: Environment variable management
---
## Project Structure
```
backend/igny8_core/
├── auth/ # Multi-tenancy and authentication
│ ├── models.py # Account, User, Plan, Site, Sector, Industry models
│ ├── views.py # Account, User, Site, Sector ViewSets
│ ├── serializers.py # Account, User, Plan serializers
│ └── urls.py # Auth module URLs
├── modules/ # Feature modules
│ ├── planner/ # Keywords, Clusters, Ideas
│ │ ├── models.py # Keywords, Clusters, ContentIdeas models
│ │ ├── views.py # KeywordViewSet, ClusterViewSet, ContentIdeasViewSet
│ │ ├── tasks.py # Celery tasks for AI operations
│ │ ├── serializers.py # Model serializers
│ │ └── urls.py # Planner module URLs
│ ├── writer/ # Tasks, Content, Images
│ │ ├── models.py # Tasks, Content, Images models
│ │ ├── views.py # TasksViewSet, ImagesViewSet
│ │ ├── tasks.py # Celery tasks for content/image generation
│ │ └── urls.py # Writer module URLs
│ ├── system/ # Settings, Prompts, Integration
│ │ ├── models.py # AIPrompt, IntegrationSettings, AuthorProfile, Strategy
│ │ ├── views.py # AIPromptViewSet, AuthorProfileViewSet
│ │ ├── integration_views.py # IntegrationSettingsViewSet, task_progress
│ │ ├── utils.py # Default prompts, prompt loading
│ │ └── urls.py # System module URLs
│ └── billing/ # Credits, Transactions, Usage
│ ├── models.py # CreditTransaction, CreditUsageLog models
│ ├── views.py # Billing ViewSets
│ └── services.py # CreditService
├── api/ # API base classes
│ ├── base.py # AccountModelViewSet, SiteSectorModelViewSet
│ └── pagination.py # CustomPageNumberPagination
├── utils/ # Shared utilities
│ ├── ai_processor.py # Unified AI interface (legacy, see AI functions)
│ └── content_normalizer.py # Content processing utilities
├── middleware/ # Custom middleware
│ ├── account.py # AccountContextMiddleware (sets request.account)
│ └── resource_tracker.py # ResourceTrackerMiddleware (API metrics)
├── ai/ # AI framework
│ ├── base.py # BaseAIFunction
│ ├── engine.py # AIEngine
│ ├── tasks.py # run_ai_task
│ ├── registry.py # Function registry
│ ├── prompts.py # PromptRegistry
│ ├── ai_core.py # AICore
│ ├── settings.py # Model configuration
│ ├── validators.py # Validation functions
│ ├── tracker.py # Progress tracking
│ └── functions/ # AI function implementations
│ ├── auto_cluster.py
│ ├── generate_ideas.py
│ ├── generate_content.py
│ ├── generate_image_prompts.py
│ └── generate_images.py
├── settings.py # Django settings
├── urls.py # Root URL configuration
└── celery.py # Celery configuration
```
---
## Models
### Base Models
#### AccountBaseModel
**File**: `auth/models.py`
**Purpose**: Base model for all account-isolated models.
**Fields**:
- `account`: ForeignKey to Account
- `created_at`: DateTimeField (auto_now_add)
- `updated_at`: DateTimeField (auto_now)
**Usage**: All models that need account isolation inherit from this.
#### SiteSectorBaseModel
**File**: `auth/models.py`
**Purpose**: Base model for models that belong to Site and Sector.
**Fields**:
- Inherits from `AccountBaseModel`
- `site`: ForeignKey to Site
- `sector`: ForeignKey to Sector
**Methods**:
- `save()`: Automatically sets `account` from `site.account` and validates sector belongs to site
**Usage**: Models like Keywords, Clusters, ContentIdeas, Tasks inherit from this.
### Auth Models
#### Account
**Table**: `igny8_accounts`
**Fields**:
- `name`: CharField
- `slug`: SlugField (unique)
- `owner`: ForeignKey to User
- `plan`: ForeignKey to Plan
- `credits`: IntegerField (default: 0)
- `status`: CharField (choices: active, suspended, trial, cancelled)
**Methods**:
- `is_system_account()`: Returns True if account is a system account
#### User
**Table**: `igny8_users`
**Inherits**: `AbstractUser`
**Fields**:
- `email`: EmailField (unique)
- `account`: ForeignKey to Account
- `role`: CharField (choices: developer, owner, admin, editor, viewer, system_bot)
**Methods**:
- `has_role(role)`: Checks if user has role
- `is_owner_or_admin()`: Checks if user is owner or admin
- `is_developer()`: Checks if user is developer
- `is_admin_or_developer()`: Checks if user is admin or developer
- `is_system_account_user()`: Checks if user belongs to system account
- `get_accessible_sites()`: Returns queryset of accessible sites
#### Plan
**Table**: `igny8_plans`
**Fields**: Extensive fields for limits (users, sites, keywords, clusters, content ideas, AI requests, word count, images, credits)
**Methods**:
- `clean()`: Validates plan limits
- `get_effective_credits_per_month()`: Returns included_credits or credits_per_month
#### Site
**Table**: `igny8_sites`
**Inherits**: `AccountBaseModel`
**Fields**:
- `name`: CharField
- `slug`: SlugField (unique per account)
- `domain`: URLField (optional)
- `industry`: ForeignKey to Industry (optional)
- `is_active`: BooleanField
- `status`: CharField
- `wp_url`: URLField (optional)
- `wp_username`: CharField (optional)
- `wp_app_password`: CharField (optional)
**Methods**:
- `get_active_sectors_count()`: Returns count of active sectors
- `can_add_sector()`: Returns True if site can add another sector (max 5)
#### Sector
**Table**: `igny8_sectors`
**Inherits**: `AccountBaseModel`
**Fields**:
- `site`: ForeignKey to Site
- `industry_sector`: ForeignKey to IndustrySector (optional)
- `name`: CharField
- `slug`: SlugField (unique per site)
- `is_active`: BooleanField
- `status`: CharField
**Validation**: Maximum 5 active sectors per site
### Planner Models
#### Keywords
**Table**: `igny8_keywords`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `keyword`: CharField
- `volume`: IntegerField (optional)
- `difficulty`: IntegerField (optional)
- `intent`: CharField (optional)
- `status`: CharField
- `cluster`: ManyToManyField to Clusters
#### Clusters
**Table**: `igny8_clusters`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `name`: CharField
- `description`: TextField (optional)
- `keywords_count`: IntegerField (calculated)
- `volume`: IntegerField (calculated)
- `status`: CharField
- `keywords`: ManyToManyField to Keywords
#### ContentIdeas
**Table**: `igny8_content_ideas`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `idea_title`: CharField
- `description`: TextField
- `content_type`: CharField
- `content_structure`: CharField
- `target_keywords`: TextField
- `keyword_cluster`: ForeignKey to Clusters
- `estimated_word_count`: IntegerField
- `status`: CharField
### Writer Models
#### Tasks
**Table**: `igny8_tasks`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `title`: CharField
- `description`: TextField
- `cluster`: ForeignKey to Clusters (optional)
- `idea`: ForeignKey to ContentIdeas (optional)
- `content_type`: CharField
- `content_structure`: CharField
- `status`: CharField
#### Content
**Table**: `igny8_content`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `task`: OneToOneField to Tasks
- `html_content`: TextField
- `word_count`: IntegerField
- `status`: CharField
- `wp_post_id`: IntegerField (optional)
- `meta_title`: CharField (optional)
- `meta_description`: TextField (optional)
- `primary_keyword`: CharField (optional)
- `secondary_keywords`: TextField (optional)
#### Images
**Table**: `igny8_images`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `task`: ForeignKey to Tasks (optional)
- `content`: ForeignKey to Content (optional)
- `image_type`: CharField (choices: featured, in_article, desktop, mobile)
- `prompt`: TextField
- `image_url`: CharField
- `status`: CharField
### System Models
#### AIPrompt
**Table**: `igny8_ai_prompts`
**Inherits**: `AccountBaseModel`
**Fields**:
- `prompt_type`: CharField
- `prompt_value`: TextField
- `function_name`: CharField (optional)
#### IntegrationSettings
**Table**: `igny8_integration_settings`
**Inherits**: `AccountBaseModel`
**Fields**:
- `integration_type`: CharField (choices: openai, runware, image_generation)
- `config`: JSONField
#### AuthorProfile
**Table**: `igny8_author_profiles`
**Inherits**: `AccountBaseModel`
**Fields**:
- `name`: CharField
- `description`: TextField
- `tone`: CharField
- `language`: CharField
#### Strategy
**Table**: `igny8_strategies`
**Inherits**: `AccountBaseModel`
**Fields**:
- `name`: CharField
- `description`: TextField
- `sector`: ForeignKey to Sector (optional)
- `prompt_types`: JSONField
- `section_logic`: JSONField
- `is_active`: BooleanField
### Billing Models
#### CreditTransaction
**Table**: `igny8_credit_transactions`
**Inherits**: `AccountBaseModel`
**Fields**:
- `transaction_type`: CharField (choices: purchase, usage, refund, adjustment)
- `amount`: IntegerField
- `balance_after`: IntegerField
- `description`: TextField (optional)
#### CreditUsageLog
**Table**: `igny8_credit_usage_logs`
**Inherits**: `AccountBaseModel`
**Fields**:
- `operation_type`: CharField
- `credits_used`: IntegerField
- `cost_usd`: DecimalField
- `details`: JSONField (optional)
---
## ViewSets
### Base ViewSets
#### AccountModelViewSet
**File**: `api/base.py`
**Purpose**: Base ViewSet with automatic account filtering.
**Methods**:
- `get_queryset()`: Filters queryset by `request.account` (with admin/developer override)
- `perform_create()`: Sets account on created objects
- `get_serializer_context()`: Adds account to serializer context
**Access Control**:
- Admin/Developer users: Bypass account filtering
- System account users: Bypass account filtering
- Regular users: Only see data from their account
#### SiteSectorModelViewSet
**File**: `api/base.py`
**Purpose**: Base ViewSet with site/sector filtering and access control.
**Inherits**: `AccountModelViewSet`
**Methods**:
- `get_queryset()`: Filters by account, accessible sites, and optional site_id/sector_id
- `perform_create()`: Validates site access and sector-site relationship
- `get_serializer_context()`: Adds accessible sites and sectors to context
**Access Control**:
- Developers: All active sites
- System account users: All active sites
- Owners/Admins: All sites in their account
- Editors/Viewers: Only sites granted via `SiteUserAccess`
### Planner ViewSets
#### KeywordViewSet
**Inherits**: `SiteSectorModelViewSet`
**Actions**:
- `list()`: List keywords with filtering
- `create()`: Create keyword
- `retrieve()`: Get keyword details
- `update()`: Update keyword
- `destroy()`: Delete keyword
- `auto_cluster()`: Auto-cluster keywords using AI
- `bulk_delete()`: Bulk delete keywords
- `bulk_update_status()`: Bulk update keyword status
- `export_csv()`: Export keywords to CSV
- `import_csv()`: Import keywords from CSV
**Filtering**:
- Search: `keyword` field
- Filters: `status`, `cluster_id`, `intent`
- Custom: `difficulty_min`, `difficulty_max`, `volume_min`, `volume_max`
- Ordering: `created_at`, `volume`, `difficulty`
#### ClusterViewSet
**Inherits**: `SiteSectorModelViewSet`
**Actions**:
- `list()`: List clusters
- `create()`: Create cluster
- `retrieve()`: Get cluster details
- `update()`: Update cluster
- `destroy()`: Delete cluster
- `auto_generate_ideas()`: Auto-generate content ideas for clusters
#### ContentIdeasViewSet
**Inherits**: `SiteSectorModelViewSet`
**Actions**:
- `list()`: List content ideas
- `create()`: Create content idea
- `retrieve()`: Get content idea details
- `update()`: Update content idea
- `destroy()`: Delete content idea
### Writer ViewSets
#### TasksViewSet
**Inherits**: `SiteSectorModelViewSet`
**Actions**:
- `list()`: List tasks
- `create()`: Create task
- `retrieve()`: Get task details
- `update()`: Update task
- `destroy()`: Delete task
- `auto_generate_content()`: Auto-generate content for tasks
- `generate_images()`: Generate images for image records
- `bulk_delete()`: Bulk delete tasks
- `bulk_update()`: Bulk update task status
**Filtering**:
- Search: `title`, `keywords`
- Filters: `status`, `cluster_id`, `content_type`, `content_structure`
- Ordering: `title`, `created_at`, `word_count`, `status`
#### ImagesViewSet
**Inherits**: `SiteSectorModelViewSet`
**Actions**:
- `list()`: List images
- `create()`: Create image
- `retrieve()`: Get image details
- `update()`: Update image
- `destroy()`: Delete image
- `generate_images()`: Generate images using AI
### System ViewSets
#### IntegrationSettingsViewSet
**Inherits**: `viewsets.ViewSet`
**Actions**:
- `list()`: List integrations
- `retrieve()`: Get integration settings
- `update()`: Save integration settings
- `save_post()`: Save integration settings (POST)
- `test_connection()`: Test API connection
- `test_openai()`: Test OpenAI connection
- `test_runware()`: Test Runware connection
- `generate_image()`: Test image generation
- `task_progress()`: Get Celery task progress
#### AIPromptViewSet
**Inherits**: `AccountModelViewSet`
**Actions**:
- `list()`: List prompts
- `create()`: Create prompt
- `retrieve()`: Get prompt details
- `update()`: Update prompt
- `destroy()`: Delete prompt
- `reset_to_default()`: Reset prompt to default value
#### AuthorProfileViewSet
**Inherits**: `AccountModelViewSet`
**Actions**:
- `list()`: List author profiles
- `create()`: Create author profile
- `retrieve()`: Get author profile details
- `update()`: Update author profile
- `destroy()`: Delete author profile
---
## Serializers
### Planner Serializers
#### KeywordSerializer
**Fields**: All Keyword model fields
**Validation**: Validates keyword uniqueness, cluster belongs to same sector
#### ClusterSerializer
**Fields**: All Cluster model fields
**Read-Only Fields**: `keywords_count`, `volume` (calculated)
#### ContentIdeasSerializer
**Fields**: All ContentIdeas model fields
### Writer Serializers
#### TasksSerializer
**Fields**: All Tasks model fields
#### ContentSerializer
**Fields**: All Content model fields
#### ImagesSerializer
**Fields**: All Images model fields
### System Serializers
#### AIPromptSerializer
**Fields**: All AIPrompt model fields
#### IntegrationSettingsSerializer
**Fields**: All IntegrationSettings model fields
---
## Celery Tasks
### AI Task Entry Point
#### run_ai_task
**File**: `ai/tasks.py`
**Purpose**: Unified Celery task entrypoint for all AI functions
**Parameters**:
- `function_name`: Function name (e.g., 'auto_cluster')
- `payload`: Function-specific payload
- `account_id`: Account ID
**Flow**:
1. Gets account from account_id
2. Gets function instance from registry
3. Creates AIEngine
4. Executes function via AIEngine
### Planner Tasks
#### auto_cluster_keywords_task
**File**: `modules/planner/tasks.py` (legacy, now uses `run_ai_task`)
**Purpose**: Auto-cluster keywords using AI
**Parameters**:
- `keyword_ids`: List of keyword IDs
- `account_id`: Account ID
- `site_id`: Site ID
- `sector_id`: Sector ID
**Progress Tracking**: Updates progress with request_steps and response_steps
#### auto_generate_ideas_task
**File**: `modules/planner/tasks.py` (legacy, now uses `run_ai_task`)
**Purpose**: Auto-generate content ideas for clusters
**Parameters**:
- `cluster_ids`: List of cluster IDs
- `account_id`: Account ID
**Progress Tracking**: Updates progress for each cluster
### Writer Tasks
#### auto_generate_content_task
**File**: `modules/writer/tasks.py` (legacy, now uses `run_ai_task`)
**Purpose**: Auto-generate content for tasks
**Parameters**:
- `task_ids`: List of task IDs
- `account_id`: Account ID
**Progress Tracking**: Updates progress for each task
#### process_image_generation_queue
**File**: `modules/writer/views.py` (via `ai/tasks.py`)
**Purpose**: Generate images for image records
**Parameters**:
- `image_ids`: List of image IDs
- `account_id`: Account ID
- `content_id`: Content ID (optional)
**Progress Tracking**: Updates progress for each image sequentially
---
## API Endpoints
### Base URL
`/api/v1/`
### Planner Endpoints
- `GET /api/v1/planner/keywords/` - List keywords
- `POST /api/v1/planner/keywords/` - Create keyword
- `GET /api/v1/planner/keywords/{id}/` - Get keyword
- `PUT /api/v1/planner/keywords/{id}/` - Update keyword
- `DELETE /api/v1/planner/keywords/{id}/` - Delete keyword
- `POST /api/v1/planner/keywords/auto_cluster/` - Auto-cluster keywords
- `POST /api/v1/planner/keywords/bulk_delete/` - Bulk delete keywords
- `POST /api/v1/planner/keywords/bulk_update_status/` - Bulk update status
- `GET /api/v1/planner/keywords/export_csv/` - Export keywords
- `POST /api/v1/planner/keywords/import_csv/` - Import keywords
- `GET /api/v1/planner/clusters/` - List clusters
- `POST /api/v1/planner/clusters/auto_generate_ideas/` - Auto-generate ideas
- `GET /api/v1/planner/ideas/` - List content ideas
### Writer Endpoints
- `GET /api/v1/writer/tasks/` - List tasks
- `POST /api/v1/writer/tasks/auto_generate_content/` - Auto-generate content
- `POST /api/v1/writer/images/generate_images/` - Generate images
### System Endpoints
- `GET /api/v1/system/settings/integrations/{pk}/` - Get integration settings
- `PUT /api/v1/system/settings/integrations/{pk}/` - Save integration settings
- `POST /api/v1/system/settings/integrations/{pk}/test_openai/` - Test OpenAI
- `POST /api/v1/system/settings/integrations/{pk}/test_runware/` - Test Runware
- `POST /api/v1/system/settings/integrations/{pk}/generate_image/` - Test image generation
- `GET /api/v1/system/settings/task_progress/{task_id}/` - Get task progress
---
## Base Classes
### AccountModelViewSet
**File**: `api/base.py`
**Purpose**: Base ViewSet with automatic account filtering
**Features**:
- Automatic account filtering
- Admin/Developer override
- Account context in serializers
### SiteSectorModelViewSet
**File**: `api/base.py`
**Purpose**: Base ViewSet with site/sector filtering
**Features**:
- Account filtering (inherited)
- Site access control
- Sector validation
- Accessible sites/sectors in serializer context
---
## Middleware
### AccountContextMiddleware
**File**: `middleware/account.py`
**Purpose**: Sets `request.account` from JWT token
**Functionality**:
- Extracts account ID from JWT token
- Loads Account object
- Sets `request.account`
### ResourceTrackerMiddleware
**File**: `middleware/resource_tracker.py`
**Purpose**: Tracks API request metrics
**Functionality**:
- Tracks CPU, memory, I/O usage
- Stores metrics in cache
- Provides metrics endpoint
---
## Utilities
### Content Normalizer
**File**: `utils/content_normalizer.py`
**Purpose**: Content processing utilities
**Functions**:
- `normalize_content()`: Converts plain text to HTML
- `_extract_body_content()`: Extracts body content from HTML
---
## Modules
### Planner Module
**Purpose**: Keyword management and content planning
**Models**: Keywords, Clusters, ContentIdeas
**ViewSets**: KeywordViewSet, ClusterViewSet, ContentIdeasViewSet
**Tasks**: Auto-cluster keywords, Auto-generate ideas
### Writer Module
**Purpose**: Content generation and management
**Models**: Tasks, Content, Images
**ViewSets**: TasksViewSet, ImagesViewSet
**Tasks**: Auto-generate content, Generate images
### System Module
**Purpose**: System configuration and AI settings
**Models**: AIPrompt, IntegrationSettings, AuthorProfile, Strategy
**ViewSets**: AIPromptViewSet, IntegrationSettingsViewSet, AuthorProfileViewSet
### Billing Module
**Purpose**: Credit management and usage tracking
**Models**: CreditTransaction, CreditUsageLog
**ViewSets**: CreditTransactionViewSet, CreditUsageLogViewSet
**Services**: CreditService (check, deduct, add, calculate credits)
---
## Summary
The IGNY8 backend provides:
1. **Multi-Tenancy**: Complete account isolation with automatic filtering
2. **RESTful API**: DRF ViewSets with consistent response format
3. **Celery Integration**: Asynchronous task processing
4. **Hierarchical Organization**: Account > Site > Sector > Content structure
5. **AI Framework**: Unified AI framework for all AI operations
6. **Progress Tracking**: Real-time progress updates for Celery tasks
7. **Module-Based Design**: Clear separation of concerns
8. **Base Classes**: Reusable ViewSets for common patterns
9. **Middleware**: Account context and resource tracking
10. **Utilities**: Content processing and AI integration
This architecture ensures scalability, maintainability, and extensibility while providing a robust foundation for the IGNY8 platform.

544
master-docs/05-AI-FRAMEWORK-IMPLEMENTATION.md Normal file
View File

@@ -0,0 +1,544 @@
# IGNY8 AI Framework Implementation Reference
**Last Updated:** 2025-01-XX
**Purpose:** Complete AI framework implementation reference covering architecture, code structure, all 5 AI functions, execution flow, progress tracking, cost tracking, prompt management, and model configuration.
---
## Table of Contents
1. [AI Framework Overview](#ai-framework-overview)
2. [Common Architecture](#common-architecture)
3. [AI Function Execution Flow](#ai-function-execution-flow)
4. [AI Functions](#ai-functions)
5. [Progress Tracking](#progress-tracking)
6. [Cost Tracking](#cost-tracking)
7. [Prompt Management](#prompt-management)
8. [Model Configuration](#model-configuration)
---
## AI Framework Overview
The IGNY8 AI framework provides a unified interface for all AI operations. All AI functions inherit from `BaseAIFunction` and are orchestrated by `AIEngine`, ensuring consistent execution, progress tracking, error handling, and cost tracking.
### Key Components
- **BaseAIFunction**: Abstract base class for all AI functions
- **AIEngine**: Central orchestrator managing lifecycle, progress, logging, cost tracking
- **AICore**: Centralized AI request handler for all AI operations
- **PromptRegistry**: Centralized prompt management with hierarchical resolution
- **Function Registry**: Lazy-loaded function registry
- **Progress Tracking**: Real-time progress updates via Celery
- **Cost Tracking**: Automatic cost and token tracking
### AI Functions
1. **Auto Cluster Keywords**: Group related keywords into semantic clusters
2. **Generate Ideas**: Generate content ideas from keyword clusters
3. **Generate Content**: Generate blog post and article content
4. **Generate Image Prompts**: Extract image prompts from content
5. **Generate Images**: Generate images using OpenAI DALL-E or Runware
---
## Common Architecture
### Core Framework Files
#### Entry Point
**File**: `backend/igny8_core/ai/tasks.py`
**Function**: `run_ai_task`
**Purpose**: Unified Celery task entrypoint for all AI functions
**Parameters**: `function_name` (str), `payload` (dict), `account_id` (int)
**Flow**: Loads function from registry → Creates AIEngine → Executes function
#### Engine Orchestrator
**File**: `backend/igny8_core/ai/engine.py`
**Class**: `AIEngine`
**Purpose**: Central orchestrator managing lifecycle, progress, logging, cost tracking
**Methods**:
- `execute` - Main execution pipeline (6 phases: INIT, PREP, AI_CALL, PARSE, SAVE, DONE)
- `_handle_error` - Centralized error handling
- `_log_to_database` - Logs to AITaskLog model
#### Base Function Class
**File**: `backend/igny8_core/ai/base.py`
**Class**: `BaseAIFunction`
**Purpose**: Abstract base class defining interface for all AI functions
**Abstract Methods**:
- `get_name` - Returns function name (e.g., 'auto_cluster')
- `prepare` - Loads and prepares data
- `build_prompt` - Builds AI prompt
- `parse_response` - Parses AI response
- `save_output` - Saves results to database
**Optional Methods**:
- `get_metadata` - Returns display name, description, phases
- `get_max_items` - Returns max items limit (or None)
- `validate` - Validates input payload (default: checks for 'ids')
- `get_model` - Returns model override (default: None, uses account default)
#### Function Registry
**File**: `backend/igny8_core/ai/registry.py`
**Functions**:
- `register_function` - Registers function class
- `register_lazy_function` - Registers lazy loader
- `get_function` - Gets function class by name (lazy loads if needed)
- `get_function_instance` - Gets function instance by name
- `list_functions` - Lists all registered functions
#### AI Core Handler
**File**: `backend/igny8_core/ai/ai_core.py`
**Class**: `AICore`
**Purpose**: Centralized AI request handler for all AI operations (text and image generation)
**Methods**:
- `run_ai_request` - Makes API call to OpenAI/Runware for text generation
- `generate_image` - Makes API call to OpenAI DALL-E or Runware for image generation
- `extract_json` - Extracts JSON from response (handles markdown code blocks)
#### Prompt Registry
**File**: `backend/igny8_core/ai/prompts.py`
**Class**: `PromptRegistry`
**Purpose**: Centralized prompt management with hierarchical resolution
**Method**: `get_prompt` - Gets prompt with resolution order:
1. Task-level prompt_override (if exists)
2. DB prompt for (account, function)
3. Default fallback from DEFAULT_PROMPTS registry
**Prompt Types**:
- `clustering` - For auto_cluster function
- `ideas` - For generate_ideas function
- `content_generation` - For generate_content function
- `image_prompt_extraction` - For extract_image_prompts function
- `image_prompt_template` - Template for formatting image prompts
- `negative_prompt` - Negative prompt for Runware image generation
#### Model Settings
**File**: `backend/igny8_core/ai/settings.py`
**Constants**: `FUNCTION_ALIASES` - Function name aliases for backward compatibility
**Functions**:
- `get_model_config(function_name, account)` - Gets model config from IntegrationSettings (account required, no fallbacks)
- Raises `ValueError` if IntegrationSettings not configured
- Returns dict with `model`, `max_tokens`, `temperature`, `response_format`
---
## AI Function Execution Flow
### Complete Execution Pipeline
```
1. API Endpoint (views.py)
↓
2. run_ai_task (tasks.py)
- Gets account from account_id
- Gets function instance from registry
- Creates AIEngine
↓
3. AIEngine.execute (engine.py)
Phase 1: INIT (0-10%)
- Calls function.validate()
- Updates progress tracker
↓
Phase 2: PREP (10-25%)
- Calls function.prepare()
- Calls function.build_prompt()
- Updates progress tracker
↓
Phase 3: AI_CALL (25-70%)
- Gets model config from settings
- Calls AICore.run_ai_request() or AICore.generate_image()
- Tracks cost and tokens
- Updates progress tracker
↓
Phase 4: PARSE (70-85%)
- Calls function.parse_response()
- Updates progress tracker
↓
Phase 5: SAVE (85-98%)
- Calls function.save_output()
- Logs credit usage
- Updates progress tracker
↓
Phase 6: DONE (98-100%)
- Logs to AITaskLog
- Returns result
```
### Progress Updates
**Progress Endpoint**: `/api/v1/system/settings/task_progress/{task_id}/`
**Response Format**:
- `state`: Task state (PENDING, PROGRESS, SUCCESS, FAILURE)
- `meta`: Progress metadata
- `phase`: Current phase (INIT, PREP, AI_CALL, PARSE, SAVE, DONE)
- `percentage`: Progress percentage (0-100)
- `message`: User-friendly message
- `request_steps`: Array of request steps
- `response_steps`: Array of response steps
- `cost`: API cost in USD
- `tokens`: Token count
---
## AI Functions
### 1. Auto Cluster Keywords
**Purpose**: Group related keywords into semantic clusters using AI
**Function Class**: `AutoClusterFunction`
**File**: `backend/igny8_core/ai/functions/auto_cluster.py`
**API Endpoint**:
- **ViewSet**: `KeywordViewSet`
- **Action**: `auto_cluster`
- **Method**: POST
- **URL Path**: `/v1/planner/keywords/auto_cluster/`
- **Payload**: `ids` (list[int]) - Keyword IDs
**Function Methods**:
- `get_name()`: Returns `'auto_cluster'`
- `validate(payload, account)`: Validates keyword IDs exist
- `prepare(payload, account)`: Loads keywords from database
- `build_prompt(data, account)`: Builds clustering prompt with keyword data
- `parse_response(response, step_tracker)`: Parses cluster JSON response
- `save_output(parsed, original_data, account, progress_tracker, step_tracker)`: Creates Cluster records and links keywords
**Input**: List of keyword IDs
**Output**: Cluster records created, keywords linked to clusters
**Progress Messages**:
- INIT: "Validating keywords"
- PREP: "Preparing keyword clustering"
- AI_CALL: "Analyzing keyword relationships"
- PARSE: "Processing cluster data"
- SAVE: "Creating clusters"
### 2. Generate Ideas
**Purpose**: Generate content ideas from keyword clusters
**Function Class**: `GenerateIdeasFunction`
**File**: `backend/igny8_core/ai/functions/generate_ideas.py`
**API Endpoint**:
- **ViewSet**: `ClusterViewSet`
- **Action**: `auto_generate_ideas`
- **Method**: POST
- **URL Path**: `/v1/planner/clusters/auto_generate_ideas/`
- **Payload**: `ids` (list[int]) - Cluster IDs
**Function Methods**:
- `get_name()`: Returns `'generate_ideas'`
- `validate(payload, account)`: Validates cluster IDs exist
- `prepare(payload, account)`: Loads clusters and keywords
- `build_prompt(data, account)`: Builds idea generation prompt with cluster data
- `parse_response(response, step_tracker)`: Parses ideas JSON response
- `save_output(parsed, original_data, account, progress_tracker, step_tracker)`: Creates ContentIdeas records
**Input**: List of cluster IDs
**Output**: ContentIdeas records created
**Progress Messages**:
- INIT: "Verifying cluster integrity"
- PREP: "Loading cluster keywords"
- AI_CALL: "Generating ideas with Igny8 Semantic AI"
- PARSE: "{count} high-opportunity idea(s) generated"
- SAVE: "Content Outline for Ideas generated"
### 3. Generate Content
**Purpose**: Generate blog post and article content from tasks
**Function Class**: `GenerateContentFunction`
**File**: `backend/igny8_core/ai/functions/generate_content.py`
**API Endpoint**:
- **ViewSet**: `TasksViewSet`
- **Action**: `auto_generate_content`
- **Method**: POST
- **URL Path**: `/v1/writer/tasks/auto_generate_content/`
- **Payload**: `ids` (list[int]) - Task IDs (max 50)
**Function Methods**:
- `get_name()`: Returns `'generate_content'`
- `get_max_items()`: Returns `50` (max tasks per batch)
- `validate(payload, account)`: Validates task IDs exist
- `prepare(payload, account)`: Loads tasks with related data
- `build_prompt(data, account)`: Builds content generation prompt with task data
- `parse_response(response, step_tracker)`: Parses content (JSON or plain text)
- `save_output(parsed, original_data, account, progress_tracker, step_tracker)`: Creates/updates Content records
**Input**: List of task IDs
**Output**: Content records created/updated with HTML content
**Progress Messages**:
- INIT: "Initializing content generation"
- PREP: "Loading tasks and building prompts"
- AI_CALL: "Generating content with AI"
- PARSE: "Processing content"
- SAVE: "Saving content"
### 4. Generate Image Prompts
**Purpose**: Extract image prompts from content for generating images
**Function Class**: `GenerateImagePromptsFunction`
**File**: `backend/igny8_core/ai/functions/generate_image_prompts.py`
**API Endpoint**:
- **ViewSet**: `TasksViewSet` (via content)
- **Action**: `generate_image_prompts`
- **Method**: POST
- **URL Path**: `/v1/writer/content/generate_image_prompts/`
- **Payload**: `ids` (list[int]) - Content IDs
**Function Methods**:
- `get_name()`: Returns `'generate_image_prompts'`
- `validate(payload, account)`: Validates content IDs exist
- `prepare(payload, account)`: Loads content records
- `build_prompt(data, account)`: Builds prompt extraction prompt with content HTML
- `parse_response(response, step_tracker)`: Parses image prompts JSON
- `save_output(parsed, original_data, account, progress_tracker, step_tracker)`: Updates Images records with prompts
**Input**: List of content IDs
**Output**: Images records updated with prompts (featured, in-article)
**Progress Messages**:
- INIT: "Validating content"
- PREP: "Preparing image prompt extraction"
- AI_CALL: "Extracting image prompts"
- PARSE: "Processing image prompts"
- SAVE: "Saving image prompts"
### 5. Generate Images
**Purpose**: Generate images using AI (OpenAI DALL-E or Runware) based on prompts
**Function Class**: `GenerateImagesFunction`
**File**: `backend/igny8_core/ai/functions/generate_images.py`
**API Endpoint**:
- **ViewSet**: `ImagesViewSet`
- **Action**: `generate_images`
- **Method**: POST
- **URL Path**: `/v1/writer/images/generate_images/`
- **Payload**: `ids` (list[int]) - Image IDs
**Function Methods**:
- `get_name()`: Returns `'generate_images'`
- `validate(payload, account)`: Validates image IDs exist and have prompts
- `prepare(payload, account)`: Loads images with prompts
- `build_prompt(data, account)`: Formats image prompt with context
- `parse_response(response, step_tracker)`: Parses image URL from API response
- `save_output(parsed, original_data, account, progress_tracker, step_tracker)`: Updates Images records with image URLs
**Input**: List of image IDs (with prompts)
**Output**: Images records updated with image URLs
**Image Generation Settings**:
- Provider: 'openai' or 'runware'
- Model: Model name (e.g., 'dall-e-3', 'runware:97@1')
- Image Type: 'realistic', 'artistic', 'cartoon'
- Max In-Article Images: Max images per content
- Image Format: 'webp', 'jpg', 'png'
- Desktop/Mobile: Boolean flags
**Progress Messages**:
- INIT: "Validating image prompts"
- PREP: "Preparing image generation"
- AI_CALL: "Creating image(s) with AI"
- PARSE: "Processing image response"
- SAVE: "Saving generated image(s)"
---
## Progress Tracking
### Progress Phases
All AI functions follow the same 6-phase execution:
1. **INIT** (0-10%): Validation phase
2. **PREP** (10-25%): Data preparation phase
3. **AI_CALL** (25-70%): AI API call phase
4. **PARSE** (70-85%): Response parsing phase
5. **SAVE** (85-98%): Database save phase
6. **DONE** (98-100%): Completion phase
### Progress Updates
**Frontend Polling**: Frontend polls `/api/v1/system/settings/task_progress/{task_id}/` every 1-2 seconds
**Progress Response**:
- `state`: Task state
- `meta`: Progress metadata
- `phase`: Current phase
- `percentage`: Progress percentage
- `message`: User-friendly message
- `request_steps`: Request steps array
- `response_steps`: Response steps array
- `cost`: API cost
- `tokens`: Token count
### Step Tracking
**Request Steps**: Tracked during prompt building and AI call
**Response Steps**: Tracked during response parsing
**Purpose**: Provides detailed logging for debugging and transparency
---
## Cost Tracking
### Cost Calculation
**Text Generation**:
- Cost calculated based on model pricing (input tokens + output tokens)
- Tracked per request
- Stored in `CostTracker`
**Image Generation**:
- Cost calculated based on provider pricing
- OpenAI DALL-E: Fixed cost per image
- Runware: Variable cost per image
- Tracked per image
### Cost Storage
**AITaskLog Model**:
- `cost`: Total cost for task
- `tokens`: Total tokens used
**CreditUsageLog Model**:
- `cost_usd`: Cost in USD
- `credits_used`: Credits deducted
---
## Prompt Management
### Prompt Resolution Order
1. **Task-Level Override**: If task has `prompt_override`, use it
2. **Database Prompt**: If account has custom prompt in database, use it
3. **Default Prompt**: Use default prompt from `DEFAULT_PROMPTS` registry
### Prompt Customization
**Per Account**: Custom prompts stored in `AIPrompt` model
**Per Function**: Different prompts for each function type
**Context Variables**: Prompts support context placeholders:
- `[IGNY8_KEYWORDS]` - Keyword list
- `[IGNY8_CLUSTERS]` - Cluster list
- `[IGNY8_CLUSTER_KEYWORDS]` - Cluster keywords
- `[IGNY8_IDEA]` - Idea data
- `[IGNY8_CLUSTER]` - Cluster data
---
## Model Configuration
### IntegrationSettings - Single Source of Truth
**IMPORTANT**: As of the refactoring completed in 2025-01-XX, the AI framework uses **IntegrationSettings only** for model configuration. There are no hardcoded defaults or fallbacks.
**IntegrationSettings Model**:
- `integration_type`: 'openai' or 'runware' (required)
- `account`: Account instance (required) - each account must configure their own models
- `is_active`: Boolean (must be True for configuration to be used)
- `config`: JSONField with model configuration (required)
- `model`: Model name (required) - e.g., 'gpt-4o-mini', 'gpt-4o', 'dall-e-3'
- `max_tokens`: Max tokens (optional, defaults to 4000)
- `temperature`: Temperature (optional, defaults to 0.7)
- `response_format`: Response format (optional, automatically set for JSON mode models)
### Model Configuration Function
**File**: `backend/igny8_core/ai/settings.py`
**Function**: `get_model_config(function_name: str, account) -> Dict[str, Any]`
**Behavior**:
- **Requires** `account` parameter (no longer optional)
- **Requires** IntegrationSettings to be configured for the account
- **Raises** `ValueError` with clear error messages if:
- Account not provided
- IntegrationSettings not found for account
- Model not configured in IntegrationSettings
- IntegrationSettings is inactive
**Error Messages**:
- Missing account: `"Account is required for model configuration"`
- Missing IntegrationSettings: `"OpenAI IntegrationSettings not configured for account {id}. Please configure OpenAI settings in the integration page."`
- Missing model: `"Model not configured in IntegrationSettings for account {id}. Please set 'model' in OpenAI integration settings."`
**Returns**:
```python
{
'model': str, # Model name from IntegrationSettings
'max_tokens': int, # From config or default 4000
'temperature': float, # From config or default 0.7
'response_format': dict, # JSON mode for supported models, or None
}
```
### Account-Specific Configuration
**Key Principle**: Each account must configure their own AI models. There are no global defaults.
**Configuration Steps**:
1. Navigate to Settings → Integrations
2. Configure OpenAI integration settings
3. Set `model` in the configuration (required)
4. Optionally set `max_tokens` and `temperature`
5. Ensure integration is active
**Supported Models**:
- Text generation: `gpt-4o-mini`, `gpt-4o`, `gpt-4-turbo`, etc.
- Image generation: `dall-e-3` (OpenAI) or `runware:97@1` (Runware)
- JSON mode: Automatically enabled for supported models (gpt-4o, gpt-4-turbo, etc.)
### Function Aliases
**File**: `backend/igny8_core/ai/settings.py`
**FUNCTION_ALIASES**: Dictionary mapping legacy function names to current names
- `cluster_keywords` → `auto_cluster`
- `auto_cluster_keywords` → `auto_cluster`
- `auto_generate_ideas` → `generate_ideas`
- `auto_generate_content` → `generate_content`
- `auto_generate_images` → `generate_images`
**Purpose**: Maintains backward compatibility with legacy function names.
### Removed Functions
The following helper functions were removed as part of the refactoring (they were never used):
- `get_model()` - Removed (use `get_model_config()['model']` instead)
- `get_max_tokens()` - Removed (use `get_model_config()['max_tokens']` instead)
- `get_temperature()` - Removed (use `get_model_config()['temperature']` instead)
**Rationale**: These functions were redundant - `get_model_config()` already returns all needed values.
---
## Summary
The IGNY8 AI framework provides:
1. **Unified Interface**: All AI functions use the same execution pipeline
2. **Consistent Execution**: All functions follow the same 6-phase flow
3. **Progress Tracking**: Real-time progress updates via Celery
4. **Cost Tracking**: Automatic cost and token tracking
5. **Error Handling**: Centralized error handling in AIEngine
6. **Prompt Management**: Hierarchical prompt resolution
7. **Model Configuration**: Per-account model overrides
8. **Database Logging**: Automatic logging to AITaskLog
9. **Extensibility**: Easy to add new AI functions
10. **Reliability**: Retry logic and error recovery
This architecture ensures consistency, maintainability, and extensibility while providing a robust foundation for all AI operations in the IGNY8 platform.

1244
master-docs/06-FUNCTIONAL-BUSINESS-LOGIC.md Normal file
View File

File diff suppressed because it is too large Load Diff

1392
master-docs/API-COMPLETE-REFERENCE.md Normal file
View File

File diff suppressed because it is too large Load Diff

273
master-docs/DIAGNOSIS-AUTHENTICATION-ISSUE.md Normal file
View File

@@ -0,0 +1,273 @@
# Authentication & Account Context Diagnosis
## Issue Summary
**Problem**: Wrong user showing without proper rights - authentication/account context mismatch
## Architecture Overview
### Request Flow
```
1. Request arrives
↓
2. Django Middleware Stack (settings.py:74-88)
- SecurityMiddleware
- WhiteNoiseMiddleware
- CorsMiddleware
- SessionMiddleware
- CommonMiddleware
- CsrfViewMiddleware
- AuthenticationMiddleware (sets request.user from session)
↓
3. AccountContextMiddleware (line 83)
- Extracts account from JWT token OR session
- Sets request.account
↓
4. DRF Authentication Classes (settings.py:210-214)
- JWTAuthentication (runs first)
- CSRFExemptSessionAuthentication
- BasicAuthentication
↓
5. View/ViewSet
- Uses request.user (from DRF auth)
- Uses request.account (from middleware OR JWTAuthentication)
```
## Critical Issues Found
### Issue #1: Duplicate Account Setting Logic
**Location**: Two places set `request.account` with different logic
1. **AccountContextMiddleware** (`backend/igny8_core/auth/middleware.py:99-106`)
```python
if account_id:
account = Account.objects.get(id=account_id)
# If user's account changed, use the new one from user object
if user.account and user.account.id != account_id:
request.account = user.account # Prioritizes user's current account
else:
request.account = account # Uses token's account
```
2. **JWTAuthentication** (`backend/igny8_core/api/authentication.py:64-80`)
```python
account_id = payload.get('account_id')
account = None
if account_id:
account = Account.objects.get(id=account_id) # Always uses token's account
if not account:
account = getattr(user, 'account', None) # Fallback only if no account_id
request.account = account # OVERWRITES middleware's account
```
**Problem**:
- Middleware validates if user's account changed and prioritizes `user.account`
- JWTAuthentication runs AFTER middleware and OVERWRITES `request.account` without validation
- This means middleware's validation is ignored
### Issue #2: User Object Loading Mismatch
**Location**: Different user loading strategies
1. **AccountContextMiddleware** (line 98)
```python
user = User.objects.select_related('account', 'account__plan').get(id=user_id)
```
- Loads user WITH account relationship (efficient, has account data)
2. **JWTAuthentication** (line 58)
```python
user = User.objects.get(id=user_id)
```
- Does NOT load account relationship
- When checking `user.account`, it triggers a separate DB query
- If account relationship is stale or missing, this can fail
**Problem**:
- JWTAuthentication's user object doesn't have account relationship loaded
- When `/me` endpoint uses `request.user.id` and then serializes with `UserSerializer`, it tries to access `user.account`
- This might trigger lazy loading which could return wrong/stale data
### Issue #3: Middleware Updates request.user (Session Auth)
**Location**: `backend/igny8_core/auth/middleware.py:32-46`
```python
if hasattr(request, 'user') and request.user and request.user.is_authenticated:
user = UserModel.objects.select_related('account', 'account__plan').get(id=request.user.id)
request.user = user # OVERWRITES request.user
request.account = user_account
```
**Problem**:
- Middleware is setting `request.user` for session authentication
- But then JWTAuthentication runs and might set a DIFFERENT user (from JWT token)
- This creates a conflict where middleware's user is overwritten
### Issue #4: Token Account vs User Account Mismatch
**Location**: Token generation vs user's current account
**Token Generation** (`backend/igny8_core/auth/utils.py:30-57`):
```python
def generate_access_token(user, account=None):
if account is None:
account = getattr(user, 'account', None)
payload = {
'user_id': user.id,
'account_id': account.id if account else None, # Token stores account_id at login time
...
}
```
**Problem**:
- Token is generated at login with `user.account` at that moment
- If user's account changes AFTER login (e.g., admin moves user to different account), token still has old `account_id`
- Middleware tries to handle this (line 103-104), but JWTAuthentication overwrites it
### Issue #5: /me Endpoint Uses request.user Without Account Relationship
**Location**: `backend/igny8_core/auth/urls.py:188-197`
```python
def get(self, request):
user = UserModel.objects.select_related('account', 'account__plan').get(id=request.user.id)
serializer = UserSerializer(user)
return success_response(data={'user': serializer.data}, request=request)
```
**Problem**:
- `/me` endpoint correctly loads user with account relationship
- BUT `request.user` (from JWTAuthentication) doesn't have account relationship loaded
- If other code uses `request.user.account` directly, it might get wrong/stale data
## Root Cause Analysis
### Primary Root Cause
**JWTAuthentication overwrites `request.account` set by middleware without validating if user's account changed**
### Secondary Issues
1. JWTAuthentication doesn't load user with account relationship (inefficient + potential stale data)
2. Middleware sets `request.user` for session auth, but JWTAuthentication might overwrite it
3. Token's `account_id` can become stale if user's account changes after login
## Data Flow Problem
### Current Flow (BROKEN)
```
1. Request with JWT token arrives
↓
2. AccountContextMiddleware runs:
- Decodes JWT token
- Gets user_id=5, account_id=10
- Loads User(id=5) with account relationship
- Checks: user.account.id = 12 (user moved to account 12)
- Sets: request.account = Account(id=12) ✅ CORRECT
↓
3. JWTAuthentication runs:
- Decodes JWT token (again)
- Gets user_id=5, account_id=10
- Loads User(id=5) WITHOUT account relationship
- Gets Account(id=10) from token
- Sets: request.account = Account(id=10) ❌ WRONG (overwrites middleware)
- Sets: request.user = User(id=5) (without account relationship)
↓
4. View uses request.account (WRONG - account 10 instead of 12)
5. View uses request.user.account (might trigger lazy load, could be stale)
```
### Expected Flow (CORRECT)
```
1. Request with JWT token arrives
↓
2. AccountContextMiddleware runs:
- Sets request.account based on token with validation
↓
3. JWTAuthentication runs:
- Sets request.user with account relationship loaded
- Does NOT overwrite request.account (respects middleware)
↓
4. View uses request.account (CORRECT - from middleware)
5. View uses request.user.account (CORRECT - loaded with relationship)
```
## Database Schema Check
### User Model
- `User.account` = ForeignKey to Account, `db_column='tenant_id'`, nullable
- Relationship: User → Account (many-to-one)
### Account Model
- `Account.owner` = ForeignKey to User
- Relationship: Account → User (many-to-one, owner)
### Potential Database Issues
- If `User.account_id` (tenant_id column) doesn't match token's `account_id`, there's a mismatch
- If user's account was changed in DB but token wasn't refreshed, token has stale account_id
## Permission System Check
### HasTenantAccess Permission
**Location**: `backend/igny8_core/api/permissions.py:25-67`
```python
def has_permission(self, request, view):
account = getattr(request, 'account', None)
# If no account in request, try to get from user
if not account and hasattr(request.user, 'account'):
account = request.user.account
# Check if user belongs to this account
if account:
user_account = request.user.account
return user_account == account or user_account.id == account.id
```
**Problem**:
- Permission checks `request.account` vs `request.user.account`
- If `request.account` is wrong (from JWTAuthentication overwrite), permission check fails
- User gets 403 Forbidden even though they should have access
## Recommendations (Diagnosis Only - No Code Changes)
### Fix Priority
1. **CRITICAL**: Make JWTAuthentication respect middleware's `request.account` OR remove duplicate logic
- Option A: JWTAuthentication should check if `request.account` already exists and not overwrite it
- Option B: Remove account setting from JWTAuthentication, let middleware handle it
2. **HIGH**: Load user with account relationship in JWTAuthentication
- Change `User.objects.get(id=user_id)` to `User.objects.select_related('account', 'account__plan').get(id=user_id)`
3. **MEDIUM**: Don't set `request.user` in middleware for JWT auth
- Middleware should only set `request.user` for session auth
- For JWT auth, let JWTAuthentication handle `request.user`
4. **LOW**: Add validation in token generation to ensure account_id matches user.account
- Or add token refresh mechanism when user's account changes
### Architecture Decision Needed
**Question**: Should `request.account` be set by:
- A) Middleware only (current middleware logic with validation)
- B) JWTAuthentication only (simpler, but loses validation)
- C) Both, but JWTAuthentication checks if middleware already set it
**Recommendation**: Option C - Middleware sets it with validation, JWTAuthentication only sets if not already set
## Files Involved
1. `backend/igny8_core/auth/middleware.py` - AccountContextMiddleware
2. `backend/igny8_core/api/authentication.py` - JWTAuthentication
3. `backend/igny8_core/auth/urls.py` - MeView endpoint
4. `backend/igny8_core/auth/utils.py` - Token generation
5. `backend/igny8_core/api/permissions.py` - HasTenantAccess permission
6. `backend/igny8_core/settings.py` - Middleware and authentication class order
## Testing Scenarios to Verify
1. **User with account_id in token matches user.account** → Should work
2. **User's account changed after login (token has old account_id)** → Currently broken
3. **User with no account in token** → Should fallback to user.account
4. **Developer/admin user** → Should bypass account checks
5. **Session auth vs JWT auth** → Both should work consistently

2055
master-docs/WORDPRESS-PLUGIN-INTEGRATION.md Normal file
View File

File diff suppressed because it is too large Load Diff