Files
igny8/docs/04-BACKEND.md
Desktop e4a6bd1160 DOCS
2025-11-12 20:22:08 +05:00

818 lines
22 KiB
Markdown

# IGNY8 Backend Documentation
**Last Updated:** 2025-01-XX
**Purpose:** Complete backend documentation covering models, views, APIs, modules, serializers, tasks, and structure.
---
## Table of Contents
1. [Backend Overview](#backend-overview)
2. [Tech Stack](#tech-stack)
3. [Project Structure](#project-structure)
4. [Models](#models)
5. [ViewSets](#viewsets)
6. [Serializers](#serializers)
7. [Celery Tasks](#celery-tasks)
8. [API Endpoints](#api-endpoints)
9. [Base Classes](#base-classes)
10. [Middleware](#middleware)
11. [Utilities](#utilities)
12. [Modules](#modules)
---
## Backend Overview
The IGNY8 backend is a Django 5.2+ application using Django REST Framework (DRF) for API endpoints. The backend follows a modular architecture with clear separation of concerns, automatic account isolation, and support for asynchronous task processing via Celery.
### Key Features
- **Multi-Tenancy**: Complete account isolation with automatic filtering
- **RESTful API**: DRF ViewSets with consistent response format
- **Celery Integration**: Asynchronous task processing for long-running operations
- **Account/Site/Sector Hierarchy**: Hierarchical data organization
- **AI Integration**: Unified AI framework for all AI operations
- **Progress Tracking**: Real-time progress updates for Celery tasks
---
## Tech Stack
### Core Technologies
- **Django 5.2+**: Web framework
- **Django REST Framework**: API framework
- **PostgreSQL**: Database
- **Celery**: Asynchronous task queue
- **Redis**: Celery broker and caching
### Key Libraries
- **django-filter**: Advanced filtering
- **djangorestframework-simplejwt**: JWT authentication
- **requests**: HTTP client for external APIs
- **python-dotenv**: Environment variable management
---
## Project Structure
```
backend/igny8_core/
├── auth/ # Multi-tenancy and authentication
│ ├── models.py # Account, User, Plan, Site, Sector, Industry models
│ ├── views.py # Account, User, Site, Sector ViewSets
│ ├── serializers.py # Account, User, Plan serializers
│ └── urls.py # Auth module URLs
├── modules/ # Feature modules
│ ├── planner/ # Keywords, Clusters, Ideas
│ │ ├── models.py # Keywords, Clusters, ContentIdeas models
│ │ ├── views.py # KeywordViewSet, ClusterViewSet, ContentIdeasViewSet
│ │ ├── tasks.py # Celery tasks for AI operations
│ │ ├── serializers.py # Model serializers
│ │ └── urls.py # Planner module URLs
│ ├── writer/ # Tasks, Content, Images
│ │ ├── models.py # Tasks, Content, Images models
│ │ ├── views.py # TasksViewSet, ImagesViewSet
│ │ ├── tasks.py # Celery tasks for content/image generation
│ │ └── urls.py # Writer module URLs
│ ├── system/ # Settings, Prompts, Integration
│ │ ├── models.py # AIPrompt, IntegrationSettings, AuthorProfile, Strategy
│ │ ├── views.py # AIPromptViewSet, AuthorProfileViewSet
│ │ ├── integration_views.py # IntegrationSettingsViewSet, task_progress
│ │ ├── utils.py # Default prompts, prompt loading
│ │ └── urls.py # System module URLs
│ └── billing/ # Credits, Transactions, Usage
│ ├── models.py # CreditTransaction, CreditUsageLog models
│ ├── views.py # Billing ViewSets
│ └── services.py # CreditService
├── api/ # API base classes
│ ├── base.py # AccountModelViewSet, SiteSectorModelViewSet
│ └── pagination.py # CustomPageNumberPagination
├── utils/ # Shared utilities
│ ├── ai_processor.py # Unified AI interface (legacy, see AI functions)
│ └── content_normalizer.py # Content processing utilities
├── middleware/ # Custom middleware
│ ├── account.py # AccountContextMiddleware (sets request.account)
│ └── resource_tracker.py # ResourceTrackerMiddleware (API metrics)
├── ai/ # AI framework
│ ├── base.py # BaseAIFunction
│ ├── engine.py # AIEngine
│ ├── tasks.py # run_ai_task
│ ├── registry.py # Function registry
│ ├── prompts.py # PromptRegistry
│ ├── ai_core.py # AICore
│ ├── settings.py # Model configuration
│ ├── validators.py # Validation functions
│ ├── tracker.py # Progress tracking
│ └── functions/ # AI function implementations
│ ├── auto_cluster.py
│ ├── generate_ideas.py
│ ├── generate_content.py
│ ├── generate_image_prompts.py
│ └── generate_images.py
├── settings.py # Django settings
├── urls.py # Root URL configuration
└── celery.py # Celery configuration
```
---
## Models
### Base Models
#### AccountBaseModel
**File**: `auth/models.py`
**Purpose**: Base model for all account-isolated models.
**Fields**:
- `account`: ForeignKey to Account
- `created_at`: DateTimeField (auto_now_add)
- `updated_at`: DateTimeField (auto_now)
**Usage**: All models that need account isolation inherit from this.
#### SiteSectorBaseModel
**File**: `auth/models.py`
**Purpose**: Base model for models that belong to Site and Sector.
**Fields**:
- Inherits from `AccountBaseModel`
- `site`: ForeignKey to Site
- `sector`: ForeignKey to Sector
**Methods**:
- `save()`: Automatically sets `account` from `site.account` and validates sector belongs to site
**Usage**: Models like Keywords, Clusters, ContentIdeas, Tasks inherit from this.
### Auth Models
#### Account
**Table**: `igny8_accounts`
**Fields**:
- `name`: CharField
- `slug`: SlugField (unique)
- `owner`: ForeignKey to User
- `plan`: ForeignKey to Plan
- `credits`: IntegerField (default: 0)
- `status`: CharField (choices: active, suspended, trial, cancelled)
**Methods**:
- `is_system_account()`: Returns True if account is a system account
#### User
**Table**: `igny8_users`
**Inherits**: `AbstractUser`
**Fields**:
- `email`: EmailField (unique)
- `account`: ForeignKey to Account
- `role`: CharField (choices: developer, owner, admin, editor, viewer, system_bot)
**Methods**:
- `has_role(role)`: Checks if user has role
- `is_owner_or_admin()`: Checks if user is owner or admin
- `is_developer()`: Checks if user is developer
- `is_admin_or_developer()`: Checks if user is admin or developer
- `is_system_account_user()`: Checks if user belongs to system account
- `get_accessible_sites()`: Returns queryset of accessible sites
#### Plan
**Table**: `igny8_plans`
**Fields**: Extensive fields for limits (users, sites, keywords, clusters, content ideas, AI requests, word count, images, credits)
**Methods**:
- `clean()`: Validates plan limits
- `get_effective_credits_per_month()`: Returns included_credits or credits_per_month
#### Site
**Table**: `igny8_sites`
**Inherits**: `AccountBaseModel`
**Fields**:
- `name`: CharField
- `slug`: SlugField (unique per account)
- `domain`: URLField (optional)
- `industry`: ForeignKey to Industry (optional)
- `is_active`: BooleanField
- `status`: CharField
- `wp_url`: URLField (optional)
- `wp_username`: CharField (optional)
- `wp_app_password`: CharField (optional)
**Methods**:
- `get_active_sectors_count()`: Returns count of active sectors
- `can_add_sector()`: Returns True if site can add another sector (max 5)
#### Sector
**Table**: `igny8_sectors`
**Inherits**: `AccountBaseModel`
**Fields**:
- `site`: ForeignKey to Site
- `industry_sector`: ForeignKey to IndustrySector (optional)
- `name`: CharField
- `slug`: SlugField (unique per site)
- `is_active`: BooleanField
- `status`: CharField
**Validation**: Maximum 5 active sectors per site
### Planner Models
#### Keywords
**Table**: `igny8_keywords`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `keyword`: CharField
- `volume`: IntegerField (optional)
- `difficulty`: IntegerField (optional)
- `intent`: CharField (optional)
- `status`: CharField
- `cluster`: ManyToManyField to Clusters
#### Clusters
**Table**: `igny8_clusters`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `name`: CharField
- `description`: TextField (optional)
- `keywords_count`: IntegerField (calculated)
- `volume`: IntegerField (calculated)
- `status`: CharField
- `keywords`: ManyToManyField to Keywords
#### ContentIdeas
**Table**: `igny8_content_ideas`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `idea_title`: CharField
- `description`: TextField
- `content_type`: CharField
- `content_structure`: CharField
- `target_keywords`: TextField
- `keyword_cluster`: ForeignKey to Clusters
- `estimated_word_count`: IntegerField
- `status`: CharField
### Writer Models
#### Tasks
**Table**: `igny8_tasks`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `title`: CharField
- `description`: TextField
- `cluster`: ForeignKey to Clusters (optional)
- `idea`: ForeignKey to ContentIdeas (optional)
- `content_type`: CharField
- `content_structure`: CharField
- `status`: CharField
#### Content
**Table**: `igny8_content`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `task`: OneToOneField to Tasks
- `html_content`: TextField
- `word_count`: IntegerField
- `status`: CharField
- `wp_post_id`: IntegerField (optional)
- `meta_title`: CharField (optional)
- `meta_description`: TextField (optional)
- `primary_keyword`: CharField (optional)
- `secondary_keywords`: TextField (optional)
#### Images
**Table**: `igny8_images`
**Inherits**: `SiteSectorBaseModel`
**Fields**:
- `task`: ForeignKey to Tasks (optional)
- `content`: ForeignKey to Content (optional)
- `image_type`: CharField (choices: featured, in_article, desktop, mobile)
- `prompt`: TextField
- `image_url`: CharField
- `status`: CharField
### System Models
#### AIPrompt
**Table**: `igny8_ai_prompts`
**Inherits**: `AccountBaseModel`
**Fields**:
- `prompt_type`: CharField
- `prompt_value`: TextField
- `function_name`: CharField (optional)
#### IntegrationSettings
**Table**: `igny8_integration_settings`
**Inherits**: `AccountBaseModel`
**Fields**:
- `integration_type`: CharField (choices: openai, runware, image_generation)
- `config`: JSONField
#### AuthorProfile
**Table**: `igny8_author_profiles`
**Inherits**: `AccountBaseModel`
**Fields**:
- `name`: CharField
- `description`: TextField
- `tone`: CharField
- `language`: CharField
#### Strategy
**Table**: `igny8_strategies`
**Inherits**: `AccountBaseModel`
**Fields**:
- `name`: CharField
- `description`: TextField
- `sector`: ForeignKey to Sector (optional)
- `prompt_types`: JSONField
- `section_logic`: JSONField
- `is_active`: BooleanField
### Billing Models
#### CreditTransaction
**Table**: `igny8_credit_transactions`
**Inherits**: `AccountBaseModel`
**Fields**:
- `transaction_type`: CharField (choices: purchase, usage, refund, adjustment)
- `amount`: IntegerField
- `balance_after`: IntegerField
- `description`: TextField (optional)
#### CreditUsageLog
**Table**: `igny8_credit_usage_logs`
**Inherits**: `AccountBaseModel`
**Fields**:
- `operation_type`: CharField
- `credits_used`: IntegerField
- `cost_usd`: DecimalField
- `details`: JSONField (optional)
---
## ViewSets
### Base ViewSets
#### AccountModelViewSet
**File**: `api/base.py`
**Purpose**: Base ViewSet with automatic account filtering.
**Methods**:
- `get_queryset()`: Filters queryset by `request.account` (with admin/developer override)
- `perform_create()`: Sets account on created objects
- `get_serializer_context()`: Adds account to serializer context
**Access Control**:
- Admin/Developer users: Bypass account filtering
- System account users: Bypass account filtering
- Regular users: Only see data from their account
#### SiteSectorModelViewSet
**File**: `api/base.py`
**Purpose**: Base ViewSet with site/sector filtering and access control.
**Inherits**: `AccountModelViewSet`
**Methods**:
- `get_queryset()`: Filters by account, accessible sites, and optional site_id/sector_id
- `perform_create()`: Validates site access and sector-site relationship
- `get_serializer_context()`: Adds accessible sites and sectors to context
**Access Control**:
- Developers: All active sites
- System account users: All active sites
- Owners/Admins: All sites in their account
- Editors/Viewers: Only sites granted via `SiteUserAccess`
### Planner ViewSets
#### KeywordViewSet
**Inherits**: `SiteSectorModelViewSet`
**Actions**:
- `list()`: List keywords with filtering
- `create()`: Create keyword
- `retrieve()`: Get keyword details
- `update()`: Update keyword
- `destroy()`: Delete keyword
- `auto_cluster()`: Auto-cluster keywords using AI
- `bulk_delete()`: Bulk delete keywords
- `bulk_update_status()`: Bulk update keyword status
- `export_csv()`: Export keywords to CSV
- `import_csv()`: Import keywords from CSV
**Filtering**:
- Search: `keyword` field
- Filters: `status`, `cluster_id`, `intent`
- Custom: `difficulty_min`, `difficulty_max`, `volume_min`, `volume_max`
- Ordering: `created_at`, `volume`, `difficulty`
#### ClusterViewSet
**Inherits**: `SiteSectorModelViewSet`
**Actions**:
- `list()`: List clusters
- `create()`: Create cluster
- `retrieve()`: Get cluster details
- `update()`: Update cluster
- `destroy()`: Delete cluster
- `auto_generate_ideas()`: Auto-generate content ideas for clusters
#### ContentIdeasViewSet
**Inherits**: `SiteSectorModelViewSet`
**Actions**:
- `list()`: List content ideas
- `create()`: Create content idea
- `retrieve()`: Get content idea details
- `update()`: Update content idea
- `destroy()`: Delete content idea
### Writer ViewSets
#### TasksViewSet
**Inherits**: `SiteSectorModelViewSet`
**Actions**:
- `list()`: List tasks
- `create()`: Create task
- `retrieve()`: Get task details
- `update()`: Update task
- `destroy()`: Delete task
- `auto_generate_content()`: Auto-generate content for tasks
- `generate_images()`: Generate images for image records
- `bulk_delete()`: Bulk delete tasks
- `bulk_update()`: Bulk update task status
**Filtering**:
- Search: `title`, `keywords`
- Filters: `status`, `cluster_id`, `content_type`, `content_structure`
- Ordering: `title`, `created_at`, `word_count`, `status`
#### ImagesViewSet
**Inherits**: `SiteSectorModelViewSet`
**Actions**:
- `list()`: List images
- `create()`: Create image
- `retrieve()`: Get image details
- `update()`: Update image
- `destroy()`: Delete image
- `generate_images()`: Generate images using AI
### System ViewSets
#### IntegrationSettingsViewSet
**Inherits**: `viewsets.ViewSet`
**Actions**:
- `list()`: List integrations
- `retrieve()`: Get integration settings
- `update()`: Save integration settings
- `save_post()`: Save integration settings (POST)
- `test_connection()`: Test API connection
- `test_openai()`: Test OpenAI connection
- `test_runware()`: Test Runware connection
- `generate_image()`: Test image generation
- `task_progress()`: Get Celery task progress
#### AIPromptViewSet
**Inherits**: `AccountModelViewSet`
**Actions**:
- `list()`: List prompts
- `create()`: Create prompt
- `retrieve()`: Get prompt details
- `update()`: Update prompt
- `destroy()`: Delete prompt
- `reset_to_default()`: Reset prompt to default value
#### AuthorProfileViewSet
**Inherits**: `AccountModelViewSet`
**Actions**:
- `list()`: List author profiles
- `create()`: Create author profile
- `retrieve()`: Get author profile details
- `update()`: Update author profile
- `destroy()`: Delete author profile
---
## Serializers
### Planner Serializers
#### KeywordSerializer
**Fields**: All Keyword model fields
**Validation**: Validates keyword uniqueness, cluster belongs to same sector
#### ClusterSerializer
**Fields**: All Cluster model fields
**Read-Only Fields**: `keywords_count`, `volume` (calculated)
#### ContentIdeasSerializer
**Fields**: All ContentIdeas model fields
### Writer Serializers
#### TasksSerializer
**Fields**: All Tasks model fields
#### ContentSerializer
**Fields**: All Content model fields
#### ImagesSerializer
**Fields**: All Images model fields
### System Serializers
#### AIPromptSerializer
**Fields**: All AIPrompt model fields
#### IntegrationSettingsSerializer
**Fields**: All IntegrationSettings model fields
---
## Celery Tasks
### AI Task Entry Point
#### run_ai_task
**File**: `ai/tasks.py`
**Purpose**: Unified Celery task entrypoint for all AI functions
**Parameters**:
- `function_name`: Function name (e.g., 'auto_cluster')
- `payload`: Function-specific payload
- `account_id`: Account ID
**Flow**:
1. Gets account from account_id
2. Gets function instance from registry
3. Creates AIEngine
4. Executes function via AIEngine
### Planner Tasks
#### auto_cluster_keywords_task
**File**: `modules/planner/tasks.py` (legacy, now uses `run_ai_task`)
**Purpose**: Auto-cluster keywords using AI
**Parameters**:
- `keyword_ids`: List of keyword IDs
- `account_id`: Account ID
- `site_id`: Site ID
- `sector_id`: Sector ID
**Progress Tracking**: Updates progress with request_steps and response_steps
#### auto_generate_ideas_task
**File**: `modules/planner/tasks.py` (legacy, now uses `run_ai_task`)
**Purpose**: Auto-generate content ideas for clusters
**Parameters**:
- `cluster_ids`: List of cluster IDs
- `account_id`: Account ID
**Progress Tracking**: Updates progress for each cluster
### Writer Tasks
#### auto_generate_content_task
**File**: `modules/writer/tasks.py` (legacy, now uses `run_ai_task`)
**Purpose**: Auto-generate content for tasks
**Parameters**:
- `task_ids`: List of task IDs
- `account_id`: Account ID
**Progress Tracking**: Updates progress for each task
#### process_image_generation_queue
**File**: `modules/writer/views.py` (via `ai/tasks.py`)
**Purpose**: Generate images for image records
**Parameters**:
- `image_ids`: List of image IDs
- `account_id`: Account ID
- `content_id`: Content ID (optional)
**Progress Tracking**: Updates progress for each image sequentially
---
## API Endpoints
### Base URL
`/api/v1/`
### Planner Endpoints
- `GET /api/v1/planner/keywords/` - List keywords
- `POST /api/v1/planner/keywords/` - Create keyword
- `GET /api/v1/planner/keywords/{id}/` - Get keyword
- `PUT /api/v1/planner/keywords/{id}/` - Update keyword
- `DELETE /api/v1/planner/keywords/{id}/` - Delete keyword
- `POST /api/v1/planner/keywords/auto_cluster/` - Auto-cluster keywords
- `POST /api/v1/planner/keywords/bulk_delete/` - Bulk delete keywords
- `POST /api/v1/planner/keywords/bulk_update_status/` - Bulk update status
- `GET /api/v1/planner/keywords/export_csv/` - Export keywords
- `POST /api/v1/planner/keywords/import_csv/` - Import keywords
- `GET /api/v1/planner/clusters/` - List clusters
- `POST /api/v1/planner/clusters/auto_generate_ideas/` - Auto-generate ideas
- `GET /api/v1/planner/ideas/` - List content ideas
### Writer Endpoints
- `GET /api/v1/writer/tasks/` - List tasks
- `POST /api/v1/writer/tasks/auto_generate_content/` - Auto-generate content
- `POST /api/v1/writer/images/generate_images/` - Generate images
### System Endpoints
- `GET /api/v1/system/settings/integrations/{pk}/` - Get integration settings
- `PUT /api/v1/system/settings/integrations/{pk}/` - Save integration settings
- `POST /api/v1/system/settings/integrations/{pk}/test_openai/` - Test OpenAI
- `POST /api/v1/system/settings/integrations/{pk}/test_runware/` - Test Runware
- `POST /api/v1/system/settings/integrations/{pk}/generate_image/` - Test image generation
- `GET /api/v1/system/settings/task_progress/{task_id}/` - Get task progress
---
## Base Classes
### AccountModelViewSet
**File**: `api/base.py`
**Purpose**: Base ViewSet with automatic account filtering
**Features**:
- Automatic account filtering
- Admin/Developer override
- Account context in serializers
### SiteSectorModelViewSet
**File**: `api/base.py`
**Purpose**: Base ViewSet with site/sector filtering
**Features**:
- Account filtering (inherited)
- Site access control
- Sector validation
- Accessible sites/sectors in serializer context
---
## Middleware
### AccountContextMiddleware
**File**: `middleware/account.py`
**Purpose**: Sets `request.account` from JWT token
**Functionality**:
- Extracts account ID from JWT token
- Loads Account object
- Sets `request.account`
### ResourceTrackerMiddleware
**File**: `middleware/resource_tracker.py`
**Purpose**: Tracks API request metrics
**Functionality**:
- Tracks CPU, memory, I/O usage
- Stores metrics in cache
- Provides metrics endpoint
---
## Utilities
### Content Normalizer
**File**: `utils/content_normalizer.py`
**Purpose**: Content processing utilities
**Functions**:
- `normalize_content()`: Converts plain text to HTML
- `_extract_body_content()`: Extracts body content from HTML
---
## Modules
### Planner Module
**Purpose**: Keyword management and content planning
**Models**: Keywords, Clusters, ContentIdeas
**ViewSets**: KeywordViewSet, ClusterViewSet, ContentIdeasViewSet
**Tasks**: Auto-cluster keywords, Auto-generate ideas
### Writer Module
**Purpose**: Content generation and management
**Models**: Tasks, Content, Images
**ViewSets**: TasksViewSet, ImagesViewSet
**Tasks**: Auto-generate content, Generate images
### System Module
**Purpose**: System configuration and AI settings
**Models**: AIPrompt, IntegrationSettings, AuthorProfile, Strategy
**ViewSets**: AIPromptViewSet, IntegrationSettingsViewSet, AuthorProfileViewSet
### Billing Module
**Purpose**: Credit management and usage tracking
**Models**: CreditTransaction, CreditUsageLog
**ViewSets**: CreditTransactionViewSet, CreditUsageLogViewSet
**Services**: CreditService (check, deduct, add, calculate credits)
---
## Summary
The IGNY8 backend provides:
1. **Multi-Tenancy**: Complete account isolation with automatic filtering
2. **RESTful API**: DRF ViewSets with consistent response format
3. **Celery Integration**: Asynchronous task processing
4. **Hierarchical Organization**: Account > Site > Sector > Content structure
5. **AI Framework**: Unified AI framework for all AI operations
6. **Progress Tracking**: Real-time progress updates for Celery tasks
7. **Module-Based Design**: Clear separation of concerns
8. **Base Classes**: Reusable ViewSets for common patterns
9. **Middleware**: Account context and resource tracking
10. **Utilities**: Content processing and AI integration
This architecture ensures scalability, maintainability, and extensibility while providing a robust foundation for the IGNY8 platform.