igny8/docs/04-BACKEND.md

IGNY8 Backend Documentation

Last Updated: 2025-01-XX
Purpose: Complete backend documentation covering models, views, APIs, modules, serializers, tasks, and structure.

---

Table of Contents

1. Backend Overview
2. Tech Stack
3. Project Structure
4. Models
5. ViewSets
6. Serializers
7. Celery Tasks
8. API Endpoints
9. Base Classes
10. Middleware
11. Utilities
12. Modules

---

Backend Overview

The IGNY8 backend is a Django 5.2+ application using Django REST Framework (DRF) for API endpoints. The backend follows a modular architecture with clear separation of concerns, automatic account isolation, and support for asynchronous task processing via Celery.

Key Features

• Multi-Tenancy: Complete account isolation with automatic filtering
• RESTful API: DRF ViewSets with consistent response format
• Celery Integration: Asynchronous task processing for long-running operations
• Account/Site/Sector Hierarchy: Hierarchical data organization
• AI Integration: Unified AI framework for all AI operations
• Progress Tracking: Real-time progress updates for Celery tasks

---

Tech Stack

Core Technologies

• Django 5.2+: Web framework
• Django REST Framework: API framework
• PostgreSQL: Database
• Celery: Asynchronous task queue
• Redis: Celery broker and caching

Key Libraries

• django-filter: Advanced filtering
• djangorestframework-simplejwt: JWT authentication
• requests: HTTP client for external APIs
• python-dotenv: Environment variable management

---

Project Structure

backend/igny8_core/
├── auth/                     # Multi-tenancy and authentication
│   ├── models.py             # Account, User, Plan, Site, Sector, Industry models
│   ├── views.py              # Account, User, Site, Sector ViewSets
│   ├── serializers.py        # Account, User, Plan serializers
│   └── urls.py               # Auth module URLs
├── modules/                  # Feature modules
│   ├── planner/              # Keywords, Clusters, Ideas
│   │   ├── models.py         # Keywords, Clusters, ContentIdeas models
│   │   ├── views.py          # KeywordViewSet, ClusterViewSet, ContentIdeasViewSet
│   │   ├── tasks.py          # Celery tasks for AI operations
│   │   ├── serializers.py    # Model serializers
│   │   └── urls.py           # Planner module URLs
│   ├── writer/               # Tasks, Content, Images
│   │   ├── models.py         # Tasks, Content, Images models
│   │   ├── views.py          # TasksViewSet, ImagesViewSet
│   │   ├── tasks.py          # Celery tasks for content/image generation
│   │   └── urls.py           # Writer module URLs
│   ├── system/               # Settings, Prompts, Integration
│   │   ├── models.py         # AIPrompt, IntegrationSettings, AuthorProfile, Strategy
│   │   ├── views.py          # AIPromptViewSet, AuthorProfileViewSet
│   │   ├── integration_views.py  # IntegrationSettingsViewSet, task_progress
│   │   ├── utils.py          # Default prompts, prompt loading
│   │   └── urls.py           # System module URLs
│   └── billing/              # Credits, Transactions, Usage
│       ├── models.py         # CreditTransaction, CreditUsageLog models
│       ├── views.py          # Billing ViewSets
│       └── services.py       # CreditService
├── api/                      # API base classes
│   ├── base.py               # AccountModelViewSet, SiteSectorModelViewSet
│   └── pagination.py         # CustomPageNumberPagination
├── utils/                    # Shared utilities
│   ├── ai_processor.py       # Unified AI interface (legacy, see AI functions)
│   └── content_normalizer.py # Content processing utilities
├── middleware/               # Custom middleware
│   ├── account.py            # AccountContextMiddleware (sets request.account)
│   └── resource_tracker.py   # ResourceTrackerMiddleware (API metrics)
├── ai/                       # AI framework
│   ├── base.py               # BaseAIFunction
│   ├── engine.py             # AIEngine
│   ├── tasks.py              # run_ai_task
│   ├── registry.py           # Function registry
│   ├── prompts.py            # PromptRegistry
│   ├── ai_core.py            # AICore
│   ├── settings.py           # Model configuration
│   ├── validators.py         # Validation functions
│   ├── tracker.py            # Progress tracking
│   └── functions/            # AI function implementations
│       ├── auto_cluster.py
│       ├── generate_ideas.py
│       ├── generate_content.py
│       ├── generate_image_prompts.py
│       └── generate_images.py
├── settings.py               # Django settings
├── urls.py                   # Root URL configuration
└── celery.py                 # Celery configuration

---

Models

Base Models

AccountBaseModel

File: auth/models.py

Purpose: Base model for all account-isolated models.

Fields:

• account: ForeignKey to Account
• created_at: DateTimeField (auto_now_add)
• updated_at: DateTimeField (auto_now)

Usage: All models that need account isolation inherit from this.

SiteSectorBaseModel

File: auth/models.py

Purpose: Base model for models that belong to Site and Sector.

Fields:

• Inherits from AccountBaseModel
• site: ForeignKey to Site
• sector: ForeignKey to Sector

Methods:

• save(): Automatically sets account from site.account and validates sector belongs to site

Usage: Models like Keywords, Clusters, ContentIdeas, Tasks inherit from this.

Auth Models

Account

Table: igny8_accounts

Fields:

• name: CharField
• slug: SlugField (unique)
• owner: ForeignKey to User
• plan: ForeignKey to Plan
• credits: IntegerField (default: 0)
• status: CharField (choices: active, suspended, trial, cancelled)

Methods:

• is_system_account(): Returns True if account is a system account

User

Table: igny8_users

Inherits: AbstractUser

Fields:

• email: EmailField (unique)
• account: ForeignKey to Account
• role: CharField (choices: developer, owner, admin, editor, viewer, system_bot)

Methods:

• has_role(role): Checks if user has role
• is_owner_or_admin(): Checks if user is owner or admin
• is_developer(): Checks if user is developer
• is_admin_or_developer(): Checks if user is admin or developer
• is_system_account_user(): Checks if user belongs to system account
• get_accessible_sites(): Returns queryset of accessible sites

Plan

Table: igny8_plans

Fields: Extensive fields for limits (users, sites, keywords, clusters, content ideas, AI requests, word count, images, credits)

Methods:

• clean(): Validates plan limits
• get_effective_credits_per_month(): Returns included_credits or credits_per_month

Site

Table: igny8_sites

Inherits: AccountBaseModel

Fields:

• name: CharField
• slug: SlugField (unique per account)
• domain: URLField (optional)
• industry: ForeignKey to Industry (optional)
• is_active: BooleanField
• status: CharField
• wp_url: URLField (optional)
• wp_username: CharField (optional)
• wp_app_password: CharField (optional)

Methods:

• get_active_sectors_count(): Returns count of active sectors
• can_add_sector(): Returns True if site can add another sector (max 5)

Sector

Table: igny8_sectors

Inherits: AccountBaseModel

Fields:

• site: ForeignKey to Site
• industry_sector: ForeignKey to IndustrySector (optional)
• name: CharField
• slug: SlugField (unique per site)
• is_active: BooleanField
• status: CharField

Validation: Maximum 5 active sectors per site

Planner Models

Keywords

Table: igny8_keywords

Inherits: SiteSectorBaseModel

Fields:

• keyword: CharField
• volume: IntegerField (optional)
• difficulty: IntegerField (optional)
• intent: CharField (optional)
• status: CharField
• cluster: ManyToManyField to Clusters

Clusters

Table: igny8_clusters

Inherits: SiteSectorBaseModel

Fields:

• name: CharField
• description: TextField (optional)
• keywords_count: IntegerField (calculated)
• volume: IntegerField (calculated)
• status: CharField
• keywords: ManyToManyField to Keywords

ContentIdeas

Table: igny8_content_ideas

Inherits: SiteSectorBaseModel

Fields:

• idea_title: CharField
• description: TextField
• content_type: CharField
• content_structure: CharField
• target_keywords: TextField
• keyword_cluster: ForeignKey to Clusters
• estimated_word_count: IntegerField
• status: CharField

Writer Models

Tasks

Table: igny8_tasks

Inherits: SiteSectorBaseModel

Fields:

• title: CharField
• description: TextField
• cluster: ForeignKey to Clusters (optional)
• idea: ForeignKey to ContentIdeas (optional)
• content_type: CharField
• content_structure: CharField
• status: CharField

Content

Table: igny8_content

Inherits: SiteSectorBaseModel

Fields:

• task: OneToOneField to Tasks
• html_content: TextField
• word_count: IntegerField
• status: CharField
• wp_post_id: IntegerField (optional)
• meta_title: CharField (optional)
• meta_description: TextField (optional)
• primary_keyword: CharField (optional)
• secondary_keywords: TextField (optional)

Images

Table: igny8_images

Inherits: SiteSectorBaseModel

Fields:

• task: ForeignKey to Tasks (optional)
• content: ForeignKey to Content (optional)
• image_type: CharField (choices: featured, in_article, desktop, mobile)
• prompt: TextField
• image_url: CharField
• status: CharField

System Models

AIPrompt

Table: igny8_ai_prompts

Inherits: AccountBaseModel

Fields:

• prompt_type: CharField
• prompt_value: TextField
• function_name: CharField (optional)

IntegrationSettings

Table: igny8_integration_settings

Inherits: AccountBaseModel

Fields:

• integration_type: CharField (choices: openai, runware, image_generation)
• config: JSONField

AuthorProfile

Table: igny8_author_profiles

Inherits: AccountBaseModel

Fields:

• name: CharField
• description: TextField
• tone: CharField
• language: CharField

Strategy

Table: igny8_strategies

Inherits: AccountBaseModel

Fields:

• name: CharField
• description: TextField
• sector: ForeignKey to Sector (optional)
• prompt_types: JSONField
• section_logic: JSONField
• is_active: BooleanField

Billing Models

CreditTransaction

Table: igny8_credit_transactions

Inherits: AccountBaseModel

Fields:

• transaction_type: CharField (choices: purchase, usage, refund, adjustment)
• amount: IntegerField
• balance_after: IntegerField
• description: TextField (optional)

CreditUsageLog

Table: igny8_credit_usage_logs

Inherits: AccountBaseModel

Fields:

• operation_type: CharField
• credits_used: IntegerField
• cost_usd: DecimalField
• details: JSONField (optional)

---

ViewSets

Base ViewSets

AccountModelViewSet

File: api/base.py

Purpose: Base ViewSet with automatic account filtering.

Methods:

• get_queryset(): Filters queryset by request.account (with admin/developer override)
• perform_create(): Sets account on created objects
• get_serializer_context(): Adds account to serializer context

Access Control:

• Admin/Developer users: Bypass account filtering
• System account users: Bypass account filtering
• Regular users: Only see data from their account

SiteSectorModelViewSet

File: api/base.py

Purpose: Base ViewSet with site/sector filtering and access control.

Inherits: AccountModelViewSet

Methods:

• get_queryset(): Filters by account, accessible sites, and optional site_id/sector_id
• perform_create(): Validates site access and sector-site relationship
• get_serializer_context(): Adds accessible sites and sectors to context

Access Control:

• Developers: All active sites
• System account users: All active sites
• Owners/Admins: All sites in their account
• Editors/Viewers: Only sites granted via SiteUserAccess

Planner ViewSets

KeywordViewSet

Inherits: SiteSectorModelViewSet

Actions:

• list(): List keywords with filtering
• create(): Create keyword
• retrieve(): Get keyword details
• update(): Update keyword
• destroy(): Delete keyword
• auto_cluster(): Auto-cluster keywords using AI
• bulk_delete(): Bulk delete keywords
• bulk_update_status(): Bulk update keyword status
• export_csv(): Export keywords to CSV
• import_csv(): Import keywords from CSV

Filtering:

• Search: keyword field
• Filters: status, cluster_id, intent
• Custom: difficulty_min, difficulty_max, volume_min, volume_max
• Ordering: created_at, volume, difficulty

ClusterViewSet

Inherits: SiteSectorModelViewSet

Actions:

• list(): List clusters
• create(): Create cluster
• retrieve(): Get cluster details
• update(): Update cluster
• destroy(): Delete cluster
• auto_generate_ideas(): Auto-generate content ideas for clusters

ContentIdeasViewSet

Inherits: SiteSectorModelViewSet

Actions:

• list(): List content ideas
• create(): Create content idea
• retrieve(): Get content idea details
• update(): Update content idea
• destroy(): Delete content idea

Writer ViewSets

TasksViewSet

Inherits: SiteSectorModelViewSet

Actions:

• list(): List tasks
• create(): Create task
• retrieve(): Get task details
• update(): Update task
• destroy(): Delete task
• auto_generate_content(): Auto-generate content for tasks
• generate_images(): Generate images for image records
• bulk_delete(): Bulk delete tasks
• bulk_update(): Bulk update task status

Filtering:

• Search: title, keywords
• Filters: status, cluster_id, content_type, content_structure
• Ordering: title, created_at, word_count, status

ImagesViewSet

Inherits: SiteSectorModelViewSet

Actions:

• list(): List images
• create(): Create image
• retrieve(): Get image details
• update(): Update image
• destroy(): Delete image
• generate_images(): Generate images using AI

System ViewSets

IntegrationSettingsViewSet

Inherits: viewsets.ViewSet

Actions:

• list(): List integrations
• retrieve(): Get integration settings
• update(): Save integration settings
• save_post(): Save integration settings (POST)
• test_connection(): Test API connection
• test_openai(): Test OpenAI connection
• test_runware(): Test Runware connection
• generate_image(): Test image generation
• task_progress(): Get Celery task progress

AIPromptViewSet

Inherits: AccountModelViewSet

Actions:

• list(): List prompts
• create(): Create prompt
• retrieve(): Get prompt details
• update(): Update prompt
• destroy(): Delete prompt
• reset_to_default(): Reset prompt to default value

AuthorProfileViewSet

Inherits: AccountModelViewSet

Actions:

• list(): List author profiles
• create(): Create author profile
• retrieve(): Get author profile details
• update(): Update author profile
• destroy(): Delete author profile

---

Serializers

Planner Serializers

KeywordSerializer

Fields: All Keyword model fields

Validation: Validates keyword uniqueness, cluster belongs to same sector

ClusterSerializer

Fields: All Cluster model fields

Read-Only Fields: keywords_count, volume (calculated)

ContentIdeasSerializer

Fields: All ContentIdeas model fields

Writer Serializers

TasksSerializer

Fields: All Tasks model fields

ContentSerializer

Fields: All Content model fields

ImagesSerializer

Fields: All Images model fields

System Serializers

AIPromptSerializer

Fields: All AIPrompt model fields

IntegrationSettingsSerializer

Fields: All IntegrationSettings model fields

---

Celery Tasks

AI Task Entry Point

run_ai_task

File: ai/tasks.py

Purpose: Unified Celery task entrypoint for all AI functions

Parameters:

• function_name: Function name (e.g., 'auto_cluster')
• payload: Function-specific payload
• account_id: Account ID

Flow:

1. Gets account from account_id
2. Gets function instance from registry
3. Creates AIEngine
4. Executes function via AIEngine

Planner Tasks

auto_cluster_keywords_task

File: modules/planner/tasks.py (legacy, now uses run_ai_task)

Purpose: Auto-cluster keywords using AI

Parameters:

• keyword_ids: List of keyword IDs
• account_id: Account ID
• site_id: Site ID
• sector_id: Sector ID

Progress Tracking: Updates progress with request_steps and response_steps

auto_generate_ideas_task

File: modules/planner/tasks.py (legacy, now uses run_ai_task)

Purpose: Auto-generate content ideas for clusters

Parameters:

• cluster_ids: List of cluster IDs
• account_id: Account ID

Progress Tracking: Updates progress for each cluster

Writer Tasks

auto_generate_content_task

File: modules/writer/tasks.py (legacy, now uses run_ai_task)

Purpose: Auto-generate content for tasks

Parameters:

• task_ids: List of task IDs
• account_id: Account ID

Progress Tracking: Updates progress for each task

process_image_generation_queue

File: modules/writer/views.py (via ai/tasks.py)

Purpose: Generate images for image records

Parameters:

• image_ids: List of image IDs
• account_id: Account ID
• content_id: Content ID (optional)

Progress Tracking: Updates progress for each image sequentially

---

API Endpoints

Base URL

/api/v1/

Planner Endpoints

• GET /api/v1/planner/keywords/ - List keywords
• POST /api/v1/planner/keywords/ - Create keyword
• GET /api/v1/planner/keywords/{id}/ - Get keyword
• PUT /api/v1/planner/keywords/{id}/ - Update keyword
• DELETE /api/v1/planner/keywords/{id}/ - Delete keyword
• POST /api/v1/planner/keywords/auto_cluster/ - Auto-cluster keywords
• POST /api/v1/planner/keywords/bulk_delete/ - Bulk delete keywords
• POST /api/v1/planner/keywords/bulk_update_status/ - Bulk update status
• GET /api/v1/planner/keywords/export_csv/ - Export keywords
• POST /api/v1/planner/keywords/import_csv/ - Import keywords
• GET /api/v1/planner/clusters/ - List clusters
• POST /api/v1/planner/clusters/auto_generate_ideas/ - Auto-generate ideas
• GET /api/v1/planner/ideas/ - List content ideas

Writer Endpoints

• GET /api/v1/writer/tasks/ - List tasks
• POST /api/v1/writer/tasks/auto_generate_content/ - Auto-generate content
• POST /api/v1/writer/images/generate_images/ - Generate images

System Endpoints

• GET /api/v1/system/settings/integrations/{pk}/ - Get integration settings
• PUT /api/v1/system/settings/integrations/{pk}/ - Save integration settings
• POST /api/v1/system/settings/integrations/{pk}/test_openai/ - Test OpenAI
• POST /api/v1/system/settings/integrations/{pk}/test_runware/ - Test Runware
• POST /api/v1/system/settings/integrations/{pk}/generate_image/ - Test image generation
• GET /api/v1/system/settings/task_progress/{task_id}/ - Get task progress

---

Base Classes

AccountModelViewSet

File: api/base.py

Purpose: Base ViewSet with automatic account filtering

Features:

• Automatic account filtering
• Admin/Developer override
• Account context in serializers

SiteSectorModelViewSet

File: api/base.py

Purpose: Base ViewSet with site/sector filtering

Features:

• Account filtering (inherited)
• Site access control
• Sector validation
• Accessible sites/sectors in serializer context

---

Middleware

AccountContextMiddleware

File: middleware/account.py

Purpose: Sets request.account from JWT token

Functionality:

• Extracts account ID from JWT token
• Loads Account object
• Sets request.account

ResourceTrackerMiddleware

File: middleware/resource_tracker.py

Purpose: Tracks API request metrics

Functionality:

• Tracks CPU, memory, I/O usage
• Stores metrics in cache
• Provides metrics endpoint

---

Utilities

Content Normalizer

File: utils/content_normalizer.py

Purpose: Content processing utilities

Functions:

• normalize_content(): Converts plain text to HTML
• _extract_body_content(): Extracts body content from HTML

---

Modules

Planner Module

Purpose: Keyword management and content planning

Models: Keywords, Clusters, ContentIdeas

ViewSets: KeywordViewSet, ClusterViewSet, ContentIdeasViewSet

Tasks: Auto-cluster keywords, Auto-generate ideas

Writer Module

Purpose: Content generation and management

Models: Tasks, Content, Images

ViewSets: TasksViewSet, ImagesViewSet

Tasks: Auto-generate content, Generate images

System Module

Purpose: System configuration and AI settings

Models: AIPrompt, IntegrationSettings, AuthorProfile, Strategy

ViewSets: AIPromptViewSet, IntegrationSettingsViewSet, AuthorProfileViewSet

Billing Module

Purpose: Credit management and usage tracking

Models: CreditTransaction, CreditUsageLog

ViewSets: CreditTransactionViewSet, CreditUsageLogViewSet

Services: CreditService (check, deduct, add, calculate credits)

---

Summary

The IGNY8 backend provides:

1. Multi-Tenancy: Complete account isolation with automatic filtering
2. RESTful API: DRF ViewSets with consistent response format
3. Celery Integration: Asynchronous task processing
4. Hierarchical Organization: Account > Site > Sector > Content structure
5. AI Framework: Unified AI framework for all AI operations
6. Progress Tracking: Real-time progress updates for Celery tasks
7. Module-Based Design: Clear separation of concerns
8. Base Classes: Reusable ViewSets for common patterns
9. Middleware: Account context and resource tracking
10. Utilities: Content processing and AI integration

This architecture ensures scalability, maintainability, and extensibility while providing a robust foundation for the IGNY8 platform.