1629 lines
76 KiB
Markdown
1629 lines
76 KiB
Markdown
# Phase 2 Build Plan — Module Builds (9 Docs)
|
||
|
||
**Purpose:** This is a single instruction set for Claude Code to build all 9 Phase 2 execution documents. Read this entire file, then build each doc one at a time.
|
||
|
||
---
|
||
|
||
## CRITICAL BASELINE RULES (from V2-Execution-Docs-Updated)
|
||
|
||
Before writing ANY doc, internalize these codebase-verified facts:
|
||
|
||
### Primary Keys
|
||
- ALL models use `BigAutoField` (integer PKs, NOT UUIDs)
|
||
- `DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'`
|
||
- Every ID in API requests/responses is an integer
|
||
|
||
### Model Names (PLURAL)
|
||
- `Clusters` (not Cluster), `Keywords` (not Keyword), `Tasks` (not Task)
|
||
- `ContentIdeas` (not ContentIdea), `Images` (not Image), `Content` (stays singular)
|
||
|
||
### App Structure
|
||
- Project root: `igny8_core/`
|
||
- SAG app: `igny8_core/sag/`
|
||
- App labels: `igny8_core_auth`, `planner`, `writer`, `automation`, `integration`, etc.
|
||
- All tables use `igny8_` prefix (e.g., `igny8_content`, `igny8_clusters`)
|
||
|
||
### Frontend Stack
|
||
- TypeScript (`.tsx` files, NOT `.jsx`)
|
||
- Zustand for state management (NOT Redux)
|
||
- Vite ^6.1.0 as build tool
|
||
- React ^19.0.0
|
||
- Vitest + Playwright for testing (NOT Selenium/Cypress)
|
||
|
||
### Verified Codebase Versions (what actually runs today)
|
||
- Python 3.11-slim (Dockerfile)
|
||
- Django >=5.2.7 (requirements.txt)
|
||
- Node 18-alpine (Dockerfile.dev)
|
||
- Celery >=5.3.0
|
||
- WP Plugin: IGNY8 WordPress Bridge v1.5.2
|
||
- These are CURRENT versions. Upgrade targets (Python 3.14, Django 6.0, etc.) are in 00B but are SEPARATE tasks.
|
||
|
||
### Model Inheritance
|
||
- `AccountBaseModel` provides: `account` FK, `created_by`, `created_at`, `updated_at`
|
||
- `SiteSectorBaseModel` extends AccountBaseModel with: `site` FK, `sector` FK
|
||
- `SoftDeletableModel` mixin provides soft delete via `SoftDeleteManager`
|
||
|
||
### Existing Models Phase 2 Extends
|
||
- `Content` (writer app) — content_type: post/page/product/taxonomy; content_structure: article/guide/comparison/review/listicle/landing_page/etc.
|
||
- `Tasks` (writer app) — writing tasks, status: pending/generating/completed/failed
|
||
- `Images` (writer app) — generated images for content
|
||
- `Clusters` (planner app) — keyword clusters, linked to SAGCluster via optional FK (added in Phase 1)
|
||
- `ContentIdeas` (planner app) — idea generation
|
||
- `Keywords` (planner app) — keyword entities
|
||
- `SAGBlueprint`, `SAGAttribute`, `SAGCluster`, `SectorAttributeTemplate` (sag app) — Phase 1 models
|
||
- `SiteIntegration` (integration app) — stores WordPress connection details
|
||
- `SyncEvent` (integration app) — logs sync operations
|
||
- `OptimizationTask` (optimization app) — exists but inactive
|
||
- `ContentTaxonomy`, `ContentTaxonomyRelation` (writer app) — taxonomy entities
|
||
|
||
### 7-Stage Automation Pipeline (existing)
|
||
| Stage | Function | AI |
|
||
|-------|----------|-----|
|
||
| 1 | Keywords → Clusters | Yes (auto_cluster) |
|
||
| 2 | Clusters → Ideas | Yes (generate_ideas) |
|
||
| 3 | Ideas → Tasks | No |
|
||
| 4 | Tasks → Content | Yes (generate_content) |
|
||
| 5 | Content → Image Prompts | Yes (generate_image_prompts) |
|
||
| 6 | Image Prompts → Images | Yes (generate_images) |
|
||
| 7 | Auto-approval → Publish | No |
|
||
|
||
Phase 2 adds Stage 8 (Socializer) and Stage 9 (Video Creator) to this pipeline.
|
||
|
||
### AI Function Pattern
|
||
- Class-based AI functions (NOT decorator-based)
|
||
- Located in `igny8_core/ai/functions/`
|
||
- Each function: `class XxxFunction(BaseAIFunction)` with `execute()` method
|
||
- AI provider abstraction via `IntegrationProvider` model + `IntegrationSettings`
|
||
|
||
### Source of Truth
|
||
- Start every doc with: `**Source of Truth:** Codebase at /data/app/igny8/`
|
||
- Codebase overrides any planning doc
|
||
|
||
---
|
||
|
||
## DOC STRUCTURE STANDARD
|
||
|
||
Every doc MUST follow this structure:
|
||
|
||
```
|
||
# [Phase.Sub] — Title
|
||
**Source of Truth:** Codebase at `/data/app/igny8/`
|
||
**Version:** 1.0
|
||
**Date:** 2026-03-23
|
||
|
||
## 1. Current State
|
||
What exists today relevant to this sub-phase.
|
||
|
||
## 2. What to Build
|
||
Complete scope.
|
||
|
||
## 3. Data Models & APIs
|
||
New/modified models, endpoints, serializers, database tables.
|
||
|
||
## 4. Implementation Steps
|
||
Ordered steps with exact commands, file paths, code snippets.
|
||
|
||
## 5. Acceptance Criteria
|
||
Testable checkboxes.
|
||
|
||
## 6. Claude Code Instructions
|
||
Specific execution guidance — file creation order, test commands, verification.
|
||
```
|
||
|
||
---
|
||
|
||
## EXISTING BACKEND CONTEXT (relevant to ALL Phase 2 modules)
|
||
|
||
### Content Model (writer app)
|
||
```python
|
||
class Content(SiteSectorBaseModel, SoftDeletableModel):
|
||
content_type = models.CharField(max_length=50) # post/page/product/taxonomy
|
||
content_structure = models.CharField(max_length=50) # article/guide/comparison/review/listicle/landing_page/etc.
|
||
title = models.CharField(max_length=500)
|
||
content_html = models.TextField()
|
||
content_text = models.TextField()
|
||
meta_title = models.CharField(max_length=200)
|
||
meta_description = models.TextField()
|
||
schema_markup = models.JSONField(null=True)
|
||
status = models.CharField() # draft/approved/published/archived
|
||
task = models.ForeignKey('Tasks', ...)
|
||
cluster = models.ForeignKey('Clusters', ...)
|
||
word_count = models.IntegerField()
|
||
# ... other fields
|
||
```
|
||
|
||
### Tasks Model (writer app)
|
||
```python
|
||
class Tasks(SiteSectorBaseModel, SoftDeletableModel):
|
||
content_type = models.CharField(max_length=50)
|
||
content_structure = models.CharField(max_length=50)
|
||
title = models.CharField(max_length=500)
|
||
brief = models.JSONField(null=True)
|
||
status = models.CharField() # pending/generating/completed/failed
|
||
cluster = models.ForeignKey('Clusters', ...)
|
||
idea = models.ForeignKey('ContentIdeas', ...)
|
||
# ... other fields
|
||
```
|
||
|
||
### Existing API Patterns
|
||
- All APIs under `/api/v1/{app_name}/`
|
||
- DRF ViewSets with ModelSerializer
|
||
- Pagination: `PageNumberPagination` (default 20)
|
||
- Authentication: JWT (SimpleJWT)
|
||
- Permissions: `IsAuthenticated` + custom `HasAccountAccess`
|
||
- Filtering: `django-filter` with `FilterSet` classes
|
||
|
||
### Celery Task Pattern
|
||
```python
|
||
# igny8_core/{app}/tasks.py
|
||
from celery import shared_task
|
||
|
||
@shared_task(bind=True, max_retries=3, default_retry_delay=60)
|
||
def task_name(self, **kwargs):
|
||
try:
|
||
# work
|
||
except Exception as exc:
|
||
self.retry(exc=exc)
|
||
```
|
||
|
||
### Credit System (billing app)
|
||
- `CreditTransaction` — tracks credit changes
|
||
- `CreditUsageLog` — logs per-operation usage
|
||
- `CreditCostConfig` — configurable cost per operation type
|
||
- Pattern: check balance → execute → deduct → log
|
||
|
||
---
|
||
|
||
## DOC 02A: content-types-extension.md
|
||
|
||
**File:** `V2-Execution-Docs/02A-content-types-extension.md`
|
||
**Scope:** Extend the content generation pipeline beyond blog posts to support pages, products, services, company pages, comparison pages, and brand pages — each with type-specific AI prompts, presets, and structure templates.
|
||
**Depends On:** 01E (blueprint-aware pipeline provides stage 0 context + type-specific routing)
|
||
|
||
### Source Material:
|
||
- temp/IGNY8-Project-Files/IGNY8-Consolidated-Module-Plans.md (Content Types section)
|
||
- Live Docs on Server/igny8-app-docs/plans/Future_Moduels_Features/Content_Types_Writing_Plan.md
|
||
- Live Docs on Server/igny8-app-docs/plans/final-dev-guides-of-futute-implementation/DocB-Platform-Modules-Dev-Guide.md
|
||
|
||
### What to Cover:
|
||
|
||
**1. Current State:**
|
||
- Content model already has `content_type` (post/page/product/taxonomy) and `content_structure` (article/guide/comparison/review/listicle/landing_page/etc.)
|
||
- Only `post` type is actively used by the generation pipeline
|
||
- AI function `generate_content` produces blog-style articles regardless of content_type
|
||
- No type-specific prompts, structure templates, or presets exist
|
||
|
||
**2. What to Build — 6 Content Type Extensions:**
|
||
|
||
**Type 1: Pages** (content_type=page)
|
||
- Structures: landing_page, about_page, contact_page, home_page, generic_page
|
||
- Page-specific presets: section-based layouts rather than linear article flow
|
||
- Landing pages: hero + feature blocks + CTA sections
|
||
- AI prompt: professional, conversion-focused, concise, benefit-driven
|
||
- Schema: WebPage, AboutPage, ContactPage as appropriate
|
||
|
||
**Type 2: Products** (content_type=product)
|
||
- Structures: product_page, product_comparison, product_roundup
|
||
- WooCommerce-ready fields: price_range, features JSON, specifications JSON, pros_cons JSON
|
||
- AI prompt: feature-benefit mapping, comparison tables, spec sheets, buyer personas
|
||
- Schema: Product with offers, aggregateRating, review
|
||
- Image requirements: product hero, feature highlights, comparison visuals
|
||
|
||
**Type 3: Services** (content_type=service)
|
||
- Structures: service_page, service_area_page, service_comparison
|
||
- Fields: process_steps JSON, outcomes JSON, pricing_tiers JSON, areas_served JSON, faqs JSON
|
||
- AI prompt: problem-solution, trust-building, process explanation, CTA-heavy
|
||
- Schema: Service, ProfessionalService, LocalBusiness+hasOfferCatalog
|
||
- Geographic targeting: generate area-specific variations from base service
|
||
|
||
**Type 4: Company Pages** (content_type=page, content_structure=company_page)
|
||
- Structures: about_company, team_page, careers_page, press_page
|
||
- AI prompt: brand voice emphasis, story-driven, credibility markers
|
||
- Schema: Organization, AboutPage
|
||
|
||
**Type 5: Comparison Pages** (content_type=post, content_structure=comparison)
|
||
- Structures: versus_page (A vs B), multi_comparison (top 5/10 list), alternative_page
|
||
- Fields: comparison_items JSON [{name, features, pros, cons, rating, verdict}]
|
||
- AI prompt: objective analysis, comparison tables, winner selection with reasoning
|
||
- Schema: Article with itemListElement (for multi-comparison)
|
||
|
||
**Type 6: Brand Pages** (content_type=page, content_structure=brand_page)
|
||
- Structures: brand_overview, brand_review, brand_alternative
|
||
- AI prompt: brand-focused, factual, includes company background
|
||
- Use case: sites that review/compare brands in their industry
|
||
|
||
**3. Data Models & APIs:**
|
||
|
||
New/modified fields on existing models:
|
||
- `Tasks.structure_template` — JSONField storing section layout for the content type
|
||
- `Tasks.type_presets` — JSONField with type-specific generation parameters
|
||
- `Content.sections` — JSONField for section-based content (pages, landing pages)
|
||
- `Content.structured_data` — JSONField for type-specific data (product specs, service steps, comparison items)
|
||
|
||
New model:
|
||
- `ContentTypeTemplate` (extends AccountBaseModel)
|
||
- `content_type` CharField
|
||
- `content_structure` CharField
|
||
- `template_name` CharField
|
||
- `section_layout` JSONField — ordered sections with types
|
||
- `ai_prompt_template` TextField — base prompt for this type
|
||
- `default_schema_type` CharField
|
||
- `default_word_count_range` JSONField — {min, max}
|
||
- `is_default` BooleanField — system-provided vs custom
|
||
- Table: `igny8_content_type_templates`
|
||
|
||
New endpoints:
|
||
- `GET /api/v1/writer/content-type-templates/` — list templates by type
|
||
- `POST /api/v1/writer/content-type-templates/` — create custom template
|
||
- `GET /api/v1/writer/content-type-templates/{id}/preview/` — preview generated structure
|
||
|
||
Modified endpoints:
|
||
- `POST /api/v1/writer/tasks/` — extend to accept content_type + content_structure + structure_template
|
||
- `POST /api/v1/writer/generate/` — extend to route to type-specific AI prompt
|
||
|
||
**4. AI Function Extensions:**
|
||
- Extend `GenerateContentFunction` with type routing
|
||
- Type-specific prompt templates in `AIPrompt` model (system app)
|
||
- Each type gets: system prompt, structure instructions, example output, schema generation instructions
|
||
- Blueprint context injection (from Phase 1): cluster type → content type mapping
|
||
|
||
**5. Blueprint Integration (from Phase 1):**
|
||
- SAGCluster.cluster_type maps to content_type:
|
||
- hub → page (landing_page), service → service, product → product
|
||
- comparison → post (comparison), brand → page (brand_page)
|
||
- informational → post (article/guide), commercial → post/page
|
||
- When pipeline Stage 0 runs, it reads cluster_type and sets appropriate content_type + content_structure on the Task
|
||
|
||
**Credit Costs:** Same as current (4-8 credits per generation) — type routing doesn't change cost, AI does more structured work but same token volume.
|
||
|
||
**Cross-references:** 01E (blueprint-aware pipeline — type routing), 01C (cluster types feed content type mapping), 02B (taxonomy content is a content_type=taxonomy extension), 02G (schema generation per type), 03B (content sync maps types to WP post types)
|
||
|
||
---
|
||
|
||
## DOC 02B: taxonomy-term-content.md
|
||
|
||
**File:** `V2-Execution-Docs/02B-taxonomy-term-content.md`
|
||
**Scope:** Make taxonomy terms first-class SEO content pages — generate rich landing page content for categories, tags, and SAG attribute terms, map terms to clusters, sync to WordPress.
|
||
**Depends On:** 02A (content_type=taxonomy uses the extended type system)
|
||
|
||
### Source Material:
|
||
- Live Docs on Server/igny8-app-docs/plans/Future_Moduels_Features/Taxonomy_Term_Content_Plan.md
|
||
- temp/IGNY8-Project-Files/IGNY8-Consolidated-Module-Plans.md (Taxonomy section)
|
||
- temp/IGNY8-Project-Files/SAG-IGNY8-Implementation.md (attribute → taxonomy mapping)
|
||
|
||
### What to Cover:
|
||
|
||
**1. Current State:**
|
||
- `ContentTaxonomy` model exists — stores external taxonomy references (name, slug, external_id, taxonomy_type)
|
||
- `ContentTaxonomyRelation` exists — links Content to ContentTaxonomy
|
||
- Taxonomy sync from WordPress is stubbed but not fully functional
|
||
- `Content.content_type='taxonomy'` exists as an option but not used in generation
|
||
- No content generation for terms — terms are metadata only
|
||
|
||
**2. What to Build:**
|
||
|
||
**Taxonomy Sync (WordPress → IGNY8):**
|
||
- Restore and complete ContentSyncService taxonomy sync
|
||
- Fetch: categories, tags, WooCommerce product_cat/product_tag/pa_* attributes
|
||
- Store in ContentTaxonomy with `taxonomy_type` (category/tag/product_cat/product_tag/attribute)
|
||
- Map terms to clusters using keyword overlap + semantic similarity
|
||
|
||
**Cluster Mapping Service:**
|
||
- Shared `cluster_mapping_service.py` used by multiple modules
|
||
- Algorithm: keyword overlap (40%) + semantic embedding similarity (40%) + title match (20%)
|
||
- Maps each ContentTaxonomy to a primary `Clusters` record + optional secondary_cluster_ids
|
||
- Confidence threshold: 0.6 (auto-map), <0.6 (suggest for manual review)
|
||
|
||
**Term Content Generation:**
|
||
- Content structures for taxonomy: `category_archive`, `tag_archive`, `attribute_archive`, `cluster_hub`
|
||
- Each generates: H1 title, rich description (500-1500 words), FAQ section (5-8 Q&As), related terms, meta title/description
|
||
- AI function: `GenerateTermContentFunction` — takes term name, assigned cluster keywords, existing content titles under term, parent/sibling terms
|
||
- Output: structured JSON with sections (intro, overview, FAQ, related)
|
||
- Schema: CollectionPage with itemListElement
|
||
|
||
**Term Content Sync (IGNY8 → WordPress):**
|
||
- Push generated content to WordPress term descriptions
|
||
- Store in term meta: `_igny8_term_content` (HTML), `_igny8_term_faq` (JSON), `_igny8_term_related_terms` (JSON)
|
||
- Endpoint: `POST /wp-json/igny8/v1/terms/{id}/content`
|
||
|
||
**3. Data Models & APIs:**
|
||
|
||
Modified models:
|
||
- `ContentTaxonomy` — add fields:
|
||
- `cluster_id` ForeignKey to Clusters (nullable)
|
||
- `secondary_cluster_ids` JSONField (list of ints)
|
||
- `mapping_confidence` FloatField
|
||
- `mapping_status` CharField (auto_mapped/manual_mapped/unmapped/suggested)
|
||
- `term_content` TextField (generated rich content HTML)
|
||
- `term_faq` JSONField (list of {question, answer})
|
||
- `meta_title` CharField
|
||
- `meta_description` TextField
|
||
- `content_status` CharField (none/generating/generated/published)
|
||
|
||
New endpoints:
|
||
- `GET /api/v1/writer/taxonomy/terms/?site_id=X` — list terms with mapping status
|
||
- `POST /api/v1/writer/taxonomy/terms/sync/` — trigger WordPress → IGNY8 sync
|
||
- `POST /api/v1/writer/taxonomy/terms/{id}/map-cluster/` — manual cluster assignment
|
||
- `GET /api/v1/writer/taxonomy/terms/{id}/cluster-suggestions/` — AI cluster suggestions
|
||
- `POST /api/v1/writer/taxonomy/terms/create-tasks/` — bulk create content generation tasks for terms
|
||
- `POST /api/v1/writer/taxonomy/terms/{id}/publish/` — push term content to WordPress
|
||
- `GET /api/v1/writer/taxonomy/terms/unmapped/` — terms needing cluster assignment
|
||
|
||
**Credit Costs:** 1 credit sync (per batch), 4-6 credits per term content generation, 3-5 credits optimization
|
||
|
||
**Cross-references:** 02A (content_type=taxonomy type system), 01A (SAGAttribute → taxonomy mapping), 01D (wizard creates initial taxonomy plan), 03B (connected plugin receives term content), 03C (theme renders term landing pages)
|
||
|
||
---
|
||
|
||
## DOC 02C: gsc-integration.md
|
||
|
||
**File:** `V2-Execution-Docs/02C-gsc-integration.md`
|
||
**Scope:** Google Search Console integration — OAuth connection, URL Inspection API (2K/day quota), auto-indexing for new content, re-inspection schedule, search analytics dashboard, plugin-side GSC data sync.
|
||
**Depends On:** 01A (SAG models exist for blueprint-aware indexing priorities) — can build in PARALLEL with Phase 1 modules
|
||
|
||
### Source Material:
|
||
- Live Docs on Server/igny8-app-docs/plans/gsc_integratin.md
|
||
- Live Docs on Server/igny8-app-docs/plans/Future_Moduels_Features/GSC_plnning_to_be_updated_consistent_with_supported_API_and_Igny8_app.md
|
||
- Live Docs on Server/igny8-app-docs/plans/final-dev-guides-of-futute-implementation/DocB-Platform-Modules-Dev-Guide.md
|
||
|
||
### What to Cover:
|
||
|
||
**1. Current State:**
|
||
- No GSC integration exists in IGNY8
|
||
- `SiteIntegration` model in integration app stores WordPress connections but no Google connections
|
||
- No OAuth infrastructure for Google APIs
|
||
- WordPress plugin v1.5.2 has no GSC features
|
||
- Integration app exists with `SyncEvent` model for logging
|
||
|
||
**2. What to Build:**
|
||
|
||
**OAuth 2.0 Connection:**
|
||
- Google Cloud project with Search Console API + URL Inspection API enabled
|
||
- OAuth 2.0 flow: IGNY8 backend → Google consent → callback → store encrypted tokens
|
||
- Scopes: `https://www.googleapis.com/auth/webmasters.readonly`, `https://www.googleapis.com/auth/indexing`
|
||
- Token refresh via background Celery task (tokens expire after 1 hour)
|
||
- Multi-property support (one account can have multiple GSC properties)
|
||
|
||
**URL Inspection API (core feature):**
|
||
- Google API: `POST https://searchconsole.googleapis.com/v1/urlInspection/index:inspect`
|
||
- Request: `{inspectionUrl, siteUrl}`
|
||
- Response: verdict (PASS/PARTIAL/FAIL/NEUTRAL), coverageState, indexingState, crawledAs, robotsTxtState, lastCrawlTime
|
||
- **Quota: 2,000 inspections/day** per property, resets midnight PT
|
||
- Rate limit: 1 request per 3 seconds (600/30min safe limit)
|
||
|
||
**Indexing Queue System:**
|
||
- `IndexingQueue` model with priority-based processing
|
||
- Priority levels:
|
||
- 100: Auto-indexing newly published content (highest)
|
||
- 90: Auto re-inspection of previously submitted
|
||
- 70: Manual inspection request
|
||
- 50: Manual inspection (info only, no submit)
|
||
- 30: Scheduled re-inspection
|
||
- Celery task processes queue respecting daily quota + rate limits
|
||
- Dashboard shows quota usage: X/2000 used today
|
||
|
||
**Re-Inspection Schedule:**
|
||
- After initial inspection → schedule follow-up checks
|
||
- Check 1: 24 hours after submission
|
||
- Check 2: 3 days after
|
||
- Check 3: 6 days after
|
||
- Check 4: 13 days after
|
||
- If still not indexed after Check 4 → mark "manual_review_needed", stop auto-checking
|
||
- Track status progression per URL
|
||
|
||
**Search Analytics API:**
|
||
- Google API: `POST https://searchconsole.googleapis.com/v3/sites/{siteUrl}/searchAnalytics/query`
|
||
- Dimensions: page, query, country, device, date
|
||
- Metrics: clicks, impressions, ctr, position
|
||
- Date range: up to 16 months historical
|
||
- Cache results in `MetricsCache` with TTL (refresh daily via Celery)
|
||
- Frontend dashboard: filterable by date range, page, keyword
|
||
|
||
**Auto-Indexing for New Content:**
|
||
- When content status changes to `published` → auto-queue for inspection
|
||
- Priority 100 (highest)
|
||
- If content already has a URL via publishing → inspect that URL
|
||
- Blueprint-aware priority: hub pages get inspected before supporting articles
|
||
|
||
**Plugin-Side GSC Status Sync:**
|
||
- IGNY8 backend pushes index statuses to WordPress plugin
|
||
- Endpoint: `POST /wp-json/igny8/v1/gsc/status-sync`
|
||
- Plugin displays index status badges on post list table
|
||
- Statuses: ⏳ pending_inspection, ✓ indexed, ✗ not_indexed, ➡ indexing_requested, 🚫 error_noindex
|
||
|
||
**3. Data Models & APIs:**
|
||
|
||
New models (all in integration app):
|
||
- `GSCConnection` (extends AccountBaseModel)
|
||
- `site` ForeignKey to Site
|
||
- `google_email` CharField
|
||
- `access_token` TextField (encrypted)
|
||
- `refresh_token` TextField (encrypted)
|
||
- `token_expiry` DateTimeField
|
||
- `gsc_property_url` CharField — e.g., `sc-domain:example.com`
|
||
- `status` CharField (active/expired/revoked)
|
||
- Table: `igny8_gsc_connections`
|
||
|
||
- `URLInspectionRecord` (extends SiteSectorBaseModel)
|
||
- `url` URLField
|
||
- `content` ForeignKey to Content (nullable — not all URLs are IGNY8 content)
|
||
- `last_inspection_result` JSONField — full Google response
|
||
- `verdict` CharField (PASS/PARTIAL/FAIL/NEUTRAL)
|
||
- `coverage_state` CharField
|
||
- `indexing_state` CharField
|
||
- `last_crawled` DateTimeField (nullable)
|
||
- `last_inspected` DateTimeField
|
||
- `inspection_count` IntegerField (default 0)
|
||
- `next_inspection` DateTimeField (nullable)
|
||
- `status` CharField (pending_inspection/indexed/not_indexed/indexing_requested/error_noindex/manual_review)
|
||
- Table: `igny8_url_inspection_records`
|
||
|
||
- `IndexingQueue` (extends SiteSectorBaseModel)
|
||
- `url` URLField
|
||
- `url_inspection_record` ForeignKey to URLInspectionRecord (nullable)
|
||
- `priority` IntegerField (30-100)
|
||
- `status` CharField (queued/processing/completed/failed/quota_exceeded)
|
||
- `date_added` DateTimeField (auto_now_add)
|
||
- `date_processed` DateTimeField (nullable)
|
||
- `error_message` TextField (nullable)
|
||
- Table: `igny8_indexing_queue`
|
||
|
||
- `GSCMetricsCache` (extends SiteSectorBaseModel)
|
||
- `metric_type` CharField (search_analytics/page_performance/keyword_performance)
|
||
- `dimension_filters` JSONField — what was queried
|
||
- `data` JSONField — cached results
|
||
- `date_range_start` DateField
|
||
- `date_range_end` DateField
|
||
- `expires_at` DateTimeField
|
||
- Table: `igny8_gsc_metrics_cache`
|
||
|
||
- `GSCDailyQuota` (extends SiteSectorBaseModel)
|
||
- `date` DateField
|
||
- `inspections_used` IntegerField (default 0)
|
||
- `quota_limit` IntegerField (default 2000)
|
||
- Table: `igny8_gsc_daily_quota`
|
||
|
||
New endpoints:
|
||
- `POST /api/v1/integration/gsc/connect/` — initiate OAuth flow (returns redirect URL)
|
||
- `GET /api/v1/integration/gsc/callback/` — OAuth callback (stores tokens)
|
||
- `DELETE /api/v1/integration/gsc/disconnect/` — revoke + delete connection
|
||
- `GET /api/v1/integration/gsc/properties/` — list connected GSC properties
|
||
- `GET /api/v1/integration/gsc/quota/` — today's quota usage
|
||
- `POST /api/v1/integration/gsc/inspect/` — manual URL inspection (add to queue)
|
||
- `GET /api/v1/integration/gsc/inspections/?site_id=X` — list inspection records with filters
|
||
- `GET /api/v1/integration/gsc/inspections/{id}/` — single inspection detail
|
||
- `POST /api/v1/integration/gsc/inspect/bulk/` — bulk queue URLs
|
||
- `GET /api/v1/integration/gsc/analytics/` — search analytics data (cached)
|
||
- `GET /api/v1/integration/gsc/analytics/keywords/` — keyword performance
|
||
- `GET /api/v1/integration/gsc/analytics/pages/` — page performance
|
||
- `GET /api/v1/integration/gsc/analytics/export/` — CSV export
|
||
|
||
Celery tasks:
|
||
- `process_indexing_queue` — runs every 5 minutes, processes up to quota
|
||
- `refresh_gsc_tokens` — runs hourly, refreshes expiring tokens
|
||
- `fetch_search_analytics` — runs daily, caches metrics for all connected sites
|
||
- `schedule_reinspections` — runs daily, adds due re-inspections to queue
|
||
|
||
**Credit Costs:** 1 credit OAuth connection setup, 0.1 per 100 URL inspections, 0.05 per indexing request, 0.5/month/site for analytics caching
|
||
|
||
**Cross-references:** 01E (blueprint-aware pipeline triggers auto-indexing after publish), 02E (GSC impressions data feeds backlink KPI dashboard), 02F (GSC position data identifies optimization candidates), 03A (WP plugin standalone has GSC dashboard), 03B (connected mode syncs index statuses to WP)
|
||
|
||
---
|
||
|
||
## DOC 02D: linker-internal.md
|
||
|
||
**File:** `V2-Execution-Docs/02D-linker-internal.md`
|
||
**Scope:** SAG-based internal linking engine — 7 link types, scoring algorithm, anchor text strategy, density rules, link audit, new content mode, and existing content remediation.
|
||
**Depends On:** 01G (SAG health monitoring provides blueprint + cluster data for link planning)
|
||
|
||
### Source Material:
|
||
- temp/IGNY8-Project-Files/SAG-Doc3-Interlinking-Specification-PLAN.md (PRIMARY — comprehensive interlinking spec)
|
||
- Live Docs on Server/igny8-app-docs/plans/Future_Moduels_Features/Linker_Module_Plan.md
|
||
- temp/IGNY8-Project-Files/SAG-IGNY8-Implementation.md (3-layer architecture, linking rules)
|
||
|
||
### What to Cover:
|
||
|
||
**1. Current State:**
|
||
- `linker` Django app exists in INSTALLED_APPS but is **inactive** (behind feature flag)
|
||
- No models, views, or URLs currently active
|
||
- Content model has no link-related fields
|
||
- No link map, no link scoring, no anchor management
|
||
- Internal linking is entirely manual in WordPress
|
||
|
||
**2. What to Build — SAG-Based Internal Linking Engine:**
|
||
|
||
**Page Hierarchy (from SAG structure):**
|
||
- T1: Homepage — 1 page, 5-15 outbound / N/A inbound
|
||
- Sector landing pages — top-level category pages
|
||
- T2/T3: Cluster hubs — pillar pages per SAGCluster, 5-20 outbound / 3+ inbound
|
||
- T4: Supporting content — blog articles within clusters, 2-12 outbound / 1+ inbound
|
||
- Term pages — attribute taxonomy pages, 3+ outbound / 0 required
|
||
- Product/Service pages — 2-5 outbound / 1+ inbound
|
||
|
||
**7 Link Types:**
|
||
|
||
1. **Vertical Upward** (Supporting → Hub)
|
||
- MANDATORY: every supporting article MUST link to its cluster hub
|
||
- Limit: 1 per article (one hub)
|
||
- Placement: first 2 paragraphs
|
||
- Anchor: primary keyword 60%, hub title 30%, natural 10%
|
||
|
||
2. **Vertical Downward** (Hub → Supporting)
|
||
- Hub lists ALL its supporting articles in "Related Articles" section
|
||
- Additional contextual links in hub body
|
||
- No cap (all supporting content should be linked)
|
||
|
||
3. **Horizontal Sibling** (Supporting ↔ Supporting)
|
||
- Same-cluster articles linking to each other
|
||
- Max 2 per article
|
||
- Based on natural content overlap
|
||
|
||
4. **Cross-Cluster** (Hub ↔ Hub)
|
||
- Hubs sharing an attribute value can cross-link
|
||
- Max 2 per hub
|
||
- Scored by shared attribute overlap
|
||
|
||
5. **Taxonomy Contextual** (Term Page → Hubs)
|
||
- Term pages link to ALL cluster hubs using that attribute
|
||
- No cap
|
||
- Auto-generated from SAG attribute → cluster mapping
|
||
|
||
6. **Breadcrumb**
|
||
- Home → Sector → [Attribute] → Hub → Current Page
|
||
- Auto-generated from SAG hierarchy
|
||
- Every page gets breadcrumbs
|
||
|
||
7. **Related Content**
|
||
- 2-3 links in "Related Reading" section at end of article
|
||
- Scored by: semantic similarity + user journey logic (what to read next)
|
||
- Cross-cluster allowed for related content
|
||
|
||
**Link Scoring Algorithm (5 factors):**
|
||
- Shared attribute values: 40% weight
|
||
- Target page authority (inbound link count): 25%
|
||
- Keyword overlap: 20%
|
||
- Content recency: 10%
|
||
- Link count gap (pages with fewest inbound links get boost): 5%
|
||
- Score 0-100, threshold for automatic linking: 60+
|
||
|
||
**Anchor Text Rules:**
|
||
- Max 8 words, min 2 words, grammatically natural
|
||
- No exact-match anchor used >3 times to same target URL
|
||
- Anchor distribution per target: primary keyword 60%, page title 30%, natural phrase 10%
|
||
- Diversification audit: flag if any single anchor >40% of links to a target
|
||
|
||
**Link Density Rules (outbound per page type, by word count):**
|
||
| Page Type | <1000 words | 1000-2000 | 2000+ |
|
||
|-----------|------------|-----------|-------|
|
||
| Hub | 5-10 | 10-15 | 15-20 |
|
||
| Blog | 2-5 | 3-8 | 4-12 |
|
||
| Product/Service | 2-3 | 3-5 | 3-5 |
|
||
| Term Page | 3+ | 3+ | unlimited |
|
||
|
||
**Two Modes:**
|
||
|
||
**A. New Content Mode (integrated with pipeline):**
|
||
- After Stage 4 (content generated), before Stage 7 (publish):
|
||
- Calculate link targets using scoring algorithm
|
||
- Insert links into content_html at natural positions
|
||
- Store link plan in SAGLink records
|
||
- If content is a hub → auto-generate "Related Articles" section
|
||
|
||
**B. Existing Content Remediation (audit + fix):**
|
||
- Site-wide link audit: crawl all published content, build link map
|
||
- Identify: orphan pages (0 inbound), over-linked pages (>density max), missing mandatory links (supporting without hub link)
|
||
- Generate fix recommendations with priority scoring
|
||
- Batch application: insert missing links with AI-selected anchor text
|
||
|
||
**3. Data Models & APIs:**
|
||
|
||
New models (in linker app):
|
||
- `SAGLink` (extends SiteSectorBaseModel)
|
||
- `blueprint` ForeignKey to SAGBlueprint (nullable)
|
||
- `source_content` ForeignKey to Content
|
||
- `target_content` ForeignKey to Content
|
||
- `link_type` CharField (vertical_up/vertical_down/horizontal/cross_cluster/taxonomy/breadcrumb/related)
|
||
- `anchor_text` CharField(max_length=200)
|
||
- `anchor_type` CharField (primary_keyword/page_title/natural/branded)
|
||
- `placement_zone` CharField (in_body/related_section/breadcrumb/sidebar)
|
||
- `placement_position` IntegerField (paragraph number for in-body)
|
||
- `score` FloatField (0-100)
|
||
- `status` CharField (planned/inserted/verified/broken/removed)
|
||
- `is_mandatory` BooleanField (True for vertical_up)
|
||
- `inserted_at` DateTimeField (nullable)
|
||
- Table: `igny8_sag_links`
|
||
|
||
- `SAGLinkAudit` (extends SiteSectorBaseModel)
|
||
- `blueprint` ForeignKey to SAGBlueprint (nullable)
|
||
- `audit_date` DateTimeField
|
||
- `total_links` IntegerField
|
||
- `missing_mandatory` IntegerField
|
||
- `orphan_pages` IntegerField
|
||
- `broken_links` IntegerField
|
||
- `over_linked_pages` IntegerField
|
||
- `under_linked_pages` IntegerField
|
||
- `cluster_scores` JSONField — {cluster_id: {score, missing, issues[]}}
|
||
- `recommendations` JSONField — [{content_id, action, link_type, target_id, anchor_suggestion, priority}]
|
||
- `overall_health_score` FloatField (0-100)
|
||
- Table: `igny8_sag_link_audits`
|
||
|
||
- `LinkMap` (extends SiteSectorBaseModel)
|
||
- `source_url` URLField
|
||
- `source_content` ForeignKey to Content (nullable)
|
||
- `target_url` URLField
|
||
- `target_content` ForeignKey to Content (nullable)
|
||
- `anchor_text` CharField
|
||
- `is_internal` BooleanField
|
||
- `is_follow` BooleanField
|
||
- `position` CharField (in_content/navigation/footer/sidebar)
|
||
- `last_verified` DateTimeField
|
||
- `status` CharField (active/broken/removed)
|
||
- Table: `igny8_link_map`
|
||
|
||
Modified models:
|
||
- `Content` — add fields:
|
||
- `link_plan` JSONField (nullable) — planned links before insertion
|
||
- `links_inserted` BooleanField (default False)
|
||
- `inbound_link_count` IntegerField (default 0)
|
||
- `outbound_link_count` IntegerField (default 0)
|
||
|
||
New endpoints:
|
||
- `GET /api/v1/linker/links/?site_id=X` — list all SAGLinks with filters
|
||
- `POST /api/v1/linker/links/plan/` — generate link plan for a content piece
|
||
- `POST /api/v1/linker/links/insert/` — insert planned links into content HTML
|
||
- `POST /api/v1/linker/links/batch-insert/` — batch insert for multiple content
|
||
- `GET /api/v1/linker/audit/?site_id=X` — latest audit results
|
||
- `POST /api/v1/linker/audit/run/` — trigger site-wide link audit
|
||
- `GET /api/v1/linker/audit/recommendations/?site_id=X` — get fix recommendations
|
||
- `POST /api/v1/linker/audit/apply/` — apply recommended fixes (batch)
|
||
- `GET /api/v1/linker/link-map/?site_id=X` — full link map
|
||
- `GET /api/v1/linker/orphans/?site_id=X` — orphan pages
|
||
- `GET /api/v1/linker/health/?site_id=X` — cluster-level link health scores
|
||
- `GET /api/v1/linker/content/{id}/links/` — all links for a specific content
|
||
|
||
Celery tasks:
|
||
- `generate_link_plan` — runs after content generation, before publish
|
||
- `run_link_audit` — scheduled weekly or triggered manually
|
||
- `verify_links` — check for broken links (HTTP status check)
|
||
- `rebuild_link_map` — full crawl of published content
|
||
|
||
**Cluster-Level Health Score (per cluster):**
|
||
- Hub published and linked: 25 points
|
||
- All supporting articles have mandatory uplink: 25 points
|
||
- At least 1 cross-cluster link: 15 points
|
||
- Term pages link to hub: 15 points
|
||
- No broken links: 10 points
|
||
- Density within range: 10 points
|
||
- Total: 0-100 per cluster, averaged for site-wide score
|
||
|
||
**Credit Costs:** 1 credit per audit, 0.5 per 1-5 links generated (AI anchor text), 3-5 credits full site audit
|
||
|
||
**Cross-references:** 01G (SAG health score incorporates link health), 01A (SAGBlueprint/SAGCluster data), 02E (external backlinks complement internal links), 02F (optimizer uses link opportunities), 03A (WP plugin standalone has internal linking module), 03C (theme renders breadcrumbs + related content)
|
||
|
||
---
|
||
|
||
## DOC 02E: linker-external-backlinks.md
|
||
|
||
**File:** `V2-Execution-Docs/02E-linker-external-backlinks.md`
|
||
**Scope:** SAG-based external backlink campaign engine — page tier assignment, country-specific strategies, FatGrid/PRNews.io/EIN Presswire/Linking News integration, campaign generation from blueprints, anchor text planning, quality scoring, tipping point detection, KPI dashboard.
|
||
**Depends On:** 02D (internal linking established) + 02C (GSC data for KPI tracking)
|
||
|
||
### Source Material:
|
||
- temp/IGNY8-Project-Files/SAG-Doc4-External-Backlink-Campaign-PLAN.md (PRIMARY — comprehensive backlink spec)
|
||
- temp/IGNY8-Project-Files/IGNY8-Consolidated-Module-Plans.md (Backlinks section)
|
||
- Igny8 V2 New Final Plans/Linker Internal & External/IGNY8_Linker_FatGrid_PR_Analysis.md.pdf
|
||
|
||
### What to Cover:
|
||
|
||
**1. Current State:**
|
||
- No backlink management in IGNY8
|
||
- No external API integrations (FatGrid, PRNews.io, etc.)
|
||
- No campaign generation, tracking, or KPI monitoring
|
||
- Backlink building is entirely manual and external to IGNY8
|
||
|
||
**2. What to Build — SAG-Based Backlink Campaign Engine:**
|
||
|
||
**Hub-Only Strategy (core principle):**
|
||
- Backlinks target ONLY T1-T3 pages (homepage + cluster hubs + key service/product pages)
|
||
- Budget: 70-85% to T1-T3, 15-30% to T4-T5 (authority magnets), 0% to term/product pages
|
||
- SAG internal linking (02D) distributes authority from hubs to 70+ supporting pages
|
||
- Typically 20-30 target pages per site
|
||
|
||
**Page Tier Assignment:**
|
||
| Tier | Pages | Links/Page | Description |
|
||
|------|-------|------------|-------------|
|
||
| T1 | Homepage (1 page) | 10-30 | Brand authority anchor |
|
||
| T2 | Top 40% hubs by search volume | 5-15 | Primary money pages |
|
||
| T3 | Remaining hubs + products/services | 3-10 | Supporting money pages |
|
||
| T4 | Supporting blog articles | 1-4 | Content authority |
|
||
| T5 | Authority magnets (guides, tools) | 2-6 | Link bait pages |
|
||
|
||
**Country-Specific Strategies (4 profiles):**
|
||
|
||
**Pakistan:** 8-month timeline, $2-5K budget, target DR 25-30, exact match 5-10%, easiest competition
|
||
**Canada:** 12-month timeline, $3-7K budget, target DR 35-40, exact match 3-7%
|
||
**UK:** 14-month timeline, $3-9K budget, target DR 35-40, exact match 3-7%
|
||
**USA:** 18-month timeline, $5-13K budget, target DR 40-45, exact match 2-5%, hardest competition
|
||
|
||
Each profile includes: link_mix (budget/mid/premium/free percentages), anchor_text_mix per type, tier_allocation, monthly_velocity (ramp_up/peak/cruise/maintenance phases), quality_thresholds, geo_strategy, kpi_milestones.
|
||
|
||
**Anchor Text Mix (country-specific):**
|
||
| Type | PK | UK | CA | USA |
|
||
|------|-----|-----|-----|------|
|
||
| Branded | 30-35% | 35-40% | 35-40% | 35-45% |
|
||
| Naked URL | 15-20% | 15-20% | 15-20% | 15-20% |
|
||
| Generic | 15-20% | 15-20% | 15-20% | 15-20% |
|
||
| Partial Match | 15-20% | 12-18% | 12-18% | 10-15% |
|
||
| Exact Match | 5-10% | 3-7% | 3-7% | 2-5% |
|
||
| LSI/Topical | 5-10% | 5-10% | 5-10% | 5-8% |
|
||
| Brand+Keyword | — | 3-5% | 3-5% | 3-5% |
|
||
|
||
**Campaign Generation Algorithm:**
|
||
1. Load SAGBlueprint → identify all published hub pages
|
||
2. Assign tiers based on search volume data (from Keywords model)
|
||
3. Select country profile → calculate referring_domains_needed
|
||
4. links_per_tier = referring_domains_needed × tier_allocation_%
|
||
5. budget_estimate = links × cost_per_link × link_mix_%
|
||
6. Distribute across monthly velocity curve (ramp→peak→cruise→maintenance)
|
||
7. Assign pages to months by priority (KD, volume, commercial value)
|
||
8. Pre-generate 3 anchor text variants per page per type
|
||
9. Set quality requirements per country threshold
|
||
|
||
**Marketplace Integrations:**
|
||
|
||
**FatGrid API:**
|
||
- Base: `https://api.fatgrid.com/api/public`
|
||
- Auth: API key header
|
||
- Endpoints: Domain Lookup, Marketplace Browse (filterable by DR, traffic, price, niche), Bulk Domain Lookup (1K)
|
||
- 15+ marketplaces: Collaborator.pro, PRNews.io, Adsy.com, WhitePress.com, Bazoom.com, MeUp.com, etc.
|
||
- IGNY8 uses FatGrid to find and filter link placement opportunities
|
||
|
||
**PR Distribution (3 tiers):**
|
||
- PR Basic: EIN Presswire ($99-499/release) → AP News, Bloomberg, 115+ US TV
|
||
- PR Premium: PRNews.io ($500-5K/placement) → Yahoo Finance, Forbes-tier
|
||
- PR Enterprise: Linking News white-label ($500-2K/distribution) → ABC, NBC, FOX, Yahoo, Bloomberg
|
||
|
||
**Quality Scoring (per backlink opportunity):**
|
||
Auto-checkable factors (7 points): organic traffic >500/mo, DR/DA >threshold, indexed in Google, not on GP farm blocklist, traffic trend stable/growing
|
||
Manual review factors (4 points): outbound links <100, niche relevance, editorial quality, dofollow confirmed
|
||
Thresholds: PK ≥5, UK ≥6, CA ≥6, USA ≥7 (out of 11 points)
|
||
|
||
**Authority Tipping Point Detection:**
|
||
Monitor for 3+ simultaneous indicators:
|
||
- DR reached country target
|
||
- Pages with GSC impressions >100 but 0 SAGBacklinks start getting clicks
|
||
- Un-linked pages rank on page 2-3
|
||
- New content ranks passively without dedicated links
|
||
- Keywords in top 10 exceed threshold (PK:10+, UK/CA:15+, USA:20+)
|
||
When triggered → recommend: reduce velocity, shift budget to content
|
||
|
||
**Dead Link Monitoring:**
|
||
- Periodic HTTP checks on all placed backlinks
|
||
- Status tracking: live → dead/removed
|
||
- Impact scoring: how much authority was lost
|
||
- Auto-generate replacement recommendations
|
||
- Reserve 10-15% monthly budget for replacements
|
||
|
||
**3. Data Models & APIs:**
|
||
|
||
New models (in linker app):
|
||
- `SAGCampaign` (extends SiteSectorBaseModel)
|
||
- `blueprint` ForeignKey to SAGBlueprint (nullable)
|
||
- `country_code` CharField(max_length=3) — PK/UK/CA/USA
|
||
- `status` CharField (draft/active/paused/completed)
|
||
- `tier_assignments` JSONField — {content_id: tier_level}
|
||
- `total_links_target` IntegerField
|
||
- `budget_estimate_min` DecimalField
|
||
- `budget_estimate_max` DecimalField
|
||
- `timeline_months` IntegerField
|
||
- `monthly_plan` JSONField — [{month, links_target, pages[], budget}]
|
||
- `anchor_text_plan` JSONField — {content_id: [{text, type, allocated}]}
|
||
- `country_profile` JSONField — full profile snapshot at creation
|
||
- `kpi_data` JSONField — monthly snapshots
|
||
- `started_at` DateTimeField (nullable)
|
||
- Table: `igny8_sag_campaigns`
|
||
|
||
- `SAGBacklink` (extends SiteSectorBaseModel)
|
||
- `campaign` ForeignKey to SAGCampaign
|
||
- `blueprint` ForeignKey to SAGBlueprint (nullable)
|
||
- `target_content` ForeignKey to Content
|
||
- `target_url` URLField
|
||
- `target_tier` CharField (T1/T2/T3/T4/T5)
|
||
- `source_url` URLField (nullable — may not be known at planning)
|
||
- `source_domain` CharField (nullable)
|
||
- `source_dr` IntegerField (nullable)
|
||
- `source_traffic` IntegerField (nullable)
|
||
- `anchor_text` CharField
|
||
- `anchor_type` CharField (branded/naked_url/generic/partial_match/exact_match/lsi/brand_keyword)
|
||
- `link_type` CharField (guest_post/niche_edit/pr_distribution/directory/resource_page/broken_link/haro)
|
||
- `marketplace` CharField (nullable) — fatgrid, prnews, ein, linking_news, manual
|
||
- `cost` DecimalField (nullable)
|
||
- `quality_score` FloatField (nullable, 0-11)
|
||
- `country_relevant` BooleanField (default True)
|
||
- `date_ordered` DateField (nullable)
|
||
- `date_live` DateField (nullable)
|
||
- `date_last_checked` DateField (nullable)
|
||
- `status` CharField (planned/ordered/live/dead/replaced/rejected)
|
||
- `notes` TextField (nullable)
|
||
- Table: `igny8_sag_backlinks`
|
||
|
||
- `CampaignKPISnapshot` (extends SiteSectorBaseModel)
|
||
- `campaign` ForeignKey to SAGCampaign
|
||
- `snapshot_date` DateField
|
||
- `dr` FloatField (nullable)
|
||
- `da` FloatField (nullable)
|
||
- `referring_domains` IntegerField
|
||
- `new_links_this_month` IntegerField
|
||
- `links_by_tier` JSONField
|
||
- `cost_this_month` DecimalField
|
||
- `cost_per_link_avg` DecimalField
|
||
- `keywords_top_10` IntegerField
|
||
- `keywords_top_20` IntegerField
|
||
- `keywords_top_50` IntegerField
|
||
- `organic_traffic` IntegerField (nullable — from GSC)
|
||
- `impressions` IntegerField (nullable — from GSC)
|
||
- `pages_ranking_without_backlinks` IntegerField
|
||
- `tipping_point_indicators` JSONField — {indicator: True/False}
|
||
- `tipping_point_triggered` BooleanField
|
||
- Table: `igny8_campaign_kpi_snapshots`
|
||
|
||
New endpoints:
|
||
- `GET /api/v1/linker/campaigns/?site_id=X` — list campaigns
|
||
- `POST /api/v1/linker/campaigns/generate/` — AI generates campaign from blueprint + country
|
||
- `GET /api/v1/linker/campaigns/{id}/` — campaign detail with monthly plan
|
||
- `PUT /api/v1/linker/campaigns/{id}/` — update campaign (adjust plan)
|
||
- `POST /api/v1/linker/campaigns/{id}/activate/` — start campaign
|
||
- `GET /api/v1/linker/campaigns/{id}/kpi/` — KPI snapshots
|
||
- `POST /api/v1/linker/campaigns/{id}/kpi/snapshot/` — record monthly KPI
|
||
- `GET /api/v1/linker/backlinks/?campaign_id=X` — list backlinks with filters
|
||
- `POST /api/v1/linker/backlinks/` — add backlink record
|
||
- `PUT /api/v1/linker/backlinks/{id}/` — update status/details
|
||
- `POST /api/v1/linker/backlinks/check/` — trigger dead link check
|
||
- `GET /api/v1/linker/marketplace/search/` — FatGrid marketplace search (proxy)
|
||
- `GET /api/v1/linker/marketplace/domain/{domain}/` — FatGrid domain lookup (proxy)
|
||
- `GET /api/v1/linker/tipping-point/?campaign_id=X` — tipping point analysis
|
||
|
||
Celery tasks:
|
||
- `check_backlink_status` — weekly, HTTP check all live backlinks
|
||
- `record_kpi_snapshot` — monthly, pull DR/DA from Ahrefs API + GSC data
|
||
- `evaluate_tipping_point` — monthly, after KPI snapshot
|
||
- `generate_replacement_recommendations` — when dead links detected
|
||
|
||
**Credit Costs:** 2 credits campaign generation, 1 credit audit, 0.5 per KPI snapshot, 1 credit dead link detection, FatGrid API calls consume FatGrid credits (separate billing)
|
||
|
||
**Cross-references:** 02D (internal linking distributes authority from hubs), 02C (GSC data feeds KPIs), 01A (SAGBlueprint provides structure), 04A (managed services sell backlink packages)
|
||
|
||
---
|
||
|
||
## DOC 02F: optimizer.md
|
||
|
||
**File:** `V2-Execution-Docs/02F-optimizer.md`
|
||
**Scope:** Cluster-aligned content optimization — keyword coverage analysis, heading restructure, schema gap detection, before/after scoring, batch optimization.
|
||
**Depends On:** 02B (taxonomy term content provides term → cluster mapping context)
|
||
|
||
### Source Material:
|
||
- Live Docs on Server/igny8-app-docs/plans/Future_Moduels_Features/Optimizer_Module_Plan.md
|
||
- temp/IGNY8-Project-Files/IGNY8-Consolidated-Module-Plans.md (Optimizer section)
|
||
- Live Docs on Server/igny8-app-docs/plans/final-dev-guides-of-futute-implementation/DocB-Platform-Modules-Dev-Guide.md
|
||
|
||
### What to Cover:
|
||
|
||
**1. Current State:**
|
||
- `optimization` Django app exists in INSTALLED_APPS, is **inactive** (behind feature flag)
|
||
- `OptimizationTask` model exists but has minimal fields
|
||
- AI function `optimize_content` exists as one of the 7 registered functions — but only does basic rewriting
|
||
- No cluster-alignment, no keyword coverage analysis, no schema gap detection, no before/after scoring
|
||
- No integration with SAG data
|
||
|
||
**2. What to Build — Cluster-Aligned Optimization Engine:**
|
||
|
||
**Cluster Matching (auto-assign optimization context):**
|
||
- Analyze content title, headings, keyword density
|
||
- Match against cluster keyword sets using scoring: keyword overlap (40%), semantic similarity (40%), title match (20%)
|
||
- If content has no cluster assignment → suggest best match
|
||
- If confidence < 0.6 → flag for manual review
|
||
|
||
**Keyword Coverage Analysis:**
|
||
- For a given cluster, load all Keywords from that cluster
|
||
- Scan content_html for each keyword presence (exact + partial + semantic)
|
||
- Report: covered keywords, missing keywords, keyword density per term
|
||
- Target: cover 70%+ of cluster keywords in hub content, 40%+ in supporting
|
||
|
||
**Heading Restructure:**
|
||
- Analyze H1/H2/H3 hierarchy for SEO best practices
|
||
- Check: single H1, H2s contain target keywords, logical hierarchy, no skipped levels
|
||
- Suggest additions/modifications based on missing keyword themes
|
||
- AI rewrites headings maintaining meaning while incorporating keywords
|
||
|
||
**Content Rewrite (intent-aligned):**
|
||
- Classify content intent: informational / commercial / transactional
|
||
- Adjust structure based on intent:
|
||
- Informational: expand explanations, add examples, increase depth
|
||
- Commercial: add comparison tables, pros/cons, feature highlights
|
||
- Transactional: strengthen CTAs, add trust signals, streamline path
|
||
- Expand thin content (<500 words) to minimum viable length
|
||
- Compress bloated content (remove redundancy)
|
||
- Add missing sections identified by keyword coverage
|
||
|
||
**Schema Gap Detection:**
|
||
- Check existing `schema_markup` JSON-LD against content type expectations
|
||
- Expected schema by type: Article (post), Product (product), Service (service), FAQPage (if FAQ present), BreadcrumbList (all), Organization (company pages)
|
||
- Identify missing required fields per schema type
|
||
- Generate corrected/complete schema JSON-LD
|
||
|
||
**Before/After Scoring:**
|
||
- Content Quality Score (0-100): keyword coverage (30%), heading structure (20%), content depth (20%), readability (15%), schema completeness (15%)
|
||
- Track `score_before` and `score_after` for every optimization
|
||
- Dashboard shows optimization impact across all content
|
||
|
||
**Batch Optimization:**
|
||
- Select multiple content by: cluster, score threshold, content type, date range
|
||
- Queue as Celery tasks with priority ordering
|
||
- Progress tracking per batch
|
||
|
||
**3. Data Models & APIs:**
|
||
|
||
Modified model — `OptimizationTask` (optimization app, extend existing):
|
||
- Add fields:
|
||
- `content` ForeignKey to Content
|
||
- `primary_cluster` ForeignKey to Clusters (nullable)
|
||
- `secondary_clusters` JSONField (list of int IDs)
|
||
- `keyword_targets` JSONField — [{keyword, target_density, current_density, status}]
|
||
- `optimization_type` CharField (full_rewrite/heading_only/schema_only/keyword_coverage/batch)
|
||
- `intent_classification` CharField (informational/commercial/transactional)
|
||
- `score_before` FloatField (nullable)
|
||
- `score_after` FloatField (nullable)
|
||
- `content_before` TextField — snapshot of original HTML
|
||
- `content_after` TextField — optimized HTML (nullable)
|
||
- `metadata_before` JSONField — {meta_title, meta_description, headings[]}
|
||
- `metadata_after` JSONField (nullable)
|
||
- `schema_before` JSONField (nullable)
|
||
- `schema_after` JSONField (nullable)
|
||
- `structure_changes` JSONField — [{change_type, description, before, after}]
|
||
- `confidence_score` FloatField (nullable) — AI confidence in changes
|
||
- `applied` BooleanField (default False)
|
||
- `applied_at` DateTimeField (nullable)
|
||
- `status` CharField (pending/analyzing/optimizing/review/applied/rejected)
|
||
|
||
New endpoints:
|
||
- `POST /api/v1/optimizer/analyze/` — analyze single content piece (returns scores + recommendations)
|
||
- `POST /api/v1/optimizer/optimize/` — run full optimization (returns preview)
|
||
- `POST /api/v1/optimizer/preview/` — preview changes without applying
|
||
- `POST /api/v1/optimizer/apply/{id}/` — apply optimized version to Content record
|
||
- `POST /api/v1/optimizer/reject/{id}/` — reject optimization, keep original
|
||
- `POST /api/v1/optimizer/batch/` — queue batch optimization
|
||
- `GET /api/v1/optimizer/tasks/?site_id=X` — list optimization tasks with filters
|
||
- `GET /api/v1/optimizer/tasks/{id}/` — optimization detail with before/after
|
||
- `GET /api/v1/optimizer/tasks/{id}/diff/` — HTML diff view
|
||
- `GET /api/v1/optimizer/cluster-suggestions/?content_id=X` — suggest cluster for unassigned content
|
||
- `POST /api/v1/optimizer/assign-cluster/` — assign cluster to content
|
||
- `GET /api/v1/optimizer/dashboard/?site_id=X` — optimization stats (avg score improvement, count by status)
|
||
|
||
Celery tasks:
|
||
- `run_optimization` — processes single optimization task
|
||
- `run_batch_optimization` — processes batch with concurrency control
|
||
- `identify_optimization_candidates` — weekly scan: find content with score < threshold
|
||
|
||
**Credit Costs:** 2 credits analysis, 5-8 credits full rewrite, 1 credit schema-only, 15-25 credits batch (10 items)
|
||
|
||
**Cross-references:** 02B (taxonomy terms get cluster context for optimization), 02D (optimizer identifies internal link opportunities → feeds to linker), 02G (schema generation is a sub-feature), 02C (GSC position data identifies pages that need optimization), 01E (blueprint-aware pipeline sets initial quality bar)
|
||
|
||
---
|
||
|
||
## DOC 02G: rich-schema-serp.md
|
||
|
||
**File:** `V2-Execution-Docs/02G-rich-schema-serp.md`
|
||
**Scope:** Rich schema JSON-LD generation for 10 types, on-page SERP elements (TL;DR, TOC, tables, definitions, PAA), retroactive enhancement engine for existing content.
|
||
**Depends On:** 02A (content types determine which schema to apply)
|
||
|
||
### Source Material:
|
||
- temp/IGNY8-Project-Files/IGNY8-Rich-Schema-SERP-Enhancement-Module.docx
|
||
- temp/IGNY8-Project-Files/IGNY8-Consolidated-Module-Plans.md (Schema section)
|
||
- Live Docs on Server/igny8-app-docs/plans/final-dev-guides-of-futute-implementation/DocB-Platform-Modules-Dev-Guide.md
|
||
|
||
### What to Cover:
|
||
|
||
**1. Current State:**
|
||
- `Content.schema_markup` JSONField exists — used to store schema during generation
|
||
- Current AI `generate_content` sometimes includes basic Article schema
|
||
- No systematic schema generation by type
|
||
- No on-page SERP element injection
|
||
- No schema validation
|
||
- No retroactive enhancement of existing content
|
||
|
||
**2. What to Build:**
|
||
|
||
**10 JSON-LD Schema Types:**
|
||
|
||
1. **Article / BlogPosting** — for post content_type
|
||
- Fields: headline, datePublished, dateModified, author (Person/Organization), publisher, image, description, mainEntityOfPage, wordCount, articleSection
|
||
|
||
2. **Product** — for product content_type
|
||
- Fields: name, description, image, brand, offers (price, priceCurrency, availability, url), aggregateRating, review, sku, gtin
|
||
|
||
3. **Service** — for service content_type
|
||
- Fields: name, description, provider (Organization), serviceType, areaServed, hasOfferCatalog, offers
|
||
|
||
4. **LocalBusiness** — for sites with physical location
|
||
- Fields: name, address, telephone, openingHours, geo, image, priceRange, sameAs, hasMap
|
||
|
||
5. **Organization** — site-wide
|
||
- Fields: name, url, logo, sameAs[], contactPoint, foundingDate, founders
|
||
|
||
6. **BreadcrumbList** — all pages
|
||
- Auto-generated from SAG hierarchy or WordPress breadcrumb trail
|
||
- Fields: itemListElement [{position, name, item(URL)}]
|
||
|
||
7. **FAQPage** — content with FAQ sections
|
||
- Auto-detected from H2/H3 question patterns or explicit FAQ blocks
|
||
- Fields: mainEntity [{@type: Question, name, acceptedAnswer: {text}}]
|
||
|
||
8. **HowTo** — step-by-step content
|
||
- Detected from ordered lists with process indicators
|
||
- Fields: name, step [{@type: HowToStep, name, text, image, url}], totalTime, estimatedCost
|
||
|
||
9. **VideoObject** — content with video embeds (Phase 2I integration)
|
||
- Fields: name, description, thumbnailUrl, uploadDate, duration, contentUrl, embedUrl
|
||
|
||
10. **WebSite + SearchAction** — site-wide (homepage)
|
||
- Fields: name, url, potentialAction (SearchAction with query-input)
|
||
|
||
**On-Page SERP Elements (injected into content_html):**
|
||
|
||
1. **TL;DR Box** — 2-3 sentence summary at top of article, in styled box
|
||
2. **Table of Contents** — auto-generated from H2/H3 headings, with anchor links
|
||
3. **Key Takeaways** — bullet list of main points, in styled box
|
||
4. **Definition Boxes** — detect defined terms, create highlight boxes
|
||
5. **Comparison Tables** — for comparison content, structured HTML tables
|
||
6. **People Also Ask** — AI-generated related questions with answers
|
||
7. **Statistics Callouts** — detect numbers/stats in text, create visual callouts
|
||
8. **Pro/Con Boxes** — for review/comparison content
|
||
|
||
**Retroactive Enhancement Engine:**
|
||
- Scan all published content
|
||
- For each: determine applicable schema types + missing SERP elements
|
||
- Generate schema JSON-LD + inject SERP elements
|
||
- Preview mode: show changes before applying
|
||
- Batch processing with priority (highest-traffic pages first)
|
||
|
||
**Schema Validation:**
|
||
- Validate against Google Rich Results requirements (not just schema.org)
|
||
- Check required fields per type
|
||
- Test with Google Rich Results Test API (where available)
|
||
- Warning system for incomplete/invalid schema
|
||
- Auto-fix common issues (missing required fields with sensible defaults)
|
||
|
||
**3. Data Models & APIs:**
|
||
|
||
New models (in writer app or new schema app):
|
||
- `SchemaTemplate` (extends AccountBaseModel)
|
||
- `schema_type` CharField (article/product/service/localbusiness/organization/breadcrumb/faq/howto/video/website)
|
||
- `content_type_match` CharField — which content_type this applies to
|
||
- `content_structure_match` CharField (nullable) — further filter
|
||
- `template_json` JSONField — template with placeholder fields
|
||
- `required_fields` JSONField — list of required field paths
|
||
- `is_default` BooleanField
|
||
- Table: `igny8_schema_templates`
|
||
|
||
- `SERPEnhancement` (extends SiteSectorBaseModel)
|
||
- `content` ForeignKey to Content
|
||
- `enhancement_type` CharField (tldr/toc/key_takeaways/definition/comparison_table/paa/stats_callout/pro_con)
|
||
- `html_snippet` TextField — generated HTML to inject
|
||
- `insertion_point` CharField — where in content (top/after_intro/before_h2_N/bottom)
|
||
- `status` CharField (generated/inserted/removed)
|
||
- `generated_at` DateTimeField
|
||
- Table: `igny8_serp_enhancements`
|
||
|
||
- `SchemaValidationResult` (extends SiteSectorBaseModel)
|
||
- `content` ForeignKey to Content
|
||
- `schema_type` CharField
|
||
- `is_valid` BooleanField
|
||
- `errors` JSONField
|
||
- `warnings` JSONField
|
||
- `validated_at` DateTimeField
|
||
- Table: `igny8_schema_validation_results`
|
||
|
||
Modified models:
|
||
- `Content.schema_markup` — already exists, now systematically populated
|
||
- `Content.serp_elements` — new JSONField tracking which enhancements are active
|
||
|
||
New endpoints:
|
||
- `POST /api/v1/writer/schema/generate/` — generate schema for single content
|
||
- `POST /api/v1/writer/schema/validate/` — validate existing schema
|
||
- `POST /api/v1/writer/schema/batch-generate/` — batch schema generation
|
||
- `GET /api/v1/writer/schema/templates/` — list schema templates
|
||
- `POST /api/v1/writer/serp/enhance/` — generate SERP elements for content
|
||
- `POST /api/v1/writer/serp/batch-enhance/` — batch enhancement
|
||
- `GET /api/v1/writer/serp/preview/{content_id}/` — preview enhancements
|
||
- `POST /api/v1/writer/serp/apply/{content_id}/` — apply enhancements to content HTML
|
||
- `GET /api/v1/writer/schema/audit/?site_id=X` — schema coverage audit
|
||
- `POST /api/v1/writer/schema/retroactive/` — trigger retroactive enhancement scan
|
||
|
||
Celery tasks:
|
||
- `generate_schema_for_content` — after content generation, auto-generate schema
|
||
- `retroactive_schema_scan` — scan existing content, generate missing schemas
|
||
- `validate_schemas_batch` — periodic validation of all schemas
|
||
|
||
**Credit Costs:** 1 credit schema generation, 0.5 per SERP element, 0.1 schema validation, 8-12 credits batch (10 items)
|
||
|
||
**Cross-references:** 02A (content type determines schema type), 02F (optimizer detects schema gaps), 03A (WP plugin standalone has schema module), 03B (connected mode pushes schema to WP via bulk endpoint)
|
||
|
||
---
|
||
|
||
## DOC 02H: socializer.md
|
||
|
||
**File:** `V2-Execution-Docs/02H-socializer.md`
|
||
**Scope:** Native Django social media posting — 5 platforms (LinkedIn, Twitter/X, Facebook, Instagram, TikTok), OAuth connections, AI content adaptation, scheduling/calendar, engagement tracking, Stage 8 pipeline integration, credit system.
|
||
**Depends On:** 01E (blueprint-aware pipeline — Stage 8 hooks in after publish)
|
||
|
||
### Source Material:
|
||
- Live Docs on Server/igny8-app-docs/plans/SOCIALIZER-AND-VIDEO-CREATOR-MODULES.md
|
||
- Live Docs on Server/igny8-app-docs/plans/Future_Moduels_Features/Socializer_Video_Modules_Plan.md
|
||
- Live Docs on Server/igny8-app-docs/plans/Future_Moduels_Features/FB_LinkedIn_Auth_flow_Steps.md
|
||
- temp/IGNY8-Project-Files/IGNY8-Consolidated-Module-Plans.md (Socializer section)
|
||
|
||
### What to Cover:
|
||
|
||
**1. Current State:**
|
||
- No social media functionality in IGNY8
|
||
- No OAuth connections to any social platform
|
||
- No social post generation, scheduling, or analytics
|
||
- The automation pipeline ends at Stage 7 (publish to WordPress)
|
||
|
||
**2. What to Build — Social Media Engine (Stage 8):**
|
||
|
||
**Platform Support:**
|
||
|
||
| Platform | Max Length | Key Format | OAuth | API |
|
||
|----------|-----------|------------|-------|-----|
|
||
| LinkedIn | 1,300 chars | Professional, industry insights | OAuth 2.0 (3-legged) | Marketing API v2 |
|
||
| Twitter/X | 280 chars (thread option) | Concise, hashtags, hooks | OAuth 2.0 (PKCE) | API v2 |
|
||
| Facebook | ~63,206 chars (500 optimal) | Conversational, visual | OAuth 2.0 (Facebook Login) | Graph API v18+ |
|
||
| Instagram | 2,200 chars, 30 hashtags | Visual-first, hashtag-heavy | Via Facebook Business | Graph API (IG) |
|
||
| TikTok | varies | Gen-Z tone, trending hooks | OAuth 2.0 | Content Posting API |
|
||
|
||
**OAuth Infrastructure:**
|
||
- Reusable OAuth service: `igny8_core/integration/oauth/`
|
||
- Platform-specific handlers: `linkedin_oauth.py`, `twitter_oauth.py`, `facebook_oauth.py`, `tiktok_oauth.py`
|
||
- Token encryption (same pattern as GSC tokens in 02C)
|
||
- Auto-refresh via Celery
|
||
- Multi-account support per platform per site
|
||
|
||
**Facebook + LinkedIn Auth Flow (detailed):**
|
||
- Facebook: App Review required for `pages_manage_posts`, `pages_read_engagement` permissions
|
||
- Flow: Login → Page Selection → Permission Grant → Store page_access_token
|
||
- Webhook: Subscribe to page feed for engagement updates
|
||
- LinkedIn: Partner Program for Marketing API access
|
||
- Flow: Authorization → Organization Selection → Store access_token
|
||
- Scopes: `w_member_social`, `r_organization_social`, `w_organization_social`
|
||
|
||
**AI Content Adaptation:**
|
||
- Input: Content record (title, content_html, meta_description, cluster keywords)
|
||
- Per platform, AI generates:
|
||
- Adapted text matching platform tone + length limits
|
||
- Hashtag set (topic-aware, trending check)
|
||
- CTA appropriate to platform
|
||
- Image sizing recommendations
|
||
- Twitter thread generation: split long content into coherent thread (2-10 tweets)
|
||
|
||
**Post Types:**
|
||
1. **Announcement** — link post with title + excerpt + URL
|
||
2. **Highlights** — key takeaways from article + CTA
|
||
3. **Quote Card** — branded insight/statistic as image
|
||
4. **FAQ Snippet** — single FAQ from content as post
|
||
5. **Carousel** (future) — multi-image posts
|
||
|
||
**Image Sizing per Platform:**
|
||
- LinkedIn: 1200×627px
|
||
- Twitter: 1600×900px
|
||
- Facebook: 1200×630px
|
||
- Instagram: 1080×1080px (square), 1080×1350px (portrait)
|
||
- TikTok: 1080×1920px
|
||
|
||
**Scheduling & Calendar:**
|
||
- Best-time slots per platform (configurable defaults + per-account)
|
||
- Queue system: posts scheduled → Celery processes at scheduled time
|
||
- Frequency caps: max posts per day per platform (default: 2)
|
||
- Cooldown: 4-6 hours between posts to same platform
|
||
- Calendar view: visual week/month layout with drag-drop rescheduling
|
||
- Bulk scheduling: schedule social posts for multiple content at once
|
||
|
||
**Engagement Tracking:**
|
||
- Celery task fetches engagement every 6 hours per post
|
||
- Metrics: likes, comments, shares/retweets, impressions, engagement_rate
|
||
- Aggregate dashboard: per-platform performance, best performing posts, optimal times
|
||
- Attribution: link clicks tracked via UTM parameters appended to URLs
|
||
|
||
**Stage 8 Pipeline Integration:**
|
||
- After Stage 7 (publish to WP) → auto-trigger Stage 8 if social accounts connected
|
||
- Generate platform-adapted posts for each connected account
|
||
- Auto-schedule based on best-time algorithm
|
||
- If manual review enabled → posts go to "pending_review" status
|
||
- If auto-publish enabled → posts queue immediately
|
||
|
||
**3. Data Models & APIs:**
|
||
|
||
New models (in new `social` app or extend `integration`):
|
||
- `SocialAccount` (extends SiteSectorBaseModel)
|
||
- `platform` CharField (linkedin/twitter/facebook/instagram/tiktok)
|
||
- `platform_account_id` CharField — external ID
|
||
- `account_name` CharField — display name
|
||
- `account_type` CharField (personal/page/organization)
|
||
- `access_token` TextField (encrypted)
|
||
- `refresh_token` TextField (encrypted, nullable)
|
||
- `token_expiry` DateTimeField (nullable)
|
||
- `permissions` JSONField — granted scopes
|
||
- `status` CharField (active/expired/revoked/error)
|
||
- `settings` JSONField — {auto_publish, max_per_day, best_times[], cooldown_hours}
|
||
- Table: `igny8_social_accounts`
|
||
|
||
- `SocialPost` (extends SiteSectorBaseModel)
|
||
- `content` ForeignKey to Content (nullable — can be standalone post)
|
||
- `social_account` ForeignKey to SocialAccount
|
||
- `platform` CharField (denormalized for filtering)
|
||
- `post_type` CharField (announcement/highlights/quote_card/faq_snippet/carousel)
|
||
- `text` TextField
|
||
- `hashtags` JSONField (list of strings)
|
||
- `media_urls` JSONField (list of image/video URLs)
|
||
- `link_url` URLField (nullable)
|
||
- `utm_params` JSONField (nullable)
|
||
- `thread_posts` JSONField (nullable — for Twitter threads: [{text, media_url}])
|
||
- `scheduled_at` DateTimeField (nullable)
|
||
- `published_at` DateTimeField (nullable)
|
||
- `platform_post_id` CharField (nullable — external post ID after publishing)
|
||
- `platform_post_url` URLField (nullable)
|
||
- `status` CharField (draft/scheduled/publishing/published/failed/cancelled)
|
||
- `error_message` TextField (nullable)
|
||
- Table: `igny8_social_posts`
|
||
|
||
- `SocialEngagement` (extends models.Model)
|
||
- `social_post` ForeignKey to SocialPost
|
||
- `likes` IntegerField (default 0)
|
||
- `comments` IntegerField (default 0)
|
||
- `shares` IntegerField (default 0)
|
||
- `impressions` IntegerField (default 0)
|
||
- `clicks` IntegerField (default 0)
|
||
- `engagement_rate` FloatField (default 0.0)
|
||
- `raw_data` JSONField — full platform response
|
||
- `fetched_at` DateTimeField
|
||
- Table: `igny8_social_engagement`
|
||
|
||
New endpoints:
|
||
- `POST /api/v1/social/accounts/connect/{platform}/` — initiate OAuth
|
||
- `GET /api/v1/social/accounts/callback/{platform}/` — OAuth callback
|
||
- `GET /api/v1/social/accounts/?site_id=X` — list connected accounts
|
||
- `DELETE /api/v1/social/accounts/{id}/` — disconnect account
|
||
- `PUT /api/v1/social/accounts/{id}/settings/` — update account settings
|
||
- `POST /api/v1/social/posts/generate/` — AI generate posts for content (all connected platforms)
|
||
- `POST /api/v1/social/posts/` — create manual post
|
||
- `GET /api/v1/social/posts/?site_id=X` — list posts with filters
|
||
- `PUT /api/v1/social/posts/{id}/` — edit draft/scheduled post
|
||
- `POST /api/v1/social/posts/{id}/publish/` — publish immediately
|
||
- `POST /api/v1/social/posts/{id}/schedule/` — schedule for later
|
||
- `DELETE /api/v1/social/posts/{id}/` — cancel/delete
|
||
- `POST /api/v1/social/posts/bulk-generate/` — generate for multiple content
|
||
- `POST /api/v1/social/posts/bulk-schedule/` — schedule multiple posts
|
||
- `GET /api/v1/social/calendar/?site_id=X&month=YYYY-MM` — calendar view
|
||
- `GET /api/v1/social/analytics/?site_id=X` — aggregate analytics
|
||
- `GET /api/v1/social/analytics/posts/?site_id=X` — per-post analytics
|
||
|
||
Celery tasks:
|
||
- `publish_scheduled_posts` — runs every minute, publishes due posts
|
||
- `refresh_social_tokens` — runs hourly, refreshes expiring tokens
|
||
- `fetch_engagement` — runs every 6 hours, fetches metrics for recent posts
|
||
- `generate_social_posts_stage8` — triggered by pipeline after Stage 7
|
||
|
||
**Credit Costs:** 1 credit per platform adaptation, 0.5 hashtag generation, 2 credits thread (Twitter), 5 credits carousel, 3-10 credits image resizing/generation, 15-25 credits full suite (5 platforms)
|
||
|
||
**Cross-references:** 01E (pipeline Stage 8 integration), 02I (video creator shares social posting for video content), 03A (WP plugin standalone has share buttons — different from this; this is posting FROM IGNY8), 04A (managed services include social media management)
|
||
|
||
---
|
||
|
||
## DOC 02I: video-creator.md
|
||
|
||
**File:** `V2-Execution-Docs/02I-video-creator.md`
|
||
**Scope:** AI video creation pipeline — script generation from articles, TTS voiceover (cloud + self-hosted), FFmpeg composition, auto-subtitles, publishing to YouTube/Instagram/TikTok, Stage 9 pipeline integration, GPU requirements.
|
||
**Depends On:** 02H (socializer provides OAuth infrastructure + publishing APIs for video platforms) + 0F (self-hosted AI infra for TTS + image generation)
|
||
|
||
### Source Material:
|
||
- Live Docs on Server/igny8-app-docs/plans/SOCIALIZER-AND-VIDEO-CREATOR-MODULES.md
|
||
- Live Docs on Server/igny8-app-docs/plans/Future_Moduels_Features/Socializer_Video_Modules_Plan.md
|
||
- temp/IGNY8-Project-Files/IGNY8-Consolidated-Module-Plans.md (Video section)
|
||
|
||
### What to Cover:
|
||
|
||
**1. Current State:**
|
||
- No video creation capability in IGNY8
|
||
- No TTS, no FFmpeg pipeline, no video publishing
|
||
- Images exist (generated by Stages 5-6) — these feed into video as visual assets
|
||
- Self-hosted AI infra (Phase 0F) provides GPU for TTS and image generation
|
||
- Socializer (02H) provides OAuth connections to YouTube/Instagram/TikTok
|
||
|
||
**2. What to Build — AI Video Creation Pipeline (Stage 9):**
|
||
|
||
**Video Types:**
|
||
| Type | Duration | Aspect | Primary Platform |
|
||
|------|----------|--------|-----------------|
|
||
| Short | 30-90s | 9:16 vertical | YouTube Shorts, Reels, TikTok |
|
||
| Medium | 60-180s | 9:16 or 16:9 | TikTok, YouTube |
|
||
| Long | 5-15m | 16:9 horizontal | YouTube |
|
||
|
||
**Platform Specs:**
|
||
- YouTube Long: MP4 H.264, AAC audio, up to 12h, 1920×1080, max 256GB
|
||
- YouTube Shorts: ≤60s, 1080×1920, same encoding
|
||
- Instagram Reels: ≤90s, 1080×1920, max 650MB
|
||
- TikTok: ≤10m, 1080×1920, max 72MB
|
||
|
||
**5-Stage Video Pipeline:**
|
||
|
||
**Stage 1 — Script Generation (AI):**
|
||
- Input: Content record (title, content_html, meta_description, keywords, images)
|
||
- AI extracts: key points, compelling hook (from meta_description), intro, body points, CTA
|
||
- Output structure:
|
||
```json
|
||
{
|
||
"hook": "text (3-5 sec)",
|
||
"intro": "text (10-15 sec)",
|
||
"points": [{"text": "...", "duration_est": 20, "visual_cue": "show chart", "text_overlay": "Key stat"}],
|
||
"cta": "text (5-10 sec)",
|
||
"chapter_markers": [{"time": 0, "title": "Intro"}, ...],
|
||
"total_estimated_duration": 120
|
||
}
|
||
```
|
||
- SEO: AI generates platform-specific title, description, tags for each target platform
|
||
|
||
**Stage 2 — Voiceover (TTS):**
|
||
- Cloud providers:
|
||
- OpenAI TTS: $15-30 per 1M characters, voices: alloy/echo/fable/onyx/nova/shimmer
|
||
- ElevenLabs: plan-based pricing, higher quality, voice cloning
|
||
- Self-hosted (via 0F GPU):
|
||
- Coqui XTTS-v2: good quality, free, multi-language
|
||
- Bark: expressive speech, slower
|
||
- Piper TTS: fast, lightweight
|
||
- Features: voice selection, speed control, multi-language support
|
||
- Output: WAV/MP3 audio file + word-level timestamps (for subtitle sync)
|
||
|
||
**Stage 3 — Visual Assets:**
|
||
- Sources:
|
||
- Article images from `Images` model (already generated in pipeline)
|
||
- AI-generated scenes (Runware/DALL-E/Stable Diffusion via 0F)
|
||
- Stock footage APIs: Pexels, Pixabay (free, API key)
|
||
- Text overlay frames (rendered via Pillow)
|
||
- Code snippet frames (via Pygments syntax highlighting)
|
||
- Ken Burns effect on still images (zoom/pan)
|
||
- Transition effects between scenes
|
||
|
||
**Stage 4 — Video Composition (FFmpeg + MoviePy):**
|
||
- Libraries: FFmpeg (encoding), MoviePy (high-level), Pillow (overlays), pydub (audio)
|
||
- Process:
|
||
1. Create visual timeline from script sections
|
||
2. Assign visuals to each section (image/video clip per point)
|
||
3. Add text overlays at specified timestamps
|
||
4. Mix voiceover audio with background music (royalty-free, 20% volume)
|
||
5. Apply transitions between sections
|
||
6. Render to target resolution/format
|
||
- Presets:
|
||
- `youtube_long`: 1920×1080, 3-15m, H.264/AAC
|
||
- `youtube_short`: 1080×1920, 30-60s, H.264/AAC
|
||
- `instagram_reel`: 1080×1920, 30-90s, H.264/AAC
|
||
- `tiktok`: 1080×1920, 30-180s, H.264/AAC
|
||
|
||
**Stage 5 — SEO & Publishing:**
|
||
- Auto-generate: SRT subtitle file from TTS timestamps
|
||
- AI thumbnails: hero image with title text overlay
|
||
- Platform-specific metadata: title (optimized per platform), description (with timestamps for YouTube), tags, category
|
||
- Publishing via platform APIs (reuse OAuth from 02H)
|
||
- Confirmation logging
|
||
|
||
**User Flow:**
|
||
1. Select content → choose video type (short/medium/long) → target platforms
|
||
2. AI generates script → user reviews/edits script
|
||
3. Select voice → preview audio → approve
|
||
4. Auto-assign visuals → user can swap images → preview composition
|
||
5. Render video → preview final → approve
|
||
6. Publish to selected platforms → track performance
|
||
|
||
**Separate Celery Queue:**
|
||
- Video rendering is CPU/GPU intensive
|
||
- Dedicated `video` queue: `celery -A igny8_core worker -Q video --concurrency=1`
|
||
- Long-running tasks: 5-30 minutes per video render
|
||
- Progress tracking via Celery result backend
|
||
|
||
**3. Data Models & APIs:**
|
||
|
||
New models (in new `video` app):
|
||
- `VideoProject` (extends SiteSectorBaseModel)
|
||
- `content` ForeignKey to Content (nullable — can be standalone)
|
||
- `project_type` CharField (short/medium/long)
|
||
- `target_platforms` JSONField (list of platform strings)
|
||
- `status` CharField (draft/scripting/voiceover/composing/rendering/review/published/failed)
|
||
- `settings` JSONField — {voice_id, voice_provider, music_track, transition_style}
|
||
- Table: `igny8_video_projects`
|
||
|
||
- `VideoScript` (extends models.Model)
|
||
- `project` OneToOneField to VideoProject
|
||
- `script_text` TextField — full narration text
|
||
- `sections` JSONField — [{text, duration_est, visual_cue, text_overlay}]
|
||
- `hook` TextField
|
||
- `cta` TextField
|
||
- `chapter_markers` JSONField
|
||
- `total_estimated_duration` IntegerField (seconds)
|
||
- `seo_metadata` JSONField — {platform: {title, description, tags}}
|
||
- `version` IntegerField (default 1)
|
||
- Table: `igny8_video_scripts`
|
||
|
||
- `VideoAsset` (extends models.Model)
|
||
- `project` ForeignKey to VideoProject
|
||
- `asset_type` CharField (image/footage/music/overlay/subtitle)
|
||
- `source` CharField (article_image/ai_generated/stock_pexels/stock_pixabay/uploaded/rendered)
|
||
- `file_path` CharField — path in media storage
|
||
- `file_url` URLField (nullable)
|
||
- `duration` FloatField (nullable — for video/audio assets)
|
||
- `section_index` IntegerField (nullable — which script section)
|
||
- `order` IntegerField (default 0)
|
||
- Table: `igny8_video_assets`
|
||
|
||
- `RenderedVideo` (extends models.Model)
|
||
- `project` ForeignKey to VideoProject
|
||
- `preset` CharField (youtube_long/youtube_short/instagram_reel/tiktok)
|
||
- `resolution` CharField (1920x1080/1080x1920)
|
||
- `duration` FloatField (seconds)
|
||
- `file_size` BigIntegerField (bytes)
|
||
- `file_path` CharField
|
||
- `file_url` URLField (nullable)
|
||
- `subtitle_file_path` CharField (nullable — SRT)
|
||
- `thumbnail_path` CharField (nullable)
|
||
- `render_started_at` DateTimeField
|
||
- `render_completed_at` DateTimeField (nullable)
|
||
- `status` CharField (queued/rendering/completed/failed)
|
||
- Table: `igny8_rendered_videos`
|
||
|
||
- `PublishedVideo` (extends models.Model)
|
||
- `rendered_video` ForeignKey to RenderedVideo
|
||
- `social_account` ForeignKey to SocialAccount (from 02H)
|
||
- `platform` CharField
|
||
- `platform_video_id` CharField (nullable)
|
||
- `published_url` URLField (nullable)
|
||
- `title` CharField
|
||
- `description` TextField
|
||
- `tags` JSONField
|
||
- `thumbnail_url` URLField (nullable)
|
||
- `published_at` DateTimeField (nullable)
|
||
- `status` CharField (publishing/published/failed/removed)
|
||
- Table: `igny8_published_videos`
|
||
|
||
- `VideoEngagement` (extends models.Model)
|
||
- `published_video` ForeignKey to PublishedVideo
|
||
- `views` IntegerField (default 0)
|
||
- `likes` IntegerField (default 0)
|
||
- `comments` IntegerField (default 0)
|
||
- `shares` IntegerField (default 0)
|
||
- `watch_time_seconds` IntegerField (default 0)
|
||
- `avg_view_duration` FloatField (default 0.0)
|
||
- `raw_data` JSONField
|
||
- `fetched_at` DateTimeField
|
||
- Table: `igny8_video_engagement`
|
||
|
||
New endpoints:
|
||
- `POST /api/v1/video/projects/` — create video project
|
||
- `GET /api/v1/video/projects/?site_id=X` — list projects
|
||
- `GET /api/v1/video/projects/{id}/` — project detail
|
||
- `POST /api/v1/video/scripts/generate/` — AI generate script from content
|
||
- `PUT /api/v1/video/scripts/{project_id}/` — edit script
|
||
- `POST /api/v1/video/voiceover/generate/` — generate TTS audio
|
||
- `POST /api/v1/video/voiceover/preview/` — preview voice sample (short clip)
|
||
- `GET /api/v1/video/assets/{project_id}/` — list project assets
|
||
- `POST /api/v1/video/assets/{project_id}/` — add/replace asset
|
||
- `POST /api/v1/video/render/` — queue video render
|
||
- `GET /api/v1/video/render/{id}/status/` — render progress
|
||
- `GET /api/v1/video/rendered/{project_id}/` — list rendered videos
|
||
- `POST /api/v1/video/publish/` — publish to platform
|
||
- `GET /api/v1/video/analytics/?site_id=X` — aggregate video analytics
|
||
- `GET /api/v1/video/analytics/{published_video_id}/` — single video analytics
|
||
|
||
Celery tasks (on `video` queue):
|
||
- `generate_video_script` — AI script generation
|
||
- `generate_voiceover` — TTS processing
|
||
- `render_video` — FFmpeg/MoviePy composition (long-running, 5-30min)
|
||
- `generate_thumbnail` — AI thumbnail creation
|
||
- `generate_subtitles` — SRT from TTS timestamps
|
||
- `publish_video` — upload to platform API
|
||
- `fetch_video_engagement` — periodic metric fetch
|
||
- `video_pipeline_stage9` — full pipeline: script → voice → render → publish
|
||
|
||
**System Requirements:**
|
||
- FFmpeg installed on server (add to Docker image)
|
||
- MoviePy, Pillow, pydub Python packages
|
||
- Sufficient disk space for video temp files (cleanup after publish)
|
||
- Self-hosted GPU (from 0F) for TTS + AI image generation
|
||
- Cloud fallback if GPU unavailable
|
||
|
||
**Credit Costs:**
|
||
- Script generation: 5 credits
|
||
- TTS: 10/min standard (cloud), 2/min (self-hosted)
|
||
- TTS HD (ElevenLabs): 20/min
|
||
- Visual asset generation: 15-50 credits (depends on count)
|
||
- Thumbnail: 3-10 credits
|
||
- Composition/render: 5 credits
|
||
- SEO metadata per platform: 1 credit
|
||
- **Short-form total: 40-80 credits**
|
||
- **Long-form total: 100-250 credits**
|
||
|
||
**Cross-references:** 02H (socializer provides SocialAccount model + OAuth for publishing), 0F (self-hosted AI provides GPU for TTS + image gen), 01E (pipeline Stage 9 integration), 02G (VideoObject schema generated for published videos)
|
||
|
||
---
|
||
|
||
## BUILD SEQUENCE
|
||
|
||
Claude Code builds these docs in the following order:
|
||
|
||
```
|
||
Phase 2 Doc Build Order
|
||
═══════════════════════
|
||
|
||
1. 02A — Content Types Extension
|
||
(Foundation: extends Content model for all other modules)
|
||
|
||
2. 02B — Taxonomy Term Content
|
||
(Builds on 02A content type system)
|
||
|
||
3. 02C — GSC Integration
|
||
(Independent, can parallel with 02A/02B in execution)
|
||
|
||
4. 02G — Rich Schema SERP
|
||
(Depends on 02A content types for schema mapping)
|
||
|
||
5. 02F — Optimizer
|
||
(Uses 02B cluster mapping + 02G schema generation)
|
||
|
||
6. 02D — Linker Internal
|
||
(Needs SAG data + content types established)
|
||
|
||
7. 02E — Linker External/Backlinks
|
||
(Extends 02D + uses 02C GSC data)
|
||
|
||
8. 02H — Socializer
|
||
(Independent OAuth infra, Stage 8)
|
||
|
||
9. 02I — Video Creator
|
||
(Uses 02H OAuth + needs GPU from 0F, Stage 9)
|
||
```
|
||
|
||
For each doc:
|
||
1. Read this plan's section for that doc
|
||
2. Read the source files listed in "Source Material"
|
||
3. Read the relevant existing V2 docs referenced in "Cross-references"
|
||
4. Write the doc following the DOC STRUCTURE STANDARD
|
||
5. Verify all model names are PLURAL (Clusters, Keywords, Tasks, etc.)
|
||
6. Verify all PKs are BigAutoField (integer)
|
||
7. Verify all frontend references use .tsx + Zustand
|
||
8. Verify all API paths follow `/api/v1/{app_name}/` pattern
|
||
9. Save to `V2-Execution-Docs/02X-filename.md`
|
||
|
||
---
|
||
|
||
## SOURCE FILES TO READ
|
||
|
||
For each doc, read the source files in the "Source Material" sections above. All files are under:
|
||
`/data/app/igny8/` (on server) or `EcoSystem/Projects/APPS/Igny8/` (local docs)
|
||
|
||
Key source files across all Phase 2 docs:
|
||
1. `temp/IGNY8-Project-Files/IGNY8-Consolidated-Module-Plans.md` — ALL modules overview
|
||
2. `temp/IGNY8-Project-Files/SAG-Doc3-Interlinking-Specification-PLAN.md` — 02D primary
|
||
3. `temp/IGNY8-Project-Files/SAG-Doc4-External-Backlink-Campaign-PLAN.md` — 02E primary
|
||
4. `temp/IGNY8-Project-Files/SAG-IGNY8-Implementation.md` — SAG architecture context
|
||
5. `temp/IGNY8-Project-Files/IGNY8-Rich-Schema-SERP-Enhancement-Module.docx` — 02G primary
|
||
6. `Live Docs on Server/igny8-app-docs/plans/Future_Moduels_Features/` — all module plans
|
||
7. `Live Docs on Server/igny8-app-docs/plans/SOCIALIZER-AND-VIDEO-CREATOR-MODULES.md` — 02H/02I
|
||
8. `Live Docs on Server/igny8-app-docs/plans/final-dev-guides-of-futute-implementation/DocB-Platform-Modules-Dev-Guide.md` — all modules dev guide
|
||
|
||
Always cross-reference with existing V2 docs:
|
||
- `00-MASTER-EXECUTION-PLAN.md` — baseline rules, model inventory
|
||
- `01A-sag-data-foundation.md` — SAGBlueprint, SAGAttribute, SAGCluster models
|
||
- `01E-blueprint-aware-pipeline.md` — pipeline stage context
|
||
- `01G-sag-health-monitoring.md` — health scoring context
|
||
|
||
---
|
||
|
||
*This document is the single instruction set for building all 9 Phase 2 execution documents. Each doc section above contains the complete spec needed to write a self-contained execution doc that Claude Code can pick up and build independently.*
|