igny8/docs/40-WORKFLOWS/CONTENT-PIPELINE.md

Content Pipeline Workflow

Last Verified: December 25, 2025

---

Overview

The IGNY8 content pipeline transforms raw keywords into published WordPress articles through a multi-stage workflow. This can run manually (step-by-step) or automatically via the Automation module.

---

Pipeline Stages

┌─────────────────────────────────────────────────────────────────────────┐
│                         CONTENT PIPELINE                                 │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐          │
│  │ KEYWORDS │───►│ CLUSTERS │───►│  IDEAS   │───►│  TASKS   │          │
│  │  Stage 1 │    │  Stage 2 │    │  Stage 3 │    │  Stage 4 │          │
│  └──────────┘    └──────────┘    └──────────┘    └──────────┘          │
│                                                                          │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐          │
│  │ CONTENT  │───►│  IMAGES  │───►│  REVIEW  │───►│ PUBLISH  │          │
│  │  Stage 5 │    │  Stage 6 │    │  Stage 7 │    │  Stage 8 │          │
│  └──────────┘    └──────────┘    └──────────┘    └──────────┘          │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

---

Stage 1: Keywords

Module: Planner
Status Values: pending, clustered, used, archived

Input

• Seed keywords (manually added or from SEO tools)
• Optional: search volume, difficulty, CPC data

Process

1. User adds keywords via UI or bulk import
2. Keywords validated and deduplicated
3. Assigned to site + sector

Output

• Keyword records in Keyword model
• Status: pending

Credit Usage

• None (free operation)

---

Stage 2: Clustering

Module: Planner
AI Function: AutoClusterKeywords

Input

• Selected pending keywords (2-100)

Process

1. AI analyzes semantic relationships
2. Groups keywords by topic/intent
3. Creates cluster with name + description

Output

• Cluster records created
• Keywords linked to clusters
• Keyword status → clustered

Credit Usage

• 1 idea credit per clustering operation

---

Stage 3: Ideas

Module: Planner
AI Function: GenerateContentIdeas

Input

• Cluster with 2+ keywords

Process

1. AI generates content idea titles
2. Creates brief description for each
3. Suggests primary + secondary keywords

Output

• ContentIdea records created
• Status: pending

Credit Usage

• 1 idea credit per idea generated

---

Stage 4: Tasks

Module: Writer
Status Values: pending, in_progress, completed, cancelled

Input

• Selected content ideas

Process

1. Ideas converted to tasks
2. Task gets content brief + keywords
3. Optional: set due date, assign user

Output

• Task records created
• Status: pending
• Ideas status → used

Credit Usage

• None (free operation)

---

Stage 5: Content Generation

Module: Writer
AI Function: GenerateContent

Input

• Task with title + keywords + brief

Process

1. AI generates full article content
2. Creates structured HTML output
3. Adds meta title + description

Output

• Content record created
• Full HTML body
• SEO metadata
• Task status → completed

Credit Usage

• 1 content credit per generation

---

Stage 6: Image Generation

Module: Writer
AI Function: GenerateImages

Input

• Content record
• Number of images (default: 1-3)

Process

1. AI analyzes content for image prompts
2. Generates images via DALL-E/Runware
3. Creates thumbnail + full versions

Output

• ContentImage records created
• Image URLs + alt text
• Featured image assigned

Credit Usage

• 1 image credit per image generated

---

Stage 7: Review

Module: Writer
Status Values: draft, review, approved, published

Input

• Generated content + images

Process

1. Content displayed in rich editor
2. User reviews + edits if needed
3. User approves for publishing

Output

• Content status → approved
• Any manual edits saved

Credit Usage

• None (free operation)
• Regeneration costs additional credits

---

Stage 8: Publishing

Module: Publisher
Integration: WordPress REST API

Input

• Approved content
• WordPress integration credentials

Process

1. Content formatted for WordPress
2. Images uploaded to WP media
3. Post created with categories/tags
4. Status set to draft/published

Output

• PublishingRecord created
• WordPress post ID stored
• Content status → published

Credit Usage

• None (free operation)

---

Automation Mode

When running via Automation module:

1. Configuration - Set limits per stage
2. Execution - Pipeline runs automatically
3. Pacing - Configurable delays between operations
4. Monitoring - Real-time status updates

Automation Config Options

Stage Limits:
- clustering_limit: Max keywords to cluster
- ideas_limit: Max ideas to generate
- content_limit: Max content to generate
- image_limit: Max images to generate
- publish_limit: Max content to publish

Timing:
- delay_between_operations: Seconds between API calls
- max_runtime: Maximum run duration

Behavior:
- auto_approve: Skip review stage
- auto_publish: Publish immediately
- stop_on_error: Halt pipeline on failure

---

Data Flow Diagram

Seed Keywords
     │
     ▼
┌─────────────────┐
│    Keyword      │ status: pending
│    Model        │ belongs_to: site, sector
└────────┬────────┘
         │ AI: AutoCluster
         ▼
┌─────────────────┐
│    Cluster      │ keywords: [...]
│    Model        │ belongs_to: site, sector
└────────┬────────┘
         │ AI: GenerateIdeas
         ▼
┌─────────────────┐
│  ContentIdea    │ primary_keyword, secondaries
│    Model        │ cluster_id, status
└────────┬────────┘
         │ Convert to Task
         ▼
┌─────────────────┐
│     Task        │ idea_id, brief
│    Model        │ assigned_to, status
└────────┬────────┘
         │ AI: GenerateContent
         ▼
┌─────────────────┐
│    Content      │ task_id, body, meta
│    Model        │ status: draft
└────────┬────────┘
         │ AI: GenerateImages
         ▼
┌─────────────────┐
│ ContentImage    │ content_id, url
│    Model        │ alt_text, is_featured
└────────┬────────┘
         │ User Review
         ▼
┌─────────────────┐
│    Content      │ status: approved
│    (updated)    │
└────────┬────────┘
         │ WordPress API
         ▼
┌─────────────────┐
│PublishingRecord │ content_id, wp_post_id
│    Model        │ status, published_at
└─────────────────┘

---

Error Handling

Stage	Common Errors	Recovery
Clustering	API timeout	Retry with smaller batch
Ideas	API rate limit	Wait and retry
Content	Insufficient credits	Add credits, retry
Images	Image API failure	Skip images, continue
Publish	WordPress auth fail	Reauth integration

---

Monitoring

Pipeline Stats (Dashboard)

• Keywords pending clustering
• Ideas pending task creation
• Tasks pending generation
• Content pending review
• Content pending publish

Automation Logs

• Run ID + timestamps
• Stage + item processed
• Success/failure status
• Credit deductions
• Error messages