# Content Pipeline Workflow

**Last Verified:** December 25, 2025

---

## Overview

The IGNY8 content pipeline transforms raw keywords into published WordPress articles through a multi-stage workflow. This can run manually (step-by-step) or automatically via the Automation module.

---

## Pipeline Stages

```
┌─────────────────────────────────────────────────────────────────────────┐
│                         CONTENT PIPELINE                                 │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐          │
│  │ KEYWORDS │───►│ CLUSTERS │───►│  IDEAS   │───►│  TASKS   │          │
│  │  Stage 1 │    │  Stage 2 │    │  Stage 3 │    │  Stage 4 │          │
│  └──────────┘    └──────────┘    └──────────┘    └──────────┘          │
│                                                                          │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐          │
│  │ CONTENT  │───►│  IMAGES  │───►│  REVIEW  │───►│ PUBLISH  │          │
│  │  Stage 5 │    │  Stage 6 │    │  Stage 7 │    │  Stage 8 │          │
│  └──────────┘    └──────────┘    └──────────┘    └──────────┘          │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘
```

---

## Stage 1: Keywords

**Module:** Planner  
**Status Values:** `pending`, `clustered`, `used`, `archived`

### Input
- Seed keywords (manually added or from SEO tools)
- Optional: search volume, difficulty, CPC data

### Process
1. User adds keywords via UI or bulk import
2. Keywords validated and deduplicated
3. Assigned to site + sector

### Output
- Keyword records in `Keyword` model
- Status: `pending`

### Credit Usage
- None (free operation)

---

## Stage 2: Clustering

**Module:** Planner  
**AI Function:** `AutoClusterKeywords`

### Input
- Selected pending keywords (2-100)

### Process
1. AI analyzes semantic relationships
2. Groups keywords by topic/intent
3. Creates cluster with name + description

### Output
- `Cluster` records created
- Keywords linked to clusters
- Keyword status → `clustered`

### Credit Usage
- 1 idea credit per clustering operation

---

## Stage 3: Ideas

**Module:** Planner  
**AI Function:** `GenerateContentIdeas`

### Input
- Cluster with 2+ keywords

### Process
1. AI generates content idea titles
2. Creates brief description for each
3. Suggests primary + secondary keywords

### Output
- `ContentIdea` records created
- Status: `pending`

### Credit Usage
- 1 idea credit per idea generated

---

## Stage 4: Tasks

**Module:** Writer  
**Status Values:** `pending`, `in_progress`, `completed`, `cancelled`

### Input
- Selected content ideas

### Process
1. Ideas converted to tasks
2. Task gets content brief + keywords
3. Optional: set due date, assign user

### Output
- `Task` records created
- Status: `pending`
- Ideas status → `used`

### Credit Usage
- None (free operation)

---

## Stage 5: Content Generation

**Module:** Writer  
**AI Function:** `GenerateContent`

### Input
- Task with title + keywords + brief

### Process
1. AI generates full article content
2. Creates structured HTML output
3. Adds meta title + description

### Output
- `Content` record created
- Full HTML body
- SEO metadata
- Task status → `completed`

### Credit Usage
- 1 content credit per generation

---

## Stage 6: Image Generation

**Module:** Writer  
**AI Function:** `GenerateImages`

### Input
- Content record
- Number of images (default: 1-3)

### Process
1. AI analyzes content for image prompts
2. Generates images via DALL-E/Runware
3. Creates thumbnail + full versions

### Output
- `ContentImage` records created
- Image URLs + alt text
- Featured image assigned

### Credit Usage
- 1 image credit per image generated

---

## Stage 7: Review

**Module:** Writer  
**Status Values:** `draft`, `review`, `approved`, `published`

### Input
- Generated content + images

### Process
1. Content displayed in rich editor
2. User reviews + edits if needed
3. User approves for publishing

### Output
- Content status → `approved`
- Any manual edits saved

### Credit Usage
- None (free operation)
- Regeneration costs additional credits

---

## Stage 8: Publishing

**Module:** Publisher  
**Integration:** WordPress REST API

### Input
- Approved content
- WordPress integration credentials

### Process
1. Content formatted for WordPress
2. Images uploaded to WP media
3. Post created with categories/tags
4. Status set to draft/published

### Output
- `PublishingRecord` created
- WordPress post ID stored
- Content status → `published`

### Credit Usage
- None (free operation)

---

## Automation Mode

When running via Automation module:

1. **Configuration** - Set limits per stage
2. **Execution** - Pipeline runs automatically
3. **Pacing** - Configurable delays between operations
4. **Monitoring** - Real-time status updates

### Automation Config Options

```
Stage Limits:
- clustering_limit: Max keywords to cluster
- ideas_limit: Max ideas to generate
- content_limit: Max content to generate
- image_limit: Max images to generate
- publish_limit: Max content to publish

Timing:
- delay_between_operations: Seconds between API calls
- max_runtime: Maximum run duration

Behavior:
- auto_approve: Skip review stage
- auto_publish: Publish immediately
- stop_on_error: Halt pipeline on failure
```

---

## Data Flow Diagram

```
Seed Keywords
     │
     ▼
┌─────────────────┐
│    Keyword      │ status: pending
│    Model        │ belongs_to: site, sector
└────────┬────────┘
         │ AI: AutoCluster
         ▼
┌─────────────────┐
│    Cluster      │ keywords: [...]
│    Model        │ belongs_to: site, sector
└────────┬────────┘
         │ AI: GenerateIdeas
         ▼
┌─────────────────┐
│  ContentIdea    │ primary_keyword, secondaries
│    Model        │ cluster_id, status
└────────┬────────┘
         │ Convert to Task
         ▼
┌─────────────────┐
│     Task        │ idea_id, brief
│    Model        │ assigned_to, status
└────────┬────────┘
         │ AI: GenerateContent
         ▼
┌─────────────────┐
│    Content      │ task_id, body, meta
│    Model        │ status: draft
└────────┬────────┘
         │ AI: GenerateImages
         ▼
┌─────────────────┐
│ ContentImage    │ content_id, url
│    Model        │ alt_text, is_featured
└────────┬────────┘
         │ User Review
         ▼
┌─────────────────┐
│    Content      │ status: approved
│    (updated)    │
└────────┬────────┘
         │ WordPress API
         ▼
┌─────────────────┐
│PublishingRecord │ content_id, wp_post_id
│    Model        │ status, published_at
└─────────────────┘
```

---

## Error Handling

| Stage | Common Errors | Recovery |
|-------|---------------|----------|
| Clustering | API timeout | Retry with smaller batch |
| Ideas | API rate limit | Wait and retry |
| Content | Insufficient credits | Add credits, retry |
| Images | Image API failure | Skip images, continue |
| Publish | WordPress auth fail | Reauth integration |

---

## Monitoring

### Pipeline Stats (Dashboard)

- Keywords pending clustering
- Ideas pending task creation
- Tasks pending generation
- Content pending review
- Content pending publish

### Automation Logs

- Run ID + timestamps
- Stage + item processed
- Success/failure status
- Credit deductions
- Error messages