igny8/STATUS-UNIFIED-STRUCTURE-PLAN-v2.md

Unified Status Structure Plan - Keywords, Clusters, Ideas (CORRECTED)

Date: December 3, 2025
Objective: Implement unified, workflow-driven status values across all three modules

---

🎯 Key Concept: Status vs Workflow

✅ WORKFLOW STATUS (Auto-updated by AI/System)

These statuses represent the workflow progression:

• Keywords: new → mapped
• Clusters: new → mapped
• Ideas: new → queued → completed

🔘 OPTIONAL MANUAL STATUS (User control only)

disabled = NOT a workflow step, but a filter status

• Does NOT affect workflow
• Manually set by user only
• Excludes items from all automated processes
• Excludes from dashboard metrics/suggestions
• Can be toggled on/off anytime

Important: disabled is a data attribute, not a workflow state. Items with disabled=true are simply ignored by all processes.

---

📋 Module Status Definitions

KEYWORDS Module

Workflow Statuses: new → mapped
Optional Filter Status: disabled (manual user control)

Status	Type	Meaning	When Set	Process Behavior
new	Workflow	Keyword attached to site, awaiting cluster assignment	User adds keyword via import/UI	Included in auto-cluster AI suggestions
mapped	Workflow	Keyword assigned to a cluster	AI auto_cluster completes or manual cluster assignment	Can be used for idea generation; included in cluster-based workflows
disabled	Filter	Manually excluded from processes	User manually sets	Excluded from auto-cluster, idea generation, dashboard (optional)

Workflow Transition:

new ──[AI auto_cluster]──> mapped
 │
 └──[User toggle]──> disabled (any time, no workflow impact)

---

CLUSTERS Module

Workflow Statuses: new → mapped
Optional Filter Status: disabled (manual user control)

Status	Type	Meaning	When Set	Process Behavior
new	Workflow	Cluster created; ready for idea generation	AI clustering creates OR user manually creates	Included in auto-generate-ideas suggestions
mapped	Workflow	Ideas generated from this cluster	AI generate_ideas completes	Ideas are ready; can be queued to writer
disabled	Filter	Manually excluded from processes	User manually sets	Excluded from idea generation, suggestion lists, dashboard (optional)

Workflow Transition:

new ──[AI generate_ideas]──> mapped
 │
 └──[User toggle]──> disabled (any time, no workflow impact)

---

IDEAS Module

Workflow Statuses: new → queued → completed
Optional Filter Status: disabled (manual user control)

Status	Type	Meaning	When Set	Process Behavior
new	Workflow	Idea generated by AI; awaiting queue to writer	AI generate_ideas creates	Included in bulk-queue suggestions
queued	Workflow	Queued to writer; Task record created in Writer module	User bulk-queues ideas to writer	Task is assigned; waiting for content generation
completed	Workflow	Content generated from task (tracked via Task.status='completed')	generate_content AI completes; Content record created	Final automated state; content ready for publishing/deployment
disabled	Filter	Manually excluded from processes	User manually sets	Excluded from queue suggestions, bulk operations, dashboard (optional)

Workflow Transition:

new ──[User bulk_queue]──> queued ──[Writer AI: generate_content]──> completed
 │
 └──[User toggle]──> disabled (any time, no workflow impact)

Note on completed:

• When a Task's status becomes completed, the related Idea automatically becomes completed
• No separate published status needed; publishing is a separate content deployment action
• One idea = one content piece through task tracking

---

🔄 Process Exclusion Rules

When disabled=true (filter status):

Items are excluded from:

• ✗ AI process suggestions (auto-cluster, generate-ideas)
• ✗ Bulk operation selections (queue to writer, generate content)
• ✗ Dashboard workflow metrics (unless explicitly shown)
• ✗ Progress calculations (% mapped, % queued, etc.)

When workflow status applies:

• ✅ Included in relevant processes
• ✅ Included in dashboard metrics
• ✅ Included in AI suggestions
• ✅ Can be bulk-operated on

---

📊 Status Update Flow

═══ KEYWORDS WORKFLOW ═══
1. User imports SeedKeywords
   └─> Keywords created with status='new'

2. Auto-cluster AI runs
   └─> Keywords assigned to clusters
   └─> Keywords status changes to 'mapped'

3. [Optional] User manually disables keyword
   └─> Keywords.disabled = true
   └─> Excluded from all processes


═══ CLUSTERS WORKFLOW ═══
1. Auto-cluster AI creates/updates clusters
   └─> Clusters created with status='new'

2. AI generate-ideas runs on 'new' clusters
   └─> Ideas created for cluster
   └─> Clusters status changes to 'mapped'

3. [Optional] User manually disables cluster
   └─> Clusters.disabled = true
   └─> Excluded from all processes


═══ IDEAS WORKFLOW ═══
1. AI generate-ideas creates ideas for clusters
   └─> Ideas created with status='new'

2. User bulk-queues ideas to writer
   └─> Task created in Writer module
   └─> Ideas status changes to 'queued'

3. Writer AI (generate-content) creates content
   └─> Content record created
   └─> Task.status = 'completed'
   └─> Ideas.status = 'completed' [Auto-sync from Task]

4. [Optional] User manually disables idea
   └─> Ideas.disabled = true
   └─> Excluded from all processes

---

🔧 Implementation Changes

Backend Model Changes

1. /backend/igny8_core/business/planning/models.py

Keywords Model:

# REMOVE old STATUS_CHOICES
# OLD:
STATUS_CHOICES = [
    ('active', 'Active'),
    ('pending', 'Pending'),
    ('archived', 'Archived'),
]

# NEW:
STATUS_CHOICES = [
    ('new', 'New'),
    ('mapped', 'Mapped'),
]
status = models.CharField(
    max_length=50, 
    choices=STATUS_CHOICES, 
    default='new'
)

# ADD new filter field:
disabled = models.BooleanField(default=False, help_text="Exclude from processes")

Clusters Model:

# ADD STATUS_CHOICES (currently hardcoded as default='active', no choices)
# CURRENT: status = models.CharField(max_length=50, default='active')

# NEW:
STATUS_CHOICES = [
    ('new', 'New'),
    ('mapped', 'Mapped'),
]
status = models.CharField(
    max_length=50,
    choices=STATUS_CHOICES,
    default='new'
)

# ADD new filter field:
disabled = models.BooleanField(default=False, help_text="Exclude from processes")

ContentIdeas Model:

# UPDATE STATUS_CHOICES
# REMOVE 'scheduled', 'published'
STATUS_CHOICES = [
    ('new', 'New'),
    ('queued', 'Queued'),
    ('completed', 'Completed'),
]
status = models.CharField(
    max_length=50,
    choices=STATUS_CHOICES,
    default='new'
)

# ADD new filter field:
disabled = models.BooleanField(default=False, help_text="Exclude from processes")

---

Backend AI Function Updates

2. /backend/igny8_core/ai/functions/auto_cluster.py

Location: Line ~297 in save_output()

# CURRENT:
status='active'

# NEW:
status='mapped'  # When keywords are assigned to cluster

---

3. /backend/igny8_core/ai/functions/generate_ideas.py

Location: After ideas created in save_output()

# NEW: Add after creating ideas
# Update cluster status from 'new' to 'mapped' after ideas generated
for cluster in clusters_used:
    if cluster.status == 'new':
        cluster.status = 'mapped'
        cluster.save()

---

4. /backend/igny8_core/ai/functions/generate_content.py

Location: Line ~318 in save_output()

# CURRENT:
task.status = 'completed'
task.save()

# NEW: Auto-sync idea status from task
task.status = 'completed'
task.save()

# NEW: Update related idea to 'completed'
if hasattr(task, 'idea') and task.idea:
    idea = task.idea
    idea.status = 'completed'
    idea.save()

---

Backend API Updates

5. /backend/igny8_core/modules/planner/views.py

Queue ideas to writer (Line ~1084):

# CURRENT:
idea.status = 'scheduled'

# NEW:
idea.status = 'queued'

CSV import defaults (Line ~563):

# CURRENT:
status=row.get('status', 'pending') or 'pending'

# NEW:
status=row.get('status', 'new') or 'new'

Filter out disabled items in list views:

# NEW: When returning lists, exclude disabled=true (optional, configurable)
queryset = queryset.filter(disabled=False)  # Or show both with filter option

---

Frontend Configuration Updates

6. /frontend/src/config/pages/keywords.config.tsx

Table Column - Status Badge (Lines ~230-248):

// CURRENT:
new: 'amber', active: 'green', archived: 'red'

// NEW:
new: 'amber', mapped: 'green'

// ADD: Display 'disabled' badge if disabled=true

Filter Dropdown (Lines ~310-318):

// CURRENT:
options: ['pending', 'active', 'archived']

// NEW:
options: ['new', 'mapped'],
additionalFilters: [
  { key: 'disabled', label: 'Show Disabled', type: 'checkbox', default: false }
]

Form Field Default (Lines ~560-570):

// CURRENT:
default: 'pending'

// NEW:
default: 'new'

// ADD: disabled checkbox in form

---

7. /frontend/src/config/pages/clusters.config.tsx

Table Column - Status Badge (Lines ~190-200):

// CURRENT:
(missing or wrong values)

// NEW:
new: 'amber', mapped: 'green'

// ADD: Display 'disabled' badge if disabled=true

Filter Dropdown (Lines ~240-253):

// CURRENT:
options: ['new', 'idea', 'mapped']

// NEW:
options: ['new', 'mapped'],
additionalFilters: [
  { key: 'disabled', label: 'Show Disabled', type: 'checkbox', default: false }
]

Form Field (Lines ~405-418):

// ADD: disabled checkbox in form
// Keep status default as 'new'

---

8. /frontend/src/config/pages/ideas.config.tsx

Table Column - Status Badge (Lines ~170-185):

// CURRENT:
new: 'amber', scheduled: 'blue', completed: 'blue', published: 'green'

// NEW:
new: 'amber', queued: 'blue', completed: 'green'

// ADD: Display 'disabled' badge if disabled=true

Filter Dropdown (Lines ~218-228):

// CURRENT:
options: ['new', 'scheduled', 'completed', 'published']

// NEW:
options: ['new', 'queued', 'completed'],
additionalFilters: [
  { key: 'disabled', label: 'Show Disabled', type: 'checkbox', default: false }
]

Form Field (Lines ~372-385):

// REMOVE: 'published' option
// ADD: disabled checkbox in form
// Keep status default as 'new'

---

Frontend Dashboard

9. /frontend/src/pages/Planner/Dashboard.tsx

Status Metrics:

// Update calculations:
// - Keywords: new vs mapped
// - Clusters: new vs mapped
// - Ideas: new vs queued vs completed

// Optional: Add disabled count metrics

---

📍 Change Summary

Total Files: 9

File	Changes	Type
models.py	Add STATUS_CHOICES for Clusters; Update Keywords/Ideas; Add disabled field to all 3	Backend
auto_cluster.py	Line 297: status='active' → status='mapped'	Backend AI
generate_ideas.py	After ideas created: Set cluster status='mapped'	Backend AI
generate_content.py	Line 318: Also sync idea.status='completed' from task	Backend AI
views.py	Queue ideas (line 1084): 'scheduled' → 'queued'; CSV import: 'pending' → 'new'	Backend API
keywords.config.tsx	Update badge/filter/form for new/mapped; Add disabled checkbox	Frontend Config
clusters.config.tsx	Update badge/filter/form for new/mapped; Add disabled checkbox	Frontend Config
ideas.config.tsx	Update badge/filter/form for new/queued/completed; Add disabled checkbox	Frontend Config
Dashboard.tsx	Update metrics calculations for new status values	Frontend Page

---

✅ Workflow Validations

Keywords

✓ new → mapped (via auto_cluster)
✓ Can toggle disabled at any time (no workflow impact)
✓ Disabled items excluded from auto-cluster suggestions

Clusters

✓ new → mapped (via generate_ideas)
✓ Can toggle disabled at any time (no workflow impact)
✓ Disabled items excluded from idea generation

Ideas

✓ new → queued (via bulk_queue_to_writer)
✓ queued → completed (via generate_content, tracked via Task.status)
✓ Can toggle disabled at any time (no workflow impact)
✓ Disabled items excluded from queue suggestions

---

🗑️ Data Migration Strategy

Django Migration File

File: backend/igny8_core/business/planning/migrations/NNNN_unified_status_fields.py

# Forward migration:
# 1. Add disabled=BooleanField(default=False) to all 3 models
# 2. Add STATUS_CHOICES to Clusters model
# 3. Update Keywords.status data:
#    - pending → new
#    - active → mapped
#    - archived → mapped + set disabled=True
# 4. Update Clusters.status data:
#    - active (with ideas_count > 0) → mapped
#    - active (with ideas_count = 0) → new
# 5. Update ContentIdeas.status data:
#    - scheduled → queued
#    - published → completed
#    - new → new (no change)

# Reverse migration:
# 1. Remove disabled field from all 3 models
# 2. Restore old STATUS_CHOICES
# 3. Reverse data transformations:
#    - Keywords: new→pending, mapped→active, disabled=True→archived
#    - Clusters: Remove STATUS_CHOICES, set all to 'active'
#    - Ideas: queued→scheduled, completed→published

---

🧪 Testing Checklist

• Keywords: status='new' by default
• Keywords: auto_cluster sets status='mapped'
• Clusters: status='new' by default
• Clusters: generate_ideas sets status='mapped'
• Ideas: status='new' by default
• Ideas: bulk_queue_to_writer sets status='queued'
• Ideas: generate_content sets status='completed' (via task sync)
• All modules: disabled=true excludes from processes
• All modules: disabled checkbox toggles correctly
• Dashboard: metrics use new status values
• Filters: disabled checkbox hides disabled items by default
• Migration: old data transforms correctly

---

🎯 Key Differences from Previous Plan

Previous	Now Corrected
disabled was workflow step	disabled is filter status only
3 workflows statuses per module	2 workflow statuses per module
Published status for Ideas	Completed is final; publish is separate action
Ideas tracked separately	Ideas auto-sync from Task status
Unclear disabled behavior	Clear: disabled excluded from all processes

---

Status: ✅ Plan Complete & Corrected Ready for: Implementation Phase 1 (Backend Models)