igny8/STATUS-UNIFIED-STRUCTURE-PLAN-v2.md

# Unified Status Structure Plan - Keywords, Clusters, Ideas (CORRECTED)

**Date:** December 3, 2025  
**Objective:** Implement unified, workflow-driven status values across all three modules  

---

## 🎯 Key Concept: Status vs Workflow

### ✅ WORKFLOW STATUS (Auto-updated by AI/System)
These statuses represent the workflow progression:
- Keywords: `new` → `mapped`
- Clusters: `new` → `mapped`
- Ideas: `new` → `queued` → `completed`

### 🔘 OPTIONAL MANUAL STATUS (User control only)
`disabled` = **NOT a workflow step**, but a **filter status**
- Does NOT affect workflow
- Manually set by user only
- Excludes items from all automated processes
- Excludes from dashboard metrics/suggestions
- Can be toggled on/off anytime

**Important:** `disabled` is a **data attribute**, not a workflow state. Items with `disabled=true` are simply ignored by all processes.

---

## 📋 Module Status Definitions

### KEYWORDS Module
**Workflow Statuses:** `new` → `mapped`  
**Optional Filter Status:** `disabled` (manual user control)

| Status | Type | Meaning | When Set | Process Behavior |
|--------|------|---------|----------|-----------------|
| **new** | Workflow | Keyword attached to site, awaiting cluster assignment | User adds keyword via import/UI | Included in auto-cluster AI suggestions |
| **mapped** | Workflow | Keyword assigned to a cluster | AI auto_cluster completes or manual cluster assignment | Can be used for idea generation; included in cluster-based workflows |
| **disabled** | Filter | Manually excluded from processes | User manually sets | Excluded from auto-cluster, idea generation, dashboard (optional) |

**Workflow Transition:**
```
new ──[AI auto_cluster]──> mapped
 │
 └──[User toggle]──> disabled (any time, no workflow impact)
```

---

### CLUSTERS Module
**Workflow Statuses:** `new` → `mapped`  
**Optional Filter Status:** `disabled` (manual user control)

| Status | Type | Meaning | When Set | Process Behavior |
|--------|------|---------|----------|-----------------|
| **new** | Workflow | Cluster created; ready for idea generation | AI clustering creates OR user manually creates | Included in auto-generate-ideas suggestions |
| **mapped** | Workflow | Ideas generated from this cluster | AI generate_ideas completes | Ideas are ready; can be queued to writer |
| **disabled** | Filter | Manually excluded from processes | User manually sets | Excluded from idea generation, suggestion lists, dashboard (optional) |

**Workflow Transition:**
```
new ──[AI generate_ideas]──> mapped
 │
 └──[User toggle]──> disabled (any time, no workflow impact)
```

---

### IDEAS Module
**Workflow Statuses:** `new` → `queued` → `completed`  
**Optional Filter Status:** `disabled` (manual user control)

| Status | Type | Meaning | When Set | Process Behavior |
|--------|------|---------|----------|-----------------|
| **new** | Workflow | Idea generated by AI; awaiting queue to writer | AI generate_ideas creates | Included in bulk-queue suggestions |
| **queued** | Workflow | Queued to writer; Task record created in Writer module | User bulk-queues ideas to writer | Task is assigned; waiting for content generation |
| **completed** | Workflow | Content generated from task (tracked via Task.status='completed') | generate_content AI completes; Content record created | Final automated state; content ready for publishing/deployment |
| **disabled** | Filter | Manually excluded from processes | User manually sets | Excluded from queue suggestions, bulk operations, dashboard (optional) |

**Workflow Transition:**
```
new ──[User bulk_queue]──> queued ──[Writer AI: generate_content]──> completed
 │
 └──[User toggle]──> disabled (any time, no workflow impact)
```

**Note on `completed`:**
- When a Task's status becomes `completed`, the related Idea automatically becomes `completed`
- No separate `published` status needed; publishing is a separate content deployment action
- One idea = one content piece through task tracking

---

## 🔄 Process Exclusion Rules

### When `disabled=true` (filter status):
Items are **excluded from**:
- ✗ AI process suggestions (auto-cluster, generate-ideas)
- ✗ Bulk operation selections (queue to writer, generate content)
- ✗ Dashboard workflow metrics (unless explicitly shown)
- ✗ Progress calculations (% mapped, % queued, etc.)

### When workflow status applies:
- ✅ Included in relevant processes
- ✅ Included in dashboard metrics
- ✅ Included in AI suggestions
- ✅ Can be bulk-operated on

---

## 📊 Status Update Flow

```
═══ KEYWORDS WORKFLOW ═══
1. User imports SeedKeywords
   └─> Keywords created with status='new'

2. Auto-cluster AI runs
   └─> Keywords assigned to clusters
   └─> Keywords status changes to 'mapped'

3. [Optional] User manually disables keyword
   └─> Keywords.disabled = true
   └─> Excluded from all processes


═══ CLUSTERS WORKFLOW ═══
1. Auto-cluster AI creates/updates clusters
   └─> Clusters created with status='new'

2. AI generate-ideas runs on 'new' clusters
   └─> Ideas created for cluster
   └─> Clusters status changes to 'mapped'

3. [Optional] User manually disables cluster
   └─> Clusters.disabled = true
   └─> Excluded from all processes


═══ IDEAS WORKFLOW ═══
1. AI generate-ideas creates ideas for clusters
   └─> Ideas created with status='new'

2. User bulk-queues ideas to writer
   └─> Task created in Writer module
   └─> Ideas status changes to 'queued'

3. Writer AI (generate-content) creates content
   └─> Content record created
   └─> Task.status = 'completed'
   └─> Ideas.status = 'completed' [Auto-sync from Task]

4. [Optional] User manually disables idea
   └─> Ideas.disabled = true
   └─> Excluded from all processes
```

---

## 🔧 Implementation Changes

### Backend Model Changes

#### 1. `/backend/igny8_core/business/planning/models.py`

**Keywords Model:**
```python
# REMOVE old STATUS_CHOICES
# OLD:
STATUS_CHOICES = [
    ('active', 'Active'),
    ('pending', 'Pending'),
    ('archived', 'Archived'),
]

# NEW:
STATUS_CHOICES = [
    ('new', 'New'),
    ('mapped', 'Mapped'),
]
status = models.CharField(
    max_length=50, 
    choices=STATUS_CHOICES, 
    default='new'
)

# ADD new filter field:
disabled = models.BooleanField(default=False, help_text="Exclude from processes")
```

**Clusters Model:**
```python
# ADD STATUS_CHOICES (currently hardcoded as default='active', no choices)
# CURRENT: status = models.CharField(max_length=50, default='active')

# NEW:
STATUS_CHOICES = [
    ('new', 'New'),
    ('mapped', 'Mapped'),
]
status = models.CharField(
    max_length=50,
    choices=STATUS_CHOICES,
    default='new'
)

# ADD new filter field:
disabled = models.BooleanField(default=False, help_text="Exclude from processes")
```

**ContentIdeas Model:**
```python
# UPDATE STATUS_CHOICES
# REMOVE 'scheduled', 'published'
STATUS_CHOICES = [
    ('new', 'New'),
    ('queued', 'Queued'),
    ('completed', 'Completed'),
]
status = models.CharField(
    max_length=50,
    choices=STATUS_CHOICES,
    default='new'
)

# ADD new filter field:
disabled = models.BooleanField(default=False, help_text="Exclude from processes")
```

---

### Backend AI Function Updates

#### 2. `/backend/igny8_core/ai/functions/auto_cluster.py`

**Location: Line ~297 in save_output()**
```python
# CURRENT:
status='active'

# NEW:
status='mapped'  # When keywords are assigned to cluster
```

---

#### 3. `/backend/igny8_core/ai/functions/generate_ideas.py`

**Location: After ideas created in save_output()**
```python
# NEW: Add after creating ideas
# Update cluster status from 'new' to 'mapped' after ideas generated
for cluster in clusters_used:
    if cluster.status == 'new':
        cluster.status = 'mapped'
        cluster.save()
```

---

#### 4. `/backend/igny8_core/ai/functions/generate_content.py`

**Location: Line ~318 in save_output()**
```python
# CURRENT:
task.status = 'completed'
task.save()

# NEW: Auto-sync idea status from task
task.status = 'completed'
task.save()

# NEW: Update related idea to 'completed'
if hasattr(task, 'idea') and task.idea:
    idea = task.idea
    idea.status = 'completed'
    idea.save()
```

---

### Backend API Updates

#### 5. `/backend/igny8_core/modules/planner/views.py`

**Queue ideas to writer (Line ~1084):**
```python
# CURRENT:
idea.status = 'scheduled'

# NEW:
idea.status = 'queued'
```

**CSV import defaults (Line ~563):**
```python
# CURRENT:
status=row.get('status', 'pending') or 'pending'

# NEW:
status=row.get('status', 'new') or 'new'
```

**Filter out disabled items in list views:**
```python
# NEW: When returning lists, exclude disabled=true (optional, configurable)
queryset = queryset.filter(disabled=False)  # Or show both with filter option
```

---

### Frontend Configuration Updates

#### 6. `/frontend/src/config/pages/keywords.config.tsx`

**Table Column - Status Badge (Lines ~230-248):**
```typescript
// CURRENT:
new: 'amber', active: 'green', archived: 'red'

// NEW:
new: 'amber', mapped: 'green'

// ADD: Display 'disabled' badge if disabled=true
```

**Filter Dropdown (Lines ~310-318):**
```typescript
// CURRENT:
options: ['pending', 'active', 'archived']

// NEW:
options: ['new', 'mapped'],
additionalFilters: [
  { key: 'disabled', label: 'Show Disabled', type: 'checkbox', default: false }
]
```

**Form Field Default (Lines ~560-570):**
```typescript
// CURRENT:
default: 'pending'

// NEW:
default: 'new'

// ADD: disabled checkbox in form
```

---

#### 7. `/frontend/src/config/pages/clusters.config.tsx`

**Table Column - Status Badge (Lines ~190-200):**
```typescript
// CURRENT:
(missing or wrong values)

// NEW:
new: 'amber', mapped: 'green'

// ADD: Display 'disabled' badge if disabled=true
```

**Filter Dropdown (Lines ~240-253):**
```typescript
// CURRENT:
options: ['new', 'idea', 'mapped']

// NEW:
options: ['new', 'mapped'],
additionalFilters: [
  { key: 'disabled', label: 'Show Disabled', type: 'checkbox', default: false }
]
```

**Form Field (Lines ~405-418):**
```typescript
// ADD: disabled checkbox in form
// Keep status default as 'new'
```

---

#### 8. `/frontend/src/config/pages/ideas.config.tsx`

**Table Column - Status Badge (Lines ~170-185):**
```typescript
// CURRENT:
new: 'amber', scheduled: 'blue', completed: 'blue', published: 'green'

// NEW:
new: 'amber', queued: 'blue', completed: 'green'

// ADD: Display 'disabled' badge if disabled=true
```

**Filter Dropdown (Lines ~218-228):**
```typescript
// CURRENT:
options: ['new', 'scheduled', 'completed', 'published']

// NEW:
options: ['new', 'queued', 'completed'],
additionalFilters: [
  { key: 'disabled', label: 'Show Disabled', type: 'checkbox', default: false }
]
```

**Form Field (Lines ~372-385):**
```typescript
// REMOVE: 'published' option
// ADD: disabled checkbox in form
// Keep status default as 'new'
```

---

### Frontend Dashboard

#### 9. `/frontend/src/pages/Planner/Dashboard.tsx`

**Status Metrics:**
```typescript
// Update calculations:
// - Keywords: new vs mapped
// - Clusters: new vs mapped
// - Ideas: new vs queued vs completed

// Optional: Add disabled count metrics
```

---

## 📍 Change Summary

### Total Files: 9

| File | Changes | Type |
|------|---------|------|
| **models.py** | Add STATUS_CHOICES for Clusters; Update Keywords/Ideas; Add `disabled` field to all 3 | Backend |
| **auto_cluster.py** | Line 297: `status='active'` → `status='mapped'` | Backend AI |
| **generate_ideas.py** | After ideas created: Set cluster `status='mapped'` | Backend AI |
| **generate_content.py** | Line 318: Also sync `idea.status='completed'` from task | Backend AI |
| **views.py** | Queue ideas (line 1084): `'scheduled'` → `'queued'`; CSV import: `'pending'` → `'new'` | Backend API |
| **keywords.config.tsx** | Update badge/filter/form for new/mapped; Add disabled checkbox | Frontend Config |
| **clusters.config.tsx** | Update badge/filter/form for new/mapped; Add disabled checkbox | Frontend Config |
| **ideas.config.tsx** | Update badge/filter/form for new/queued/completed; Add disabled checkbox | Frontend Config |
| **Dashboard.tsx** | Update metrics calculations for new status values | Frontend Page |

---

## ✅ Workflow Validations

### Keywords
```
✓ new → mapped (via auto_cluster)
✓ Can toggle disabled at any time (no workflow impact)
✓ Disabled items excluded from auto-cluster suggestions
```

### Clusters
```
✓ new → mapped (via generate_ideas)
✓ Can toggle disabled at any time (no workflow impact)
✓ Disabled items excluded from idea generation
```

### Ideas
```
✓ new → queued (via bulk_queue_to_writer)
✓ queued → completed (via generate_content, tracked via Task.status)
✓ Can toggle disabled at any time (no workflow impact)
✓ Disabled items excluded from queue suggestions
```

---

## 🗑️ Data Migration Strategy

### Django Migration File
**File:** `backend/igny8_core/business/planning/migrations/NNNN_unified_status_fields.py`

```python
# Forward migration:
# 1. Add disabled=BooleanField(default=False) to all 3 models
# 2. Add STATUS_CHOICES to Clusters model
# 3. Update Keywords.status data:
#    - pending → new
#    - active → mapped
#    - archived → mapped + set disabled=True
# 4. Update Clusters.status data:
#    - active (with ideas_count > 0) → mapped
#    - active (with ideas_count = 0) → new
# 5. Update ContentIdeas.status data:
#    - scheduled → queued
#    - published → completed
#    - new → new (no change)

# Reverse migration:
# 1. Remove disabled field from all 3 models
# 2. Restore old STATUS_CHOICES
# 3. Reverse data transformations:
#    - Keywords: new→pending, mapped→active, disabled=True→archived
#    - Clusters: Remove STATUS_CHOICES, set all to 'active'
#    - Ideas: queued→scheduled, completed→published
```

---

## 🧪 Testing Checklist

- [ ] Keywords: status='new' by default
- [ ] Keywords: auto_cluster sets status='mapped'
- [ ] Clusters: status='new' by default
- [ ] Clusters: generate_ideas sets status='mapped'
- [ ] Ideas: status='new' by default
- [ ] Ideas: bulk_queue_to_writer sets status='queued'
- [ ] Ideas: generate_content sets status='completed' (via task sync)
- [ ] All modules: disabled=true excludes from processes
- [ ] All modules: disabled checkbox toggles correctly
- [ ] Dashboard: metrics use new status values
- [ ] Filters: disabled checkbox hides disabled items by default
- [ ] Migration: old data transforms correctly

---

## 🎯 Key Differences from Previous Plan

| Previous | Now Corrected |
|----------|---------------|
| `disabled` was workflow step | `disabled` is filter status only |
| 3 workflows statuses per module | 2 workflow statuses per module |
| Published status for Ideas | Completed is final; publish is separate action |
| Ideas tracked separately | Ideas auto-sync from Task status |
| Unclear disabled behavior | Clear: disabled excluded from all processes |

---

**Status:** ✅ Plan Complete & Corrected
**Ready for:** Implementation Phase 1 (Backend Models)