old automation cleanup adn status feilds of planner udpate
This commit is contained in:
IGNY8 VPS (Salman)
541
STATUS-UNIFIED-STRUCTURE-PLAN-v2.md
Normal file
541
STATUS-UNIFIED-STRUCTURE-PLAN-v2.md
Normal file
@@ -0,0 +1,541 @@
|
||||
# Unified Status Structure Plan - Keywords, Clusters, Ideas (CORRECTED)
|
||||
|
||||
**Date:** December 3, 2025
|
||||
**Objective:** Implement unified, workflow-driven status values across all three modules
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Key Concept: Status vs Workflow
|
||||
|
||||
### ✅ WORKFLOW STATUS (Auto-updated by AI/System)
|
||||
These statuses represent the workflow progression:
|
||||
- Keywords: `new` → `mapped`
|
||||
- Clusters: `new` → `mapped`
|
||||
- Ideas: `new` → `queued` → `completed`
|
||||
|
||||
### 🔘 OPTIONAL MANUAL STATUS (User control only)
|
||||
`disabled` = **NOT a workflow step**, but a **filter status**
|
||||
- Does NOT affect workflow
|
||||
- Manually set by user only
|
||||
- Excludes items from all automated processes
|
||||
- Excludes from dashboard metrics/suggestions
|
||||
- Can be toggled on/off anytime
|
||||
|
||||
**Important:** `disabled` is a **data attribute**, not a workflow state. Items with `disabled=true` are simply ignored by all processes.
|
||||
|
||||
---
|
||||
|
||||
## 📋 Module Status Definitions
|
||||
|
||||
### KEYWORDS Module
|
||||
**Workflow Statuses:** `new` → `mapped`
|
||||
**Optional Filter Status:** `disabled` (manual user control)
|
||||
|
||||
| Status | Type | Meaning | When Set | Process Behavior |
|
||||
|--------|------|---------|----------|-----------------|
|
||||
| **new** | Workflow | Keyword attached to site, awaiting cluster assignment | User adds keyword via import/UI | Included in auto-cluster AI suggestions |
|
||||
| **mapped** | Workflow | Keyword assigned to a cluster | AI auto_cluster completes or manual cluster assignment | Can be used for idea generation; included in cluster-based workflows |
|
||||
| **disabled** | Filter | Manually excluded from processes | User manually sets | Excluded from auto-cluster, idea generation, dashboard (optional) |
|
||||
|
||||
**Workflow Transition:**
|
||||
```
|
||||
new ──[AI auto_cluster]──> mapped
|
||||
│
|
||||
└──[User toggle]──> disabled (any time, no workflow impact)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### CLUSTERS Module
|
||||
**Workflow Statuses:** `new` → `mapped`
|
||||
**Optional Filter Status:** `disabled` (manual user control)
|
||||
|
||||
| Status | Type | Meaning | When Set | Process Behavior |
|
||||
|--------|------|---------|----------|-----------------|
|
||||
| **new** | Workflow | Cluster created; ready for idea generation | AI clustering creates OR user manually creates | Included in auto-generate-ideas suggestions |
|
||||
| **mapped** | Workflow | Ideas generated from this cluster | AI generate_ideas completes | Ideas are ready; can be queued to writer |
|
||||
| **disabled** | Filter | Manually excluded from processes | User manually sets | Excluded from idea generation, suggestion lists, dashboard (optional) |
|
||||
|
||||
**Workflow Transition:**
|
||||
```
|
||||
new ──[AI generate_ideas]──> mapped
|
||||
│
|
||||
└──[User toggle]──> disabled (any time, no workflow impact)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### IDEAS Module
|
||||
**Workflow Statuses:** `new` → `queued` → `completed`
|
||||
**Optional Filter Status:** `disabled` (manual user control)
|
||||
|
||||
| Status | Type | Meaning | When Set | Process Behavior |
|
||||
|--------|------|---------|----------|-----------------|
|
||||
| **new** | Workflow | Idea generated by AI; awaiting queue to writer | AI generate_ideas creates | Included in bulk-queue suggestions |
|
||||
| **queued** | Workflow | Queued to writer; Task record created in Writer module | User bulk-queues ideas to writer | Task is assigned; waiting for content generation |
|
||||
| **completed** | Workflow | Content generated from task (tracked via Task.status='completed') | generate_content AI completes; Content record created | Final automated state; content ready for publishing/deployment |
|
||||
| **disabled** | Filter | Manually excluded from processes | User manually sets | Excluded from queue suggestions, bulk operations, dashboard (optional) |
|
||||
|
||||
**Workflow Transition:**
|
||||
```
|
||||
new ──[User bulk_queue]──> queued ──[Writer AI: generate_content]──> completed
|
||||
│
|
||||
└──[User toggle]──> disabled (any time, no workflow impact)
|
||||
```
|
||||
|
||||
**Note on `completed`:**
|
||||
- When a Task's status becomes `completed`, the related Idea automatically becomes `completed`
|
||||
- No separate `published` status needed; publishing is a separate content deployment action
|
||||
- One idea = one content piece through task tracking
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Process Exclusion Rules
|
||||
|
||||
### When `disabled=true` (filter status):
|
||||
Items are **excluded from**:
|
||||
- ✗ AI process suggestions (auto-cluster, generate-ideas)
|
||||
- ✗ Bulk operation selections (queue to writer, generate content)
|
||||
- ✗ Dashboard workflow metrics (unless explicitly shown)
|
||||
- ✗ Progress calculations (% mapped, % queued, etc.)
|
||||
|
||||
### When workflow status applies:
|
||||
- ✅ Included in relevant processes
|
||||
- ✅ Included in dashboard metrics
|
||||
- ✅ Included in AI suggestions
|
||||
- ✅ Can be bulk-operated on
|
||||
|
||||
---
|
||||
|
||||
## 📊 Status Update Flow
|
||||
|
||||
```
|
||||
═══ KEYWORDS WORKFLOW ═══
|
||||
1. User imports SeedKeywords
|
||||
└─> Keywords created with status='new'
|
||||
|
||||
2. Auto-cluster AI runs
|
||||
└─> Keywords assigned to clusters
|
||||
└─> Keywords status changes to 'mapped'
|
||||
|
||||
3. [Optional] User manually disables keyword
|
||||
└─> Keywords.disabled = true
|
||||
└─> Excluded from all processes
|
||||
|
||||
|
||||
═══ CLUSTERS WORKFLOW ═══
|
||||
1. Auto-cluster AI creates/updates clusters
|
||||
└─> Clusters created with status='new'
|
||||
|
||||
2. AI generate-ideas runs on 'new' clusters
|
||||
└─> Ideas created for cluster
|
||||
└─> Clusters status changes to 'mapped'
|
||||
|
||||
3. [Optional] User manually disables cluster
|
||||
└─> Clusters.disabled = true
|
||||
└─> Excluded from all processes
|
||||
|
||||
|
||||
═══ IDEAS WORKFLOW ═══
|
||||
1. AI generate-ideas creates ideas for clusters
|
||||
└─> Ideas created with status='new'
|
||||
|
||||
2. User bulk-queues ideas to writer
|
||||
└─> Task created in Writer module
|
||||
└─> Ideas status changes to 'queued'
|
||||
|
||||
3. Writer AI (generate-content) creates content
|
||||
└─> Content record created
|
||||
└─> Task.status = 'completed'
|
||||
└─> Ideas.status = 'completed' [Auto-sync from Task]
|
||||
|
||||
4. [Optional] User manually disables idea
|
||||
└─> Ideas.disabled = true
|
||||
└─> Excluded from all processes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Implementation Changes
|
||||
|
||||
### Backend Model Changes
|
||||
|
||||
#### 1. `/backend/igny8_core/business/planning/models.py`
|
||||
|
||||
**Keywords Model:**
|
||||
```python
|
||||
# REMOVE old STATUS_CHOICES
|
||||
# OLD:
|
||||
STATUS_CHOICES = [
|
||||
('active', 'Active'),
|
||||
('pending', 'Pending'),
|
||||
('archived', 'Archived'),
|
||||
]
|
||||
|
||||
# NEW:
|
||||
STATUS_CHOICES = [
|
||||
('new', 'New'),
|
||||
('mapped', 'Mapped'),
|
||||
]
|
||||
status = models.CharField(
|
||||
max_length=50,
|
||||
choices=STATUS_CHOICES,
|
||||
default='new'
|
||||
)
|
||||
|
||||
# ADD new filter field:
|
||||
disabled = models.BooleanField(default=False, help_text="Exclude from processes")
|
||||
```
|
||||
|
||||
**Clusters Model:**
|
||||
```python
|
||||
# ADD STATUS_CHOICES (currently hardcoded as default='active', no choices)
|
||||
# CURRENT: status = models.CharField(max_length=50, default='active')
|
||||
|
||||
# NEW:
|
||||
STATUS_CHOICES = [
|
||||
('new', 'New'),
|
||||
('mapped', 'Mapped'),
|
||||
]
|
||||
status = models.CharField(
|
||||
max_length=50,
|
||||
choices=STATUS_CHOICES,
|
||||
default='new'
|
||||
)
|
||||
|
||||
# ADD new filter field:
|
||||
disabled = models.BooleanField(default=False, help_text="Exclude from processes")
|
||||
```
|
||||
|
||||
**ContentIdeas Model:**
|
||||
```python
|
||||
# UPDATE STATUS_CHOICES
|
||||
# REMOVE 'scheduled', 'published'
|
||||
STATUS_CHOICES = [
|
||||
('new', 'New'),
|
||||
('queued', 'Queued'),
|
||||
('completed', 'Completed'),
|
||||
]
|
||||
status = models.CharField(
|
||||
max_length=50,
|
||||
choices=STATUS_CHOICES,
|
||||
default='new'
|
||||
)
|
||||
|
||||
# ADD new filter field:
|
||||
disabled = models.BooleanField(default=False, help_text="Exclude from processes")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Backend AI Function Updates
|
||||
|
||||
#### 2. `/backend/igny8_core/ai/functions/auto_cluster.py`
|
||||
|
||||
**Location: Line ~297 in save_output()**
|
||||
```python
|
||||
# CURRENT:
|
||||
status='active'
|
||||
|
||||
# NEW:
|
||||
status='mapped' # When keywords are assigned to cluster
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 3. `/backend/igny8_core/ai/functions/generate_ideas.py`
|
||||
|
||||
**Location: After ideas created in save_output()**
|
||||
```python
|
||||
# NEW: Add after creating ideas
|
||||
# Update cluster status from 'new' to 'mapped' after ideas generated
|
||||
for cluster in clusters_used:
|
||||
if cluster.status == 'new':
|
||||
cluster.status = 'mapped'
|
||||
cluster.save()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 4. `/backend/igny8_core/ai/functions/generate_content.py`
|
||||
|
||||
**Location: Line ~318 in save_output()**
|
||||
```python
|
||||
# CURRENT:
|
||||
task.status = 'completed'
|
||||
task.save()
|
||||
|
||||
# NEW: Auto-sync idea status from task
|
||||
task.status = 'completed'
|
||||
task.save()
|
||||
|
||||
# NEW: Update related idea to 'completed'
|
||||
if hasattr(task, 'idea') and task.idea:
|
||||
idea = task.idea
|
||||
idea.status = 'completed'
|
||||
idea.save()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Backend API Updates
|
||||
|
||||
#### 5. `/backend/igny8_core/modules/planner/views.py`
|
||||
|
||||
**Queue ideas to writer (Line ~1084):**
|
||||
```python
|
||||
# CURRENT:
|
||||
idea.status = 'scheduled'
|
||||
|
||||
# NEW:
|
||||
idea.status = 'queued'
|
||||
```
|
||||
|
||||
**CSV import defaults (Line ~563):**
|
||||
```python
|
||||
# CURRENT:
|
||||
status=row.get('status', 'pending') or 'pending'
|
||||
|
||||
# NEW:
|
||||
status=row.get('status', 'new') or 'new'
|
||||
```
|
||||
|
||||
**Filter out disabled items in list views:**
|
||||
```python
|
||||
# NEW: When returning lists, exclude disabled=true (optional, configurable)
|
||||
queryset = queryset.filter(disabled=False) # Or show both with filter option
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Frontend Configuration Updates
|
||||
|
||||
#### 6. `/frontend/src/config/pages/keywords.config.tsx`
|
||||
|
||||
**Table Column - Status Badge (Lines ~230-248):**
|
||||
```typescript
|
||||
// CURRENT:
|
||||
new: 'amber', active: 'green', archived: 'red'
|
||||
|
||||
// NEW:
|
||||
new: 'amber', mapped: 'green'
|
||||
|
||||
// ADD: Display 'disabled' badge if disabled=true
|
||||
```
|
||||
|
||||
**Filter Dropdown (Lines ~310-318):**
|
||||
```typescript
|
||||
// CURRENT:
|
||||
options: ['pending', 'active', 'archived']
|
||||
|
||||
// NEW:
|
||||
options: ['new', 'mapped'],
|
||||
additionalFilters: [
|
||||
{ key: 'disabled', label: 'Show Disabled', type: 'checkbox', default: false }
|
||||
]
|
||||
```
|
||||
|
||||
**Form Field Default (Lines ~560-570):**
|
||||
```typescript
|
||||
// CURRENT:
|
||||
default: 'pending'
|
||||
|
||||
// NEW:
|
||||
default: 'new'
|
||||
|
||||
// ADD: disabled checkbox in form
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 7. `/frontend/src/config/pages/clusters.config.tsx`
|
||||
|
||||
**Table Column - Status Badge (Lines ~190-200):**
|
||||
```typescript
|
||||
// CURRENT:
|
||||
(missing or wrong values)
|
||||
|
||||
// NEW:
|
||||
new: 'amber', mapped: 'green'
|
||||
|
||||
// ADD: Display 'disabled' badge if disabled=true
|
||||
```
|
||||
|
||||
**Filter Dropdown (Lines ~240-253):**
|
||||
```typescript
|
||||
// CURRENT:
|
||||
options: ['new', 'idea', 'mapped']
|
||||
|
||||
// NEW:
|
||||
options: ['new', 'mapped'],
|
||||
additionalFilters: [
|
||||
{ key: 'disabled', label: 'Show Disabled', type: 'checkbox', default: false }
|
||||
]
|
||||
```
|
||||
|
||||
**Form Field (Lines ~405-418):**
|
||||
```typescript
|
||||
// ADD: disabled checkbox in form
|
||||
// Keep status default as 'new'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 8. `/frontend/src/config/pages/ideas.config.tsx`
|
||||
|
||||
**Table Column - Status Badge (Lines ~170-185):**
|
||||
```typescript
|
||||
// CURRENT:
|
||||
new: 'amber', scheduled: 'blue', completed: 'blue', published: 'green'
|
||||
|
||||
// NEW:
|
||||
new: 'amber', queued: 'blue', completed: 'green'
|
||||
|
||||
// ADD: Display 'disabled' badge if disabled=true
|
||||
```
|
||||
|
||||
**Filter Dropdown (Lines ~218-228):**
|
||||
```typescript
|
||||
// CURRENT:
|
||||
options: ['new', 'scheduled', 'completed', 'published']
|
||||
|
||||
// NEW:
|
||||
options: ['new', 'queued', 'completed'],
|
||||
additionalFilters: [
|
||||
{ key: 'disabled', label: 'Show Disabled', type: 'checkbox', default: false }
|
||||
]
|
||||
```
|
||||
|
||||
**Form Field (Lines ~372-385):**
|
||||
```typescript
|
||||
// REMOVE: 'published' option
|
||||
// ADD: disabled checkbox in form
|
||||
// Keep status default as 'new'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Frontend Dashboard
|
||||
|
||||
#### 9. `/frontend/src/pages/Planner/Dashboard.tsx`
|
||||
|
||||
**Status Metrics:**
|
||||
```typescript
|
||||
// Update calculations:
|
||||
// - Keywords: new vs mapped
|
||||
// - Clusters: new vs mapped
|
||||
// - Ideas: new vs queued vs completed
|
||||
|
||||
// Optional: Add disabled count metrics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📍 Change Summary
|
||||
|
||||
### Total Files: 9
|
||||
|
||||
| File | Changes | Type |
|
||||
|------|---------|------|
|
||||
| **models.py** | Add STATUS_CHOICES for Clusters; Update Keywords/Ideas; Add `disabled` field to all 3 | Backend |
|
||||
| **auto_cluster.py** | Line 297: `status='active'` → `status='mapped'` | Backend AI |
|
||||
| **generate_ideas.py** | After ideas created: Set cluster `status='mapped'` | Backend AI |
|
||||
| **generate_content.py** | Line 318: Also sync `idea.status='completed'` from task | Backend AI |
|
||||
| **views.py** | Queue ideas (line 1084): `'scheduled'` → `'queued'`; CSV import: `'pending'` → `'new'` | Backend API |
|
||||
| **keywords.config.tsx** | Update badge/filter/form for new/mapped; Add disabled checkbox | Frontend Config |
|
||||
| **clusters.config.tsx** | Update badge/filter/form for new/mapped; Add disabled checkbox | Frontend Config |
|
||||
| **ideas.config.tsx** | Update badge/filter/form for new/queued/completed; Add disabled checkbox | Frontend Config |
|
||||
| **Dashboard.tsx** | Update metrics calculations for new status values | Frontend Page |
|
||||
|
||||
---
|
||||
|
||||
## ✅ Workflow Validations
|
||||
|
||||
### Keywords
|
||||
```
|
||||
✓ new → mapped (via auto_cluster)
|
||||
✓ Can toggle disabled at any time (no workflow impact)
|
||||
✓ Disabled items excluded from auto-cluster suggestions
|
||||
```
|
||||
|
||||
### Clusters
|
||||
```
|
||||
✓ new → mapped (via generate_ideas)
|
||||
✓ Can toggle disabled at any time (no workflow impact)
|
||||
✓ Disabled items excluded from idea generation
|
||||
```
|
||||
|
||||
### Ideas
|
||||
```
|
||||
✓ new → queued (via bulk_queue_to_writer)
|
||||
✓ queued → completed (via generate_content, tracked via Task.status)
|
||||
✓ Can toggle disabled at any time (no workflow impact)
|
||||
✓ Disabled items excluded from queue suggestions
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🗑️ Data Migration Strategy
|
||||
|
||||
### Django Migration File
|
||||
**File:** `backend/igny8_core/business/planning/migrations/NNNN_unified_status_fields.py`
|
||||
|
||||
```python
|
||||
# Forward migration:
|
||||
# 1. Add disabled=BooleanField(default=False) to all 3 models
|
||||
# 2. Add STATUS_CHOICES to Clusters model
|
||||
# 3. Update Keywords.status data:
|
||||
# - pending → new
|
||||
# - active → mapped
|
||||
# - archived → mapped + set disabled=True
|
||||
# 4. Update Clusters.status data:
|
||||
# - active (with ideas_count > 0) → mapped
|
||||
# - active (with ideas_count = 0) → new
|
||||
# 5. Update ContentIdeas.status data:
|
||||
# - scheduled → queued
|
||||
# - published → completed
|
||||
# - new → new (no change)
|
||||
|
||||
# Reverse migration:
|
||||
# 1. Remove disabled field from all 3 models
|
||||
# 2. Restore old STATUS_CHOICES
|
||||
# 3. Reverse data transformations:
|
||||
# - Keywords: new→pending, mapped→active, disabled=True→archived
|
||||
# - Clusters: Remove STATUS_CHOICES, set all to 'active'
|
||||
# - Ideas: queued→scheduled, completed→published
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing Checklist
|
||||
|
||||
- [ ] Keywords: status='new' by default
|
||||
- [ ] Keywords: auto_cluster sets status='mapped'
|
||||
- [ ] Clusters: status='new' by default
|
||||
- [ ] Clusters: generate_ideas sets status='mapped'
|
||||
- [ ] Ideas: status='new' by default
|
||||
- [ ] Ideas: bulk_queue_to_writer sets status='queued'
|
||||
- [ ] Ideas: generate_content sets status='completed' (via task sync)
|
||||
- [ ] All modules: disabled=true excludes from processes
|
||||
- [ ] All modules: disabled checkbox toggles correctly
|
||||
- [ ] Dashboard: metrics use new status values
|
||||
- [ ] Filters: disabled checkbox hides disabled items by default
|
||||
- [ ] Migration: old data transforms correctly
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Key Differences from Previous Plan
|
||||
|
||||
| Previous | Now Corrected |
|
||||
|----------|---------------|
|
||||
| `disabled` was workflow step | `disabled` is filter status only |
|
||||
| 3 workflows statuses per module | 2 workflow statuses per module |
|
||||
| Published status for Ideas | Completed is final; publish is separate action |
|
||||
| Ideas tracked separately | Ideas auto-sync from Task status |
|
||||
| Unclear disabled behavior | Clear: disabled excluded from all processes |
|
||||
|
||||
---
|
||||
|
||||
**Status:** ✅ Plan Complete & Corrected
|
||||
**Ready for:** Implementation Phase 1 (Backend Models)
|
||||
|
||||
Reference in New Issue
Block a user