salman/igny8
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs

Browse Source
This commit is contained in:
IGNY8 VPS (Salman)
2025-12-04 13:38:54 +00:00
parent ab4724cba4
commit 1fc7d3717d
6 changed files with 3634 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

237
app-packaging-backaup-plan.md Normal file
View File

@@ -0,0 +1,237 @@
bluesalman: gothrough readme and tell me which folders/files inside backeedn adn frontedn folders are acutaly part of igny8 app, and which folders are part of some tech stack which will be enever modifed by app only isntaleld when tech stack isntaleled
GitHub Copilot: I'll create a short plan, then list the backend and frontend directories to inspect their contents.
Created 3 todos
Now I'll list both directories (backend and frontend) to collect the items to classify.
Read [](file:///data/app/igny8/backend)
Read [](file:///data/app/igny8/frontend)
Marking plan steps completed and finishing classification.
Completed: *List backend files* (1/3)
Progress: I scanned backend and frontend and classified items below.
**Backend — Classified**
- **App (igny8 application core):**
- `igny8_core/`: The Django project / app code (settings, apps, views, models, tasks). This is the main application source you will modify.
- `manage.py`: Django management entrypoint — part of the app.
- `migrations/`: Django DB migrations — app schema changes (source-controlled).
- `staticfiles/`: Collected static assets (app assets). May be generated but belongs to the app.
- `requirements.txt`: Dependency manifest for the app (used to install the Python packages the app needs). Not runtime third-party code, but part of the app repo and should be version-controlled.
- **Repo tooling / app-maintenance scripts (part of the repo; maintained by your team):**
- `check_api_response.py`, `check_recent_keywords.py`, `cleanup_structure_categories.py`, `diagnose_generate_content.py`, `inject_test_data.py`, `sync_idea_status.py`, `test_tags_categories.py`, `verify_*.py`, `fix_*` scripts, `final_verify.py`, `deploy_automation.sh`, `example_csv/`: These are maintenance, verification, or migration scripts that are part of the repository and used by developers/ops. They are not third-party code but are repo-level tooling you may edit.
- **Tech stack / infra artifacts (should not be modified by the app runtime; installed or produced by infra):**
- `.venv/`: Virtual environment — environment artifact; not part of app source and typically excluded from commits.
- `Dockerfile`: Container build file — infra/build config (used by ops or CI).
- `backup_postgres_20251120_232816.sql`, `db_backup_20251120_232646.sqlite3`, `rename_fields_migration.sql`: Database backups / SQL dumps — artifacts, not app source.
- `celerybeat-schedule`: Celery scheduler state file — runtime artifact.
- `logs/`: Runtime logs — generated artifacts.
- Any large generated files (e.g., DB dumps) are infra artifacts and shouldn't be edited as source.
Notes:
- Some items like `deploy_automation.sh` or `fix_*` scripts are operational but live in the repo — treat them as repository tooling (editable by you), whereas backups, `.venv`, and docker images/configs are infra artifacts.
**Frontend — Classified**
- **App (igny8 frontend source / code you will modify):**
- `src/`: Application source (React/Vue/Svelte/TSX/etc.) — primary frontend code to edit.
- `public/`, `index.html`, `marketing.html`, `banner.png`: Static assets and HTML entrypoints — part of the app.
- `package.json`, `package-lock.json`: NPM manifests — define packages the frontend needs (repo-managed).
- `eslint.config.js`, `postcss.config.js`, `vite.config.ts`, `vitest.config.ts`, `tsconfig*.json`: Build and tooling config that belong to the frontend source and are edited as part of app development.
- README.md, `DESIGN_SYSTEM.md`, `MIGRATION_GUIDE.md`: Documentation for the frontend — repo docs.
- **Tech stack / infra artifacts (installed/generated by stack — do not edit at runtime):**
- `node_modules/`: Installed dependencies — generated by `npm` / `pnpm` / `yarn`; should not be manually edited and typically ignored by VCS.
- `dist/`: Built production bundle — generated artifact (output of the build).
- `Caddyfile`, `Caddyfile.marketing`, `Dockerfile`, `Dockerfile.dev`, `Dockerfile.marketing`, `Dockerfile.marketing.dev`: Webserver and container build configs — deployment/infra files. They belong to ops/build but do live in the repo; treat as infra configuration, not application logic.
- `LICENSE.md`, other top-level docs: docs, not runtime app code.
Notes:
- Build-tool config files (e.g., `vite.config.ts`, `tsconfig.json`) are part of the app repository and are changed by developers. By contrast, `node_modules` and `dist` are installed/generated by the tech stack and should not be edited directly.
Quick summary (one-line):
- Edit: igny8_core, `manage.py`, migrations, src, public, `package.json`, and similar source/config files — these are part of the igny8 app.
- Don't manually edit: runtime/generated/installed artifacts like `.venv/`, `node_modules/`, `dist/`, DB backups, `celerybeat-schedule`, and use `Dockerfile`/`Caddyfile` only as infra config (ops-managed).
Would you like:
- a) A file added to the repo (e.g., `ARCHITECTURE.md` or `IGNY8-CONTENTS.md`) that labels each top-level item as "app" vs "infra", or
- b) I can produce a JSON/CSV mapping of items for automation (useful for CI), or
- c) Walk through any specific file/folder you want deeper inspection of?
Which option do you prefer?
bluesalman: ok now cehck teh strcutre of /data adn recrussively all its fodlers including igny8/
waht can we do if we want to make igny8 folder as the only full app foodler requried to move to any new location,
and jsut install the portainer as current envirnpmaent and isntall teh app package on nwe locatink build the package and it runs there,
teh curretn setup isnto standrd as it ocntians some node )moduels dist and env,
venv i know can be removed completely
so waht do you think
GitHub Copilot: I'll make a short plan, then list data and the key subfolders (app, igny8, and their important children) so I can analyze what to keep, remove, or package.
Created 4 todos
Read [](file:///data)
Updating the plan: mark the scan step done (noting data access blocked) and keep other steps ready. Then I'll provide analysis, exact commands, Docker and non-Docker packaging options, and a sample `docker-compose.yml` and `install.sh`.
bluesalman: waht happend you are stucl
GitHub Copilot: I'll create a short todo plan, then list data, app, and igny8 to inspect contents before proposing a packaging strategy.
Created 4 todos
Read [](file:///data)
Now I'll run a quick directory size and shallow tree under the workspace `igny8/` so we can see which folders (like `node_modules`, `.venv`, `dist`) are large and where they are.
Ran terminal command: cd /data/app/igny8 && echo 'Top-level sizes:' && du -sh * 2>/dev/null | sort -h && echo && echo 'Directory tree (depth 3):' && find . -maxdepth 3 -type d | sed 's|^\./||' | sort
Updating the todo list: mark the listing steps done and the analysis/packaging step in-progress.
Starting: *Analyze and propose packaging* (4/4)
Progress update: I inspected the workspace `igny8/` (sizes and a shallow tree). Next I analyzed which files to keep/exclude and drafted packaging, cleanup commands, and a sample `docker-compose.yml` plus a `prepare_package.sh` plan. Below are clear findings and actionable steps.
**Findings**
- **Large/generated artifacts present:**
- .venv — virtualenv (can be removed for packaging).
- node_modules, node_modules — installed npm dependencies (do not include in package).
- dist — built frontend assets (can be produced during build; you can include or build on deploy).
- DB dumps and scheduler/state files appear under backend (e.g., `*.sql`, `*.sqlite3`, `celerybeat-schedule`) — these are artifacts and should not be bundled as app source.
- **App source & config (keep these):**
- igny8_core, manage.py, migrations, requirements.txt — Django app and deps manifest.
- src, public, package.json and build config files — frontend source and build config.
- `Dockerfile` / `Dockerfile.*` and `Caddyfile` — infra config that should remain in the repo (useful for containerization).
- **Repo tooling and docs:** `deploy_automation.sh`, `verify_*.py`, docs, README.md — keep in repo; useful for ops/dev.
**Goal**
Make `igny8/` a single, relocatable application folder that you can move to a new host, then deploy via Portainer (or Docker Compose) without shipping generated artifacts like `node_modules`, `.venv`, or large backups.
Recommendations and exact steps
- **Strategy (recommended): Docker image + single `docker-compose.yml` stack**
- Keep only source, config, Dockerfiles and manifests in the app folder.
- Exclude generated artifacts from the package (`.venv`, `node_modules`, `dist`, backups, logs).
- Provide a `docker-compose.yml` that builds the backend and frontend images from the repo and defines services: `db` (Postgres), `redis`, backend (Gunicorn/Django), `worker` (Celery), and frontend (built and served by Caddy or served as static in a small nginx/caddy image).
- Deploy via Portainer by uploading the `docker-compose.yml` (Portainer can build images from contexts if configured), or pre-build images, push to container registry, and use compose with image references.
- **Alternative:** Tarball + install script that runs `docker compose build` and `docker compose up -d` on the new host.
- **Safety note:** Always create a backup or branch of the repo before mass-deleting artifacts.
Exact commands to produce a clean package (run from igny8):
```bash
# make a copy first (safe)
cp -a /data/app/igny8 /tmp/igny8-package-source
# inside the copy, remove generated artifacts
cd /tmp/igny8-package-source
rm -rf backend/.venv
rm -rf backend/*.sql backend/*.sqlite3 backend/celerybeat-schedule backend/logs || true
rm -rf frontend/node_modules sites/node_modules
rm -rf frontend/dist
# optional: remove any large backups in root
rm -f backup_*.sql || true
# create archive
tar -czf /tmp/igny8-clean.tgz .
# verify size
ls -lh /tmp/igny8-clean.tgz
```
Suggested `.dockerignore` (place in repo root) to keep image builds small:
```
backend/.venv
frontend/node_modules
sites/node_modules
frontend/dist
*.sql
*.sqlite3
logs/
*.log
.git
node_modules
```
Sample minimal `docker-compose.yml` (adapt to your app's env vars and Dockerfiles):
```yaml
version: '3.8'
services:
db:
image: postgres:15
environment:
POSTGRES_DB: igny8
POSTGRES_USER: igny8
POSTGRES_PASSWORD: changeme
volumes:
- db_data:/var/lib/postgresql/data
redis:
image: redis:7
command: redis-server --appendonly yes
backend:
build:
context: ./backend
dockerfile: Dockerfile
depends_on:
- db
- redis
environment:
DATABASE_URL: postgres://igny8:changeme@db:5432/igny8
REDIS_URL: redis://redis:6379/0
ports:
- "8000:8000"
volumes:
- static_volume:/app/staticfiles
worker:
build:
context: ./backend
dockerfile: Dockerfile
command: celery -A igny8_core worker --loglevel=info
depends_on:
- redis
- db
frontend:
build:
context: ./frontend
dockerfile: Dockerfile
ports:
- "3000:80"
depends_on:
- backend
volumes:
db_data:
static_volume:
```
Notes:
- The Dockerfile should install Python deps from `requirements.txt`, collect static, run migrations in a start-up script, and run Gunicorn. If not present, we can produce a small `Dockerfile`.
- The Dockerfile should `npm ci && npm run build` and copy `dist/` into a small static-server image (Caddy or nginx). If you prefer to serve frontend via the backend/CDN, build and copy the static files into backend static volume.
What I'd do next (I can do these for you)
- Option A (recommended): Create `docker-compose.yml` and `prepare_package.sh` in the repo (I can add both files), plus `.dockerignore` and a minimal Dockerfile / Dockerfile if missing. Then test `docker compose up --build` locally.
- Option B: Create an archive script only (if you prefer to handle Dockerfiles yourself).
Would you like me to:
- **(1)** Create the `prepare_package.sh`, `.dockerignore`, and a `docker-compose.yml` in the repository and test the compose up here, or
- **(2)** Only produce the files and commands you can run on your target host (no changes to repo), or
- **(3)** Generate a simple Dockerfile and Dockerfile for a fully self-contained build?
Pick an option and I’ll implement it.

753
docs/automation/auto-cluster-validation-fix-plan.md Normal file
View File

@@ -0,0 +1,753 @@
# Auto-Cluster Validation Fix Plan
**Date:** December 4, 2025
**Status:** Design Phase
**Priority:** MEDIUM
---
## 🎯 OBJECTIVE
Add validation to prevent auto-cluster from running with less than 5 keywords, and ensure both manual auto-cluster and automation pipeline use the same shared validation logic to maintain consistency.
---
## 🔍 CURRENT STATE ANALYSIS
### Current Behavior
**Auto-Cluster Function:**
- Located in: `backend/igny8_core/ai/functions/auto_cluster.py`
- No minimum keyword validation
- Accepts any number of keywords (even 1)
- May produce poor quality clusters with insufficient data
**Automation Pipeline:**
- Located in: `backend/igny8_core/business/automation/services/automation_service.py`
- Uses auto-cluster in Stage 1
- No pre-check for minimum keywords
- May waste credits on insufficient data
### Problems
1. ❌ **No Minimum Check:** Auto-cluster runs with 1-4 keywords
2. ❌ **Poor Results:** AI cannot create meaningful clusters with < 5 keywords
3. ❌ **Wasted Credits:** Charges credits for insufficient analysis
4. ❌ **Inconsistent Validation:** No shared validation between manual and automation
5. ❌ **User Confusion:** Error occurs during processing, not at selection
---
## ✅ PROPOSED SOLUTION
### Validation Strategy
**Single Source of Truth:**
- Create one validation function
- Use it in both auto-cluster function AND automation pipeline
- Consistent error messages
- No code duplication
**Error Behavior:**
- **Manual Auto-Cluster:** Return error before API call
- **Automation Pipeline:** Skip Stage 1 with warning in logs
---
## 📋 IMPLEMENTATION PLAN
### Step 1: Create Shared Validation Module
**New File:** `backend/igny8_core/ai/validators/cluster_validators.py`
```python
"""
Cluster-specific validators
Shared between auto-cluster function and automation pipeline
"""
import logging
from typing import Dict, List
logger = logging.getLogger(__name__)
def validate_minimum_keywords(
keyword_ids: List[int],
account=None,
min_required: int = 5
) -> Dict:
"""
Validate that sufficient keywords are available for clustering
Args:
keyword_ids: List of keyword IDs to cluster
account: Account object for filtering
min_required: Minimum number of keywords required (default: 5)
Returns:
Dict with 'valid' (bool) and 'error' (str) or 'count' (int)
"""
from igny8_core.modules.planner.models import Keywords
# Build queryset
queryset = Keywords.objects.filter(id__in=keyword_ids, status='new')
if account:
queryset = queryset.filter(account=account)
# Count available keywords
count = queryset.count()
# Validate minimum
if count < min_required:
return {
'valid': False,
'error': f'Insufficient keywords for clustering. Need at least {min_required} keywords, but only {count} available.',
'count': count,
'required': min_required
}
return {
'valid': True,
'count': count,
'required': min_required
}
def validate_keyword_selection(
selected_ids: List[int],
available_count: int,
min_required: int = 5
) -> Dict:
"""
Validate keyword selection (for frontend validation)
Args:
selected_ids: List of selected keyword IDs
available_count: Total count of available keywords
min_required: Minimum required
Returns:
Dict with validation result
"""
selected_count = len(selected_ids)
# Check if any keywords selected
if selected_count == 0:
return {
'valid': False,
'error': 'No keywords selected',
'type': 'NO_SELECTION'
}
# Check if enough selected
if selected_count < min_required:
return {
'valid': False,
'error': f'Please select at least {min_required} keywords. Currently selected: {selected_count}',
'type': 'INSUFFICIENT_SELECTION',
'selected': selected_count,
'required': min_required
}
# Check if enough available (even if not all selected)
if available_count < min_required:
return {
'valid': False,
'error': f'Not enough keywords available. Need at least {min_required} keywords, but only {available_count} exist.',
'type': 'INSUFFICIENT_AVAILABLE',
'available': available_count,
'required': min_required
}
return {
'valid': True,
'selected': selected_count,
'available': available_count,
'required': min_required
}
```
### Step 2: Update Auto-Cluster Function
**File:** `backend/igny8_core/ai/functions/auto_cluster.py`
**Add import:**
```python
from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
```
**Update validate() method:**
```python
def validate(self, payload: dict, account=None) -> Dict:
"""Validate keyword IDs and minimum count"""
result = super().validate(payload, account)
if not result['valid']:
return result
keyword_ids = payload.get('keyword_ids', [])
if not keyword_ids:
return {'valid': False, 'error': 'No keyword IDs provided'}
# NEW: Validate minimum keywords using shared validator
min_validation = validate_minimum_keywords(
keyword_ids=keyword_ids,
account=account,
min_required=5 # Configurable constant
)
if not min_validation['valid']:
# Log the validation failure
logger.warning(
f"[AutoCluster] Validation failed: {min_validation['error']}"
)
return min_validation
# Log successful validation
logger.info(
f"[AutoCluster] Validation passed: {min_validation['count']} keywords available (min: {min_validation['required']})"
)
return {'valid': True}
```
### Step 3: Update Automation Pipeline
**File:** `backend/igny8_core/business/automation/services/automation_service.py`
**Add import:**
```python
from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
```
**Update run_stage_1() method:**
```python
def run_stage_1(self):
"""Stage 1: Keywords → Clusters (AI)"""
stage_number = 1
stage_name = "Keywords → Clusters (AI)"
start_time = time.time()
# Query pending keywords
pending_keywords = Keywords.objects.filter(
site=self.site,
status='new'
)
total_count = pending_keywords.count()
# NEW: Pre-stage validation for minimum keywords
keyword_ids = list(pending_keywords.values_list('id', flat=True))
min_validation = validate_minimum_keywords(
keyword_ids=keyword_ids,
account=self.account,
min_required=5
)
if not min_validation['valid']:
# Log validation failure
self.logger.log_stage_start(
self.run.run_id, self.account.id, self.site.id,
stage_number, stage_name, total_count
)
error_msg = min_validation['error']
self.logger.log_stage_error(
self.run.run_id, self.account.id, self.site.id,
stage_number, error_msg
)
# Skip stage with proper result
self.run.stage_1_result = {
'keywords_processed': 0,
'clusters_created': 0,
'skipped': True,
'skip_reason': error_msg,
'credits_used': 0
}
self.run.current_stage = 2
self.run.save()
logger.warning(f"[AutomationService] Stage 1 skipped: {error_msg}")
return
# Log stage start
self.logger.log_stage_start(
self.run.run_id, self.account.id, self.site.id,
stage_number, stage_name, total_count
)
# ... rest of existing stage logic ...
```
### Step 4: Update API Endpoint
**File:** `backend/igny8_core/modules/planner/views.py` (KeywordsViewSet)
**Update auto_cluster action:**
```python
@action(detail=False, methods=['post'], url_path='auto_cluster', url_name='auto_cluster')
def auto_cluster(self, request):
"""Auto-cluster keywords using AI"""
from igny8_core.ai.tasks import run_ai_task
from igny8_core.ai.validators.cluster_validators import validate_minimum_keywords
account = getattr(request, 'account', None)
keyword_ids = request.data.get('ids', [])
if not keyword_ids:
return error_response(
error='No keyword IDs provided',
status_code=status.HTTP_400_BAD_REQUEST,
request=request
)
# NEW: Validate minimum keywords BEFORE queuing task
validation = validate_minimum_keywords(
keyword_ids=keyword_ids,
account=account,
min_required=5
)
if not validation['valid']:
return error_response(
error=validation['error'],
status_code=status.HTTP_400_BAD_REQUEST,
request=request,
extra_data={
'count': validation.get('count'),
'required': validation.get('required')
}
)
# Validation passed - proceed with clustering
account_id = account.id if account else None
try:
if hasattr(run_ai_task, 'delay'):
task = run_ai_task.delay(
function_name='auto_cluster',
payload={'keyword_ids': keyword_ids},
account_id=account_id
)
return success_response(
data={'task_id': str(task.id)},
message=f'Auto-cluster started with {validation["count"]} keywords',
request=request
)
else:
# Synchronous fallback
result = run_ai_task(
function_name='auto_cluster',
payload={'keyword_ids': keyword_ids},
account_id=account_id
)
return success_response(data=result, request=request)
except Exception as e:
logger.error(f"Failed to start auto-cluster: {e}", exc_info=True)
return error_response(
error=f'Failed to start clustering: {str(e)}',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
request=request
)
```
### Step 5: Add Frontend Validation (Optional but Recommended)
**File:** `frontend/src/pages/Planner/Keywords.tsx`
**Update handleAutoCluster function:**
```typescript
const handleAutoCluster = async () => {
try {
const selectedIds = selectedKeywords.map(k => k.id);
// Frontend validation (pre-check before API call)
if (selectedIds.length < 5) {
toast.error(
`Please select at least 5 keywords for auto-clustering. Currently selected: ${selectedIds.length}`,
{ duration: 5000 }
);
return;
}
// Check total available
const availableCount = keywords.filter(k => k.status === 'new').length;
if (availableCount < 5) {
toast.error(
`Not enough keywords available. Need at least 5 keywords, but only ${availableCount} exist.`,
{ duration: 5000 }
);
return;
}
// Proceed with API call
const result = await autoClusterKeywords(selectedIds);
if (result.task_id) {
toast.success(`Auto-cluster started with ${selectedIds.length} keywords`);
setTaskId(result.task_id);
} else {
toast.error('Failed to start auto-cluster');
}
} catch (error: any) {
// Backend validation error (in case frontend check was bypassed)
const errorMsg = error.response?.data?.error || error.message;
toast.error(errorMsg);
}
};
```
---
## 🗂️ FILE STRUCTURE
### New Files
```
backend/igny8_core/ai/validators/
├── __init__.py
└── cluster_validators.py (NEW)
```
### Modified Files
```
backend/igny8_core/ai/functions/auto_cluster.py
backend/igny8_core/business/automation/services/automation_service.py
backend/igny8_core/modules/planner/views.py
frontend/src/pages/Planner/Keywords.tsx
```
---
## 🧪 TESTING PLAN
### Unit Tests
**File:** `backend/igny8_core/ai/validators/tests/test_cluster_validators.py`
```python
import pytest
from django.test import TestCase
from igny8_core.ai.validators.cluster_validators import (
validate_minimum_keywords,
validate_keyword_selection
)
from igny8_core.modules.planner.models import Keywords
from igny8_core.auth.models import Account, Site
class ClusterValidatorsTestCase(TestCase):
def setUp(self):
self.account = Account.objects.create(name='Test Account')
self.site = Site.objects.create(name='Test Site', account=self.account)
def test_validate_minimum_keywords_success(self):
"""Test with sufficient keywords (>= 5)"""
# Create 10 keywords
keyword_ids = []
for i in range(10):
kw = Keywords.objects.create(
keyword=f'keyword {i}',
status='new',
account=self.account,
site=self.site
)
keyword_ids.append(kw.id)
result = validate_minimum_keywords(keyword_ids, self.account)
assert result['valid'] is True
assert result['count'] == 10
assert result['required'] == 5
def test_validate_minimum_keywords_failure(self):
"""Test with insufficient keywords (< 5)"""
# Create only 3 keywords
keyword_ids = []
for i in range(3):
kw = Keywords.objects.create(
keyword=f'keyword {i}',
status='new',
account=self.account,
site=self.site
)
keyword_ids.append(kw.id)
result = validate_minimum_keywords(keyword_ids, self.account)
assert result['valid'] is False
assert 'Insufficient keywords' in result['error']
assert result['count'] == 3
assert result['required'] == 5
def test_validate_minimum_keywords_edge_case_exactly_5(self):
"""Test with exactly 5 keywords (boundary)"""
keyword_ids = []
for i in range(5):
kw = Keywords.objects.create(
keyword=f'keyword {i}',
status='new',
account=self.account,
site=self.site
)
keyword_ids.append(kw.id)
result = validate_minimum_keywords(keyword_ids, self.account)
assert result['valid'] is True
assert result['count'] == 5
def test_validate_keyword_selection_insufficient(self):
"""Test frontend selection validation"""
result = validate_keyword_selection(
selected_ids=[1, 2, 3], # Only 3
available_count=10,
min_required=5
)
assert result['valid'] is False
assert result['type'] == 'INSUFFICIENT_SELECTION'
assert result['selected'] == 3
assert result['required'] == 5
```
### Integration Tests
```python
class AutoClusterIntegrationTestCase(TestCase):
def test_auto_cluster_with_insufficient_keywords(self):
"""Test auto-cluster endpoint rejects < 5 keywords"""
# Create only 3 keywords
keyword_ids = self._create_keywords(3)
response = self.client.post(
'/api/planner/keywords/auto_cluster/',
data={'ids': keyword_ids},
HTTP_AUTHORIZATION=f'Bearer {self.token}'
)
assert response.status_code == 400
assert 'Insufficient keywords' in response.json()['error']
def test_automation_skips_stage_1_with_insufficient_keywords(self):
"""Test automation skips Stage 1 if < 5 keywords"""
# Create only 2 keywords
self._create_keywords(2)
# Start automation
run_id = self.automation_service.start_automation('manual')
# Verify Stage 1 was skipped
run = AutomationRun.objects.get(run_id=run_id)
assert run.stage_1_result['skipped'] is True
assert 'Insufficient keywords' in run.stage_1_result['skip_reason']
assert run.current_stage == 2 # Moved to next stage
```
### Manual Test Cases
- [ ] **Test 1:** Try auto-cluster with 0 keywords selected
- Expected: Error message "No keywords selected"
- [ ] **Test 2:** Try auto-cluster with 3 keywords selected
- Expected: Error message "Please select at least 5 keywords. Currently selected: 3"
- [ ] **Test 3:** Try auto-cluster with exactly 5 keywords
- Expected: Success, clustering starts
- [ ] **Test 4:** Run automation with 2 keywords in site
- Expected: Stage 1 skipped with warning in logs
- [ ] **Test 5:** Run automation with 10 keywords in site
- Expected: Stage 1 runs normally
---
## 📊 ERROR MESSAGES
### Frontend (User-Facing)
**No Selection:**
```
❌ No keywords selected
Please select keywords to cluster.
```
**Insufficient Selection:**
```
❌ Please select at least 5 keywords for auto-clustering
Currently selected: 3 keywords
You need at least 5 keywords to create meaningful clusters.
```
**Insufficient Available:**
```
❌ Not enough keywords available
Need at least 5 keywords, but only 2 exist.
Add more keywords before running auto-cluster.
```
### Backend (Logs)
**Validation Failed:**
```
[AutoCluster] Validation failed: Insufficient keywords for clustering. Need at least 5 keywords, but only 3 available.
```
**Validation Passed:**
```
[AutoCluster] Validation passed: 15 keywords available (min: 5)
```
**Automation Stage Skipped:**
```
[AutomationService] Stage 1 skipped: Insufficient keywords for clustering. Need at least 5 keywords, but only 2 available.
```
---
## 🎯 CONFIGURATION
### Constants File
**File:** `backend/igny8_core/ai/constants.py` (or create if doesn't exist)
```python
"""
AI Function Configuration Constants
"""
# Cluster Configuration
MIN_KEYWORDS_FOR_CLUSTERING = 5 # Minimum keywords needed for meaningful clusters
OPTIMAL_KEYWORDS_FOR_CLUSTERING = 20 # Recommended for best results
# Other AI limits...
```
**Usage in validators:**
```python
from igny8_core.ai.constants import MIN_KEYWORDS_FOR_CLUSTERING
def validate_minimum_keywords(keyword_ids, account=None):
min_required = MIN_KEYWORDS_FOR_CLUSTERING
# ... validation logic
```
---
## 🔄 SHARED VALIDATION PATTERN
### Why This Approach Works
**✅ Single Source of Truth:**
- One function: `validate_minimum_keywords()`
- Used by both auto-cluster function and automation
- Update in one place applies everywhere
**✅ Consistent Behavior:**
- Same error messages
- Same validation logic
- Same minimum requirements
**✅ Easy to Maintain:**
- Want to change minimum from 5 to 10? Change one constant
- Want to add new validation? Add to one function
- Want to test? Test one module
**✅ No Code Duplication:**
- DRY principle followed
- Reduces bugs from inconsistency
- Easier code review
### Pattern for Future Validators
```python
# backend/igny8_core/ai/validators/content_validators.py
def validate_minimum_content_length(content_text: str, min_words: int = 100):
"""
Shared validator for content minimum length
Used by: GenerateContentFunction, Automation Stage 4, Content creation
"""
word_count = len(content_text.split())
if word_count < min_words:
return {
'valid': False,
'error': f'Content too short. Minimum {min_words} words required, got {word_count}.'
}
return {'valid': True, 'word_count': word_count}
```
---
## 🚀 IMPLEMENTATION STEPS
### Phase 1: Create Validator (Day 1)
- [ ] Create `cluster_validators.py`
- [ ] Implement `validate_minimum_keywords()`
- [ ] Implement `validate_keyword_selection()`
- [ ] Write unit tests
### Phase 2: Integrate Backend (Day 1)
- [ ] Update `AutoClusterFunction.validate()`
- [ ] Update `AutomationService.run_stage_1()`
- [ ] Update `KeywordsViewSet.auto_cluster()`
- [ ] Write integration tests
### Phase 3: Frontend (Day 2)
- [ ] Add frontend validation in Keywords page
- [ ] Add user-friendly error messages
- [ ] Test error scenarios
### Phase 4: Testing & Deployment (Day 2)
- [ ] Run all tests
- [ ] Manual QA testing
- [ ] Deploy to production
- [ ] Monitor first few auto-cluster runs
---
## 🎯 SUCCESS CRITERIA
✅ Auto-cluster returns error if < 5 keywords selected
✅ Automation skips Stage 1 if < 5 keywords available
✅ Both use same validation function (no duplication)
✅ Clear error messages guide users
✅ Frontend validation provides instant feedback
✅ Backend validation catches edge cases
✅ All tests pass
✅ No regression in existing functionality
---
## 📈 FUTURE ENHANCEMENTS
### V2 Features
1. **Configurable Minimum:**
- Allow admin to set minimum via settings
- Default: 5, Range: 3-20
2. **Quality Scoring:**
- Show quality indicator based on keyword count
- 5-10: "Fair", 11-20: "Good", 21+: "Excellent"
3. **Smart Recommendations:**
- "You have 4 keywords. Add 1 more for best results"
- "15 keywords selected. Good for clustering!"
4. **Batch Size Validation:**
- Warn if too many keywords selected (> 100)
- Suggest splitting into multiple runs
---
## END OF PLAN
This plan ensures robust, consistent validation for auto-cluster across all entry points (manual and automation) using shared, well-tested validation logic.

725
docs/automation/automation-progress-ux-improvement-plan.md Normal file
View File

@@ -0,0 +1,725 @@
# Automation Progress UX Improvement Plan
**Date:** December 4, 2025
**Status:** Design Phase
**Priority:** MEDIUM
---
## 🎯 OBJECTIVE
Improve the automation progress tracking UX to show **real-time processing status** for currently processing items, making it easier for users to understand what's happening during automation runs.
---
## 🔍 CURRENT STATE ANALYSIS
### Current Behavior
**What Users See Now:**
1. A "Current State" card that shows the stage being processed
2. Stage number and status (e.g., "Stage 3: Ideas → Tasks")
3. **BUT:** No visibility into which specific records are being processed
4. **Problem:** User only knows when a full stage completes
**Example Current Experience:**
```
┌─────────────────────────────────────┐
│ Current State: Stage 2 │
│ Clusters → Ideas (AI) │
│ │
│ Status: Processing │
└─────────────────────────────────────┘
[User waits... no updates until stage completes]
```
### User Pain Points
1. ❌ **No Record-Level Progress:** Can't see which keywords/ideas/content are being processed
2. ❌ **No Queue Visibility:** Don't know what's coming up next
3. ❌ **No Item Count Progress:** "Processing 15 of 50 keywords..." is missing
4. ❌ **Card Position:** Current state card is at bottom, requires scrolling
5. ❌ **No Percentage Progress:** Just a spinner, no quantitative feedback
---
## ✅ PROPOSED SOLUTION
### New Design Concept
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ 🔄 AUTOMATION IN PROGRESS │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67% │
│ │
│ Stage 2: Clusters → Ideas (AI) │
│Column 1 │
│ Currently Processing: │
│ • "Best SEO tools for small business" (Cluster #42) │
│ Column 2 │
│ Up Next: │
│ • "Content marketing automation platforms" │
│ • "AI-powered content creation tools" │
│ Sinngle row centered │
│ Progress: 34/50 clusters processed │
└─────────────────────────────────────────────────────────────────────────────┘
[STAGES SECTION BELOW - All 7 stages in grid view]
```
---
## 📐 DETAILED DESIGN SPECIFICATIONS
### 1. Card Repositioning
**Move from:** Bottom of page (after stages)
**Move to:** Top of page (above stages section)
**Layout:** Max width 1200px horizontal card
**Visibility:** Only shown when `currentRun?.status === 'running'`
### 2. Card Structure
#### Header Section
- **Left:** Large stage number icon (animated pulse)
- **Center:** Stage name + type badge (AI/Local/Manual)
- **Right:** Percentage complete (calculated from processed/total)
#### Progress Bar
- **Type:** Animated linear progress bar
- **Colors:**
- Blue for active stage
- Green for completed
- Gray for pending
- **Updates:** Refresh every 3-5 seconds via polling
#### Currently Processing Section
- **For Keywords Stage:**
```
Currently Processing:
• "keyword 1"
• "keyword 2"
• "keyword 3"
+ 47 more in queue
```
- **For Ideas Stage:**
```
Currently Processing:
• "10 Ways to Improve SEO Rankings"
Up Next:
• "Content Marketing Best Practices 2025"
• "AI Tools for Content Writers"
```
- **For Content Stage:**
```
Currently Processing:
• "How to Use ChatGPT for Content Creation" (2,500 words)
Up Next:
• "Best AI Image Generators in 2025"
```
#### Record Counter
```
Progress: [current]/[total] [items] processed
Example: Progress: 15/50 keywords processed
```
### 3. Refresh Strategy
**Polling Approach:**
```typescript
// Poll every 3 seconds while automation is running
useEffect(() => {
if (currentRun?.status === 'running') {
const interval = setInterval(() => {
// Refresh ONLY the current processing data
fetchCurrentProcessingState();
}, 3000);
return () => clearInterval(interval);
}
}, [currentRun]);
```
**Partial Refresh:**
- Only refresh the "Currently Processing" component
- Don't reload entire page
- Don't re-fetch stage cards
- Smooth transition (no flickering)
---
## 🗄️ BACKEND CHANGES REQUIRED
### New API Endpoint
**URL:** `GET /api/automation/current_processing/`
**Params:** `?site_id={id}&run_id={run_id}`
**Response Format:**
```json
{
"run_id": "abc123",
"current_stage": 2,
"stage_name": "Clusters → Ideas",
"stage_type": "AI",
"total_items": 50,
"processed_items": 34,
"percentage": 68,
"currently_processing": [
{
"id": 42,
"title": "Best SEO tools for small business",
"type": "cluster"
}
],
"up_next": [
{
"id": 43,
"title": "Content marketing automation platforms",
"type": "cluster"
},
{
"id": 44,
"title": "AI-powered content creation tools",
"type": "cluster"
}
],
"remaining_count": 16
}
```
### Implementation in AutomationService
**File:** `backend/igny8_core/business/automation/services/automation_service.py`
**Add method:**
```python
def get_current_processing_state(self) -> dict:
"""
Get real-time processing state for current automation run
"""
if not self.run or self.run.status != 'running':
return None
stage = self.run.current_stage
# Get stage-specific data
if stage == 1: # Keywords → Clusters
queue = Keywords.objects.filter(
site=self.site, status='new'
).order_by('id')
return {
'stage_number': 1,
'stage_name': 'Keywords → Clusters',
'stage_type': 'AI',
'total_items': queue.count() + self._get_processed_count(stage),
'processed_items': self._get_processed_count(stage),
'currently_processing': self._get_current_items(queue, 3),
'up_next': self._get_next_items(queue, 2, skip=3),
}
elif stage == 2: # Clusters → Ideas
queue = Clusters.objects.filter(
site=self.site, status='new', disabled=False
).order_by('id')
return {
'stage_number': 2,
'stage_name': 'Clusters → Ideas',
'stage_type': 'AI',
'total_items': queue.count() + self._get_processed_count(stage),
'processed_items': self._get_processed_count(stage),
'currently_processing': self._get_current_items(queue, 1),
'up_next': self._get_next_items(queue, 2, skip=1),
}
# ... similar for stages 3-6
def _get_processed_count(self, stage: int) -> int:
"""Get count of items processed in current stage"""
result_key = f'stage_{stage}_result'
result = getattr(self.run, result_key, {})
# Extract appropriate count from result
if stage == 1:
return result.get('keywords_processed', 0)
elif stage == 2:
return result.get('clusters_processed', 0)
# ... etc
def _get_current_items(self, queryset, count: int) -> list:
"""Get currently processing items"""
items = queryset[:count]
return [
{
'id': item.id,
'title': getattr(item, 'keyword', None) or
getattr(item, 'cluster_name', None) or
getattr(item, 'idea_title', None) or
getattr(item, 'title', None),
'type': queryset.model.__name__.lower()
}
for item in items
]
```
### Add View in AutomationViewSet
**File:** `backend/igny8_core/business/automation/views.py`
```python
@action(detail=False, methods=['get'], url_path='current_processing')
def current_processing(self, request):
"""Get current processing state for active automation run"""
site_id = request.GET.get('site_id')
run_id = request.GET.get('run_id')
if not site_id or not run_id:
return error_response(
error='site_id and run_id required',
status_code=400,
request=request
)
try:
run = AutomationRun.objects.get(run_id=run_id, site_id=site_id)
if run.status != 'running':
return success_response(data=None, request=request)
service = AutomationService.from_run_id(run_id)
state = service.get_current_processing_state()
return success_response(data=state, request=request)
except AutomationRun.DoesNotExist:
return error_response(
error='Run not found',
status_code=404,
request=request
)
```
---
## 🎨 FRONTEND CHANGES REQUIRED
### 1. New Component: CurrentProcessingCard
**File:** `frontend/src/components/Automation/CurrentProcessingCard.tsx`
```typescript
interface CurrentProcessingCardProps {
runId: string;
siteId: number;
currentStage: number;
onComplete?: () => void;
}
const CurrentProcessingCard: React.FC<CurrentProcessingCardProps> = ({
runId,
siteId,
currentStage,
onComplete
}) => {
const [processingState, setProcessingState] = useState<ProcessingState | null>(null);
// Poll every 3 seconds
useEffect(() => {
const fetchState = async () => {
const state = await automationService.getCurrentProcessing(siteId, runId);
setProcessingState(state);
// If stage completed, trigger refresh
if (state && state.processed_items === state.total_items) {
onComplete?.();
}
};
fetchState();
const interval = setInterval(fetchState, 3000);
return () => clearInterval(interval);
}, [siteId, runId]);
if (!processingState) return null;
const percentage = Math.round(
(processingState.processed_items / processingState.total_items) * 100
);
return (
<div className="bg-blue-50 dark:bg-blue-900/20 border-2 border-blue-500 rounded-lg p-6 mb-6">
{/* Header */}
<div className="flex items-center justify-between mb-4">
<div className="flex items-center gap-3">
<div className="animate-pulse">
<BoltIcon className="w-8 h-8 text-blue-600" />
</div>
<div>
<h2 className="text-2xl font-bold text-gray-900 dark:text-white">
Automation In Progress
</h2>
<p className="text-sm text-gray-600 dark:text-gray-400">
Stage {currentStage}: {processingState.stage_name}
<span className="ml-2 px-2 py-0.5 bg-blue-100 dark:bg-blue-900 text-blue-700 dark:text-blue-300 rounded text-xs">
{processingState.stage_type}
</span>
</p>
</div>
</div>
<div className="text-right">
<div className="text-4xl font-bold text-blue-600">{percentage}%</div>
<div className="text-sm text-gray-600 dark:text-gray-400">
{processingState.processed_items}/{processingState.total_items} processed
</div>
</div>
</div>
{/* Progress Bar */}
<div className="mb-6">
<div className="w-full bg-gray-200 dark:bg-gray-700 rounded-full h-3">
<div
className="bg-blue-600 h-3 rounded-full transition-all duration-500"
style={{ width: `${percentage}%` }}
/>
</div>
</div>
{/* Currently Processing */}
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
<div>
<h3 className="text-sm font-semibold text-gray-700 dark:text-gray-300 mb-2">
Currently Processing:
</h3>
<div className="space-y-1">
{processingState.currently_processing.map((item, idx) => (
<div key={idx} className="flex items-start gap-2 text-sm">
<span className="text-blue-600 mt-1">•</span>
<span className="text-gray-800 dark:text-gray-200 font-medium">
{item.title}
</span>
</div>
))}
</div>
</div>
<div>
<h3 className="text-sm font-semibold text-gray-700 dark:text-gray-300 mb-2">
Up Next:
</h3>
<div className="space-y-1">
{processingState.up_next.map((item, idx) => (
<div key={idx} className="flex items-start gap-2 text-sm">
<span className="text-gray-400 mt-1">•</span>
<span className="text-gray-600 dark:text-gray-400">
{item.title}
</span>
</div>
))}
{processingState.remaining_count > processingState.up_next.length && (
<div className="text-xs text-gray-500 mt-2">
+ {processingState.remaining_count - processingState.up_next.length} more in queue
</div>
)}
</div>
</div>
</div>
</div>
);
};
```
### 2. Update AutomationPage.tsx
**File:** `frontend/src/pages/Automation/AutomationPage.tsx`
```typescript
// Add new import
import CurrentProcessingCard from '../../components/Automation/CurrentProcessingCard';
// In the component
return (
<div className="p-6">
<PageMeta title="Automation" description="AI automation pipeline" />
{/* Current Processing Card - MOVE TO TOP */}
{currentRun?.status === 'running' && (
<CurrentProcessingCard
runId={currentRun.run_id}
siteId={selectedSite.id}
currentStage={currentRun.current_stage}
onComplete={() => {
// Refresh full page metrics when stage completes
loadAutomationData();
}}
/>
)}
{/* Metrics Cards */}
<div className="grid grid-cols-1 md:grid-cols-3 gap-6 mb-6">
{/* ... existing metrics ... */}
</div>
{/* Stages Section */}
<ComponentCard>
<h2 className="text-xl font-semibold mb-4">Pipeline Stages</h2>
{/* ... existing stages ... */}
</ComponentCard>
{/* Rest of the page ... */}
</div>
);
```
### 3. Add Service Method
**File:** `frontend/src/services/automationService.ts`
```typescript
export interface ProcessingState {
run_id: string;
current_stage: number;
stage_name: string;
stage_type: 'AI' | 'Local' | 'Manual';
total_items: number;
processed_items: number;
percentage: number;
currently_processing: Array<{
id: number;
title: string;
type: string;
}>;
up_next: Array<{
id: number;
title: string;
type: string;
}>;
remaining_count: number;
}
// Add to automationService
getCurrentProcessing: async (
siteId: number,
runId: string
): Promise<ProcessingState | null> => {
return fetchAPI(
buildUrl('/current_processing/', { site_id: siteId, run_id: runId })
);
},
```
---
## 🧪 TESTING PLAN
### Unit Tests
- [ ] Test `get_current_processing_state()` for each stage
- [ ] Test `_get_processed_count()` calculation
- [ ] Test `_get_current_items()` formatting
- [ ] Test API endpoint with various run states
### Integration Tests
- [ ] Test polling updates every 3 seconds
- [ ] Test stage completion triggers full refresh
- [ ] Test card disappears when automation completes
- [ ] Test with 0 items (edge case)
- [ ] Test with 1000+ items (performance)
### Visual/UX Tests
- [ ] Card positioned at top of page
- [ ] Progress bar animates smoothly
- [ ] Record names display correctly
- [ ] Responsive design (mobile/tablet/desktop)
- [ ] Dark mode support
- [ ] Loading states
- [ ] Error states
---
## 📊 STAGE-SPECIFIC DISPLAY FORMATS
### Stage 1: Keywords → Clusters
```
Currently Processing:
• "best seo tools"
• "content marketing platforms"
• "ai writing assistants"
+ 47 more keywords in queue
Progress: 3/50 keywords processed
```
### Stage 2: Clusters → Ideas
```
Currently Processing:
• "SEO Tools and Software" (Cluster #12)
Up Next:
• "Content Marketing Strategies"
• "AI Content Generation"
Progress: 12/25 clusters processed
```
### Stage 3: Ideas → Tasks
```
Currently Processing:
• "10 Best SEO Tools for 2025"
Up Next:
• "How to Create Content with AI"
• "Content Marketing ROI Calculator"
Progress: 8/30 ideas processed
```
### Stage 4: Tasks → Content
```
Currently Processing:
• "Ultimate Guide to SEO in 2025" (2,500 words)
Up Next:
• "AI Content Creation Best Practices"
Progress: 5/15 tasks processed
```
### Stage 5: Content → Image Prompts
```
Currently Processing:
• "How to Use ChatGPT for Content" (Extracting 3 image prompts)
Up Next:
• "Best AI Image Generators 2025"
Progress: 10/15 content pieces processed
```
### Stage 6: Image Prompts → Images
```
Currently Processing:
• Featured image for "SEO Guide 2025"
Up Next:
• In-article image #1 for "SEO Guide 2025"
• In-article image #2 for "SEO Guide 2025"
Progress: 15/45 images generated
```
### Stage 7: Manual Review Gate
```
Automation Complete! ✅
Ready for Review:
• "Ultimate Guide to SEO in 2025"
• "AI Content Creation Best Practices"
• "Best Image Generators 2025"
+ 12 more content pieces
Total: 15 content pieces ready for review
```
---
## 🎯 SUCCESS METRICS
### User Experience
✅ Users can see **exactly what's being processed** at any moment
✅ Users know **what's coming up next** in the queue
✅ Users can estimate **remaining time** based on progress
✅ Users get **quantitative feedback** (percentage, counts)
✅ Users see **smooth, non-disruptive updates** (no page flicker)
### Technical
✅ Polling interval: 3 seconds (balance between freshness and load)
✅ API response time: < 200ms
✅ Component re-render: Only the processing card, not entire page
✅ Memory usage: No memory leaks from polling
✅ Error handling: Graceful degradation if API fails
---
## 🚀 IMPLEMENTATION PHASES
### Phase 1: Backend (1-2 days)
- [ ] Implement `get_current_processing_state()` method
- [ ] Add `/current_processing/` API endpoint
- [ ] Test with all 7 stages
- [ ] Add unit tests
### Phase 2: Frontend (2-3 days)
- [ ] Create `CurrentProcessingCard` component
- [ ] Add polling logic with cleanup
- [ ] Style with Tailwind (match existing design system)
- [ ] Add dark mode support
- [ ] Integrate into `AutomationPage`
### Phase 3: Testing & Refinement (1-2 days)
- [ ] Integration testing
- [ ] Performance testing
- [ ] UX testing
- [ ] Bug fixes
### Phase 4: Deployment
- [ ] Deploy backend changes
- [ ] Deploy frontend changes
- [ ] Monitor first automation runs
- [ ] Collect user feedback
---
## 🔄 FUTURE ENHANCEMENTS
### V2 Features (Post-MVP)
1. **Estimated Time Remaining:**
```
Progress: 15/50 keywords processed
Estimated time remaining: ~8 minutes
```
2. **Stage-Level Progress Bar:**
- Each stage shows its own mini progress bar
- Visual indicator of which stages are complete
3. **Click to View Details:**
- Click on a record name to see modal with details
- Preview generated content/images
4. **Pause/Resume from Card:**
- Add pause button directly in the card
- Quick action without scrolling
5. **Export Processing Log:**
- Download real-time processing log
- CSV of all processed items with timestamps
---
## END OF PLAN
This plan provides a comprehensive UX improvement for automation progress tracking, making the process transparent and user-friendly while maintaining system performance.

403
docs/automation/automation-stage-6-image-generation-fix.md Normal file
View File

@@ -0,0 +1,403 @@
# Automation Stage 6 - Image Generation Fix Plan
**Date:** December 4, 2025
**Status:** Analysis Complete - Implementation Required
**Priority:** HIGH
---
## 🔍 PROBLEM IDENTIFICATION
### Current Issue
Stage 6 of the automation pipeline (Image Prompts → Generated Images) is **NOT running correctly**. The issue stems from using the wrong AI function for image generation.
### Root Cause Analysis
**Current Implementation (INCORRECT):**
```python
# File: backend/igny8_core/business/automation/services/automation_service.py
# Line ~935
engine = AIEngine(account=self.account)
result = engine.execute(
fn=GenerateImagesFunction(),
payload={'image_ids': [image.id]} # ❌ WRONG
)
```
**Why It Fails:**
1. `GenerateImagesFunction()` expects:
- Input: `{'ids': [task_ids]}` (Task IDs, NOT Image IDs)
- Purpose: Extract prompts from Tasks and generate images for tasks
- Use case: When you have Tasks with content but no images
2. Automation Stage 6 has:
- Input: Images records with `status='pending'` (already have prompts)
- Purpose: Generate actual image URLs from existing prompts
- Context: Images were created in Stage 5 by `GenerateImagePromptsFunction`
### How Other Stages Work Correctly
**Stage 1:** Keywords → Clusters
```python
engine.execute(
fn=AutoClusterFunction(),
payload={'keyword_ids': keyword_ids} # ✅ Correct
)
```
**Stage 2:** Clusters → Ideas
```python
engine.execute(
fn=GenerateIdeasFunction(),
payload={'cluster_ids': cluster_ids} # ✅ Correct
)
```
**Stage 4:** Tasks → Content
```python
engine.execute(
fn=GenerateContentFunction(),
payload={'ids': task_ids} # ✅ Correct
)
```
**Stage 5:** Content → Image Prompts
```python
engine.execute(
fn=GenerateImagePromptsFunction(),
payload={'ids': content_ids} # ✅ Correct
)
```
**Stage 6:** Image Prompts → Images (BROKEN)
```python
# Currently uses GenerateImagesFunction (WRONG)
# Should use process_image_generation_queue (CORRECT)
```
---
## ✅ CORRECT SOLUTION
### The Right Approach
**Use `process_image_generation_queue` Celery task** - This is the same approach used by:
1. Writer/Images page (`/writer/images/generate_images/` endpoint)
2. Manual image generation from prompts
**Evidence from Working Code:**
```python
# File: backend/igny8_core/modules/writer/views.py
# ImagesViewSet.generate_images()
from igny8_core.ai.tasks import process_image_generation_queue
task = process_image_generation_queue.delay(
image_ids=image_ids, # ✅ Accepts image_ids
account_id=account_id,
content_id=content_id
)
```
**What `process_image_generation_queue` Does:**
1. ✅ Accepts `image_ids` (list of Image record IDs)
2. ✅ Each Image record already has a `prompt` field (populated by Stage 5)
3. ✅ Generates images sequentially with progress tracking
4. ✅ Updates Images records: `status='pending'` → `status='generated'`
5. ✅ Downloads and saves images locally
6. ✅ Automatically handles credits deduction
7. ✅ Supports multiple providers (OpenAI, Runware)
8. ✅ Handles errors gracefully (continues on failure)
---
## 📋 IMPLEMENTATION PLAN
### Changes Required
**File:** `backend/igny8_core/business/automation/services/automation_service.py`
**Location:** `run_stage_6()` method (lines 874-1022)
### Step 1: Import the Correct Task
**Current:**
```python
from igny8_core.ai.functions.generate_images import GenerateImagesFunction
```
**Add:**
```python
from igny8_core.ai.tasks import process_image_generation_queue
```
### Step 2: Modify Stage 6 Logic
**Replace this block (lines ~920-945):**
```python
# INCORRECT - Delete this
for idx, image in enumerate(image_list, 1):
try:
content_title = image.content.title if image.content else 'Unknown'
self.logger.log_stage_progress(
self.run.run_id, self.account.id, self.site.id,
stage_number, f"Generating image {idx}/{total_images}: {image.image_type} for '{content_title}'"
)
# Call AI function via AIEngine
engine = AIEngine(account=self.account)
result = engine.execute(
fn=GenerateImagesFunction(),
payload={'image_ids': [image.id]} # ❌ WRONG
)
# Monitor task
task_id = result.get('task_id')
if task_id:
self._wait_for_task(task_id, stage_number, f"Image for '{content_title}'", continue_on_error=True)
images_processed += 1
```
**With this:**
```python
# CORRECT - Use process_image_generation_queue
for idx, image in enumerate(image_list, 1):
try:
content_title = image.content.title if image.content else 'Unknown'
self.logger.log_stage_progress(
self.run.run_id, self.account.id, self.site.id,
stage_number, f"Generating image {idx}/{total_images}: {image.image_type} for '{content_title}'"
)
# Call process_image_generation_queue directly (same as Writer/Images page)
from igny8_core.ai.tasks import process_image_generation_queue
# Queue the task
if hasattr(process_image_generation_queue, 'delay'):
task = process_image_generation_queue.delay(
image_ids=[image.id],
account_id=self.account.id,
content_id=image.content.id if image.content else None
)
task_id = str(task.id)
else:
# Fallback for testing (synchronous)
result = process_image_generation_queue(
image_ids=[image.id],
account_id=self.account.id,
content_id=image.content.id if image.content else None
)
task_id = None
# Monitor task (if async)
if task_id:
self._wait_for_task(task_id, stage_number, f"Image for '{content_title}'", continue_on_error=True)
images_processed += 1
```
### Step 3: Update Logging
The logging structure remains the same, just update the log messages to reflect the correct process:
```python
self.logger.log_stage_progress(
self.run.run_id, self.account.id, self.site.id,
stage_number, f"Image generation task queued for '{content_title}' ({images_processed}/{total_images})"
)
```
### Step 4: No Changes Needed For
✅ Stage 5 (Image Prompt Extraction) - Already correct
✅ Images table structure - Already has all required fields
✅ Progress tracking - Already implemented in `process_image_generation_queue`
✅ Credits deduction - Automatic in `process_image_generation_queue`
✅ Error handling - Built into the task with `continue_on_error=True`
---
## 🔄 HOW IT WORKS (CORRECTED FLOW)
### Stage 5: Content → Image Prompts
```
Input: Content (status='draft', no images)
AI: GenerateImagePromptsFunction
Output: Images (status='pending', prompt='...')
```
### Stage 6: Image Prompts → Generated Images (FIXED)
```
Input: Images (status='pending', has prompt)
Task: process_image_generation_queue (Celery task)
AI: Calls OpenAI/Runware API with prompt
Output: Images (status='generated', image_url='https://...', image_path='/path/to/file')
```
### What Happens in process_image_generation_queue
1. **Load Image Record:**
- Get Image by ID
- Read existing `prompt` field (created in Stage 5)
- Get Content for template formatting
2. **Format Prompt:**
- Use image_prompt_template from PromptRegistry
- Format: `"Create a {image_type} image for '{post_title}'. Prompt: {image_prompt}"`
- Handle model-specific limits (DALL-E 3: 4000 chars, DALL-E 2: 1000 chars)
3. **Generate Image:**
- Call `AICore.generate_image()`
- Uses configured provider (OpenAI/Runware)
- Uses configured model (dall-e-3, runware:97@1, etc.)
- Respects image size settings
4. **Download & Save:**
- Download image from URL
- Save to `/data/app/igny8/frontend/public/images/ai-images/`
- Update Image record with both `image_url` and `image_path`
5. **Update Status:**
- `status='pending'` → `status='generated'`
- Triggers automatic Content status update (if all images generated)
6. **Deduct Credits:**
- Automatic via `AICore` credit system
- Records in `AIUsageLog`
---
## 🧪 TESTING CHECKLIST
### Pre-Deployment Tests
- [ ] **Unit Test:** Verify `process_image_generation_queue` works with single image
- [ ] **Integration Test:** Run Stage 6 with 3-5 pending images
- [ ] **Error Handling:** Test with invalid image ID
- [ ] **Credits:** Verify credits are deducted correctly
- [ ] **Multi-Provider:** Test with both OpenAI and Runware
### Post-Deployment Validation
- [ ] **Full Pipeline:** Run Automation from Stage 1 → Stage 7
- [ ] **Verify Stage 5 Output:** Images created with `status='pending'` and prompts
- [ ] **Verify Stage 6 Output:** Images updated to `status='generated'` with URLs
- [ ] **Check Downloads:** Images saved to `/data/app/igny8/frontend/public/images/ai-images/`
- [ ] **Monitor Logs:** Review automation logs for Stage 6 completion
- [ ] **Credits Report:** Confirm Stage 6 credits recorded in automation results
### Success Criteria
✅ Stage 6 completes without errors
✅ All pending images get generated
✅ Images are downloaded and accessible
✅ Content status automatically updates when all images generated
✅ Credits are properly deducted and logged
✅ Automation proceeds to Stage 7 (Manual Review Gate)
---
## 📊 COMPARISON: BEFORE vs AFTER
### BEFORE (Broken)
```python
# ❌ WRONG APPROACH
GenerateImagesFunction()
- Expects: task_ids
- Purpose: Extract prompts from Tasks
- Problem: Doesn't work with Images that already have prompts
```
**Result:** Stage 6 fails, images never generated
### AFTER (Fixed)
```python
# ✅ CORRECT APPROACH
process_image_generation_queue()
- Accepts: image_ids
- Purpose: Generate images from existing prompts
- Works with: Images (status='pending' with prompts)
```
**Result:** Stage 6 succeeds, images generated sequentially with progress tracking
---
## 🔒 SAFETY & ROLLBACK
### Backup Plan
If the fix causes issues:
1. **Rollback Code:**
- Git revert the automation_service.py changes
- Automation still works for Stages 1-5
2. **Manual Workaround:**
- Users can manually generate images from Writer/Images page
- This uses the same `process_image_generation_queue` task
3. **No Data Loss:**
- Stage 5 already created Images with prompts
- These remain in database and can be processed anytime
---
## 📝 IMPLEMENTATION STEPS
1. **Update Code:** Modify `run_stage_6()` as documented above
2. **Test Locally:** Run automation with test data
3. **Code Review:** Verify changes match working Writer/Images implementation
4. **Deploy:** Push to production
5. **Monitor:** Watch first automation run for Stage 6 completion
6. **Validate:** Check images generated and credits deducted
---
## 🎯 EXPECTED OUTCOME
After implementing this fix:
✅ **Stage 6 will work correctly** - Images generate from prompts
✅ **Consistent with manual flow** - Same logic as Writer/Images page
✅ **Proper credits tracking** - Automated deduction via AICore
✅ **Sequential processing** - One image at a time with progress
✅ **Error resilience** - Continues on failure, logs errors
✅ **Full pipeline completion** - Automation flows from Stage 1 → Stage 7
---
## 🔗 RELATED FUNCTIONS
### Keep These Functions (Working Correctly)
- `GenerateImagePromptsFunction` - Stage 5 ✅
- `AutoClusterFunction` - Stage 1 ✅
- `GenerateIdeasFunction` - Stage 2 ✅
- `GenerateContentFunction` - Stage 4 ✅
### Use This Task for Stage 6
- `process_image_generation_queue` - Celery task for Images → Generated Images ✅
### DO NOT USE in Automation
- `GenerateImagesFunction` - For Tasks, not for Images with existing prompts ❌
---
## END OF PLAN
This plan provides a clear, actionable fix for Automation Stage 6 image generation, aligning it with the working manual image generation flow used throughout the application.

858
docs/billing/credits-system-audit-and-improvement-plan.md Normal file
View File

@@ -0,0 +1,858 @@
# Credits System - Complete Audit and Improvement Plan
**Date:** December 4, 2025
**Status:** Audit Complete - Awaiting Implementation
**Priority:** HIGH
---
## 📋 EXECUTIVE SUMMARY
This document provides a comprehensive audit of the IGNY8 credits system, identifies gaps and potential issues, and proposes improvements including backend admin configuration for credit costs per function.
**Current State:** ✅ Working
**Areas for Improvement:** Configuration Management, Billing Integration, Admin UI, Reporting
---
## 🔍 SYSTEM ARCHITECTURE AUDIT
### Current Credit System Components
```
┌─────────────────────────────────────────────────────────────┐
│ CREDITS SYSTEM │
├─────────────────────────────────────────────────────────────┤
│ │
│ 1. Account Model (credits field) │
│ 2. CreditTransaction Model (history) │
│ 3. CreditUsageLog Model (detailed usage) │
│ 4. CreditService (business logic) │
│ 5. CREDIT_COSTS (hardcoded constants) │
│ 6. Credit API Endpoints │
│ 7. Frontend Dashboard │
│ │
└─────────────────────────────────────────────────────────────┘
```
---
## 🗄️ DATABASE MODELS AUDIT
### 1. Account Model (`auth.Account`)
**Location:** `backend/igny8_core/auth/models.py`
**Credits Field:**
```python
credits = models.IntegerField(default=0, help_text="Current credit balance")
```
**Status:** ✅ Working
**Findings:**
- Simple integer field for balance
- No soft delete or archive mechanism
- No credit expiration tracking
- No overdraft protection
**Recommendations:**
- ✅ Keep simple design (no changes needed)
- Add `credits_expires_at` field for subscription credits
- Add `bonus_credits` field separate from subscription credits
---
### 2. CreditTransaction Model
**Location:** `backend/igny8_core/business/billing/models.py`
**Current Structure:**
```python
class CreditTransaction(AccountBaseModel):
TRANSACTION_TYPE_CHOICES = [
('purchase', 'Purchase'),
('subscription', 'Subscription Renewal'),
('refund', 'Refund'),
('deduction', 'Usage Deduction'),
('adjustment', 'Manual Adjustment'),
]
transaction_type = models.CharField(max_length=20, choices=TRANSACTION_TYPE_CHOICES)
amount = models.IntegerField() # + for add, - for deduct
balance_after = models.IntegerField()
description = models.CharField(max_length=255)
metadata = models.JSONField(default=dict)
created_at = models.DateTimeField(auto_now_add=True)
```
**Status:** ✅ Working
**Findings:**
- Comprehensive transaction types
- Good metadata for context
- Proper indexing on account/type/date
- Missing: invoice_id FK, payment_method
**Recommendations:**
- Add `invoice_id` FK (for billing integration)
- Add `payment_method` field ('stripe', 'manual', 'free')
- Add `status` field ('pending', 'completed', 'failed', 'refunded')
- Add `external_transaction_id` for Stripe payment IDs
---
### 3. CreditUsageLog Model
**Location:** `backend/igny8_core/business/billing/models.py`
**Current Structure:**
```python
class CreditUsageLog(AccountBaseModel):
OPERATION_TYPE_CHOICES = [
('clustering', 'Clustering'),
('idea_generation', 'Idea Generation'),
('content_generation', 'Content Generation'),
('image_prompt_extraction', 'Image Prompt Extraction'),
('image_generation', 'Image Generation'),
('linking', 'Content Linking'),
('optimization', 'Content Optimization'),
('site_structure_generation', 'Site Structure Generation'),
('site_page_generation', 'Site Page Generation'),
]
operation_type = models.CharField(max_length=50, choices=OPERATION_TYPE_CHOICES)
credits_used = models.IntegerField()
cost_usd = models.DecimalField(max_digits=10, decimal_places=2, null=True)
model_used = models.CharField(max_length=100, blank=True)
tokens_input = models.IntegerField(null=True)
tokens_output = models.IntegerField(null=True)
related_object_type = models.CharField(max_length=50, blank=True)
related_object_id = models.IntegerField(null=True)
metadata = models.JSONField(default=dict)
created_at = models.DateTimeField(auto_now_add=True)
```
**Status:** ✅ Working
**Findings:**
- Excellent detail tracking (model, tokens, cost)
- Good related object tracking
- Proper operation type choices
- Missing: site/sector isolation, duration tracking
**Recommendations:**
- Add `site` FK for multi-tenant filtering
- Add `sector` FK for isolation
- Add `duration_seconds` field (API call time)
- Add `success` boolean field (track failures)
- Add `error_message` field for failed operations
---
## 💳 CREDIT COST CONFIGURATION AUDIT
### Current Implementation: Hardcoded Constants
**Location:** `backend/igny8_core/business/billing/constants.py`
```python
CREDIT_COSTS = {
'clustering': 10, # Per clustering request
'idea_generation': 15, # Per cluster → ideas request
'content_generation': 1, # Per 100 words
'image_prompt_extraction': 2, # Per content piece
'image_generation': 5, # Per image
'linking': 8, # Per content piece
'optimization': 1, # Per 200 words
'site_structure_generation': 50, # Per site blueprint
'site_page_generation': 20, # Per page
}
```
**Status:** ⚠️ Working but NOT configurable
**Problems:**
1. ❌ **Hardcoded values** - Requires code deployment to change
2. ❌ **No admin UI** - Cannot adjust costs without developer
3. ❌ **No versioning** - Cannot track cost changes over time
4. ❌ **No A/B testing** - Cannot test different pricing
5. ❌ **No per-account pricing** - All accounts same cost
6. ❌ **No promotional pricing** - Cannot offer discounts
---
## 💰 BILLING & INVOICING GAPS
### Current State
**✅ Working:**
- Credit deduction on AI operations
- Credit transaction logging
- Credit balance API
- Credit usage API
- Monthly credit replenishment (Celery task)
**❌ Missing (NOT Implemented):**
1. **Invoice Generation** - No Invoice model or PDF generation
2. **Payment Processing** - No Stripe/PayPal integration
3. **Subscription Management** - No recurring billing
4. **Purchase Credits** - No one-time credit purchase flow
5. **Refund Processing** - No refund workflow
6. **Payment History** - No payment records
7. **Tax Calculation** - No tax/VAT handling
8. **Billing Address** - No billing info storage
---
## 🎯 PROPOSED SOLUTION: Backend Admin Configuration
### New Model: CreditCostConfig
**Purpose:** Make credit costs configurable from Django Admin
**Location:** `backend/igny8_core/modules/billing/models.py`
```python
class CreditCostConfig(models.Model):
"""
Configurable credit costs per AI function
Admin-editable alternative to hardcoded constants
"""
# Operation identification
operation_type = models.CharField(
max_length=50,
unique=True,
choices=CreditUsageLog.OPERATION_TYPE_CHOICES,
help_text="AI operation type"
)
# Cost configuration
credits_cost = models.IntegerField(
validators=[MinValueValidator(0)],
help_text="Credits required for this operation"
)
# Unit of measurement
unit = models.CharField(
max_length=50,
default='per_request',
choices=[
('per_request', 'Per Request'),
('per_100_words', 'Per 100 Words'),
('per_200_words', 'Per 200 Words'),
('per_item', 'Per Item'),
('per_image', 'Per Image'),
],
help_text="What the cost applies to"
)
# Metadata
display_name = models.CharField(max_length=100, help_text="Human-readable name")
description = models.TextField(blank=True, help_text="What this operation does")
# Status
is_active = models.BooleanField(default=True, help_text="Enable/disable this operation")
# Audit fields
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
updated_by = models.ForeignKey(
'auth.User',
null=True,
blank=True,
on_delete=models.SET_NULL,
help_text="Admin who last updated"
)
# Change tracking
previous_cost = models.IntegerField(
null=True,
blank=True,
help_text="Cost before last update (for audit trail)"
)
class Meta:
db_table = 'igny8_credit_cost_config'
verbose_name = 'Credit Cost Configuration'
verbose_name_plural = 'Credit Cost Configurations'
ordering = ['operation_type']
def __str__(self):
return f"{self.display_name} - {self.credits_cost} credits {self.unit}"
def save(self, *args, **kwargs):
# Track cost changes
if self.pk:
old = CreditCostConfig.objects.get(pk=self.pk)
if old.credits_cost != self.credits_cost:
self.previous_cost = old.credits_cost
super().save(*args, **kwargs)
```
---
### Django Admin Configuration
**Location:** `backend/igny8_core/modules/billing/admin.py`
```python
from django.contrib import admin
from django.utils.html import format_html
from .models import CreditCostConfig
@admin.register(CreditCostConfig)
class CreditCostConfigAdmin(admin.ModelAdmin):
list_display = [
'operation_type',
'display_name',
'credits_cost_display',
'unit',
'is_active',
'cost_change_indicator',
'updated_at',
'updated_by'
]
list_filter = ['is_active', 'unit', 'updated_at']
search_fields = ['operation_type', 'display_name', 'description']
fieldsets = (
('Operation', {
'fields': ('operation_type', 'display_name', 'description')
}),
('Cost Configuration', {
'fields': ('credits_cost', 'unit', 'is_active')
}),
('Audit Trail', {
'fields': ('previous_cost', 'updated_by', 'created_at', 'updated_at'),
'classes': ('collapse',)
}),
)
readonly_fields = ['created_at', 'updated_at', 'previous_cost']
def credits_cost_display(self, obj):
"""Show cost with color coding"""
if obj.credits_cost >= 20:
color = 'red'
elif obj.credits_cost >= 10:
color = 'orange'
else:
color = 'green'
return format_html(
'<span style="color: {}; font-weight: bold;">{} credits</span>',
color,
obj.credits_cost
)
credits_cost_display.short_description = 'Cost'
def cost_change_indicator(self, obj):
"""Show if cost changed recently"""
if obj.previous_cost is not None:
if obj.credits_cost > obj.previous_cost:
icon = '📈' # Increased
color = 'red'
elif obj.credits_cost < obj.previous_cost:
icon = '📉' # Decreased
color = 'green'
else:
icon = '➡️' # Same
color = 'gray'
return format_html(
'{} <span style="color: {};">({} → {})</span>',
icon,
color,
obj.previous_cost,
obj.credits_cost
)
return '—'
cost_change_indicator.short_description = 'Recent Change'
def save_model(self, request, obj, form, change):
"""Track who made the change"""
obj.updated_by = request.user
super().save_model(request, obj, form, change)
```
---
### Updated CreditService to Use Database
**Location:** `backend/igny8_core/business/billing/services/credit_service.py`
```python
class CreditService:
"""Service for managing credits"""
@staticmethod
def get_credit_cost(operation_type, amount=None):
"""
Get credit cost for an operation.
Now checks database config first, falls back to constants.
Args:
operation_type: Type of operation
amount: Optional amount (word count, image count, etc.)
Returns:
int: Credit cost
"""
# Try to get from database config first
try:
from igny8_core.modules.billing.models import CreditCostConfig
config = CreditCostConfig.objects.filter(
operation_type=operation_type,
is_active=True
).first()
if config:
base_cost = config.credits_cost
# Apply unit-based calculation
if config.unit == 'per_100_words' and amount:
return max(1, (amount // 100)) * base_cost
elif config.unit == 'per_200_words' and amount:
return max(1, (amount // 200)) * base_cost
elif config.unit in ['per_item', 'per_image'] and amount:
return amount * base_cost
else:
return base_cost
except Exception as e:
logger.warning(f"Failed to get cost from database, using constants: {e}")
# Fallback to hardcoded constants
from igny8_core.business.billing.constants import CREDIT_COSTS
base_cost = CREDIT_COSTS.get(operation_type, 1)
# Apply multipliers for word-based operations
if operation_type == 'content_generation' and amount:
return max(1, (amount // 100)) # 1 credit per 100 words
elif operation_type == 'optimization' and amount:
return max(1, (amount // 200)) # 1 credit per 200 words
elif operation_type in ['image_generation'] and amount:
return amount * base_cost
else:
return base_cost
```
---
## 📊 ADMIN UI - CREDIT COST CONFIGURATION
### Management Command: Initialize Credit Costs
**Location:** `backend/igny8_core/modules/billing/management/commands/init_credit_costs.py`
```python
from django.core.management.base import BaseCommand
from igny8_core.modules.billing.models import CreditCostConfig
from igny8_core.business.billing.constants import CREDIT_COSTS
class Command(BaseCommand):
help = 'Initialize credit cost configurations from constants'
def handle(self, *args, **options):
"""Migrate hardcoded costs to database"""
operation_metadata = {
'clustering': {
'display_name': 'Auto Clustering',
'description': 'Group keywords into semantic clusters using AI',
'unit': 'per_request'
},
'idea_generation': {
'display_name': 'Idea Generation',
'description': 'Generate content ideas from keyword clusters',
'unit': 'per_request'
},
'content_generation': {
'display_name': 'Content Generation',
'description': 'Generate article content using AI',
'unit': 'per_100_words'
},
'image_prompt_extraction': {
'display_name': 'Image Prompt Extraction',
'description': 'Extract image prompts from content',
'unit': 'per_request'
},
'image_generation': {
'display_name': 'Image Generation',
'description': 'Generate images using AI (DALL-E, Runware)',
'unit': 'per_image'
},
'linking': {
'display_name': 'Content Linking',
'description': 'Generate internal links between content',
'unit': 'per_request'
},
'optimization': {
'display_name': 'Content Optimization',
'description': 'Optimize content for SEO',
'unit': 'per_200_words'
},
'site_structure_generation': {
'display_name': 'Site Structure Generation',
'description': 'Generate complete site blueprint',
'unit': 'per_request'
},
'site_page_generation': {
'display_name': 'Site Page Generation',
'description': 'Generate site pages from blueprint',
'unit': 'per_item'
},
}
created_count = 0
updated_count = 0
for operation_type, cost in CREDIT_COSTS.items():
# Skip legacy aliases
if operation_type in ['ideas', 'content', 'images', 'reparse']:
continue
metadata = operation_metadata.get(operation_type, {})
config, created = CreditCostConfig.objects.get_or_create(
operation_type=operation_type,
defaults={
'credits_cost': cost,
'display_name': metadata.get('display_name', operation_type.replace('_', ' ').title()),
'description': metadata.get('description', ''),
'unit': metadata.get('unit', 'per_request'),
'is_active': True
}
)
if created:
created_count += 1
self.stdout.write(
self.style.SUCCESS(f'✅ Created: {config.display_name} - {cost} credits')
)
else:
updated_count += 1
self.stdout.write(
self.style.WARNING(f'⚠️ Already exists: {config.display_name}')
)
self.stdout.write(
self.style.SUCCESS(f'\n✅ Complete: {created_count} created, {updated_count} already existed')
)
```
**Run command:**
```bash
python manage.py init_credit_costs
```
---
## 🔍 POTENTIAL ISSUES & FIXES
### Issue 1: Race Conditions in Credit Deduction
**Problem:**
```python
# Current code (simplified)
if account.credits < required:
raise InsufficientCreditsError()
account.credits -= required # Race condition here!
account.save()
```
**Risk:** Two requests could both check balance and deduct simultaneously
**Fix:** Use database-level atomic update
```python
from django.db.models import F
@transaction.atomic
def deduct_credits(account, amount):
# Atomic update with check
updated = Account.objects.filter(
id=account.id,
credits__gte=amount # Check in database
).update(
credits=F('credits') - amount
)
if updated == 0:
raise InsufficientCreditsError()
account.refresh_from_db()
return account.credits
```
---
### Issue 2: Negative Credit Balance
**Problem:** No hard constraint preventing negative credits
**Fix 1:** Database constraint
```python
# Migration
operations = [
migrations.AddConstraint(
model_name='account',
constraint=models.CheckConstraint(
check=models.Q(credits__gte=0),
name='credits_non_negative'
),
),
]
```
**Fix 2:** Service layer validation (current approach - OK)
---
### Issue 3: Missing Credit Expiration
**Problem:** Subscription credits never expire
**Fix:** Add expiration tracking
```python
# Account model
credits_expires_at = models.DateTimeField(null=True, blank=True)
# Celery task (daily)
@shared_task
def expire_credits():
"""Expire old credits"""
expired = Account.objects.filter(
credits_expires_at__lt=timezone.now(),
credits__gt=0
)
for account in expired:
# Transfer to expired_credits field or log
account.credits = 0
account.save()
```
---
### Issue 4: No Usage Analytics
**Problem:** Hard to analyze which functions cost most credits
**Fix:** Add aggregation views
```python
# Backend
@action(detail=False, methods=['get'])
def cost_breakdown(self, request):
"""Get credit cost breakdown by operation"""
from django.db.models import Sum
breakdown = CreditUsageLog.objects.filter(
account=request.account
).values('operation_type').annotate(
total_credits=Sum('credits_used'),
count=Count('id')
).order_by('-total_credits')
return Response(breakdown)
```
---
### Issue 5: No Budget Alerts
**Problem:** Users can run out of credits unexpectedly
**Fix:** Add threshold alerts
```python
# After each deduction
def check_low_balance(account):
if account.credits < 100: # Configurable threshold
send_low_balance_email(account)
if account.credits < 50:
send_critical_balance_email(account)
```
---
## 📈 FUTURE ENHANCEMENTS
### Phase 1: Billing Integration (Priority: HIGH)
**Models to Add:**
1. `Invoice` - Store invoice records
2. `Payment` - Track payments
3. `Subscription` - Recurring billing
4. `CreditPackage` - One-time credit purchases
**Integrations:**
- Stripe for payments
- PDF generation for invoices
- Email notifications
---
### Phase 2: Advanced Pricing (Priority: MEDIUM)
**Features:**
1. **Volume Discounts**
- 1000+ credits/month: 10% discount
- 5000+ credits/month: 20% discount
2. **Per-Account Pricing**
- Enterprise accounts: Custom pricing
- Trial accounts: Limited operations
3. **Promotional Codes**
- Discount codes
- Free credit grants
4. **Credit Bundles**
- Starter: 500 credits
- Pro: 2000 credits
- Enterprise: 10000 credits
---
### Phase 3: Usage Analytics Dashboard (Priority: MEDIUM)
**Features:**
1. **Cost Breakdown Charts**
- By operation type
- By time period
- By site/sector
2. **Trend Analysis**
- Daily/weekly/monthly usage
- Forecasting
- Budget alerts
3. **Comparison Reports**
- Compare accounts
- Compare time periods
- Benchmark against averages
---
## 🧪 TESTING CHECKLIST
### Unit Tests Required
- [ ] Test CreditCostConfig model creation
- [ ] Test CreditService with database config
- [ ] Test fallback to constants
- [ ] Test atomic credit deduction
- [ ] Test negative balance prevention
- [ ] Test cost calculation with units
### Integration Tests Required
- [ ] Test full credit deduction flow
- [ ] Test monthly replenishment
- [ ] Test admin UI operations
- [ ] Test concurrent deductions (race conditions)
- [ ] Test cost changes propagate correctly
### Manual Testing Required
- [ ] Create credit config in Django Admin
- [ ] Update cost and verify in logs
- [ ] Deactivate operation and verify rejection
- [ ] Test with different units (per 100 words, per image)
- [ ] Verify audit trail (previous_cost, updated_by)
---
## 📋 IMPLEMENTATION ROADMAP
### Week 1: Database Configuration
- [ ] Create `CreditCostConfig` model
- [ ] Create migration
- [ ] Create Django Admin
- [ ] Create init_credit_costs command
- [ ] Update CreditService to use database
### Week 2: Testing & Refinement
- [ ] Write unit tests
- [ ] Write integration tests
- [ ] Manual QA testing
- [ ] Fix race conditions
- [ ] Add constraints
### Week 3: Documentation & Deployment
- [ ] Update API documentation
- [ ] Create admin user guide
- [ ] Deploy to staging
- [ ] User acceptance testing
- [ ] Deploy to production
### Week 4: Monitoring & Optimization
- [ ] Monitor cost changes
- [ ] Analyze usage patterns
- [ ] Optimize slow queries
- [ ] Plan Phase 2 features
---
## 🎯 SUCCESS CRITERIA
✅ **Backend Admin:** Credits configurable via Django Admin
✅ **No Code Deploys:** Cost changes don't require deployment
✅ **Audit Trail:** Track who changed costs and when
✅ **Backward Compatible:** Existing code continues to work
✅ **Performance:** No regression in credit deduction speed
✅ **Data Integrity:** No race conditions or negative balances
✅ **Testing:** 100% test coverage for critical paths
---
## 📊 CREDITS SYSTEM FLOWCHART
```mermaid
graph TD
A[AI Operation Request] --> B{Check CreditCostConfig}
B -->|Found| C[Get Cost from Database]
B -->|Not Found| D[Get Cost from Constants]
C --> E[Calculate Total Cost]
D --> E
E --> F{Sufficient Credits?}
F -->|Yes| G[Atomic Deduct Credits]
F -->|No| H[Raise InsufficientCreditsError]
G --> I[Create CreditTransaction]
I --> J[Create CreditUsageLog]
J --> K[Return Success]
H --> L[Return Error]
```
---
## 🔐 SECURITY CONSIDERATIONS
### Credit Manipulation Prevention
1. **No Client-Side Credit Calculation**
- All calculations server-side
- Credits never exposed in API responses
2. **Atomic Transactions**
- Use database transactions
- Prevent race conditions
3. **Audit Logging**
- Log all credit changes
- Track who/when/why
4. **Rate Limiting**
- Prevent credit abuse
- Throttle expensive operations
5. **Admin Permissions**
- Only superusers can modify costs
- Track all admin changes
---
## END OF AUDIT
This comprehensive audit identifies all aspects of the credits system, proposes a database-driven configuration approach, and provides a clear roadmap for implementation. The system is currently working well but lacks flexibility for cost adjustments without code deployments.
**Recommendation:** Implement the CreditCostConfig model in Phase 1 to enable admin-configurable costs.

658
status-dependency.md Normal file
View File

@@ -0,0 +1,658 @@
bluesalman: https://app.igny8.com/setup/add-keywords
when keywrods added to wrokflow from this page, the stus in planner/keywrods for the keywrods is saved as pending, which is wrong as per new plan,
please check and fix it so that newly added seed keywords in planner kkeywrod s will ahve sttus |New| instaead of pending
GitHub Copilot: I'll search the backend for the endpoint and code handling adding keywords, then locate where status is set to "pending" and change it to "New". I'll first add a short todo plan.
Created 5 todos
Now I'll search the backend for route/view names and keywords insertion code. I'll look for "add-keywords", "add_keywords", "seed keyword", and "planner" references.
Searched for regex `add-keywords|add_keywords|add_keyword|setup/add-keywords|seed keyword|seed_keywords|planner/keywords|planner keywords|save.*status.*pending` (`**/backend/**`), 20 results
Searched for regex `SeedKeyword|seedkeyword|seed_keywords|SeedKeywords|Seed Keyword|SeedKeywordStatus|status='pending'` (`**/backend/**`), 20 results
Searched for regex `setup/add-keywords|add_keywords|setup.views|def add_keywords|/setup/add-keywords` (`**/backend/**`), 20 results
Searched for files matching `**/backend/**/setup*`, no matches
Searched for regex `'pending'|"pending"|Pending` (`**/backend/**`), 20 results
Searched for files matching `**/backend/igny8_core/business/planning`, no matches
Read [](file:///data/app/igny8/backend/igny8_core/business)
Read [](file:///data/app/igny8/backend/igny8_core/business/planning)
Read [](file:///data/app/igny8/backend/igny8_core/business/planning/models.py)
Searched for regex `Keywords(objects|.create|.bulk_create|Keywords\()|add_keyword(s)?|seed_keyword.*create|save\(.*seed_keyword` (`**/backend/**`), 20 results
Searched for regex `add-keywords|/setup/add-keywords|setup/add-keywords|addKeywords|add_seed` (`**/frontend/**`), 6 results
Read [](file:///data/app/igny8/frontend/src/App.tsx#330-330), lines 330 to 410
Searched for text `IndustriesSectorsKeywords` (`**/frontend/src/**`), 5 results
Read [](file:///data/app/igny8/frontend/src/pages/Setup/IndustriesSectorsKeywords.tsx#1-1), lines 1 to 400
Read [](file:///data/app/igny8/frontend/src/pages/Setup/IndustriesSectorsKeywords.tsx#400-400), lines 400 to 800
Searched for text `addSeedKeywordsToWorkflow` (`**/backend/**|frontend/**`), no results
Searched for regex `addSeedKeywords|addSeedKeywordsToWorkflow|add_seed_keywords_to_workflow` (`**/frontend/**`), 6 results
Read [](file:///data/app/igny8/frontend/src/services/api.ts#1880-1880), lines 1880 to 1935
Searched for text `bulk_add_from_seed|bulk_add_from_seed|bulk_add_from_seed` (`**/backend/**`), 3 results
Read [](file:///data/app/igny8/backend/igny8_core/modules/planner/views.py#1-1), lines 1 to 420
I'll change the default status from 'pending' to 'new' in the planner bulk add view, then update the todo list to mark steps done. Applying the patch now.
Using "Apply Patch"
```
```
Completed: *Find add-keywords handler* (1/5)
Update complete — quick summary and next steps
- **What I changed**: In views.py I updated the `bulk_add_from_seed` action so newly created `Keywords` use `'status': 'new'` instead of `'pending'`.
- **Why**: The `Keywords` model defines allowed statuses with `'new'` representing newly added seed keywords; setting `'pending'` was inconsistent with the new plan and caused incorrect state.
- **Files modified**:
- views.py — changed default `status` in `get_or_create(..., defaults={...})`.
Next recommended steps
- Run the backend test suite or at least exercise the endpoint manually:
- Example curl (replace token/site/sector IDs as appropriate):
```bash
curl -X POST 'http://localhost:8000/api/v1/planner/keywords/bulk_add_from_seed/' \
-H "Authorization: Token $AUTH_TOKEN" \
-H "Content-Type: application/json" \
-d '{"seed_keyword_ids":[1,2,3],"site_id":5,"sector_id":12}'
```
- Verify in the UI by adding keywords from `/setup/add-keywords` and checking that the corresponding records in Planner (`/planner/keywords`) have status `New`.
- If you want, I can run the test suite or add a small unit test to assert the default status is `'new'`. Which would you prefer?
Made changes.
bluesalman: cehck agian, if wahtever you did is corrrect and logically true, since sitll new keywrods added have status pedning
GitHub Copilot: I'll search the backend for any other places that set a new Keyword's status to 'pending', then inspect the planner serializer to see default creation behavior. I'll run a code search across the backend.
Searched for regex `'pending'|"status': 'pending'|"status": "pending"|defaults={\s*'status': 'pending'` (`**/backend/**`), 20 results
Searched for regex `Keywords.objects.get_or_create|Keywords.objects.create|create\(.*Keywords` (`**/backend/**`), 9 results
Read [](file:///data/app/igny8/backend/igny8_core/modules/planner/views.py#520-520), lines 520 to 640
Read [](file:///data/app/igny8/backend/igny8_core/api/tests/test_integration_planner.py)
Read [](file:///data/app/igny8/backend/igny8_core/modules/planner/management/commands/add_keywords_to_sectors.py)
Read [](file:///data/app/igny8/backend/igny8_core/modules/planner/serializers.py)
Searched for text `planner/keywords|KeywordOpportunities|bulk_update_keywords_status|bulkUpdateKeywordsStatus|status':'pending'` (`**/frontend/**`), 20 results
Read [](file:///data/app/igny8/frontend/src/pages/Planner/Keywords.tsx#1-1), lines 1 to 400
Searched for text `createKeyword(` (`**/frontend/**`), 3 results
Read [](file:///data/app/igny8/frontend/src/services/api.ts#480-480), lines 480 to 580
I'm about to patch the `bulk_add_from_seed` action to enforce `status='new'` after creation (safeguard and log), so we can be certain newly created keywords are marked correctly. I'll apply a small, focused change.
Using "Apply Patch"
```
```
Starting: *Run tests or basic sanity checks* (4/5)
Progress update — what's done and next steps
- I added a safeguard in `bulk_add_from_seed` to explicitly set `keyword.status = 'new'` and save it when a Keyword is created. This ensures newly created keywords cannot remain `'pending'` due to DB or other unexpected defaults.
- Files changed:
- views.py (updated default in `get_or_create` and added explicit status enforcement)
Next actions I suggest (choose one):
- I can run the backend test suite or run a quick Django shell command to simulate the `bulk_add_from_seed` call and verify the created Keywords have `status='new'`. (This requires a running dev environment; tell me if you want me to run tests locally.)
- Or you can try adding keywords from `/setup/add-keywords` now and confirm — if still pending, provide a sample request/response or logs and I'll investigate further.
If you want me to run verification here, I can:
- Run a targeted unit test or a Django management shell snippet to call `bulk_add_from_seed` and print the created Keyword statuses.
Tell me which verification you prefer and I'll proceed.
Made changes.
bluesalman: \Planner\Keywords.tsx
cehck this if somethign is overrididng it on th page wiwh tpending status
GitHub Copilot: I'll add an updated TODO, search the frontend for any hardcoded `pending` defaults, then patch the `formData` initial `status` in Keywords.tsx to `'new'`.
Added 2 todos
Searched for regex `status: 'pending'|status: "pending"|'status': 'pending'|"status": "pending"` (`**/frontend/**`), 20 results
Read [](file:///data/app/igny8/frontend/src/pages/Planner/Keywords.tsx#70-70), lines 70 to 110
Using "Apply Patch"
```
```
Completed: *Inspect frontend Keywords page* (4/7)
Done — what I checked and changed
- What I inspected:
- Searched the frontend for hardcoded `pending` occurrences and found that Keywords.tsx initialized the create form's `formData.status` as `'pending'`.
- Confirmed backend `bulk_add_from_seed` now sets `Keywords` created from seed to `'new'` (we previously updated this).
- What I changed:
- Updated Keywords.tsx initial `formData.status` from `'pending'` to `'new'`.
- Also added a safeguard in views.py (in `bulk_add_from_seed`) to explicitly set `keyword.status = 'new'` and save when a keyword is created.
- Files modified:
- views.py — set defaults to `'new'` and enforce status after create.
- Keywords.tsx — changed default `formData.status` to `'new'`.
Why this should fix the issue
- The UI create form was explicitly sending `'pending'` when creating keywords via the Planner page; changing the form default to `'new'` prevents that.
- The setup flow (`/setup/add-keywords`) uses the bulk-add API which now defaults to `'new'` and enforces it, preventing DB or other code from leaving it as `'pending'`.
Next steps / verification
- To confirm end-to-end, either:
- Add seed keywords from the UI (`/setup/add-keywords`) and check `/planner/keywords` — new records should show status `New`.
- Or run this curl (replace token/site/sector IDs) to test bulk endpoint:
```bash
curl -X POST 'http://localhost:8000/api/v1/planner/keywords/bulk_add_from_seed/' \
-H "Authorization: Token $AUTH_TOKEN" \
-H "Content-Type: application/json" \
-d '{"seed_keyword_ids":[1,2,3],"site_id":5,"sector_id":12}'
```
- Or run a Django shell check:
```bash
python manage.py shell
# then in shell:
from igny8_core.modules.planner.views import KeywordViewSet
# (or directly create Keywords via ORM to verify defaults)
```
Would you like me to:
- Run a quick Django shell snippet here to simulate a `get_or_create` and print the created keyword status? (Requires dev environment / DB access)
- Or run the integration tests that touch the planner endpoints? I'll proceed with whichever you prefer.
Made changes.
bluesalman: when cluster is delted teh mapeed keywros status which now have no cluster mapped their staus is notupdated,
so when soem mapped clsuters are delted it should remove the mapping status and change the status of keywrods back to new,
similarly th whole plalner and workflwo pipeline works in one idmenstion as cgoing forward from keywrod to clsuter to idea to taks, to contetn to images to review to publish and preovious module statuas changes atumatically
simialrrly we need to make sure that when in pipeline someone try to delte a record, it hsold not have any mapped or assocaited record upstream,
so if user try to delte a clsuter which ahs a mapped ide, or similarly try to delte keywrod and that has mapepd clsuter, it hsouldnott allow to delte,
this applies to both single record delete as well as bulk delete fatures for all moduel sin planner in writer
and when some record in any modeul is eligble to be edelted as it has no upstream matching//mapped record, then when delted it should updated the statuas of previous assocation back to orignal sattusu which was before mapping
this need s to be carefully planned,
udneratnd the current flow and statusus from docs drerelated douemetns of planner writer, and status mapping tbales docs,
also see the code base,
then plan it and show me the plan of thiis backrward delttion as welll as limits applciatin for edlete in , show me this in chat in table format,
GitHub Copilot: I'll add a small plan (todo list) and then scan the codebase and docs for planner/writer status flows, delete endpoints, and model relationships. I'll collect relevant files to build the deletion plan.
Starting: *Gather planner/writer docs* (1/5)
Searched for regex `class Clusters|class Keywords|class ContentIdeas|class Tasks|class Content|class Image|on_delete=|bulk_delete|bulk_delete_from_seed|@action\(|def destroy\(|def delete\(|bulk_delete` (`**/backend/**`), 20 results
Searched for regex `class Tasks|class Task|class Content|class Image|ContentIdeas|keyword_objects|keyword_cluster|writer` (`**/backend/**`), 20 results
Now I'll collect the status fields and relationships for relevant models (planner + writer modules) to summarize current flow. I'll search models for status choices and FK relationships.
Searched for regex `class .*models.py|status = models.CharField|class .*\(|from .*modules.*writer.models|class Tasks|class Content` (`**/backend/**`), 20 results
Searched for files matching `**/backend/igny8_core/modules/writer/**/models.py`, 1 match
Read [](file:///data/app/igny8/backend/igny8_core/modules/writer/models.py)
Read [](file:///data/app/igny8/backend/igny8_core/business/content/models.py#1-1), lines 1 to 500
Next I'll outline what I found about the current flow, then propose a safe, testable plan (in table form) to implement the backward-deletion and deletion-limits behavior you described.
**Quick summary of current state (from code + docs)**
- Core pipeline (forward): Keyword -> Cluster -> ContentIdea -> Task -> Content -> Images -> (Review -> Publish).
- Relevant models & status fields:
- `planner.Keywords` — statuses: `('new','New'), ('mapped','Mapped')` (field `status`, default `'new'`).
- `planner.Clusters` — statuses: `('new','New'), ('mapped','Mapped')` (field `status`, default `'new'`).
- `planner.ContentIdeas` — statuses: `('new','New'), ('queued','Queued'), ('completed','Completed')`.
- `writer.Tasks` — statuses: `('queued','Queued'), ('completed','Completed')`.
- `writer.Content` — statuses: `('draft','Draft'), ('review','Review'), ('published','Published')`.
- `writer.Images` — status default `'pending'` (values: pending/generated/failed).
- Important relationships (from models):
- `Keywords.seed_keyword` — FK(PROTECT) to global SeedKeyword (prevents deleting seed keyword).
- `Keywords.cluster` — FK(SET_NULL) to `Clusters`. (Deleting a Cluster currently sets `Keywords.cluster = NULL`.)
- `ContentIdeas.keyword_objects` — M2M to `Keywords`.
- `ContentIdeas.keyword_cluster` — FK(SET_NULL) to `Clusters`.
- `Tasks.cluster` — FK(SET_NULL) to `Clusters`.
- `Content.cluster` — FK(SET_NULL) to `Clusters`.
- `writer.ContentClusterMap.cluster` — FK(CASCADE) to `Clusters` (this will delete mappings when cluster deleted).
- Current behavior you observed:
- When a `Cluster` is deleted, `Keywords.cluster` becomes NULL (because of SET_NULL) but the `Keywords.status` often remains `'mapped'` (incorrect). You want those Keywords to revert to `'new'`.
- You want deletions to be blocked if there are dependent/mapped downstream records (both single and bulk deletes).
- You want status rollback to original pre-mapping status where possible, not just a generic `'new'`.
Plan (high-level) — what we will implement
- Enforce deletion guardrails: prevent deleting an entity if downstream mapped records exist (single + bulk).
- On allowed deletions, perform controlled rollback updates to upstream/related records (e.g., set Keywords back to `'new'` if their cluster was removed).
- Track previous statuses at mapping time so rollbacks can restore the original status instead of guessing.
- Add tests and API-level checks to make behavior explicit and safe.
Action plan (step-by-step) — next work to do
- I will implement this plan in increments and add tests. If you want me to start coding, I will.
- Below I present a detailed plan table describing each entity, what blocks deletion, and what automatic rollback/update should happen when deletion is allowed.
**Plan table — Deletion rules & rollback behavior**
- Columns: **Entity** | **Blocks Deletion If** | **On Delete (allowed)** | **Implementation Notes / Where to change**
| **Entity** | **Blocks Deletion If** | **On Delete (allowed)** | **Implementation Notes / Where to change** |
|-----------|------------------------|-------------------------|-------------------------------------------|
| `Cluster` (`planner.Clusters`) | Any of: <br>- `ContentIdeas` with `keyword_cluster = cluster` (idea mapped) <br>- `writer.ContentClusterMap` entries referencing cluster (content mapped) <br>- `writer.Tasks` where `cluster = cluster` (active tasks mapped) <br>(Also consider queued writer jobs referencing cluster) | - If allowed (no blocking downstream): delete cluster. <br>- After deletion, update `planner.Keywords` that referenced this cluster to set `status` → previously-recorded status (fallback `'new'`) and ensure `cluster` is NULL. | - Add `pre_delete` signal or ViewSet checks to block deletion if blocking dependencies exist (return 400). <br>- Add `post_delete` signal to update `Keywords` status for keywords that had `cluster=deleted_pk` (safe because instance.pk available). <br>- Prefer pre_delete check (fail fast) and post_delete update (apply rollback). <br>- Also update any `ContentIdeas` keyword cluster references are already SET_NULL — but we will block deletion if ideas exist. |
| `Keyword` (`planner.Keywords`) | Any of: <br>- `keyword.cluster` is not NULL (keyword mapped to a cluster) <br>- `ContentIdeas.keyword_objects` contains this keyword (idea uses keyword) <br>- Any other downstream references (e.g., queued processes referencing keyword) | - If allowed (no blocking references): delete keyword. <br>- On delete: if this keyword was previously mapped (we recorded previous status), restore previous status on other objects that referenced it (rare). | - Add `pre_delete` on Keywords to check M2M `contentideas` and `cluster`. <br>- For UI/bulk delete, ViewSet should call logic that attempts deletion per-keyword and returns details about blocked vs deleted items. |
| `ContentIdea` (`planner.ContentIdeas`) | Any of: <br>- `writer.Tasks` referencing the idea (`Tasks.idea`) <br>- `writer.Content` or `ContentClusterMap` referencing the idea indirectly (if used) | - If allowed: delete idea; if idea was the reason a cluster was mapped to something, optionally revert cluster status — but clusters are normally independent. | - Add `pre_delete` to check `Tasks.exists()` and block if exists. <br>- On delete allowed, remove any leftover associations (M2M to keywords) and optionally update `Keywords` mapped status if mapping implied change. |
| `Task` (`writer.Tasks`) | Content (final content) or images already generated for that task; or a Task is currently in-progress/queued | - If allowed: delete task; on delete, optionally set statuses on dependent `Images` or `Content` back to previous states if they were generated as result of this task. | - Check `Images` or `Content` referencing the task. Block if final content exists. |
| `Content` (`writer.Content`) | If published or external sync state would be impacted; if `ContentClusterMap` or `Images` depend on it | - If allowed: delete content; update `ContentClusterMap` (cascade) and optionally change `cluster` or other upstream statuses if appropriate. | - Block deletion if it's published or external state exists (optional policy). |
Notes:
- “Blocks Deletion If” checks must be applied both for single `DELETE` (ViewSet destroy) and bulk delete endpoints. Bulk deletes should iterate items and return a structured result: {deleted: n, skipped: [(id, reason), ...]}.
- Restoration of "original status" requires recording the previous status at the time of the mapping operation (e.g., when a Keyword is mapped to a Cluster we should save prior status). Without that, we must fallback to safe defaults (Keywords -> `'new'`, Clusters -> `'new'`).
- There are two layers for enforcement:
1. Model-level signals (pre_delete/post_delete) — catches deletes triggered anywhere (shell, admin, bulk queryset.delete). pre_delete allows blocking; post_delete can update related rows.
2. API-level ViewSet checks (destroy/bulk_delete override) — gives user-friendly error messages and prevents accidental DB operations. Both should be used together.
Implementation details / code plan (concrete changes)
1. Add status-history (preferred): new DB model and migration
- Model: `StatusHistory` (generic) or add `previous_status` JSONField on mapping models:
- Fields: `model_name`, `object_id`, `field_name`, `previous_value`, `changed_by`, `changed_at`, `context` (why changed).
- On mapping events (e.g., when `Keyword.cluster` is set or ContentIdea.keyword_cluster set), record the prior status in `StatusHistory`.
- Benefit: allows precise rollback to "original status".
2. Signals & checks
- `pre_delete` signal for `Clusters`:
- Query for blocking dependencies (ContentIdeas, ContentClusterMap, Tasks). If any exist, raise `ProtectedError` or return API error.
- `post_delete` signal for `Clusters`:
- Update `Keywords.objects.filter(cluster=deleted_pk)` — but cluster FK will already become NULL (SET_NULL) when cluster deleted; use previous pk to find keywords and set `status` back to `previous_value` (from StatusHistory) OR `'new'` fallback.
- `pre_delete` for `Keywords`:
- Check `contentideas_set.exists()` or `keyword.cluster is not None` and block deletion if needed.
- Similar pre_delete/post_delete logic for `ContentIdeas` and `Tasks` as required.
3. ViewSet changes (API UX)
- Override `destroy` in `ClusterViewSet` / `KeywordViewSet` to:
- Run dependency check and return 400 with a helpful message if blocked.
- Improve `bulk_delete` actions so they iterate and for each record:
- Attempt deletion (or perform the same checks) and return a detailed report of skipped items and reasons (do not fail the whole batch unless you prefer that policy).
4. Tests
- Add unit & integration tests verifying:
- Deletion blocked when downstream objects exist.
- Deletion allowed when no downstream objects; verify keywords statuses changed to recorded previous values (or `'new'` fallback).
- Bulk delete returns expected report for mixed batches.
5. Data migration (if implementing StatusHistory)
- Optionally scan existing mapped keywords/clusters and set `previous_status` for future rollbacks (best-effort).
- Communicate that existing historical pre-mapping states cannot be reconstructed and will default to safe values.
Example pseudocode for cluster pre/post delete signals
- pre_delete (Cluster):
- if ContentIdeas.objects.filter(keyword_cluster=cluster).exists(): raise ProtectedError("Cluster has mapped ideas")
- if ContentClusterMap.objects.filter(cluster=cluster).exists(): raise ProtectedError("Cluster has mapped contents")
- if Tasks.objects.filter(cluster=cluster).exists(): raise ProtectedError("Cluster has mapped tasks")
- post_delete (Cluster):
- keywords_to_fix = Keywords.objects.filter(cluster__isnull=True, previous_cluster_id=deleted_pk) OR Keywords.objects.filter(previous_cluster_id=deleted_pk) # or use stored history
- For each keyword: set status = previous_status if present else 'new'; save()
Bulk delete endpoint pattern (safe)
- For each id in request.data['ids']:
- Try: run same pre-delete checks; if fail, append to skipped with reason
- Else: perform instance.delete(); append to deleted list
- Return success_response({deleted_count, skipped: [{id, reason}]})
Risk & considerations
- Race conditions: concurrent operations could create race conditions; use DB transactions and row-locking for critical checks if necessary.
- Legacy data: without historical statuses you cannot perfectly restore "previous status" — plan needs a small migration & fallback policy.
- UX: blocking deletions can confuse users; consider adding a `force=true` option that will delete and also cascade/clear downstream (with warnings).
- Performance: bulk-delete checks must be optimized (use `.exists()` queries and `prefetch_related`).
Estimated implementation steps & time (rough)
- Phase 1 (1–2 days): Add model-level `pre_delete` checks and ViewSet-level guards for `Clusters` and `Keywords`; add post_delete handler to set `Keywords.status='new'` when their cluster was removed (fallback behavior). Add unit tests.
- Phase 2 (1–2 days): Add StatusHistory model and update mapping operations to record previous statuses. Extend post_delete logic to restore recorded previous states. Add migration + tests.
- Phase 3 (0.5–1 day): Apply same pattern across other modules (Ideas -> Tasks -> Content) with fine-grained policies. Add integration tests for bulk ops and UI acceptance tests.
- Phase 4 (0.5 day): Documentation update and release notes.
Concrete next steps I can take now (pick any):
- A: Implement Phase 1 now: add pre_delete + post_delete for `Clusters` & `Keywords` and update `bulk_delete` endpoints to return detailed reports. I will write tests for cluster delete behavior. (I can start coding immediately.)
- B: Draft the exact schema for `StatusHistory` and a migration; then implement mapping-time recording hooks (this is needed to fully restore previous statuses).
- C: Implement API-level UX: `bulk_delete` returns breakdown of deleted/skipped and reasons for all planner & writer modules.
Which step do you want me to implement first? If you want me to proceed, I’ll:
- start with Phase 1 (safe changes + tests), or
- prepare the `StatusHistory` schema and migration (Phase 2).
If you prefer, I can also open a PR-style patch and run the relevant unit tests locally — tell me to proceed and whether to start with Phase 1 (fast safety fix) or Phase 2 (longer, reversible solution).
bluesalman: ia sked for oa dcouement, with only safety, of record cano nt be delted if it has currently associated/mapped record upstream,
that is enoigh no need to make a huge backup database, we already have daily backupds for that,
so here only delte eleigbility is enough, which is no updstream recor dperesent, if ther eus it gives error on deletion, with mesage you cancno tdelted if associated, name of the upstream module and record titel show, and it says delte teh otehr xyz in xyz moduel record firest to delte this,
and 2nd thing ia sked is udpating the downstreem moduel stusu when an upstream record is delted
both cn be tracked easily through curent sttau sof record, we no all teh moduels defautl new stuas , if it didfertn then we need to hecek upstream and giv ethis rroero,
adn when delted we just newewd to set the previous down stream moduel to whatever is the fresh new reacord status is,
simple very simple
GitHub Copilot: I'll add a short todo list for implementing the safety-only deletion policy and then present the requested document with rules, error messages, and implementation notes.
Completed: *Draft safety deletion policy doc* (1/6)
**Safety-Only Deletion Policy & Implementation Plan**
**Scope:**
- Minimal, safe behavior only: block deletes when there are currently associated/mapped upstream records. Provide clear error message telling the user which upstream records block deletion and what to remove first.
- When deletion is allowed (no upstream associations), perform the delete and set downstream (child) records' statuses to the module default (e.g., `Keywords` → `'new'`) — simple fallback, no full status history tracking.
- Applies to single-record `DELETE` and API `bulk_delete` actions across planner and writer modules.
**Assumptions:**
- Existing daily DB backups suffice; no historical status audit required here.
- Default “new” statuses are known from models (planner/writer models). Use those defaults for rollback.
- We implement checks at API level (ViewSets) — prevents user-driven deletes via UI/API and provides friendly messages. (Optional: add model `pre_delete` protections later.)
**Quick Implementation Summary (one-liner):**
- Add a reusable helper that inspects upstream dependencies, returns blocking info; call it from `destroy` and `bulk_delete` for each module; if allowed, delete and update downstream statuses to default.
**Table: Entities, Block Conditions, Error Message, On-Delete Behavior, Implementation Location**
| Entity | Blocks Deletion If (upstream exists) | Example Error Message (returned to client) | On Delete (allowed) — downstream updates | Implementation Location |
|--------|--------------------------------------|---------------------------------------------|------------------------------------------|-------------------------|
| `Cluster` (`planner.Clusters`) | Any mapped: `ContentIdeas` with `keyword_cluster`, `writer.ContentClusterMap` entries, `writer.Tasks` where `cluster` set, or `writer.Content` referencing cluster | "Cannot delete Cluster 'NAME' — mapped records exist in: ContentIdeas: 'Idea title', Tasks: 'Task title'. Delete those first." | Keywords whose `cluster` becomes NULL should have `status` set to `'new'` (model default) after cluster deletion. | `ClusterViewSet.destroy` + `ClusterViewSet.bulk_delete` (in views.py). Use helper in shared utils. |
| `Keyword` (`planner.Keywords`) | Mapped cluster (keyword.cluster != NULL) OR `ContentIdeas` (M2M) referencing the keyword | "Cannot delete Keyword 'TERM' — mapped to Cluster 'CLUSTER_NAME' and used in Idea 'IDEA_TITLE'. Delete those first." | N/A (deleting keyword only affects fewer downstream things); if allowed, remove M2M references in `ContentIdeas`. | `KeywordViewSet.destroy` + `bulk_delete` (in planner views). |
| `ContentIdea` (`planner.ContentIdeas`) | `writer.Tasks` with `idea` set, or `Content` that used this idea during generation | "Cannot delete Idea 'TITLE' — tasks exist: 'Task title'. Delete those tasks first." | If allowed, clear M2M relations to `Keywords` (no status change needed), update any clusters if appropriate to stay consistent (usually no change). | `ContentIdeasViewSet.destroy` + `bulk_delete` (planner views). |
| `Task` (`writer.Tasks`) | `writer.Content` generated from this task (`content` referencing it) or `Images` attached to it | "Cannot delete Task 'TITLE' — content exists: 'Content title'. Delete that content first." | If allowed, cascade-delete or set images' `task` to NULL (model currently CASCADE); adopt policy: block if `Content` exists. | `TaskViewSet.destroy` + `bulk_delete` (writer views). |
| `Content` (`writer.Content`) | Published content or `ContentClusterMap`/`Images` referencing it (depending on policy) | "Cannot delete Content 'TITLE' — images or content mappings exist. Delete images/mappings first." | If allowed, `ContentClusterMap` entries cascade; update any upstream mapping flags if needed (e.g., cluster mapped counts). | `ContentViewSet.destroy` + `bulk_delete` (writer views). |
**API behavior / Error format**
- Single delete (DELETE /{module}/{id}/): return HTTP 400 with JSON:
- { "success": false, "error": "Cannot delete X: mapped records present", "details": [ { "module": "ContentIdeas", "id": 123, "title": "Idea title" }, ... ], "message": "Delete the listed records in module Y before deleting this." }
- Bulk delete (POST /.../bulk_delete/): iterate IDs and return per-id result:
- { "deleted": [1,2], "skipped": [ { "id": 3, "reason": "Mapped records exist", "details": [...] }, ... ] }
**Simple Helper (design, pseudo-code)**
- Add a small helper util `check_delete_blockers(instance)` that:
- Accepts a model instance (Cluster/Keyword/Idea/Task/Content).
- Runs fast `.exists()` queries for known upstream relationships.
- Returns list of blocking tuples: [(module_name, id, title_field), ...] or empty list if safe.
- Implement in `igny8_core.modules.planner.utils` or `igny8_core.api.utils` for reuse.
Example (concise Python-like pseudocode):
```python
def check_delete_blockers_cluster(cluster):
blockers = []
# Ideas
qs = ContentIdeas.objects.filter(keyword_cluster=cluster).values('id','idea_title')[:5]
blockers += [('ContentIdeas', r['id'], r['idea_title']) for r in qs]
# Writer mappings
from igny8_core.business.content.models import ContentClusterMap, Tasks, Content
if ContentClusterMap.objects.filter(cluster=cluster).exists():
cm = ContentClusterMap.objects.filter(cluster=cluster).values('id','content_id')[:5]
# map to titles if possible
if Tasks.objects.filter(cluster=cluster).exists():
t = Tasks.objects.filter(cluster=cluster).values('id','title')[:5]
blockers += [('Tasks', r['id'], r['title']) for r in t]
if Content.objects.filter(cluster=cluster).exists():
c = Content.objects.filter(cluster=cluster).values('id','title')[:5]
blockers += [('Content', r['id'], r['title']) for r in c]
return blockers
```
**Using the helper in ViewSets**
- In `destroy(self, request, *args, **kwargs)`:
- instance = self.get_object()
- blockers = check_delete_blockers_{model}(instance)
- if blockers: return error_response(...) with details and message
- else: proceed with `instance.delete()` and after delete run downstream simple updates:
- e.g., for Cluster: Keywords.objects.filter(cluster=instance).update(status='new', cluster=None)
- (Because cluster FK uses `SET_NULL` in models, you might need to find keywords that had cluster==pk before deletion. Best approach: perform updates before deletion: Keywords.objects.filter(cluster=instance).update(cluster=None, status='new') then delete cluster to avoid race.)
**Bulk delete approach**
- For each id:
- Try to fetch object
- Run the same `check_delete_blockers`
- If blockers -> add to `skipped` with reasons
- Else -> perform controlled update of downstream fields (set cluster->NULL, status->default) then delete, add to `deleted`
- Return both lists; HTTP 200.
**Downstream status update (simple):**
- Before deleting an upstream record, find child records and set their status to default (from model definitions). Examples:
- When deleting `Cluster`:
- Keywords: `Keywords.objects.filter(cluster=cluster).update(cluster=None, status='new')`
- ContentIdeas: if cluster deletion allowed, they shouldn't exist (we block), but if model had SET_NULL, ensure consistent state.
- When deleting `Keyword`:
- Remove from `ContentIdeas.keyword_objects` M2M (or block if used in ideas per policy).
- Important: run these updates inside a DB transaction so partial states do not occur.
**User-friendly message examples**
- Single delete error:
- "Cannot delete Cluster 'Mobile Apps' — mapped records exist: ContentIdeas: 'How to build mobile apps' (ID 412), Tasks: 'Write mobile apps guide' (ID 776). Delete those first."
- Bulk delete report:
- { "deleted": [5,7], "skipped": [ {"id":3,"reason":"Mapped records exist","details":[{"module":"ContentIdeas","id":412,"title":"How to build mobile apps"}]} ] }
**Testing to add (minimal)**
- Unit tests:
- Attempt to delete a `Cluster` with a `ContentIdea` pointing to it -> assert 400 and error contains idea title.
- Delete a `Cluster` with no upstream -> perform deletion and assert `Keywords` that referenced cluster now have `status == 'new'` and `cluster == None`.
- Bulk-delete mix of deletable and blocked items -> assert correct `deleted`/`skipped` lists.
- Integration tests for planner → writer interactions:
- Create cluster → create content mapping → attempt cluster delete -> assert blocked.
**Developer notes & safe rollout**
- Implement API-level checks first (fast). This prevents UI/API deletes and gives friendly messages. Tests added before merging. No DB schema changes needed.
- Optional later improvement: add a small `StatusHistory` or `previous_status` if precise restoration ever required.
- For backwards compatibility: existing code that uses `.delete()` outside viewsets (e.g., admin or scripts) won't get the friendly error unless we also add `pre_delete` model handlers — recommend adding `pre_delete` signals after the API behavior is in place.
**Minimal code snippet to return an error in a ViewSet (DRF style)**
```python
from igny8_core.api.response import error_response, success_response
def destroy(self, request, *args, **kwargs):
instance = self.get_object()
blockers = check_delete_blockers_cluster(instance)
if blockers:
details = [{'module': m, 'id': id, 'title': title} for m,id,title in blockers]
return error_response(
error=f"Cannot delete Cluster '{instance.name}' - mapped records exist",
request=request,
status_code=status.HTTP_400_BAD_REQUEST,
data={'details': details, 'message': 'Delete the listed records first'}
)
# Safe path: update child records then delete inside transaction
with transaction.atomic():
Keywords.objects.filter(cluster=instance).update(cluster=None, status='new')
instance.delete()
return success_response(data={'deleted_id': kwargs.get('pk')}, request=request)
```
**Next steps (recommended immediate actions)**
- I can implement Phase 1 changes (API-level checks + bulk_delete reporting + tests) now. This is low-risk and requires no migrations.
- If you want, I will:
- Implement helper utils, update `ClusterViewSet.destroy` and `bulk_delete` and `KeywordViewSet` similarly, and add unit tests for the blocking behavior.
- Run tests locally and show results.
Which immediate action do you want me to take now:
- A) Implement Phase 1 (API-level checks + bulk delete + tests), or
- B) Only produce the final doc (above) for handoff and stop here?
bluesalman: only make th edocument
GitHub Copilot: **Safety Deletion Policy — Backward-Deletion Rules & Implementation Doc**
**Purpose:**
- Define a minimal, safe deletion policy: forbid deletes when there are currently associated/mapped upstream records; provide a clear error message listing blocking upstream records and how to resolve; when a deletion is permitted, update downstream statuses to the module default (no historical audit).
- This is an API-level safety document only — no DB audit/history required.
**Scope:**
- Planner + Writer modules (primary entities): `Clusters`, `Keywords`, `ContentIdeas`, `Tasks`, `Content`, `Images`.
- Applies to single-record `DELETE` and API `bulk_delete` actions.
- Behavior must be consistent and user-friendly (clear messages).
**High-level Rules (short):**
- Block delete if any upstream/mapped record exists. Return HTTP 400 with structured details (module, id, title) and an instruction to delete upstream records first.
- For an allowed delete: in a DB transaction, update affected downstream child records to model defaults (e.g., `Keywords.status = 'new'`) then delete the record.
- Bulk delete: iterate IDs, perform same checks per item, do not abort whole batch. Return per-id results (deleted/skipped with reason).
**Entity Rules Table**
| Entity | Blocks Deletion If (upstream exists) | Error Message (client) | On Delete (allowed) — downstream updates | Implementation Notes |
|---|---:|---|---|---|
| Cluster (`planner.Clusters`) | Any mapped items: `ContentIdeas` (keyword_cluster), `writer.ContentClusterMap`, `writer.Tasks` with `cluster`, `writer.Content` referencing cluster | "Cannot delete Cluster 'NAME' — mapped records exist in: ContentIdeas: 'Idea title' (ID). Delete those first." | Before delete: set `Keywords` referencing cluster → `cluster = NULL` and `status = 'new'` (model default). Then delete cluster. | Implement in `ClusterViewSet.destroy` and `bulk_delete`. Use helper `check_delete_blockers(cluster)`. Run updates inside a `transaction.atomic()` and perform child updates before `instance.delete()` to avoid orphan visibility. |
| Keyword (`planner.Keywords`) | `keyword.cluster != NULL` OR M2M `ContentIdeas` referencing this keyword | "Cannot delete Keyword 'TERM' — mapped to Cluster 'CLUSTER' and used in Idea 'IDEA_TITLE'. Delete those first." | If allowed, remove M2M references (orphan removal). No status rollback needed for upstream. | Implement in `KeywordViewSet.destroy` + `bulk_delete`. For M2M, block deletion unless explicit policy allows cascade. |
| ContentIdea (`planner.ContentIdeas`) | `writer.Tasks` where `idea` set; `Content` generated from idea | "Cannot delete Idea 'TITLE' — tasks exist: 'Task title'. Delete those tasks first." | If allowed, clear M2M `keyword_objects`. | Implement in `ContentIdeasViewSet`. Block if `Tasks.exists()`; otherwise delete inside transaction. |
| Task (`writer.Tasks`) | Exists `Content` generated from the task OR final images attached | "Cannot delete Task 'TITLE' — content exists: 'Content title'. Delete that content first." | If allowed, delete or clear `Images` per policy (prefer blocking if final content exists). | Implement in writer `TaskViewSet`. Prefer strict block if `Content` exists. |
| Content (`writer.Content`) | Published content or `ContentClusterMap`/`Images` referencing it | "Cannot delete Content 'TITLE' — images or mappings exist. Delete them first." | If allowed, delete `ContentClusterMap` entries (they `CASCADE`) and `Images` as per current model behavior. Update cluster or counters if needed. | Implement in writer `ContentViewSet`. Consider blocking published content deletion by policy. |
| Images (`writer.Images`) | Typically safe to delete unless other policies apply (e.g., external sync) | "Cannot delete Image 'ID' — blocked by external sync." | If allowed, delete image record. | Implement in `ImagesViewSet` if present; otherwise admin allowed. |
**API Response Examples**
- Single-delete error (HTTP 400):
{
"success": false,
"error": "Cannot delete Cluster 'Mobile Apps' — mapped records exist",
"details": [
{"module": "ContentIdeas", "id": 412, "title": "How to build mobile apps"},
{"module": "Tasks", "id": 776, "title": "Write mobile apps guide"}
],
"message": "Delete the listed records in their modules first, then retry."
}
- Bulk delete response (HTTP 200):
{
"deleted": [5,7],
"skipped": [
{"id":3,"reason":"Mapped records exist","details":[{"module":"ContentIdeas","id":412,"title":"How to build mobile apps"}]}
],
"message":"Some items skipped due to mapped records. See details."
}
- Success (single delete):
{
"success": true,
"data": {"deleted_id": 12}
}
**Helper design (concise)**
- New small utility in `igny8_core.api.utils` (or `igny8_core.modules.planner.utils`) with functions:
- `check_delete_blockers_for_cluster(cluster) -> List[dict]`
- `check_delete_blockers_for_keyword(keyword) -> List[dict]`
- (others as needed)
- Implementation pattern:
- Fast `.exists()` queries and `.values('id', 'title_field')[:5]` to show up to N blockers (avoid heavy scans).
- Return a short list of blockers with module name, id, display title.
Example pseudocode:
def check_delete_blockers_cluster(cluster):
blockers = []
ideas = ContentIdeas.objects.filter(keyword_cluster=cluster).values('id','idea_title')[:5]
blockers += [ {'module':'ContentIdeas','id':r['id'],'title':r['idea_title']} for r in ideas ]
tasks = Tasks.objects.filter(cluster=cluster).values('id','title')[:5]
blockers += [ {'module':'Tasks','id':r['id'],'title':r['title']} for r in tasks ]
if ContentClusterMap.objects.filter(cluster=cluster).exists():
# get sample content titles
return blockers
**Delete execution flow (safe sequence)**
- For single delete in ViewSet:
1. Fetch instance via `self.get_object()`.
2. Run `blockers = check_delete_blockers_*`.
3. If `blockers`: return `error_response` (HTTP 400) with `details`.
4. Else: inside `transaction.atomic()`:
- Update children to defaults BEFORE deleting upstream:
- Example: `Keywords.objects.filter(cluster=instance).update(cluster=None, status='new')`
- Call `instance.delete()`.
- Return success response.
- For bulk delete:
- Iterate ids; for each id, apply same procedure; collect `deleted` and `skipped` with reasons; return both lists. Do not abort entire batch on one failure.
**Downstream status update rules (simple fallback)**
- When an upstream mapping is removed (allowed delete):
- Use model default for downstream status:
- `Keywords` default: `'new'` → set status `'new'`.
- `Clusters` default: `'new'` (if applicable).
- `ContentIdeas` default: `'new'`.
- `Tasks` default: `'queued'`.
- `Content` default: `'draft'`.
- `Images` default: `'pending'`.
- Implementation: set downstream `status` explicitly in the pre-delete update inside the same transaction.
**DB & Concurrency notes**
- Run updates + delete in a single `transaction.atomic()` to avoid transient inconsistent states.
- Prefer updating children before deletion to avoid races where FK ON DELETE behavior may set NULL after delete. Example: call `Keywords.objects.filter(cluster=instance).update(cluster=None, status='new')` then `instance.delete()`.
**Testing checklist (minimal, priority)**
- Block tests:
- Attempt to delete a `Cluster` that has `ContentIdeas` → assert HTTP 400 and response includes idea title.
- Bulk delete mix of deletable and blocked clusters → assert deleted/skipped counts & reasons.
- Allowed-delete tests:
- Delete a `Cluster` with no blockers → assert `Keywords` previously referencing it have `status == 'new'` and `cluster == None`.
- Regression check:
- Ensure existing `bulk_delete` endpoints preserve format but return the enhanced report.
**Developer implementation notes**
- Where to change:
- Planner viewset: views.py — implement `destroy` override and enhanced `bulk_delete`.
- Writer viewsets: `backend/igny8_core/modules/writer/*` (or business/content viewsets) — add `destroy` + `bulk_delete` checks similarly.
- Add reusable helper module: `backend/igny8_core/api/utils/delete_checks.py` (or similar).
- No DB migrations are required for the minimal policy.
- Optional later improvement: add `StatusHistory` model if precise original-status restoration is desired.
**UX / Messaging guidance**
- Keep messages short and actionable: include the blocking module, record title, and next step (delete the blocking records first). Example:
- "Cannot delete Cluster 'NAME' — mapped records exist in ContentIdeas: 'Idea title' (ID). Delete those items in Planner → Ideas first, then retry."
- For bulk operations, provide counts and a link or list to the specific blocking records.
**Rollout / Deployment**
- Implement API-level checks and tests, deploy to staging. Verify UI flows (single and bulk deletes) show messages and blocked items correctly.
- After staging verification, deploy to production. No DB schema migration required.
---
If you want, I can now:
- Produce a ready-to-copy developer ticket/PR patch that adds the helper and one example ViewSet implementation (e.g., `ClusterViewSet.destroy` + `bulk_delete`) and unit tests — small, safe Phase 1 patch.
- Or stop here with this document for handoff.
Which do you prefer?