docs updates
This commit is contained in:
IGNY8 VPS (Salman)
@@ -1,335 +0,0 @@
|
||||
# Status Implementation Tables - Complete Reference
|
||||
|
||||
**Date:** December 3, 2025
|
||||
|
||||
---
|
||||
|
||||
## 🎯 KEYWORDS MODULE
|
||||
|
||||
### Backend Model
|
||||
**File:** `/backend/igny8_core/business/planning/models.py`
|
||||
|
||||
| Field | Type | Choices | Default | Required | Description |
|
||||
|-------|------|---------|---------|----------|-------------|
|
||||
| `status` | CharField(50) | `new`, `mapped` | `new` | ✅ Yes | Workflow status |
|
||||
| `disabled` | BooleanField | - | `False` | ✅ Yes | Filter control (exclude from processes) |
|
||||
|
||||
---
|
||||
|
||||
### Frontend Table Column
|
||||
**File:** `/frontend/src/config/pages/keywords.config.tsx` (Lines ~230-248)
|
||||
|
||||
| Status Value | Badge Color | Badge Label | Display When |
|
||||
|--------------|-------------|-------------|--------------|
|
||||
| `new` | Amber/Yellow | New | Keyword not yet clustered |
|
||||
| `mapped` | Green | Mapped | Keyword assigned to cluster |
|
||||
| *(disabled=true)* | Red/Gray | Disabled | User manually disabled (optional display) |
|
||||
|
||||
---
|
||||
|
||||
### Frontend Filter Dropdown
|
||||
**File:** `/frontend/src/config/pages/keywords.config.tsx` (Lines ~310-318)
|
||||
|
||||
| Filter Type | Options | Default | Description |
|
||||
|-------------|---------|---------|-------------|
|
||||
| Status Select | `new`, `mapped` | All | Workflow status filter |
|
||||
| Disabled Checkbox | Show/Hide disabled | Hide disabled | Filter control |
|
||||
|
||||
---
|
||||
|
||||
### Frontend Form Fields
|
||||
**File:** `/frontend/src/config/pages/keywords.config.tsx` (Lines ~560-570)
|
||||
|
||||
| Field | Type | Options | Default | Required | Editable |
|
||||
|-------|------|---------|---------|----------|----------|
|
||||
| Status | Select | `new`, `mapped` | `new` | ✅ Yes | ✅ Yes |
|
||||
| Disabled | Checkbox | true/false | `false` | ❌ No | ✅ Yes |
|
||||
|
||||
---
|
||||
|
||||
## 🎯 CLUSTERS MODULE
|
||||
|
||||
### Backend Model
|
||||
**File:** `/backend/igny8_core/business/planning/models.py`
|
||||
|
||||
| Field | Type | Choices | Default | Required | Description |
|
||||
|-------|------|---------|---------|----------|-------------|
|
||||
| `status` | CharField(50) | `new`, `mapped` | `new` | ✅ Yes | Workflow status |
|
||||
| `disabled` | BooleanField | - | `False` | ✅ Yes | Filter control (exclude from processes) |
|
||||
|
||||
---
|
||||
|
||||
### Frontend Table Column
|
||||
**File:** `/frontend/src/config/pages/clusters.config.tsx` (Lines ~190-200)
|
||||
|
||||
| Status Value | Badge Color | Badge Label | Display When |
|
||||
|--------------|-------------|-------------|--------------|
|
||||
| `new` | Amber/Yellow | New | Cluster created, no ideas generated yet |
|
||||
| `mapped` | Green | Mapped | Ideas generated from cluster |
|
||||
| *(disabled=true)* | Red/Gray | Disabled | User manually disabled (optional display) |
|
||||
|
||||
---
|
||||
|
||||
### Frontend Filter Dropdown
|
||||
**File:** `/frontend/src/config/pages/clusters.config.tsx` (Lines ~240-253)
|
||||
|
||||
| Filter Type | Options | Default | Description |
|
||||
|-------------|---------|---------|-------------|
|
||||
| Status Select | `new`, `mapped` | All | Workflow status filter |
|
||||
| Disabled Checkbox | Show/Hide disabled | Hide disabled | Filter control |
|
||||
|
||||
---
|
||||
|
||||
### Frontend Form Fields
|
||||
**File:** `/frontend/src/config/pages/clusters.config.tsx` (Lines ~405-418)
|
||||
|
||||
| Field | Type | Options | Default | Required | Editable |
|
||||
|-------|------|---------|---------|----------|----------|
|
||||
| Status | Select | `new`, `mapped` | `new` | ✅ Yes | ✅ Yes |
|
||||
| Disabled | Checkbox | true/false | `false` | ❌ No | ✅ Yes |
|
||||
|
||||
---
|
||||
|
||||
## 🎯 IDEAS MODULE
|
||||
|
||||
### Backend Model
|
||||
**File:** `/backend/igny8_core/business/planning/models.py`
|
||||
|
||||
| Field | Type | Choices | Default | Required | Description |
|
||||
|-------|------|---------|---------|----------|-------------|
|
||||
| `status` | CharField(50) | `new`, `queued`, `completed` | `new` | ✅ Yes | Workflow status |
|
||||
| `disabled` | BooleanField | - | `False` | ✅ Yes | Filter control (exclude from processes) |
|
||||
|
||||
---
|
||||
|
||||
### Frontend Table Column
|
||||
**File:** `/frontend/src/config/pages/ideas.config.tsx` (Lines ~170-185)
|
||||
|
||||
| Status Value | Badge Color | Badge Label | Display When |
|
||||
|--------------|-------------|-------------|--------------|
|
||||
| `new` | Amber/Yellow | New | Idea generated, not queued yet |
|
||||
| `queued` | Blue | Queued | Task created in Writer module |
|
||||
| `completed` | Green | Completed | Content generated (Task completed) |
|
||||
| *(disabled=true)* | Red/Gray | Disabled | User manually disabled (optional display) |
|
||||
|
||||
---
|
||||
|
||||
### Frontend Filter Dropdown
|
||||
**File:** `/frontend/src/config/pages/ideas.config.tsx` (Lines ~218-228)
|
||||
|
||||
| Filter Type | Options | Default | Description |
|
||||
|-------------|---------|---------|-------------|
|
||||
| Status Select | `new`, `queued`, `completed` | All | Workflow status filter |
|
||||
| Disabled Checkbox | Show/Hide disabled | Hide disabled | Filter control |
|
||||
|
||||
---
|
||||
|
||||
### Frontend Form Fields
|
||||
**File:** `/frontend/src/config/pages/ideas.config.tsx` (Lines ~372-385)
|
||||
|
||||
| Field | Type | Options | Default | Required | Editable |
|
||||
|-------|------|---------|---------|----------|----------|
|
||||
| Status | Select | `new`, `queued`, `completed` | `new` | ✅ Yes | ✅ Yes |
|
||||
| Disabled | Checkbox | true/false | `false` | ❌ No | ✅ Yes |
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
# 🔄 STATUS TRANSITION TABLES
|
||||
|
||||
## KEYWORDS Status Transitions
|
||||
|
||||
| Current Status | Trigger/Action | Next Status | Auto/Manual | Updated By |
|
||||
|----------------|----------------|-------------|-------------|------------|
|
||||
| `new` | AI auto_cluster runs | `mapped` | 🤖 Auto | `auto_cluster.py` Line 297 |
|
||||
| `new` | User manually assigns to cluster | `mapped` | 👤 Manual | User form edit |
|
||||
| `new` | User toggles disabled | *(stays new)* + `disabled=true` | 👤 Manual | User form edit |
|
||||
| `mapped` | User toggles disabled | *(stays mapped)* + `disabled=true` | 👤 Manual | User form edit |
|
||||
| *(any)* | User re-enables | *(stays same)* + `disabled=false` | 👤 Manual | User form edit |
|
||||
|
||||
**Workflow Path:**
|
||||
```
|
||||
new ──[auto_cluster AI]──> mapped
|
||||
│
|
||||
└──[user toggle disabled]──> (status unchanged, disabled flag set)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLUSTERS Status Transitions
|
||||
|
||||
| Current Status | Trigger/Action | Next Status | Auto/Manual | Updated By |
|
||||
|----------------|----------------|-------------|-------------|------------|
|
||||
| `new` | AI generate_ideas runs | `mapped` | 🤖 Auto | `generate_ideas.py` (new code) |
|
||||
| `new` | User manually creates ideas | `mapped` | 👤 Manual | User workflow |
|
||||
| `new` | User toggles disabled | *(stays new)* + `disabled=true` | 👤 Manual | User form edit |
|
||||
| `mapped` | User toggles disabled | *(stays mapped)* + `disabled=true` | 👤 Manual | User form edit |
|
||||
| *(any)* | User re-enables | *(stays same)* + `disabled=false` | 👤 Manual | User form edit |
|
||||
|
||||
**Workflow Path:**
|
||||
```
|
||||
new ──[generate_ideas AI]──> mapped
|
||||
│
|
||||
└──[user toggle disabled]──> (status unchanged, disabled flag set)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## IDEAS Status Transitions
|
||||
|
||||
| Current Status | Trigger/Action | Next Status | Auto/Manual | Updated By |
|
||||
|----------------|----------------|-------------|-------------|------------|
|
||||
| `new` | User bulk-queues ideas to writer | `queued` | 👤 Manual | `views.py` Line 1084 |
|
||||
| `queued` | Writer AI generates content | `completed` | 🤖 Auto | `generate_content.py` Line 318 (syncs from Task) |
|
||||
| `queued` | Task status becomes 'completed' | `completed` | 🤖 Auto | Auto-sync from Task.status |
|
||||
| `new` | User toggles disabled | *(stays new)* + `disabled=true` | 👤 Manual | User form edit |
|
||||
| `queued` | User toggles disabled | *(stays queued)* + `disabled=true` | 👤 Manual | User form edit |
|
||||
| `completed` | User toggles disabled | *(stays completed)* + `disabled=true` | 👤 Manual | User form edit |
|
||||
| *(any)* | User re-enables | *(stays same)* + `disabled=false` | 👤 Manual | User form edit |
|
||||
|
||||
**Workflow Path:**
|
||||
```
|
||||
new ──[user bulk_queue]──> queued ──[generate_content AI]──> completed
|
||||
│ │ │
|
||||
└──[user toggle disabled]────┴───────────[user toggle]──────────┘
|
||||
(status unchanged, disabled flag set)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
# 📊 COMBINED STATUS OVERVIEW
|
||||
|
||||
## All Modules - Status Values
|
||||
|
||||
| Module | Workflow Statuses | Filter Status | Total Unique Values |
|
||||
|--------|-------------------|---------------|---------------------|
|
||||
| **Keywords** | `new`, `mapped` | `disabled` (boolean) | 2 status + 1 flag |
|
||||
| **Clusters** | `new`, `mapped` | `disabled` (boolean) | 2 status + 1 flag |
|
||||
| **Ideas** | `new`, `queued`, `completed` | `disabled` (boolean) | 3 status + 1 flag |
|
||||
|
||||
---
|
||||
|
||||
## Process Inclusion Matrix
|
||||
|
||||
| Status | Auto-Cluster AI | Generate-Ideas AI | Bulk-Queue | Generate-Content AI | Dashboard Metrics |
|
||||
|--------|-----------------|-------------------|------------|---------------------|-------------------|
|
||||
| Keywords: `new` | ✅ Included | ❌ N/A | ❌ N/A | ❌ N/A | ✅ Counted |
|
||||
| Keywords: `mapped` | ❌ Already clustered | ✅ Used for ideas | ❌ N/A | ❌ N/A | ✅ Counted |
|
||||
| Keywords: `disabled=true` | ❌ Excluded | ❌ Excluded | ❌ N/A | ❌ N/A | ❌ Excluded |
|
||||
| Clusters: `new` | ❌ N/A | ✅ Included | ❌ N/A | ❌ N/A | ✅ Counted |
|
||||
| Clusters: `mapped` | ❌ N/A | ❌ Already has ideas | ✅ Ideas can queue | ❌ N/A | ✅ Counted |
|
||||
| Clusters: `disabled=true` | ❌ N/A | ❌ Excluded | ❌ Excluded | ❌ N/A | ❌ Excluded |
|
||||
| Ideas: `new` | ❌ N/A | ❌ N/A | ✅ Included | ❌ N/A | ✅ Counted |
|
||||
| Ideas: `queued` | ❌ N/A | ❌ N/A | ❌ Already queued | ✅ Included | ✅ Counted |
|
||||
| Ideas: `completed` | ❌ N/A | ❌ N/A | ❌ Already done | ❌ Already done | ✅ Counted |
|
||||
| Ideas: `disabled=true` | ❌ N/A | ❌ N/A | ❌ Excluded | ❌ Excluded | ❌ Excluded |
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
# 🔄 DATA MIGRATION TRANSITION TABLE
|
||||
|
||||
## Keywords Migration
|
||||
|
||||
| Old Status | New Status | New Disabled Flag | Logic |
|
||||
|------------|------------|-------------------|-------|
|
||||
| `pending` | `new` | `false` | Direct mapping |
|
||||
| `active` | `mapped` | `false` | Keyword was clustered |
|
||||
| `archived` | `mapped` | `true` | Preserve data, mark as disabled |
|
||||
|
||||
**SQL Preview:**
|
||||
```sql
|
||||
-- Keywords migration
|
||||
UPDATE igny8_keywords
|
||||
SET status = 'new', disabled = false
|
||||
WHERE status = 'pending';
|
||||
|
||||
UPDATE igny8_keywords
|
||||
SET status = 'mapped', disabled = false
|
||||
WHERE status = 'active';
|
||||
|
||||
UPDATE igny8_keywords
|
||||
SET status = 'mapped', disabled = true
|
||||
WHERE status = 'archived';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Clusters Migration
|
||||
|
||||
| Old Status | New Status | New Disabled Flag | Logic |
|
||||
|------------|------------|-------------------|-------|
|
||||
| `active` (with ideas_count > 0) | `mapped` | `false` | Cluster has ideas |
|
||||
| `active` (with ideas_count = 0) | `new` | `false` | Cluster has no ideas yet |
|
||||
| *(no archived status exists)* | - | - | - |
|
||||
|
||||
**SQL Preview:**
|
||||
```sql
|
||||
-- Clusters migration
|
||||
UPDATE igny8_clusters
|
||||
SET status = 'mapped', disabled = false
|
||||
WHERE ideas_count > 0;
|
||||
|
||||
UPDATE igny8_clusters
|
||||
SET status = 'new', disabled = false
|
||||
WHERE ideas_count = 0;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ideas Migration
|
||||
|
||||
| Old Status | New Status | New Disabled Flag | Logic |
|
||||
|------------|------------|-------------------|-------|
|
||||
| `new` | `new` | `false` | No change |
|
||||
| `scheduled` | `queued` | `false` | Rename to match writer workflow |
|
||||
| `published` | `completed` | `false` | Publishing is separate deployment |
|
||||
| *(no archived status)* | - | - | - |
|
||||
|
||||
**SQL Preview:**
|
||||
```sql
|
||||
-- Ideas migration
|
||||
UPDATE igny8_content_ideas
|
||||
SET status = 'new', disabled = false
|
||||
WHERE status = 'new';
|
||||
|
||||
UPDATE igny8_content_ideas
|
||||
SET status = 'queued', disabled = false
|
||||
WHERE status = 'scheduled';
|
||||
|
||||
UPDATE igny8_content_ideas
|
||||
SET status = 'completed', disabled = false
|
||||
WHERE status = 'published';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
# 📋 SUMMARY COMPARISON TABLE
|
||||
|
||||
## Before vs After
|
||||
|
||||
| Module | Current Status Values | New Status Values | Current Filter | New Filter |
|
||||
|--------|----------------------|-------------------|----------------|------------|
|
||||
| **Keywords** | `pending`, `active`, `archived` | `new`, `mapped` | Status dropdown | Status dropdown + disabled checkbox |
|
||||
| **Clusters** | `active` (hardcoded, no choices) | `new`, `mapped` | Status dropdown | Status dropdown + disabled checkbox |
|
||||
| **Ideas** | `new`, `scheduled`, `published` | `new`, `queued`, `completed` | Status dropdown | Status dropdown + disabled checkbox |
|
||||
|
||||
---
|
||||
|
||||
## Status Count Changes
|
||||
|
||||
| Module | Before | After | Change |
|
||||
|--------|--------|-------|--------|
|
||||
| **Keywords** | 3 status values | 2 workflow + 1 boolean flag | -1 status (simplified) |
|
||||
| **Clusters** | 1 hardcoded value | 2 workflow + 1 boolean flag | +1 status (proper choices) |
|
||||
| **Ideas** | 3 status values | 3 workflow + 1 boolean flag | Same count (renamed) |
|
||||
|
||||
---
|
||||
|
||||
**END OF IMPLEMENTATION TABLES**
|
||||
1150
docs/automation/AI-FUNCTIONS-COMPLETE-REFERENCE.md
Normal file
1150
docs/automation/AI-FUNCTIONS-COMPLETE-REFERENCE.md
Normal file
File diff suppressed because it is too large
Load Diff
@@ -1,299 +0,0 @@
|
||||
# Automation Implementation - Deployment Checklist
|
||||
|
||||
## ✅ Completed Components
|
||||
|
||||
### Backend
|
||||
- [x] Database models created (`AutomationConfig`, `AutomationRun`)
|
||||
- [x] AutomationLogger service (file-based logging)
|
||||
- [x] AutomationService orchestrator (7-stage pipeline)
|
||||
- [x] API endpoints (`AutomationViewSet`)
|
||||
- [x] Celery tasks (scheduled checks, run execution, resume)
|
||||
- [x] URL routing registered
|
||||
- [x] Celery beat schedule configured
|
||||
- [x] Migration file created
|
||||
|
||||
### Frontend
|
||||
- [x] TypeScript API service (`automationService.ts`)
|
||||
- [x] Main dashboard page (`AutomationPage.tsx`)
|
||||
- [x] StageCard component
|
||||
- [x] ActivityLog component
|
||||
- [x] ConfigModal component
|
||||
- [x] RunHistory component
|
||||
|
||||
### Documentation
|
||||
- [x] Comprehensive README (`AUTOMATION-IMPLEMENTATION-README.md`)
|
||||
- [x] Original plan corrected (`automation-plan.md`)
|
||||
|
||||
## ⏳ Remaining Tasks
|
||||
|
||||
### 1. Run Database Migration
|
||||
|
||||
```bash
|
||||
cd /data/app/igny8/backend
|
||||
python manage.py migrate
|
||||
```
|
||||
|
||||
This will create the `automation_config` and `automation_run` tables.
|
||||
|
||||
### 2. Register Frontend Route
|
||||
|
||||
Add to your React Router configuration:
|
||||
|
||||
```typescript
|
||||
import AutomationPage from './pages/Automation/AutomationPage';
|
||||
|
||||
// In your route definitions:
|
||||
{
|
||||
path: '/automation',
|
||||
element: <AutomationPage />,
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Add Navigation Link
|
||||
|
||||
Add link to main navigation menu:
|
||||
|
||||
```typescript
|
||||
{
|
||||
name: 'Automation',
|
||||
href: '/automation',
|
||||
icon: /* automation icon */
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Verify Infrastructure
|
||||
|
||||
**Celery Worker**
|
||||
```bash
|
||||
# Check if running
|
||||
docker ps | grep celery
|
||||
|
||||
# Start if needed
|
||||
docker-compose up -d celery
|
||||
```
|
||||
|
||||
**Celery Beat**
|
||||
```bash
|
||||
# Check if running
|
||||
docker ps | grep beat
|
||||
|
||||
# Start if needed
|
||||
docker-compose up -d celery-beat
|
||||
```
|
||||
|
||||
**Redis/Cache**
|
||||
```bash
|
||||
# Verify cache backend in settings.py
|
||||
CACHES = {
|
||||
'default': {
|
||||
'BACKEND': 'django.core.cache.backends.redis.RedisCache',
|
||||
'LOCATION': 'redis://redis:6379/1',
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Create Log Directory
|
||||
|
||||
```bash
|
||||
mkdir -p /data/app/igny8/backend/logs/automation
|
||||
chmod 755 /data/app/igny8/backend/logs/automation
|
||||
```
|
||||
|
||||
### 6. Test API Endpoints
|
||||
|
||||
```bash
|
||||
# Get config (should return default config)
|
||||
curl -X GET "http://localhost:8000/api/v1/automation/config/?site_id=1" \
|
||||
-H "Authorization: Bearer YOUR_TOKEN"
|
||||
|
||||
# Estimate credits
|
||||
curl -X GET "http://localhost:8000/api/v1/automation/estimate/?site_id=1" \
|
||||
-H "Authorization: Bearer YOUR_TOKEN"
|
||||
```
|
||||
|
||||
### 7. Test Frontend
|
||||
|
||||
1. Navigate to `/automation` page
|
||||
2. Click [Configure] - modal should open
|
||||
3. Save configuration
|
||||
4. Click [Run Now] - should trigger automation
|
||||
5. Verify real-time updates in stage cards
|
||||
6. Check activity log is streaming
|
||||
|
||||
### 8. Test Scheduled Automation
|
||||
|
||||
1. Enable automation in config
|
||||
2. Set scheduled time to 1 minute from now
|
||||
3. Wait for next hour (beat checks hourly at :00)
|
||||
4. Verify automation starts automatically
|
||||
|
||||
### 9. Monitor First Run
|
||||
|
||||
Watch logs in real-time:
|
||||
|
||||
```bash
|
||||
# Backend logs
|
||||
tail -f /data/app/igny8/backend/logs/automation/{account_id}/{site_id}/{run_id}/automation_run.log
|
||||
|
||||
# Celery worker logs
|
||||
docker logs -f <celery_container>
|
||||
|
||||
# Django logs
|
||||
docker logs -f <backend_container>
|
||||
```
|
||||
|
||||
### 10. Verify Database Records
|
||||
|
||||
```python
|
||||
from igny8_core.business.automation.models import AutomationConfig, AutomationRun
|
||||
|
||||
# Check config created
|
||||
AutomationConfig.objects.all()
|
||||
|
||||
# Check runs recorded
|
||||
AutomationRun.objects.all()
|
||||
|
||||
# View stage results
|
||||
run = AutomationRun.objects.latest('started_at')
|
||||
print(run.stage_1_result)
|
||||
print(run.stage_2_result)
|
||||
# ... etc
|
||||
```
|
||||
|
||||
## Quick Start Commands
|
||||
|
||||
```bash
|
||||
# 1. Run migration
|
||||
cd /data/app/igny8/backend
|
||||
python manage.py migrate
|
||||
|
||||
# 2. Create log directory
|
||||
mkdir -p logs/automation
|
||||
chmod 755 logs/automation
|
||||
|
||||
# 3. Restart services
|
||||
docker-compose restart celery celery-beat
|
||||
|
||||
# 4. Verify Celery beat schedule
|
||||
docker exec <celery_container> celery -A igny8_core inspect scheduled
|
||||
|
||||
# 5. Test automation (Django shell)
|
||||
python manage.py shell
|
||||
>>> from igny8_core.business.automation.services import AutomationService
|
||||
>>> from igny8_core.modules.system.models import Account, Site
|
||||
>>> account = Account.objects.first()
|
||||
>>> site = Site.objects.first()
|
||||
>>> service = AutomationService(account, site)
|
||||
>>> service.estimate_credits() # Should return number
|
||||
>>> # Don't run start_automation() yet - test via UI first
|
||||
```
|
||||
|
||||
## Expected Behavior
|
||||
|
||||
### First Successful Run
|
||||
|
||||
1. **Stage 1**: Process keywords → create clusters (2-5 min)
|
||||
2. **Stage 2**: Generate ideas from clusters (1-2 min per cluster)
|
||||
3. **Stage 3**: Create tasks from ideas (instant)
|
||||
4. **Stage 4**: Generate content from tasks (3-5 min per task)
|
||||
5. **Stage 5**: Extract image prompts (1-2 min per content)
|
||||
6. **Stage 6**: Generate images (2-3 min per image)
|
||||
7. **Stage 7**: Count content ready for review (instant)
|
||||
|
||||
Total time: 15-45 minutes depending on batch sizes
|
||||
|
||||
### Stage Results Example
|
||||
|
||||
```json
|
||||
{
|
||||
"stage_1_result": {
|
||||
"keywords_processed": 20,
|
||||
"clusters_created": 4,
|
||||
"batches_run": 1,
|
||||
"credits_used": 4
|
||||
},
|
||||
"stage_2_result": {
|
||||
"clusters_processed": 4,
|
||||
"ideas_created": 16,
|
||||
"credits_used": 8
|
||||
},
|
||||
"stage_3_result": {
|
||||
"ideas_processed": 16,
|
||||
"tasks_created": 16,
|
||||
"batches_run": 1
|
||||
},
|
||||
"stage_4_result": {
|
||||
"tasks_processed": 16,
|
||||
"content_created": 16,
|
||||
"total_words": 40000,
|
||||
"credits_used": 80
|
||||
},
|
||||
"stage_5_result": {
|
||||
"content_processed": 16,
|
||||
"prompts_created": 64,
|
||||
"credits_used": 32
|
||||
},
|
||||
"stage_6_result": {
|
||||
"images_processed": 64,
|
||||
"images_generated": 64,
|
||||
"content_moved_to_review": 16,
|
||||
"credits_used": 128
|
||||
},
|
||||
"stage_7_result": {
|
||||
"ready_for_review": 16,
|
||||
"content_ids": [1, 2, 3, ...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Module not found" errors
|
||||
- Restart Django server after adding new models
|
||||
- Run `python manage.py collectstatic` if needed
|
||||
|
||||
### "Table does not exist" errors
|
||||
- Run migration: `python manage.py migrate`
|
||||
|
||||
### "No module named automation"
|
||||
- Check `__init__.py` files exist in all directories
|
||||
- Verify imports in `urls.py`
|
||||
|
||||
### Celery tasks not running
|
||||
- Check worker is running: `docker ps | grep celery`
|
||||
- Check beat is running: `docker ps | grep beat`
|
||||
- Verify tasks registered: `celery -A igny8_core inspect registered`
|
||||
|
||||
### Logs not appearing
|
||||
- Check directory permissions: `ls -la logs/automation`
|
||||
- Check AutomationLogger.start_run() creates directories
|
||||
- Verify log file path in code matches actual filesystem
|
||||
|
||||
### Frontend errors
|
||||
- Check API service imported correctly
|
||||
- Verify route registered in router
|
||||
- Check for TypeScript compilation errors
|
||||
- Verify API endpoints returning expected data
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Migration runs without errors
|
||||
- [ ] Frontend `/automation` page loads
|
||||
- [ ] Config modal opens and saves
|
||||
- [ ] Credit estimate shows reasonable number
|
||||
- [ ] "Run Now" starts automation successfully
|
||||
- [ ] Stage cards update in real-time
|
||||
- [ ] Activity log shows progress
|
||||
- [ ] All 7 stages complete successfully
|
||||
- [ ] Content moved to review status
|
||||
- [ ] Run History table shows completed run
|
||||
- [ ] Scheduled automation triggers at configured time
|
||||
|
||||
## Post-Deployment
|
||||
|
||||
1. Monitor first few runs closely
|
||||
2. Adjust batch sizes based on performance
|
||||
3. Set up alerts for failed runs
|
||||
4. Document any issues encountered
|
||||
5. Train users on automation features
|
||||
6. Gather feedback for improvements
|
||||
1053
docs/automation/AUTOMATION-IMPLEMENTATION-ANALYSIS-CORRECTED.md
Normal file
1053
docs/automation/AUTOMATION-IMPLEMENTATION-ANALYSIS-CORRECTED.md
Normal file
File diff suppressed because it is too large
Load Diff
@@ -1,383 +0,0 @@
|
||||
# AI Automation Pipeline - Implementation Complete
|
||||
|
||||
## Overview
|
||||
|
||||
The IGNY8 AI Automation Pipeline is a fully automated content creation system that orchestrates existing AI functions into a 7-stage pipeline, transforming keywords into published content without manual intervention.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Backend Components
|
||||
|
||||
#### 1. Models (`/backend/igny8_core/business/automation/models.py`)
|
||||
|
||||
**AutomationConfig**
|
||||
- Per-site configuration for automation
|
||||
- Fields: `is_enabled`, `frequency` (daily/weekly/monthly), `scheduled_time`, batch sizes for all 7 stages
|
||||
- OneToOne relationship with Site model
|
||||
|
||||
**AutomationRun**
|
||||
- Tracks execution of automation runs
|
||||
- Fields: `run_id`, `status`, `current_stage`, `stage_1_result` through `stage_7_result` (JSON), `total_credits_used`
|
||||
- Status choices: running, paused, completed, failed
|
||||
|
||||
#### 2. Services
|
||||
|
||||
**AutomationLogger** (`services/automation_logger.py`)
|
||||
- File-based logging system
|
||||
- Log structure: `logs/automation/{account_id}/{site_id}/{run_id}/`
|
||||
- Files: `automation_run.log`, `stage_1.log` through `stage_7.log`
|
||||
- Methods: `start_run()`, `log_stage_start()`, `log_stage_progress()`, `log_stage_complete()`, `log_stage_error()`
|
||||
|
||||
**AutomationService** (`services/automation_service.py`)
|
||||
- Core orchestrator for automation pipeline
|
||||
- Methods:
|
||||
- `start_automation()` - Initialize new run with credit check
|
||||
- `run_stage_1()` through `run_stage_7()` - Execute each pipeline stage
|
||||
- `pause_automation()`, `resume_automation()` - Control run execution
|
||||
- `estimate_credits()` - Pre-run credit estimation
|
||||
- `from_run_id()` - Create service from existing run
|
||||
|
||||
#### 3. API Endpoints (`views.py`)
|
||||
|
||||
All endpoints at `/api/v1/automation/`:
|
||||
|
||||
- `GET /config/?site_id=123` - Get automation configuration
|
||||
- `PUT /update_config/?site_id=123` - Update configuration
|
||||
- `POST /run_now/?site_id=123` - Trigger immediate run
|
||||
- `GET /current_run/?site_id=123` - Get active run status
|
||||
- `POST /pause/?run_id=abc` - Pause running automation
|
||||
- `POST /resume/?run_id=abc` - Resume paused automation
|
||||
- `GET /history/?site_id=123` - Get past runs (last 20)
|
||||
- `GET /logs/?run_id=abc&lines=100` - Get run logs
|
||||
- `GET /estimate/?site_id=123` - Estimate credits needed
|
||||
|
||||
#### 4. Celery Tasks (`tasks.py`)
|
||||
|
||||
**check_scheduled_automations**
|
||||
- Runs hourly via Celery Beat
|
||||
- Checks AutomationConfig records for scheduled runs
|
||||
- Triggers automation based on frequency and scheduled_time
|
||||
|
||||
**run_automation_task**
|
||||
- Main background task that executes all 7 stages sequentially
|
||||
- Called by `run_now` API endpoint or scheduled trigger
|
||||
- Handles errors and updates AutomationRun status
|
||||
|
||||
**resume_automation_task**
|
||||
- Resumes paused automation from `current_stage`
|
||||
- Called by `resume` API endpoint
|
||||
|
||||
#### 5. Database Migration
|
||||
|
||||
Located at `/backend/igny8_core/business/automation/migrations/0001_initial.py`
|
||||
|
||||
Run with: `python manage.py migrate`
|
||||
|
||||
### Frontend Components
|
||||
|
||||
#### 1. Service (`/frontend/src/services/automationService.ts`)
|
||||
|
||||
TypeScript API client with methods matching backend endpoints:
|
||||
- `getConfig()`, `updateConfig()`, `runNow()`, `getCurrentRun()`
|
||||
- `pause()`, `resume()`, `getHistory()`, `getLogs()`, `estimate()`
|
||||
|
||||
#### 2. Pages
|
||||
|
||||
**AutomationPage** (`pages/Automation/AutomationPage.tsx`)
|
||||
- Main dashboard at `/automation`
|
||||
- Displays current run status, stage progress, activity log, history
|
||||
- Real-time polling (5s interval when run is active)
|
||||
- Controls: Run Now, Pause, Resume, Configure
|
||||
|
||||
#### 3. Components
|
||||
|
||||
**StageCard** (`components/Automation/StageCard.tsx`)
|
||||
- Visual representation of each stage (1-7)
|
||||
- Shows status: pending (⏳), active (🔄), complete (✅)
|
||||
- Displays stage results (items processed, credits used, etc.)
|
||||
|
||||
**ActivityLog** (`components/Automation/ActivityLog.tsx`)
|
||||
- Real-time log viewer with terminal-style display
|
||||
- Auto-refreshes every 3 seconds
|
||||
- Configurable line count (50, 100, 200, 500)
|
||||
|
||||
**ConfigModal** (`components/Automation/ConfigModal.tsx`)
|
||||
- Modal for editing automation settings
|
||||
- Fields: Enable/disable, frequency, scheduled time, batch sizes
|
||||
- Form validation and save
|
||||
|
||||
**RunHistory** (`components/Automation/RunHistory.tsx`)
|
||||
- Table of past automation runs
|
||||
- Columns: run_id, status, trigger, started, completed, credits, stage
|
||||
- Status badges with color coding
|
||||
|
||||
## 7-Stage Pipeline
|
||||
|
||||
### Stage 1: Keywords → Clusters (AI)
|
||||
- **Query**: `Keywords` with `status='new'`, `cluster__isnull=True`, `disabled=False`
|
||||
- **Batch Size**: Default 20 keywords
|
||||
- **AI Function**: `AutoCluster().execute()`
|
||||
- **Output**: Creates `Clusters` records
|
||||
- **Credits**: ~1 per 5 keywords
|
||||
|
||||
### Stage 2: Clusters → Ideas (AI)
|
||||
- **Query**: `Clusters` with `status='new'`, exclude those with existing ideas
|
||||
- **Batch Size**: Default 1 cluster
|
||||
- **AI Function**: `GenerateIdeas().execute()`
|
||||
- **Output**: Creates `ContentIdeas` records
|
||||
- **Credits**: ~2 per cluster
|
||||
|
||||
### Stage 3: Ideas → Tasks (Local Queue)
|
||||
- **Query**: `ContentIdeas` with `status='new'`
|
||||
- **Batch Size**: Default 20 ideas
|
||||
- **Operation**: Local database creation (no AI)
|
||||
- **Output**: Creates `Tasks` records with status='queued'
|
||||
- **Credits**: 0 (local operation)
|
||||
|
||||
### Stage 4: Tasks → Content (AI)
|
||||
- **Query**: `Tasks` with `status='queued'`, `content__isnull=True`
|
||||
- **Batch Size**: Default 1 task
|
||||
- **AI Function**: `GenerateContent().execute()`
|
||||
- **Output**: Creates `Content` records with status='draft'
|
||||
- **Credits**: ~5 per content (2500 words avg)
|
||||
|
||||
### Stage 5: Content → Image Prompts (AI)
|
||||
- **Query**: `Content` with `status='draft'`, `images_count=0` (annotated)
|
||||
- **Batch Size**: Default 1 content
|
||||
- **AI Function**: `GenerateImagePromptsFunction().execute()`
|
||||
- **Output**: Creates `Images` records with status='pending' (contains prompts)
|
||||
- **Credits**: ~2 per content (4 prompts avg)
|
||||
|
||||
### Stage 6: Image Prompts → Generated Images (AI)
|
||||
- **Query**: `Images` with `status='pending'`
|
||||
- **Batch Size**: Default 1 image
|
||||
- **AI Function**: `GenerateImages().execute()`
|
||||
- **Output**: Updates `Images` to status='generated' with `image_url`
|
||||
- **Side Effect**: Automatically sets `Content.status='review'` when all images complete (via `ai/tasks.py:723`)
|
||||
- **Credits**: ~2 per image
|
||||
|
||||
### Stage 7: Manual Review Gate
|
||||
- **Query**: `Content` with `status='review'`
|
||||
- **Operation**: Count only, no processing
|
||||
- **Output**: Returns list of content IDs ready for review
|
||||
- **Credits**: 0
|
||||
|
||||
## Key Design Principles
|
||||
|
||||
### 1. NO Duplication of AI Function Logic
|
||||
|
||||
The automation system ONLY handles:
|
||||
- Batch selection and sequencing
|
||||
- Stage orchestration
|
||||
- Credit estimation and checking
|
||||
- Progress tracking and logging
|
||||
- Scheduling and triggers
|
||||
|
||||
It does NOT handle:
|
||||
- Credit deduction (done by `AIEngine.execute()` at line 395)
|
||||
- Status updates (done within AI functions)
|
||||
- Progress tracking (StepTracker emits events automatically)
|
||||
|
||||
### 2. Correct Image Model Understanding
|
||||
|
||||
- **NO separate ImagePrompts model** - this was a misunderstanding
|
||||
- `Images` model serves dual purpose:
|
||||
- `status='pending'` = has prompt, needs image URL
|
||||
- `status='generated'` = has image_url
|
||||
- Stage 5 creates Images records with prompts
|
||||
- Stage 6 updates same records with URLs
|
||||
|
||||
### 3. Automatic Content Status Changes
|
||||
|
||||
- `Content.status` changes from 'draft' to 'review' automatically
|
||||
- Happens in `ai/tasks.py:723` when all images complete
|
||||
- Automation does NOT manually update this status
|
||||
|
||||
### 4. Distributed Locking
|
||||
|
||||
- Uses Django cache with `automation_lock_{site.id}` key
|
||||
- 6-hour timeout to prevent deadlocks
|
||||
- Released on completion, pause, or failure
|
||||
|
||||
## Configuration
|
||||
|
||||
### Schedule Configuration UI
|
||||
|
||||
Located at `/automation` page → [Configure] button
|
||||
|
||||
**Options:**
|
||||
- **Enable/Disable**: Toggle automation on/off
|
||||
- **Frequency**: Daily, Weekly (Mondays), Monthly (1st)
|
||||
- **Scheduled Time**: Time of day to run (24-hour format)
|
||||
- **Batch Sizes**: Per-stage item counts
|
||||
|
||||
**Defaults:**
|
||||
- Stage 1: 20 keywords
|
||||
- Stage 2: 1 cluster
|
||||
- Stage 3: 20 ideas
|
||||
- Stage 4: 1 task
|
||||
- Stage 5: 1 content
|
||||
- Stage 6: 1 image
|
||||
|
||||
### Credit Estimation
|
||||
|
||||
Before starting, system estimates:
|
||||
- Stage 1: keywords_count / 5
|
||||
- Stage 2: clusters_count * 2
|
||||
- Stage 4: tasks_count * 5
|
||||
- Stage 5: content_count * 2
|
||||
- Stage 6: content_count * 8 (4 images * 2 credits avg)
|
||||
|
||||
Requires 20% buffer: `account.credits_balance >= estimated * 1.2`
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
### Backend
|
||||
|
||||
1. ✅ Models created in `business/automation/models.py`
|
||||
2. ✅ Services created (`AutomationLogger`, `AutomationService`)
|
||||
3. ✅ Views created (`AutomationViewSet`)
|
||||
4. ✅ URLs registered in `igny8_core/urls.py`
|
||||
5. ✅ Celery tasks created (`check_scheduled_automations`, `run_automation_task`, `resume_automation_task`)
|
||||
6. ✅ Celery beat schedule updated in `celery.py`
|
||||
7. ⏳ Migration created (needs to run: `python manage.py migrate`)
|
||||
|
||||
### Frontend
|
||||
|
||||
8. ✅ API service created (`services/automationService.ts`)
|
||||
9. ✅ Main page created (`pages/Automation/AutomationPage.tsx`)
|
||||
10. ✅ Components created (`StageCard`, `ActivityLog`, `ConfigModal`, `RunHistory`)
|
||||
11. ⏳ Route registration (add to router: `/automation` → `AutomationPage`)
|
||||
|
||||
### Infrastructure
|
||||
|
||||
12. ⏳ Celery worker running (for background tasks)
|
||||
13. ⏳ Celery beat running (for scheduled checks)
|
||||
14. ⏳ Redis/cache backend configured (for distributed locks)
|
||||
15. ⏳ Log directory writable: `/data/app/igny8/backend/logs/automation/`
|
||||
|
||||
## Usage
|
||||
|
||||
### Manual Trigger
|
||||
|
||||
1. Navigate to `/automation` page
|
||||
2. Verify credit balance is sufficient (shows in header)
|
||||
3. Click [Run Now] button
|
||||
4. Monitor progress in real-time:
|
||||
- Stage cards show current progress
|
||||
- Activity log shows detailed logs
|
||||
- Credits used updates live
|
||||
|
||||
### Scheduled Automation
|
||||
|
||||
1. Navigate to `/automation` page
|
||||
2. Click [Configure] button
|
||||
3. Enable automation
|
||||
4. Set frequency and time
|
||||
5. Configure batch sizes
|
||||
6. Save configuration
|
||||
7. Automation will run automatically at scheduled time
|
||||
|
||||
### Pause/Resume
|
||||
|
||||
- During active run, click [Pause] to halt execution
|
||||
- Click [Resume] to continue from current stage
|
||||
- Useful for credit management or issue investigation
|
||||
|
||||
### Viewing History
|
||||
|
||||
- Run History table shows last 20 runs
|
||||
- Filter by status, date, trigger type
|
||||
- Click run_id to view detailed logs
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Log Files
|
||||
|
||||
Located at: `logs/automation/{account_id}/{site_id}/{run_id}/`
|
||||
|
||||
- `automation_run.log` - Main activity log
|
||||
- `stage_1.log` through `stage_7.log` - Stage-specific logs
|
||||
|
||||
### Database Records
|
||||
|
||||
**AutomationRun** table tracks:
|
||||
- Current status and stage
|
||||
- Stage results (JSON)
|
||||
- Credits used
|
||||
- Error messages
|
||||
- Timestamps
|
||||
|
||||
**AutomationConfig** table tracks:
|
||||
- Last run timestamp
|
||||
- Next scheduled run
|
||||
- Configuration changes
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Run stuck in "running" status
|
||||
|
||||
1. Check Celery worker logs: `docker logs <celery_container>`
|
||||
2. Check for cache lock: `redis-cli GET automation_lock_<site_id>`
|
||||
3. Manually release lock if needed: `redis-cli DEL automation_lock_<site_id>`
|
||||
4. Update run status: `AutomationRun.objects.filter(run_id='...').update(status='failed')`
|
||||
|
||||
### Insufficient credits
|
||||
|
||||
1. Check estimate: GET `/api/v1/automation/estimate/?site_id=123`
|
||||
2. Add credits via billing page
|
||||
3. Retry run
|
||||
|
||||
### Stage failures
|
||||
|
||||
1. View logs: GET `/api/v1/automation/logs/?run_id=...`
|
||||
2. Check `error_message` field in AutomationRun
|
||||
3. Verify AI function is working: test individually via existing UI
|
||||
4. Check credit balance mid-run
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
1. Email notifications on completion/failure
|
||||
2. Slack/webhook integrations
|
||||
3. Per-stage retry logic
|
||||
4. Partial run resumption after failure
|
||||
5. Advanced scheduling (specific days, multiple times)
|
||||
6. Content preview before Stage 7
|
||||
7. Auto-publish to WordPress option
|
||||
8. Credit usage analytics and forecasting
|
||||
|
||||
## File Locations Summary
|
||||
|
||||
```
|
||||
backend/igny8_core/business/automation/
|
||||
├── __init__.py
|
||||
├── models.py # AutomationConfig, AutomationRun
|
||||
├── views.py # AutomationViewSet (API endpoints)
|
||||
├── tasks.py # Celery tasks
|
||||
├── urls.py # URL routing
|
||||
├── migrations/
|
||||
│ ├── __init__.py
|
||||
│ └── 0001_initial.py # Database schema
|
||||
└── services/
|
||||
├── __init__.py
|
||||
├── automation_logger.py # File logging service
|
||||
└── automation_service.py # Core orchestrator
|
||||
|
||||
frontend/src/
|
||||
├── services/
|
||||
│ └── automationService.ts # API client
|
||||
├── pages/Automation/
|
||||
│ └── AutomationPage.tsx # Main dashboard
|
||||
└── components/Automation/
|
||||
├── StageCard.tsx # Stage status display
|
||||
├── ActivityLog.tsx # Log viewer
|
||||
├── ConfigModal.tsx # Settings modal
|
||||
└── RunHistory.tsx # Past runs table
|
||||
```
|
||||
|
||||
## Credits
|
||||
|
||||
Implemented according to `automation-plan.md` with corrections for:
|
||||
- Image model structure (no separate ImagePrompts)
|
||||
- AI function internal logic (no duplication)
|
||||
- Content status changes (automatic in background)
|
||||
Reference in New Issue
Block a user