1186 lines
41 KiB
Markdown
1186 lines
41 KiB
Markdown
# AI Automation Pipeline - Implementation Blueprint
|
|
**Date:** December 3, 2025
|
|
**Purpose:** Site-level automation orchestrating existing AI functions into sequential 7-stage pipeline
|
|
|
|
---
|
|
|
|
## 🎯 EXECUTIVE SUMMARY
|
|
|
|
### What We're Building
|
|
A **site-level automation page** (`/automation`) that orchestrates 6 existing AI functions + 1 local function into a **strictly sequential** 7-stage pipeline for hands-free content generation from keywords to draft content ready for review.
|
|
|
|
### Core Principles
|
|
✅ **Zero Duplication** - Reuse all existing AI functions (auto_cluster, generate_ideas, generate_content, generate_image_prompts, generate_images)
|
|
✅ **Strictly Sequential** - Stage N+1 ONLY starts when Stage N is 100% complete
|
|
✅ **Batch Processing** - Within each stage, process items in configurable batches with queues visible
|
|
✅ **Site-Level Scope** - NO sector filtering - operates on entire site's data
|
|
✅ **Observable** - Real-time batch progress, detailed queue counts, stage-by-stage logs
|
|
✅ **Safe Execution** - Distributed locks, credit reservations, idempotent stages, Celery task chaining
|
|
|
|
### Sequential Stage Execution
|
|
- **Stage completes** → Trigger next stage automatically
|
|
- **Within stage** → Process batches sequentially until queue empty
|
|
- **Between stages** → Hard stop, verify completion, then proceed
|
|
- **Never parallel** - Only 1 stage active at a time per site
|
|
|
|
### Automation Stops Before Publishing
|
|
- **Stage 7 (Manual Review Gate)** - Automation ends when content is draft+images ready
|
|
- User manually reviews and publishes via existing bulk actions
|
|
- No automated publishing to WordPress (human oversight required)
|
|
|
|
---
|
|
|
|
## 🏗️ SCOPE & DATA MODEL
|
|
|
|
### Site-Level Operation (NO Sector)
|
|
```
|
|
┌─────────────────────────────────────┐
|
|
│ Site: "example.com" │
|
|
│ ├─ Keywords (ALL sectors combined) │
|
|
│ ├─ Clusters (ALL sectors combined) │
|
|
│ ├─ Ideas (ALL sectors combined) │
|
|
│ └─ Tasks (ALL sectors combined) │
|
|
└─────────────────────────────────────┘
|
|
|
|
UI: Only Site selector at top
|
|
Database queries: Filter by site_id only
|
|
No sector dropdown in automation page
|
|
```
|
|
|
|
### Why Site-Level?
|
|
- **Simplicity** - User manages automation per website, not per topic
|
|
- **Unified Progress** - See total content pipeline for entire site
|
|
- **Flexible Sectors** - Content can span multiple sectors naturally
|
|
- **Easier Scheduling** - One automation config per site
|
|
|
|
---
|
|
|
|
## 📋 EXISTING AI FUNCTIONS (Reused, Not Duplicated)
|
|
|
|
| Function | File | Input | Output | Credits | Already Works |
|
|
|----------|------|-------|--------|---------|---------------|
|
|
| **auto_cluster** | `ai/functions/auto_cluster.py` | Keyword IDs (max 20) | Clusters created | 1 per 5 keywords | ✅ Yes |
|
|
| **generate_ideas** | `ai/functions/generate_ideas.py` | Cluster IDs (max 5) | Ideas created | 2 per cluster | ✅ Yes |
|
|
| **generate_content** | `ai/functions/generate_content.py` | Task IDs (1 at a time) | Content draft | 1 per 500 words | ✅ Yes |
|
|
| **generate_image_prompts** | `ai/functions/generate_image_prompts.py` | Content IDs | Image prompts | 0.5 per prompt | ✅ Yes |
|
|
| **generate_images** | `ai/functions/generate_images.py` | Image prompt IDs | Generated images | 1-4 per image | ✅ Yes |
|
|
| **bulk_queue_to_writer** | `modules/planner/views.py#L1084` | Idea IDs | Tasks created | 0 (local) | ✅ Yes |
|
|
|
|
**All functions already:**
|
|
- Have async Celery tasks
|
|
- Return task_id for progress tracking
|
|
- Deduct credits automatically
|
|
- Update model statuses (new → mapped → queued → completed)
|
|
- Handle errors gracefully
|
|
|
|
---
|
|
|
|
## 🏗️ NEW COMPONENTS TO BUILD
|
|
|
|
### Phase 1: Backend Infrastructure
|
|
|
|
#### 1.1 Database Models
|
|
**File:** `backend/igny8_core/business/automation/models.py`
|
|
|
|
```python
|
|
class AutomationRun(SiteSectorBaseModel):
|
|
"""Track each automation run"""
|
|
run_id = models.CharField(max_length=100, unique=True, db_index=True)
|
|
# Format: run_20251203_140523_manual or run_20251204_020000_scheduled
|
|
|
|
trigger_type = models.CharField(max_length=20, choices=[
|
|
('manual', 'Manual'),
|
|
('scheduled', 'Scheduled')
|
|
])
|
|
|
|
status = models.CharField(max_length=20, choices=[
|
|
('running', 'Running'),
|
|
('paused', 'Paused'),
|
|
('completed', 'Completed'),
|
|
('failed', 'Failed')
|
|
], default='running')
|
|
|
|
current_stage = models.IntegerField(default=1) # 1-7
|
|
|
|
started_at = models.DateTimeField(auto_now_add=True)
|
|
completed_at = models.DateTimeField(null=True, blank=True)
|
|
|
|
total_credits_used = models.IntegerField(default=0)
|
|
|
|
# Stage results (JSON)
|
|
stage_1_result = models.JSONField(default=dict, blank=True) # {clusters_created: 8, keywords_processed: 47}
|
|
stage_2_result = models.JSONField(default=dict, blank=True) # {ideas_created: 56}
|
|
stage_3_result = models.JSONField(default=dict, blank=True) # {tasks_created: 56}
|
|
stage_4_result = models.JSONField(default=dict, blank=True) # {content_created: 56}
|
|
stage_5_result = models.JSONField(default=dict, blank=True) # {prompts_created: 224}
|
|
stage_6_result = models.JSONField(default=dict, blank=True) # {images_created: 224}
|
|
stage_7_result = models.JSONField(default=dict, blank=True) # {ready_for_review: 56}
|
|
|
|
error_message = models.TextField(blank=True, null=True)
|
|
|
|
class Meta:
|
|
ordering = ['-started_at']
|
|
indexes = [
|
|
models.Index(fields=['run_id']),
|
|
models.Index(fields=['site', 'sector', '-started_at']),
|
|
]
|
|
|
|
|
|
class AutomationConfig(SiteSectorBaseModel):
|
|
"""Store automation schedule and settings per site/sector"""
|
|
|
|
is_enabled = models.BooleanField(default=False)
|
|
|
|
# Schedule
|
|
frequency = models.CharField(max_length=20, choices=[
|
|
('daily', 'Daily'),
|
|
('weekly', 'Weekly'),
|
|
('monthly', 'Monthly')
|
|
], default='daily')
|
|
|
|
scheduled_time = models.TimeField(default='02:00') # 2:00 AM
|
|
|
|
# Batch sizes (sensible defaults from plan)
|
|
stage_1_batch_size = models.IntegerField(default=20) # Keywords per batch
|
|
stage_2_batch_size = models.IntegerField(default=1) # Clusters at a time
|
|
stage_3_batch_size = models.IntegerField(default=20) # Ideas per batch
|
|
stage_4_batch_size = models.IntegerField(default=1) # Tasks (sequential)
|
|
stage_5_batch_size = models.IntegerField(default=1) # Content at a time
|
|
stage_6_batch_size = models.IntegerField(default=1) # Images (auto-handled)
|
|
|
|
last_run_at = models.DateTimeField(null=True, blank=True)
|
|
next_run_at = models.DateTimeField(null=True, blank=True)
|
|
|
|
class Meta:
|
|
unique_together = [['site', 'sector']]
|
|
```
|
|
|
|
#### 1.2 Logging Service
|
|
**File:** `backend/igny8_core/business/automation/services/automation_logger.py`
|
|
|
|
```python
|
|
import os
|
|
import logging
|
|
from datetime import datetime
|
|
from django.conf import settings
|
|
|
|
class AutomationLogger:
|
|
"""File-based logging for automation runs"""
|
|
|
|
def __init__(self):
|
|
self.base_path = os.path.join(settings.BASE_DIR, 'logs', 'automation')
|
|
|
|
def start_run(self, account_id, site_id, sector_id, trigger_type):
|
|
"""Create run_id and log directory"""
|
|
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
|
|
run_id = f"run_{timestamp}_{trigger_type}"
|
|
|
|
# Create directory: logs/automation/{account_id}/{site_id}/{sector_id}/{run_id}/
|
|
log_dir = os.path.join(
|
|
self.base_path,
|
|
str(account_id),
|
|
str(site_id),
|
|
str(sector_id),
|
|
run_id
|
|
)
|
|
os.makedirs(log_dir, exist_ok=True)
|
|
|
|
# Create main log file
|
|
main_log = os.path.join(log_dir, 'automation_run.log')
|
|
with open(main_log, 'w') as f:
|
|
f.write(f"{'='*60}\\n")
|
|
f.write(f"AUTOMATION RUN: {run_id}\\n")
|
|
f.write(f"Started: {datetime.now()}\\n")
|
|
f.write(f"Trigger: {trigger_type}\\n")
|
|
f.write(f"{'='*60}\\n\\n")
|
|
|
|
return run_id
|
|
|
|
def log_stage_start(self, run_id, stage_number, stage_name, pending_count, account_id, site_id, sector_id):
|
|
"""Log start of a stage"""
|
|
stage_file = self._get_stage_file(run_id, stage_number, account_id, site_id, sector_id)
|
|
timestamp = datetime.now().strftime('%H:%M:%S')
|
|
|
|
with open(stage_file, 'a') as f:
|
|
f.write(f"\\n{'='*60}\\n")
|
|
f.write(f"STAGE {stage_number}: {stage_name}\\n")
|
|
f.write(f"Started: {datetime.now()}\\n")
|
|
f.write(f"{'='*60}\\n\\n")
|
|
f.write(f"{timestamp} - Found {pending_count} items to process\\n")
|
|
|
|
def log_stage_progress(self, run_id, stage_number, message, account_id, site_id, sector_id):
|
|
"""Log progress within a stage"""
|
|
stage_file = self._get_stage_file(run_id, stage_number, account_id, site_id, sector_id)
|
|
timestamp = datetime.now().strftime('%H:%M:%S')
|
|
|
|
with open(stage_file, 'a') as f:
|
|
f.write(f"{timestamp} - {message}\\n")
|
|
|
|
# Also append to main log
|
|
main_file = self._get_main_log(run_id, account_id, site_id, sector_id)
|
|
with open(main_file, 'a') as f:
|
|
f.write(f"{timestamp} - Stage {stage_number}: {message}\\n")
|
|
|
|
def log_stage_complete(self, run_id, stage_number, stage_name, processed_count, time_elapsed, credits_used, account_id, site_id, sector_id):
|
|
"""Log completion of a stage"""
|
|
stage_file = self._get_stage_file(run_id, stage_number, account_id, site_id, sector_id)
|
|
|
|
with open(stage_file, 'a') as f:
|
|
f.write(f"\\n{'='*60}\\n")
|
|
f.write(f"STAGE {stage_number} COMPLETE\\n")
|
|
f.write(f"Total Time: {time_elapsed}\\n")
|
|
f.write(f"Processed: {processed_count} items\\n")
|
|
f.write(f"Credits Used: {credits_used}\\n")
|
|
f.write(f"{'='*60}\\n")
|
|
|
|
def log_stage_error(self, run_id, stage_number, error_message, account_id, site_id, sector_id):
|
|
"""Log error in a stage"""
|
|
stage_file = self._get_stage_file(run_id, stage_number, account_id, site_id, sector_id)
|
|
timestamp = datetime.now().strftime('%H:%M:%S')
|
|
|
|
with open(stage_file, 'a') as f:
|
|
f.write(f"\\n{timestamp} - ERROR: {error_message}\\n")
|
|
|
|
def get_activity_log(self, run_id, account_id, site_id, sector_id, last_n=50):
|
|
"""Get last N lines from main log"""
|
|
main_file = self._get_main_log(run_id, account_id, site_id, sector_id)
|
|
|
|
if not os.path.exists(main_file):
|
|
return []
|
|
|
|
with open(main_file, 'r') as f:
|
|
lines = f.readlines()
|
|
return lines[-last_n:] # Last 50 lines
|
|
|
|
def _get_stage_file(self, run_id, stage_number, account_id, site_id, sector_id):
|
|
"""Get path to stage log file"""
|
|
log_dir = os.path.join(
|
|
self.base_path,
|
|
str(account_id),
|
|
str(site_id),
|
|
str(sector_id),
|
|
run_id
|
|
)
|
|
return os.path.join(log_dir, f"stage_{stage_number}.log")
|
|
|
|
def _get_main_log(self, run_id, account_id, site_id, sector_id):
|
|
"""Get path to main log file"""
|
|
log_dir = os.path.join(
|
|
self.base_path,
|
|
str(account_id),
|
|
str(site_id),
|
|
str(sector_id),
|
|
run_id
|
|
)
|
|
return os.path.join(log_dir, 'automation_run.log')
|
|
```
|
|
|
|
#### 1.3 Automation Service (Core Orchestrator)
|
|
**File:** `backend/igny8_core/business/automation/services/automation_service.py`
|
|
|
|
```python
|
|
import time
|
|
from datetime import datetime, timedelta
|
|
from django.utils import timezone
|
|
from igny8_core.business.automation.models import AutomationRun, AutomationConfig
|
|
from igny8_core.business.automation.services.automation_logger import AutomationLogger
|
|
from igny8_core.business.planning.models import Keywords, Clusters, ContentIdeas
|
|
from igny8_core.business.content.models import Tasks, Content, Images
|
|
from igny8_core.business.billing.services.credit_service import CreditService
|
|
|
|
# Import existing services (NO DUPLICATION)
|
|
from igny8_core.business.planning.services.clustering_service import ClusteringService
|
|
from igny8_core.business.planning.services.ideas_service import IdeasService
|
|
from igny8_core.ai.functions.generate_content import GenerateContentFunction
|
|
from igny8_core.ai.functions.generate_image_prompts import GenerateImagePromptsFunction
|
|
from igny8_core.ai.functions.generate_images import GenerateImagesFunction
|
|
|
|
|
|
class AutomationService:
|
|
"""
|
|
Orchestrates the 7-stage automation pipeline.
|
|
Reuses all existing AI functions - zero duplication.
|
|
"""
|
|
|
|
def __init__(self, account, site, sector):
|
|
self.account = account
|
|
self.site = site
|
|
self.sector = sector
|
|
self.logger = AutomationLogger()
|
|
self.credit_service = CreditService()
|
|
|
|
# Existing services
|
|
self.clustering_service = ClusteringService()
|
|
self.ideas_service = IdeasService()
|
|
self.content_function = GenerateContentFunction()
|
|
self.prompts_function = GenerateImagePromptsFunction()
|
|
self.images_function = GenerateImagesFunction()
|
|
|
|
self.run = None
|
|
self.config = None
|
|
|
|
def start_automation(self, trigger_type='manual'):
|
|
\"\"\"
|
|
Main entry point for automation.
|
|
Creates run record, executes stages sequentially.
|
|
\"\"\"
|
|
try:
|
|
# Create run record
|
|
run_id = self.logger.start_run(
|
|
self.account.id,
|
|
self.site.id,
|
|
self.sector.id,
|
|
trigger_type
|
|
)
|
|
|
|
self.run = AutomationRun.objects.create(
|
|
run_id=run_id,
|
|
trigger_type=trigger_type,
|
|
account=self.account,
|
|
site=self.site,
|
|
sector=self.sector,
|
|
status='running',
|
|
current_stage=1
|
|
)
|
|
|
|
# Load config (for batch sizes)
|
|
self.config = AutomationConfig.objects.filter(
|
|
site=self.site,
|
|
sector=self.sector
|
|
).first()
|
|
|
|
if not self.config:
|
|
# Create default config
|
|
self.config = AutomationConfig.objects.create(
|
|
site=self.site,
|
|
sector=self.sector,
|
|
account=self.account
|
|
)
|
|
|
|
# Execute stages sequentially
|
|
self.run_stage_1() # Keywords → Clusters
|
|
self.run_stage_2() # Clusters → Ideas
|
|
self.run_stage_3() # Ideas → Tasks
|
|
self.run_stage_4() # Tasks → Content
|
|
self.run_stage_5() # Content → Image Prompts
|
|
self.run_stage_6() # Image Prompts → Images
|
|
self.run_stage_7() # Manual Review Gate
|
|
|
|
# Mark complete
|
|
self.run.status = 'completed'
|
|
self.run.completed_at = timezone.now()
|
|
self.run.save()
|
|
|
|
return {
|
|
'success': True,
|
|
'run_id': run_id,
|
|
'message': 'Automation completed successfully'
|
|
}
|
|
|
|
except Exception as e:
|
|
if self.run:
|
|
self.run.status = 'failed'
|
|
self.run.error_message = str(e)
|
|
self.run.save()
|
|
|
|
return {
|
|
'success': False,
|
|
'error': str(e)
|
|
}
|
|
|
|
def run_stage_1(self):
|
|
\"\"\"Stage 1: Keywords (status='new') → Clusters (AI)\"\"\"
|
|
stage_start = time.time()
|
|
stage_number = 1
|
|
stage_name = "Keywords → Clusters (AI)"
|
|
|
|
# Find pending keywords (status='new', cluster_id=null)
|
|
pending_keywords = Keywords.objects.filter(
|
|
site=self.site,
|
|
sector=self.sector,
|
|
status='new',
|
|
cluster__isnull=True,
|
|
disabled=False
|
|
)
|
|
|
|
total_count = pending_keywords.count()
|
|
|
|
if total_count == 0:
|
|
self.logger.log_stage_progress(
|
|
self.run.run_id, stage_number,
|
|
"No pending keywords found - skipping stage",
|
|
self.account.id, self.site.id, self.sector.id
|
|
)
|
|
self.run.current_stage = 2
|
|
self.run.save()
|
|
return
|
|
|
|
self.logger.log_stage_start(
|
|
self.run.run_id, stage_number, stage_name, total_count,
|
|
self.account.id, self.site.id, self.sector.id
|
|
)
|
|
|
|
# Process in batches (default 20 per batch)
|
|
batch_size = self.config.stage_1_batch_size
|
|
keyword_ids = list(pending_keywords.values_list('id', flat=True))
|
|
|
|
clusters_created = 0
|
|
keywords_processed = 0
|
|
credits_used = 0
|
|
|
|
for i in range(0, len(keyword_ids), batch_size):
|
|
batch = keyword_ids[i:i+batch_size]
|
|
batch_num = (i // batch_size) + 1
|
|
total_batches = (len(keyword_ids) + batch_size - 1) // batch_size
|
|
|
|
self.logger.log_stage_progress(
|
|
self.run.run_id, stage_number,
|
|
f"Processing batch {batch_num}/{total_batches} ({len(batch)} keywords)",
|
|
self.account.id, self.site.id, self.sector.id
|
|
)
|
|
|
|
# Call existing ClusteringService (REUSE - NO DUPLICATION)
|
|
result = self.clustering_service.cluster_keywords(
|
|
keyword_ids=batch,
|
|
account=self.account,
|
|
sector_id=self.sector.id
|
|
)
|
|
|
|
if result.get('success'):
|
|
clusters_created += result.get('clusters_created', 0)
|
|
keywords_processed += len(batch)
|
|
credits_used += result.get('credits_used', 0)
|
|
|
|
self.logger.log_stage_progress(
|
|
self.run.run_id, stage_number,
|
|
f"Batch {batch_num} complete: {result.get('clusters_created', 0)} clusters created",
|
|
self.account.id, self.site.id, self.sector.id
|
|
)
|
|
|
|
# Save stage result
|
|
elapsed = time.time() - stage_start
|
|
self.run.stage_1_result = {
|
|
'clusters_created': clusters_created,
|
|
'keywords_processed': keywords_processed
|
|
}
|
|
self.run.total_credits_used += credits_used
|
|
self.run.current_stage = 2
|
|
self.run.save()
|
|
|
|
self.logger.log_stage_complete(
|
|
self.run.run_id, stage_number, stage_name,
|
|
keywords_processed, f"{elapsed:.0f}s", credits_used,
|
|
self.account.id, self.site.id, self.sector.id
|
|
)
|
|
|
|
def run_stage_2(self):
|
|
\"\"\"Stage 2: Clusters (status='new', no ideas) → Ideas (AI)\"\"\"
|
|
# Similar structure to stage_1
|
|
# Calls existing IdeasService.generate_ideas()
|
|
pass
|
|
|
|
def run_stage_3(self):
|
|
\"\"\"Stage 3: Ideas (status='new') → Tasks (Local queue)\"\"\"
|
|
# Calls existing bulk_queue_to_writer endpoint logic
|
|
pass
|
|
|
|
def run_stage_4(self):
|
|
\"\"\"Stage 4: Tasks (status='queued') → Content (AI)\"\"\"
|
|
# Calls existing GenerateContentFunction
|
|
# Process one task at a time (sequential)
|
|
pass
|
|
|
|
def run_stage_5(self):
|
|
\"\"\"Stage 5: Content (draft) → Image Prompts (AI)\"\"\"
|
|
# Calls existing GenerateImagePromptsFunction
|
|
pass
|
|
|
|
def run_stage_6(self):
|
|
\"\"\"Stage 6: Image Prompts (pending) → Images (AI)\"\"\"
|
|
# Calls existing GenerateImagesFunction
|
|
# Handles batching automatically
|
|
pass
|
|
|
|
def run_stage_7(self):
|
|
\"\"\"Stage 7: Manual Review Gate (STOP)\"\"\"
|
|
# Just count content ready for review
|
|
# Log final status
|
|
# Automation ends here
|
|
pass
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 2: API Endpoints
|
|
|
|
#### 2.1 Automation ViewSet
|
|
**File:** `backend/igny8_core/modules/automation/views.py` (NEW MODULE)
|
|
|
|
```python
|
|
from rest_framework import viewsets
|
|
from rest_framework.decorators import action
|
|
from igny8_core.api.base import SiteSectorModelViewSet
|
|
from igny8_core.api.response import success_response, error_response
|
|
from igny8_core.business.automation.models import AutomationRun, AutomationConfig
|
|
from igny8_core.business.automation.services.automation_service import AutomationService
|
|
from igny8_core.business.automation.services.automation_logger import AutomationLogger
|
|
|
|
class AutomationViewSet(SiteSectorModelViewSet):
|
|
"""API endpoints for automation"""
|
|
|
|
@action(detail=False, methods=['post'])
|
|
def run_now(self, request):
|
|
"""Trigger manual automation run"""
|
|
account = request.account
|
|
site = request.site
|
|
sector = request.sector
|
|
|
|
# Start automation
|
|
service = AutomationService(account, site, sector)
|
|
result = service.start_automation(trigger_type='manual')
|
|
|
|
if result['success']:
|
|
return success_response(data={'run_id': result['run_id']})
|
|
else:
|
|
return error_response(message=result['error'])
|
|
|
|
@action(detail=False, methods=['get'])
|
|
def current_run(self, request):
|
|
"""Get current/latest automation run status"""
|
|
site = request.site
|
|
sector = request.sector
|
|
|
|
run = AutomationRun.objects.filter(
|
|
site=site,
|
|
sector=sector
|
|
).order_by('-started_at').first()
|
|
|
|
if not run:
|
|
return success_response(data={'run': None})
|
|
|
|
# Get activity log
|
|
logger = AutomationLogger()
|
|
activity = logger.get_activity_log(
|
|
run.run_id,
|
|
run.account_id,
|
|
run.site_id,
|
|
run.sector_id,
|
|
last_n=50
|
|
)
|
|
|
|
return success_response(data={
|
|
'run': {
|
|
'run_id': run.run_id,
|
|
'status': run.status,
|
|
'current_stage': run.current_stage,
|
|
'trigger_type': run.trigger_type,
|
|
'started_at': run.started_at,
|
|
'total_credits_used': run.total_credits_used,
|
|
'stage_1_result': run.stage_1_result,
|
|
'stage_2_result': run.stage_2_result,
|
|
'stage_3_result': run.stage_3_result,
|
|
'stage_4_result': run.stage_4_result,
|
|
'stage_5_result': run.stage_5_result,
|
|
'stage_6_result': run.stage_6_result,
|
|
'stage_7_result': run.stage_7_result,
|
|
},
|
|
'activity_log': activity
|
|
})
|
|
|
|
@action(detail=False, methods=['get', 'put'])
|
|
def config(self, request):
|
|
"""Get/Update automation configuration"""
|
|
site = request.site
|
|
sector = request.sector
|
|
account = request.account
|
|
|
|
if request.method == 'GET':
|
|
config, created = AutomationConfig.objects.get_or_create(
|
|
site=site,
|
|
sector=sector,
|
|
defaults={'account': account}
|
|
)
|
|
|
|
return success_response(data={
|
|
'is_enabled': config.is_enabled,
|
|
'frequency': config.frequency,
|
|
'scheduled_time': config.scheduled_time,
|
|
'next_run_at': config.next_run_at,
|
|
'stage_1_batch_size': config.stage_1_batch_size,
|
|
'stage_2_batch_size': config.stage_2_batch_size,
|
|
'stage_3_batch_size': config.stage_3_batch_size,
|
|
'stage_4_batch_size': config.stage_4_batch_size,
|
|
'stage_5_batch_size': config.stage_5_batch_size,
|
|
'stage_6_batch_size': config.stage_6_batch_size,
|
|
})
|
|
|
|
elif request.method == 'PUT':
|
|
# Update configuration
|
|
config, created = AutomationConfig.objects.get_or_create(
|
|
site=site,
|
|
sector=sector,
|
|
defaults={'account': account}
|
|
)
|
|
|
|
# Update fields from request
|
|
config.is_enabled = request.data.get('is_enabled', config.is_enabled)
|
|
config.frequency = request.data.get('frequency', config.frequency)
|
|
config.scheduled_time = request.data.get('scheduled_time', config.scheduled_time)
|
|
config.save()
|
|
|
|
return success_response(message='Configuration updated')
|
|
```
|
|
|
|
#### 2.2 URL Configuration
|
|
**File:** `backend/igny8_core/urls/api_urls.py` (ADD)
|
|
|
|
```python
|
|
# Add to router
|
|
router.register(r'automation', AutomationViewSet, basename='automation')
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 3: Celery Scheduled Task
|
|
|
|
#### 3.1 Periodic Task for Scheduled Runs
|
|
**File:** `backend/igny8_core/tasks/automation_tasks.py` (NEW)
|
|
|
|
```python
|
|
from celery import shared_task
|
|
from django.utils import timezone
|
|
from datetime import datetime, timedelta
|
|
from igny8_core.business.automation.models import AutomationConfig
|
|
from igny8_core.business.automation.services.automation_service import AutomationService
|
|
from igny8_core.auth.models import Account, Site, Sector
|
|
|
|
@shared_task(name='run_scheduled_automation')
|
|
def run_scheduled_automation():
|
|
\"\"\"
|
|
Celery beat task - runs every hour, checks if any configs need to run
|
|
\"\"\"
|
|
now = timezone.now()
|
|
|
|
# Find configs that:
|
|
# 1. Are enabled
|
|
# 2. Have next_run_at <= now
|
|
configs = AutomationConfig.objects.filter(
|
|
is_enabled=True,
|
|
next_run_at__lte=now
|
|
)
|
|
|
|
for config in configs:
|
|
try:
|
|
# Load related objects
|
|
account = config.account
|
|
site = config.site
|
|
sector = config.sector
|
|
|
|
# Start automation
|
|
service = AutomationService(account, site, sector)
|
|
service.start_automation(trigger_type='scheduled')
|
|
|
|
# Calculate next run time
|
|
if config.frequency == 'daily':
|
|
next_run = now + timedelta(days=1)
|
|
elif config.frequency == 'weekly':
|
|
next_run = now + timedelta(weeks=1)
|
|
elif config.frequency == 'monthly':
|
|
next_run = now + timedelta(days=30)
|
|
|
|
# Set time to scheduled_time
|
|
next_run = next_run.replace(
|
|
hour=config.scheduled_time.hour,
|
|
minute=config.scheduled_time.minute,
|
|
second=0,
|
|
microsecond=0
|
|
)
|
|
|
|
config.last_run_at = now
|
|
config.next_run_at = next_run
|
|
config.save()
|
|
|
|
except Exception as e:
|
|
# Log error but continue with other configs
|
|
print(f"Error running scheduled automation for {config.id}: {e}")
|
|
continue
|
|
```
|
|
|
|
#### 3.2 Register Celery Beat Schedule
|
|
**File:** `backend/igny8_core/celery.py` (UPDATE)
|
|
|
|
```python
|
|
app.conf.beat_schedule = {
|
|
# ... existing schedules ...
|
|
'run-scheduled-automation': {
|
|
'task': 'run_scheduled_automation',
|
|
'schedule': crontab(minute=0), # Every hour on the hour
|
|
},
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 4: Frontend Components
|
|
|
|
#### 4.1 Automation Page Component
|
|
**File:** `frontend/src/pages/Automation/Dashboard.tsx` (NEW)
|
|
|
|
```tsx
|
|
import { useState, useEffect } from 'react';
|
|
import { useInterval } from '../../hooks/useInterval';
|
|
import { automationApi } from '../../services/api/automationApi';
|
|
import { useSiteStore } from '../../stores/siteStore';
|
|
import { useSectorStore } from '../../stores/sectorStore';
|
|
import Button from '../../components/ui/button/Button';
|
|
import { BoltIcon, PlayIcon, PauseIcon, SettingsIcon } from '../../icons';
|
|
import StageCard from './components/StageCard';
|
|
import ActivityLog from './components/ActivityLog';
|
|
import ConfigModal from './components/ConfigModal';
|
|
|
|
export default function AutomationDashboard() {
|
|
const { activeSite } = useSiteStore();
|
|
const { activeSector } = useSectorStore();
|
|
|
|
const [currentRun, setCurrentRun] = useState(null);
|
|
const [activityLog, setActivityLog] = useState([]);
|
|
const [config, setConfig] = useState(null);
|
|
const [showConfigModal, setShowConfigModal] = useState(false);
|
|
const [isRunning, setIsRunning] = useState(false);
|
|
|
|
// Poll current run status every 3 seconds when running
|
|
useInterval(() => {
|
|
if (activeSite && activeSector) {
|
|
loadCurrentRun();
|
|
}
|
|
}, currentRun?.status === 'running' ? 3000 : null);
|
|
|
|
useEffect(() => {
|
|
if (activeSite && activeSector) {
|
|
loadCurrentRun();
|
|
loadConfig();
|
|
}
|
|
}, [activeSite?.id, activeSector?.id]);
|
|
|
|
const loadCurrentRun = async () => {
|
|
const response = await automationApi.getCurrentRun();
|
|
if (response.success) {
|
|
setCurrentRun(response.data.run);
|
|
setActivityLog(response.data.activity_log || []);
|
|
}
|
|
};
|
|
|
|
const loadConfig = async () => {
|
|
const response = await automationApi.getConfig();
|
|
if (response.success) {
|
|
setConfig(response.data);
|
|
}
|
|
};
|
|
|
|
const handleRunNow = async () => {
|
|
setIsRunning(true);
|
|
const response = await automationApi.runNow();
|
|
if (response.success) {
|
|
// Start polling
|
|
loadCurrentRun();
|
|
}
|
|
setIsRunning(false);
|
|
};
|
|
|
|
return (
|
|
<div className="p-6 space-y-6">
|
|
{/* Header */}
|
|
<div className="flex items-center justify-between">
|
|
<div>
|
|
<h1 className="text-2xl font-bold text-gray-900 dark:text-white flex items-center gap-2">
|
|
<BoltIcon className="w-7 h-7 text-warning-500" />
|
|
AI Automation Pipeline
|
|
</h1>
|
|
<p className="text-sm text-gray-600 dark:text-gray-400 mt-1">
|
|
Automated content generation from keywords to review
|
|
</p>
|
|
</div>
|
|
|
|
<div className="flex gap-2">
|
|
<Button
|
|
onClick={handleRunNow}
|
|
disabled={isRunning || currentRun?.status === 'running'}
|
|
icon={<PlayIcon className="w-4 h-4" />}
|
|
variant="primary"
|
|
>
|
|
{isRunning ? 'Starting...' : 'Run Now'}
|
|
</Button>
|
|
|
|
<Button
|
|
onClick={() => setShowConfigModal(true)}
|
|
icon={<SettingsIcon className="w-4 h-4" />}
|
|
variant="secondary"
|
|
>
|
|
Configure
|
|
</Button>
|
|
</div>
|
|
</div>
|
|
|
|
{/* Schedule Info */}
|
|
{config && (
|
|
<div className="bg-blue-50 dark:bg-blue-900/20 border border-blue-200 dark:border-blue-800 rounded-lg p-4">
|
|
<div className="flex items-center justify-between">
|
|
<div>
|
|
<div className="text-sm font-medium text-blue-900 dark:text-blue-100">
|
|
{config.is_enabled ? '⏰ Scheduled' : '⏸ Paused'}
|
|
</div>
|
|
<div className="text-sm text-blue-700 dark:text-blue-300 mt-1">
|
|
{config.is_enabled
|
|
? `Next Run: ${config.next_run_at} (${config.frequency})`
|
|
: 'Scheduling disabled'
|
|
}
|
|
</div>
|
|
</div>
|
|
|
|
<div className="text-right">
|
|
<div className="text-2xl font-bold text-blue-900 dark:text-blue-100">
|
|
{currentRun?.total_credits_used || 0}
|
|
</div>
|
|
<div className="text-xs text-blue-700 dark:text-blue-300">
|
|
Credits Used
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
)}
|
|
|
|
{/* Pipeline Overview */}
|
|
<div className="bg-white dark:bg-gray-800 rounded-lg p-6 border border-gray-200 dark:border-gray-700">
|
|
<h2 className="text-lg font-semibold mb-4">Pipeline Overview</h2>
|
|
<div className="flex items-center gap-2 text-sm">
|
|
<span className="px-3 py-1 bg-amber-100 dark:bg-amber-900/30 text-amber-900 dark:text-amber-200 rounded">
|
|
Keywords
|
|
</span>
|
|
<span>→</span>
|
|
<span className="px-3 py-1 bg-green-100 dark:bg-green-900/30 text-green-900 dark:text-green-200 rounded">
|
|
Clusters
|
|
</span>
|
|
<span>→</span>
|
|
<span className="px-3 py-1 bg-purple-100 dark:bg-purple-900/30 text-purple-900 dark:text-purple-200 rounded">
|
|
Ideas
|
|
</span>
|
|
<span>→</span>
|
|
<span className="px-3 py-1 bg-blue-100 dark:bg-blue-900/30 text-blue-900 dark:text-blue-200 rounded">
|
|
Tasks
|
|
</span>
|
|
<span>→</span>
|
|
<span className="px-3 py-1 bg-orange-100 dark:bg-orange-900/30 text-orange-900 dark:text-orange-200 rounded">
|
|
Content
|
|
</span>
|
|
<span>→</span>
|
|
<span className="px-3 py-1 bg-gray-100 dark:bg-gray-700 text-gray-900 dark:text-gray-200 rounded">
|
|
Review
|
|
</span>
|
|
</div>
|
|
|
|
{currentRun && (
|
|
<div className="mt-4 flex items-center gap-4">
|
|
<div className="flex-1 bg-gray-200 dark:bg-gray-700 rounded-full h-2">
|
|
<div
|
|
className="bg-blue-500 h-2 rounded-full transition-all duration-500"
|
|
style={{ width: `${(currentRun.current_stage / 7) * 100}%` }}
|
|
/>
|
|
</div>
|
|
<span className="text-sm font-medium">
|
|
Stage {currentRun.current_stage}/7
|
|
</span>
|
|
</div>
|
|
)}
|
|
</div>
|
|
|
|
{/* Stage Cards */}
|
|
<div className="space-y-4">
|
|
<StageCard
|
|
number={1}
|
|
name="Keywords → Clusters (AI)"
|
|
status={getStageStatus(currentRun, 1)}
|
|
result={currentRun?.stage_1_result}
|
|
/>
|
|
|
|
<StageCard
|
|
number={2}
|
|
name="Clusters → Ideas (AI)"
|
|
status={getStageStatus(currentRun, 2)}
|
|
result={currentRun?.stage_2_result}
|
|
/>
|
|
|
|
<StageCard
|
|
number={3}
|
|
name="Ideas → Tasks (Local)"
|
|
status={getStageStatus(currentRun, 3)}
|
|
result={currentRun?.stage_3_result}
|
|
/>
|
|
|
|
<StageCard
|
|
number={4}
|
|
name="Tasks → Content (AI)"
|
|
status={getStageStatus(currentRun, 4)}
|
|
result={currentRun?.stage_4_result}
|
|
/>
|
|
|
|
<StageCard
|
|
number={5}
|
|
name="Content → Image Prompts (AI)"
|
|
status={getStageStatus(currentRun, 5)}
|
|
result={currentRun?.stage_5_result}
|
|
/>
|
|
|
|
<StageCard
|
|
number={6}
|
|
name="Image Prompts → Images (AI)"
|
|
status={getStageStatus(currentRun, 6)}
|
|
result={currentRun?.stage_6_result}
|
|
/>
|
|
|
|
<StageCard
|
|
number={7}
|
|
name="Manual Review Gate"
|
|
status={getStageStatus(currentRun, 7)}
|
|
result={currentRun?.stage_7_result}
|
|
isManualGate
|
|
/>
|
|
</div>
|
|
|
|
{/* Activity Log */}
|
|
<ActivityLog log={activityLog} />
|
|
|
|
{/* Config Modal */}
|
|
{showConfigModal && (
|
|
<ConfigModal
|
|
isOpen={showConfigModal}
|
|
onClose={() => setShowConfigModal(false)}
|
|
config={config}
|
|
onSave={loadConfig}
|
|
/>
|
|
)}
|
|
</div>
|
|
);
|
|
}
|
|
|
|
function getStageStatus(run, stageNumber) {
|
|
if (!run) return 'waiting';
|
|
if (run.status === 'running' && run.current_stage === stageNumber) return 'running';
|
|
if (run.current_stage > stageNumber) return 'completed';
|
|
return 'waiting';
|
|
}
|
|
```
|
|
|
|
#### 4.2 Add to Sidebar Menu
|
|
**File:** `frontend/src/layouts/AppSidebar.tsx` (UPDATE)
|
|
|
|
```tsx
|
|
// Add after Sites menu item
|
|
{
|
|
path: '/automation',
|
|
label: 'Automation',
|
|
icon: <BoltIcon className="w-5 h-5" />,
|
|
badge: automationRunning ? { text: 'Running', color: 'green' } : null
|
|
},
|
|
```
|
|
|
|
#### 4.3 API Service
|
|
**File:** `frontend/src/services/api/automationApi.ts` (NEW)
|
|
|
|
```typescript
|
|
import { apiClient } from './apiClient';
|
|
|
|
export const automationApi = {
|
|
runNow: () => apiClient.post('/automation/run_now/'),
|
|
|
|
getCurrentRun: () => apiClient.get('/automation/current_run/'),
|
|
|
|
getConfig: () => apiClient.get('/automation/config/'),
|
|
|
|
updateConfig: (data: any) => apiClient.put('/automation/config/', data),
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 5: Implementation Checklist
|
|
|
|
#### Week 1: Backend Foundation
|
|
- [ ] Create `automation` module directory structure
|
|
- [ ] Implement `AutomationRun` and `AutomationConfig` models
|
|
- [ ] Run migrations
|
|
- [ ] Implement `AutomationLogger` service
|
|
- [ ] Test file logging manually
|
|
|
|
#### Week 2: Core Service
|
|
- [ ] Implement `AutomationService` class
|
|
- [ ] Implement `run_stage_1()` (Keywords → Clusters)
|
|
- [ ] Test stage 1 in isolation
|
|
- [ ] Implement `run_stage_2()` (Clusters → Ideas)
|
|
- [ ] Test stage 2 in isolation
|
|
- [ ] Implement remaining stages 3-7
|
|
|
|
#### Week 3: API & Scheduling
|
|
- [ ] Create `AutomationViewSet` with endpoints
|
|
- [ ] Test manual run via API
|
|
- [ ] Implement Celery periodic task
|
|
- [ ] Test scheduled runs
|
|
- [ ] Add error handling and rollback
|
|
|
|
#### Week 4: Frontend
|
|
- [ ] Create Automation page component
|
|
- [ ] Implement StageCard component
|
|
- [ ] Implement ActivityLog component
|
|
- [ ] Implement ConfigModal component
|
|
- [ ] Add to sidebar menu
|
|
- [ ] Test full UI flow
|
|
|
|
#### Week 5: Testing & Polish
|
|
- [ ] End-to-end testing (manual + scheduled)
|
|
- [ ] Load testing (100+ keywords)
|
|
- [ ] Credit calculation verification
|
|
- [ ] Log file verification
|
|
- [ ] UI polish and responsiveness
|
|
- [ ] Documentation update
|
|
|
|
---
|
|
|
|
### Phase 6: Safety Mechanisms
|
|
|
|
#### 6.1 Pause/Resume
|
|
```python
|
|
# In AutomationService
|
|
def pause_run(self):
|
|
self.run.status = 'paused'
|
|
self.run.save()
|
|
|
|
def resume_run(self):
|
|
self.run.status = 'running'
|
|
self.run.save()
|
|
# Resume from current_stage
|
|
```
|
|
|
|
#### 6.2 Rollback on Error
|
|
```python
|
|
# Each stage wraps in try-except
|
|
try:
|
|
self.run_stage_1()
|
|
except Exception as e:
|
|
self.logger.log_stage_error(self.run.run_id, 1, str(e))
|
|
self.run.status = 'failed'
|
|
self.run.error_message = str(e)
|
|
self.run.save()
|
|
# Optionally: rollback created records
|
|
raise
|
|
```
|
|
|
|
#### 6.3 Credit Pre-Check
|
|
```python
|
|
# Before starting, estimate total credits needed
|
|
def estimate_credits(self):
|
|
keywords = Keywords.objects.filter(status='new', cluster__isnull=True).count()
|
|
clusters = Clusters.objects.filter(ideas__isnull=True).count()
|
|
# ... etc
|
|
|
|
total_estimate = (keywords / 5) + (clusters * 2) + ...
|
|
|
|
if self.account.credits_balance < total_estimate:
|
|
raise InsufficientCreditsError(f"Need ~{total_estimate} credits")
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 7: Monitoring & Observability
|
|
|
|
#### 7.1 Dashboard Metrics
|
|
- Total runs (today/week/month)
|
|
- Success rate
|
|
- Average credits per run
|
|
- Average time per stage
|
|
- Content pieces generated
|
|
|
|
#### 7.2 Alerts
|
|
- Email when run completes
|
|
- Email on failure
|
|
- Slack notification (optional)
|
|
|
|
---
|
|
|
|
## 🎨 UI/UX HIGHLIGHTS
|
|
|
|
### Rich Visual Design
|
|
- **Stage Cards** with status badges (waiting/running/completed/failed)
|
|
- **Live Progress Bar** for current stage
|
|
- **Activity Feed** with timestamps and color-coded messages
|
|
- **Credit Counter** with real-time updates
|
|
- **Schedule Badge** showing next run time
|
|
|
|
### User Experience
|
|
- **One-Click Run** - Single "Run Now" button
|
|
- **Real-Time Updates** - Auto-refreshes every 3 seconds when running
|
|
- **Clear Status** - Visual indicators for each stage
|
|
- **Easy Config** - Modal for schedule settings
|
|
- **Error Clarity** - Detailed error messages with stage number
|
|
|
|
---
|
|
|
|
## 🔧 TROUBLESHOOTING GUIDE
|
|
|
|
### Issue: Stage stuck in "running"
|
|
**Solution:**
|
|
1. Check `/logs/automation/{account}/{site}/{sector}/{run_id}/stage_X.log`
|
|
2. Look for last log entry
|
|
3. Check Celery worker logs
|
|
4. Manually mark stage complete or restart
|
|
|
|
### Issue: Credits deducted but no results
|
|
**Solution:**
|
|
1. Check stage log for AI task_id
|
|
2. Query task progress endpoint
|
|
3. Verify AI function completed
|
|
4. Rollback transaction if needed
|
|
|
|
### Issue: Duplicate clusters created
|
|
**Solution:**
|
|
1. Add unique constraint on cluster name per sector
|
|
2. Check deduplication logic in ClusteringService
|
|
3. Review stage_1 logs for batch processing
|
|
|
|
---
|
|
|
|
## 📊 SUCCESS METRICS
|
|
|
|
After implementation, measure:
|
|
- **Automation adoption rate** (% of sites using scheduled runs)
|
|
- **Content generation volume** (pieces per day/week)
|
|
- **Time savings** (manual hours vs automated)
|
|
- **Credit efficiency** (credits per content piece)
|
|
- **Error rate** (failed runs / total runs)
|
|
|
|
---
|
|
|
|
## 🚀 FUTURE ENHANCEMENTS
|
|
|
|
### Phase 8: Advanced Features
|
|
- **Conditional stages** (skip if no data)
|
|
- **Parallel processing** (multiple tasks at once in stage 4)
|
|
- **Smart scheduling** (avoid peak hours)
|
|
- **A/B testing** (test different prompts)
|
|
- **Content quality scoring** (auto-reject low scores)
|
|
|
|
### Phase 9: Integrations
|
|
- **WordPress auto-publish** (with approval workflow)
|
|
- **Analytics tracking** (measure content performance)
|
|
- **Social media posting** (auto-share published content)
|
|
|
|
---
|
|
|
|
**END OF IMPLEMENTATION PLAN**
|
|
|
|
This plan provides a complete, production-ready automation system that:
|
|
✅ Reuses all existing AI functions (zero duplication)
|
|
✅ Modular and maintainable (each stage independent)
|
|
✅ Observable and debuggable (file logs + database records)
|
|
✅ Safe and reliable (error handling + rollback)
|
|
✅ Rich UI/UX (real-time updates + visual feedback)
|
|
✅ Scalable (handles 100+ keywords efficiently)
|