salman/igny8
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
316cafab1b1a1c46d318924c206a77b1b93db0ed
igny8/docs/automation/AUTOMATION-IMPLEMENTATION-README.md
IGNY8 VPS (Salman) 316cafab1b automation fixes
2025-12-03 12:24:59 +00:00

13 KiB
Raw Blame History

AI Automation Pipeline - Implementation Complete

Overview

The IGNY8 AI Automation Pipeline is a fully automated content creation system that orchestrates existing AI functions into a 7-stage pipeline, transforming keywords into published content without manual intervention.

Architecture

Backend Components

1. Models (/backend/igny8_core/business/automation/models.py)

AutomationConfig

  • Per-site configuration for automation
  • Fields: is_enabled, frequency (daily/weekly/monthly), scheduled_time, batch sizes for all 7 stages
  • OneToOne relationship with Site model

AutomationRun

  • Tracks execution of automation runs
  • Fields: run_id, status, current_stage, stage_1_result through stage_7_result (JSON), total_credits_used
  • Status choices: running, paused, completed, failed

2. Services

AutomationLogger (services/automation_logger.py)

  • File-based logging system
  • Log structure: logs/automation/{account_id}/{site_id}/{run_id}/
  • Files: automation_run.log, stage_1.log through stage_7.log
  • Methods: start_run(), log_stage_start(), log_stage_progress(), log_stage_complete(), log_stage_error()

AutomationService (services/automation_service.py)

  • Core orchestrator for automation pipeline
  • Methods:
    • start_automation() - Initialize new run with credit check
    • run_stage_1() through run_stage_7() - Execute each pipeline stage
    • pause_automation(), resume_automation() - Control run execution
    • estimate_credits() - Pre-run credit estimation
    • from_run_id() - Create service from existing run

3. API Endpoints (views.py)

All endpoints at /api/v1/automation/:

  • GET /config/?site_id=123 - Get automation configuration
  • PUT /update_config/?site_id=123 - Update configuration
  • POST /run_now/?site_id=123 - Trigger immediate run
  • GET /current_run/?site_id=123 - Get active run status
  • POST /pause/?run_id=abc - Pause running automation
  • POST /resume/?run_id=abc - Resume paused automation
  • GET /history/?site_id=123 - Get past runs (last 20)
  • GET /logs/?run_id=abc&lines=100 - Get run logs
  • GET /estimate/?site_id=123 - Estimate credits needed

4. Celery Tasks (tasks.py)

check_scheduled_automations

  • Runs hourly via Celery Beat
  • Checks AutomationConfig records for scheduled runs
  • Triggers automation based on frequency and scheduled_time

run_automation_task

  • Main background task that executes all 7 stages sequentially
  • Called by run_now API endpoint or scheduled trigger
  • Handles errors and updates AutomationRun status

resume_automation_task

  • Resumes paused automation from current_stage
  • Called by resume API endpoint

5. Database Migration

Located at /backend/igny8_core/business/automation/migrations/0001_initial.py

Run with: python manage.py migrate

Frontend Components

1. Service (/frontend/src/services/automationService.ts)

TypeScript API client with methods matching backend endpoints:

  • getConfig(), updateConfig(), runNow(), getCurrentRun()
  • pause(), resume(), getHistory(), getLogs(), estimate()

2. Pages

AutomationPage (pages/Automation/AutomationPage.tsx)

  • Main dashboard at /automation
  • Displays current run status, stage progress, activity log, history
  • Real-time polling (5s interval when run is active)
  • Controls: Run Now, Pause, Resume, Configure

3. Components

StageCard (components/Automation/StageCard.tsx)

  • Visual representation of each stage (1-7)
  • Shows status: pending (⏳), active (🔄), complete (✅)
  • Displays stage results (items processed, credits used, etc.)

ActivityLog (components/Automation/ActivityLog.tsx)

  • Real-time log viewer with terminal-style display
  • Auto-refreshes every 3 seconds
  • Configurable line count (50, 100, 200, 500)

ConfigModal (components/Automation/ConfigModal.tsx)

  • Modal for editing automation settings
  • Fields: Enable/disable, frequency, scheduled time, batch sizes
  • Form validation and save

RunHistory (components/Automation/RunHistory.tsx)

  • Table of past automation runs
  • Columns: run_id, status, trigger, started, completed, credits, stage
  • Status badges with color coding

7-Stage Pipeline

Stage 1: Keywords → Clusters (AI)

  • Query: Keywords with status='new', cluster__isnull=True, disabled=False
  • Batch Size: Default 20 keywords
  • AI Function: AutoCluster().execute()
  • Output: Creates Clusters records
  • Credits: ~1 per 5 keywords

Stage 2: Clusters → Ideas (AI)

  • Query: Clusters with status='new', exclude those with existing ideas
  • Batch Size: Default 1 cluster
  • AI Function: GenerateIdeas().execute()
  • Output: Creates ContentIdeas records
  • Credits: ~2 per cluster

Stage 3: Ideas → Tasks (Local Queue)

  • Query: ContentIdeas with status='new'
  • Batch Size: Default 20 ideas
  • Operation: Local database creation (no AI)
  • Output: Creates Tasks records with status='queued'
  • Credits: 0 (local operation)

Stage 4: Tasks → Content (AI)

  • Query: Tasks with status='queued', content__isnull=True
  • Batch Size: Default 1 task
  • AI Function: GenerateContent().execute()
  • Output: Creates Content records with status='draft'
  • Credits: ~5 per content (2500 words avg)

Stage 5: Content → Image Prompts (AI)

  • Query: Content with status='draft', images_count=0 (annotated)
  • Batch Size: Default 1 content
  • AI Function: GenerateImagePromptsFunction().execute()
  • Output: Creates Images records with status='pending' (contains prompts)
  • Credits: ~2 per content (4 prompts avg)

Stage 6: Image Prompts → Generated Images (AI)

  • Query: Images with status='pending'
  • Batch Size: Default 1 image
  • AI Function: GenerateImages().execute()
  • Output: Updates Images to status='generated' with image_url
  • Side Effect: Automatically sets Content.status='review' when all images complete (via ai/tasks.py:723)
  • Credits: ~2 per image

Stage 7: Manual Review Gate

  • Query: Content with status='review'
  • Operation: Count only, no processing
  • Output: Returns list of content IDs ready for review
  • Credits: 0

Key Design Principles

1. NO Duplication of AI Function Logic

The automation system ONLY handles:

  • Batch selection and sequencing
  • Stage orchestration
  • Credit estimation and checking
  • Progress tracking and logging
  • Scheduling and triggers

It does NOT handle:

  • Credit deduction (done by AIEngine.execute() at line 395)
  • Status updates (done within AI functions)
  • Progress tracking (StepTracker emits events automatically)

2. Correct Image Model Understanding

  • NO separate ImagePrompts model - this was a misunderstanding
  • Images model serves dual purpose:
    • status='pending' = has prompt, needs image URL
    • status='generated' = has image_url
  • Stage 5 creates Images records with prompts
  • Stage 6 updates same records with URLs

3. Automatic Content Status Changes

  • Content.status changes from 'draft' to 'review' automatically
  • Happens in ai/tasks.py:723 when all images complete
  • Automation does NOT manually update this status

4. Distributed Locking

  • Uses Django cache with automation_lock_{site.id} key
  • 6-hour timeout to prevent deadlocks
  • Released on completion, pause, or failure

Configuration

Schedule Configuration UI

Located at /automation page → [Configure] button

Options:

  • Enable/Disable: Toggle automation on/off
  • Frequency: Daily, Weekly (Mondays), Monthly (1st)
  • Scheduled Time: Time of day to run (24-hour format)
  • Batch Sizes: Per-stage item counts

Defaults:

  • Stage 1: 20 keywords
  • Stage 2: 1 cluster
  • Stage 3: 20 ideas
  • Stage 4: 1 task
  • Stage 5: 1 content
  • Stage 6: 1 image

Credit Estimation

Before starting, system estimates:

  • Stage 1: keywords_count / 5
  • Stage 2: clusters_count * 2
  • Stage 4: tasks_count * 5
  • Stage 5: content_count * 2
  • Stage 6: content_count * 8 (4 images * 2 credits avg)

Requires 20% buffer: account.credits_balance >= estimated * 1.2

Deployment Checklist

Backend

  1. ✅ Models created in business/automation/models.py
  2. ✅ Services created (AutomationLogger, AutomationService)
  3. ✅ Views created (AutomationViewSet)
  4. ✅ URLs registered in igny8_core/urls.py
  5. ✅ Celery tasks created (check_scheduled_automations, run_automation_task, resume_automation_task)
  6. ✅ Celery beat schedule updated in celery.py
  7. ⏳ Migration created (needs to run: python manage.py migrate)

Frontend

  1. ✅ API service created (services/automationService.ts)
  2. ✅ Main page created (pages/Automation/AutomationPage.tsx)
  3. ✅ Components created (StageCard, ActivityLog, ConfigModal, RunHistory)
  4. ⏳ Route registration (add to router: /automation → AutomationPage)

Infrastructure

  1. ⏳ Celery worker running (for background tasks)
  2. ⏳ Celery beat running (for scheduled checks)
  3. ⏳ Redis/cache backend configured (for distributed locks)
  4. ⏳ Log directory writable: /data/app/igny8/backend/logs/automation/

Usage

Manual Trigger

  1. Navigate to /automation page
  2. Verify credit balance is sufficient (shows in header)
  3. Click [Run Now] button
  4. Monitor progress in real-time:
    • Stage cards show current progress
    • Activity log shows detailed logs
    • Credits used updates live

Scheduled Automation

  1. Navigate to /automation page
  2. Click [Configure] button
  3. Enable automation
  4. Set frequency and time
  5. Configure batch sizes
  6. Save configuration
  7. Automation will run automatically at scheduled time

Pause/Resume

  • During active run, click [Pause] to halt execution
  • Click [Resume] to continue from current stage
  • Useful for credit management or issue investigation

Viewing History

  • Run History table shows last 20 runs
  • Filter by status, date, trigger type
  • Click run_id to view detailed logs

Monitoring

Log Files

Located at: logs/automation/{account_id}/{site_id}/{run_id}/

  • automation_run.log - Main activity log
  • stage_1.log through stage_7.log - Stage-specific logs

Database Records

AutomationRun table tracks:

  • Current status and stage
  • Stage results (JSON)
  • Credits used
  • Error messages
  • Timestamps

AutomationConfig table tracks:

  • Last run timestamp
  • Next scheduled run
  • Configuration changes

Troubleshooting

Run stuck in "running" status

  1. Check Celery worker logs: docker logs <celery_container>
  2. Check for cache lock: redis-cli GET automation_lock_<site_id>
  3. Manually release lock if needed: redis-cli DEL automation_lock_<site_id>
  4. Update run status: AutomationRun.objects.filter(run_id='...').update(status='failed')

Insufficient credits

  1. Check estimate: GET /api/v1/automation/estimate/?site_id=123
  2. Add credits via billing page
  3. Retry run

Stage failures

  1. View logs: GET /api/v1/automation/logs/?run_id=...
  2. Check error_message field in AutomationRun
  3. Verify AI function is working: test individually via existing UI
  4. Check credit balance mid-run

Future Enhancements

  1. Email notifications on completion/failure
  2. Slack/webhook integrations
  3. Per-stage retry logic
  4. Partial run resumption after failure
  5. Advanced scheduling (specific days, multiple times)
  6. Content preview before Stage 7
  7. Auto-publish to WordPress option
  8. Credit usage analytics and forecasting

File Locations Summary

backend/igny8_core/business/automation/
├── __init__.py
├── models.py                   # AutomationConfig, AutomationRun
├── views.py                    # AutomationViewSet (API endpoints)
├── tasks.py                    # Celery tasks
├── urls.py                     # URL routing
├── migrations/
│   ├── __init__.py
│   └── 0001_initial.py        # Database schema
└── services/
    ├── __init__.py
    ├── automation_logger.py    # File logging service
    └── automation_service.py   # Core orchestrator

frontend/src/
├── services/
│   └── automationService.ts    # API client
├── pages/Automation/
│   └── AutomationPage.tsx      # Main dashboard
└── components/Automation/
    ├── StageCard.tsx           # Stage status display
    ├── ActivityLog.tsx         # Log viewer
    ├── ConfigModal.tsx         # Settings modal
    └── RunHistory.tsx          # Past runs table

Credits

Implemented according to automation-plan.md with corrections for:

  • Image model structure (no separate ImagePrompts)
  • AI function internal logic (no duplication)
  • Content status changes (automatic in background)
Reference in New Issue View Git Blame Copy Permalink