Files
igny8/docs/08-AUTOMATION-IMPLEMENTATION-PLAN.md
2025-11-30 13:02:27 +05:00

28 KiB

IGNY8 Automation System Implementation Plan

Overview

This document provides a detailed, step-by-step implementation plan for completing the IGNY8 Automation system. The backend infrastructure is already in place; this plan focuses on extending functionality and building the frontend interface.

Current State Analysis

Already Implemented (Backend)

  1. Data Models (backend/igny8_core/business/automation/models.py)

  2. Core Services (backend/igny8_core/business/automation/services/)

  3. API Endpoints (backend/igny8_core/modules/automation/views.py)

Missing Components

  1. Backend Extensions:

    • ActionExecutor only supports 3 actions (cluster_keywords, generate_ideas, generate_content)
    • Missing: generate_images, generate_site_structure, generate_page_content, optimize_content, publish_to_wordpress
    • No Celery task integration for async execution
    • No progress tracking integration
    • No webhook support for event triggers
  2. Frontend (Complete):

    • No automation module pages
    • No rule management interface
    • No workflow visualizer
    • No execution monitor
    • No progress tracking UI
  3. Integration:

    • No connection to existing AI functions
    • No WordPress publish integration
    • No credit cost estimation

Implementation Phases

Phase 1: Backend Extensions (Week 1)

1.1 Extend ActionExecutor

File: backend/igny8_core/business/automation/services/action_executor.py

Changes Required:

# Add new action methods to ActionExecutor class

def _execute_generate_images(self, params, rule):
    """Execute generate images action"""
    content_id = params.get('content_id')
    provider = params.get('provider', 'dall-e')
    
    try:
        from igny8_core.business.content.services.image_generation_service import ImageGenerationService
        service = ImageGenerationService()
        
        result = service.generate_images(
            content_id=content_id,
            provider=provider,
            account=rule.account
        )
        return result
    except Exception as e:
        logger.error(f"Error generating images: {str(e)}", exc_info=True)
        return {'success': False, 'error': str(e)}

def _execute_generate_site_structure(self, params, rule):
    """Execute generate site structure action"""
    site_id = params.get('site_id') or (rule.site.id if rule.site else None)
    
    try:
        from igny8_core.business.sites.services.site_generation_service import SiteGenerationService
        service = SiteGenerationService()
        
        result = service.generate_structure(
            site_id=site_id,
            account=rule.account
        )
        return result
    except Exception as e:
        logger.error(f"Error generating site structure: {str(e)}", exc_info=True)
        return {'success': False, 'error': str(e)}

def _execute_generate_page_content(self, params, rule):
    """Execute generate page content action"""
    page_ids = params.get('page_ids', [])
    
    try:
        from igny8_core.business.sites.services.page_generation_service import PageGenerationService
        service = PageGenerationService()
        
        result = service.generate_content(
            page_ids=page_ids,
            account=rule.account
        )
        return result
    except Exception as e:
        logger.error(f"Error generating page content: {str(e)}", exc_info=True)
        return {'success': False, 'error': str(e)}

def _execute_optimize_content(self, params, rule):
    """Execute optimize content action"""
    content_ids = params.get('content_ids', [])
    
    try:
        from igny8_core.business.optimizer.services.optimization_service import OptimizationService
        service = OptimizationService()
        
        result = service.optimize_content(
            content_ids=content_ids,
            account=rule.account
        )
        return result
    except Exception as e:
        logger.error(f"Error optimizing content: {str(e)}", exc_info=True)
        return {'success': False, 'error': str(e)}

def _execute_publish_to_wordpress(self, params, rule):
    """Execute publish to WordPress action"""
    content_ids = params.get('content_ids', [])
    integration_id = params.get('integration_id')
    
    try:
        from igny8_core.business.integration.services.wordpress_service import WordPressService
        service = WordPressService()
        
        result = service.publish_content(
            content_ids=content_ids,
            integration_id=integration_id,
            account=rule.account
        )
        return result
    except Exception as e:
        logger.error(f"Error publishing to WordPress: {str(e)}", exc_info=True)
        return {'success': False, 'error': str(e)}

# Update execute() method to handle new action types
def execute(self, action, context, rule):
    """Execute a single action"""
    action_type = action.get('type')
    params = action.get('params', {})
    
    action_map = {
        'cluster_keywords': self._execute_cluster_keywords,
        'generate_ideas': self._execute_generate_ideas,
        'generate_content': self._execute_generate_content,
        'generate_images': self._execute_generate_images,
        'generate_site_structure': self._execute_generate_site_structure,
        'generate_page_content': self._execute_generate_page_content,
        'optimize_content': self._execute_optimize_content,
        'publish_to_wordpress': self._execute_publish_to_wordpress,
    }
    
    executor = action_map.get(action_type)
    if executor:
        return executor(params, rule)
    else:
        logger.warning(f"Unknown action type: {action_type}")
        return {'success': False, 'error': f'Unknown action type: {action_type}'}

Estimated Time: 4 hours

1.2 Create Celery Tasks

File: backend/igny8_core/business/automation/tasks.py

Changes Required:

"""
Celery tasks for automation
"""
from celery import shared_task
from django.utils import timezone
from igny8_core.business.automation.models import AutomationRule, ScheduledTask
from igny8_core.business.automation.services.automation_service import AutomationService
import logging

logger = logging.getLogger(__name__)


@shared_task(bind=True, name='automation.execute_rule')
def execute_rule_task(self, rule_id, context=None):
    """
    Execute an automation rule asynchronously
    
    Args:
        rule_id: AutomationRule ID
        context: Optional context dict
    """
    try:
        rule = AutomationRule.objects.get(id=rule_id)
        service = AutomationService()
        
        # Update task progress
        self.update_state(
            state='PROGRESS',
            meta={'phase': 'Initializing', 'progress': 0}
        )
        
        result = service.execute_rule(rule, context or {})
        
        self.update_state(
            state='SUCCESS',
            meta={'phase': 'Completed', 'progress': 100, 'result': result}
        )
        
        return result
    except Exception as e:
        logger.error(f"Error executing rule {rule_id}: {str(e)}", exc_info=True)
        self.update_state(
            state='FAILURE',
            meta={'error': str(e)}
        )
        raise


@shared_task(name='automation.execute_scheduled_rules')
def execute_scheduled_rules_task():
    """
    Execute all scheduled rules that are due
    Called by Celery Beat
    """
    service = AutomationService()
    result = service.execute_scheduled_rules()
    logger.info(f"Scheduled rules execution: {result}")
    return result


@shared_task(name='automation.cleanup_old_tasks')
def cleanup_old_tasks():
    """
    Clean up old scheduled tasks
    Keep last 30 days only
    """
    from datetime import timedelta
    cutoff_date = timezone.now() - timedelta(days=30)
    
    deleted_count = ScheduledTask.objects.filter(
        created_at__lt=cutoff_date,
        status__in=['completed', 'failed', 'cancelled']
    ).delete()[0]
    
    logger.info(f"Cleaned up {deleted_count} old scheduled tasks")
    return {'deleted': deleted_count}

Estimated Time: 2 hours

1.3 Add Progress Tracking

File: backend/igny8_core/business/automation/services/automation_service.py

Changes Required:

# Add to execute_rule method

def execute_rule(self, rule, context=None):
    """Execute an automation rule with progress tracking"""
    
    # ... existing code ...
    
    # Execute via rule engine with progress tracking
    try:
        from celery import current_task
        
        # Update progress for each action
        total_actions = len(rule.actions)
        for idx, action in enumerate(rule.actions):
            if current_task:
                current_task.update_state(
                    state='PROGRESS',
                    meta={
                        'phase': f'Executing action {idx + 1}/{total_actions}',
                        'progress': int((idx / total_actions) * 100),
                        'action': action.get('type')
                    }
                )
            
            # Execute action
            action_result = self.rule_engine.execute_action(action, context, rule)
            
            # Store result
            results.append(action_result)
        
        # ... rest of existing code ...

Estimated Time: 2 hours

1.4 Add API Endpoints

File: backend/igny8_core/modules/automation/views.py

Changes Required:

# Add new endpoints to AutomationRuleViewSet

@action(detail=True, methods=['get'], url_path='progress', url_name='progress')
def get_progress(self, request, pk=None):
    """Get execution progress for a rule"""
    rule = self.get_object()
    
    # Get latest task for this rule
    from celery.result import AsyncResult
    from igny8_core.business.automation.models import ScheduledTask
    
    latest_task = ScheduledTask.objects.filter(
        automation_rule=rule,
        status='running'
    ).order_by('-created_at').first()
    
    if not latest_task or not latest_task.metadata.get('task_id'):
        return success_response(
            data={'status': 'idle'},
            request=request
        )
    
    task_id = latest_task.metadata['task_id']
    task_result = AsyncResult(task_id)
    
    return success_response(
        data={
            'status': task_result.state,
            'progress': task_result.info.get('progress', 0) if task_result.info else 0,
            'phase': task_result.info.get('phase', 'Unknown') if task_result.info else 'Unknown',
            'result': task_result.result if task_result.successful() else None
        },
        request=request
    )

@action(detail=False, methods=['get'], url_path='action-types', url_name='action-types')
def get_action_types(self, request):
    """Get available action types"""
    action_types = [
        {
            'type': 'cluster_keywords',
            'name': 'Cluster Keywords',
            'description': 'Automatically cluster keywords by topic',
            'params': ['keyword_ids', 'sector_id']
        },
        {
            'type': 'generate_ideas',
            'name': 'Generate Ideas',
            'description': 'Generate content ideas from clusters',
            'params': ['cluster_ids']
        },
        {
            'type': 'generate_content',
            'name': 'Generate Content',
            'description': 'Generate article content from tasks',
            'params': ['task_ids']
        },
        {
            'type': 'generate_images',
            'name': 'Generate Images',
            'description': 'Generate images for content',
            'params': ['content_id', 'provider']
        },
        {
            'type': 'generate_site_structure',
            'name': 'Generate Site Structure',
            'description': 'Generate website structure',
            'params': ['site_id']
        },
        {
            'type': 'generate_page_content',
            'name': 'Generate Page Content',
            'description': 'Generate content for pages',
            'params': ['page_ids']
        },
        {
            'type': 'optimize_content',
            'name': 'Optimize Content',
            'description': 'Optimize content for SEO',
            'params': ['content_ids']
        },
        {
            'type': 'publish_to_wordpress',
            'name': 'Publish to WordPress',
            'description': 'Publish content to WordPress',
            'params': ['content_ids', 'integration_id']
        }
    ]
    
    return success_response(
        data=action_types,
        request=request
    )

@action(detail=False, methods=['get'], url_path='execution-history', url_name='execution-history')
def get_execution_history(self, request):
    """Get execution history for all rules"""
    from igny8_core.business.automation.models import ScheduledTask
    
    tasks = ScheduledTask.objects.filter(
        automation_rule__account=request.account
    ).select_related('automation_rule').order_by('-executed_at')[:50]
    
    serializer = ScheduledTaskSerializer(tasks, many=True)
    
    return success_response(
        data=serializer.data,
        request=request
    )

Estimated Time: 3 hours

Phase 2: Frontend Foundation (Week 2)

2.1 Create Automation Store

File: frontend/src/stores/automationStore.ts (NEW)

Implementation:

import { create } from 'zustand';
import { automationApi } from '../api/automation.api';

interface AutomationRule {
  id: string;
  name: string;
  description: string;
  trigger: 'schedule' | 'event' | 'manual';
  schedule?: string;
  conditions: any[];
  actions: any[];
  is_active: boolean;
  status: 'active' | 'inactive' | 'paused';
  last_executed_at?: string;
  execution_count: number;
}

interface ScheduledTask {
  id: string;
  automation_rule: string;
  scheduled_at: string;
  executed_at?: string;
  status: 'pending' | 'running' | 'completed' | 'failed' | 'cancelled';
  result: any;
  error_message?: string;
}

interface AutomationStore {
  // State
  rules: AutomationRule[];
  tasks: ScheduledTask[];
  actionTypes: any[];
  loading: boolean;
  error: string | null;
  executionProgress: {
    ruleId: string | null;
    status: string;
    progress: number;
    phase: string;
  } | null;
  
  // Actions
  fetchRules: () => Promise<void>;
  fetchRule: (id: string) => Promise<AutomationRule>;
  createRule: (rule: Partial<AutomationRule>) => Promise<AutomationRule>;
  updateRule: (id: string, rule: Partial<AutomationRule>) => Promise<AutomationRule>;
  deleteRule: (id: string) => Promise<void>;
  executeRule: (id: string, context?: any) => Promise<void>;
  fetchActionTypes: () => Promise<void>;
  fetchExecutionHistory: () => Promise<void>;
  fetchProgress: (ruleId: string) => Promise<void>;
  clearError: () => void;
}

export const useAutomationStore = create<AutomationStore>((set, get) => ({
  // Initial state
  rules: [],
  tasks: [],
  actionTypes: [],
  loading: false,
  error: null,
  executionProgress: null,
  
  // Actions
  fetchRules: async () => {
    set({ loading: true, error: null });
    try {
      const response = await automationApi.getRules();
      set({ rules: response.data, loading: false });
    } catch (error: any) {
      set({ error: error.message, loading: false });
    }
  },
  
  fetchRule: async (id: string) => {
    set({ loading: true, error: null });
    try {
      const response = await automationApi.getRule(id);
      set({ loading: false });
      return response.data;
    } catch (error: any) {
      set({ error: error.message, loading: false });
      throw error;
    }
  },
  
  createRule: async (rule: Partial<AutomationRule>) => {
    set({ loading: true, error: null });
    try {
      const response = await automationApi.createRule(rule);
      set(state => ({
        rules: [...state.rules, response.data],
        loading: false
      }));
      return response.data;
    } catch (error: any) {
      set({ error: error.message, loading: false });
      throw error;
    }
  },
  
  updateRule: async (id: string, rule: Partial<AutomationRule>) => {
    set({ loading: true, error: null });
    try {
      const response = await automationApi.updateRule(id, rule);
      set(state => ({
        rules: state.rules.map(r => r.id === id ? response.data : r),
        loading: false
      }));
      return response.data;
    } catch (error: any) {
      set({ error: error.message, loading: false });
      throw error;
    }
  },
  
  deleteRule: async (id: string) => {
    set({ loading: true, error: null });
    try {
      await automationApi.deleteRule(id);
      set(state => ({
        rules: state.rules.filter(r => r.id !== id),
        loading: false
      }));
    } catch (error: any) {
      set({ error: error.message, loading: false });
      throw error;
    }
  },
  
  executeRule: async (id: string, context?: any) => {
    set({ loading: true, error: null });
    try {
      await automationApi.executeRule(id, context);
      set({ loading: false });
      
      // Start polling for progress
      const pollProgress = setInterval(async () => {
        await get().fetchProgress(id);
        const progress = get().executionProgress;
        if (progress && (progress.status === 'SUCCESS' || progress.status === 'FAILURE')) {
          clearInterval(pollProgress);
        }
      }, 2000);
    } catch (error: any) {
      set({ error: error.message, loading: false });
      throw error;
    }
  },
  
  fetchActionTypes: async () => {
    try {
      const response = await automationApi.getActionTypes();
      set({ actionTypes: response.data });
    } catch (error: any) {
      set({ error: error.message });
    }
  },
  
  fetchExecutionHistory: async () => {
    set({ loading: true, error: null });
    try {
      const response = await automationApi.getExecutionHistory();
      set({ tasks: response.data, loading: false });
    } catch (error: any) {
      set({ error: error.message, loading: false });
    }
  },
  
  fetchProgress: async (ruleId: string) => {
    try {
      const response = await automationApi.getProgress(ruleId);
      set({ executionProgress: { ruleId, ...response.data } });
    } catch (error: any) {
      console.error('Error fetching progress:', error);
    }
  },
  
  clearError: () => set({ error: null })
}));

Estimated Time: 3 hours

2.2 Create Automation API Client

File: frontend/src/api/automation.api.ts (NEW)

Implementation:

import apiClient from './client';

export const automationApi = {
  // Rules
  getRules: () => apiClient.get('/automation/rules/'),
  getRule: (id: string) => apiClient.get(`/automation/rules/${id}/`),
  createRule: (data: any) => apiClient.post('/automation/rules/', data),
  updateRule: (id: string, data: any) => apiClient.patch(`/automation/rules/${id}/`, data),
  deleteRule: (id: string) => apiClient.delete(`/automation/rules/${id}/`),
  executeRule: (id: string, context?: any) => 
    apiClient.post(`/automation/rules/${id}/execute/`, { context }),
  
  // Progress
  getProgress: (ruleId: string) => 
    apiClient.get(`/automation/rules/${ruleId}/progress/`),
  
  // Action types
  getActionTypes: () => apiClient.get('/automation/rules/action-types/'),
  
  // Execution history
  getExecutionHistory: () => apiClient.get('/automation/rules/execution-history/'),
  
  // Scheduled tasks
  getTasks: () => apiClient.get('/automation/scheduled-tasks/'),
  getTask: (id: string) => apiClient.get(`/automation/scheduled-tasks/${id}/`)
};

Estimated Time: 1 hour

Phase 3: Frontend UI Components (Week 3)

3.1 Rule List Page

File: frontend/src/pages/automation/AutomationRulesPage.tsx (NEW)

Features:

  • Table view of all automation rules
  • Status indicators (active/inactive/paused)
  • Last execution time and count
  • Quick actions (execute, edit, delete)
  • Create new rule button

Estimated Time: 6 hours

3.2 Rule Editor Modal

File: frontend/src/components/automation/RuleEditorModal.tsx (NEW)

Features:

  • Form for rule name, description, trigger type
  • Schedule configuration (cron expression builder)
  • Condition builder (field, operator, value)
  • Action builder (type selector, parameter inputs)
  • Save/cancel buttons

Estimated Time: 8 hours

3.3 Workflow Visualizer

File: frontend/src/components/automation/WorkflowVisualizer.tsx (NEW)

Features:

  • Visual flowchart of rule actions
  • Drag-and-drop action reordering
  • Action configuration panel
  • Connection lines between actions
  • Status indicators during execution

Estimated Time: 12 hours

3.4 Execution Monitor

File: frontend/src/components/automation/ExecutionMonitor.tsx (NEW)

Features:

  • Real-time progress display
  • Current phase indicator
  • Progress percentage
  • Action-by-action status
  • Error display
  • Cancel execution button

Estimated Time: 6 hours

3.5 Execution History

File: frontend/src/components/automation/ExecutionHistory.tsx (NEW)

Features:

  • Table of past executions
  • Status badges (completed/failed/cancelled)
  • Execution time and duration
  • Result summary
  • Error details (if failed)
  • Re-run button

Estimated Time: 4 hours

Phase 4: Integration & Testing (Week 4)

4.1 Backend Integration Tests

File: backend/igny8_core/business/automation/tests/test_automation.py (NEW)

Test Cases:

  • Rule creation and validation
  • Condition evaluation
  • Action execution
  • Progress tracking
  • Error handling
  • Credit checking

Estimated Time: 8 hours

4.2 Frontend Component Tests

File: frontend/src/components/automation/__tests__/ (NEW)

Test Cases:

  • Rule list rendering
  • Rule editor form validation
  • Workflow visualizer interactions
  • Execution monitor updates
  • API integration

Estimated Time: 6 hours

4.3 End-to-End Testing

Manual Testing:

  • Create automation rule
  • Execute rule manually
  • Monitor progress
  • Verify results
  • Test scheduled execution
  • Test error scenarios

Estimated Time: 4 hours

4.4 Documentation

Files to Update:

  • User guide for automation module
  • API documentation
  • Developer guide for adding new actions

Estimated Time: 4 hours

Implementation Timeline

Week 1: Backend Extensions

  • Day 1-2: Extend ActionExecutor with all action types
  • Day 3: Create Celery tasks
  • Day 4: Add progress tracking
  • Day 5: Add API endpoints and testing

Week 2: Frontend Foundation

  • Day 1: Create automation store
  • Day 2: Create API client
  • Day 3: Setup routing and page structure
  • Day 4-5: Build rule list page

Week 3: Frontend UI

  • Day 1-2: Build rule editor modal
  • Day 3-4: Build workflow visualizer
  • Day 5: Build execution monitor and history

Week 4: Integration & Polish

  • Day 1-2: Backend integration tests
  • Day 3: Frontend component tests
  • Day 4: End-to-end testing
  • Day 5: Documentation and polish

Resource Requirements

Development Team

  • 1 Backend Developer (Full-time, 4 weeks)
  • 1 Frontend Developer (Full-time, 4 weeks)
  • 1 QA Engineer (Part-time, Week 4)

Infrastructure

  • Development environment
  • Staging environment for testing
  • Celery worker and beat scheduler
  • Redis for task queue

Risk Assessment

High Risk

  1. Workflow Visualizer Complexity

    • Mitigation: Use existing React Flow library
    • Fallback: Simpler list-based interface
  2. Real-time Progress Tracking

    • Mitigation: Implement polling with WebSocket fallback
    • Fallback: Manual refresh button

Medium Risk

  1. Action Execution Failures

    • Mitigation: Comprehensive error handling and retry logic
    • Fallback: Manual intervention and re-execution
  2. Credit Estimation Accuracy

    • Mitigation: Conservative estimates with buffer
    • Fallback: Manual credit approval for large operations

Low Risk

  1. UI/UX Complexity
    • Mitigation: Iterative design with user feedback
    • Fallback: Simplified interface

Success Criteria

Functional Requirements

  • Users can create automation rules with conditions and actions
  • Rules can be executed manually or on schedule
  • Real-time progress tracking during execution
  • Execution history with success/failure status
  • All AI functions available as actions
  • WordPress publishing integration

Performance Requirements

  • Rule execution starts within 2 seconds
  • Progress updates every 2 seconds
  • UI remains responsive during execution
  • Support for 100+ concurrent rule executions

Quality Requirements

  • 80%+ test coverage
  • Zero critical bugs
  • Comprehensive documentation
  • User acceptance testing passed

Post-Implementation

Monitoring

  • Track rule execution success rate
  • Monitor execution duration
  • Track credit consumption
  • Monitor error rates

Optimization

  • Optimize slow-running actions
  • Improve credit estimation accuracy
  • Enhance error messages
  • Add more action types based on usage

Future Enhancements

  • Conditional branching (if/else logic)
  • Parallel action execution
  • Action templates and presets
  • Rule sharing and marketplace
  • Advanced scheduling (multiple schedules per rule)
  • Webhook triggers for external events

Appendix

Action Type Reference

Action Type Description Parameters Credits
cluster_keywords Cluster keywords by topic keyword_ids, sector_id 10
generate_ideas Generate content ideas cluster_ids 15
generate_content Generate article content task_ids 50
generate_images Generate images content_id, provider 1-4
generate_site_structure Generate site structure site_id 20
generate_page_content Generate page content page_ids 30
optimize_content Optimize for SEO content_ids 10
publish_to_wordpress Publish to WordPress content_ids, integration_id 5

Condition Operators

Operator Description Example
equals Exact match status equals "draft"
not_equals Not equal status not_equals "published"
contains String contains title contains "guide"
greater_than Numeric > word_count greater_than 1000
less_than Numeric < word_count less_than 5000
in Value in list status in ["draft", "pending"]
not_in Value not in list status not_in ["archived"]

API Endpoint Summary

Method Endpoint Description
GET /api/v1/automation/rules/ List all rules
POST /api/v1/automation/rules/ Create new rule
GET /api/v1/automation/rules/{id}/ Get rule details
PUT /api/v1/automation/rules/{id}/ Update rule
DELETE /api/v1/automation/rules/{id}/ Delete rule
POST /api/v1/automation/rules/{id}/execute/ Execute rule
GET /api/v1/automation/rules/{id}/progress/ Get execution progress
GET /api/v1/automation/rules/action-types/ Get available actions
GET /api/v1/automation/rules/execution-history/ Get execution history
GET /api/v1/automation/scheduled-tasks/ List scheduled tasks

Document Version: 1.0
Created: 2024-11-30
Author: IGNY8 Development Team
Status: Ready for Implementation