# IGNY8 Automation System Implementation Plan ## Overview This document provides a detailed, step-by-step implementation plan for completing the IGNY8 Automation system. The backend infrastructure is already in place; this plan focuses on extending functionality and building the frontend interface. ## Current State Analysis ### ✅ Already Implemented (Backend) 1. **Data Models** ([`backend/igny8_core/business/automation/models.py`](backend/igny8_core/business/automation/models.py:1)) - [`AutomationRule`](backend/igny8_core/business/automation/models.py:11): Complete with trigger types, conditions, actions - [`ScheduledTask`](backend/igny8_core/business/automation/models.py:93): Task scheduling and tracking 2. **Core Services** ([`backend/igny8_core/business/automation/services/`](backend/igny8_core/business/automation/services/)) - [`AutomationService`](backend/igny8_core/business/automation/services/automation_service.py:15): Main orchestrator - [`RuleEngine`](backend/igny8_core/business/automation/services/rule_engine.py:15): Condition evaluation - [`ConditionEvaluator`](backend/igny8_core/business/automation/services/condition_evaluator.py:15): Field path resolution - [`ActionExecutor`](backend/igny8_core/business/automation/services/action_executor.py:13): Action execution (partial) 3. **API Endpoints** ([`backend/igny8_core/modules/automation/views.py`](backend/igny8_core/modules/automation/views.py:1)) - [`AutomationRuleViewSet`](backend/igny8_core/modules/automation/views.py:29): CRUD + execute endpoint - [`ScheduledTaskViewSet`](backend/igny8_core/modules/automation/views.py:76): Task management ### ❌ Missing Components 1. **Backend Extensions:** - ActionExecutor only supports 3 actions (cluster_keywords, generate_ideas, generate_content) - Missing: generate_images, generate_site_structure, generate_page_content, optimize_content, publish_to_wordpress - No Celery task integration for async execution - No progress tracking integration - No webhook support for event triggers 2. **Frontend (Complete):** - No automation module pages - No rule management interface - No workflow visualizer - No execution monitor - No progress tracking UI 3. **Integration:** - No connection to existing AI functions - No WordPress publish integration - No credit cost estimation ## Implementation Phases ### Phase 1: Backend Extensions (Week 1) #### 1.1 Extend ActionExecutor **File:** [`backend/igny8_core/business/automation/services/action_executor.py`](backend/igny8_core/business/automation/services/action_executor.py:1) **Changes Required:** ```python # Add new action methods to ActionExecutor class def _execute_generate_images(self, params, rule): """Execute generate images action""" content_id = params.get('content_id') provider = params.get('provider', 'dall-e') try: from igny8_core.business.content.services.image_generation_service import ImageGenerationService service = ImageGenerationService() result = service.generate_images( content_id=content_id, provider=provider, account=rule.account ) return result except Exception as e: logger.error(f"Error generating images: {str(e)}", exc_info=True) return {'success': False, 'error': str(e)} def _execute_generate_site_structure(self, params, rule): """Execute generate site structure action""" site_id = params.get('site_id') or (rule.site.id if rule.site else None) try: from igny8_core.business.sites.services.site_generation_service import SiteGenerationService service = SiteGenerationService() result = service.generate_structure( site_id=site_id, account=rule.account ) return result except Exception as e: logger.error(f"Error generating site structure: {str(e)}", exc_info=True) return {'success': False, 'error': str(e)} def _execute_generate_page_content(self, params, rule): """Execute generate page content action""" page_ids = params.get('page_ids', []) try: from igny8_core.business.sites.services.page_generation_service import PageGenerationService service = PageGenerationService() result = service.generate_content( page_ids=page_ids, account=rule.account ) return result except Exception as e: logger.error(f"Error generating page content: {str(e)}", exc_info=True) return {'success': False, 'error': str(e)} def _execute_optimize_content(self, params, rule): """Execute optimize content action""" content_ids = params.get('content_ids', []) try: from igny8_core.business.optimizer.services.optimization_service import OptimizationService service = OptimizationService() result = service.optimize_content( content_ids=content_ids, account=rule.account ) return result except Exception as e: logger.error(f"Error optimizing content: {str(e)}", exc_info=True) return {'success': False, 'error': str(e)} def _execute_publish_to_wordpress(self, params, rule): """Execute publish to WordPress action""" content_ids = params.get('content_ids', []) integration_id = params.get('integration_id') try: from igny8_core.business.integration.services.wordpress_service import WordPressService service = WordPressService() result = service.publish_content( content_ids=content_ids, integration_id=integration_id, account=rule.account ) return result except Exception as e: logger.error(f"Error publishing to WordPress: {str(e)}", exc_info=True) return {'success': False, 'error': str(e)} # Update execute() method to handle new action types def execute(self, action, context, rule): """Execute a single action""" action_type = action.get('type') params = action.get('params', {}) action_map = { 'cluster_keywords': self._execute_cluster_keywords, 'generate_ideas': self._execute_generate_ideas, 'generate_content': self._execute_generate_content, 'generate_images': self._execute_generate_images, 'generate_site_structure': self._execute_generate_site_structure, 'generate_page_content': self._execute_generate_page_content, 'optimize_content': self._execute_optimize_content, 'publish_to_wordpress': self._execute_publish_to_wordpress, } executor = action_map.get(action_type) if executor: return executor(params, rule) else: logger.warning(f"Unknown action type: {action_type}") return {'success': False, 'error': f'Unknown action type: {action_type}'} ``` **Estimated Time:** 4 hours #### 1.2 Create Celery Tasks **File:** [`backend/igny8_core/business/automation/tasks.py`](backend/igny8_core/business/automation/tasks.py:1) **Changes Required:** ```python """ Celery tasks for automation """ from celery import shared_task from django.utils import timezone from igny8_core.business.automation.models import AutomationRule, ScheduledTask from igny8_core.business.automation.services.automation_service import AutomationService import logging logger = logging.getLogger(__name__) @shared_task(bind=True, name='automation.execute_rule') def execute_rule_task(self, rule_id, context=None): """ Execute an automation rule asynchronously Args: rule_id: AutomationRule ID context: Optional context dict """ try: rule = AutomationRule.objects.get(id=rule_id) service = AutomationService() # Update task progress self.update_state( state='PROGRESS', meta={'phase': 'Initializing', 'progress': 0} ) result = service.execute_rule(rule, context or {}) self.update_state( state='SUCCESS', meta={'phase': 'Completed', 'progress': 100, 'result': result} ) return result except Exception as e: logger.error(f"Error executing rule {rule_id}: {str(e)}", exc_info=True) self.update_state( state='FAILURE', meta={'error': str(e)} ) raise @shared_task(name='automation.execute_scheduled_rules') def execute_scheduled_rules_task(): """ Execute all scheduled rules that are due Called by Celery Beat """ service = AutomationService() result = service.execute_scheduled_rules() logger.info(f"Scheduled rules execution: {result}") return result @shared_task(name='automation.cleanup_old_tasks') def cleanup_old_tasks(): """ Clean up old scheduled tasks Keep last 30 days only """ from datetime import timedelta cutoff_date = timezone.now() - timedelta(days=30) deleted_count = ScheduledTask.objects.filter( created_at__lt=cutoff_date, status__in=['completed', 'failed', 'cancelled'] ).delete()[0] logger.info(f"Cleaned up {deleted_count} old scheduled tasks") return {'deleted': deleted_count} ``` **Estimated Time:** 2 hours #### 1.3 Add Progress Tracking **File:** [`backend/igny8_core/business/automation/services/automation_service.py`](backend/igny8_core/business/automation/services/automation_service.py:1) **Changes Required:** ```python # Add to execute_rule method def execute_rule(self, rule, context=None): """Execute an automation rule with progress tracking""" # ... existing code ... # Execute via rule engine with progress tracking try: from celery import current_task # Update progress for each action total_actions = len(rule.actions) for idx, action in enumerate(rule.actions): if current_task: current_task.update_state( state='PROGRESS', meta={ 'phase': f'Executing action {idx + 1}/{total_actions}', 'progress': int((idx / total_actions) * 100), 'action': action.get('type') } ) # Execute action action_result = self.rule_engine.execute_action(action, context, rule) # Store result results.append(action_result) # ... rest of existing code ... ``` **Estimated Time:** 2 hours #### 1.4 Add API Endpoints **File:** [`backend/igny8_core/modules/automation/views.py`](backend/igny8_core/modules/automation/views.py:1) **Changes Required:** ```python # Add new endpoints to AutomationRuleViewSet @action(detail=True, methods=['get'], url_path='progress', url_name='progress') def get_progress(self, request, pk=None): """Get execution progress for a rule""" rule = self.get_object() # Get latest task for this rule from celery.result import AsyncResult from igny8_core.business.automation.models import ScheduledTask latest_task = ScheduledTask.objects.filter( automation_rule=rule, status='running' ).order_by('-created_at').first() if not latest_task or not latest_task.metadata.get('task_id'): return success_response( data={'status': 'idle'}, request=request ) task_id = latest_task.metadata['task_id'] task_result = AsyncResult(task_id) return success_response( data={ 'status': task_result.state, 'progress': task_result.info.get('progress', 0) if task_result.info else 0, 'phase': task_result.info.get('phase', 'Unknown') if task_result.info else 'Unknown', 'result': task_result.result if task_result.successful() else None }, request=request ) @action(detail=False, methods=['get'], url_path='action-types', url_name='action-types') def get_action_types(self, request): """Get available action types""" action_types = [ { 'type': 'cluster_keywords', 'name': 'Cluster Keywords', 'description': 'Automatically cluster keywords by topic', 'params': ['keyword_ids', 'sector_id'] }, { 'type': 'generate_ideas', 'name': 'Generate Ideas', 'description': 'Generate content ideas from clusters', 'params': ['cluster_ids'] }, { 'type': 'generate_content', 'name': 'Generate Content', 'description': 'Generate article content from tasks', 'params': ['task_ids'] }, { 'type': 'generate_images', 'name': 'Generate Images', 'description': 'Generate images for content', 'params': ['content_id', 'provider'] }, { 'type': 'generate_site_structure', 'name': 'Generate Site Structure', 'description': 'Generate website structure', 'params': ['site_id'] }, { 'type': 'generate_page_content', 'name': 'Generate Page Content', 'description': 'Generate content for pages', 'params': ['page_ids'] }, { 'type': 'optimize_content', 'name': 'Optimize Content', 'description': 'Optimize content for SEO', 'params': ['content_ids'] }, { 'type': 'publish_to_wordpress', 'name': 'Publish to WordPress', 'description': 'Publish content to WordPress', 'params': ['content_ids', 'integration_id'] } ] return success_response( data=action_types, request=request ) @action(detail=False, methods=['get'], url_path='execution-history', url_name='execution-history') def get_execution_history(self, request): """Get execution history for all rules""" from igny8_core.business.automation.models import ScheduledTask tasks = ScheduledTask.objects.filter( automation_rule__account=request.account ).select_related('automation_rule').order_by('-executed_at')[:50] serializer = ScheduledTaskSerializer(tasks, many=True) return success_response( data=serializer.data, request=request ) ``` **Estimated Time:** 3 hours ### Phase 2: Frontend Foundation (Week 2) #### 2.1 Create Automation Store **File:** `frontend/src/stores/automationStore.ts` (NEW) **Implementation:** ```typescript import { create } from 'zustand'; import { automationApi } from '../api/automation.api'; interface AutomationRule { id: string; name: string; description: string; trigger: 'schedule' | 'event' | 'manual'; schedule?: string; conditions: any[]; actions: any[]; is_active: boolean; status: 'active' | 'inactive' | 'paused'; last_executed_at?: string; execution_count: number; } interface ScheduledTask { id: string; automation_rule: string; scheduled_at: string; executed_at?: string; status: 'pending' | 'running' | 'completed' | 'failed' | 'cancelled'; result: any; error_message?: string; } interface AutomationStore { // State rules: AutomationRule[]; tasks: ScheduledTask[]; actionTypes: any[]; loading: boolean; error: string | null; executionProgress: { ruleId: string | null; status: string; progress: number; phase: string; } | null; // Actions fetchRules: () => Promise; fetchRule: (id: string) => Promise; createRule: (rule: Partial) => Promise; updateRule: (id: string, rule: Partial) => Promise; deleteRule: (id: string) => Promise; executeRule: (id: string, context?: any) => Promise; fetchActionTypes: () => Promise; fetchExecutionHistory: () => Promise; fetchProgress: (ruleId: string) => Promise; clearError: () => void; } export const useAutomationStore = create((set, get) => ({ // Initial state rules: [], tasks: [], actionTypes: [], loading: false, error: null, executionProgress: null, // Actions fetchRules: async () => { set({ loading: true, error: null }); try { const response = await automationApi.getRules(); set({ rules: response.data, loading: false }); } catch (error: any) { set({ error: error.message, loading: false }); } }, fetchRule: async (id: string) => { set({ loading: true, error: null }); try { const response = await automationApi.getRule(id); set({ loading: false }); return response.data; } catch (error: any) { set({ error: error.message, loading: false }); throw error; } }, createRule: async (rule: Partial) => { set({ loading: true, error: null }); try { const response = await automationApi.createRule(rule); set(state => ({ rules: [...state.rules, response.data], loading: false })); return response.data; } catch (error: any) { set({ error: error.message, loading: false }); throw error; } }, updateRule: async (id: string, rule: Partial) => { set({ loading: true, error: null }); try { const response = await automationApi.updateRule(id, rule); set(state => ({ rules: state.rules.map(r => r.id === id ? response.data : r), loading: false })); return response.data; } catch (error: any) { set({ error: error.message, loading: false }); throw error; } }, deleteRule: async (id: string) => { set({ loading: true, error: null }); try { await automationApi.deleteRule(id); set(state => ({ rules: state.rules.filter(r => r.id !== id), loading: false })); } catch (error: any) { set({ error: error.message, loading: false }); throw error; } }, executeRule: async (id: string, context?: any) => { set({ loading: true, error: null }); try { await automationApi.executeRule(id, context); set({ loading: false }); // Start polling for progress const pollProgress = setInterval(async () => { await get().fetchProgress(id); const progress = get().executionProgress; if (progress && (progress.status === 'SUCCESS' || progress.status === 'FAILURE')) { clearInterval(pollProgress); } }, 2000); } catch (error: any) { set({ error: error.message, loading: false }); throw error; } }, fetchActionTypes: async () => { try { const response = await automationApi.getActionTypes(); set({ actionTypes: response.data }); } catch (error: any) { set({ error: error.message }); } }, fetchExecutionHistory: async () => { set({ loading: true, error: null }); try { const response = await automationApi.getExecutionHistory(); set({ tasks: response.data, loading: false }); } catch (error: any) { set({ error: error.message, loading: false }); } }, fetchProgress: async (ruleId: string) => { try { const response = await automationApi.getProgress(ruleId); set({ executionProgress: { ruleId, ...response.data } }); } catch (error: any) { console.error('Error fetching progress:', error); } }, clearError: () => set({ error: null }) })); ``` **Estimated Time:** 3 hours #### 2.2 Create Automation API Client **File:** `frontend/src/api/automation.api.ts` (NEW) **Implementation:** ```typescript import apiClient from './client'; export const automationApi = { // Rules getRules: () => apiClient.get('/automation/rules/'), getRule: (id: string) => apiClient.get(`/automation/rules/${id}/`), createRule: (data: any) => apiClient.post('/automation/rules/', data), updateRule: (id: string, data: any) => apiClient.patch(`/automation/rules/${id}/`, data), deleteRule: (id: string) => apiClient.delete(`/automation/rules/${id}/`), executeRule: (id: string, context?: any) => apiClient.post(`/automation/rules/${id}/execute/`, { context }), // Progress getProgress: (ruleId: string) => apiClient.get(`/automation/rules/${ruleId}/progress/`), // Action types getActionTypes: () => apiClient.get('/automation/rules/action-types/'), // Execution history getExecutionHistory: () => apiClient.get('/automation/rules/execution-history/'), // Scheduled tasks getTasks: () => apiClient.get('/automation/scheduled-tasks/'), getTask: (id: string) => apiClient.get(`/automation/scheduled-tasks/${id}/`) }; ``` **Estimated Time:** 1 hour ### Phase 3: Frontend UI Components (Week 3) #### 3.1 Rule List Page **File:** `frontend/src/pages/automation/AutomationRulesPage.tsx` (NEW) **Features:** - Table view of all automation rules - Status indicators (active/inactive/paused) - Last execution time and count - Quick actions (execute, edit, delete) - Create new rule button **Estimated Time:** 6 hours #### 3.2 Rule Editor Modal **File:** `frontend/src/components/automation/RuleEditorModal.tsx` (NEW) **Features:** - Form for rule name, description, trigger type - Schedule configuration (cron expression builder) - Condition builder (field, operator, value) - Action builder (type selector, parameter inputs) - Save/cancel buttons **Estimated Time:** 8 hours #### 3.3 Workflow Visualizer **File:** `frontend/src/components/automation/WorkflowVisualizer.tsx` (NEW) **Features:** - Visual flowchart of rule actions - Drag-and-drop action reordering - Action configuration panel - Connection lines between actions - Status indicators during execution **Estimated Time:** 12 hours #### 3.4 Execution Monitor **File:** `frontend/src/components/automation/ExecutionMonitor.tsx` (NEW) **Features:** - Real-time progress display - Current phase indicator - Progress percentage - Action-by-action status - Error display - Cancel execution button **Estimated Time:** 6 hours #### 3.5 Execution History **File:** `frontend/src/components/automation/ExecutionHistory.tsx` (NEW) **Features:** - Table of past executions - Status badges (completed/failed/cancelled) - Execution time and duration - Result summary - Error details (if failed) - Re-run button **Estimated Time:** 4 hours ### Phase 4: Integration & Testing (Week 4) #### 4.1 Backend Integration Tests **File:** `backend/igny8_core/business/automation/tests/test_automation.py` (NEW) **Test Cases:** - Rule creation and validation - Condition evaluation - Action execution - Progress tracking - Error handling - Credit checking **Estimated Time:** 8 hours #### 4.2 Frontend Component Tests **File:** `frontend/src/components/automation/__tests__/` (NEW) **Test Cases:** - Rule list rendering - Rule editor form validation - Workflow visualizer interactions - Execution monitor updates - API integration **Estimated Time:** 6 hours #### 4.3 End-to-End Testing **Manual Testing:** - Create automation rule - Execute rule manually - Monitor progress - Verify results - Test scheduled execution - Test error scenarios **Estimated Time:** 4 hours #### 4.4 Documentation **Files to Update:** - User guide for automation module - API documentation - Developer guide for adding new actions **Estimated Time:** 4 hours ## Implementation Timeline ### Week 1: Backend Extensions - **Day 1-2:** Extend ActionExecutor with all action types - **Day 3:** Create Celery tasks - **Day 4:** Add progress tracking - **Day 5:** Add API endpoints and testing ### Week 2: Frontend Foundation - **Day 1:** Create automation store - **Day 2:** Create API client - **Day 3:** Setup routing and page structure - **Day 4-5:** Build rule list page ### Week 3: Frontend UI - **Day 1-2:** Build rule editor modal - **Day 3-4:** Build workflow visualizer - **Day 5:** Build execution monitor and history ### Week 4: Integration & Polish - **Day 1-2:** Backend integration tests - **Day 3:** Frontend component tests - **Day 4:** End-to-end testing - **Day 5:** Documentation and polish ## Resource Requirements ### Development Team - 1 Backend Developer (Full-time, 4 weeks) - 1 Frontend Developer (Full-time, 4 weeks) - 1 QA Engineer (Part-time, Week 4) ### Infrastructure - Development environment - Staging environment for testing - Celery worker and beat scheduler - Redis for task queue ## Risk Assessment ### High Risk 1. **Workflow Visualizer Complexity** - Mitigation: Use existing React Flow library - Fallback: Simpler list-based interface 2. **Real-time Progress Tracking** - Mitigation: Implement polling with WebSocket fallback - Fallback: Manual refresh button ### Medium Risk 1. **Action Execution Failures** - Mitigation: Comprehensive error handling and retry logic - Fallback: Manual intervention and re-execution 2. **Credit Estimation Accuracy** - Mitigation: Conservative estimates with buffer - Fallback: Manual credit approval for large operations ### Low Risk 1. **UI/UX Complexity** - Mitigation: Iterative design with user feedback - Fallback: Simplified interface ## Success Criteria ### Functional Requirements - ✅ Users can create automation rules with conditions and actions - ✅ Rules can be executed manually or on schedule - ✅ Real-time progress tracking during execution - ✅ Execution history with success/failure status - ✅ All AI functions available as actions - ✅ WordPress publishing integration ### Performance Requirements - ✅ Rule execution starts within 2 seconds - ✅ Progress updates every 2 seconds - ✅ UI remains responsive during execution - ✅ Support for 100+ concurrent rule executions ### Quality Requirements - ✅ 80%+ test coverage - ✅ Zero critical bugs - ✅ Comprehensive documentation - ✅ User acceptance testing passed ## Post-Implementation ### Monitoring - Track rule execution success rate - Monitor execution duration - Track credit consumption - Monitor error rates ### Optimization - Optimize slow-running actions - Improve credit estimation accuracy - Enhance error messages - Add more action types based on usage ### Future Enhancements - Conditional branching (if/else logic) - Parallel action execution - Action templates and presets - Rule sharing and marketplace - Advanced scheduling (multiple schedules per rule) - Webhook triggers for external events ## Appendix ### Action Type Reference | Action Type | Description | Parameters | Credits | |------------|-------------|------------|---------| | cluster_keywords | Cluster keywords by topic | keyword_ids, sector_id | 10 | | generate_ideas | Generate content ideas | cluster_ids | 15 | | generate_content | Generate article content | task_ids | 50 | | generate_images | Generate images | content_id, provider | 1-4 | | generate_site_structure | Generate site structure | site_id | 20 | | generate_page_content | Generate page content | page_ids | 30 | | optimize_content | Optimize for SEO | content_ids | 10 | | publish_to_wordpress | Publish to WordPress | content_ids, integration_id | 5 | ### Condition Operators | Operator | Description | Example | |----------|-------------|---------| | equals | Exact match | status equals "draft" | | not_equals | Not equal | status not_equals "published" | | contains | String contains | title contains "guide" | | greater_than | Numeric > | word_count greater_than 1000 | | less_than | Numeric < | word_count less_than 5000 | | in | Value in list | status in ["draft", "pending"] | | not_in | Value not in list | status not_in ["archived"] | ### API Endpoint Summary | Method | Endpoint | Description | |--------|----------|-------------| | GET | /api/v1/automation/rules/ | List all rules | | POST | /api/v1/automation/rules/ | Create new rule | | GET | /api/v1/automation/rules/{id}/ | Get rule details | | PUT | /api/v1/automation/rules/{id}/ | Update rule | | DELETE | /api/v1/automation/rules/{id}/ | Delete rule | | POST | /api/v1/automation/rules/{id}/execute/ | Execute rule | | GET | /api/v1/automation/rules/{id}/progress/ | Get execution progress | | GET | /api/v1/automation/rules/action-types/ | Get available actions | | GET | /api/v1/automation/rules/execution-history/ | Get execution history | | GET | /api/v1/automation/scheduled-tasks/ | List scheduled tasks | --- **Document Version:** 1.0 **Created:** 2024-11-30 **Author:** IGNY8 Development Team **Status:** Ready for Implementation