Automation Part 1
This commit is contained in:
IGNY8 VPS (Salman)
299
AUTOMATION-DEPLOYMENT-CHECKLIST.md
Normal file
299
AUTOMATION-DEPLOYMENT-CHECKLIST.md
Normal file
@@ -0,0 +1,299 @@
|
||||
# Automation Implementation - Deployment Checklist
|
||||
|
||||
## ✅ Completed Components
|
||||
|
||||
### Backend
|
||||
- [x] Database models created (`AutomationConfig`, `AutomationRun`)
|
||||
- [x] AutomationLogger service (file-based logging)
|
||||
- [x] AutomationService orchestrator (7-stage pipeline)
|
||||
- [x] API endpoints (`AutomationViewSet`)
|
||||
- [x] Celery tasks (scheduled checks, run execution, resume)
|
||||
- [x] URL routing registered
|
||||
- [x] Celery beat schedule configured
|
||||
- [x] Migration file created
|
||||
|
||||
### Frontend
|
||||
- [x] TypeScript API service (`automationService.ts`)
|
||||
- [x] Main dashboard page (`AutomationPage.tsx`)
|
||||
- [x] StageCard component
|
||||
- [x] ActivityLog component
|
||||
- [x] ConfigModal component
|
||||
- [x] RunHistory component
|
||||
|
||||
### Documentation
|
||||
- [x] Comprehensive README (`AUTOMATION-IMPLEMENTATION-README.md`)
|
||||
- [x] Original plan corrected (`automation-plan.md`)
|
||||
|
||||
## ⏳ Remaining Tasks
|
||||
|
||||
### 1. Run Database Migration
|
||||
|
||||
```bash
|
||||
cd /data/app/igny8/backend
|
||||
python manage.py migrate
|
||||
```
|
||||
|
||||
This will create the `automation_config` and `automation_run` tables.
|
||||
|
||||
### 2. Register Frontend Route
|
||||
|
||||
Add to your React Router configuration:
|
||||
|
||||
```typescript
|
||||
import AutomationPage from './pages/Automation/AutomationPage';
|
||||
|
||||
// In your route definitions:
|
||||
{
|
||||
path: '/automation',
|
||||
element: <AutomationPage />,
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Add Navigation Link
|
||||
|
||||
Add link to main navigation menu:
|
||||
|
||||
```typescript
|
||||
{
|
||||
name: 'Automation',
|
||||
href: '/automation',
|
||||
icon: /* automation icon */
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Verify Infrastructure
|
||||
|
||||
**Celery Worker**
|
||||
```bash
|
||||
# Check if running
|
||||
docker ps | grep celery
|
||||
|
||||
# Start if needed
|
||||
docker-compose up -d celery
|
||||
```
|
||||
|
||||
**Celery Beat**
|
||||
```bash
|
||||
# Check if running
|
||||
docker ps | grep beat
|
||||
|
||||
# Start if needed
|
||||
docker-compose up -d celery-beat
|
||||
```
|
||||
|
||||
**Redis/Cache**
|
||||
```bash
|
||||
# Verify cache backend in settings.py
|
||||
CACHES = {
|
||||
'default': {
|
||||
'BACKEND': 'django.core.cache.backends.redis.RedisCache',
|
||||
'LOCATION': 'redis://redis:6379/1',
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Create Log Directory
|
||||
|
||||
```bash
|
||||
mkdir -p /data/app/igny8/backend/logs/automation
|
||||
chmod 755 /data/app/igny8/backend/logs/automation
|
||||
```
|
||||
|
||||
### 6. Test API Endpoints
|
||||
|
||||
```bash
|
||||
# Get config (should return default config)
|
||||
curl -X GET "http://localhost:8000/api/v1/automation/config/?site_id=1" \
|
||||
-H "Authorization: Bearer YOUR_TOKEN"
|
||||
|
||||
# Estimate credits
|
||||
curl -X GET "http://localhost:8000/api/v1/automation/estimate/?site_id=1" \
|
||||
-H "Authorization: Bearer YOUR_TOKEN"
|
||||
```
|
||||
|
||||
### 7. Test Frontend
|
||||
|
||||
1. Navigate to `/automation` page
|
||||
2. Click [Configure] - modal should open
|
||||
3. Save configuration
|
||||
4. Click [Run Now] - should trigger automation
|
||||
5. Verify real-time updates in stage cards
|
||||
6. Check activity log is streaming
|
||||
|
||||
### 8. Test Scheduled Automation
|
||||
|
||||
1. Enable automation in config
|
||||
2. Set scheduled time to 1 minute from now
|
||||
3. Wait for next hour (beat checks hourly at :00)
|
||||
4. Verify automation starts automatically
|
||||
|
||||
### 9. Monitor First Run
|
||||
|
||||
Watch logs in real-time:
|
||||
|
||||
```bash
|
||||
# Backend logs
|
||||
tail -f /data/app/igny8/backend/logs/automation/{account_id}/{site_id}/{run_id}/automation_run.log
|
||||
|
||||
# Celery worker logs
|
||||
docker logs -f <celery_container>
|
||||
|
||||
# Django logs
|
||||
docker logs -f <backend_container>
|
||||
```
|
||||
|
||||
### 10. Verify Database Records
|
||||
|
||||
```python
|
||||
from igny8_core.business.automation.models import AutomationConfig, AutomationRun
|
||||
|
||||
# Check config created
|
||||
AutomationConfig.objects.all()
|
||||
|
||||
# Check runs recorded
|
||||
AutomationRun.objects.all()
|
||||
|
||||
# View stage results
|
||||
run = AutomationRun.objects.latest('started_at')
|
||||
print(run.stage_1_result)
|
||||
print(run.stage_2_result)
|
||||
# ... etc
|
||||
```
|
||||
|
||||
## Quick Start Commands
|
||||
|
||||
```bash
|
||||
# 1. Run migration
|
||||
cd /data/app/igny8/backend
|
||||
python manage.py migrate
|
||||
|
||||
# 2. Create log directory
|
||||
mkdir -p logs/automation
|
||||
chmod 755 logs/automation
|
||||
|
||||
# 3. Restart services
|
||||
docker-compose restart celery celery-beat
|
||||
|
||||
# 4. Verify Celery beat schedule
|
||||
docker exec <celery_container> celery -A igny8_core inspect scheduled
|
||||
|
||||
# 5. Test automation (Django shell)
|
||||
python manage.py shell
|
||||
>>> from igny8_core.business.automation.services import AutomationService
|
||||
>>> from igny8_core.modules.system.models import Account, Site
|
||||
>>> account = Account.objects.first()
|
||||
>>> site = Site.objects.first()
|
||||
>>> service = AutomationService(account, site)
|
||||
>>> service.estimate_credits() # Should return number
|
||||
>>> # Don't run start_automation() yet - test via UI first
|
||||
```
|
||||
|
||||
## Expected Behavior
|
||||
|
||||
### First Successful Run
|
||||
|
||||
1. **Stage 1**: Process keywords → create clusters (2-5 min)
|
||||
2. **Stage 2**: Generate ideas from clusters (1-2 min per cluster)
|
||||
3. **Stage 3**: Create tasks from ideas (instant)
|
||||
4. **Stage 4**: Generate content from tasks (3-5 min per task)
|
||||
5. **Stage 5**: Extract image prompts (1-2 min per content)
|
||||
6. **Stage 6**: Generate images (2-3 min per image)
|
||||
7. **Stage 7**: Count content ready for review (instant)
|
||||
|
||||
Total time: 15-45 minutes depending on batch sizes
|
||||
|
||||
### Stage Results Example
|
||||
|
||||
```json
|
||||
{
|
||||
"stage_1_result": {
|
||||
"keywords_processed": 20,
|
||||
"clusters_created": 4,
|
||||
"batches_run": 1,
|
||||
"credits_used": 4
|
||||
},
|
||||
"stage_2_result": {
|
||||
"clusters_processed": 4,
|
||||
"ideas_created": 16,
|
||||
"credits_used": 8
|
||||
},
|
||||
"stage_3_result": {
|
||||
"ideas_processed": 16,
|
||||
"tasks_created": 16,
|
||||
"batches_run": 1
|
||||
},
|
||||
"stage_4_result": {
|
||||
"tasks_processed": 16,
|
||||
"content_created": 16,
|
||||
"total_words": 40000,
|
||||
"credits_used": 80
|
||||
},
|
||||
"stage_5_result": {
|
||||
"content_processed": 16,
|
||||
"prompts_created": 64,
|
||||
"credits_used": 32
|
||||
},
|
||||
"stage_6_result": {
|
||||
"images_processed": 64,
|
||||
"images_generated": 64,
|
||||
"content_moved_to_review": 16,
|
||||
"credits_used": 128
|
||||
},
|
||||
"stage_7_result": {
|
||||
"ready_for_review": 16,
|
||||
"content_ids": [1, 2, 3, ...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Module not found" errors
|
||||
- Restart Django server after adding new models
|
||||
- Run `python manage.py collectstatic` if needed
|
||||
|
||||
### "Table does not exist" errors
|
||||
- Run migration: `python manage.py migrate`
|
||||
|
||||
### "No module named automation"
|
||||
- Check `__init__.py` files exist in all directories
|
||||
- Verify imports in `urls.py`
|
||||
|
||||
### Celery tasks not running
|
||||
- Check worker is running: `docker ps | grep celery`
|
||||
- Check beat is running: `docker ps | grep beat`
|
||||
- Verify tasks registered: `celery -A igny8_core inspect registered`
|
||||
|
||||
### Logs not appearing
|
||||
- Check directory permissions: `ls -la logs/automation`
|
||||
- Check AutomationLogger.start_run() creates directories
|
||||
- Verify log file path in code matches actual filesystem
|
||||
|
||||
### Frontend errors
|
||||
- Check API service imported correctly
|
||||
- Verify route registered in router
|
||||
- Check for TypeScript compilation errors
|
||||
- Verify API endpoints returning expected data
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Migration runs without errors
|
||||
- [ ] Frontend `/automation` page loads
|
||||
- [ ] Config modal opens and saves
|
||||
- [ ] Credit estimate shows reasonable number
|
||||
- [ ] "Run Now" starts automation successfully
|
||||
- [ ] Stage cards update in real-time
|
||||
- [ ] Activity log shows progress
|
||||
- [ ] All 7 stages complete successfully
|
||||
- [ ] Content moved to review status
|
||||
- [ ] Run History table shows completed run
|
||||
- [ ] Scheduled automation triggers at configured time
|
||||
|
||||
## Post-Deployment
|
||||
|
||||
1. Monitor first few runs closely
|
||||
2. Adjust batch sizes based on performance
|
||||
3. Set up alerts for failed runs
|
||||
4. Document any issues encountered
|
||||
5. Train users on automation features
|
||||
6. Gather feedback for improvements
|
||||
Reference in New Issue
Block a user