autmation final reaftocrs and setitgns dafautls
This commit is contained in:
IGNY8 VPS (Salman)
403
docs/40-WORKFLOWS/AUTOMATION-AND-PUBLISHING-SCHEDULING.md
Normal file
403
docs/40-WORKFLOWS/AUTOMATION-AND-PUBLISHING-SCHEDULING.md
Normal file
@@ -0,0 +1,403 @@
|
||||
# Automation & Publishing Scheduling System
|
||||
|
||||
> **Last Updated:** January 18, 2026
|
||||
> **System:** IGNY8 Celery Task Scheduling
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
IGNY8 uses **Celery Beat** to schedule two distinct but related systems:
|
||||
|
||||
1. **Automation Pipeline** - Processes content through 7 stages (Keywords → Published)
|
||||
2. **Publishing Scheduler** - Schedules and publishes approved content to WordPress
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ CELERY BEAT SCHEDULER │
|
||||
│ (Persistent Schedule Store) │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────┼─────────────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Every Hour │ │ Every Hour │ │ Every 5 min │
|
||||
│ at :05 │ │ at :00 │ │ │
|
||||
└───────────────┘ └─────────────────┘ └─────────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ check_ │ │schedule_approved│ │process_scheduled│
|
||||
│ scheduled_ │ │ _content │ │ _publications │
|
||||
│ automations │ │ │ │ │
|
||||
└───────────────┘ └─────────────────┘ └─────────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Automation │ │ Content gets │ │ Publishes to │
|
||||
│ Pipeline │ │ scheduled_ │ │ WordPress via │
|
||||
│ (7 stages) │ │ publish_at set │ │ API │
|
||||
└───────────────┘ └─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Automation Scheduling
|
||||
|
||||
### Celery Task Configuration
|
||||
|
||||
| Task Name | Schedule | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| `automation.check_scheduled_automations` | Every hour at `:05` | Check if any automation configs should run |
|
||||
| `automation.check_test_triggers` | Every minute | Check for admin test triggers (exits early if none) |
|
||||
|
||||
### How It Works
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────┐
|
||||
│ AUTOMATION SCHEDULING FLOW │
|
||||
└──────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Celery Beat AutomationConfig Result
|
||||
│ │ │
|
||||
│ (Every hour at :05) │ │
|
||||
▼ │ │
|
||||
┌─────────────┐ │ │
|
||||
│ 02:05 check │ ─── Hour ────► scheduled_hour == 2? │
|
||||
│ 03:05 check │ ─── Hour ────► scheduled_hour == 3? │
|
||||
│ 04:05 check │ ─── Hour ────► scheduled_hour == 4? │
|
||||
│ ... │ │ │
|
||||
└─────────────┘ │ │
|
||||
▼ │
|
||||
┌────────────────┐ │
|
||||
│ Match Found? │ │
|
||||
└────────────────┘ │
|
||||
│ │ │
|
||||
Yes No │
|
||||
│ │ │
|
||||
▼ └─► Skip │
|
||||
┌────────────────┐ │
|
||||
│ Check Blocks: │ │
|
||||
│ • 23hr block │ │
|
||||
│ • Already │ │
|
||||
│ running? │ │
|
||||
└────────────────┘ │
|
||||
│ │
|
||||
Pass │
|
||||
│ │
|
||||
▼ │
|
||||
┌────────────────┐ │
|
||||
│ Start Run │ ────────────────────────► Run Started
|
||||
│ • Set last_run │
|
||||
│ • Queue task │
|
||||
└────────────────┘
|
||||
```
|
||||
|
||||
### Hourly Matching Logic
|
||||
|
||||
The scheduler runs at `:05` of every hour and checks if `scheduled_hour == current_hour`:
|
||||
|
||||
| Celery Runs At | Checks Hour | Example Config |
|
||||
|----------------|-------------|----------------|
|
||||
| `02:05` | `2` | `scheduled_time = 02:00` → Matches |
|
||||
| `03:05` | `3` | `scheduled_time = 03:00` → Matches |
|
||||
| `14:05` | `14` | `scheduled_time = 14:00` (2 PM) → Matches |
|
||||
|
||||
**Note:** Users select hour only (1-12 with AM/PM in UI), stored as `HH:00` format. The `:05` offset ensures the check happens after the configured hour begins.
|
||||
|
||||
### Test Mode (Admin Feature)
|
||||
|
||||
Admins can trigger automations without waiting for the hourly schedule:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────┐
|
||||
│ TEST TRIGGER FLOW │
|
||||
└──────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Admin Action Celery (Every Minute) Result
|
||||
│ │ │
|
||||
▼ │ │
|
||||
┌─────────────────┐ │ │
|
||||
│ Enable test mode│ │ │
|
||||
│ Set trigger time│ │ │
|
||||
│ (or "now") │ │ │
|
||||
└─────────────────┘ │ │
|
||||
│ │ │
|
||||
└───────────────────────────────────▼ │
|
||||
┌─────────────────┐ │
|
||||
│check_test_ │ │
|
||||
│triggers task │ │
|
||||
└─────────────────┘ │
|
||||
│ │
|
||||
▼ │
|
||||
┌─────────────────┐ │
|
||||
│test_trigger_at │ │
|
||||
│<= now? │ │
|
||||
└─────────────────┘ │
|
||||
│ │ │
|
||||
Yes No │
|
||||
│ └─► Wait │
|
||||
▼ │
|
||||
┌─────────────────┐ │
|
||||
│Start test run │──────────────► Run Started
|
||||
│trigger_type= │ (type='test')
|
||||
│'test' │
|
||||
│Clear trigger_at │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
**Test mode fields:**
|
||||
- `test_mode_enabled` - Enable test functionality for this config
|
||||
- `test_trigger_at` - When to trigger (set to `now` for immediate)
|
||||
|
||||
**Admin actions:**
|
||||
- 🧪 **Trigger test run** - Sets both fields and triggers at next minute
|
||||
- 🧹 **Clear test mode** - Disables test mode and clears trigger
|
||||
|
||||
### Frequency Rules
|
||||
|
||||
| Frequency | When It Runs | Additional Condition |
|
||||
|-----------|--------------|----------------------|
|
||||
| `daily` | Every day | `scheduled_hour == current_hour` |
|
||||
| `weekly` | Mondays only | `weekday() == 0` + hour match |
|
||||
| `monthly` | 1st of month only | `day == 1` + hour match |
|
||||
|
||||
### Duplicate Prevention (23-Hour Block)
|
||||
|
||||
After a successful run, the config won't run again for **23 hours**:
|
||||
|
||||
```python
|
||||
if config.last_run_at:
|
||||
time_since_last_run = now - config.last_run_at
|
||||
if time_since_last_run < timedelta(hours=23):
|
||||
# SKIP - already ran recently
|
||||
```
|
||||
|
||||
### Schedule Change Behavior
|
||||
|
||||
When `scheduled_time`, `frequency`, or `is_enabled` changes:
|
||||
- `last_run_at` is reset to `None`
|
||||
- `next_run_at` is recalculated
|
||||
- Automation becomes eligible immediately at the next hourly check
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Current time: 12:30
|
||||
You change scheduled_time from 02:00 → 12:00 (12 PM)
|
||||
Hour 12 was already checked at 12:05
|
||||
→ Automation will run tomorrow at 12:05 (daily) or next valid day (weekly/monthly)
|
||||
```
|
||||
|
||||
**Pro tip:** Use test mode to trigger immediately after schedule changes.
|
||||
|
||||
---
|
||||
|
||||
## 2. Publishing Scheduling
|
||||
|
||||
### Celery Task Configuration
|
||||
|
||||
| Task Name | Schedule | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| `publishing.schedule_approved_content` | Every hour at `:00` | Schedule approved content for future publishing |
|
||||
| `publishing.process_scheduled_publications` | Every 5 min | Publish content where `scheduled_publish_at <= now` |
|
||||
|
||||
### Publishing Flow
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────┐
|
||||
│ PUBLISHING SCHEDULING FLOW │
|
||||
└──────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
EVERY HOUR (:00) EVERY 5 MINUTES
|
||||
│ │
|
||||
▼ ▼
|
||||
┌────────────────────────┐ ┌────────────────────────┐
|
||||
│schedule_approved_content│ │process_scheduled_ │
|
||||
│ │ │publications │
|
||||
└────────────────────────┘ └────────────────────────┘
|
||||
│ │
|
||||
▼ │
|
||||
┌────────────────────────┐ │
|
||||
│ For each Site with │ │
|
||||
│ auto_publish_enabled: │ │
|
||||
│ │ │
|
||||
│ 1. Find approved │ │
|
||||
│ content │ │
|
||||
│ 2. Calculate slots │ │
|
||||
│ based on settings │ │
|
||||
│ 3. Set scheduled_ │ │
|
||||
│ publish_at │ │
|
||||
│ 4. Set site_status= │ │
|
||||
│ 'scheduled' │ │
|
||||
└────────────────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌───────────────────────────────────────────────┐
|
||||
│ DATABASE │
|
||||
│ ┌─────────────────────────────────────────┐ │
|
||||
│ │ Content Table │ │
|
||||
│ │ • status = 'approved' │ │
|
||||
│ │ • site_status = 'scheduled' │ │
|
||||
│ │ • scheduled_publish_at = <datetime> │ │
|
||||
│ └─────────────────────────────────────────┘ │
|
||||
└───────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌────────────────────────┐
|
||||
│ process_scheduled_ │
|
||||
│ publications │
|
||||
│ │
|
||||
│ WHERE: │
|
||||
│ • site_status = │
|
||||
│ 'scheduled' │
|
||||
│ • scheduled_publish_at │
|
||||
│ <= NOW │
|
||||
└────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌────────────────────────┐
|
||||
│ Publish to WordPress │
|
||||
│ via PublisherService │
|
||||
│ │
|
||||
│ On Success: │
|
||||
│ • site_status = │
|
||||
│ 'published' │
|
||||
│ • Set wp_post_id │
|
||||
└────────────────────────┘
|
||||
```
|
||||
|
||||
### Scheduling Modes
|
||||
|
||||
| Mode | Behavior |
|
||||
|------|----------|
|
||||
| `immediate` | Content scheduled for `now`, picked up within 5 minutes |
|
||||
| `time_slots` | Content scheduled at specific times (e.g., 9am, 2pm, 6pm) |
|
||||
| `stagger` | Content spread evenly across publish hours |
|
||||
|
||||
### Publishing Limits
|
||||
|
||||
| Limit | Description |
|
||||
|-------|-------------|
|
||||
| `daily_publish_limit` | Max posts per day |
|
||||
| `weekly_publish_limit` | Max posts per week |
|
||||
| `monthly_publish_limit` | Max posts per month |
|
||||
| `queue_limit` | Max items to schedule at once (default: 100) |
|
||||
|
||||
---
|
||||
|
||||
## 3. System Relationship
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ COMPLETE CONTENT LIFECYCLE │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
AUTOMATION PIPELINE PUBLISHING PIPELINE
|
||||
(Every 15 min check) (Hourly + Every 5 min)
|
||||
│ │
|
||||
▼ │
|
||||
┌─────────────────┐ │
|
||||
│ Stage 1-6 │ │
|
||||
│ Keywords → │ │
|
||||
│ Content Created │ │
|
||||
└─────────────────┘ │
|
||||
│ │
|
||||
▼ │
|
||||
┌─────────────────┐ │
|
||||
│ Stage 7 │ │
|
||||
│ Auto-Approval │ status='approved' │
|
||||
│ (if enabled) │ ─────────────────────────────────► │
|
||||
└─────────────────┘ │
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│schedule_approved_ │
|
||||
│content (hourly) │
|
||||
│ │
|
||||
│ Sets: │
|
||||
│ • scheduled_publish │
|
||||
│ _at │
|
||||
│ • site_status = │
|
||||
│ 'scheduled' │
|
||||
└─────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│process_scheduled_ │
|
||||
│publications (5 min) │
|
||||
│ │
|
||||
│ Publishes to WP │
|
||||
│ • site_status = │
|
||||
│ 'published' │
|
||||
└─────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Celery Beat Schedule Summary
|
||||
|
||||
| Task | Schedule | Crontab |
|
||||
|------|----------|---------|
|
||||
| `automation.check_scheduled_automations` | Every 15 min | `minute='0,15,30,45'` |
|
||||
| `publishing.schedule_approved_content` | Hourly | `minute=0` |
|
||||
| `publishing.process_scheduled_publications` | Every 5 min | `minute='*/5'` |
|
||||
|
||||
---
|
||||
|
||||
## 5. Key Configuration Tables
|
||||
|
||||
### AutomationConfig
|
||||
|
||||
| Field | Type | Purpose |
|
||||
|-------|------|---------|
|
||||
| `is_enabled` | Boolean | Enable/disable scheduling |
|
||||
| `frequency` | Choice | `daily`, `weekly`, `monthly` |
|
||||
| `scheduled_time` | Time | When to run (e.g., `02:00`) |
|
||||
| `last_run_at` | DateTime | Last successful run (23hr block) |
|
||||
| `next_run_at` | DateTime | Calculated next run time |
|
||||
|
||||
### PublishingSettings
|
||||
|
||||
| Field | Type | Purpose |
|
||||
|-------|------|---------|
|
||||
| `auto_publish_enabled` | Boolean | Enable auto-scheduling |
|
||||
| `scheduling_mode` | Choice | `immediate`, `time_slots`, `stagger` |
|
||||
| `publish_days` | Array | `['mon', 'tue', ...]` |
|
||||
| `publish_time_slots` | Array | `['09:00', '14:00', '18:00']` |
|
||||
| `daily_publish_limit` | Integer | Max posts/day |
|
||||
|
||||
---
|
||||
|
||||
## 6. Troubleshooting
|
||||
|
||||
### Automation Didn't Run
|
||||
|
||||
| Check | Solution |
|
||||
|-------|----------|
|
||||
| `is_enabled = False` | Enable the automation |
|
||||
| Time not in current window | Wait for next window or change time |
|
||||
| `last_run_at` within 23hrs | Wait for 23hr block to expire |
|
||||
| Another run in progress | Wait for current run to complete |
|
||||
| Config updated after window | Schedule moved to next occurrence |
|
||||
|
||||
### Content Not Publishing
|
||||
|
||||
| Check | Solution |
|
||||
|-------|----------|
|
||||
| `auto_publish_enabled = False` | Enable in PublishingSettings |
|
||||
| Content `status != 'approved'` | Approve content first |
|
||||
| No `wp_api_key` on Site | Configure WordPress integration |
|
||||
| Daily/weekly limit reached | Wait for limit reset or increase |
|
||||
|
||||
---
|
||||
|
||||
## 7. Logs
|
||||
|
||||
Monitor these logs for scheduling issues:
|
||||
|
||||
```bash
|
||||
# Automation scheduling
|
||||
docker logs igny8_celery_worker 2>&1 | grep "AutomationTask"
|
||||
|
||||
# Publishing scheduling
|
||||
docker logs igny8_celery_worker 2>&1 | grep "schedule_approved_content\|process_scheduled"
|
||||
```
|
||||
Reference in New Issue
Block a user