flower celery
This commit is contained in:
IGNY8 VPS (Salman)
199
docs/BACKUP-SYSTEM.md
Normal file
199
docs/BACKUP-SYSTEM.md
Normal file
@@ -0,0 +1,199 @@
|
||||
# IGNY8 Application Packaging & Backup System
|
||||
|
||||
## Overview
|
||||
|
||||
This system creates portable, clean packages of the IGNY8 application containing only source code (no dependencies or generated artifacts).
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Packaging Script
|
||||
**Location**: `/data/app/igny8/scripts/package_app.sh`
|
||||
|
||||
**What it does**:
|
||||
- Creates compressed archive of application source code
|
||||
- Excludes: `node_modules/`, `.venv/`, `dist/`, `*.sql`, `*.log`, build artifacts
|
||||
- Includes: Source code, Docker configs, documentation
|
||||
- Generates SHA256 checksum for verification
|
||||
- Auto-cleans packages older than 7 days
|
||||
- Package size: ~44 MB (compressed)
|
||||
|
||||
**Manual Usage**:
|
||||
```bash
|
||||
/data/app/igny8/scripts/package_app.sh
|
||||
```
|
||||
|
||||
### 2. Automated Backup Tasks
|
||||
**Location**: `/data/app/igny8/backend/igny8_core/tasks/backup.py`
|
||||
|
||||
**Celery Tasks**:
|
||||
|
||||
1. **`backup.create_daily_app_package`**
|
||||
- Runs daily at 00:00 UTC
|
||||
- Creates full application package
|
||||
- Logs success/failure
|
||||
- Returns status dict
|
||||
|
||||
2. **`backup.cleanup_old_packages`**
|
||||
- Runs weekly (Monday at 01:00 UTC)
|
||||
- Removes packages older than 30 days
|
||||
- Keeps recent backups safe
|
||||
|
||||
### 3. Celery Beat Schedule
|
||||
**Location**: `/data/app/igny8/backend/igny8_core/celery.py`
|
||||
|
||||
Configured schedules:
|
||||
```python
|
||||
'daily-app-package-backup': {
|
||||
'task': 'backup.create_daily_app_package',
|
||||
'schedule': crontab(hour=0, minute=0), # Daily at midnight
|
||||
},
|
||||
'weekly-package-cleanup': {
|
||||
'task': 'backup.cleanup_old_packages',
|
||||
'schedule': crontab(hour=1, minute=0, day_of_week=1), # Monday 01:00
|
||||
},
|
||||
```
|
||||
|
||||
## Backup Storage
|
||||
|
||||
**Location**: `/data/backups/igny8/`
|
||||
|
||||
**Files created**:
|
||||
```
|
||||
/data/backups/igny8/
|
||||
├── igny8-app-20251210_161527.tar.gz # Compressed package
|
||||
├── igny8-app-20251210_161527.tar.gz.sha256 # Checksum file
|
||||
├── igny8-app-20251211_000000.tar.gz
|
||||
├── igny8-app-20251211_000000.tar.gz.sha256
|
||||
└── ... (7-30 days of backups retained)
|
||||
```
|
||||
|
||||
## Package Contents
|
||||
|
||||
### ✅ INCLUDED (Source code only - ~56 MB uncompressed):
|
||||
- `backend/igny8_core/` - Django application
|
||||
- `backend/manage.py`, `requirements.txt`
|
||||
- `frontend/src/`, `frontend/public/`
|
||||
- `frontend/package.json`, config files
|
||||
- `docs/`, `README.md`
|
||||
- Dockerfiles, docker-compose files
|
||||
|
||||
### ❌ EXCLUDED (Auto-generated - will be rebuilt):
|
||||
- `backend/.venv/` - Python virtual environment
|
||||
- `frontend/node_modules/` - NPM dependencies
|
||||
- `frontend/dist/` - Build output
|
||||
- `*.sql`, `*.sqlite3` - Database files
|
||||
- `logs/` - Runtime logs
|
||||
- `celerybeat-schedule` - Celery state
|
||||
- `staticfiles/` - Collected static files
|
||||
- `__pycache__/`, `*.pyc` - Python cache
|
||||
|
||||
## Restoration Process
|
||||
|
||||
To restore from a package on a new server:
|
||||
|
||||
```bash
|
||||
# 1. Extract package
|
||||
cd /data/app
|
||||
tar -xzf /path/to/igny8-app-YYYYMMDD_HHMMSS.tar.gz
|
||||
|
||||
# 2. Verify checksum (optional but recommended)
|
||||
sha256sum -c igny8-app-YYYYMMDD_HHMMSS.tar.gz.sha256
|
||||
|
||||
# 3. Follow INSTALLATION.md to:
|
||||
# - Build Docker images (installs all dependencies)
|
||||
# - Start services
|
||||
# - Run migrations
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Check backup status**:
|
||||
```bash
|
||||
# List recent backups
|
||||
ls -lh /data/backups/igny8/
|
||||
|
||||
# View Celery Beat schedule
|
||||
docker exec igny8_celery_beat celery -A igny8_core inspect scheduled
|
||||
|
||||
# View Celery logs
|
||||
docker logs igny8_celery_beat
|
||||
docker logs igny8_celery_worker
|
||||
```
|
||||
|
||||
**Verify backup worked**:
|
||||
```bash
|
||||
# Check if backup ran
|
||||
ls -lt /data/backups/igny8/ | head -5
|
||||
|
||||
# Verify package integrity
|
||||
cd /data/backups/igny8/
|
||||
sha256sum -c igny8-app-*.tar.gz.sha256
|
||||
```
|
||||
|
||||
## Retention Policy
|
||||
|
||||
- **Daily backups**: Kept for 7 days in packaging script
|
||||
- **Long-term backups**: Kept for 30 days by cleanup task
|
||||
- **Manual backups**: Never auto-deleted (run script manually with different output dir)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Backup didn't run at midnight**:
|
||||
```bash
|
||||
# Check Celery Beat is running
|
||||
docker ps | grep celery_beat
|
||||
|
||||
# Check Beat logs
|
||||
docker logs igny8_celery_beat
|
||||
|
||||
# Manually trigger backup
|
||||
docker exec igny8_celery_worker celery -A igny8_core call backup.create_daily_app_package
|
||||
```
|
||||
|
||||
**Insufficient disk space**:
|
||||
```bash
|
||||
# Check disk usage
|
||||
df -h /data
|
||||
|
||||
# Clean old backups manually
|
||||
find /data/backups/igny8/ -name "*.tar.gz" -mtime +30 -delete
|
||||
```
|
||||
|
||||
**Package too large**:
|
||||
- Check no large files accidentally included
|
||||
- Verify `.git` folder is excluded
|
||||
- Ensure `node_modules`, `.venv` excluded
|
||||
|
||||
## Security Notes
|
||||
|
||||
1. **Packages contain source code** - store securely
|
||||
2. **No secrets in package** - `.env` is excluded
|
||||
3. **Checksum verification** - always verify SHA256 before restoring
|
||||
4. **Backup the backups** - Consider copying to remote storage
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Action | Command |
|
||||
|--------|---------|
|
||||
| Create backup now | `/data/app/igny8/scripts/package_app.sh` |
|
||||
| List backups | `ls -lh /data/backups/igny8/` |
|
||||
| Verify package | `sha256sum -c package.tar.gz.sha256` |
|
||||
| Extract package | `tar -xzf package.tar.gz` |
|
||||
| Check schedule | `docker exec igny8_celery_beat celery -A igny8_core inspect scheduled` |
|
||||
| View logs | `docker logs igny8_celery_beat` |
|
||||
|
||||
## Configuration
|
||||
|
||||
To change backup schedule, edit `/data/app/igny8/backend/igny8_core/celery.py`:
|
||||
|
||||
```python
|
||||
'daily-app-package-backup': {
|
||||
'task': 'backup.create_daily_app_package',
|
||||
'schedule': crontab(hour=0, minute=0), # Change time here
|
||||
},
|
||||
```
|
||||
|
||||
Then restart Celery:
|
||||
```bash
|
||||
docker restart igny8_celery_beat igny8_celery_worker
|
||||
```
|
||||
159
docs/PACKAGING-SETUP-COMPLETE.md
Normal file
159
docs/PACKAGING-SETUP-COMPLETE.md
Normal file
@@ -0,0 +1,159 @@
|
||||
# IGNY8 Application Packaging & Backup - Setup Complete ✅
|
||||
|
||||
## What Was Created
|
||||
|
||||
### 1. Packaging Script
|
||||
**File**: `/data/app/igny8/scripts/package_app.sh`
|
||||
- ✅ Creates portable application packages
|
||||
- ✅ Excludes all generated artifacts (node_modules, .venv, dist, logs, etc.)
|
||||
- ✅ Generates SHA256 checksums for verification
|
||||
- ✅ Auto-cleans old backups (7+ days)
|
||||
- ✅ Package size: ~44 MB (compressed from ~56 MB source)
|
||||
|
||||
### 2. Automated Backup Tasks
|
||||
**File**: `/data/app/igny8/backend/igny8_core/tasks/backup.py`
|
||||
- ✅ `backup.create_daily_app_package` - Daily backup task
|
||||
- ✅ `backup.cleanup_old_packages` - Weekly cleanup task
|
||||
- ✅ Full error handling and logging
|
||||
|
||||
### 3. Celery Beat Schedule
|
||||
**File**: `/data/app/igny8/backend/igny8_core/celery.py`
|
||||
- ✅ Daily backup at 00:00 UTC
|
||||
- ✅ Weekly cleanup on Mondays at 01:00 UTC
|
||||
|
||||
### 4. Documentation
|
||||
- ✅ `/data/app/igny8/INSTALLATION.md` - Complete installation guide
|
||||
- ✅ `/data/app/igny8/docs/BACKUP-SYSTEM.md` - Backup system documentation
|
||||
|
||||
## First Backup Created
|
||||
|
||||
```
|
||||
📦 Package: /data/backups/igny8/igny8-app-20251210_161527.tar.gz
|
||||
📏 Size: 44 MB
|
||||
🔐 SHA256: 5aa8fe458bcb75d1f4a455746a484015a81147bbfa8e8b1285e99195e2deacbd
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### Automatic Daily Backups (00:00 UTC)
|
||||
1. Celery Beat triggers `backup.create_daily_app_package` at midnight
|
||||
2. Task executes `/data/app/igny8/scripts/package_app.sh`
|
||||
3. Script creates compressed archive with date stamp
|
||||
4. Package saved to `/data/backups/igny8/`
|
||||
5. SHA256 checksum file generated
|
||||
6. Old backups (7+ days) auto-deleted
|
||||
|
||||
### What Gets Packaged
|
||||
**INCLUDED** (~56 MB source):
|
||||
- `backend/igny8_core/` - All Django code
|
||||
- `backend/manage.py`, `requirements.txt`
|
||||
- `frontend/src/`, `public/` - All React code
|
||||
- `frontend/package.json`, build configs
|
||||
- `docs/`, `README.md`
|
||||
- Dockerfiles, docker-compose files
|
||||
|
||||
**EXCLUDED** (auto-generated):
|
||||
- `node_modules/` - NPM dependencies (297 MB)
|
||||
- `.venv/` - Python virtual environment (140 MB)
|
||||
- `dist/` - Build outputs
|
||||
- `*.sql`, `*.sqlite3` - Databases
|
||||
- `logs/` - Runtime logs
|
||||
- Cache files, compiled files
|
||||
|
||||
### Restoration Process
|
||||
```bash
|
||||
# On new server:
|
||||
1. Extract: tar -xzf igny8-app-YYYYMMDD_HHMMSS.tar.gz
|
||||
2. Verify: sha256sum -c igny8-app-YYYYMMDD_HHMMSS.tar.gz.sha256
|
||||
3. Follow INSTALLATION.md to deploy
|
||||
```
|
||||
|
||||
## Testing & Verification
|
||||
|
||||
### Manual Test (Already Done ✅)
|
||||
```bash
|
||||
/data/app/igny8/scripts/package_app.sh
|
||||
# Result: Package created successfully at /data/backups/igny8/
|
||||
```
|
||||
|
||||
### Verify Celery Tasks
|
||||
```bash
|
||||
# Restart Celery to load new tasks
|
||||
docker restart igny8_celery_worker igny8_celery_beat
|
||||
|
||||
# Check scheduled tasks
|
||||
docker exec igny8_celery_beat celery -A igny8_core inspect scheduled
|
||||
|
||||
# View Beat logs
|
||||
docker logs igny8_celery_beat
|
||||
|
||||
# Manually trigger backup (for testing)
|
||||
docker exec igny8_celery_worker celery -A igny8_core call backup.create_daily_app_package
|
||||
```
|
||||
|
||||
### Monitor Backups
|
||||
```bash
|
||||
# List all backups
|
||||
ls -lh /data/backups/igny8/
|
||||
|
||||
# Check latest backup
|
||||
ls -lt /data/backups/igny8/ | head -3
|
||||
|
||||
# Verify checksum
|
||||
cd /data/backups/igny8/
|
||||
sha256sum -c igny8-app-*.tar.gz.sha256
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Wait for first automated backup** (will run at 00:00 UTC tonight)
|
||||
2. **Verify backup runs successfully**:
|
||||
```bash
|
||||
# Check tomorrow morning
|
||||
ls -lt /data/backups/igny8/
|
||||
```
|
||||
3. **Optional: Add remote backup** (copy to S3, Google Drive, etc.)
|
||||
|
||||
## Retention Policy
|
||||
|
||||
- **Packaging script**: Deletes backups > 7 days old
|
||||
- **Cleanup task**: Deletes backups > 30 days old (weekly check)
|
||||
- **Result**: 7-30 days of backups retained automatically
|
||||
|
||||
## Package Portability
|
||||
|
||||
✅ **Portable** - Package contains only source code
|
||||
✅ **Small** - 44 MB compressed (vs ~500+ MB with dependencies)
|
||||
✅ **Clean** - No generated files, logs, or secrets
|
||||
✅ **Verified** - SHA256 checksum ensures integrity
|
||||
✅ **Self-contained** - Everything needed to rebuild app
|
||||
|
||||
## Deployment to New Server
|
||||
|
||||
From the package, you get a clean IGNY8 installation:
|
||||
|
||||
1. Extract package (44 MB)
|
||||
2. Install Docker
|
||||
3. Build images (Docker installs dependencies automatically)
|
||||
4. Start services
|
||||
5. Initialize database
|
||||
6. **Ready to run!**
|
||||
|
||||
**Dependencies installed automatically by Docker**:
|
||||
- Python packages from `requirements.txt`
|
||||
- Node packages from `package.json`
|
||||
- PostgreSQL, Redis containers
|
||||
- All build tools and dependencies
|
||||
|
||||
## Summary
|
||||
|
||||
✅ **Automated daily backups** at midnight
|
||||
✅ **Portable packages** - only source code
|
||||
✅ **Auto-cleanup** - keeps 7-30 days
|
||||
✅ **Checksum verification** - ensures integrity
|
||||
✅ **Easy restoration** - extract and deploy
|
||||
✅ **Documentation** - complete guides created
|
||||
|
||||
**Total disk usage for backups**: ~44 MB × 7-30 days = **308 MB - 1.3 GB**
|
||||
|
||||
Much better than storing everything (would be 10+ GB with node_modules, venv, etc.)!
|
||||
Reference in New Issue
Block a user