663 lines
18 KiB
Markdown
663 lines
18 KiB
Markdown
# Phase 6: Data Backup & Cleanup Guide
|
|
|
|
**Version:** 1.0
|
|
**Created:** January 9, 2026
|
|
**Purpose:** Pre-V1.0 Launch Database Preparation
|
|
|
|
---
|
|
|
|
## 📋 Table of Contents
|
|
|
|
1. [Overview](#overview)
|
|
2. [What Was Created](#what-was-created)
|
|
3. [When to Use](#when-to-use)
|
|
4. [Pre-Execution Checklist](#pre-execution-checklist)
|
|
5. [Command 1: Export System Config](#command-1-export-system-config)
|
|
6. [Command 2: Cleanup User Data](#command-2-cleanup-user-data)
|
|
7. [Complete Workflow](#complete-workflow)
|
|
8. [Safety Measures](#safety-measures)
|
|
9. [Rollback Procedures](#rollback-procedures)
|
|
10. [FAQ](#faq)
|
|
|
|
---
|
|
|
|
## 📖 Overview
|
|
|
|
Phase 6 provides two Django management commands to safely prepare your IGNY8 database for V1.0 production launch:
|
|
|
|
1. **Export System Configuration** - Backs up all system settings to JSON files
|
|
2. **Cleanup User Data** - Removes all test/development user data while preserving system configuration
|
|
|
|
### Why These Commands?
|
|
|
|
- **Clean Start**: Launch V1.0 with a pristine database
|
|
- **Configuration Preservation**: Keep all your carefully configured settings
|
|
- **Safety First**: Multiple safety checks and dry-run options
|
|
- **Audit Trail**: Complete metadata and logging
|
|
|
|
---
|
|
|
|
## 🛠️ What Was Created
|
|
|
|
### File Locations
|
|
|
|
```
|
|
backend/igny8_core/management/commands/
|
|
├── export_system_config.py # System configuration backup
|
|
└── cleanup_user_data.py # User data cleanup
|
|
```
|
|
|
|
### Command 1: `export_system_config.py`
|
|
|
|
**Purpose**: Exports all system configuration to JSON files for backup and version control.
|
|
|
|
**What it exports:**
|
|
- ✅ Subscription Plans (Starter, Growth, Scale)
|
|
- ✅ Credit Cost Configurations
|
|
- ✅ AI Model Settings (OpenAI, Anthropic, etc.)
|
|
- ✅ Global Integration Settings
|
|
- ✅ Industries and Sectors
|
|
- ✅ Seed Keywords (reference data)
|
|
- ✅ Author Profiles
|
|
- ✅ AI Prompts and Variables
|
|
|
|
**What it creates:**
|
|
- Individual JSON files for each data type
|
|
- `export_metadata.json` with timestamp and statistics
|
|
- Organized folder structure in `backups/config/`
|
|
|
|
### Command 2: `cleanup_user_data.py`
|
|
|
|
**Purpose**: Safely removes all user-generated test data before production launch.
|
|
|
|
**What it deletes:**
|
|
- 🗑️ Sites and Site Settings
|
|
- 🗑️ Keywords, Clusters, Ideas
|
|
- 🗑️ Tasks, Content, Images
|
|
- 🗑️ Publishing Records
|
|
- 🗑️ WordPress Sync Events
|
|
- 🗑️ Credit Transactions and Usage Logs
|
|
- 🗑️ Automation Runs
|
|
- 🗑️ Notifications
|
|
- 🗑️ Orders
|
|
|
|
**What it preserves:**
|
|
- ✅ User Accounts (admin users)
|
|
- ✅ System Configuration (all settings from export)
|
|
- ✅ Plans and Pricing
|
|
- ✅ AI Models and Prompts
|
|
- ✅ Industries and Sectors
|
|
|
|
---
|
|
|
|
## ⏰ When to Use
|
|
|
|
### Correct Timing
|
|
|
|
✅ **Use these commands when:**
|
|
- You're preparing for V1.0 production launch
|
|
- You've completed all testing and configuration
|
|
- You want to start production with clean data
|
|
- All system settings (Plans, AI models, prompts) are finalized
|
|
|
|
❌ **Do NOT use these commands when:**
|
|
- You're still in active development
|
|
- You haven't backed up your configurations
|
|
- You're unsure about your system settings
|
|
- You're in production with live users
|
|
|
|
### Recommended Timeline
|
|
|
|
```
|
|
Day -7: Final configuration review
|
|
Day -5: Export system config (first backup)
|
|
Day -3: Test commands in staging
|
|
Day -2: Export system config (final backup)
|
|
Day -1: Cleanup user data in staging
|
|
Day 0: Launch day - cleanup in production
|
|
```
|
|
|
|
---
|
|
|
|
## ✅ Pre-Execution Checklist
|
|
|
|
Before running ANY Phase 6 command, complete this checklist:
|
|
|
|
### Environment Verification
|
|
|
|
- [ ] Confirm you're in the correct environment (staging vs production)
|
|
- [ ] Check `ENVIRONMENT` setting in Django settings
|
|
- [ ] Verify database connection is correct
|
|
- [ ] Ensure you have full database backup
|
|
|
|
### System State
|
|
|
|
- [ ] All Plans configured and tested
|
|
- [ ] All AI prompts finalized
|
|
- [ ] All credit costs verified
|
|
- [ ] All industries/sectors populated
|
|
- [ ] Seed keywords imported
|
|
|
|
### Safety Backups
|
|
|
|
- [ ] Full database dump exists
|
|
- [ ] Previous export exists (if available)
|
|
- [ ] Media files backed up
|
|
- [ ] Environment variables documented
|
|
|
|
### Access & Permissions
|
|
|
|
- [ ] You have Django shell access
|
|
- [ ] You have database backup access
|
|
- [ ] You have rollback permissions
|
|
- [ ] Stakeholders notified
|
|
|
|
---
|
|
|
|
## 📤 Command 1: Export System Config
|
|
|
|
### Basic Usage
|
|
|
|
```bash
|
|
cd /data/app/igny8/backend
|
|
python manage.py export_system_config
|
|
```
|
|
|
|
### With Custom Output Directory
|
|
|
|
```bash
|
|
python manage.py export_system_config --output-dir=/path/to/backup
|
|
```
|
|
|
|
### Step-by-Step Execution
|
|
|
|
#### Step 1: Navigate to Backend
|
|
|
|
```bash
|
|
cd /data/app/igny8/backend
|
|
```
|
|
|
|
#### Step 2: Run Export
|
|
|
|
```bash
|
|
python manage.py export_system_config --output-dir=../backups/config/$(date +%Y%m%d)
|
|
```
|
|
|
|
#### Step 3: Verify Output
|
|
|
|
```bash
|
|
ls -la ../backups/config/$(date +%Y%m%d)/
|
|
```
|
|
|
|
Expected output:
|
|
```
|
|
plans.json # Subscription plans
|
|
credit_costs.json # Credit cost configurations
|
|
ai_models.json # AI model settings
|
|
global_integrations.json # Integration settings
|
|
industries.json # Industry master data
|
|
sectors.json # Sector master data
|
|
seed_keywords.json # Reference keywords
|
|
author_profiles.json # Writing style profiles
|
|
prompts.json # AI prompts
|
|
prompt_variables.json # Prompt variables
|
|
export_metadata.json # Export timestamp & stats
|
|
```
|
|
|
|
#### Step 4: Verify Data
|
|
|
|
Check one of the exports:
|
|
```bash
|
|
cat ../backups/config/$(date +%Y%m%d)/plans.json | head -20
|
|
```
|
|
|
|
#### Step 5: Commit to Version Control
|
|
|
|
```bash
|
|
cd /data/app/igny8
|
|
git add backups/config/
|
|
git commit -m "Backup: V1.0 system configuration export"
|
|
git push
|
|
```
|
|
|
|
### What The Output Looks Like
|
|
|
|
```
|
|
Exporting system configuration to: /data/app/igny8/backups/config/20260109
|
|
|
|
✓ Exported 3 Subscription Plans → plans.json
|
|
✓ Exported 12 Credit Cost Configurations → credit_costs.json
|
|
✓ Exported 4 AI Model Configurations → ai_models.json
|
|
✓ Exported 1 Global Integration Settings → global_integrations.json
|
|
✓ Exported 15 Industries → industries.json
|
|
✓ Exported 47 Sectors → sectors.json
|
|
✓ Exported 523 Seed Keywords → seed_keywords.json
|
|
✓ Exported 3 Author Profiles → author_profiles.json
|
|
✓ Exported 8 AI Prompts → prompts.json
|
|
✓ Exported 12 Prompt Variables → prompt_variables.json
|
|
|
|
✓ Metadata saved to export_metadata.json
|
|
|
|
======================================================================
|
|
System Configuration Export Complete!
|
|
|
|
Successful: 10 exports
|
|
Failed: 0 exports
|
|
Location: /data/app/igny8/backups/config/20260109
|
|
======================================================================
|
|
```
|
|
|
|
### Troubleshooting
|
|
|
|
**Problem**: "No module named 'django'"
|
|
```bash
|
|
# Solution: Activate virtual environment or use Docker
|
|
docker-compose exec backend python manage.py export_system_config
|
|
```
|
|
|
|
**Problem**: "Permission denied" when writing files
|
|
```bash
|
|
# Solution: Check directory permissions
|
|
mkdir -p ../backups/config
|
|
chmod 755 ../backups/config
|
|
```
|
|
|
|
**Problem**: Empty JSON files
|
|
```bash
|
|
# Solution: Verify data exists in database
|
|
python manage.py shell
|
|
>>> from igny8_core.modules.billing.models import Plan
|
|
>>> Plan.objects.count()
|
|
```
|
|
|
|
---
|
|
|
|
## 🗑️ Command 2: Cleanup User Data
|
|
|
|
### ⚠️ CRITICAL WARNING
|
|
|
|
**THIS COMMAND PERMANENTLY DELETES DATA**
|
|
|
|
- Cannot be undone without database restore
|
|
- Removes ALL user-generated content
|
|
- Should ONLY be run before production launch
|
|
- ALWAYS run `--dry-run` first
|
|
|
|
### Safety Features
|
|
|
|
1. **Dry-Run Mode**: Preview deletions without actually deleting
|
|
2. **Confirmation Prompt**: Must type "DELETE ALL DATA" to proceed
|
|
3. **Production Protection**: Blocked in production environment (unless explicitly allowed)
|
|
4. **Transaction Safety**: All deletions in atomic transaction
|
|
5. **Detailed Logging**: Shows exactly what was deleted
|
|
|
|
### Usage: Dry Run (Always First!)
|
|
|
|
```bash
|
|
cd /data/app/igny8/backend
|
|
python manage.py cleanup_user_data --dry-run
|
|
```
|
|
|
|
### Dry Run Output Example
|
|
|
|
```
|
|
======================================================================
|
|
DRY RUN - No data will be deleted
|
|
======================================================================
|
|
|
|
✓ Would delete 1,234 Notifications
|
|
✓ Would delete 5,678 Credit Usage Logs
|
|
✓ Would delete 456 Credit Transactions
|
|
✓ Would delete 23 Orders
|
|
✓ Would delete 8,901 WordPress Sync Events
|
|
✓ Would delete 234 Publishing Records
|
|
✓ Would delete 45 Automation Runs
|
|
✓ Would delete 3,456 Images
|
|
✓ Would delete 2,345 Content
|
|
✓ Would delete 4,567 Tasks
|
|
✓ Would delete 5,678 Content Ideas
|
|
✓ Would delete 1,234 Clusters
|
|
✓ Would delete 9,876 Keywords
|
|
✓ Would delete 12 Sites
|
|
|
|
→ Keeping 3 Users (not deleted)
|
|
|
|
Total records to delete: 43,739
|
|
|
|
======================================================================
|
|
To proceed with actual deletion, run:
|
|
python manage.py cleanup_user_data --confirm
|
|
======================================================================
|
|
```
|
|
|
|
### Usage: Actual Cleanup
|
|
|
|
```bash
|
|
python manage.py cleanup_user_data --confirm
|
|
```
|
|
|
|
**You will be prompted:**
|
|
```
|
|
======================================================================
|
|
⚠️ DELETING ALL USER DATA - THIS CANNOT BE UNDONE!
|
|
======================================================================
|
|
|
|
Type "DELETE ALL DATA" to proceed:
|
|
```
|
|
|
|
**Type exactly:** `DELETE ALL DATA`
|
|
|
|
### Actual Cleanup Output
|
|
|
|
```
|
|
Proceeding with deletion...
|
|
|
|
✓ Deleted 1,234 Notifications
|
|
✓ Deleted 5,678 Credit Usage Logs
|
|
✓ Deleted 456 Credit Transactions
|
|
✓ Deleted 23 Orders
|
|
✓ Deleted 8,901 WordPress Sync Events
|
|
✓ Deleted 234 Publishing Records
|
|
✓ Deleted 45 Automation Runs
|
|
✓ Deleted 3,456 Images
|
|
✓ Deleted 2,345 Content
|
|
✓ Deleted 4,567 Tasks
|
|
✓ Deleted 5,678 Content Ideas
|
|
✓ Deleted 1,234 Clusters
|
|
✓ Deleted 9,876 Keywords
|
|
✓ Deleted 12 Sites
|
|
|
|
======================================================================
|
|
User Data Cleanup Complete!
|
|
|
|
Total records deleted: 43,739
|
|
Failed deletions: 0
|
|
======================================================================
|
|
```
|
|
|
|
### Production Environment Protection
|
|
|
|
If you try to run cleanup in production:
|
|
|
|
```
|
|
⚠️ BLOCKED: Cannot run cleanup in PRODUCTION environment!
|
|
|
|
To allow this, temporarily set ENVIRONMENT to "staging" in settings.
|
|
```
|
|
|
|
To override (ONLY if absolutely necessary):
|
|
|
|
```python
|
|
# In settings.py - TEMPORARY
|
|
ENVIRONMENT = 'staging' # Change back after cleanup!
|
|
```
|
|
|
|
---
|
|
|
|
## 🔄 Complete Workflow
|
|
|
|
### Full Pre-Launch Procedure
|
|
|
|
```bash
|
|
# ========================================
|
|
# STEP 1: FULL DATABASE BACKUP
|
|
# ========================================
|
|
cd /data/app/igny8/backend
|
|
pg_dump -h localhost -U postgres igny8_db > ../backups/$(date +%Y%m%d)_pre_v1_full_backup.sql
|
|
|
|
# Verify backup exists and has content
|
|
ls -lh ../backups/$(date +%Y%m%d)_pre_v1_full_backup.sql
|
|
head -50 ../backups/$(date +%Y%m%d)_pre_v1_full_backup.sql
|
|
|
|
|
|
# ========================================
|
|
# STEP 2: EXPORT SYSTEM CONFIGURATION
|
|
# ========================================
|
|
python manage.py export_system_config --output-dir=../backups/config/$(date +%Y%m%d)
|
|
|
|
# Verify exports
|
|
ls -la ../backups/config/$(date +%Y%m%d)/
|
|
|
|
# Review critical configs
|
|
cat ../backups/config/$(date +%Y%m%d)/plans.json | python -m json.tool | head -30
|
|
cat ../backups/config/$(date +%Y%m%d)/credit_costs.json | python -m json.tool | head -30
|
|
|
|
|
|
# ========================================
|
|
# STEP 3: COMMIT CONFIGS TO GIT
|
|
# ========================================
|
|
cd /data/app/igny8
|
|
git add backups/config/
|
|
git commit -m "Pre-V1.0: System configuration backup $(date +%Y%m%d)"
|
|
git push
|
|
|
|
|
|
# ========================================
|
|
# STEP 4: BACKUP MEDIA FILES
|
|
# ========================================
|
|
cd /data/app/igny8
|
|
tar -czf backups/$(date +%Y%m%d)_media_backup.tar.gz backend/media/
|
|
|
|
|
|
# ========================================
|
|
# STEP 5: DRY RUN CLEANUP (REVIEW CAREFULLY)
|
|
# ========================================
|
|
cd backend
|
|
python manage.py cleanup_user_data --dry-run
|
|
|
|
# Review the counts - make sure they're expected
|
|
|
|
|
|
# ========================================
|
|
# STEP 6: ACTUAL CLEANUP (POINT OF NO RETURN)
|
|
# ========================================
|
|
python manage.py cleanup_user_data --confirm
|
|
# Type: DELETE ALL DATA
|
|
|
|
|
|
# ========================================
|
|
# STEP 7: VERIFY CLEANUP
|
|
# ========================================
|
|
python manage.py shell << 'EOF'
|
|
from igny8_core.auth.models import Site, CustomUser
|
|
from igny8_core.business.planning.models import Keywords
|
|
from igny8_core.modules.billing.models import Plan
|
|
|
|
print(f"Sites: {Site.objects.count()} (should be 0)")
|
|
print(f"Keywords: {Keywords.objects.count()} (should be 0)")
|
|
print(f"Users: {CustomUser.objects.count()} (admins preserved)")
|
|
print(f"Plans: {Plan.objects.count()} (should have your plans)")
|
|
EOF
|
|
|
|
|
|
# ========================================
|
|
# STEP 8: TEST APPLICATION
|
|
# ========================================
|
|
python manage.py runserver 0.0.0.0:8000 &
|
|
# Visit app and verify:
|
|
# - Can login as admin
|
|
# - Dashboard loads (empty state)
|
|
# - Plans visible in settings
|
|
# - Can create new user account
|
|
|
|
|
|
# ========================================
|
|
# STEP 9: TAG RELEASE
|
|
# ========================================
|
|
cd /data/app/igny8
|
|
git tag -a v1.0.0-clean -m "V1.0.0 - Clean database ready for launch"
|
|
git push origin v1.0.0-clean
|
|
```
|
|
|
|
---
|
|
|
|
## 🛡️ Safety Measures
|
|
|
|
### Built-in Protections
|
|
|
|
1. **Atomic Transactions**: All deletions in single transaction - all or nothing
|
|
2. **Production Check**: Requires explicit override in production
|
|
3. **Confirmation Prompt**: Must type exact phrase
|
|
4. **Dry Run**: See exactly what will be deleted
|
|
5. **Detailed Logging**: Know what was deleted and any failures
|
|
|
|
### Manual Safety Checklist
|
|
|
|
Before running cleanup:
|
|
|
|
- [ ] **Full database backup** exists and verified
|
|
- [ ] **System config export** completed and committed to git
|
|
- [ ] **Media files** backed up
|
|
- [ ] **Dry run reviewed** and counts are expected
|
|
- [ ] **Stakeholders notified** of pending cleanup
|
|
- [ ] **Rollback plan** documented and tested
|
|
- [ ] **Off-hours execution** scheduled (if production)
|
|
- [ ] **Monitoring ready** to catch any issues
|
|
|
|
### Additional Recommendations
|
|
|
|
1. **Staging First**: Always test in staging environment first
|
|
2. **Screenshot Evidence**: Take screenshots of dry-run output
|
|
3. **Communication**: Notify team before and after
|
|
4. **Timing**: Run during low-traffic hours
|
|
5. **Verification**: Test application immediately after
|
|
|
|
---
|
|
|
|
## 🔙 Rollback Procedures
|
|
|
|
### If Something Goes Wrong
|
|
|
|
#### During Cleanup (Transaction Failed)
|
|
|
|
No action needed - atomic transaction will automatically rollback.
|
|
|
|
#### After Cleanup (Need to Restore)
|
|
|
|
```bash
|
|
# OPTION 1: Restore from PostgreSQL backup
|
|
cd /data/app/igny8
|
|
psql -U postgres -d igny8_db < backups/20260109_pre_v1_full_backup.sql
|
|
|
|
# OPTION 2: Restore specific tables (if partial restore needed)
|
|
pg_restore -U postgres -d igny8_db -t "specific_table" backups/20260109_pre_v1_full_backup.sql
|
|
|
|
# OPTION 3: Restore from Docker backup (if using Docker)
|
|
docker-compose exec -T db psql -U postgres igny8_db < backups/20260109_pre_v1_full_backup.sql
|
|
```
|
|
|
|
#### Restore Media Files
|
|
|
|
```bash
|
|
cd /data/app/igny8
|
|
tar -xzf backups/20260109_media_backup.tar.gz -C backend/
|
|
```
|
|
|
|
#### Verify Restore
|
|
|
|
```bash
|
|
cd backend
|
|
python manage.py shell << 'EOF'
|
|
from igny8_core.auth.models import Site
|
|
from igny8_core.business.planning.models import Keywords
|
|
print(f"Sites restored: {Site.objects.count()}")
|
|
print(f"Keywords restored: {Keywords.objects.count()}")
|
|
EOF
|
|
```
|
|
|
|
---
|
|
|
|
## ❓ FAQ
|
|
|
|
### Q: Can I run these commands multiple times?
|
|
|
|
**A:**
|
|
- **Export Config**: Yes, safe to run multiple times. Creates timestamped backups.
|
|
- **Cleanup**: Yes, but after first cleanup there's nothing left to delete (idempotent).
|
|
|
|
### Q: What if I only want to delete some data?
|
|
|
|
**A:** These commands are all-or-nothing by design for safety. To delete specific data, use Django admin or write a custom management command.
|
|
|
|
### Q: Can I restore individual items from the export?
|
|
|
|
**A:** Yes! The JSON files use Django's standard serialization format. Use `python manage.py loaddata <file>.json` to restore.
|
|
|
|
### Q: Will this affect my development environment?
|
|
|
|
**A:** Only if you run it there. These commands work on whatever database your Django settings point to.
|
|
|
|
### Q: How long does cleanup take?
|
|
|
|
**A:** Depends on data volume. Typical ranges:
|
|
- Small (< 10k records): 1-5 seconds
|
|
- Medium (10k-100k): 5-30 seconds
|
|
- Large (> 100k): 30-120 seconds
|
|
|
|
### Q: What if cleanup fails halfway?
|
|
|
|
**A:** Can't happen - it's wrapped in an atomic transaction. Either everything deletes or nothing does.
|
|
|
|
### Q: Do I need to stop the application?
|
|
|
|
**A:** Recommended but not required. Stopping the app prevents race conditions during cleanup.
|
|
|
|
### Q: Can I schedule these as cron jobs?
|
|
|
|
**A:**
|
|
- **Export**: Yes, great for automated backups
|
|
- **Cleanup**: No, should only be run manually with explicit confirmation
|
|
|
|
### Q: What about Django migrations?
|
|
|
|
**A:** Cleanup only deletes data, not schema. All tables and migrations remain intact.
|
|
|
|
### Q: How do I know if my system config is complete?
|
|
|
|
**A:** Run the export and review the counts in `export_metadata.json`. Compare with your documentation.
|
|
|
|
---
|
|
|
|
## 📞 Support
|
|
|
|
### If You Need Help
|
|
|
|
1. **Check this guide** thoroughly first
|
|
2. **Review error messages** carefully
|
|
3. **Test in staging** before production
|
|
4. **Contact team** if unsure about any step
|
|
|
|
### Emergency Contacts
|
|
|
|
- **Database Issues**: DBA team
|
|
- **Application Issues**: Backend team
|
|
- **Configuration Questions**: System admin
|
|
- **Rollback Needed**: All hands on deck!
|
|
|
|
---
|
|
|
|
## ✅ Success Criteria
|
|
|
|
After completing Phase 6, you should have:
|
|
|
|
- ✅ Multiple timestamped config exports in `backups/config/`
|
|
- ✅ Full database SQL backup in `backups/`
|
|
- ✅ Media files backup in `backups/`
|
|
- ✅ Zero user-generated data in database
|
|
- ✅ All system configurations intact
|
|
- ✅ Application starts and loads empty state
|
|
- ✅ Admin can log in
|
|
- ✅ New users can sign up
|
|
- ✅ Plans visible and functional
|
|
- ✅ Git tag created for v1.0.0-clean
|
|
|
|
---
|
|
|
|
**Document Version:** 1.0
|
|
**Last Updated:** January 9, 2026
|
|
**Next Review:** After V1.0 Launch
|
|
|
|
---
|
|
|
|
*This guide is part of the IGNY8 Pre-Launch Preparation (Phase 6)*
|