igny8/docs/plans/PHASE-6-BACKUP-CLEANUP-GUIDE.md

# Phase 6: Data Backup & Cleanup Guide

**Version:** 1.0  
**Created:** January 9, 2026  
**Purpose:** Pre-V1.0 Launch Database Preparation

---

## 📋 Table of Contents

1. [Overview](#overview)
2. [What Was Created](#what-was-created)
3. [When to Use](#when-to-use)
4. [Pre-Execution Checklist](#pre-execution-checklist)
5. [Command 1: Export System Config](#command-1-export-system-config)
6. [Command 2: Cleanup User Data](#command-2-cleanup-user-data)
7. [Complete Workflow](#complete-workflow)
8. [Safety Measures](#safety-measures)
9. [Rollback Procedures](#rollback-procedures)
10. [FAQ](#faq)

---

## 📖 Overview

Phase 6 provides two Django management commands to safely prepare your IGNY8 database for V1.0 production launch:

1. **Export System Configuration** - Backs up all system settings to JSON files
2. **Cleanup User Data** - Removes all test/development user data while preserving system configuration

### Why These Commands?

- **Clean Start**: Launch V1.0 with a pristine database
- **Configuration Preservation**: Keep all your carefully configured settings
- **Safety First**: Multiple safety checks and dry-run options
- **Audit Trail**: Complete metadata and logging

---

## 🛠️ What Was Created

### File Locations

```
backend/igny8_core/management/commands/
├── export_system_config.py    # System configuration backup
└── cleanup_user_data.py        # User data cleanup
```

### Command 1: `export_system_config.py`

**Purpose**: Exports all system configuration to JSON files for backup and version control.

**What it exports:**
- ✅ Subscription Plans (Starter, Growth, Scale)
- ✅ Credit Cost Configurations
- ✅ AI Model Settings (OpenAI, Anthropic, etc.)
- ✅ Global Integration Settings
- ✅ Industries and Sectors
- ✅ Seed Keywords (reference data)
- ✅ Author Profiles
- ✅ AI Prompts and Variables

**What it creates:**
- Individual JSON files for each data type
- `export_metadata.json` with timestamp and statistics
- Organized folder structure in `backups/config/`

### Command 2: `cleanup_user_data.py`

**Purpose**: Safely removes all user-generated test data before production launch.

**What it deletes:**
- 🗑️ Sites and Site Settings
- 🗑️ Keywords, Clusters, Ideas
- 🗑️ Tasks, Content, Images
- 🗑️ Publishing Records
- 🗑️ WordPress Sync Events
- 🗑️ Credit Transactions and Usage Logs
- 🗑️ Automation Runs
- 🗑️ Notifications
- 🗑️ Orders

**What it preserves:**
- ✅ User Accounts (admin users)
- ✅ System Configuration (all settings from export)
- ✅ Plans and Pricing
- ✅ AI Models and Prompts
- ✅ Industries and Sectors

---

## ⏰ When to Use

### Correct Timing

✅ **Use these commands when:**
- You're preparing for V1.0 production launch
- You've completed all testing and configuration
- You want to start production with clean data
- All system settings (Plans, AI models, prompts) are finalized

❌ **Do NOT use these commands when:**
- You're still in active development
- You haven't backed up your configurations
- You're unsure about your system settings
- You're in production with live users

### Recommended Timeline

```
Day -7: Final configuration review
Day -5: Export system config (first backup)
Day -3: Test commands in staging
Day -2: Export system config (final backup)
Day -1: Cleanup user data in staging
Day  0: Launch day - cleanup in production
```

---

## ✅ Pre-Execution Checklist

Before running ANY Phase 6 command, complete this checklist:

### Environment Verification

- [ ] Confirm you're in the correct environment (staging vs production)
- [ ] Check `ENVIRONMENT` setting in Django settings
- [ ] Verify database connection is correct
- [ ] Ensure you have full database backup

### System State

- [ ] All Plans configured and tested
- [ ] All AI prompts finalized
- [ ] All credit costs verified
- [ ] All industries/sectors populated
- [ ] Seed keywords imported

### Safety Backups

- [ ] Full database dump exists
- [ ] Previous export exists (if available)
- [ ] Media files backed up
- [ ] Environment variables documented

### Access & Permissions

- [ ] You have Django shell access
- [ ] You have database backup access
- [ ] You have rollback permissions
- [ ] Stakeholders notified

---

## 📤 Command 1: Export System Config

### Basic Usage

```bash
cd /data/app/igny8/backend
python manage.py export_system_config
```

### With Custom Output Directory

```bash
python manage.py export_system_config --output-dir=/path/to/backup
```

### Step-by-Step Execution

#### Step 1: Navigate to Backend

```bash
cd /data/app/igny8/backend
```

#### Step 2: Run Export

```bash
python manage.py export_system_config --output-dir=../backups/config/$(date +%Y%m%d)
```

#### Step 3: Verify Output

```bash
ls -la ../backups/config/$(date +%Y%m%d)/
```

Expected output:
```
plans.json                    # Subscription plans
credit_costs.json             # Credit cost configurations
ai_models.json                # AI model settings
global_integrations.json      # Integration settings
industries.json               # Industry master data
sectors.json                  # Sector master data
seed_keywords.json            # Reference keywords
author_profiles.json          # Writing style profiles
prompts.json                  # AI prompts
prompt_variables.json         # Prompt variables
export_metadata.json          # Export timestamp & stats
```

#### Step 4: Verify Data

Check one of the exports:
```bash
cat ../backups/config/$(date +%Y%m%d)/plans.json | head -20
```

#### Step 5: Commit to Version Control

```bash
cd /data/app/igny8
git add backups/config/
git commit -m "Backup: V1.0 system configuration export"
git push
```

### What The Output Looks Like

```
Exporting system configuration to: /data/app/igny8/backups/config/20260109

✓ Exported    3 Subscription Plans              → plans.json
✓ Exported   12 Credit Cost Configurations      → credit_costs.json
✓ Exported    4 AI Model Configurations         → ai_models.json
✓ Exported    1 Global Integration Settings     → global_integrations.json
✓ Exported   15 Industries                      → industries.json
✓ Exported   47 Sectors                         → sectors.json
✓ Exported  523 Seed Keywords                   → seed_keywords.json
✓ Exported    3 Author Profiles                 → author_profiles.json
✓ Exported    8 AI Prompts                      → prompts.json
✓ Exported   12 Prompt Variables                → prompt_variables.json

✓ Metadata saved to export_metadata.json

======================================================================
System Configuration Export Complete!

  Successful: 10 exports
  Failed:     0 exports
  Location:   /data/app/igny8/backups/config/20260109
======================================================================
```

### Troubleshooting

**Problem**: "No module named 'django'"
```bash
# Solution: Activate virtual environment or use Docker
docker-compose exec backend python manage.py export_system_config
```

**Problem**: "Permission denied" when writing files
```bash
# Solution: Check directory permissions
mkdir -p ../backups/config
chmod 755 ../backups/config
```

**Problem**: Empty JSON files
```bash
# Solution: Verify data exists in database
python manage.py shell
>>> from igny8_core.modules.billing.models import Plan
>>> Plan.objects.count()
```

---

## 🗑️ Command 2: Cleanup User Data

### ⚠️ CRITICAL WARNING

**THIS COMMAND PERMANENTLY DELETES DATA**

- Cannot be undone without database restore
- Removes ALL user-generated content
- Should ONLY be run before production launch
- ALWAYS run `--dry-run` first

### Safety Features

1. **Dry-Run Mode**: Preview deletions without actually deleting
2. **Confirmation Prompt**: Must type "DELETE ALL DATA" to proceed
3. **Production Protection**: Blocked in production environment (unless explicitly allowed)
4. **Transaction Safety**: All deletions in atomic transaction
5. **Detailed Logging**: Shows exactly what was deleted

### Usage: Dry Run (Always First!)

```bash
cd /data/app/igny8/backend
python manage.py cleanup_user_data --dry-run
```

### Dry Run Output Example

```
======================================================================
DRY RUN - No data will be deleted
======================================================================

  ✓ Would delete   1,234 Notifications
  ✓ Would delete   5,678 Credit Usage Logs
  ✓ Would delete     456 Credit Transactions
  ✓ Would delete      23 Orders
  ✓ Would delete   8,901 WordPress Sync Events
  ✓ Would delete     234 Publishing Records
  ✓ Would delete      45 Automation Runs
  ✓ Would delete   3,456 Images
  ✓ Would delete   2,345 Content
  ✓ Would delete   4,567 Tasks
  ✓ Would delete   5,678 Content Ideas
  ✓ Would delete   1,234 Clusters
  ✓ Would delete   9,876 Keywords
  ✓ Would delete      12 Sites

  → Keeping          3 Users (not deleted)

  Total records to delete: 43,739

======================================================================
To proceed with actual deletion, run:
  python manage.py cleanup_user_data --confirm
======================================================================
```

### Usage: Actual Cleanup

```bash
python manage.py cleanup_user_data --confirm
```

**You will be prompted:**
```
======================================================================
⚠️  DELETING ALL USER DATA - THIS CANNOT BE UNDONE!
======================================================================

Type "DELETE ALL DATA" to proceed:
```

**Type exactly:** `DELETE ALL DATA`

### Actual Cleanup Output

```
Proceeding with deletion...

✓ Deleted  1,234 Notifications
✓ Deleted  5,678 Credit Usage Logs
✓ Deleted    456 Credit Transactions
✓ Deleted     23 Orders
✓ Deleted  8,901 WordPress Sync Events
✓ Deleted    234 Publishing Records
✓ Deleted     45 Automation Runs
✓ Deleted  3,456 Images
✓ Deleted  2,345 Content
✓ Deleted  4,567 Tasks
✓ Deleted  5,678 Content Ideas
✓ Deleted  1,234 Clusters
✓ Deleted  9,876 Keywords
✓ Deleted     12 Sites

======================================================================
User Data Cleanup Complete!

  Total records deleted: 43,739
  Failed deletions:      0
======================================================================
```

### Production Environment Protection

If you try to run cleanup in production:

```
⚠️  BLOCKED: Cannot run cleanup in PRODUCTION environment!

To allow this, temporarily set ENVIRONMENT to "staging" in settings.
```

To override (ONLY if absolutely necessary):

```python
# In settings.py - TEMPORARY
ENVIRONMENT = 'staging'  # Change back after cleanup!
```

---

## 🔄 Complete Workflow

### Full Pre-Launch Procedure

```bash
# ========================================
# STEP 1: FULL DATABASE BACKUP
# ========================================
cd /data/app/igny8/backend
pg_dump -h localhost -U postgres igny8_db > ../backups/$(date +%Y%m%d)_pre_v1_full_backup.sql

# Verify backup exists and has content
ls -lh ../backups/$(date +%Y%m%d)_pre_v1_full_backup.sql
head -50 ../backups/$(date +%Y%m%d)_pre_v1_full_backup.sql


# ========================================
# STEP 2: EXPORT SYSTEM CONFIGURATION
# ========================================
python manage.py export_system_config --output-dir=../backups/config/$(date +%Y%m%d)

# Verify exports
ls -la ../backups/config/$(date +%Y%m%d)/

# Review critical configs
cat ../backups/config/$(date +%Y%m%d)/plans.json | python -m json.tool | head -30
cat ../backups/config/$(date +%Y%m%d)/credit_costs.json | python -m json.tool | head -30


# ========================================
# STEP 3: COMMIT CONFIGS TO GIT
# ========================================
cd /data/app/igny8
git add backups/config/
git commit -m "Pre-V1.0: System configuration backup $(date +%Y%m%d)"
git push


# ========================================
# STEP 4: BACKUP MEDIA FILES
# ========================================
cd /data/app/igny8
tar -czf backups/$(date +%Y%m%d)_media_backup.tar.gz backend/media/


# ========================================
# STEP 5: DRY RUN CLEANUP (REVIEW CAREFULLY)
# ========================================
cd backend
python manage.py cleanup_user_data --dry-run

# Review the counts - make sure they're expected


# ========================================
# STEP 6: ACTUAL CLEANUP (POINT OF NO RETURN)
# ========================================
python manage.py cleanup_user_data --confirm
# Type: DELETE ALL DATA


# ========================================
# STEP 7: VERIFY CLEANUP
# ========================================
python manage.py shell << 'EOF'
from igny8_core.auth.models import Site, CustomUser
from igny8_core.business.planning.models import Keywords
from igny8_core.modules.billing.models import Plan

print(f"Sites: {Site.objects.count()} (should be 0)")
print(f"Keywords: {Keywords.objects.count()} (should be 0)")
print(f"Users: {CustomUser.objects.count()} (admins preserved)")
print(f"Plans: {Plan.objects.count()} (should have your plans)")
EOF


# ========================================
# STEP 8: TEST APPLICATION
# ========================================
python manage.py runserver 0.0.0.0:8000 &
# Visit app and verify:
# - Can login as admin
# - Dashboard loads (empty state)
# - Plans visible in settings
# - Can create new user account


# ========================================
# STEP 9: TAG RELEASE
# ========================================
cd /data/app/igny8
git tag -a v1.0.0-clean -m "V1.0.0 - Clean database ready for launch"
git push origin v1.0.0-clean
```

---

## 🛡️ Safety Measures

### Built-in Protections

1. **Atomic Transactions**: All deletions in single transaction - all or nothing
2. **Production Check**: Requires explicit override in production
3. **Confirmation Prompt**: Must type exact phrase
4. **Dry Run**: See exactly what will be deleted
5. **Detailed Logging**: Know what was deleted and any failures

### Manual Safety Checklist

Before running cleanup:

- [ ] **Full database backup** exists and verified
- [ ] **System config export** completed and committed to git
- [ ] **Media files** backed up
- [ ] **Dry run reviewed** and counts are expected
- [ ] **Stakeholders notified** of pending cleanup
- [ ] **Rollback plan** documented and tested
- [ ] **Off-hours execution** scheduled (if production)
- [ ] **Monitoring ready** to catch any issues

### Additional Recommendations

1. **Staging First**: Always test in staging environment first
2. **Screenshot Evidence**: Take screenshots of dry-run output
3. **Communication**: Notify team before and after
4. **Timing**: Run during low-traffic hours
5. **Verification**: Test application immediately after

---

## 🔙 Rollback Procedures

### If Something Goes Wrong

#### During Cleanup (Transaction Failed)

No action needed - atomic transaction will automatically rollback.

#### After Cleanup (Need to Restore)

```bash
# OPTION 1: Restore from PostgreSQL backup
cd /data/app/igny8
psql -U postgres -d igny8_db < backups/20260109_pre_v1_full_backup.sql

# OPTION 2: Restore specific tables (if partial restore needed)
pg_restore -U postgres -d igny8_db -t "specific_table" backups/20260109_pre_v1_full_backup.sql

# OPTION 3: Restore from Docker backup (if using Docker)
docker-compose exec -T db psql -U postgres igny8_db < backups/20260109_pre_v1_full_backup.sql
```

#### Restore Media Files

```bash
cd /data/app/igny8
tar -xzf backups/20260109_media_backup.tar.gz -C backend/
```

#### Verify Restore

```bash
cd backend
python manage.py shell << 'EOF'
from igny8_core.auth.models import Site
from igny8_core.business.planning.models import Keywords
print(f"Sites restored: {Site.objects.count()}")
print(f"Keywords restored: {Keywords.objects.count()}")
EOF
```

---

## ❓ FAQ

### Q: Can I run these commands multiple times?

**A:** 
- **Export Config**: Yes, safe to run multiple times. Creates timestamped backups.
- **Cleanup**: Yes, but after first cleanup there's nothing left to delete (idempotent).

### Q: What if I only want to delete some data?

**A:** These commands are all-or-nothing by design for safety. To delete specific data, use Django admin or write a custom management command.

### Q: Can I restore individual items from the export?

**A:** Yes! The JSON files use Django's standard serialization format. Use `python manage.py loaddata <file>.json` to restore.

### Q: Will this affect my development environment?

**A:** Only if you run it there. These commands work on whatever database your Django settings point to.

### Q: How long does cleanup take?

**A:** Depends on data volume. Typical ranges:
- Small (< 10k records): 1-5 seconds
- Medium (10k-100k): 5-30 seconds  
- Large (> 100k): 30-120 seconds

### Q: What if cleanup fails halfway?

**A:** Can't happen - it's wrapped in an atomic transaction. Either everything deletes or nothing does.

### Q: Do I need to stop the application?

**A:** Recommended but not required. Stopping the app prevents race conditions during cleanup.

### Q: Can I schedule these as cron jobs?

**A:** 
- **Export**: Yes, great for automated backups
- **Cleanup**: No, should only be run manually with explicit confirmation

### Q: What about Django migrations?

**A:** Cleanup only deletes data, not schema. All tables and migrations remain intact.

### Q: How do I know if my system config is complete?

**A:** Run the export and review the counts in `export_metadata.json`. Compare with your documentation.

---

## 📞 Support

### If You Need Help

1. **Check this guide** thoroughly first
2. **Review error messages** carefully
3. **Test in staging** before production
4. **Contact team** if unsure about any step

### Emergency Contacts

- **Database Issues**: DBA team
- **Application Issues**: Backend team
- **Configuration Questions**: System admin
- **Rollback Needed**: All hands on deck!

---

## ✅ Success Criteria

After completing Phase 6, you should have:

- ✅ Multiple timestamped config exports in `backups/config/`
- ✅ Full database SQL backup in `backups/`
- ✅ Media files backup in `backups/`
- ✅ Zero user-generated data in database
- ✅ All system configurations intact
- ✅ Application starts and loads empty state
- ✅ Admin can log in
- ✅ New users can sign up
- ✅ Plans visible and functional
- ✅ Git tag created for v1.0.0-clean

---

**Document Version:** 1.0  
**Last Updated:** January 9, 2026  
**Next Review:** After V1.0 Launch

---

*This guide is part of the IGNY8 Pre-Launch Preparation (Phase 6)*