docs-reorg

2025-12-11 01:23:17 +05:00
parent 50aafd9ce3
commit 20fdd3b295
4 changed files with 0 additions and 0 deletions
--- a/docs/package-backup/BACKUP-SYSTEM.md
+++ b/docs/package-backup/BACKUP-SYSTEM.md
--- a/docs/package-backup/INSTALLATION.md
+++ b/docs/package-backup/INSTALLATION.md
@@ -0,0 +1,431 @@
+# IGNY8 Application - Installation Guide
+
+**Complete step-by-step installation on a clean Ubuntu server**
+
+---
+
+## Prerequisites
+
+- **Clean Ubuntu 22.04 LTS or later** (stable version)
+- **Minimum Requirements**: 4GB RAM, 20GB disk space, 2 CPU cores
+- **Root or sudo access**
+- **Internet connection**
+
+---
+
+## Installation Steps
+
+### 1. Initial System Setup
+
+```bash
+# Update system
+sudo apt update && sudo apt upgrade -y
+
+# Install required system packages
+sudo apt install -y \
+    git \
+    curl \
+    wget \
+    ca-certificates \
+    gnupg \
+    lsb-release
+```
+
+---
+
+### 2. Install Docker & Docker Compose
+
+```bash
+# Add Docker's official GPG key
+sudo mkdir -p /etc/apt/keyrings
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+
+# Set up Docker repository
+echo \
+  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
+  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+
+# Install Docker Engine
+sudo apt update
+sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+
+# Verify installation
+docker --version
+docker compose version
+
+# Add current user to docker group (optional - for non-root usage)
+sudo usermod -aG docker $USER
+newgrp docker
+```
+
+---
+
+### 3. Install Portainer (Optional - for GUI management)
+
+```bash
+# Create Docker network
+docker network create igny8_net
+
+# Create Portainer volume
+docker volume create portainer_data
+
+# Run Portainer
+docker run -d \
+  --name portainer \
+  --restart always \
+  -p 9443:9443 \
+  -p 8000:8000 \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  -v portainer_data:/data \
+  --network igny8_net \
+  portainer/portainer-ce:latest
+
+# Access Portainer at: https://YOUR_SERVER_IP:9443
+```
+
+---
+
+### 4. Get IGNY8 Application
+
+**Option A: Clone from Git Repository**
+```bash
+# Create application directory
+sudo mkdir -p /data/app
+cd /data/app
+
+# Clone repository
+git clone <YOUR_IGNY8_REPO_URL> igny8
+cd igny8
+```
+
+**Option B: Upload Application Package**
+```bash
+# Create application directory
+sudo mkdir -p /data/app
+cd /data/app
+
+# Upload igny8.tar.gz to server, then extract
+tar -xzf igny8.tar.gz
+cd igny8
+```
+
+---
+
+### 5. Configure Environment Variables
+
+```bash
+# Create environment file
+cat > .env << 'EOF'
+# Database Configuration
+DB_HOST=postgres
+DB_NAME=igny8_db
+DB_USER=igny8
+DB_PASSWORD=CHANGE_THIS_PASSWORD
+
+# Redis Configuration
+REDIS_HOST=redis
+REDIS_PORT=6379
+
+# Django Settings
+DEBUG=False
+SECRET_KEY=CHANGE_THIS_TO_RANDOM_50_CHAR_STRING
+ALLOWED_HOSTS=*
+
+# Security
+USE_SECURE_COOKIES=True
+USE_SECURE_PROXY_HEADER=True
+
+# Application URLs
+VITE_BACKEND_URL=https://api.yourdomain.com/api
+EOF
+
+# Generate Django secret key
+python3 -c 'from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())' >> .env.secret
+```
+
+---
+
+### 6. Prepare Application Structure
+
+**What gets installed automatically (NO manual work needed):**
+
+✅ **Backend (Python packages)**: Installed inside Docker container from `requirements.txt`
+- Django, Celery, DRF, PostgreSQL drivers, etc.
+- No local Python venv needed
+
+✅ **Frontend (Node packages)**: Installed inside Docker container from `package.json`
+- React, Vite, TailwindCSS, etc.
+- No local node_modules needed
+
+✅ **Database**: PostgreSQL runs in Docker container
+
+✅ **Cache**: Redis runs in Docker container
+
+**Your IGNY8 folder should contain ONLY:**
+```
+igny8/
+├── backend/
+│   ├── igny8_core/          # Django application code
+│   ├── manage.py            # Django management
+│   ├── requirements.txt     # Python dependencies
+│   └── Dockerfile           # Backend container build
+├── frontend/
+│   ├── src/                 # React application code
+│   ├── public/              # Static assets
+│   ├── package.json         # Node dependencies
+│   └── Dockerfile.dev       # Frontend container build
+├── docker-compose.app.yml   # Application stack definition
+├── README.md
+└── docs/
+
+# These should NOT be present (automatically generated):
+❌ backend/.venv/            # Will be built in Docker
+❌ frontend/node_modules/    # Will be built in Docker
+❌ frontend/dist/            # Will be built in Docker
+❌ *.sql, *.sqlite3          # Database backups
+❌ logs/                     # Runtime logs
+```
+
+---
+
+### 7. Build Docker Images
+
+```bash
+cd /data/app/igny8
+
+# Build backend image
+cd backend
+docker build -t igny8-backend:latest -f Dockerfile .
+
+# Build frontend dev image
+cd ../frontend
+docker build -t igny8-frontend-dev:latest -f Dockerfile.dev .
+
+# Build marketing dev image (if needed)
+docker build -t igny8-marketing-dev:latest -f Dockerfile.marketing.dev .
+
+cd ..
+```
+
+---
+
+### 8. Start Infrastructure Services
+
+```bash
+# Create infrastructure stack (Postgres, Redis, Caddy, etc.)
+cd /data/app
+
+# Start infrastructure services
+docker compose -f docker-compose.yml up -d
+
+# Verify infrastructure is running
+docker ps
+```
+
+---
+
+### 9. Start IGNY8 Application
+
+```bash
+# Start application stack
+cd /data/app/igny8
+docker compose -f docker-compose.app.yml up -d
+
+# Check all containers are running
+docker ps -a
+
+# View logs
+docker logs igny8_backend
+docker logs igny8_frontend
+docker logs igny8_celery_worker
+```
+
+---
+
+### 10. Initialize Database
+
+```bash
+# Run migrations
+docker exec igny8_backend python manage.py migrate
+
+# Create superuser
+docker exec -it igny8_backend python manage.py createsuperuser
+
+# Collect static files (if needed)
+docker exec igny8_backend python manage.py collectstatic --noinput
+```
+
+---
+
+### 11. Verify Installation
+
+**Check services:**
+```bash
+# Backend API
+curl http://localhost:8011/api/v1/system/status/
+
+# Frontend
+curl http://localhost:8021
+
+# Access application
+# Backend API: http://YOUR_SERVER_IP:8011
+# Frontend App: http://YOUR_SERVER_IP:8021
+# Marketing: http://YOUR_SERVER_IP:8023
+```
+
+**Check container health:**
+```bash
+docker ps --format "table {{.Names}}\t{{.Status}}"
+```
+
+---
+
+## What Happens Automatically vs. Manually
+
+### ✅ AUTOMATIC (Docker handles everything):
+
+1. **Python environment setup**: Docker builds venv inside container
+2. **Install Python packages**: From `requirements.txt`
+3. **Node.js environment setup**: Docker installs Node in container
+4. **Install npm packages**: From `package.json`
+5. **Build frontend**: Vite builds in dev mode
+6. **Database setup**: PostgreSQL container starts
+7. **Cache setup**: Redis container starts
+8. **Networking**: Docker creates networks
+
+### 🔧 MANUAL (You must do):
+
+1. **Install Docker** on the host machine
+2. **Clone/upload** the IGNY8 application folder
+3. **Create `.env`** file with passwords and secrets
+4. **Build Docker images** (`docker build`)
+5. **Start services** (`docker compose up`)
+6. **Run database migrations** (`docker exec ... migrate`)
+7. **Create admin user** (`docker exec ... createsuperuser`)
+
+---
+
+## Post-Installation
+
+### Set up SSL/HTTPS (Production)
+
+```bash
+# Update Caddyfile with your domain
+# Edit /data/app/igny8/frontend/Caddyfile
+# Caddy will automatically get SSL certificates from Let's Encrypt
+```
+
+### Set up backups
+
+```bash
+# Database backup script
+cat > /data/app/backup.sh << 'EOF'
+#!/bin/bash
+DATE=$(date +%Y%m%d_%H%M%S)
+docker exec igny8_postgres pg_dump -U igny8 igny8_db > /data/backups/pg/igny8_$DATE.sql
+EOF
+
+chmod +x /data/app/backup.sh
+
+# Add to crontab for daily backups
+crontab -e
+# Add: 0 2 * * * /data/app/backup.sh
+```
+
+---
+
+## Updating the Application
+
+```bash
+# Pull latest code
+cd /data/app/igny8
+git pull origin main
+
+# Rebuild images
+docker compose -f docker-compose.app.yml build
+
+# Restart services
+docker compose -f docker-compose.app.yml down
+docker compose -f docker-compose.app.yml up -d
+
+# Run new migrations
+docker exec igny8_backend python manage.py migrate
+```
+
+---
+
+## Troubleshooting
+
+**Containers won't start:**
+```bash
+# Check logs
+docker logs igny8_backend
+docker logs igny8_postgres
+
+# Check container status
+docker ps -a
+```
+
+**Database connection errors:**
+```bash
+# Verify postgres is running
+docker ps | grep postgres
+
+# Check network
+docker network inspect igny8_net
+```
+
+**Port conflicts:**
+```bash
+# Check what's using the port
+sudo lsof -i :8011
+sudo lsof -i :8021
+```
+
+---
+
+## Summary: Fresh Ubuntu to Running IGNY8
+
+**Time estimate: 15-30 minutes**
+
+1. ✅ Install Docker (5 min)
+2. ✅ Install Portainer - optional (2 min)
+3. ✅ Clone/upload IGNY8 app (2 min)
+4. ✅ Create `.env` file (3 min)
+5. ✅ Build Docker images (5-10 min - downloads dependencies)
+6. ✅ Start infrastructure (2 min)
+7. ✅ Start application (2 min)
+8. ✅ Initialize database (3 min)
+9. ✅ Verify and test (2 min)
+
+**Total manual commands: ~20 commands**
+**Everything else: Automated by Docker**
+
+---
+
+## Files You Need to Package
+
+**Minimum files to transfer to new server:**
+```
+igny8/
+├── backend/
+│   ├── igny8_core/          ← Your Django code
+│   ├── manage.py
+│   ├── requirements.txt     ← Python deps list
+│   └── Dockerfile
+├── frontend/
+│   ├── src/                 ← Your React code
+│   ├── public/
+│   ├── package.json         ← Node deps list
+│   └── Dockerfile.dev
+├── docker-compose.app.yml   ← Stack definition
+├── .env                     ← Secrets (create on new server)
+└── README.md
+
+Total size: ~10-20 MB (without node_modules, .venv, dist)
+```
+
+**DO NOT include:**
+- `.venv/`, `node_modules/`, `dist/`, `*.sql`, `logs/`, `celerybeat-schedule`
+
+These are automatically created by Docker during build.
--- a/docs/package-backup/PACKAGING-SETUP-COMPLETE.md
+++ b/docs/package-backup/PACKAGING-SETUP-COMPLETE.md
--- a/docs/package-backup/UNDER-OBSERVATION.md
+++ b/docs/package-backup/UNDER-OBSERVATION.md
@@ -0,0 +1,118 @@
+# UNDER OBSERVATION
+
+## Issue: User Logged Out During Image Prompt Generation (Dec 10, 2025)
+
+### Original Problem
+User performed workflow: auto-cluster → generate ideas → queue to writer → generate content → generate image prompt. During image prompt generation (near completion), user was automatically logged out.
+
+### Investigation Timeline
+
+**Initial Analysis:**
+- Suspected backend container restarts invalidating sessions
+- Docker ps showed all containers up 19+ minutes - NO RESTARTS during incident
+- Backend logs showed: `[IsAuthenticatedAndActive] DENIED: User not authenticated` and `Client error: Authentication credentials were not provided`
+- Token was not being sent with API requests
+
+**Root Cause Identified:**
+The logout was NOT caused by backend issues or container restarts. It was caused by **frontend state corruption during HMR (Hot Module Reload)** triggered by code changes made to fix an unrelated useLocation() error.
+
+**What Actually Happened:**
+
+1. **Commit 5fb3687854d9aadfc5d604470f3712004b23243c** - Already had proper fix for useLocation() error (Suspense outside Routes)
+
+2. **Additional "fixes" applied on Dec 10, 2025:**
+   - Changed `cacheDir: "/tmp/vite-cache"` in vite.config.ts
+   - Moved BrowserRouter above ErrorBoundary in main.tsx
+   - Added `watch.interval: 100` and `fs.strict: false`
+
+3. **These changes triggered:**
+   - Vite cache stored in /tmp got wiped on container operations
+   - Full rebuild with HMR
+   - Component tree restructuring (BrowserRouter position change)
+   - Auth store (Zustand persist) lost state during rapid unmount/remount cycle
+   - Frontend started making API calls WITHOUT Authorization header
+   - Backend correctly rejected unauthenticated requests
+   - Frontend logout() triggered
+
+### Fix Applied
+**Reverted the problematic changes:**
+- Removed `cacheDir: "/tmp/vite-cache"` - let Vite use default node_modules/.vite
+- Restored BrowserRouter position inside ErrorBoundary/ThemeProvider (original structure)
+- Removed `watch.interval` and `fs.strict` additions
+
+**Kept the actual fixes:**
+- Backend: Removed `IsSystemAccountOrDeveloper` from IntegrationSettingsViewSet class-level permissions
+- Backend: Auto-cluster `extra_data` → `debug_info` parameter fix
+- Frontend: Suspense wrapping Routes (from commit 5fb3687) - THIS was the real useLocation() fix
+
+### What to Watch For
+
+**1. useLocation() Error After Container Restarts**
+- **Symptom:** "useLocation() may be used only in the context of a <Router> component"
+- **Where:** Keywords page, other planner/writer module pages (50-60% of pages)
+- **If it happens:** 
+  - Check if Vite cache is stale
+  - Clear node_modules/.vite inside frontend container: `docker compose exec igny8_frontend rm -rf /app/node_modules/.vite`
+  - Restart frontend container
+  - DO NOT change cacheDir or component tree structure
+
+**2. Auth State Loss During Development**
+- **Symptom:** Random logouts during active sessions, "Authentication credentials were not provided"
+- **Triggers:** 
+  - HMR with significant component tree changes
+  - Rapid container restarts during development
+  - Changes to context provider order in main.tsx
+- **Prevention:**
+  - Avoid restructuring main.tsx component tree
+  - Test auth persistence after any main.tsx changes
+  - Monitor browser console for localStorage errors during HMR
+
+**3. Permission Errors for Normal Users**
+- **Symptom:** "You do not have permission to perform this action" for valid users with complete account setup
+- **Check:**
+  - Backend logs for permission class debug output: `[IsAuthenticatedAndActive]`, `[IsViewerOrAbove]`, `[HasTenantAccess]`
+  - Verify user has role='owner' and is_active=True
+  - Ensure viewset doesn't have `IsSystemAccountOrDeveloper` at class level for endpoints normal users need
+
+**4. Celery Task Progress Polling 403 Errors**
+- **Symptom:** Task progress endpoint returns 403 for normal users
+- **Root cause:** ViewSet class-level permissions blocking action-level overrides
+- **Solution:** Ensure IntegrationSettingsViewSet permission_classes doesn't include IsSystemAccountOrDeveloper
+
+### Lessons Learned
+
+1. **Don't layer fixes on top of fixes** - Identify root cause first
+2. **Vite cache location matters** - /tmp gets wiped, breaking HMR state persistence
+3. **Component tree structure is fragile** - Moving BrowserRouter breaks auth rehydration timing
+4. **Container uptime ≠ code stability** - HMR can cause issues without restart
+5. **Permission debugging** - Added logging to permission classes was critical for diagnosis
+6. **The original fix was already correct** - Commit 5fb3687 had it right, additional "improvements" broke it
+
+### Files Modified (Reverted)
+- `frontend/vite.config.ts` - Removed cacheDir and watch config changes
+- `frontend/src/main.tsx` - Restored original component tree structure
+
+### Files Modified (Kept)
+- `backend/igny8_core/modules/system/integration_views.py` - Removed IsSystemAccountOrDeveloper
+- `backend/igny8_core/modules/planner/views.py` - Fixed extra_data → debug_info
+- `backend/igny8_core/api/permissions.py` - Added debug logging (can be removed later)
+
+### Status
+**RESOLVED** - Auth state stable, backend permissions correct, useLocation fix preserved.
+
+**ADDITIONAL FIX (Dec 10, 2025 - Evening):**
+1. **Permission Fix**: Fixed image generation task progress polling 403 errors
+   - Root cause: `IsSystemAccountOrDeveloper` was still in class-level permissions
+   - Solution: Moved to `get_permissions()` method to allow action-level overrides
+   - `task_progress` and `get_image_generation_settings` now accessible to all authenticated users
+   - Save/test operations still restricted to system accounts
+
+2. **System Account Fallback**: Fixed "Image generation settings not found" for normal users
+   - Root cause: IntegrationSettings are account-specific - normal users don't have their own settings
+   - Only super user account (aws-admin) has configured API keys
+   - Solution: Added fallback to system account (aws-admin) settings in `process_image_generation_queue` task
+   - When user's account doesn't have IntegrationSettings, falls back to system account
+   - Allows normal users to use centralized API keys managed by super users
+   - Files modified: `backend/igny8_core/ai/tasks.py`
+
+**Monitor for 48 hours** - Watch for any recurrence of useLocation errors or auth issues after container restarts. Test image generation with normal user accounts (paid-2).