only restart issue and logout issue debugging added
This commit is contained in:
174
CONTAINER-RESTART-DEBUGGING.md
Normal file
174
CONTAINER-RESTART-DEBUGGING.md
Normal file
@@ -0,0 +1,174 @@
|
|||||||
|
# Container Restart and Auto-Logout Debugging Setup
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
Added comprehensive logging to track container restarts and automatic user logouts.
|
||||||
|
|
||||||
|
## Changes Made
|
||||||
|
|
||||||
|
### 1. Container Lifecycle Logging
|
||||||
|
|
||||||
|
#### Backend Container (`backend/container_startup.sh`)
|
||||||
|
- Logs container startup time, hostname, and PID
|
||||||
|
- Detects and logs container restarts (by checking for previous PID file)
|
||||||
|
- Logs environment configuration (Python version, Django settings, DB host)
|
||||||
|
- Warns when restarts are detected and suggests checking Docker logs for SIGTERM signals
|
||||||
|
- Integrated into Dockerfile as ENTRYPOINT
|
||||||
|
|
||||||
|
#### Frontend Container (`frontend/container_startup.sh`)
|
||||||
|
- Logs container startup time, hostname, and PID
|
||||||
|
- Detects and logs container restarts
|
||||||
|
- Logs Node/NPM versions and Vite configuration
|
||||||
|
- Checks for git directory presence and warns about file watching
|
||||||
|
- Shows last git commit when detected
|
||||||
|
- Integrated into Dockerfile.dev as ENTRYPOINT
|
||||||
|
|
||||||
|
### 2. Vite File Watching Fix (`frontend/vite.config.ts`)
|
||||||
|
|
||||||
|
**ROOT CAUSE IDENTIFIED:** `usePolling: true` was watching ALL files including `.git` directory, causing container restarts on git commits.
|
||||||
|
|
||||||
|
**Fix Applied:**
|
||||||
|
```typescript
|
||||||
|
watch: {
|
||||||
|
usePolling: true,
|
||||||
|
ignored: [
|
||||||
|
'**/node_modules/**',
|
||||||
|
'**/.git/**', // CRITICAL: Ignore git directory
|
||||||
|
'**/dist/**',
|
||||||
|
'**/build/**',
|
||||||
|
'**/.vscode/**',
|
||||||
|
'**/.idea/**',
|
||||||
|
],
|
||||||
|
interval: 1000, // Poll every 1 second instead of default
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Auto-Logout Logging (`backend/igny8_core/auth/middleware.py`)
|
||||||
|
|
||||||
|
Added detailed logging for all automatic logout scenarios:
|
||||||
|
|
||||||
|
#### Session Contamination - Account ID Mismatch
|
||||||
|
```python
|
||||||
|
logger.warning(
|
||||||
|
f"[AUTO-LOGOUT] Session contamination: account_id mismatch. "
|
||||||
|
f"Session={stored_account_id}, Current={request.account.id}, "
|
||||||
|
f"User={request.user.id}, Path={request.path}, IP={request.META.get('REMOTE_ADDR')}"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Session Contamination - User ID Mismatch
|
||||||
|
```python
|
||||||
|
logger.warning(
|
||||||
|
f"[AUTO-LOGOUT] Session contamination: user_id mismatch. "
|
||||||
|
f"Session={stored_user_id}, Current={request.user.id}, "
|
||||||
|
f"Account={request.account.id if request.account else None}, "
|
||||||
|
f"Path={request.path}, IP={request.META.get('REMOTE_ADDR')}"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Account/Plan Validation Failures
|
||||||
|
```python
|
||||||
|
logger.warning(
|
||||||
|
f"[AUTO-LOGOUT] Account/plan validation failed: {error}. "
|
||||||
|
f"User={request.user.id}, Account={getattr(request, 'account', None)}, "
|
||||||
|
f"Path={request.path}, IP={request.META.get('REMOTE_ADDR')}"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Logging Configuration (`backend/igny8_core/settings.py`)
|
||||||
|
|
||||||
|
Added two new loggers:
|
||||||
|
- `auth.middleware` - Captures all authentication and auto-logout events
|
||||||
|
- `container.lifecycle` - Captures container startup/restart events
|
||||||
|
|
||||||
|
Both log to console (captured by Docker logs).
|
||||||
|
|
||||||
|
## How to Use
|
||||||
|
|
||||||
|
### Viewing Container Restart Logs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check backend container logs for restart events
|
||||||
|
docker logs igny8_backend 2>&1 | grep "CONTAINER-STARTUP\|RESTART"
|
||||||
|
|
||||||
|
# Check frontend container logs for restart events
|
||||||
|
docker logs igny8_frontend 2>&1 | grep "CONTAINER-STARTUP\|RESTART"
|
||||||
|
|
||||||
|
# Monitor in real-time
|
||||||
|
docker logs -f igny8_backend
|
||||||
|
```
|
||||||
|
|
||||||
|
### Viewing Auto-Logout Logs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check for automatic logout events
|
||||||
|
docker logs igny8_backend 2>&1 | grep "AUTO-LOGOUT"
|
||||||
|
|
||||||
|
# Filter by specific logout reason
|
||||||
|
docker logs igny8_backend 2>&1 | grep "AUTO-LOGOUT.*contamination"
|
||||||
|
docker logs igny8_backend 2>&1 | grep "AUTO-LOGOUT.*validation failed"
|
||||||
|
|
||||||
|
# See recent logout events with context
|
||||||
|
docker logs --since 1h igny8_backend 2>&1 | grep -A2 -B2 "AUTO-LOGOUT"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Correlating Events
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# See both container restarts and logouts together
|
||||||
|
docker logs --since 2h igny8_backend 2>&1 | grep -E "CONTAINER-STARTUP|AUTO-LOGOUT|SIGTERM"
|
||||||
|
|
||||||
|
# Check if git commits correlate with restarts
|
||||||
|
git log --since="2 hours ago" --format="%ai %s" && \
|
||||||
|
docker logs --since 2h igny8_frontend 2>&1 | grep "CONTAINER-STARTUP"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Next Steps to Deploy
|
||||||
|
|
||||||
|
1. **Rebuild Docker images:**
|
||||||
|
```bash
|
||||||
|
cd /data/app/igny8/backend
|
||||||
|
docker build -t igny8-backend:latest -f Dockerfile .
|
||||||
|
|
||||||
|
cd /data/app/igny8/frontend
|
||||||
|
docker build -t igny8-frontend-dev:latest -f Dockerfile.dev .
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Restart containers:**
|
||||||
|
```bash
|
||||||
|
cd /data/app/igny8
|
||||||
|
docker compose -f docker-compose.app.yml down
|
||||||
|
docker compose -f docker-compose.app.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Verify logging is working:**
|
||||||
|
```bash
|
||||||
|
docker logs igny8_backend 2>&1 | head -30
|
||||||
|
docker logs igny8_frontend 2>&1 | head -30
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Test git commit trigger (should NOT restart now):**
|
||||||
|
```bash
|
||||||
|
cd /data/app/igny8
|
||||||
|
echo "test" >> README.md
|
||||||
|
git add README.md
|
||||||
|
git commit -m "test commit"
|
||||||
|
|
||||||
|
# Wait 5 seconds and check - containers should NOT restart
|
||||||
|
sleep 5
|
||||||
|
docker ps --filter "name=igny8" --format "{{.Names}}: {{.Status}}"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Expected Outcomes
|
||||||
|
|
||||||
|
1. **Git commits should NO LONGER trigger container restarts** because `.git` is now ignored by Vite's file watcher
|
||||||
|
2. **Every container restart will be logged** with timestamp and reason
|
||||||
|
3. **Every automatic logout will be logged** with user ID, account ID, reason, path, and IP address
|
||||||
|
4. **You can correlate restarts with git operations** to verify the fix is working
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
If containers still restart after git commits:
|
||||||
|
1. Check if the new images were built and deployed
|
||||||
|
2. Verify the vite.config.ts changes are present in the running container
|
||||||
|
3. Check Docker logs to see what triggered the restart
|
||||||
|
4. Look for HMR messages in frontend logs mentioning `.git` files
|
||||||
@@ -22,6 +22,10 @@ RUN pip install --upgrade pip \
|
|||||||
# Copy full project
|
# Copy full project
|
||||||
COPY . /app/
|
COPY . /app/
|
||||||
|
|
||||||
|
# Copy startup script
|
||||||
|
COPY container_startup.sh /app/
|
||||||
|
RUN chmod +x /app/container_startup.sh
|
||||||
|
|
||||||
# Collect static files for WhiteNoise (skip during build if DB not available)
|
# Collect static files for WhiteNoise (skip during build if DB not available)
|
||||||
# Will be run during container startup if needed
|
# Will be run during container startup if needed
|
||||||
RUN python manage.py collectstatic --noinput || echo "Skipping collectstatic during build"
|
RUN python manage.py collectstatic --noinput || echo "Skipping collectstatic during build"
|
||||||
@@ -32,5 +36,7 @@ ENV DJANGO_SETTINGS_MODULE=igny8_core.settings
|
|||||||
# Expose port for Gunicorn (matches Portainer docker-compose config)
|
# Expose port for Gunicorn (matches Portainer docker-compose config)
|
||||||
EXPOSE 8010
|
EXPOSE 8010
|
||||||
|
|
||||||
|
# Use startup script as entrypoint to log container lifecycle
|
||||||
# Start using Gunicorn (matches Portainer docker-compose config)
|
# Start using Gunicorn (matches Portainer docker-compose config)
|
||||||
|
ENTRYPOINT ["/app/container_startup.sh"]
|
||||||
CMD ["gunicorn", "igny8_core.wsgi:application", "--bind", "0.0.0.0:8010"]
|
CMD ["gunicorn", "igny8_core.wsgi:application", "--bind", "0.0.0.0:8010"]
|
||||||
|
|||||||
47
backend/container_startup.sh
Normal file
47
backend/container_startup.sh
Normal file
@@ -0,0 +1,47 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Container Startup Logger
|
||||||
|
# Logs container lifecycle events for debugging restarts
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "=========================================="
|
||||||
|
echo "[CONTAINER-STARTUP] $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
echo "Container: igny8_backend"
|
||||||
|
echo "Hostname: $(hostname)"
|
||||||
|
echo "PID: $$"
|
||||||
|
echo "=========================================="
|
||||||
|
|
||||||
|
# Log environment info
|
||||||
|
echo "[INFO] Python version: $(python --version 2>&1)"
|
||||||
|
echo "[INFO] Django settings: ${DJANGO_SETTINGS_MODULE:-igny8_core.settings}"
|
||||||
|
echo "[INFO] Debug mode: ${DEBUG:-False}"
|
||||||
|
echo "[INFO] Database host: ${DB_HOST:-not set}"
|
||||||
|
|
||||||
|
# Check if this is a restart (look for previous process artifacts)
|
||||||
|
if [ -f /tmp/container_pid ]; then
|
||||||
|
PREV_PID=$(cat /tmp/container_pid)
|
||||||
|
echo "[WARNING] Previous container PID found: $PREV_PID"
|
||||||
|
echo "[WARNING] This appears to be a RESTART event"
|
||||||
|
echo "[WARNING] Check Docker logs for SIGTERM/SIGKILL signals"
|
||||||
|
else
|
||||||
|
echo "[INFO] First startup (no previous PID file found)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Save current PID
|
||||||
|
echo $$ > /tmp/container_pid
|
||||||
|
|
||||||
|
# Run database migrations (will skip if up to date)
|
||||||
|
echo "[INFO] Running database migrations..."
|
||||||
|
python manage.py migrate --noinput || echo "[WARNING] Migration failed or skipped"
|
||||||
|
|
||||||
|
# Collect static files (skip if already done)
|
||||||
|
echo "[INFO] Collecting static files..."
|
||||||
|
python manage.py collectstatic --noinput || echo "[WARNING] Collectstatic failed or skipped"
|
||||||
|
|
||||||
|
echo "=========================================="
|
||||||
|
echo "[CONTAINER-STARTUP] Initialization complete"
|
||||||
|
echo "[CONTAINER-STARTUP] Starting Gunicorn..."
|
||||||
|
echo "=========================================="
|
||||||
|
|
||||||
|
# Execute the CMD passed to the script (Gunicorn command)
|
||||||
|
exec "$@"
|
||||||
@@ -2,11 +2,14 @@
|
|||||||
Multi-Account Middleware
|
Multi-Account Middleware
|
||||||
Extracts account from JWT token and injects into request context
|
Extracts account from JWT token and injects into request context
|
||||||
"""
|
"""
|
||||||
|
import logging
|
||||||
from django.utils.deprecation import MiddlewareMixin
|
from django.utils.deprecation import MiddlewareMixin
|
||||||
from django.http import JsonResponse
|
from django.http import JsonResponse
|
||||||
from django.contrib.auth import logout
|
from django.contrib.auth import logout
|
||||||
from rest_framework import status
|
from rest_framework import status
|
||||||
|
|
||||||
|
logger = logging.getLogger('auth.middleware')
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import jwt
|
import jwt
|
||||||
JWT_AVAILABLE = True
|
JWT_AVAILABLE = True
|
||||||
@@ -54,6 +57,11 @@ class AccountContextMiddleware(MiddlewareMixin):
|
|||||||
stored_user_id = request.session.get('_user_id')
|
stored_user_id = request.session.get('_user_id')
|
||||||
if stored_account_id and stored_account_id != request.account.id:
|
if stored_account_id and stored_account_id != request.account.id:
|
||||||
# Session contamination detected - force logout
|
# Session contamination detected - force logout
|
||||||
|
logger.warning(
|
||||||
|
f"[AUTO-LOGOUT] Session contamination: account_id mismatch. "
|
||||||
|
f"Session={stored_account_id}, Current={request.account.id}, "
|
||||||
|
f"User={request.user.id}, Path={request.path}, IP={request.META.get('REMOTE_ADDR')}"
|
||||||
|
)
|
||||||
logout(request)
|
logout(request)
|
||||||
return JsonResponse(
|
return JsonResponse(
|
||||||
{'success': False, 'error': 'Session integrity violation detected. Please login again.'},
|
{'success': False, 'error': 'Session integrity violation detected. Please login again.'},
|
||||||
@@ -61,6 +69,12 @@ class AccountContextMiddleware(MiddlewareMixin):
|
|||||||
)
|
)
|
||||||
if stored_user_id and stored_user_id != request.user.id:
|
if stored_user_id and stored_user_id != request.user.id:
|
||||||
# Session contamination detected - force logout
|
# Session contamination detected - force logout
|
||||||
|
logger.warning(
|
||||||
|
f"[AUTO-LOGOUT] Session contamination: user_id mismatch. "
|
||||||
|
f"Session={stored_user_id}, Current={request.user.id}, "
|
||||||
|
f"Account={request.account.id if request.account else None}, "
|
||||||
|
f"Path={request.path}, IP={request.META.get('REMOTE_ADDR')}"
|
||||||
|
)
|
||||||
logout(request)
|
logout(request)
|
||||||
return JsonResponse(
|
return JsonResponse(
|
||||||
{'success': False, 'error': 'Session integrity violation detected. Please login again.'},
|
{'success': False, 'error': 'Session integrity violation detected. Please login again.'},
|
||||||
@@ -173,9 +187,14 @@ class AccountContextMiddleware(MiddlewareMixin):
|
|||||||
"""Logout session users (if any) and return a consistent JSON error."""
|
"""Logout session users (if any) and return a consistent JSON error."""
|
||||||
try:
|
try:
|
||||||
if hasattr(request, 'user') and request.user and request.user.is_authenticated:
|
if hasattr(request, 'user') and request.user and request.user.is_authenticated:
|
||||||
|
logger.warning(
|
||||||
|
f"[AUTO-LOGOUT] Account/plan validation failed: {error}. "
|
||||||
|
f"User={request.user.id}, Account={getattr(request, 'account', None)}, "
|
||||||
|
f"Path={request.path}, IP={request.META.get('REMOTE_ADDR')}"
|
||||||
|
)
|
||||||
logout(request)
|
logout(request)
|
||||||
except Exception:
|
except Exception as e:
|
||||||
pass
|
logger.error(f"[AUTO-LOGOUT] Error during logout: {e}")
|
||||||
|
|
||||||
return JsonResponse(
|
return JsonResponse(
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -604,6 +604,16 @@ LOGGING = {
|
|||||||
'level': 'INFO',
|
'level': 'INFO',
|
||||||
'propagate': False,
|
'propagate': False,
|
||||||
},
|
},
|
||||||
|
'auth.middleware': {
|
||||||
|
'handlers': ['console'],
|
||||||
|
'level': 'INFO',
|
||||||
|
'propagate': False,
|
||||||
|
},
|
||||||
|
'container.lifecycle': {
|
||||||
|
'handlers': ['console'],
|
||||||
|
'level': 'INFO',
|
||||||
|
'propagate': False,
|
||||||
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -12,9 +12,15 @@ RUN npm install
|
|||||||
# Copy source code (will be mounted as volume, but needed for initial setup)
|
# Copy source code (will be mounted as volume, but needed for initial setup)
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
|
# Copy startup script
|
||||||
|
COPY container_startup.sh /app/
|
||||||
|
RUN chmod +x /app/container_startup.sh
|
||||||
|
|
||||||
# Expose Vite dev server port
|
# Expose Vite dev server port
|
||||||
EXPOSE 5173
|
EXPOSE 5173
|
||||||
|
|
||||||
|
# Use startup script as entrypoint to log container lifecycle
|
||||||
# Start Vite dev server with host binding for Docker
|
# Start Vite dev server with host binding for Docker
|
||||||
|
ENTRYPOINT ["/app/container_startup.sh"]
|
||||||
CMD ["npm", "run", "dev", "--", "--host", "0.0.0.0", "--port", "5173"]
|
CMD ["npm", "run", "dev", "--", "--host", "0.0.0.0", "--port", "5173"]
|
||||||
|
|
||||||
|
|||||||
44
frontend/container_startup.sh
Normal file
44
frontend/container_startup.sh
Normal file
@@ -0,0 +1,44 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# Frontend Container Startup Logger
|
||||||
|
# Logs container lifecycle events for debugging restarts
|
||||||
|
|
||||||
|
echo "=========================================="
|
||||||
|
echo "[CONTAINER-STARTUP] $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
echo "Container: igny8_frontend"
|
||||||
|
echo "Hostname: $(hostname)"
|
||||||
|
echo "PID: $$"
|
||||||
|
echo "=========================================="
|
||||||
|
|
||||||
|
# Log environment info
|
||||||
|
echo "[INFO] Node version: $(node --version 2>&1)"
|
||||||
|
echo "[INFO] NPM version: $(npm --version 2>&1)"
|
||||||
|
echo "[INFO] Vite backend URL: ${VITE_BACKEND_URL:-not set}"
|
||||||
|
echo "[INFO] Working directory: $(pwd)"
|
||||||
|
|
||||||
|
# Check if this is a restart
|
||||||
|
if [ -f /tmp/container_pid ]; then
|
||||||
|
PREV_PID=$(cat /tmp/container_pid)
|
||||||
|
echo "[WARNING] Previous container PID found: $PREV_PID"
|
||||||
|
echo "[WARNING] This appears to be a RESTART event"
|
||||||
|
echo "[WARNING] Check Docker logs for SIGTERM/SIGKILL signals"
|
||||||
|
else
|
||||||
|
echo "[INFO] First startup (no previous PID file found)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Save current PID
|
||||||
|
echo $$ > /tmp/container_pid
|
||||||
|
|
||||||
|
# Check for git directory changes (common restart trigger)
|
||||||
|
if [ -d "/app/.git" ]; then
|
||||||
|
echo "[INFO] Git directory detected in /app/.git"
|
||||||
|
echo "[WARNING] Git operations may trigger container restarts due to Vite file watching"
|
||||||
|
echo "[INFO] Last git commit: $(cd /app && git log -1 --format='%h %s' 2>/dev/null || echo 'N/A')"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=========================================="
|
||||||
|
echo "[CONTAINER-STARTUP] Initialization complete"
|
||||||
|
echo "[CONTAINER-STARTUP] Starting Vite dev server..."
|
||||||
|
echo "=========================================="
|
||||||
|
|
||||||
|
# Execute the CMD passed to the script
|
||||||
|
exec "$@"
|
||||||
@@ -190,6 +190,15 @@ export default defineConfig(({ mode, command }) => {
|
|||||||
allowedHosts: true,
|
allowedHosts: true,
|
||||||
watch: {
|
watch: {
|
||||||
usePolling: true, // Needed for file watching in Docker
|
usePolling: true, // Needed for file watching in Docker
|
||||||
|
ignored: [
|
||||||
|
'**/node_modules/**',
|
||||||
|
'**/.git/**', // CRITICAL: Ignore git directory to prevent restart on commits
|
||||||
|
'**/dist/**',
|
||||||
|
'**/build/**',
|
||||||
|
'**/.vscode/**',
|
||||||
|
'**/.idea/**',
|
||||||
|
],
|
||||||
|
interval: 1000, // Poll every 1 second (default is 100ms)
|
||||||
},
|
},
|
||||||
hmr: {
|
hmr: {
|
||||||
// Behind reverse proxy - explicitly set host and port to public domain
|
// Behind reverse proxy - explicitly set host and port to public domain
|
||||||
|
|||||||
Reference in New Issue
Block a user