diff --git a/unified-api/API-IMPLEMENTATION-PLAN-SECTION1.md b/unified-api/API-IMPLEMENTATION-PLAN-SECTION1.md deleted file mode 100644 index 35018e52..00000000 --- a/unified-api/API-IMPLEMENTATION-PLAN-SECTION1.md +++ /dev/null @@ -1,1102 +0,0 @@ -# API Implementation Plan - Section 1: Unified Response Format - -**Date:** 2025-01-XX -**Status:** Planning -**Priority:** High -**Related Document:** `API-ENDPOINTS-ANALYSIS.md` - ---- - -## Executive Summary - -This document outlines the implementation plan for **Section 1: Define a Unified Response Format** across the IGNY8 API layer. The goal is to ensure every API endpoint in all modules responds with a consistent JSON structure, improving API predictability, frontend integration, and developer experience. - -**Target Format:** -```json -{ - "success": true | false, - "data": {...} | [...], - "message": "Human-readable success/failure message", - "error": "Top-level error message (if any)", - "errors": { "field": ["Error list"] } (optional, for validation failures) -} -``` - ---- - -## Table of Contents - -1. [Current State Analysis](#current-state-analysis) -2. [Implementation Tasks](#implementation-tasks) -3. [Task 1: Define Global Response Wrapper Functions](#task-1-define-global-response-wrapper-functions) -4. [Task 2: Refactor All API Views and ViewSets](#task-2-refactor-all-api-views-and-viewsets) -5. [Task 3: Patch DRF Auto-generated Responses (Optional)](#task-3-patch-drf-auto-generated-responses-optional) -6. [Task 4: Update Exception Handlers](#task-4-update-exception-handlers) -7. [Task 5: Validation via Postman/Frontend Logs](#task-5-validation-via-postmanfrontend-logs) -8. [Task 6: Documentation Update](#task-6-documentation-update) -9. [Task 7: Changelog Entry](#task-7-changelog-entry) -10. [Testing Strategy](#testing-strategy) -11. [Rollout Plan](#rollout-plan) -12. [Success Criteria](#success-criteria) - ---- - -## Current State Analysis - -### Current Response Format Issues - -Based on `API-ENDPOINTS-ANALYSIS.md`, the following inconsistencies exist: - -1. **Mixed Response Formats:** - - Some endpoints return `{ success: true, data: ... }` - - Some endpoints return DRF standard format `{ count, next, previous, results: [...] }` - - Some endpoints return custom formats - - Error responses vary: `{ error: "..." }` vs `{ success: false, message: "..." }` - -2. **Affected Modules:** - - **Auth Module**: 15+ endpoints (register, login, user management, etc.) - - **Planner Module**: 20+ endpoints (keywords, clusters, ideas) - - **Writer Module**: 20+ endpoints (tasks, content, images) - - **System Module**: 30+ endpoints (settings, prompts, integrations) - - **Billing Module**: 10+ endpoints (credits, transactions, usage) - -3. **Base Classes to Update:** - - `AccountModelViewSet` - Used by many ViewSets - - `SiteSectorModelViewSet` - Used by Planner/Writer modules - - Custom ViewSets with overridden `create`, `update`, `destroy` methods - ---- - -## Implementation Tasks - -### Overview - -| Task ID | Task Name | Priority | Estimated Effort | Dependencies | -|---------|-----------|----------|------------------|--------------| -| 1.1 | Create response wrapper functions | High | 2 hours | None | -| 1.2 | Update exception handlers | High | 3 hours | 1.1 | -| 2.1 | Audit all ViewSets | Medium | 4 hours | None | -| 2.2 | Refactor Auth module ViewSets | High | 6 hours | 1.1, 1.2 | -| 2.3 | Refactor Planner module ViewSets | High | 8 hours | 1.1, 1.2 | -| 2.4 | Refactor Writer module ViewSets | High | 8 hours | 1.1, 1.2 | -| 2.5 | Refactor System module ViewSets | High | 10 hours | 1.1, 1.2 | -| 2.6 | Refactor Billing module ViewSets | High | 4 hours | 1.1, 1.2 | -| 3.1 | Patch DRF auto-generated responses (optional) | Low | 4 hours | 1.1 | -| 5.1 | Manual testing | High | 8 hours | 2.2-2.6 | -| 6.1 | Update documentation | Medium | 4 hours | 5.1 | - -**Total Estimated Effort:** ~61 hours - ---- - -## Task 1: Define Global Response Wrapper Functions - -### Goal -Create reusable helper functions that standardize all API responses. - -### Implementation Steps - -#### Step 1.1: Create Response Utility Module - -**File:** `backend/igny8_core/api/response.py` - -**Implementation:** -```python -""" -Unified API Response Format Utilities - -This module provides helper functions to ensure all API endpoints -return a consistent response format. -""" - -from rest_framework.response import Response -from rest_framework import status - - -def success_response(data=None, message=None, status_code=status.HTTP_200_OK): - """ - Create a standardized success response. - - Args: - data: Response data (dict, list, or any serializable object) - message: Optional human-readable success message - status_code: HTTP status code (default: 200) - - Returns: - Response: DRF Response object with unified format - - Example: - return success_response( - data={"id": 1, "name": "Example"}, - message="Resource created successfully" - ) - """ - response_data = { - "success": True, - "data": data, - } - - if message: - response_data["message"] = message - - return Response(response_data, status=status_code) - - -def error_response( - error, - errors=None, - status_code=status.HTTP_400_BAD_REQUEST, - message=None -): - """ - Create a standardized error response. - - Args: - error: Top-level error message (string) - errors: Optional field-specific validation errors (dict) - status_code: HTTP status code (default: 400) - message: Optional additional message (deprecated, use error) - - Returns: - Response: DRF Response object with unified format - - Example: - return error_response( - error="Validation failed", - errors={"email": ["Invalid email format"]}, - status_code=status.HTTP_400_BAD_REQUEST - ) - """ - response_data = { - "success": False, - "error": error, - } - - if errors: - response_data["errors"] = errors - - # Backward compatibility: if message is provided, use it as error - if message and not error: - response_data["error"] = message - - return Response(response_data, status=status_code) - - -def paginated_response(paginated_data, message=None): - """ - Create a standardized paginated response. - - This wraps DRF's pagination response to include success flag - and optional message while preserving pagination metadata. - - Args: - paginated_data: DRF paginated response data (dict with count, next, previous, results) - message: Optional human-readable message - - Returns: - Response: DRF Response object with unified format - - Example: - paginator = CustomPageNumberPagination() - page = paginator.paginate_queryset(queryset, request) - serializer = MySerializer(page, many=True) - paginated_data = paginator.get_paginated_response(serializer.data).data - return paginated_response(paginated_data, message="Keywords retrieved successfully") - """ - response_data = { - "success": True, - **paginated_data # Unpack count, next, previous, results - } - - if message: - response_data["message"] = message - - return Response(response_data) -``` - -#### Step 1.2: Update `__init__.py` for Easy Import - -**File:** `backend/igny8_core/api/__init__.py` - -**Add:** -```python -from .response import success_response, error_response, paginated_response - -__all__ = ['success_response', 'error_response', 'paginated_response'] -``` - -#### Step 1.3: Create Unit Tests - -**File:** `backend/igny8_core/api/tests/test_response.py` - -**Test Cases:** -- Test `success_response` with data only -- Test `success_response` with data and message -- Test `success_response` with custom status code (201, 204) -- Test `error_response` with error only -- Test `error_response` with error and errors dict -- Test `error_response` with custom status codes (400, 403, 404, 500) -- Test `paginated_response` with standard pagination data -- Test `paginated_response` with message - -**Estimated Time:** 2 hours - ---- - -## Task 2: Refactor All API Views and ViewSets - -### Goal -Update all ViewSets to use the new response wrapper functions. - -### Implementation Strategy - -#### Step 2.1: Audit All ViewSets - -**Action Items:** -1. List all ViewSets in each module -2. Identify custom `create`, `update`, `destroy`, `list`, `retrieve` methods -3. Identify custom action methods (`@action` decorator) -4. Document current response formats -5. Create refactoring checklist - -**Files to Audit:** - -**Auth Module:** -- `backend/auth/api/views.py` - UsersViewSet, AccountsViewSet, SiteViewSet, etc. -- `backend/auth/api/viewsets.py` - Custom authentication views - -**Planner Module:** -- `backend/planner/api/viewsets.py` - KeywordViewSet, ClusterViewSet, ContentIdeasViewSet - -**Writer Module:** -- `backend/writer/api/viewsets.py` - TasksViewSet, ContentViewSet, ImagesViewSet - -**System Module:** -- `backend/system/api/viewsets.py` - AIPromptViewSet, AuthorProfileViewSet, StrategyViewSet -- `backend/system/api/viewsets.py` - IntegrationSettingsViewSet, SystemSettingsViewSet, etc. - -**Billing Module:** -- `backend/billing/api/viewsets.py` - CreditBalanceViewSet, CreditUsageViewSet, CreditTransactionViewSet - -**Estimated Time:** 4 hours - -#### Step 2.2: Refactor Auth Module ViewSets - -**Priority Endpoints:** -1. `POST /api/v1/auth/register/` - Registration endpoint -2. `POST /api/v1/auth/login/` - Login endpoint -3. `POST /api/v1/auth/change-password/` - Password change -4. `GET /api/v1/auth/me/` - Current user info -5. All ViewSet CRUD operations -6. Custom actions (invite, activate, etc.) - -**Example Refactoring:** - -**Before:** -```python -class RegisterView(APIView): - def post(self, request): - # ... validation logic ... - return Response({ - "success": True, - "message": "Registration successful", - "user": user_data - }, status=201) -``` - -**After:** -```python -from igny8_core.api.response import success_response - -class RegisterView(APIView): - def post(self, request): - # ... validation logic ... - return success_response( - data={"user": user_data}, - message="Registration successful", - status_code=status.HTTP_201_CREATED - ) -``` - -**Estimated Time:** 6 hours - -#### Step 2.3: Refactor Planner Module ViewSets - -**Priority Endpoints:** -1. KeywordViewSet - All CRUD + custom actions (bulk_delete, auto_cluster, etc.) -2. ClusterViewSet - All CRUD + custom actions (auto_generate_ideas) -3. ContentIdeasViewSet - All CRUD + custom actions (bulk_queue_to_writer) - -**Example Refactoring:** - -**Before:** -```python -class KeywordViewSet(SiteSectorModelViewSet): - @action(detail=False, methods=['post']) - def bulk_delete(self, request): - ids = request.data.get('ids', []) - deleted_count = Keyword.objects.filter(id__in=ids).delete()[0] - return Response({"deleted_count": deleted_count}) -``` - -**After:** -```python -from igny8_core.api.response import success_response, error_response - -class KeywordViewSet(SiteSectorModelViewSet): - @action(detail=False, methods=['post']) - def bulk_delete(self, request): - ids = request.data.get('ids', []) - if not ids: - return error_response( - error="No IDs provided", - status_code=status.HTTP_400_BAD_REQUEST - ) - deleted_count = Keyword.objects.filter(id__in=ids).delete()[0] - return success_response( - data={"deleted_count": deleted_count}, - message=f"Successfully deleted {deleted_count} keywords" - ) -``` - -**Estimated Time:** 8 hours - -#### Step 2.4: Refactor Writer Module ViewSets - -**Priority Endpoints:** -1. TasksViewSet - All CRUD + custom actions (auto_generate_content, bulk_update) -2. ContentViewSet - All CRUD + custom actions (generate_image_prompts) -3. ImagesViewSet - All CRUD + custom actions (generate_images, bulk_update) - -**Estimated Time:** 8 hours - -#### Step 2.5: Refactor System Module ViewSets - -**Priority Endpoints:** -1. AIPromptViewSet - All CRUD + custom actions (save, reset, by_type) -2. IntegrationSettingsViewSet - Custom URL patterns (save, test, generate) -3. All Settings ViewSets (SystemSettings, AccountSettings, UserSettings, etc.) - -**Special Considerations:** -- IntegrationSettingsViewSet uses custom URL patterns (not standard ViewSet routes) -- Task progress endpoint returns Celery task status (may need special handling) - -**Estimated Time:** 10 hours - -#### Step 2.6: Refactor Billing Module ViewSets - -**Priority Endpoints:** -1. CreditBalanceViewSet - Custom action (balance) -2. CreditUsageViewSet - Read-only + custom actions (summary, limits) -3. CreditTransactionViewSet - Read-only - -**Estimated Time:** 4 hours - -### Refactoring Checklist Template - -For each ViewSet, check: - -- [ ] Import response wrapper functions -- [ ] Update `create` method (if overridden) -- [ ] Update `update` method (if overridden) -- [ ] Update `destroy` method (if overridden) -- [ ] Update `list` method (if overridden) - Use `paginated_response` for paginated lists -- [ ] Update `retrieve` method (if overridden) -- [ ] Update all custom `@action` methods -- [ ] Update error handling to use `error_response` -- [ ] Test each endpoint manually -- [ ] Verify response format matches specification - ---- - -## Task 3: Patch DRF Auto-generated Responses (Optional) - -### Goal -Ensure DRF's auto-generated responses (from ModelViewSet default methods) also follow the unified format. - -### Implementation Steps - -#### Step 3.1: Create Base API View Class - -**File:** `backend/igny8_core/api/views.py` - -**Implementation:** -```python -""" -Base API View Classes with Unified Response Format -""" - -from rest_framework.views import APIView -from rest_framework.viewsets import ModelViewSet -from rest_framework.response import Response -from rest_framework import status -from .response import success_response, error_response, paginated_response - - -class CustomAPIView(APIView): - """ - Base APIView that automatically wraps responses in unified format. - - Override finalize_response to ensure all responses follow the format. - """ - - def finalize_response(self, request, response, *args, **kwargs): - """ - Override to wrap DRF responses in unified format. - - This is called for all responses, including exceptions. - """ - # If response is already in unified format, return as-is - if isinstance(response.data, dict) and 'success' in response.data: - return super().finalize_response(request, response, *args, **kwargs) - - # Wrap standard DRF responses - if response.status_code < 400: - # Success response - if isinstance(response.data, dict): - # Check if it's paginated response - if 'results' in response.data and 'count' in response.data: - # Paginated response - return super().finalize_response( - request, - paginated_response(response.data), - *args, - **kwargs - ) - else: - # Regular success response - return super().finalize_response( - request, - success_response(response.data), - *args, - **kwargs - ) - else: - # List or other data - return super().finalize_response( - request, - success_response(response.data), - *args, - **kwargs - ) - else: - # Error response - error_msg = str(response.data) if response.data else "An error occurred" - errors = response.data if isinstance(response.data, dict) else None - return super().finalize_response( - request, - error_response( - error=error_msg, - errors=errors, - status_code=response.status_code - ), - *args, - **kwargs - ) - - -class CustomModelViewSet(ModelViewSet): - """ - Base ModelViewSet that automatically wraps responses in unified format. - - Inherit from this instead of ModelViewSet to get automatic response formatting. - """ - - def finalize_response(self, request, response, *args, **kwargs): - """ - Override to wrap DRF responses in unified format. - """ - # Same logic as CustomAPIView - if isinstance(response.data, dict) and 'success' in response.data: - return super().finalize_response(request, response, *args, **kwargs) - - if response.status_code < 400: - if isinstance(response.data, dict) and 'results' in response.data: - return super().finalize_response( - request, - paginated_response(response.data), - *args, - **kwargs - ) - else: - return super().finalize_response( - request, - success_response(response.data), - *args, - **kwargs - ) - else: - error_msg = str(response.data) if response.data else "An error occurred" - errors = response.data if isinstance(response.data, dict) else None - return super().finalize_response( - request, - error_response( - error=error_msg, - errors=errors, - status_code=response.status_code - ), - *args, - **kwargs - ) -``` - -#### Step 3.2: Update Base ViewSets (Optional) - -**Decision Point:** -- **Option A:** Update `AccountModelViewSet` and `SiteSectorModelViewSet` to inherit from `CustomModelViewSet` -- **Option B:** Keep current inheritance, manually wrap responses in each ViewSet - -**Recommendation:** Option B (manual wrapping) for more control and explicit behavior. - -**Estimated Time:** 4 hours (if implemented) - ---- - -## Task 4: Update Exception Handlers - -### Goal -Ensure all exceptions (400, 403, 404, 500) are wrapped in the unified error format. - -### Implementation Steps - -#### Step 4.1: Create Custom Exception Handler - -**File:** `backend/igny8_core/api/exception_handlers.py` - -**Implementation:** -```python -""" -Custom Exception Handlers for Unified Response Format -""" - -from rest_framework.views import exception_handler as drf_exception_handler -from rest_framework import status -from .response import error_response - - -def custom_exception_handler(exc, context): - """ - Custom exception handler that wraps all exceptions in unified format. - - This handler is called for all exceptions raised in API views. - It ensures all error responses follow the unified format. - - Args: - exc: The exception instance - context: Dictionary containing request, view, and args - - Returns: - Response: Error response in unified format, or None to use default - """ - # Call DRF's default exception handler first - response = drf_exception_handler(exc, context) - - if response is not None: - # Extract error message and details - error_message = "An error occurred" - errors = None - status_code = response.status_code - - # Handle different exception types - if hasattr(exc, 'detail'): - # DRF exceptions have 'detail' attribute - if isinstance(exc.detail, dict): - # Validation errors - use as errors dict - errors = exc.detail - # Extract first error message as top-level error - if errors: - first_key = list(errors.keys())[0] - first_error = errors[first_key] - if isinstance(first_error, list) and first_error: - error_message = f"{first_key}: {first_error[0]}" - else: - error_message = f"{first_key}: {first_error}" - else: - error_message = "Validation failed" - elif isinstance(exc.detail, list): - # List of errors - error_message = exc.detail[0] if exc.detail else "An error occurred" - errors = {"non_field_errors": exc.detail} - else: - # String error message - error_message = str(exc.detail) - else: - # Generic exception - error_message = str(exc) if exc else "An error occurred" - - # Map status codes to appropriate error messages - if status_code == status.HTTP_400_BAD_REQUEST: - if not error_message or error_message == "An error occurred": - error_message = "Bad request" - elif status_code == status.HTTP_401_UNAUTHORIZED: - error_message = "Authentication required" if error_message == "An error occurred" else error_message - elif status_code == status.HTTP_403_FORBIDDEN: - error_message = "Permission denied" if error_message == "An error occurred" else error_message - elif status_code == status.HTTP_404_NOT_FOUND: - error_message = "Resource not found" if error_message == "An error occurred" else error_message - elif status_code == status.HTTP_500_INTERNAL_SERVER_ERROR: - error_message = "Internal server error" if error_message == "An error occurred" else error_message - - # Return unified error response - return error_response( - error=error_message, - errors=errors, - status_code=status_code - ) - - # Return None to use default exception handling for unhandled exceptions - return None -``` - -#### Step 4.2: Register Exception Handler in Settings - -**File:** `backend/igny8_core/settings.py` - -**Update REST_FRAMEWORK configuration:** -```python -REST_FRAMEWORK = { - 'DEFAULT_AUTHENTICATION_CLASSES': [ - 'igny8_core.api.authentication.JWTAuthentication', - 'igny8_core.api.authentication.CSRFExemptSessionAuthentication', - 'rest_framework.authentication.BasicAuthentication', - ], - 'DEFAULT_PERMISSION_CLASSES': [ - 'rest_framework.permissions.AllowAny', - ], - 'EXCEPTION_HANDLER': 'igny8_core.api.exception_handlers.custom_exception_handler', # Add this - # ... other settings ... -} -``` - -#### Step 4.3: Test Exception Handling - -**Test Cases:** -1. ValidationError (400) - Should return `{ success: false, error: "...", errors: {...} }` -2. PermissionDenied (403) - Should return `{ success: false, error: "Permission denied" }` -3. NotFound (404) - Should return `{ success: false, error: "Resource not found" }` -4. Generic Exception (500) - Should return `{ success: false, error: "Internal server error" }` - -**Estimated Time:** 3 hours - ---- - -## Task 5: Validation via Postman/Frontend Logs - -### Goal -Manually test endpoints to ensure responses match the unified format. - -### Implementation Steps - -#### Step 5.1: Create Test Checklist - -**Test Endpoints by Module:** - -**Auth Module (5 endpoints):** -- [ ] `POST /api/v1/auth/register/` - Success and error cases -- [ ] `POST /api/v1/auth/login/` - Success and error cases -- [ ] `POST /api/v1/auth/change-password/` - Success and error cases -- [ ] `GET /api/v1/auth/me/` - Success and unauthorized cases -- [ ] `GET /api/v1/auth/users/` - List with pagination - -**Planner Module (6 endpoints):** -- [ ] `GET /api/v1/planner/keywords/` - List with pagination -- [ ] `POST /api/v1/planner/keywords/` - Create success and validation error -- [ ] `POST /api/v1/planner/keywords/bulk_delete/` - Success and error -- [ ] `POST /api/v1/planner/keywords/auto_cluster/` - Success response -- [ ] `GET /api/v1/planner/clusters/` - List with pagination -- [ ] `POST /api/v1/planner/ideas/bulk_queue_to_writer/` - Success response - -**Writer Module (6 endpoints):** -- [ ] `GET /api/v1/writer/tasks/` - List with pagination -- [ ] `POST /api/v1/writer/tasks/` - Create success and validation error -- [ ] `POST /api/v1/writer/tasks/auto_generate_content/` - Success response -- [ ] `GET /api/v1/writer/content/` - List with pagination -- [ ] `POST /api/v1/writer/images/generate_images/` - Success response -- [ ] `GET /api/v1/writer/images/{id}/file/` - Success response - -**System Module (5 endpoints):** -- [ ] `GET /api/v1/system/prompts/` - List with pagination -- [ ] `POST /api/v1/system/prompts/save/` - Success and error -- [ ] `GET /api/v1/system/settings/integrations/{pk}/` - Success response -- [ ] `POST /api/v1/system/settings/integrations/{pk}/test/` - Success and error -- [ ] `GET /api/v1/system/status/` - Success response - -**Billing Module (3 endpoints):** -- [ ] `GET /api/v1/billing/credits/balance/balance/` - Success response -- [ ] `GET /api/v1/billing/credits/usage/` - List with pagination -- [ ] `GET /api/v1/billing/credits/usage/summary/` - Success response - -#### Step 5.2: Create Postman Collection - -**File:** `unified-api/postman/API-Response-Format-Tests.postman_collection.json` - -**Include:** -- All test endpoints from checklist -- Success case tests -- Error case tests (validation, 404, 403, 500) -- Response format validation (JSON schema) - -#### Step 5.3: Test Frontend Integration - -**Action Items:** -1. Check browser console for API response logs -2. Verify frontend code handles new response format -3. Test error handling in frontend -4. Verify pagination still works correctly -5. Check for any breaking changes in response structure - -**Files to Check:** -- `frontend/src/services/api.ts` - API client -- Frontend stores (Zustand) - Response parsing -- Error handling components - -**Estimated Time:** 8 hours - ---- - -## Task 6: Documentation Update - -### Goal -Update all relevant documentation to reflect the new unified response format. - -### Implementation Steps - -#### Step 6.1: Update Backend Implementation Docs - -**File:** `docs/04-BACKEND-IMPLEMENTATION.md` - -**Add Section: "Standard Response Format"** - -```markdown -## Standard Response Format - -All API endpoints return responses in a unified JSON format: - -### Success Response - -```json -{ - "success": true, - "data": { - // Response data (object or array) - }, - "message": "Optional success message" -} -``` - -### Error Response - -```json -{ - "success": false, - "error": "Top-level error message", - "errors": { - "field_name": ["Field-specific error messages"] - } -} -``` - -### Paginated Response - -```json -{ - "success": true, - "count": 100, - "next": "http://api.igny8.com/api/v1/endpoint/?page=2", - "previous": null, - "results": [ ... ], - "message": "Optional message" -} -``` - -### Status Codes - -- `200 OK` - Success -- `201 Created` - Resource created successfully -- `400 Bad Request` - Validation error or bad request -- `401 Unauthorized` - Authentication required -- `403 Forbidden` - Permission denied -- `404 Not Found` - Resource not found -- `500 Internal Server Error` - Server error -``` - -#### Step 6.2: Update API Endpoints Analysis - -**File:** `unified-api/API-ENDPOINTS-ANALYSIS.md` - -**Update Section:** "Response Format Standards" - -- Mark as "Implemented" or "Standardized" -- Add examples using new format -- Update any outdated examples - -#### Step 6.3: Update API Schema (if using Swagger/OpenAPI) - -**If using drf-spectacular or similar:** - -**File:** `backend/igny8_core/api/schema.py` (if exists) - -**Add response schema:** -```python -from drf_spectacular.utils import extend_schema, OpenApiResponse -from drf_spectacular.types import OpenApiTypes - -# Define unified response schemas -SUCCESS_RESPONSE_SCHEMA = { - "type": "object", - "properties": { - "success": {"type": "boolean", "example": True}, - "data": {"type": "object"}, - "message": {"type": "string", "nullable": True} - } -} - -ERROR_RESPONSE_SCHEMA = { - "type": "object", - "properties": { - "success": {"type": "boolean", "example": False}, - "error": {"type": "string"}, - "errors": {"type": "object", "nullable": True} - } -} -``` - -#### Step 6.4: Create API Design Guide Section - -**File:** `docs/API-DESIGN-GUIDE.md` (create if doesn't exist) - -**Add:** -- Response format policy -- When to use `success_response` vs `error_response` -- Pagination guidelines -- Error message best practices -- Status code guidelines - -**Estimated Time:** 4 hours - ---- - -## Task 7: Changelog Entry - -### Goal -Document the changes in the project changelog. - -### Implementation Steps - -#### Step 7.1: Update CHANGELOG.md - -**File:** `CHANGELOG.md` (or similar) - -**Add Entry:** -```markdown -## [Unreleased] - 2025-01-XX - -### Changed -- **API Response Format**: Unified all API responses under standardized success/error format - - All endpoints now return `{ success, data, message, error, errors }` format - - Created `igny8_core.api.response` module with `success_response()`, `error_response()`, and `paginated_response()` helpers - - Updated custom exception handler to wrap all errors in unified format - - Refactored all ViewSets across Auth, Planner, Writer, System, and Billing modules - - Updated documentation with response format examples - -### Affected Areas -- API Layer (`igny8_core/api/`) -- Auth Module (`auth/api/`) -- Planner Module (`planner/api/`) -- Writer Module (`writer/api/`) -- System Module (`system/api/`) -- Billing Module (`billing/api/`) - -### Breaking Changes -- ⚠️ **Response Format Change**: All API responses now follow unified format - - Frontend code may need updates if it directly accesses response fields - - Paginated responses now include `success: true` and optional `message` field - - Error responses now use `error` field instead of `message` for top-level errors - -### Migration Guide -1. Update frontend API client to handle new response format -2. Update error handling to check `success` field -3. Update pagination handling to account for `success` and `message` fields in paginated responses -``` - -**Estimated Time:** 1 hour - ---- - -## Testing Strategy - -### Unit Tests - -**File:** `backend/igny8_core/api/tests/test_response.py` - -**Test Cases:** -- Response wrapper functions -- Exception handler -- Edge cases (None data, empty lists, etc.) - -### Integration Tests - -**Test Cases:** -- End-to-end API calls -- Error scenarios (validation, 404, 500) -- Pagination with unified format -- Authentication errors - -### Manual Testing - -**Checklist:** -- [ ] All endpoints return unified format -- [ ] Success responses have `success: true` -- [ ] Error responses have `success: false` -- [ ] Pagination includes `success: true` -- [ ] Error messages are human-readable -- [ ] Validation errors include field-specific errors - ---- - -## Rollout Plan - -### Phase 1: Foundation (Week 1) -- ✅ Task 1: Create response wrapper functions -- ✅ Task 4: Update exception handlers -- ✅ Unit tests for response utilities - -### Phase 2: Module Refactoring (Week 2-3) -- ✅ Task 2.2: Refactor Auth module -- ✅ Task 2.3: Refactor Planner module -- ✅ Task 2.4: Refactor Writer module -- ✅ Task 2.5: Refactor System module -- ✅ Task 2.6: Refactor Billing module - -### Phase 3: Testing & Validation (Week 4) -- ✅ Task 5: Manual testing -- ✅ Frontend integration testing -- ✅ Bug fixes and adjustments - -### Phase 4: Documentation & Release (Week 5) -- ✅ Task 6: Documentation updates -- ✅ Task 7: Changelog entry -- ✅ Release to staging -- ✅ Production deployment - ---- - -## Success Criteria - -### Definition of Done - -1. ✅ All API endpoints return unified response format -2. ✅ Response wrapper functions are unit tested -3. ✅ Exception handler wraps all errors correctly -4. ✅ All ViewSets use response wrappers -5. ✅ Documentation is updated -6. ✅ Frontend integration tested and working -7. ✅ No breaking changes for critical endpoints (or migration guide provided) -8. ✅ Postman collection updated with new format -9. ✅ Changelog entry created - -### Metrics - -- **Coverage:** 100% of API endpoints use unified format -- **Test Coverage:** >90% for response utilities -- **Documentation:** All modules documented -- **Breaking Changes:** Documented and migration guide provided - ---- - -## Risk Assessment - -### Risks - -1. **Breaking Changes:** Frontend may break if it expects old format - - **Mitigation:** Provide migration guide, test frontend integration early - -2. **Performance Impact:** Wrapping responses may add overhead - - **Mitigation:** Minimal overhead, test performance if concerned - -3. **Incomplete Migration:** Some endpoints may be missed - - **Mitigation:** Comprehensive audit checklist, automated tests - -4. **Exception Handler Conflicts:** Custom exception handling may conflict - - **Mitigation:** Test thoroughly, handle edge cases - -### Rollback Plan - -If issues arise: -1. Revert exception handler changes (keep in settings but disable) -2. Keep response wrapper functions (non-breaking) -3. Gradually roll back ViewSet changes if needed -4. Document issues for future fixes - ---- - -## Appendix - -### Response Format Examples - -#### Success with Data -```json -{ - "success": true, - "data": { - "id": 1, - "name": "Example Keyword", - "status": "active" - }, - "message": "Keyword created successfully" -} -``` - -#### Success with List -```json -{ - "success": true, - "data": [ - {"id": 1, "name": "Keyword 1"}, - {"id": 2, "name": "Keyword 2"} - ], - "message": "Keywords retrieved successfully" -} -``` - -#### Error with Validation -```json -{ - "success": false, - "error": "Validation failed", - "errors": { - "email": ["Invalid email format"], - "password": ["Password must be at least 8 characters"] - } -} -``` - -#### Error without Details -```json -{ - "success": false, - "error": "Resource not found" -} -``` - -#### Paginated Response -```json -{ - "success": true, - "count": 150, - "next": "http://api.igny8.com/api/v1/planner/keywords/?page=2", - "previous": null, - "results": [ - {"id": 1, "name": "Keyword 1"}, - {"id": 2, "name": "Keyword 2"} - ], - "message": "Keywords retrieved successfully" -} -``` - ---- - -**Document Status:** Implementation Plan -**Last Updated:** 2025-01-XX -**Next Review:** After Phase 1 completion - diff --git a/unified-api/API-IMPLEMENTATION-PLAN-SECTION2.md b/unified-api/API-IMPLEMENTATION-PLAN-SECTION2.md deleted file mode 100644 index 5577673b..00000000 --- a/unified-api/API-IMPLEMENTATION-PLAN-SECTION2.md +++ /dev/null @@ -1,1357 +0,0 @@ -# API Implementation Plan - Section 2: Standardize Authentication and Authorization - -**Date:** 2025-01-XX -**Status:** Planning -**Priority:** High -**Related Document:** `API-ENDPOINTS-ANALYSIS.md` - ---- - -## Executive Summary - -This document outlines the implementation plan for **Section 2: Standardize Authentication and Authorization** across the IGNY8 API layer. The goal is to ensure every API endpoint enforces consistent access control based on authenticated users, roles, and tenant/site scoping, improving security and maintainability. - -**Key Objectives:** -- Standardize permission classes across all endpoints -- Implement consistent tenant/site-based access control -- Add role-based authorization where needed -- Secure custom actions and endpoints -- Ensure proper authentication fallback mechanisms - ---- - -## Table of Contents - -1. [Current State Analysis](#current-state-analysis) -2. [Implementation Tasks](#implementation-tasks) -3. [Task 1: Define Global Permission Classes](#task-1-define-global-permission-classes) -4. [Task 2: Create Unified BaseViewSet with Permission Injection](#task-2-create-unified-baseviewset-with-permission-injection) -5. [Task 3: Audit and Refactor All ViewSets](#task-3-audit-and-refactor-all-viewsets) -6. [Task 4: Inject Role-Based Checks Where Needed](#task-4-inject-role-based-checks-where-needed) -7. [Task 5: Secure Custom Actions](#task-5-secure-custom-actions) -8. [Task 6: Validate Token and Session Auth Coexistence](#task-6-validate-token-and-session-auth-coexistence) -9. [Task 7: Frontend Sync + Fallback UX](#task-7-frontend-sync--fallback-ux) -10. [Task 8: Changelog Entry](#task-8-changelog-entry) -11. [Testing Strategy](#testing-strategy) -12. [Rollout Plan](#rollout-plan) -13. [Success Criteria](#success-criteria) - ---- - -## Current State Analysis - -### Current Authentication & Authorization Issues - -Based on `API-ENDPOINTS-ANALYSIS.md`, the following inconsistencies exist: - -1. **Inconsistent Permission Classes:** - - Default: `AllowAny` (most endpoints) - - Many ViewSets set `permission_classes = []` (explicit AllowAny) - - Some use `IsAuthenticated` - - Some use custom classes: `IsOwnerOrAdmin`, `IsEditorOrAbove` - - No consistent pattern across modules - -2. **Existing Permission Classes:** - - `IsAuthenticated` - DRF standard - - `IsOwnerOrAdmin` - Custom (owner or admin role) - - `IsEditorOrAbove` - Custom (editor, admin, or owner) - - `IsViewerOrAbove` - Custom (viewer or above) - - `AccountPermission` - Custom (account-based access) - -3. **Base Classes:** - - `AccountModelViewSet` - Has account filtering but no default permissions - - `SiteSectorModelViewSet` - Has site/sector filtering but no default permissions - -4. **Authentication Methods:** - - JWT Authentication (primary) - `igny8_core.api.authentication.JWTAuthentication` - - Session Authentication (fallback) - `CSRFExemptSessionAuthentication` - - Basic Authentication (fallback) - `rest_framework.authentication.BasicAuthentication` - -5. **Account Context:** - - `AccountContextMiddleware` sets `request.account` from JWT token - - Account filtering happens in ViewSets but not consistently enforced - ---- - -## Implementation Tasks - -### Overview - -| Task ID | Task Name | Priority | Estimated Effort | Dependencies | -|---------|-----------|----------|------------------|--------------| -| 1.1 | Create global permission classes | High | 3 hours | None | -| 2.1 | Create BaseTenantViewSet | High | 4 hours | 1.1 | -| 3.1 | Audit all ViewSets | Medium | 4 hours | None | -| 3.2 | Refactor Auth module ViewSets | High | 6 hours | 2.1 | -| 3.3 | Refactor Planner module ViewSets | High | 6 hours | 2.1 | -| 3.4 | Refactor Writer module ViewSets | High | 6 hours | 2.1 | -| 3.5 | Refactor System module ViewSets | High | 8 hours | 2.1 | -| 3.6 | Refactor Billing module ViewSets | High | 4 hours | 2.1 | -| 4.1 | Define role system | High | 2 hours | None | -| 4.2 | Create role-based permission classes | High | 4 hours | 4.1 | -| 4.3 | Apply role checks to elevated endpoints | High | 6 hours | 4.2 | -| 5.1 | Audit all custom actions | Medium | 4 hours | None | -| 5.2 | Secure custom actions | High | 8 hours | 1.1, 2.1 | -| 6.1 | Validate auth coexistence | High | 3 hours | None | -| 7.1 | Update frontend error handling | High | 4 hours | None | -| 7.2 | Update frontend role storage | Medium | 2 hours | None | -| 7.3 | Add UI guards | Medium | 4 hours | 7.2 | -| 8.1 | Create changelog entry | Low | 1 hour | All tasks | - -**Total Estimated Effort:** ~75 hours - ---- - -## Task 1: Define Global Permission Classes - -### Goal -Create reusable permission classes that enforce consistent access control across all endpoints. - -### Implementation Steps - -#### Step 1.1: Create Permission Classes Module - -**File:** `backend/igny8_core/api/permissions.py` - -**Implementation:** -```python -""" -Unified Permission Classes for IGNY8 API - -This module provides permission classes that enforce consistent -access control based on authentication, tenant access, and roles. -""" - -from rest_framework import permissions -from rest_framework.exceptions import PermissionDenied - - -class IsAuthenticatedAndActive(permissions.BasePermission): - """ - Permission class that requires user to be authenticated and active. - - This is the base permission for most endpoints. It ensures: - - User is authenticated - - User account is active - - User is not disabled - """ - - def has_permission(self, request, view): - """ - Check if user is authenticated and active. - """ - if not request.user: - return False - - if not request.user.is_authenticated: - return False - - if not request.user.is_active: - return False - - return True - - def has_object_permission(self, request, view, obj): - """ - Object-level permission check. - By default, if user passes has_permission, allow object access. - Override in subclasses for object-level checks. - """ - return self.has_permission(request, view) - - -class HasTenantAccess(permissions.BasePermission): - """ - Permission class that ensures user has access to the tenant (account). - - This permission: - - Requires user to be authenticated - - Checks that user belongs to the account (request.account) - - Allows system bots to bypass checks - - Validates account context is set - """ - - def has_permission(self, request, view): - """ - Check if user has access to the tenant/account. - """ - if not request.user or not request.user.is_authenticated: - return False - - # System bots can access all accounts - if hasattr(request.user, 'role') and request.user.role == 'system_bot': - return True - - # Check if account context is set (from middleware or token) - account = getattr(request, 'account', None) - if not account: - # No account context - deny access - return False - - # Check if user belongs to this account - user_account = getattr(request.user, 'account', None) - if not user_account: - return False - - # User must belong to the account - return user_account == account - - def has_object_permission(self, request, view, obj): - """ - Object-level permission: check if object belongs to user's account. - """ - if not request.user or not request.user.is_authenticated: - return False - - # System bots can access all - if hasattr(request.user, 'role') and request.user.role == 'system_bot': - return True - - # Check if object has account field - obj_account = getattr(obj, 'account', None) - if not obj_account: - # Object doesn't have account - allow (for non-account models) - return True - - # Get user's account - user_account = getattr(request.user, 'account', None) - if not user_account: - return False - - # Object must belong to user's account - return obj_account == user_account - - -class HasModuleAccess(permissions.BasePermission): - """ - Permission class that checks if user has access to a specific module. - - This is optional and can be used for module-level access control - (e.g., Planner, Writer, System, Billing modules). - - Usage: - permission_classes = [IsAuthenticatedAndActive, HasTenantAccess, HasModuleAccess] - # Then set module_name in ViewSet: module_name = 'planner' - """ - - def has_permission(self, request, view): - """ - Check if user has access to the module. - """ - if not request.user or not request.user.is_authenticated: - return False - - # Get module name from view - module_name = getattr(view, 'module_name', None) - if not module_name: - # No module restriction - allow - return True - - # Check if user's account has access to this module - # This can be based on subscription, plan, or account settings - account = getattr(request, 'account', None) - if not account: - return False - - # TODO: Implement module access check based on account settings - # For now, allow all authenticated users - # Example: return account.has_module_access(module_name) - return True - - -class IsAdminOrOwner(permissions.BasePermission): - """ - Permission class that requires user to have admin or owner role. - - Use this for elevated privileges like: - - Credit management - - API key management - - User administration - - Account settings - """ - - def has_permission(self, request, view): - """ - Check if user has admin or owner role. - """ - if not request.user or not request.user.is_authenticated: - return False - - # Check user role - user_role = getattr(request.user, 'role', None) - if not user_role: - return False - - # Allow admin or owner - return user_role in ['admin', 'owner'] - - def has_object_permission(self, request, view, obj): - """ - Object-level check: user must be admin or owner. - """ - return self.has_permission(request, view) - - -class IsEditorOrAbove(permissions.BasePermission): - """ - Permission class that requires editor, admin, or owner role. - - Use this for content management operations. - """ - - def has_permission(self, request, view): - """ - Check if user has editor, admin, or owner role. - """ - if not request.user or not request.user.is_authenticated: - return False - - user_role = getattr(request.user, 'role', None) - if not user_role: - return False - - # Allow editor, admin, or owner - return user_role in ['editor', 'admin', 'owner'] - - def has_object_permission(self, request, view, obj): - """ - Object-level check: user must be editor or above. - """ - return self.has_permission(request, view) - - -class IsViewerOrAbove(permissions.BasePermission): - """ - Permission class that requires viewer, editor, admin, or owner role. - - Use this for read-only operations that should be accessible to all roles. - """ - - def has_permission(self, request, view): - """ - Check if user has viewer, editor, admin, or owner role. - """ - if not request.user or not request.user.is_authenticated: - return False - - user_role = getattr(request.user, 'role', None) - if not user_role: - return False - - # Allow viewer, editor, admin, or owner - return user_role in ['viewer', 'editor', 'admin', 'owner'] - - def has_object_permission(self, request, view, obj): - """ - Object-level check: user must be viewer or above. - """ - return self.has_permission(request, view) -``` - -#### Step 1.2: Update `__init__.py` for Easy Import - -**File:** `backend/igny8_core/api/__init__.py` - -**Add:** -```python -from .permissions import ( - IsAuthenticatedAndActive, - HasTenantAccess, - HasModuleAccess, - IsAdminOrOwner, - IsEditorOrAbove, - IsViewerOrAbove, -) - -__all__ = [ - 'IsAuthenticatedAndActive', - 'HasTenantAccess', - 'HasModuleAccess', - 'IsAdminOrOwner', - 'IsEditorOrAbove', - 'IsViewerOrAbove', -] -``` - -#### Step 1.3: Create Unit Tests - -**File:** `backend/igny8_core/api/tests/test_permissions.py` - -**Test Cases:** -- Test `IsAuthenticatedAndActive` with authenticated/active user -- Test `IsAuthenticatedAndActive` with unauthenticated user -- Test `IsAuthenticatedAndActive` with inactive user -- Test `HasTenantAccess` with matching account -- Test `HasTenantAccess` with mismatched account -- Test `HasTenantAccess` with system bot -- Test `IsAdminOrOwner` with admin role -- Test `IsAdminOrOwner` with owner role -- Test `IsAdminOrOwner` with editor role (should fail) -- Test `IsEditorOrAbove` with various roles -- Test object-level permissions - -**Estimated Time:** 3 hours - ---- - -## Task 2: Create Unified BaseViewSet with Permission Injection - -### Goal -Create a base ViewSet class that automatically applies standard permissions and tenant filtering. - -### Implementation Steps - -#### Step 2.1: Create BaseTenantViewSet - -**File:** `backend/igny8_core/api/viewsets.py` - -**Implementation:** -```python -""" -Base ViewSet Classes with Unified Permissions and Tenant Filtering -""" - -from rest_framework import viewsets -from igny8_core.api.permissions import ( - IsAuthenticatedAndActive, - HasTenantAccess, -) -from igny8_core.api.base import AccountModelViewSet - - -class BaseTenantViewSet(AccountModelViewSet): - """ - Base ViewSet that automatically applies standard permissions and tenant filtering. - - This ViewSet: - - Requires authentication and active user - - Enforces tenant/account access - - Automatically filters queryset by account - - Can be extended for module-specific or role-specific access - - All module ViewSets should inherit from this instead of AccountModelViewSet. - - Usage: - class MyViewSet(BaseTenantViewSet): - queryset = MyModel.objects.all() - serializer_class = MySerializer - """ - - # Default permissions: require authentication and tenant access - permission_classes = [ - IsAuthenticatedAndActive, - HasTenantAccess, - ] - - def get_queryset(self): - """ - Get queryset filtered by account. - - This extends AccountModelViewSet.get_queryset() to ensure - proper account filtering is applied. - """ - queryset = super().get_queryset() - - # Account filtering is handled by AccountModelViewSet - # This method can be overridden in subclasses for additional filtering - - return queryset - - def perform_create(self, serializer): - """ - Override to ensure account is set on create. - """ - # Get account from request (set by middleware or authentication) - account = getattr(self.request, 'account', None) - - if account and hasattr(serializer.Meta.model, 'account'): - # Set account on the object being created - serializer.save(account=account) - else: - serializer.save() -``` - -#### Step 2.2: Update AccountModelViewSet (if needed) - -**File:** `backend/igny8_core/api/base.py` - -**Review and ensure:** -- Account filtering logic is correct -- Admin/developer override logic works -- System account bypass works correctly -- Compatible with new permission classes - -**Estimated Time:** 4 hours - ---- - -## Task 3: Audit and Refactor All ViewSets - -### Goal -Update all ViewSets to inherit from `BaseTenantViewSet` and remove redundant permission logic. - -### Implementation Strategy - -#### Step 3.1: Audit All ViewSets - -**Action Items:** -1. List all ViewSets in each module -2. Document current permission classes -3. Identify ViewSets that need role-based permissions -4. Identify ViewSets that should remain public (AllowAny) -5. Create refactoring checklist - -**Files to Audit:** - -**Auth Module:** -- `backend/auth/api/views.py` - UsersViewSet, AccountsViewSet, SiteViewSet, etc. -- `backend/auth/api/viewsets.py` - Custom authentication views - -**Planner Module:** -- `backend/planner/api/viewsets.py` - KeywordViewSet, ClusterViewSet, ContentIdeasViewSet - -**Writer Module:** -- `backend/writer/api/viewsets.py` - TasksViewSet, ContentViewSet, ImagesViewSet - -**System Module:** -- `backend/system/api/viewsets.py` - AIPromptViewSet, AuthorProfileViewSet, StrategyViewSet -- `backend/system/api/viewsets.py` - IntegrationSettingsViewSet, SystemSettingsViewSet, etc. - -**Billing Module:** -- `backend/billing/api/viewsets.py` - CreditBalanceViewSet, CreditUsageViewSet, CreditTransactionViewSet - -**Public Endpoints (should remain AllowAny):** -- `POST /api/v1/auth/register/` - Registration -- `POST /api/v1/auth/login/` - Login -- `GET /api/v1/auth/plans/` - Public plans list -- `GET /api/v1/auth/industries/` - Public industries list -- `GET /api/v1/system/status/` - System health check - -**Estimated Time:** 4 hours - -#### Step 3.2: Refactor Auth Module ViewSets - -**Priority Endpoints:** -1. `POST /api/v1/auth/register/` - Keep `AllowAny` (public) -2. `POST /api/v1/auth/login/` - Keep `AllowAny` (public) -3. `POST /api/v1/auth/change-password/` - Use `IsAuthenticatedAndActive` -4. `GET /api/v1/auth/me/` - Use `IsAuthenticatedAndActive` -5. UsersViewSet - Use `IsAdminOrOwner` (user management) -6. AccountsViewSet - Use `IsAdminOrOwner` (account management) -7. SiteViewSet - Use `IsEditorOrAbove` (site management) -8. SectorViewSet - Use `IsEditorOrAbove` (sector management) -9. PlanViewSet - Keep `AllowAny` (public read-only) -10. IndustryViewSet - Keep `AllowAny` (public read-only) - -**Example Refactoring:** - -**Before:** -```python -class UsersViewSet(AccountModelViewSet): - permission_classes = [IsOwnerOrAdmin] # Old custom class - queryset = User.objects.all() - serializer_class = UserSerializer -``` - -**After:** -```python -from igny8_core.api.viewsets import BaseTenantViewSet -from igny8_core.api.permissions import IsAdminOrOwner - -class UsersViewSet(BaseTenantViewSet): - permission_classes = [IsAuthenticatedAndActive, HasTenantAccess, IsAdminOrOwner] - queryset = User.objects.all() - serializer_class = UserSerializer -``` - -**Estimated Time:** 6 hours - -#### Step 3.3: Refactor Planner Module ViewSets - -**Priority Endpoints:** -1. KeywordViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` -2. ClusterViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` -3. ContentIdeasViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` - -**Note:** Planner module ViewSets inherit from `SiteSectorModelViewSet`, which already has account filtering. Update to use `BaseTenantViewSet` or create `BaseSiteSectorViewSet` that extends `BaseTenantViewSet`. - -**Estimated Time:** 6 hours - -#### Step 3.4: Refactor Writer Module ViewSets - -**Priority Endpoints:** -1. TasksViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` -2. ContentViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` -3. ImagesViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` - -**Estimated Time:** 6 hours - -#### Step 3.5: Refactor System Module ViewSets - -**Priority Endpoints:** -1. AIPromptViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` -2. AuthorProfileViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` -3. StrategyViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` -4. IntegrationSettingsViewSet - Use `IsAdminOrOwner` (sensitive settings) -5. SystemSettingsViewSet - Use `IsAdminOrOwner` (system settings) -6. AccountSettingsViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` -7. UserSettingsViewSet - Use `IsAuthenticatedAndActive` (user-specific) - -**Estimated Time:** 8 hours - -#### Step 3.6: Refactor Billing Module ViewSets - -**Priority Endpoints:** -1. CreditBalanceViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` -2. CreditUsageViewSet - Use `IsAuthenticatedAndActive, HasTenantAccess` -3. CreditTransactionViewSet - Use `IsAdminOrOwner` (sensitive financial data) - -**Estimated Time:** 4 hours - -### Refactoring Checklist Template - -For each ViewSet, check: - -- [ ] Inherit from `BaseTenantViewSet` (or appropriate base class) -- [ ] Set appropriate `permission_classes` -- [ ] Remove redundant permission logic -- [ ] Ensure queryset filtering works correctly -- [ ] Test authentication required -- [ ] Test tenant access enforcement -- [ ] Test role-based access (if applicable) -- [ ] Verify public endpoints remain public (AllowAny) - ---- - -## Task 4: Inject Role-Based Checks Where Needed - -### Goal -Implement role-based authorization for endpoints that require elevated privileges. - -### Implementation Steps - -#### Step 4.1: Define Role System - -**Review existing role system:** - -**File:** `backend/igny8_core/auth/models.py` (or wherever User model is defined) - -**Roles to support:** -- `owner` - Full account access -- `admin` - Administrative access -- `editor` - Content editing access -- `writer` - Content writing access -- `viewer` - Read-only access -- `system_bot` - System/internal access - -**Action Items:** -1. Verify User model has `role` field -2. Document role hierarchy and permissions -3. Create role enum or constants if needed - -**Estimated Time:** 2 hours - -#### Step 4.2: Create Role-Based Permission Classes - -**File:** `backend/igny8_core/api/permissions.py` (add to existing file) - -**Additional Permission Classes:** -```python -class RequireRole(permissions.BasePermission): - """ - Permission class that requires a specific role or set of roles. - - Usage: - permission_classes = [IsAuthenticatedAndActive, RequireRole(['admin', 'owner'])] - """ - - def __init__(self, allowed_roles): - self.allowed_roles = allowed_roles if isinstance(allowed_roles, list) else [allowed_roles] - - def has_permission(self, request, view): - if not request.user or not request.user.is_authenticated: - return False - - user_role = getattr(request.user, 'role', None) - if not user_role: - return False - - return user_role in self.allowed_roles - - -class RolePermissionMixin: - """ - Mixin that adds role-based permission checking to ViewSets. - - Usage: - class MyViewSet(BaseTenantViewSet, RolePermissionMixin): - required_roles = ['admin', 'owner'] - """ - - def get_permissions(self): - """ - Add role-based permission if required_roles is set. - """ - permissions = super().get_permissions() - - required_roles = getattr(self, 'required_roles', None) - if required_roles: - permissions.append(RequireRole(required_roles)) - - return permissions -``` - -**Estimated Time:** 4 hours - -#### Step 4.3: Apply Role Checks to Elevated Endpoints - -**Endpoints requiring elevated privileges:** - -1. **User Management:** - - `POST /api/v1/auth/users/` - Create user (admin/owner) - - `PUT /api/v1/auth/users/{id}/` - Update user (admin/owner) - - `DELETE /api/v1/auth/users/{id}/` - Delete user (admin/owner) - - `POST /api/v1/auth/users/invite/` - Invite user (admin/owner) - -2. **Account Management:** - - `POST /api/v1/auth/accounts/` - Create account (admin/owner) - - `PUT /api/v1/auth/accounts/{id}/` - Update account (admin/owner) - - `DELETE /api/v1/auth/accounts/{id}/` - Delete account (admin/owner) - -3. **Credit Management:** - - `POST /api/v1/billing/credits/transactions/` - Create transaction (admin/owner) - - Credit balance updates (admin/owner) - -4. **Integration Settings:** - - `POST /api/v1/system/settings/integrations/{pk}/save/` - Save settings (admin/owner) - - `POST /api/v1/system/settings/integrations/{pk}/test/` - Test connection (admin/owner) - -5. **System Settings:** - - `POST /api/v1/system/settings/system/` - Create system setting (admin/owner) - - `PUT /api/v1/system/settings/system/{id}/` - Update system setting (admin/owner) - -**Example Application:** - -```python -from igny8_core.api.viewsets import BaseTenantViewSet -from igny8_core.api.permissions import IsAdminOrOwner - -class UsersViewSet(BaseTenantViewSet): - permission_classes = [ - IsAuthenticatedAndActive, - HasTenantAccess, - IsAdminOrOwner, # Require admin or owner role - ] - queryset = User.objects.all() - serializer_class = UserSerializer -``` - -**Estimated Time:** 6 hours - ---- - -## Task 5: Secure Custom Actions - -### Goal -Ensure all custom actions (`@action` methods) have proper permission checks and payload validation. - -### Implementation Steps - -#### Step 5.1: Audit All Custom Actions - -**Action Items:** -1. List all `@action` decorators in all ViewSets -2. Document current permission checks (if any) -3. Identify actions that need explicit permissions -4. Identify actions that need payload validation - -**Custom Actions to Audit:** - -**Planner Module:** -- `POST /api/v1/planner/keywords/bulk_delete/` -- `POST /api/v1/planner/keywords/bulk_update_status/` -- `POST /api/v1/planner/keywords/bulk_add_from_seed/` -- `POST /api/v1/planner/keywords/auto_cluster/` -- `POST /api/v1/planner/clusters/auto_generate_ideas/` -- `POST /api/v1/planner/ideas/bulk_queue_to_writer/` - -**Writer Module:** -- `POST /api/v1/writer/tasks/auto_generate_content/` -- `POST /api/v1/writer/content/generate_image_prompts/` -- `POST /api/v1/writer/images/generate_images/` -- `POST /api/v1/writer/images/bulk_update/` - -**System Module:** -- `POST /api/v1/system/prompts/save/` -- `POST /api/v1/system/prompts/reset/` -- `POST /api/v1/system/settings/integrations/{pk}/test/` -- `POST /api/v1/system/settings/integrations/{pk}/generate/` - -**Estimated Time:** 4 hours - -#### Step 5.2: Secure Custom Actions - -**Implementation Pattern:** - -**Before:** -```python -class KeywordViewSet(SiteSectorModelViewSet): - @action(detail=False, methods=['post']) - def bulk_delete(self, request): - ids = request.data.get('ids', []) - # No permission check, no validation - deleted_count = Keyword.objects.filter(id__in=ids).delete()[0] - return Response({"deleted_count": deleted_count}) -``` - -**After:** -```python -from igny8_core.api.permissions import IsAuthenticatedAndActive, HasTenantAccess - -class KeywordViewSet(BaseTenantViewSet): - @action( - detail=False, - methods=['post'], - permission_classes=[IsAuthenticatedAndActive, HasTenantAccess] - ) - def bulk_delete(self, request): - ids = request.data.get('ids', []) - - # Validate payload - if not ids or not isinstance(ids, list): - return error_response( - error="Invalid payload: 'ids' must be a non-empty list", - status_code=status.HTTP_400_BAD_REQUEST - ) - - # Ensure all IDs belong to current user's account - queryset = self.get_queryset() # Already filtered by account - keywords = queryset.filter(id__in=ids) - - # Validate all IDs exist and belong to account - if keywords.count() != len(ids): - return error_response( - error="Some keywords not found or don't belong to your account", - status_code=status.HTTP_403_FORBIDDEN - ) - - deleted_count = keywords.delete()[0] - return success_response( - data={"deleted_count": deleted_count}, - message=f"Successfully deleted {deleted_count} keywords" - ) -``` - -**Key Security Checks:** -1. Explicit `permission_classes` on `@action` decorator -2. Validate payload structure and types -3. Use `get_queryset()` to ensure account filtering -4. Verify all IDs belong to user's account before processing -5. Return appropriate error responses for validation failures - -**Estimated Time:** 8 hours - ---- - -## Task 6: Validate Token and Session Auth Coexistence - -### Goal -Ensure both token-based and session-based authentication work correctly and are used appropriately. - -### Implementation Steps - -#### Step 6.1: Review Current Authentication Configuration - -**File:** `backend/igny8_core/settings.py` - -**Current Configuration:** -```python -REST_FRAMEWORK = { - 'DEFAULT_AUTHENTICATION_CLASSES': [ - 'igny8_core.api.authentication.JWTAuthentication', # Primary - 'igny8_core.api.authentication.CSRFExemptSessionAuthentication', # Fallback - 'rest_framework.authentication.BasicAuthentication', # Fallback - ], - 'DEFAULT_PERMISSION_CLASSES': [ - 'rest_framework.permissions.AllowAny', # Will be changed to IsAuthenticatedAndActive - ], -} -``` - -**Action Items:** -1. Verify authentication classes are in correct order (JWT first) -2. Ensure CSRFExemptSessionAuthentication is properly configured -3. Document which endpoints should use which auth method -4. Test authentication fallback behavior - -**Estimated Time:** 2 hours - -#### Step 6.2: Test Authentication Coexistence - -**Test Cases:** -1. **JWT Token Authentication:** - - Valid JWT token in `Authorization: Bearer ` header - - Should authenticate successfully - - Should set `request.user` and `request.account` - -2. **Session Authentication:** - - Valid session cookie (from Django admin or session login) - - Should authenticate successfully as fallback - - Should work for admin panel (`/admin/`) - -3. **Basic Authentication:** - - Valid HTTP Basic Auth credentials - - Should authenticate as last fallback - - Should work for API testing tools - -4. **No Authentication:** - - No token, no session, no basic auth - - Should return 401 Unauthorized for protected endpoints - - Should allow access for public endpoints (AllowAny) - -5. **Invalid Token:** - - Expired JWT token - - Invalid JWT token format - - Should fall back to session auth, then basic auth - - Should return 401 if all fail - -**Estimated Time:** 1 hour - -#### Step 6.3: Document Authentication Usage - -**File:** `docs/04-BACKEND-IMPLEMENTATION.md` (or create new auth doc) - -**Add Section: "Authentication Methods"** - -```markdown -## Authentication Methods - -### JWT Token Authentication (Primary) -- **Use Case:** Frontend application API calls -- **Header:** `Authorization: Bearer ` -- **Token Type:** Access token (15-minute expiry) -- **Token Payload:** `user_id`, `account_id`, `type: 'access'` -- **Account Context:** Automatically sets `request.account` from token - -### Session Authentication (Fallback) -- **Use Case:** Django admin panel (`/admin/`) -- **Method:** Session cookies (CSRF exempt for API) -- **Configuration:** `CSRFExemptSessionAuthentication` -- **Note:** Only used when JWT authentication fails - -### Basic Authentication (Fallback) -- **Use Case:** API testing tools (Postman, curl) -- **Method:** HTTP Basic Auth -- **Note:** Only used when JWT and session authentication fail - -### Authentication Order -1. JWT Token Authentication (tried first) -2. Session Authentication (fallback) -3. Basic Authentication (last fallback) -4. If all fail: 401 Unauthorized -``` - -**Estimated Time:** 1 hour - ---- - -## Task 7: Frontend Sync + Fallback UX - -### Goal -Ensure frontend properly handles authentication errors and role-based UI restrictions. - -### Implementation Steps - -#### Step 7.1: Update Frontend Error Handling - -**File:** `frontend/src/services/api.ts` - -**Current Error Handling:** -- Check for 401/403 responses -- Handle token refresh -- Route to login on authentication failure - -**Updates Needed:** -1. **Enhanced 401 Handling:** - ```typescript - if (response.status === 401) { - // Clear auth state - authStore.getState().logout(); - // Show user-friendly message - showNotification('Session expired. Please log in again.', 'error'); - // Redirect to login - router.push('/login'); - } - ``` - -2. **Enhanced 403 Handling:** - ```typescript - if (response.status === 403) { - // Show permission error - showNotification('You do not have permission to perform this action.', 'error'); - // Optionally redirect or show restricted UI - } - ``` - -3. **Error Response Parsing:** - ```typescript - // Parse unified error format - const errorData = await response.json(); - if (errorData.success === false) { - const errorMessage = errorData.error || 'An error occurred'; - const fieldErrors = errorData.errors || {}; - // Handle error message and field errors - } - ``` - -**Estimated Time:** 4 hours - -#### Step 7.2: Update Frontend Role Storage - -**File:** `frontend/src/stores/authStore.ts` (or similar) - -**Updates Needed:** -1. **Store Role Information:** - ```typescript - interface AuthState { - user: { - id: number; - email: string; - role: 'owner' | 'admin' | 'editor' | 'writer' | 'viewer'; - account: { - id: number; - name: string; - }; - }; - // ... other fields - } - ``` - -2. **Update on Login:** - ```typescript - const login = async (email: string, password: string) => { - const response = await fetchAPI('/v1/auth/login/', { - method: 'POST', - body: JSON.stringify({ email, password }), - }); - const data = await response.json(); - - if (data.success && data.user) { - set({ - user: data.user, - token: data.tokens.access, - // Store role for UI guards - }); - } - }; - ``` - -3. **Role Helper Functions:** - ```typescript - export const hasRole = (requiredRole: string): boolean => { - const user = authStore.getState().user; - if (!user) return false; - - const roleHierarchy = ['viewer', 'writer', 'editor', 'admin', 'owner']; - const userRoleIndex = roleHierarchy.indexOf(user.role); - const requiredRoleIndex = roleHierarchy.indexOf(requiredRole); - - return userRoleIndex >= requiredRoleIndex; - }; - - export const isAdminOrOwner = (): boolean => { - const user = authStore.getState().user; - return user?.role === 'admin' || user?.role === 'owner'; - }; - ``` - -**Estimated Time:** 2 hours - -#### Step 7.3: Add UI Guards - -**Implementation Pattern:** - -**Component-Level Guards:** -```typescript -import { hasRole, isAdminOrOwner } from '@/utils/roles'; - -function UserManagementPage() { - const user = useAuthStore((state) => state.user); - - // Check permission - if (!isAdminOrOwner()) { - return ; - } - - // Render component - return ; -} -``` - -**Button-Level Guards:** -```typescript -function SettingsPage() { - const canManageSettings = isAdminOrOwner(); - - return ( -
- - {!canManageSettings && ( - Admin access required - )} -
- ); -} -``` - -**Route-Level Guards:** -```typescript -// In router configuration -{ - path: '/admin/users', - element: , -} -``` - -**Files to Update:** -- Create `frontend/src/utils/roles.ts` - Role helper functions -- Create `frontend/src/components/ProtectedRoute.tsx` - Route guard component -- Create `frontend/src/components/AccessDenied.tsx` - Access denied component -- Update components that need role-based UI restrictions - -**Estimated Time:** 4 hours - ---- - -## Task 8: Changelog Entry - -### Goal -Document the authentication and authorization standardization changes. - -### Implementation Steps - -#### Step 8.1: Update CHANGELOG.md - -**File:** `CHANGELOG.md` (or similar) - -**Add Entry:** -```markdown -## [Unreleased] - 2025-01-XX - -### Changed -- **Authentication & Authorization**: Standardized authentication and permission enforcement across all API endpoints - - Created unified permission classes: `IsAuthenticatedAndActive`, `HasTenantAccess`, `IsAdminOrOwner`, `IsEditorOrAbove`, `IsViewerOrAbove` - - Created `BaseTenantViewSet` with automatic permission injection and tenant filtering - - Refactored all ViewSets across Auth, Planner, Writer, System, and Billing modules to use standardized permissions - - Secured all custom actions with explicit permission checks and payload validation - - Updated frontend error handling for 401/403 responses - - Added role-based UI guards and access controls - -### Security -- **Breaking Change**: Most endpoints now require authentication (changed from `AllowAny` to `IsAuthenticatedAndActive`) - - Public endpoints (register, login, plans, industries, status) remain accessible without authentication - - All other endpoints now require valid JWT token or session authentication - - Tenant/account access is now enforced at the permission level - -### Affected Areas -- API Layer (`igny8_core/api/permissions.py`, `igny8_core/api/viewsets.py`) -- Auth Module (`auth/api/`) -- Planner Module (`planner/api/`) -- Writer Module (`writer/api/`) -- System Module (`system/api/`) -- Billing Module (`billing/api/`) -- Frontend (`frontend/src/services/api.ts`, `frontend/src/stores/authStore.ts`) - -### Migration Guide -1. **Frontend Updates Required:** - - Update API client to handle 401/403 errors gracefully - - Store user role information in auth store - - Add UI guards for role-based access - - Update error messages for authentication/permission failures - -2. **API Testing:** - - All API tests must include authentication tokens - - Update test fixtures to use authenticated users - - Test role-based access for elevated endpoints - -3. **Public Endpoints:** - - Registration, login, plans, industries, and status endpoints remain public - - All other endpoints require authentication -``` - -**Estimated Time:** 1 hour - ---- - -## Testing Strategy - -### Unit Tests - -**File:** `backend/igny8_core/api/tests/test_permissions.py` - -**Test Cases:** -- Permission classes with various user states -- Tenant access validation -- Role-based permission checks -- Object-level permissions -- System bot bypass logic - -### Integration Tests - -**Test Cases:** -- End-to-end API calls with authentication -- Authentication failure scenarios (401) -- Permission denial scenarios (403) -- Tenant isolation (user can't access other accounts) -- Role-based access control -- Custom action security - -### Manual Testing - -**Checklist:** -- [ ] All endpoints require authentication (except public ones) -- [ ] Tenant access is enforced correctly -- [ ] Role-based access works for elevated endpoints -- [ ] Custom actions have proper permission checks -- [ ] Payload validation works in custom actions -- [ ] Frontend handles 401/403 errors correctly -- [ ] UI guards hide/show features based on role -- [ ] Session auth works for admin panel -- [ ] JWT token auth works for API calls - ---- - -## Rollout Plan - -### Phase 1: Foundation (Week 1) -- ✅ Task 1: Create global permission classes -- ✅ Task 2: Create BaseTenantViewSet -- ✅ Unit tests for permission classes - -### Phase 2: Module Refactoring (Week 2-3) -- ✅ Task 3.2: Refactor Auth module -- ✅ Task 3.3: Refactor Planner module -- ✅ Task 3.4: Refactor Writer module -- ✅ Task 3.5: Refactor System module -- ✅ Task 3.6: Refactor Billing module - -### Phase 3: Role-Based Access (Week 4) -- ✅ Task 4: Implement role-based checks -- ✅ Task 5: Secure custom actions -- ✅ Integration testing - -### Phase 4: Frontend & Validation (Week 5) -- ✅ Task 6: Validate auth coexistence -- ✅ Task 7: Frontend updates -- ✅ End-to-end testing -- ✅ Bug fixes and adjustments - -### Phase 5: Documentation & Release (Week 6) -- ✅ Task 8: Changelog entry -- ✅ Documentation updates -- ✅ Release to staging -- ✅ Production deployment - ---- - -## Success Criteria - -### Definition of Done - -1. ✅ All permission classes are defined and tested -2. ✅ BaseTenantViewSet is created and used by all ViewSets -3. ✅ All ViewSets have appropriate permission classes -4. ✅ All custom actions have explicit permission checks -5. ✅ Payload validation is implemented in custom actions -6. ✅ Frontend handles authentication errors correctly -7. ✅ Role-based UI guards are implemented -8. ✅ Documentation is updated -9. ✅ Changelog entry is created -10. ✅ All tests pass - -### Metrics - -- **Coverage:** 100% of endpoints have explicit permission classes -- **Security:** All endpoints enforce tenant access (except public ones) -- **Test Coverage:** >90% for permission classes -- **Breaking Changes:** Documented and migration guide provided - ---- - -## Risk Assessment - -### Risks - -1. **Breaking Changes:** Frontend may break if endpoints now require auth - - **Mitigation:** Keep public endpoints public, provide migration guide, test frontend early - -2. **Performance Impact:** Permission checks may add overhead - - **Mitigation:** Minimal overhead, cache permission checks if needed - -3. **Incomplete Migration:** Some endpoints may be missed - - **Mitigation:** Comprehensive audit checklist, automated tests - -4. **Role System Conflicts:** Existing role system may conflict - - **Mitigation:** Review existing role implementation, ensure compatibility - -### Rollback Plan - -If issues arise: -1. Revert permission class changes (keep classes but don't apply) -2. Keep BaseTenantViewSet (non-breaking) -3. Gradually roll back ViewSet changes if needed -4. Document issues for future fixes - ---- - -## Appendix - -### Permission Class Usage Guide - -#### Public Endpoints (AllowAny) -```python -class RegisterView(APIView): - permission_classes = [AllowAny] # Public endpoint -``` - -#### Standard Authenticated Endpoints -```python -class MyViewSet(BaseTenantViewSet): - # Automatically uses IsAuthenticatedAndActive + HasTenantAccess - queryset = MyModel.objects.all() -``` - -#### Role-Based Endpoints -```python -class UsersViewSet(BaseTenantViewSet): - permission_classes = [ - IsAuthenticatedAndActive, - HasTenantAccess, - IsAdminOrOwner, # Require admin or owner - ] -``` - -#### Custom Action with Explicit Permissions -```python -class MyViewSet(BaseTenantViewSet): - @action( - detail=False, - methods=['post'], - permission_classes=[IsAuthenticatedAndActive, HasTenantAccess, IsAdminOrOwner] - ) - def sensitive_action(self, request): - # Action implementation - pass -``` - ---- - -**Document Status:** Implementation Plan -**Last Updated:** 2025-01-XX -**Next Review:** After Phase 1 completion - diff --git a/unified-api/API-IMPLEMENTATION-PLAN-SECTION3.md b/unified-api/API-IMPLEMENTATION-PLAN-SECTION3.md deleted file mode 100644 index 3e9722b7..00000000 --- a/unified-api/API-IMPLEMENTATION-PLAN-SECTION3.md +++ /dev/null @@ -1,1207 +0,0 @@ -# API Implementation Plan - Section 3: Implement Consistent Error Handling - -**Date:** 2025-01-XX -**Status:** Planning -**Priority:** High -**Related Document:** `API-ENDPOINTS-ANALYSIS.md`, `API-IMPLEMENTATION-PLAN-SECTION1.md` - ---- - -## Executive Summary - -This document outlines the implementation plan for **Section 3: Implement Consistent Error Handling** across the IGNY8 API layer. The goal is to ensure all exceptions, validation errors, and system-level failures return uniform, predictable JSON responses with proper HTTP status codes and user-readable messages. - -**Key Objectives:** -- Create centralized exception handler that wraps all errors in unified format -- Unify validation error format across all endpoints -- Implement proper logging for debugging and monitoring -- Add developer-focused debug support for development environments -- Ensure consistent error responses for all failure scenarios - -**Note:** This section builds on Section 1 Task 4 (Exception Handlers) and extends it with logging, debugging, and validation error unification. - ---- - -## Table of Contents - -1. [Current State Analysis](#current-state-analysis) -2. [Implementation Tasks](#implementation-tasks) -3. [Task 1: Create Centralized Exception Handler](#task-1-create-centralized-exception-handler) -4. [Task 2: Register in Global DRF Settings](#task-2-register-in-global-drf-settings) -5. [Task 3: Unify Validation Error Format](#task-3-unify-validation-error-format) -6. [Task 4: Patch or Log Internal Server Errors](#task-4-patch-or-log-internal-server-errors) -7. [Task 5: Add Developer-Focused Debug Support](#task-5-add-developer-focused-debug-support) -8. [Task 6: Test Scenarios](#task-6-test-scenarios) -9. [Task 7: Changelog Entry](#task-7-changelog-entry) -10. [Testing Strategy](#testing-strategy) -11. [Rollout Plan](#rollout-plan) -12. [Success Criteria](#success-criteria) - ---- - -## Current State Analysis - -### Current Error Handling Issues - -Based on `API-ENDPOINTS-ANALYSIS.md` and codebase analysis: - -1. **Inconsistent Error Formats:** - - Some endpoints return `{ error: "..." }` - - Some return `{ success: false, message: "..." }` - - Some return DRF default format - - Validation errors may not be consistently formatted - -2. **Exception Handling:** - - Section 1 Task 4 creates basic exception handler - - No logging for unhandled exceptions - - No request ID tracking for error correlation - - No debug information in development mode - -3. **Validation Errors:** - - DRF serializer errors are handled automatically - - Manual validation errors may not follow unified format - - Field-specific errors may not be consistently structured - -4. **Error Logging:** - - Some errors are logged, but not consistently - - No centralized error logging strategy - - No integration with error tracking services (Sentry, etc.) - -5. **Frontend Error Handling:** - - Frontend has error parsing logic but may not handle all formats - - Error messages may not be user-friendly - - No consistent error display strategy - ---- - -## Implementation Tasks - -### Overview - -| Task ID | Task Name | Priority | Estimated Effort | Dependencies | -|---------|-----------|----------|------------------|--------------| -| 1.1 | Create centralized exception handler | High | 4 hours | Section 1 Task 1 (response.py) | -| 1.2 | Enhance exception handler with logging | High | 3 hours | 1.1 | -| 2.1 | Register exception handler in settings | High | 1 hour | 1.1 | -| 3.1 | Audit validation error usage | Medium | 3 hours | None | -| 3.2 | Unify validation error format | High | 4 hours | 1.1 | -| 4.1 | Configure error logging | High | 3 hours | None | -| 4.2 | Add request ID tracking | Medium | 2 hours | 4.1 | -| 4.3 | Integrate error tracking service (optional) | Low | 4 hours | 4.1 | -| 5.1 | Add debug mode support | Low | 3 hours | 1.1 | -| 6.1 | Create test scenarios | High | 4 hours | All tasks | -| 7.1 | Create changelog entry | Low | 1 hour | All tasks | - -**Total Estimated Effort:** ~32 hours - ---- - -## Task 1: Create Centralized Exception Handler - -### Goal -Create a comprehensive exception handler that wraps all exceptions in the unified error format with proper logging and debugging support. - -### Implementation Steps - -#### Step 1.1: Create Enhanced Exception Handler - -**File:** `backend/igny8_core/api/exception_handlers.py` - -**Note:** This extends Section 1 Task 4 with enhanced error handling, logging, and debug support. - -**Implementation:** -```python -""" -Centralized Exception Handler for IGNY8 API - -This module provides a comprehensive exception handler that: -- Wraps all exceptions in unified error format -- Logs errors for debugging and monitoring -- Provides debug information in development mode -- Tracks request IDs for error correlation -""" - -import logging -import traceback -import uuid -from django.conf import settings -from rest_framework.views import exception_handler as drf_exception_handler -from rest_framework import status -from rest_framework.response import Response -from igny8_core.api.response import error_response - -logger = logging.getLogger(__name__) - - -def get_request_id(request): - """ - Get or create request ID for error tracking. - - Request ID can be set by middleware or generated here. - """ - # Check if request ID is already set (e.g., by middleware) - request_id = getattr(request, 'request_id', None) - - if not request_id: - # Generate new request ID - request_id = str(uuid.uuid4()) - request.request_id = request_id - - return request_id - - -def extract_error_message(exc, response): - """ - Extract user-friendly error message from exception. - - Args: - exc: The exception instance - response: DRF's exception handler response - - Returns: - tuple: (error_message, errors_dict) - """ - error_message = "An error occurred" - errors = None - - # Handle DRF exceptions with 'detail' attribute - if hasattr(exc, 'detail'): - if isinstance(exc.detail, dict): - # Validation errors - use as errors dict - errors = exc.detail - # Extract first error message as top-level error - if errors: - first_key = list(errors.keys())[0] - first_error = errors[first_key] - if isinstance(first_error, list) and first_error: - error_message = f"{first_key}: {first_error[0]}" - else: - error_message = f"{first_key}: {first_error}" - else: - error_message = "Validation failed" - elif isinstance(exc.detail, list): - # List of errors - error_message = exc.detail[0] if exc.detail else "An error occurred" - errors = {"non_field_errors": exc.detail} - else: - # String error message - error_message = str(exc.detail) - elif response and hasattr(response, 'data'): - # Try to extract from response data - if isinstance(response.data, dict): - # Check for common error message fields - error_message = ( - response.data.get('error') or - response.data.get('message') or - response.data.get('detail') or - str(response.data) - ) - errors = response.data if 'error' not in response.data else None - elif isinstance(response.data, list): - error_message = response.data[0] if response.data else "An error occurred" - errors = {"non_field_errors": response.data} - else: - error_message = str(response.data) if response.data else "An error occurred" - else: - # Generic exception - error_message = str(exc) if exc else "An error occurred" - - return error_message, errors - - -def get_status_code_message(status_code): - """ - Get default error message for HTTP status code. - """ - status_messages = { - status.HTTP_400_BAD_REQUEST: "Bad request", - status.HTTP_401_UNAUTHORIZED: "Authentication required", - status.HTTP_403_FORBIDDEN: "Permission denied", - status.HTTP_404_NOT_FOUND: "Resource not found", - status.HTTP_405_METHOD_NOT_ALLOWED: "Method not allowed", - status.HTTP_409_CONFLICT: "Conflict", - status.HTTP_422_UNPROCESSABLE_ENTITY: "Unprocessable entity", - status.HTTP_429_TOO_MANY_REQUESTS: "Too many requests", - status.HTTP_500_INTERNAL_SERVER_ERROR: "Internal server error", - status.HTTP_502_BAD_GATEWAY: "Bad gateway", - status.HTTP_503_SERVICE_UNAVAILABLE: "Service unavailable", - } - return status_messages.get(status_code, "An error occurred") - - -def custom_exception_handler(exc, context): - """ - Centralized exception handler that wraps all exceptions in unified format. - - This handler: - - Wraps all exceptions in unified error format - - Logs errors with request context - - Provides debug information in development mode - - Tracks request IDs for error correlation - - Args: - exc: The exception instance - context: Dictionary containing request, view, and args - - Returns: - Response: Error response in unified format, or None to use default - """ - # Get request from context - request = context.get('request') - view = context.get('view') - - # Get request ID for tracking - request_id = None - if request: - request_id = get_request_id(request) - - # Call DRF's default exception handler first - response = drf_exception_handler(exc, context) - - # Determine status code - if response is not None: - status_code = response.status_code - else: - # Unhandled exception - default to 500 - status_code = status.HTTP_500_INTERNAL_SERVER_ERROR - - # Extract error message and details - error_message, errors = extract_error_message(exc, response) - - # Use default message if error message is generic - if error_message == "An error occurred" or not error_message: - error_message = get_status_code_message(status_code) - - # Log the error - log_level = logging.ERROR if status_code >= 500 else logging.WARNING - log_message = f"API Error [{status_code}]: {error_message}" - - if request_id: - log_message += f" (Request ID: {request_id})" - - # Include exception details in log - logger.log( - log_level, - log_message, - extra={ - 'request_id': request_id, - 'status_code': status_code, - 'exception_type': type(exc).__name__, - 'view': view.__class__.__name__ if view else None, - 'path': request.path if request else None, - 'method': request.method if request else None, - }, - exc_info=status_code >= 500, # Include traceback for server errors - ) - - # Build error response - error_response_data = { - "success": False, - "error": error_message, - } - - # Add errors dict if present - if errors: - error_response_data["errors"] = errors - - # Add request ID for error tracking - if request_id: - error_response_data["request_id"] = request_id - - # Add debug information in development mode - if settings.DEBUG: - error_response_data["debug"] = { - "exception_type": type(exc).__name__, - "exception_message": str(exc), - "view": view.__class__.__name__ if view else None, - "path": request.path if request else None, - "method": request.method if request else None, - } - - # Include traceback in debug mode for server errors - if status_code >= 500: - error_response_data["debug"]["traceback"] = traceback.format_exc() - - return Response(error_response_data, status=status_code) -``` - -#### Step 1.2: Create Request ID Middleware (Optional but Recommended) - -**File:** `backend/igny8_core/api/middleware.py` - -**Implementation:** -```python -""" -Request ID Middleware for Error Tracking -""" - -import uuid -from django.utils.deprecation import MiddlewareMixin - - -class RequestIDMiddleware(MiddlewareMixin): - """ - Middleware that adds a unique request ID to each request. - - This request ID is used for error tracking and correlation. - """ - - def process_request(self, request): - """ - Generate and attach request ID to request. - """ - # Check if request ID is already set (e.g., by reverse proxy) - request_id = request.META.get('HTTP_X_REQUEST_ID') - - if not request_id: - # Generate new request ID - request_id = str(uuid.uuid4()) - - # Attach to request - request.request_id = request_id - - # Add to response headers - # This will be done in process_response - - def process_response(self, request, response): - """ - Add request ID to response headers. - """ - request_id = getattr(request, 'request_id', None) - if request_id: - response['X-Request-ID'] = request_id - - return response -``` - -**Register in settings:** -```python -MIDDLEWARE = [ - # ... other middleware ... - 'igny8_core.api.middleware.RequestIDMiddleware', - # ... other middleware ... -] -``` - -**Estimated Time:** 4 hours (handler) + 2 hours (middleware) = 6 hours - ---- - -## Task 2: Register in Global DRF Settings - -### Goal -Register the custom exception handler in Django REST Framework settings. - -### Implementation Steps - -#### Step 2.1: Update REST_FRAMEWORK Settings - -**File:** `backend/igny8_core/settings.py` - -**Update REST_FRAMEWORK configuration:** -```python -REST_FRAMEWORK = { - 'DEFAULT_PAGINATION_CLASS': 'igny8_core.api.pagination.CustomPageNumberPagination', - 'PAGE_SIZE': 10, - 'DEFAULT_FILTER_BACKENDS': [ - 'django_filters.rest_framework.DjangoFilterBackend', - 'rest_framework.filters.SearchFilter', - 'rest_framework.filters.OrderingFilter', - ], - 'DEFAULT_PERMISSION_CLASSES': [ - 'rest_framework.permissions.AllowAny', # Will be changed in Section 2 - ], - 'DEFAULT_AUTHENTICATION_CLASSES': [ - 'igny8_core.api.authentication.JWTAuthentication', - 'igny8_core.api.authentication.CSRFExemptSessionAuthentication', - 'rest_framework.authentication.BasicAuthentication', - ], - 'EXCEPTION_HANDLER': 'igny8_core.api.exception_handlers.custom_exception_handler', # Add this -} -``` - -**Estimated Time:** 1 hour - ---- - -## Task 3: Unify Validation Error Format - -### Goal -Ensure all validation errors (serializer and manual) use the unified error format. - -### Implementation Steps - -#### Step 3.1: Audit Validation Error Usage - -**Action Items:** -1. Search for manual validation in ViewSets -2. Identify places where `error_response` is not used -3. Document current validation error patterns -4. Create refactoring checklist - -**Files to Audit:** -- All ViewSets with custom `create`, `update`, or `@action` methods -- Custom validation logic in serializers -- Manual validation in views - -**Estimated Time:** 3 hours - -#### Step 3.2: Unify Validation Error Format - -**DRF Serializer Errors:** -DRF automatically handles serializer validation errors and passes them to the exception handler. The exception handler (Task 1) will wrap them in the unified format. - -**Manual Validation Errors:** - -**Before:** -```python -class KeywordViewSet(BaseTenantViewSet): - @action(detail=False, methods=['post']) - def bulk_delete(self, request): - ids = request.data.get('ids', []) - if not ids: - return Response({ - "error": "No IDs provided" - }, status=400) -``` - -**After:** -```python -from igny8_core.api.response import error_response - -class KeywordViewSet(BaseTenantViewSet): - @action(detail=False, methods=['post']) - def bulk_delete(self, request): - ids = request.data.get('ids', []) - if not ids: - return error_response( - error="No IDs provided", - errors={"ids": ["This field is required"]}, - status_code=status.HTTP_400_BAD_REQUEST - ) -``` - -**Serializer Validation Override (if needed):** - -**File:** `backend/igny8_core/api/serializers.py` (create if needed) - -```python -""" -Base Serializer with Unified Error Format -""" - -from rest_framework import serializers -from rest_framework.exceptions import ValidationError - - -class BaseSerializer(serializers.Serializer): - """ - Base serializer that ensures validation errors follow unified format. - - DRF's default validation error handling should work with the exception handler, - but this can be used if custom validation error formatting is needed. - """ - - def validate(self, attrs): - """ - Override to add custom validation logic. - """ - return attrs - - def to_internal_value(self, data): - """ - Override to customize validation error format if needed. - """ - try: - return super().to_internal_value(data) - except ValidationError as e: - # Validation errors are already in the correct format - # The exception handler will wrap them - raise - - -class BaseModelSerializer(serializers.ModelSerializer, BaseSerializer): - """ - Base model serializer with unified error format support. - """ - pass -``` - -**Estimated Time:** 4 hours - ---- - -## Task 4: Patch or Log Internal Server Errors - -### Goal -Implement comprehensive error logging for debugging and monitoring. - -### Implementation Steps - -#### Step 4.1: Configure Error Logging - -**File:** `backend/igny8_core/settings.py` - -**Add/Update LOGGING configuration:** -```python -LOGGING = { - 'version': 1, - 'disable_existing_loggers': False, - 'formatters': { - 'verbose': { - 'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}', - 'style': '{', - }, - 'detailed': { - 'format': '{levelname} {asctime} [{request_id}] {module} {path} {method} {status_code} {message}', - 'style': '{', - }, - }, - 'filters': { - 'require_debug_false': { - '()': 'django.utils.log.RequireDebugFalse', - }, - }, - 'handlers': { - 'console': { - 'class': 'logging.StreamHandler', - 'formatter': 'verbose', - }, - 'file': { - 'class': 'logging.handlers.RotatingFileHandler', - 'filename': os.path.join(BASE_DIR, 'logs', 'django.log'), - 'maxBytes': 1024 * 1024 * 10, # 10 MB - 'backupCount': 5, - 'formatter': 'detailed', - }, - 'error_file': { - 'class': 'logging.handlers.RotatingFileHandler', - 'filename': os.path.join(BASE_DIR, 'logs', 'errors.log'), - 'maxBytes': 1024 * 1024 * 10, # 10 MB - 'backupCount': 10, - 'formatter': 'detailed', - 'level': 'ERROR', - }, - }, - 'loggers': { - 'django': { - 'handlers': ['console', 'file'], - 'level': 'INFO', - 'propagate': False, - }, - 'igny8_core.api': { - 'handlers': ['console', 'file', 'error_file'], - 'level': 'INFO', - 'propagate': False, - }, - 'django.request': { - 'handlers': ['error_file'], - 'level': 'ERROR', - 'propagate': False, - }, - }, - 'root': { - 'handlers': ['console'], - 'level': 'INFO', - }, -} -``` - -**Create logs directory:** -```bash -mkdir -p logs -touch logs/django.log logs/errors.log -``` - -**Estimated Time:** 3 hours - -#### Step 4.2: Add Request ID Tracking - -**Implementation:** -- Request ID middleware (Task 1.2) already adds request IDs -- Exception handler (Task 1.1) already logs request IDs -- Ensure request ID is included in all error responses - -**Estimated Time:** Already included in Task 1 - -#### Step 4.3: Integrate Error Tracking Service (Optional) - -**Sentry Integration Example:** - -**Install Sentry:** -```bash -pip install sentry-sdk -``` - -**File:** `backend/igny8_core/settings.py` - -**Add Sentry configuration:** -```python -import sentry_sdk -from sentry_sdk.integrations.django import DjangoIntegration -from sentry_sdk.integrations.logging import LoggingIntegration - -# Only enable Sentry in production -if not settings.DEBUG and os.getenv('SENTRY_DSN'): - sentry_sdk.init( - dsn=os.getenv('SENTRY_DSN'), - integrations=[ - DjangoIntegration(), - LoggingIntegration(level=logging.INFO, event_level=logging.ERROR), - ], - traces_sample_rate=0.1, # 10% of transactions - send_default_pii=True, # Include user information - environment=os.getenv('ENVIRONMENT', 'production'), - ) -``` - -**Update Exception Handler to Send to Sentry:** - -**File:** `backend/igny8_core/api/exception_handlers.py` - -**Add Sentry integration:** -```python -def custom_exception_handler(exc, context): - # ... existing code ... - - # Send to Sentry for server errors - if status_code >= 500: - try: - import sentry_sdk - with sentry_sdk.push_scope() as scope: - scope.set_tag("request_id", request_id) - scope.set_tag("status_code", status_code) - scope.set_context("request", { - "path": request.path if request else None, - "method": request.method if request else None, - "view": view.__class__.__name__ if view else None, - }) - sentry_sdk.capture_exception(exc) - except ImportError: - # Sentry not installed - pass - - # ... rest of handler ... -``` - -**Estimated Time:** 4 hours (optional) - ---- - -## Task 5: Add Developer-Focused Debug Support - -### Goal -Add debug information to error responses in development mode for easier debugging. - -### Implementation Steps - -#### Step 5.1: Enhance Exception Handler with Debug Mode - -**Implementation:** -The exception handler (Task 1.1) already includes debug information when `settings.DEBUG` is `True`. This includes: -- Exception type and message -- View class name -- Request path and method -- Full traceback for server errors - -**Additional Debug Features (Optional):** - -**File:** `backend/igny8_core/api/exception_handlers.py` - -**Add environment variable control:** -```python -def custom_exception_handler(exc, context): - # ... existing code ... - - # Add debug information in development mode - # Can be controlled by DEBUG setting or API_DEBUG environment variable - api_debug = settings.DEBUG or os.getenv('API_DEBUG', 'False').lower() == 'true' - - if api_debug: - error_response_data["debug"] = { - "exception_type": type(exc).__name__, - "exception_message": str(exc), - "view": view.__class__.__name__ if view else None, - "path": request.path if request else None, - "method": request.method if request else None, - "user": str(request.user) if request and hasattr(request, 'user') else None, - "account": str(request.account) if request and hasattr(request, 'account') else None, - } - - # Include traceback in debug mode for server errors - if status_code >= 500: - error_response_data["debug"]["traceback"] = traceback.format_exc() - - # Include request data (sanitized) - if request: - error_response_data["debug"]["request_data"] = { - "GET": dict(request.GET), - "POST": {k: str(v)[:100] for k, v in dict(request.POST).items()}, # Truncate long values - } - - # ... rest of handler ... -``` - -**Security Note:** -- Only include debug information when `DEBUG=True` or `API_DEBUG=True` -- Never include sensitive data (passwords, tokens) in debug output -- Sanitize request data before including in debug response - -**Estimated Time:** 3 hours - ---- - -## Task 6: Test Scenarios - -### Goal -Create comprehensive test scenarios to validate error handling works correctly. - -### Implementation Steps - -#### Step 6.1: Create Test Scenarios - -**Test Cases:** - -**1. Missing Required Field (400)** -```python -# Test: POST /api/v1/planner/keywords/ without required fields -# Expected: 400 Bad Request -{ - "success": false, - "error": "keyword: This field is required", - "errors": { - "keyword": ["This field is required"], - "site_id": ["This field is required"] - }, - "request_id": "abc-123-def" -} -``` - -**2. Invalid Token (401)** -```python -# Test: GET /api/v1/auth/me/ with invalid token -# Expected: 401 Unauthorized -{ - "success": false, - "error": "Authentication required", - "request_id": "abc-123-def" -} -``` - -**3. Permission Denied (403)** -```python -# Test: DELETE /api/v1/auth/users/1/ as non-admin user -# Expected: 403 Forbidden -{ - "success": false, - "error": "Permission denied", - "request_id": "abc-123-def" -} -``` - -**4. Not Found (404)** -```python -# Test: GET /api/v1/planner/keywords/99999/ (non-existent ID) -# Expected: 404 Not Found -{ - "success": false, - "error": "Resource not found", - "request_id": "abc-123-def" -} -``` - -**5. Server Crash (500)** -```python -# Test: Endpoint that raises unhandled exception -# Expected: 500 Internal Server Error -{ - "success": false, - "error": "Internal server error", - "request_id": "abc-123-def", - "debug": { # Only in DEBUG mode - "exception_type": "ValueError", - "exception_message": "...", - "traceback": "..." - } -} -``` - -**6. Manual Check Failure** -```python -# Test: Custom action with manual validation -# Expected: Custom error message -{ - "success": false, - "error": "No IDs provided", - "errors": { - "ids": ["This field is required"] - }, - "request_id": "abc-123-def" -} -``` - -**7. Validation Error with Multiple Fields** -```python -# Test: POST with multiple validation errors -# Expected: All errors in errors dict -{ - "success": false, - "error": "email: Invalid email format", - "errors": { - "email": ["Invalid email format"], - "password": ["Password must be at least 8 characters"], - "password_confirm": ["Passwords do not match"] - }, - "request_id": "abc-123-def" -} -``` - -#### Step 6.2: Create Automated Tests - -**File:** `backend/igny8_core/api/tests/test_exception_handlers.py` - -**Test Implementation:** -```python -""" -Tests for Exception Handler -""" - -from django.test import TestCase -from rest_framework.test import APIClient -from rest_framework import status -from django.contrib.auth import get_user_model -from igny8_core.auth.models import Account - -User = get_user_model() - - -class ExceptionHandlerTestCase(TestCase): - def setUp(self): - self.client = APIClient() - self.account = Account.objects.create(name="Test Account") - self.user = User.objects.create_user( - email="test@example.com", - password="testpass123", - account=self.account - ) - - def test_validation_error_format(self): - """Test that validation errors return unified format""" - response = self.client.post('/api/v1/planner/keywords/', {}) - self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST) - data = response.json() - self.assertFalse(data['success']) - self.assertIn('error', data) - self.assertIn('errors', data) - self.assertIn('request_id', data) - - def test_unauthorized_error_format(self): - """Test that 401 errors return unified format""" - response = self.client.get('/api/v1/auth/me/') - self.assertEqual(response.status_code, status.HTTP_401_UNAUTHORIZED) - data = response.json() - self.assertFalse(data['success']) - self.assertEqual(data['error'], 'Authentication required') - - def test_not_found_error_format(self): - """Test that 404 errors return unified format""" - self.client.force_authenticate(user=self.user) - response = self.client.get('/api/v1/planner/keywords/99999/') - self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND) - data = response.json() - self.assertFalse(data['success']) - self.assertEqual(data['error'], 'Resource not found') - - def test_request_id_present(self): - """Test that request ID is included in error responses""" - response = self.client.post('/api/v1/planner/keywords/', {}) - data = response.json() - self.assertIn('request_id', data) - self.assertIsNotNone(data['request_id']) - - def test_debug_info_in_development(self): - """Test that debug info is included when DEBUG=True""" - from django.conf import settings - original_debug = settings.DEBUG - settings.DEBUG = True - - try: - # Trigger a server error - response = self.client.get('/api/v1/nonexistent-endpoint/') - if response.status_code >= 500: - data = response.json() - if 'debug' in data: - self.assertIn('exception_type', data['debug']) - finally: - settings.DEBUG = original_debug -``` - -**Estimated Time:** 4 hours - ---- - -## Task 7: Changelog Entry - -### Goal -Document the error handling standardization changes. - -### Implementation Steps - -#### Step 7.1: Update CHANGELOG.md - -**File:** `CHANGELOG.md` (or similar) - -**Add Entry:** -```markdown -## [Unreleased] - 2025-01-XX - -### Changed -- **Error Handling**: Added centralized exception handler and standardized all API error formats - - Created `custom_exception_handler` in `igny8_core/api/exception_handlers.py` - - All exceptions now return unified error format: `{ success: false, error: "...", errors: {...}, request_id: "..." }` - - Added request ID tracking for error correlation - - Enhanced error logging with request context - - Added debug information in development mode (exception type, traceback, request details) - - Unified validation error format across all endpoints - -### Added -- **Request ID Middleware**: Added `RequestIDMiddleware` for error tracking and correlation -- **Error Logging**: Configured comprehensive error logging to files and console -- **Debug Support**: Added debug information in error responses when `DEBUG=True` - - Includes exception type, message, view, path, method - - Includes full traceback for server errors (500+) - - Includes sanitized request data - -### Security -- Debug information only included when `DEBUG=True` or `API_DEBUG=True` -- Sensitive data (passwords, tokens) excluded from debug output -- Request data sanitized before inclusion in debug responses - -### Affected Areas -- API Layer (`igny8_core/api/exception_handlers.py`, `igny8_core/api/middleware.py`) -- All API endpoints (error responses now unified) -- Error logging configuration (`settings.py`) - -### Migration Guide -1. **Frontend Updates:** - - Update error handling to check `success: false` field - - Use `error` field for top-level error message - - Use `errors` field for field-specific validation errors - - Use `request_id` for error reporting and support - -2. **Error Response Format:** - ```json - { - "success": false, - "error": "Top-level error message", - "errors": { - "field_name": ["Field-specific error messages"] - }, - "request_id": "uuid-for-tracking", - "debug": { // Only in DEBUG mode - "exception_type": "ValidationError", - "traceback": "..." - } - } - ``` - -3. **Testing:** - - All error responses now follow unified format - - Request IDs are included for error tracking - - Debug information available in development mode -``` - -**Estimated Time:** 1 hour - ---- - -## Testing Strategy - -### Unit Tests - -**File:** `backend/igny8_core/api/tests/test_exception_handlers.py` - -**Test Cases:** -- Exception handler wraps DRF exceptions correctly -- Error message extraction from various exception types -- Request ID generation and inclusion -- Debug information inclusion in development mode -- Status code mapping to error messages -- Validation error format handling - -### Integration Tests - -**Test Cases:** -- End-to-end API calls that trigger various error scenarios -- Error response format validation -- Request ID propagation -- Error logging verification -- Debug information in development mode - -### Manual Testing - -**Checklist:** -- [ ] Validation errors return unified format -- [ ] Authentication errors return unified format -- [ ] Permission errors return unified format -- [ ] Not found errors return unified format -- [ ] Server errors return unified format -- [ ] Request ID is included in all error responses -- [ ] Error logging works correctly -- [ ] Debug information appears in development mode -- [ ] Debug information is excluded in production mode -- [ ] Sensitive data is not included in debug output - ---- - -## Rollout Plan - -### Phase 1: Foundation (Week 1) -- ✅ Task 1: Create centralized exception handler -- ✅ Task 2: Register exception handler in settings -- ✅ Unit tests for exception handler - -### Phase 2: Logging & Debug (Week 2) -- ✅ Task 4: Configure error logging -- ✅ Task 5: Add debug support -- ✅ Task 1.2: Add request ID middleware (optional) - -### Phase 3: Validation & Testing (Week 3) -- ✅ Task 3: Unify validation error format -- ✅ Task 6: Create test scenarios -- ✅ Integration testing - -### Phase 4: Documentation & Release (Week 4) -- ✅ Task 7: Changelog entry -- ✅ Documentation updates -- ✅ Release to staging -- ✅ Production deployment - ---- - -## Success Criteria - -### Definition of Done - -1. ✅ Centralized exception handler is created and registered -2. ✅ All exceptions return unified error format -3. ✅ Request ID tracking is implemented -4. ✅ Error logging is configured and working -5. ✅ Debug information is available in development mode -6. ✅ Validation errors use unified format -7. ✅ All test scenarios pass -8. ✅ Documentation is updated -9. ✅ Changelog entry is created -10. ✅ No sensitive data in debug output - -### Metrics - -- **Coverage:** 100% of exceptions use unified format -- **Logging:** All errors are logged with request context -- **Test Coverage:** >90% for exception handler -- **Debug Support:** Debug information available in development mode - ---- - -## Risk Assessment - -### Risks - -1. **Debug Information Leakage:** Debug info might leak sensitive data - - **Mitigation:** Only include debug info when `DEBUG=True`, sanitize request data - -2. **Performance Impact:** Exception handling and logging may add overhead - - **Mitigation:** Minimal overhead, use async logging if needed - -3. **Breaking Changes:** Frontend may expect different error format - - **Mitigation:** Document new format, provide migration guide - -4. **Log File Growth:** Error logs may grow large - - **Mitigation:** Use rotating file handlers, set up log rotation - -### Rollback Plan - -If issues arise: -1. Revert exception handler registration (use DRF default) -2. Keep response format utilities (non-breaking) -3. Disable debug mode if sensitive data is exposed -4. Document issues for future fixes - ---- - -## Appendix - -### Error Response Format Examples - -#### Validation Error (400) -```json -{ - "success": false, - "error": "email: Invalid email format", - "errors": { - "email": ["Invalid email format"], - "password": ["Password must be at least 8 characters"] - }, - "request_id": "550e8400-e29b-41d4-a716-446655440000" -} -``` - -#### Authentication Error (401) -```json -{ - "success": false, - "error": "Authentication required", - "request_id": "550e8400-e29b-41d4-a716-446655440000" -} -``` - -#### Permission Error (403) -```json -{ - "success": false, - "error": "Permission denied", - "request_id": "550e8400-e29b-41d4-a716-446655440000" -} -``` - -#### Not Found Error (404) -```json -{ - "success": false, - "error": "Resource not found", - "request_id": "550e8400-e29b-41d4-a716-446655440000" -} -``` - -#### Server Error (500) - Production -```json -{ - "success": false, - "error": "Internal server error", - "request_id": "550e8400-e29b-41d4-a716-446655440000" -} -``` - -#### Server Error (500) - Development -```json -{ - "success": false, - "error": "Internal server error", - "request_id": "550e8400-e29b-41d4-a716-446655440000", - "debug": { - "exception_type": "ValueError", - "exception_message": "Invalid value", - "view": "KeywordViewSet", - "path": "/api/v1/planner/keywords/", - "method": "POST", - "traceback": "Traceback (most recent call last):\n ..." - } -} -``` - ---- - -**Document Status:** Implementation Plan -**Last Updated:** 2025-01-XX -**Next Review:** After Phase 1 completion - diff --git a/unified-api/API-IMPLEMENTATION-PLAN-SECTION4.md b/unified-api/API-IMPLEMENTATION-PLAN-SECTION4.md deleted file mode 100644 index f0e95600..00000000 --- a/unified-api/API-IMPLEMENTATION-PLAN-SECTION4.md +++ /dev/null @@ -1,1192 +0,0 @@ -# API Implementation Plan - Section 4: Apply Rate Limiting and Throttling Rules - -**Date:** 2025-01-XX -**Status:** Planning -**Priority:** High -**Related Document:** `API-ENDPOINTS-ANALYSIS.md` - ---- - -## Executive Summary - -This document outlines the implementation plan for **Section 4: Apply Rate Limiting and Throttling Rules** across the IGNY8 API layer. The goal is to prevent abuse, ensure fair resource usage, and enforce plan limits by introducing consistent per-user and per-function rate limits using DRF throttling and internal usage tracking. - -**Key Objectives:** -- Enable DRF throttling system with scoped rate limits -- Assign throttle scopes to all major functions (AI, auth, content, images) -- Implement plan-based throttling (optional, future phase) -- Log and expose rate limit information -- Integrate with existing credit system for quota enforcement - ---- - -## Table of Contents - -1. [Current State Analysis](#current-state-analysis) -2. [Implementation Tasks](#implementation-tasks) -3. [Task 1: Enable DRF Throttling System](#task-1-enable-drf-throttling-system) -4. [Task 2: Assign Throttle Scopes in ViewSets](#task-2-assign-throttle-scopes-in-viewsets) -5. [Task 3: Override User/Account-Level Throttles (Optional)](#task-3-override-useraccount-level-throttles-optional) -6. [Task 4: Log + Expose Rate Limit Info](#task-4-log--expose-rate-limit-info) -7. [Task 5: Advanced Daily/Monthly Quotas Per Plan](#task-5-advanced-dailymonthly-quotas-per-plan) -8. [Task 6: Testing Matrix](#task-6-testing-matrix) -9. [Task 7: Changelog Entry](#task-7-changelog-entry) -10. [Testing Strategy](#testing-strategy) -11. [Rollout Plan](#rollout-plan) -12. [Success Criteria](#success-criteria) - ---- - -## Current State Analysis - -### Current Rate Limiting Status - -Based on `API-ENDPOINTS-ANALYSIS.md`: - -1. **Rate Limiting:** ❌ **Not Implemented** - - All endpoints are currently unlimited - - No throttling configuration in settings - - No rate limit enforcement - -2. **Credit System:** - - `CreditUsageLog` model exists for tracking usage - - `CreditBalance` model tracks account credits - - Credit deduction happens but no rate limiting based on credits - -3. **Plan System:** - - `Plan` model exists with different tiers - - Plans have credit limits but no API rate limits defined - - No plan-based throttling implemented - -4. **AI Functions:** - - High-cost operations (AI generation, image generation) - - Currently no rate limiting on these expensive endpoints - - Risk of abuse and cost overruns - ---- - -## Implementation Tasks - -### Overview - -| Task ID | Task Name | Priority | Estimated Effort | Dependencies | -|---------|-----------|----------|------------------|--------------| -| 1.1 | Configure DRF throttling in settings | High | 2 hours | None | -| 1.2 | Add debug throttle bypass | High | 1 hour | 1.1 | -| 2.1 | Audit endpoints for throttle scopes | Medium | 3 hours | None | -| 2.2 | Assign throttle scopes to ViewSets | High | 6 hours | 1.1 | -| 3.1 | Create plan-based throttle class (optional) | Low | 8 hours | 1.1 | -| 4.1 | Verify throttle headers | High | 2 hours | 2.2 | -| 4.2 | Add frontend rate limit handling | Medium | 4 hours | 4.1 | -| 4.3 | Implement abuse detection logging | Medium | 3 hours | 2.2 | -| 5.1 | Integrate with credit system | Low | 6 hours | 3.1 | -| 6.1 | Create test scenarios | High | 4 hours | 2.2 | -| 7.1 | Create changelog entry | Low | 1 hour | All tasks | - -**Total Estimated Effort:** ~40 hours - ---- - -## Task 1: Enable DRF Throttling System - -### Goal -Configure Django REST Framework to use scoped rate throttling with appropriate rate limits for different endpoint types. - -### Implementation Steps - -#### Step 1.1: Configure DRF Throttling in Settings - -**File:** `backend/igny8_core/settings.py` - -**Update REST_FRAMEWORK configuration:** -```python -REST_FRAMEWORK = { - 'DEFAULT_PAGINATION_CLASS': 'igny8_core.api.pagination.CustomPageNumberPagination', - 'PAGE_SIZE': 10, - 'DEFAULT_FILTER_BACKENDS': [ - 'django_filters.rest_framework.DjangoFilterBackend', - 'rest_framework.filters.SearchFilter', - 'rest_framework.filters.OrderingFilter', - ], - 'DEFAULT_PERMISSION_CLASSES': [ - 'rest_framework.permissions.AllowAny', # Will be changed in Section 2 - ], - 'DEFAULT_AUTHENTICATION_CLASSES': [ - 'igny8_core.api.authentication.JWTAuthentication', - 'igny8_core.api.authentication.CSRFExemptSessionAuthentication', - 'rest_framework.authentication.BasicAuthentication', - ], - 'EXCEPTION_HANDLER': 'igny8_core.api.exception_handlers.custom_exception_handler', - # Throttling Configuration - 'DEFAULT_THROTTLE_CLASSES': [ - 'rest_framework.throttling.ScopedRateThrottle', - ], - 'DEFAULT_THROTTLE_RATES': { - # AI Functions - Expensive operations - 'ai_function': '10/min', # AI content generation, clustering - 'image_gen': '15/min', # Image generation - - # Content Operations - 'content_write': '30/min', # Content creation, updates - 'content_read': '100/min', # Content listing, retrieval - - # Authentication - 'auth': '20/min', # Login, register, password reset - 'auth_strict': '5/min', # Sensitive auth operations - - # Planner Operations - 'planner': '60/min', # Keyword, cluster, idea operations - 'planner_ai': '10/min', # AI-powered planner operations - - # Writer Operations - 'writer': '60/min', # Task, content management - 'writer_ai': '10/min', # AI-powered writer operations - - # System Operations - 'system': '100/min', # Settings, prompts, profiles - 'system_admin': '30/min', # Admin-only system operations - - # Billing Operations - 'billing': '30/min', # Credit queries, usage logs - 'billing_admin': '10/min', # Credit management (admin) - - # Default fallback - 'default': '100/min', # Default for endpoints without scope - }, -} -``` - -#### Step 1.2: Add Debug Throttle Bypass - -**File:** `backend/igny8_core/settings.py` - -**Add environment variable:** -```python -import os - -# Throttling Configuration -IGNY8_DEBUG_THROTTLE = os.getenv('IGNY8_DEBUG_THROTTLE', 'False').lower() == 'true' - -# Only apply throttling if not in debug mode or explicitly enabled -if not DEBUG and not IGNY8_DEBUG_THROTTLE: - REST_FRAMEWORK['DEFAULT_THROTTLE_CLASSES'] = [ - 'rest_framework.throttling.ScopedRateThrottle', - ] -else: - # In debug mode, use a no-op throttle class - REST_FRAMEWORK['DEFAULT_THROTTLE_CLASSES'] = [] -``` - -**Create Custom Throttle Class with Debug Bypass:** - -**File:** `backend/igny8_core/api/throttles.py` - -**Implementation:** -```python -""" -Custom Throttle Classes for IGNY8 API -""" - -from django.conf import settings -from rest_framework.throttling import ScopedRateThrottle - - -class DebugScopedRateThrottle(ScopedRateThrottle): - """ - Scoped rate throttle that can be bypassed in debug mode. - - Usage: - throttle_scope = 'ai_function' - """ - - def allow_request(self, request, view): - """ - Check if request should be throttled. - Bypass throttling if DEBUG=True or IGNY8_DEBUG_THROTTLE=True. - """ - # Bypass throttling in debug mode - if settings.DEBUG or getattr(settings, 'IGNY8_DEBUG_THROTTLE', False): - return True - - # Use parent class throttling logic - return super().allow_request(request, view) -``` - -**Update settings to use custom throttle:** -```python -'DEFAULT_THROTTLE_CLASSES': [ - 'igny8_core.api.throttles.DebugScopedRateThrottle', -], -``` - -**Estimated Time:** 2 hours (settings) + 1 hour (debug bypass) = 3 hours - ---- - -## Task 2: Assign Throttle Scopes in ViewSets - -### Goal -Assign appropriate throttle scopes to all ViewSets and custom actions based on their function and resource cost. - -### Implementation Steps - -#### Step 2.1: Audit Endpoints for Throttle Scopes - -**Action Items:** -1. List all ViewSets and custom actions -2. Categorize by function type (AI, content, auth, etc.) -3. Identify high-cost operations (AI, image generation) -4. Create mapping of endpoints to throttle scopes - -**Endpoint Categories:** - -**AI Functions (10/min):** -- `POST /api/v1/planner/keywords/auto_cluster/` - AI clustering -- `POST /api/v1/planner/clusters/auto_generate_ideas/` - AI idea generation -- `POST /api/v1/writer/tasks/auto_generate_content/` - AI content generation -- `POST /api/v1/writer/content/generate_image_prompts/` - AI prompt generation - -**Image Generation (15/min):** -- `POST /api/v1/writer/images/generate_images/` - Image generation -- `POST /api/v1/writer/images/auto_generate/` - Legacy image generation - -**Content Write (30/min):** -- `POST /api/v1/writer/tasks/` - Create task -- `POST /api/v1/writer/content/` - Create content -- `PUT /api/v1/writer/tasks/{id}/` - Update task -- `PUT /api/v1/writer/content/{id}/` - Update content - -**Planner AI (10/min):** -- `POST /api/v1/planner/keywords/auto_cluster/` -- `POST /api/v1/planner/clusters/auto_generate_ideas/` - -**Planner Standard (60/min):** -- `GET /api/v1/planner/keywords/` - List keywords -- `POST /api/v1/planner/keywords/` - Create keyword -- `GET /api/v1/planner/clusters/` - List clusters -- `POST /api/v1/planner/ideas/` - Create idea - -**Auth (20/min):** -- `POST /api/v1/auth/login/` -- `POST /api/v1/auth/register/` -- `POST /api/v1/auth/change-password/` - -**Auth Strict (5/min):** -- `POST /api/v1/auth/password-reset/` - If exists -- `POST /api/v1/auth/users/invite/` - User invitation - -**System Admin (30/min):** -- `POST /api/v1/system/settings/integrations/{pk}/save/` -- `POST /api/v1/system/settings/integrations/{pk}/test/` - -**Billing Admin (10/min):** -- `POST /api/v1/billing/credits/transactions/` - Create transaction - -**Estimated Time:** 3 hours - -#### Step 2.2: Assign Throttle Scopes to ViewSets - -**Example Implementation:** - -**Planner Module:** - -**File:** `backend/planner/api/viewsets.py` - -```python -from igny8_core.api.viewsets import BaseTenantViewSet - -class KeywordViewSet(BaseTenantViewSet): - throttle_scope = 'planner' # Default for standard operations - queryset = Keyword.objects.all() - serializer_class = KeywordSerializer - - @action( - detail=False, - methods=['post'], - throttle_scope='ai_function' # Override for AI operations - ) - def auto_cluster(self, request): - # AI clustering logic - pass - - -class ClusterViewSet(BaseTenantViewSet): - throttle_scope = 'planner' - queryset = Cluster.objects.all() - serializer_class = ClusterSerializer - - @action( - detail=False, - methods=['post'], - throttle_scope='planner_ai' # AI-powered operations - ) - def auto_generate_ideas(self, request): - # AI idea generation logic - pass -``` - -**Writer Module:** - -**File:** `backend/writer/api/viewsets.py` - -```python -class TasksViewSet(BaseTenantViewSet): - throttle_scope = 'writer' # Default for standard operations - queryset = Task.objects.all() - serializer_class = TaskSerializer - - @action( - detail=False, - methods=['post'], - throttle_scope='ai_function' # AI content generation - ) - def auto_generate_content(self, request): - # AI content generation logic - pass - - -class ImagesViewSet(BaseTenantViewSet): - throttle_scope = 'writer' - queryset = Image.objects.all() - serializer_class = ImageSerializer - - @action( - detail=False, - methods=['post'], - throttle_scope='image_gen' # Image generation - ) - def generate_images(self, request): - # Image generation logic - pass -``` - -**Auth Module:** - -**File:** `backend/auth/api/views.py` - -```python -from rest_framework.views import APIView -from rest_framework.throttling import ScopedRateThrottle - -class LoginView(APIView): - throttle_scope = 'auth' - throttle_classes = [DebugScopedRateThrottle] - - def post(self, request): - # Login logic - pass - - -class RegisterView(APIView): - throttle_scope = 'auth' - throttle_classes = [DebugScopedRateThrottle] - - def post(self, request): - # Registration logic - pass - - -class ChangePasswordView(APIView): - throttle_scope = 'auth' - throttle_classes = [DebugScopedRateThrottle] - - def post(self, request): - # Password change logic - pass -``` - -**System Module:** - -**File:** `backend/system/api/viewsets.py` - -```python -class IntegrationSettingsViewSet(BaseTenantViewSet): - throttle_scope = 'system' - - @action( - detail=True, - methods=['post'], - throttle_scope='system_admin' # Admin operations - ) - def save(self, request, pk=None): - # Save integration settings - pass - - @action( - detail=True, - methods=['post'], - throttle_scope='system_admin' - ) - def test(self, request, pk=None): - # Test integration connection - pass -``` - -**Billing Module:** - -**File:** `backend/billing/api/viewsets.py` - -```python -class CreditTransactionViewSet(BaseTenantViewSet): - throttle_scope = 'billing' - - # Read-only operations use 'billing' scope - # Write operations would use 'billing_admin' if implemented -``` - -**Estimated Time:** 6 hours - ---- - -## Task 3: Override User/Account-Level Throttles (Optional) - -### Goal -Implement plan-based throttling that applies different rate limits based on user's subscription plan. - -### Implementation Steps - -#### Step 3.1: Create Plan-Based Throttle Class - -**File:** `backend/igny8_core/api/throttles.py` - -**Implementation:** -```python -""" -Custom Throttle Classes for IGNY8 API -""" - -from django.conf import settings -from rest_framework.throttling import SimpleRateThrottle - - -class PlanBasedRateThrottle(SimpleRateThrottle): - """ - Rate throttle that applies different limits based on user's plan. - - Plan tiers: - - Free: 10/min for AI functions - - Pro: 100/min for AI functions - - Enterprise: Unlimited (or very high limit) - - Usage: - throttle_scope = 'ai_function' # Base scope - # Plan limits override base scope - """ - - # Plan-based rate limits - PLAN_RATES = { - 'free': { - 'ai_function': '10/min', - 'image_gen': '15/min', - 'content_write': '30/min', - }, - 'pro': { - 'ai_function': '100/min', - 'image_gen': '50/min', - 'content_write': '100/min', - }, - 'enterprise': { - 'ai_function': '1000/min', # Effectively unlimited - 'image_gen': '200/min', - 'content_write': '500/min', - }, - } - - def get_rate(self): - """ - Get rate limit based on user's plan. - """ - # Get scope from view - scope = getattr(self.view, 'throttle_scope', None) - if not scope: - return None - - # Get user's plan - user = self.request.user - if not user or not user.is_authenticated: - # Anonymous users use default rate - return self.get_default_rate(scope) - - # Get account and plan - account = getattr(user, 'account', None) - if not account: - return self.get_default_rate(scope) - - plan = getattr(account, 'plan', None) - if not plan: - return self.get_default_rate(scope) - - plan_name = plan.name.lower() if hasattr(plan, 'name') else 'free' - - # Get rate for plan and scope - plan_rates = self.PLAN_RATES.get(plan_name, self.PLAN_RATES['free']) - rate = plan_rates.get(scope, self.get_default_rate(scope)) - - return rate - - def get_default_rate(self, scope): - """ - Get default rate from settings if plan-based rate not found. - """ - throttle_rates = getattr(settings, 'REST_FRAMEWORK', {}).get('DEFAULT_THROTTLE_RATES', {}) - return throttle_rates.get(scope, '100/min') - - def get_cache_key(self, request, view): - """ - Generate cache key based on user ID and scope. - """ - if request.user.is_authenticated: - ident = request.user.pk - else: - ident = self.get_ident(request) - - scope = getattr(view, 'throttle_scope', None) - return self.cache_format % { - 'scope': scope, - 'ident': ident - } -``` - -#### Step 3.2: Apply Plan-Based Throttling - -**Update settings to use plan-based throttle for AI functions:** - -**File:** `backend/igny8_core/settings.py` - -```python -# For AI functions, use plan-based throttling -# For other functions, use scoped throttling - -# This would require custom throttle class selection per ViewSet -# Or use a mixin that selects throttle class based on scope -``` - -**Alternative: Use in specific ViewSets:** - -```python -from igny8_core.api.throttles import PlanBasedRateThrottle - -class TasksViewSet(BaseTenantViewSet): - throttle_scope = 'ai_function' - throttle_classes = [PlanBasedRateThrottle] # Use plan-based for this ViewSet - - @action(detail=False, methods=['post']) - def auto_generate_content(self, request): - # AI content generation - pass -``` - -**Estimated Time:** 8 hours (optional, future phase) - ---- - -## Task 4: Log + Expose Rate Limit Info - -### Goal -Ensure rate limit information is properly exposed to clients and logged for abuse detection. - -### Implementation Steps - -#### Step 4.1: Verify Throttle Headers - -**DRF automatically adds these headers:** -- `X-Throttle-Limit`: Maximum number of requests allowed -- `X-Throttle-Remaining`: Number of requests remaining -- `Retry-After`: Seconds to wait before retrying (when throttled) - -**Test in Postman:** -1. Make requests to throttled endpoint -2. Check response headers -3. Verify headers are present and correct - -**Example Response Headers:** -``` -HTTP/1.1 200 OK -X-Throttle-Limit: 10 -X-Throttle-Remaining: 9 -X-Throttle-Reset: 1640995200 -``` - -**When Throttled (429):** -``` -HTTP/1.1 429 Too Many Requests -X-Throttle-Limit: 10 -X-Throttle-Remaining: 0 -Retry-After: 60 -``` - -**Estimated Time:** 2 hours - -#### Step 4.2: Add Frontend Rate Limit Handling - -**File:** `frontend/src/services/api.ts` - -**Update fetchAPI to handle rate limits:** -```typescript -export async function fetchAPI(endpoint: string, options?: RequestInit) { - const response = await fetch(`${API_BASE_URL}${endpoint}`, { - ...options, - headers: { - ...options?.headers, - 'Authorization': `Bearer ${getToken()}`, - }, - }); - - // Check for rate limiting - if (response.status === 429) { - const retryAfter = response.headers.get('Retry-After'); - const throttleLimit = response.headers.get('X-Throttle-Limit'); - const throttleRemaining = response.headers.get('X-Throttle-Remaining'); - - // Show user-friendly error - showNotification( - `Rate limit exceeded. Please wait ${retryAfter} seconds before trying again.`, - 'error' - ); - - // Store rate limit info for UI - if (throttleLimit && throttleRemaining !== null) { - storeRateLimitInfo({ - limit: parseInt(throttleLimit), - remaining: parseInt(throttleRemaining), - retryAfter: retryAfter ? parseInt(retryAfter) : 60, - }); - } - - throw new Error('Rate limit exceeded'); - } - - // Check throttle headers on successful requests - const throttleLimit = response.headers.get('X-Throttle-Limit'); - const throttleRemaining = response.headers.get('X-Throttle-Remaining'); - - if (throttleLimit && throttleRemaining !== null) { - const remaining = parseInt(throttleRemaining); - const limit = parseInt(throttleLimit); - const percentage = (remaining / limit) * 100; - - // Show warning if close to limit - if (percentage < 20) { - showNotification( - `Warning: You have ${remaining} of ${limit} requests remaining.`, - 'warning' - ); - } - - // Store for UI display - storeRateLimitInfo({ - limit, - remaining, - percentage, - }); - } - - return response; -} -``` - -**Add Rate Limit Display Component:** - -**File:** `frontend/src/components/RateLimitIndicator.tsx` - -```typescript -import { useRateLimitStore } from '@/stores/rateLimitStore'; - -export function RateLimitIndicator() { - const { limit, remaining, percentage } = useRateLimitStore(); - - if (!limit) return null; - - return ( -
- API Calls: {remaining} / {limit} -
-
-
- {percentage < 20 && ( - Low on requests - )} -
- ); -} -``` - -**Estimated Time:** 4 hours - -#### Step 4.3: Implement Abuse Detection Logging - -**File:** `backend/igny8_core/api/middleware.py` - -**Add rate limit logging middleware:** -```python -""" -Rate Limit Logging Middleware -""" - -import logging -from django.utils.deprecation import MiddlewareMixin - -logger = logging.getLogger(__name__) - - -class RateLimitLoggingMiddleware(MiddlewareMixin): - """ - Middleware that logs rate limit hits for abuse detection. - """ - - def process_response(self, request, response): - """ - Log rate limit hits (429 responses). - """ - if response.status_code == 429: - # Log rate limit hit - logger.warning( - f"Rate limit exceeded: {request.method} {request.path}", - extra={ - 'user_id': request.user.id if request.user.is_authenticated else None, - 'ip_address': self.get_client_ip(request), - 'user_agent': request.META.get('HTTP_USER_AGENT'), - 'throttle_scope': getattr(request, 'throttle_scope', None), - } - ) - - # Check for repeated hits (potential abuse) - # This could be enhanced with Redis/cache to track frequency - self.check_for_abuse(request) - - return response - - def get_client_ip(self, request): - """ - Get client IP address. - """ - x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR') - if x_forwarded_for: - ip = x_forwarded_for.split(',')[0] - else: - ip = request.META.get('REMOTE_ADDR') - return ip - - def check_for_abuse(self, request): - """ - Check if user is repeatedly hitting rate limits (potential abuse). - """ - # TODO: Implement abuse detection logic - # - Track rate limit hits per user/IP - # - Alert if threshold exceeded - # - Optionally block user/IP temporarily - pass -``` - -**Register middleware:** - -**File:** `backend/igny8_core/settings.py` - -```python -MIDDLEWARE = [ - # ... other middleware ... - 'igny8_core.api.middleware.RateLimitLoggingMiddleware', - # ... other middleware ... -] -``` - -**Estimated Time:** 3 hours - ---- - -## Task 5: Advanced Daily/Monthly Quotas Per Plan - -### Goal -Implement quota-based usage limits that go beyond rate limiting, integrated with the existing credit system. - -### Implementation Steps - -#### Step 5.1: Integrate with Credit System - -**This task extends the existing credit system to enforce quotas:** - -**File:** `backend/igny8_core/api/middleware.py` - -**Add quota check middleware:** -```python -""" -Quota Enforcement Middleware -""" - -from django.http import JsonResponse -from rest_framework import status -from igny8_core.api.response import error_response - - -class QuotaEnforcementMiddleware(MiddlewareMixin): - """ - Middleware that enforces daily/monthly quotas based on plan and credits. - """ - - def process_request(self, request): - """ - Check if user has exceeded quota before processing request. - """ - # Only check for authenticated users - if not request.user or not request.user.is_authenticated: - return None - - # Get account - account = getattr(request.user, 'account', None) - if not account: - return None - - # Check if endpoint requires quota check - if not self.requires_quota_check(request): - return None - - # Check credits - from igny8_core.modules.billing.models import CreditBalance - - try: - balance = CreditBalance.objects.get(account=account) - - # Check if credits are exhausted - if balance.credits <= 0: - return error_response( - error="Credits exhausted. Please purchase more credits to continue.", - status_code=status.HTTP_402_PAYMENT_REQUIRED - ) - - # Check daily/monthly quotas (if implemented) - # This would require additional quota tracking models - - except CreditBalance.DoesNotExist: - # No balance record - allow request (will be created on first use) - pass - - return None - - def requires_quota_check(self, request): - """ - Check if this endpoint requires quota validation. - """ - # Check if it's an AI function or expensive operation - throttle_scope = getattr(request, 'throttle_scope', None) - expensive_scopes = ['ai_function', 'image_gen', 'planner_ai', 'writer_ai'] - - return throttle_scope in expensive_scopes -``` - -**Estimated Time:** 6 hours (optional, future phase) - ---- - -## Task 6: Testing Matrix - -### Goal -Create comprehensive test scenarios to validate rate limiting works correctly. - -### Implementation Steps - -#### Step 6.1: Create Test Scenarios - -**Test Cases:** - -**1. AI Function Rate Limiting (10/min)** -```python -# Test: Make 11 requests to /api/v1/writer/tasks/auto_generate_content/ -# Expected: -# - Requests 1-10: 200 OK -# - Request 11: 429 Too Many Requests -# - Headers: X-Throttle-Limit: 10, X-Throttle-Remaining: 0, Retry-After: 60 -``` - -**2. Image Generation Rate Limiting (15/min)** -```python -# Test: Make 16 requests to /api/v1/writer/images/generate_images/ -# Expected: -# - Requests 1-15: 200 OK -# - Request 16: 429 Too Many Requests -``` - -**3. Auth Rate Limiting (20/min)** -```python -# Test: Make 21 login attempts -# Expected: -# - Requests 1-20: 200 OK or 401 Unauthorized (depending on credentials) -# - Request 21: 429 Too Many Requests -``` - -**4. Debug Mode Bypass** -```python -# Test: Set IGNY8_DEBUG_THROTTLE=True -# Expected: All requests allowed regardless of rate limits -``` - -**5. Throttle Headers Verification** -```python -# Test: Make requests and check headers -# Expected: -# - X-Throttle-Limit header present -# - X-Throttle-Remaining header present and decrements -# - Retry-After header present when throttled -``` - -**6. Plan-Based Throttling (if implemented)** -```python -# Test: Free plan user vs Pro plan user -# Expected: -# - Free plan: 10/min for AI functions -# - Pro plan: 100/min for AI functions -``` - -**Automated Test Implementation:** - -**File:** `backend/igny8_core/api/tests/test_throttling.py` - -```python -""" -Tests for Rate Limiting and Throttling -""" - -from django.test import TestCase -from rest_framework.test import APIClient -from rest_framework import status -from django.contrib.auth import get_user_model -from igny8_core.auth.models import Account - -User = get_user_model() - - -class ThrottlingTestCase(TestCase): - def setUp(self): - self.client = APIClient() - self.account = Account.objects.create(name="Test Account") - self.user = User.objects.create_user( - email="test@example.com", - password="testpass123", - account=self.account - ) - self.client.force_authenticate(user=self.user) - - def test_ai_function_rate_limit(self): - """Test that AI functions are rate limited to 10/min""" - endpoint = '/api/v1/writer/tasks/auto_generate_content/' - - # Make 10 requests (should all succeed) - for i in range(10): - response = self.client.post(endpoint, {}) - self.assertIn(response.status_code, [200, 400, 500]) # Not throttled - - # 11th request should be throttled - response = self.client.post(endpoint, {}) - self.assertEqual(response.status_code, status.HTTP_429_TOO_MANY_REQUESTS) - self.assertIn('Retry-After', response.headers) - - def test_throttle_headers_present(self): - """Test that throttle headers are present in responses""" - endpoint = '/api/v1/writer/tasks/auto_generate_content/' - response = self.client.post(endpoint, {}) - - self.assertIn('X-Throttle-Limit', response.headers) - self.assertIn('X-Throttle-Remaining', response.headers) - - def test_debug_throttle_bypass(self): - """Test that throttling is bypassed in debug mode""" - from django.conf import settings - original_debug = settings.DEBUG - - try: - settings.DEBUG = True - - # Make 20 requests (should all succeed in debug mode) - endpoint = '/api/v1/writer/tasks/auto_generate_content/' - for i in range(20): - response = self.client.post(endpoint, {}) - self.assertNotEqual(response.status_code, status.HTTP_429_TOO_MANY_REQUESTS) - finally: - settings.DEBUG = original_debug -``` - -**Estimated Time:** 4 hours - ---- - -## Task 7: Changelog Entry - -### Goal -Document the rate limiting implementation in the changelog. - -### Implementation Steps - -#### Step 7.1: Update CHANGELOG.md - -**File:** `CHANGELOG.md` - -**Add Entry:** -```markdown -## [Unreleased] - 2025-01-XX - -### Added -- **Rate Limiting**: Added scoped API rate limiting for AI, image, auth, and content actions - - Configured DRF throttling with scoped rate limits - - AI functions: 10 requests/minute - - Image generation: 15 requests/minute - - Content operations: 30 requests/minute - - Authentication: 20 requests/minute - - Added `DebugScopedRateThrottle` with debug mode bypass - - Rate limit information exposed via `X-Throttle-Limit` and `X-Throttle-Remaining` headers - - Added rate limit logging for abuse detection - - Frontend rate limit handling and UI indicators - -### Changed -- All ViewSets now have explicit `throttle_scope` assignments -- Rate limiting can be bypassed in development mode via `IGNY8_DEBUG_THROTTLE=True` - -### Security -- Rate limiting prevents API abuse and ensures fair resource usage -- Abuse detection logging identifies repeated rate limit hits - -### Affected Areas -- API Layer (`igny8_core/api/throttles.py`) -- All ViewSets (throttle scope assignments) -- Frontend (`frontend/src/services/api.ts`) -- Settings (`backend/igny8_core/settings.py`) - -### Migration Guide -1. **Development:** - - Set `IGNY8_DEBUG_THROTTLE=True` to bypass throttling during development - - Or set `DEBUG=True` in Django settings - -2. **Production:** - - Rate limits are automatically enforced - - Monitor rate limit logs for abuse patterns - - Adjust rate limits in `DEFAULT_THROTTLE_RATES` if needed - -3. **Frontend:** - - Handle 429 responses gracefully - - Display rate limit information to users - - Show warnings when approaching limits -``` - -**Estimated Time:** 1 hour - ---- - -## Testing Strategy - -### Unit Tests - -**File:** `backend/igny8_core/api/tests/test_throttles.py` - -**Test Cases:** -- Throttle class rate limit enforcement -- Debug mode bypass -- Plan-based throttling (if implemented) -- Throttle header generation -- Cache key generation - -### Integration Tests - -**Test Cases:** -- End-to-end rate limiting on actual endpoints -- Throttle header presence and correctness -- Rate limit reset after time window -- Multiple users with separate rate limits -- Abuse detection logging - -### Manual Testing - -**Checklist:** -- [ ] AI functions rate limited to 10/min -- [ ] Image generation rate limited to 15/min -- [ ] Auth endpoints rate limited to 20/min -- [ ] Throttle headers present in responses -- [ ] 429 responses include Retry-After header -- [ ] Debug mode bypasses throttling -- [ ] Frontend displays rate limit warnings -- [ ] Abuse detection logs rate limit hits - ---- - -## Rollout Plan - -### Phase 1: Foundation (Week 1) -- ✅ Task 1: Configure DRF throttling -- ✅ Task 2.1: Audit endpoints -- ✅ Unit tests for throttle classes - -### Phase 2: Scope Assignment (Week 2) -- ✅ Task 2.2: Assign throttle scopes -- ✅ Task 4.1: Verify throttle headers -- ✅ Integration testing - -### Phase 3: Frontend & Logging (Week 3) -- ✅ Task 4.2: Frontend rate limit handling -- ✅ Task 4.3: Abuse detection logging -- ✅ End-to-end testing - -### Phase 4: Advanced Features (Week 4 - Optional) -- ✅ Task 3: Plan-based throttling (if needed) -- ✅ Task 5: Quota integration (if needed) - -### Phase 5: Documentation & Release (Week 5) -- ✅ Task 6: Complete test scenarios -- ✅ Task 7: Changelog entry -- ✅ Release to staging -- ✅ Production deployment - ---- - -## Success Criteria - -### Definition of Done - -1. ✅ DRF throttling configured with appropriate rate limits -2. ✅ All ViewSets have explicit throttle scopes -3. ✅ Throttle headers exposed in responses -4. ✅ Frontend handles rate limits gracefully -5. ✅ Abuse detection logging implemented -6. ✅ Debug mode bypass works correctly -7. ✅ All test scenarios pass -8. ✅ Documentation updated -9. ✅ Changelog entry created - -### Metrics - -- **Coverage:** 100% of expensive endpoints have throttle scopes -- **Security:** Rate limiting prevents abuse -- **User Experience:** Rate limit information clearly communicated -- **Performance:** Throttling has minimal overhead - ---- - -## Risk Assessment - -### Risks - -1. **Too Restrictive Limits:** Rate limits may be too low for legitimate users - - **Mitigation:** Start with conservative limits, adjust based on usage patterns - -2. **Performance Impact:** Throttling checks may add overhead - - **Mitigation:** Use efficient caching (Redis), minimal overhead expected - -3. **Frontend Breaking:** Frontend may not handle 429 responses - - **Mitigation:** Update frontend error handling, test thoroughly - -4. **Debug Mode in Production:** Accidental debug mode could bypass throttling - - **Mitigation:** Ensure DEBUG=False in production, use environment variables - -### Rollback Plan - -If issues arise: -1. Increase rate limits temporarily -2. Disable throttling for specific scopes if needed -3. Revert throttle scope assignments -4. Document issues for future fixes - ---- - -**Document Status:** Implementation Plan -**Last Updated:** 2025-01-XX -**Next Review:** After Phase 1 completion - diff --git a/unified-api/API-IMPLEMENTATION-PLAN-SECTION5.md b/unified-api/API-IMPLEMENTATION-PLAN-SECTION5.md deleted file mode 100644 index 7ea1aacd..00000000 --- a/unified-api/API-IMPLEMENTATION-PLAN-SECTION5.md +++ /dev/null @@ -1,986 +0,0 @@ -# API Implementation Plan - Section 5: Refactor Endpoints Module by Module - -**Date:** 2025-01-XX -**Status:** Planning -**Priority:** High -**Related Documents:** `API-ENDPOINTS-ANALYSIS.md`, `API-IMPLEMENTATION-PLAN-SECTION1.md`, `API-IMPLEMENTATION-PLAN-SECTION2.md`, `API-IMPLEMENTATION-PLAN-SECTION3.md`, `API-IMPLEMENTATION-PLAN-SECTION4.md` - ---- - -## Executive Summary - -This document outlines the implementation plan for **Section 5: Refactor Endpoints Module by Module** across the IGNY8 API layer. The goal is to systematically upgrade all API endpoints across all modules (Auth, Planner, Writer, System, Billing) to follow the unified interface design established in Sections 1-4, without breaking existing frontend functionality. - -**Key Objectives:** -- Create refactor checklist template for tracking progress -- Refactor one module at a time, starting with least dependent modules -- Preserve backward compatibility where needed -- Comprehensive testing before global deployment -- Update documentation with refactored endpoint status - -**Note:** This section consolidates and applies all previous sections (1-4) to actual endpoint refactoring. - ---- - -## Table of Contents - -1. [Current State Analysis](#current-state-analysis) -2. [Implementation Tasks](#implementation-tasks) -3. [Task 1: Create Refactor Checklist Template](#task-1-create-refactor-checklist-template) -4. [Task 2: Refactor One Module at a Time](#task-2-refactor-one-module-at-a-time) -5. [Task 3: Track Affected Files in Git Commit Notes](#task-3-track-affected-files-in-git-commit-notes) -6. [Task 4: Preserve Backward Compatibility](#task-4-preserve-backward-compatibility) -7. [Task 5: Final Round of Testing Before Global Deployment](#task-5-final-round-of-testing-before-global-deployment) -8. [Task 6: Documentation Update](#task-6-documentation-update) -9. [Task 7: Changelog Entry](#task-7-changelog-entry) -10. [Testing Strategy](#testing-strategy) -11. [Rollout Plan](#rollout-plan) -12. [Success Criteria](#success-criteria) - ---- - -## Current State Analysis - -### Current Endpoint Status - -Based on `API-ENDPOINTS-ANALYSIS.md`: - -1. **Total Endpoints:** 100+ endpoints across 5 modules -2. **Current State:** - - Mixed response formats (Section 1 to fix) - - Inconsistent permissions (Section 2 to fix) - - Inconsistent error handling (Section 3 to fix) - - No rate limiting (Section 4 to fix) - -3. **Module Dependencies:** - - **System Module:** Least dependent, good starting point - - **Billing Module:** Independent, good for early refactoring - - **Planner Module:** Depends on Auth, moderate dependency - - **Writer Module:** Depends on Planner, higher dependency - - **Auth Module:** Core dependency, refactor last or carefully - -4. **Frontend Integration:** - - Frontend uses these endpoints extensively - - Breaking changes would impact user experience - - Need careful testing and backward compatibility - ---- - -## Implementation Tasks - -### Overview - -| Task ID | Task Name | Priority | Estimated Effort | Dependencies | -|---------|-----------|----------|------------------|--------------| -| 1.1 | Create refactor checklist template | High | 2 hours | None | -| 2.1 | Refactor System module | High | 12 hours | Sections 1-4 | -| 2.2 | Refactor Billing module | High | 8 hours | Sections 1-4 | -| 2.3 | Refactor Planner module | High | 16 hours | Sections 1-4 | -| 2.4 | Refactor Writer module | High | 16 hours | Sections 1-4 | -| 2.5 | Refactor Auth module | High | 12 hours | Sections 1-4 | -| 3.1 | Set up Git tracking | Medium | 2 hours | None | -| 4.1 | Implement legacy format support (if needed) | Low | 6 hours | 2.1-2.5 | -| 5.1 | Frontend integration testing | High | 12 hours | 2.1-2.5 | -| 5.2 | Regression testing | High | 8 hours | 2.1-2.5 | -| 6.1 | Update documentation | High | 6 hours | 2.1-2.5 | -| 7.1 | Create changelog entry | Low | 1 hour | All tasks | - -**Total Estimated Effort:** ~101 hours (~2.5 weeks) - ---- - -## Task 1: Create Refactor Checklist Template - -### Goal -Create a comprehensive checklist template to track refactoring progress for each endpoint. - -### Implementation Steps - -#### Step 1.1: Create Checklist Document - -**File:** `docs/API-REFACTOR-CHECKLIST.md` - -**Implementation:** -```markdown -# API Refactor Checklist - -This document tracks the refactoring progress of all API endpoints to follow the unified interface design. - -## Checklist Legend - -- ✅ Complete -- ⏳ In Progress -- ❌ Not Started -- ⚠️ Issues Found -- 🔄 Needs Review - -## Refactoring Criteria - -Each endpoint must meet all of the following: - -1. **Response Format:** ✅ Uses `success_response()` or `error_response()` -2. **Error Format:** ✅ Uses unified error format with `error` and `errors` fields -3. **Permissions:** ✅ Uses appropriate permission classes (Section 2) -4. **Throttle Scope:** ✅ Has explicit `throttle_scope` assignment (Section 4) -5. **Exception Handling:** ✅ Errors handled by centralized exception handler (Section 3) -6. **Tested with Frontend:** ✅ Verified working with existing frontend code -7. **Documentation:** ✅ Updated in API documentation - ---- - -## System Module - -### AI Prompt Management - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List prompts | GET | `/api/v1/system/prompts/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Create prompt | POST | `/api/v1/system/prompts/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get prompt | GET | `/api/v1/system/prompts/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Update prompt | PUT | `/api/v1/system/prompts/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Delete prompt | DELETE | `/api/v1/system/prompts/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Save prompt | POST | `/api/v1/system/prompts/save/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Reset prompt | POST | `/api/v1/system/prompts/reset/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get by type | GET | `/api/v1/system/prompts/by_type/{type}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - -### Integration Settings - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| Get settings | GET | `/api/v1/system/settings/integrations/{pk}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Save settings | POST | `/api/v1/system/settings/integrations/{pk}/save/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Test connection | POST | `/api/v1/system/settings/integrations/{pk}/test/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Generate test | POST | `/api/v1/system/settings/integrations/{pk}/generate/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - -### Author Profiles - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List profiles | GET | `/api/v1/system/author-profiles/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Create profile | POST | `/api/v1/system/author-profiles/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get profile | GET | `/api/v1/system/author-profiles/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Update profile | PUT | `/api/v1/system/author-profiles/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Delete profile | DELETE | `/api/v1/system/author-profiles/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - ---- - -## Billing Module - -### Credit Balance - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| Get balance | GET | `/api/v1/billing/credits/balance/balance/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - -### Credit Usage - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List usage | GET | `/api/v1/billing/credits/usage/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get usage | GET | `/api/v1/billing/credits/usage/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Usage summary | GET | `/api/v1/billing/credits/usage/summary/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Usage limits | GET | `/api/v1/billing/credits/usage/limits/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - -### Credit Transactions - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List transactions | GET | `/api/v1/billing/credits/transactions/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get transaction | GET | `/api/v1/billing/credits/transactions/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - ---- - -## Planner Module - -### Keywords - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List keywords | GET | `/api/v1/planner/keywords/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Create keyword | POST | `/api/v1/planner/keywords/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get keyword | GET | `/api/v1/planner/keywords/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Update keyword | PUT | `/api/v1/planner/keywords/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Delete keyword | DELETE | `/api/v1/planner/keywords/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Bulk delete | POST | `/api/v1/planner/keywords/bulk_delete/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Bulk update status | POST | `/api/v1/planner/keywords/bulk_update_status/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Auto cluster | POST | `/api/v1/planner/keywords/auto_cluster/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Export keywords | GET | `/api/v1/planner/keywords/export/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Import keywords | POST | `/api/v1/planner/keywords/import_keywords/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - -### Clusters - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List clusters | GET | `/api/v1/planner/clusters/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Create cluster | POST | `/api/v1/planner/clusters/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get cluster | GET | `/api/v1/planner/clusters/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Update cluster | PUT | `/api/v1/planner/clusters/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Delete cluster | DELETE | `/api/v1/planner/clusters/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Auto generate ideas | POST | `/api/v1/planner/clusters/auto_generate_ideas/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - -### Content Ideas - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List ideas | GET | `/api/v1/planner/ideas/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Create idea | POST | `/api/v1/planner/ideas/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get idea | GET | `/api/v1/planner/ideas/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Update idea | PUT | `/api/v1/planner/ideas/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Delete idea | DELETE | `/api/v1/planner/ideas/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Bulk queue to writer | POST | `/api/v1/planner/ideas/bulk_queue_to_writer/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - ---- - -## Writer Module - -### Tasks - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List tasks | GET | `/api/v1/writer/tasks/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Create task | POST | `/api/v1/writer/tasks/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get task | GET | `/api/v1/writer/tasks/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Update task | PUT | `/api/v1/writer/tasks/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Delete task | DELETE | `/api/v1/writer/tasks/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Auto generate content | POST | `/api/v1/writer/tasks/auto_generate_content/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - -### Content - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List content | GET | `/api/v1/writer/content/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Create content | POST | `/api/v1/writer/content/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get content | GET | `/api/v1/writer/content/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Update content | PUT | `/api/v1/writer/content/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Delete content | DELETE | `/api/v1/writer/content/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Generate image prompts | POST | `/api/v1/writer/content/generate_image_prompts/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - -### Images - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List images | GET | `/api/v1/writer/images/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Create image | POST | `/api/v1/writer/images/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get image | GET | `/api/v1/writer/images/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Update image | PUT | `/api/v1/writer/images/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Delete image | DELETE | `/api/v1/writer/images/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Generate images | POST | `/api/v1/writer/images/generate_images/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - ---- - -## Auth Module - -### Authentication - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| Register | POST | `/api/v1/auth/register/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Login | POST | `/api/v1/auth/login/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Change password | POST | `/api/v1/auth/change-password/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get current user | GET | `/api/v1/auth/me/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - -### User Management - -| Endpoint | Method | Path | Response Format | Error Format | Permissions | Throttle Scope | Frontend Tested | Status | -|----------|--------|------|-----------------|--------------|-------------|----------------|-----------------|--------| -| List users | GET | `/api/v1/auth/users/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Create user | POST | `/api/v1/auth/users/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Get user | GET | `/api/v1/auth/users/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Update user | PUT | `/api/v1/auth/users/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Delete user | DELETE | `/api/v1/auth/users/{id}/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Invite user | POST | `/api/v1/auth/users/invite/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | -| Activate user | POST | `/api/v1/auth/users/{id}/activate/` | ⏳ | ⏳ | ⏳ | ⏳ | ❌ | ⏳ | - ---- - -## Progress Summary - -- **Total Endpoints:** ~100+ -- **Completed:** 0 -- **In Progress:** 0 -- **Not Started:** ~100+ -- **Completion:** 0% - -**Last Updated:** 2025-01-XX -``` - -**Estimated Time:** 2 hours - ---- - -## Task 2: Refactor One Module at a Time - -### Goal -Systematically refactor each module, applying all standards from Sections 1-4. - -### Refactoring Order (Least to Most Dependent) - -1. **System Module** (Least dependent) -2. **Billing Module** (Independent) -3. **Planner Module** (Depends on Auth) -4. **Writer Module** (Depends on Planner) -5. **Auth Module** (Core dependency, refactor carefully) - -### Implementation Steps - -#### Step 2.1: Refactor System Module - -**Files to Refactor:** -- `backend/system/api/viewsets.py` - AIPromptViewSet, AuthorProfileViewSet, StrategyViewSet -- `backend/system/api/viewsets.py` - IntegrationSettingsViewSet, SystemSettingsViewSet, etc. - -**Refactoring Checklist for Each ViewSet:** - -1. **Inherit from BaseTenantViewSet:** - ```python - from igny8_core.api.viewsets import BaseTenantViewSet - - class AIPromptViewSet(BaseTenantViewSet): - # Instead of AccountModelViewSet - ``` - -2. **Add Throttle Scope:** - ```python - class AIPromptViewSet(BaseTenantViewSet): - throttle_scope = 'system' - ``` - -3. **Replace Response Calls:** - ```python - # Before - return Response({"success": True, "data": ...}) - - # After - from igny8_core.api.response import success_response - return success_response(data=..., message="...") - ``` - -4. **Replace Error Responses:** - ```python - # Before - return Response({"error": "..."}, status=400) - - # After - from igny8_core.api.response import error_response - return error_response(error="...", status_code=400) - ``` - -5. **Update Custom Actions:** - ```python - @action( - detail=False, - methods=['post'], - permission_classes=[IsAuthenticatedAndActive, HasTenantAccess], - throttle_scope='system_admin' # If admin-only - ) - def save(self, request): - # Use success_response or error_response - pass - ``` - -**Estimated Time:** 12 hours - -#### Step 2.2: Refactor Billing Module - -**Files to Refactor:** -- `backend/billing/api/viewsets.py` - CreditBalanceViewSet, CreditUsageViewSet, CreditTransactionViewSet - -**Similar refactoring steps as System Module.** - -**Estimated Time:** 8 hours - -#### Step 2.3: Refactor Planner Module - -**Files to Refactor:** -- `backend/planner/api/viewsets.py` - KeywordViewSet, ClusterViewSet, ContentIdeasViewSet - -**Additional considerations:** -- AI functions need `throttle_scope='ai_function'` or `throttle_scope='planner_ai'` -- Bulk operations need proper error handling -- Export/import endpoints may need special handling - -**Estimated Time:** 16 hours - -#### Step 2.4: Refactor Writer Module - -**Files to Refactor:** -- `backend/writer/api/viewsets.py` - TasksViewSet, ContentViewSet, ImagesViewSet - -**Additional considerations:** -- AI content generation needs `throttle_scope='ai_function'` -- Image generation needs `throttle_scope='image_gen'` -- Content operations need `throttle_scope='content_write'` - -**Estimated Time:** 16 hours - -#### Step 2.5: Refactor Auth Module - -**Files to Refactor:** -- `backend/auth/api/views.py` - RegisterView, LoginView, ChangePasswordView, MeView -- `backend/auth/api/viewsets.py` - UsersViewSet, AccountsViewSet, SiteViewSet, etc. - -**Special considerations:** -- Public endpoints (register, login) should remain `AllowAny` -- Auth endpoints need `throttle_scope='auth'` -- User management needs `IsAdminOrOwner` permissions - -**Estimated Time:** 12 hours - -### Testing Each Module - -**After refactoring each module:** - -1. **Postman Testing:** - - Test all CRUD operations - - Test custom actions - - Verify response format - - Verify error format - - Check throttle headers - -2. **Frontend Testing:** - - Open frontend application - - Test module-specific workflows - - Verify no breaking changes - - Check error messages display correctly - -3. **Update Checklist:** - - Mark endpoints as complete in `API-REFACTOR-CHECKLIST.md` - - Note any issues found - - Document any special cases - ---- - -## Task 3: Track Affected Files in Git Commit Notes - -### Goal -Maintain clear Git history with well-organized commits for each module refactor. - -### Implementation Steps - -#### Step 3.1: Set Up Git Tracking - -**Commit Message Format:** -``` -chore(api): standardize [module] endpoints - -- Refactor [ViewSet] to use BaseTenantViewSet -- Add throttle_scope assignments -- Replace Response() with success_response()/error_response() -- Update permissions to use standardized classes -- Update custom actions with explicit permissions - -Affected endpoints: -- GET/POST/PUT/DELETE /api/v1/[module]/[resource]/ -- POST /api/v1/[module]/[resource]/[action]/ - -Closes #[issue-number] -Ref: API-REFACTOR-CHECKLIST.md -``` - -**Example Commits:** - -```bash -# System Module -git commit -m "chore(api): standardize system.prompts endpoints - -- Refactor AIPromptViewSet to use BaseTenantViewSet -- Add throttle_scope='system' -- Replace Response() with success_response()/error_response() -- Update permissions to IsAuthenticatedAndActive + HasTenantAccess -- Update save/reset/by_type actions with explicit permissions - -Affected endpoints: -- GET/POST/PUT/DELETE /api/v1/system/prompts/ -- POST /api/v1/system/prompts/save/ -- POST /api/v1/system/prompts/reset/ -- GET /api/v1/system/prompts/by_type/{type}/ - -Ref: docs/API-REFACTOR-CHECKLIST.md" - -# Billing Module -git commit -m "chore(api): standardize billing.credits endpoints - -- Refactor CreditBalanceViewSet, CreditUsageViewSet, CreditTransactionViewSet -- Add throttle_scope='billing' -- Replace Response() with success_response()/error_response() -- Update permissions - -Ref: docs/API-REFACTOR-CHECKLIST.md" -``` - -**Branch Strategy:** -```bash -# Create feature branch for each module -git checkout -b refactor/system-module -# ... make changes ... -git commit -m "..." -git push origin refactor/system-module - -# Create PR with checklist link -``` - -**Estimated Time:** 2 hours (setup and documentation) - ---- - -## Task 4: Preserve Backward Compatibility - -### Goal -Ensure backward compatibility if third-party systems rely on old format. - -### Implementation Steps - -#### Step 4.1: Implement Legacy Format Support (If Needed) - -**File:** `backend/igny8_core/api/response.py` - -**Add legacy format support:** -```python -def success_response(data=None, message=None, status_code=status.HTTP_200_OK, legacy=False): - """ - Create a standardized success response. - - Args: - legacy: If True, return old format for backward compatibility - """ - if legacy: - # Return old format - return Response(data, status=status_code) - - # Return unified format - response_data = { - "success": True, - "data": data, - } - - if message: - response_data["message"] = message - - return Response(response_data, status=status_code) -``` - -**Add Legacy Format Middleware:** - -**File:** `backend/igny8_core/api/middleware.py` - -```python -class LegacyFormatMiddleware(MiddlewareMixin): - """ - Middleware that converts unified format to legacy format if ?legacy=1 is present. - """ - - def process_response(self, request, response): - """ - Convert response to legacy format if legacy=1 query param is present. - """ - if request.GET.get('legacy') == '1' and hasattr(response, 'data'): - if isinstance(response.data, dict) and 'success' in response.data: - # Convert unified format to legacy - if response.data['success']: - # Success: return data directly - response.data = response.data.get('data', response.data) - else: - # Error: return error field - response.data = { - 'error': response.data.get('error'), - 'errors': response.data.get('errors'), - } - - return response -``` - -**Register middleware (if needed):** -```python -MIDDLEWARE = [ - # ... other middleware ... - 'igny8_core.api.middleware.LegacyFormatMiddleware', # Only if needed - # ... other middleware ... -] -``` - -**Mark as Deprecated:** - -**File:** `CHANGELOG.md` - -```markdown -### Deprecated -- Legacy response format (use ?legacy=1 query param) - - Will be removed in version 2.0.0 - - Migrate to unified format: `{ success, data, message, error, errors }` -``` - -**Estimated Time:** 6 hours (only if needed) - ---- - -## Task 5: Final Round of Testing Before Global Deployment - -### Goal -Comprehensive testing to ensure no breaking changes and all functionality works correctly. - -### Implementation Steps - -#### Step 5.1: Frontend Integration Testing - -**Test Workflows:** - -1. **Keyword → Cluster → Ideas → Tasks → Content → Images → Publish:** - - Create keyword - - Create cluster - - Generate ideas - - Create task from idea - - Generate content - - Generate images - - Publish content - -2. **Settings Flow:** - - Update AI prompts - - Update integration settings - - Test integration connections - - Update author profiles - -3. **Credits Flow:** - - View credit balance - - View usage logs - - View usage summary - - View transactions - -4. **Auth Flow:** - - Register new user - - Login - - Change password - - View current user - - Manage users (admin) - -**Check for:** -- ✅ No errors in browser console -- ✅ All API calls return expected format -- ✅ Error messages display correctly -- ✅ Success messages display correctly -- ✅ Tables and dropdowns load correctly -- ✅ Forms submit and validate correctly -- ✅ Notifications work (success/failure) - -#### Step 5.2: Regression Testing - -**Core Flows to Test:** - -1. **Keyword Clustering:** - - Create keywords - - Auto-cluster keywords - - Verify clusters created - -2. **Content Idea Generation:** - - Select cluster - - Auto-generate ideas - - Verify ideas created - -3. **Task Creation:** - - Queue ideas to writer - - Verify tasks created - -4. **Content Generation:** - - Auto-generate content for task - - Verify content created - -5. **Image Generation:** - - Generate images for content - - Verify images created - -6. **Publishing:** - - Publish content (if implemented) - - Verify publishing works - -**Test with Multiple:** -- User roles (owner, admin, editor, writer, viewer) -- Tenants/accounts -- Plan tiers (if plan-based throttling implemented) - -#### Step 5.3: Throttling Tests - -**Test Rate Limiting:** -- Make requests exceeding rate limits -- Verify 429 responses -- Verify throttle headers -- Verify retry-after information -- Test debug mode bypass - -**Estimated Time:** 12 hours (frontend) + 8 hours (regression) = 20 hours - ---- - -## Task 6: Documentation Update - -### Goal -Update all documentation to reflect refactored endpoints. - -### Implementation Steps - -#### Step 6.1: Update Backend Implementation Docs - -**File:** `docs/04-BACKEND-IMPLEMENTATION.md` - -**Add Section: "Refactored Endpoints Status Table"** - -```markdown -## Refactored Endpoints Status - -All endpoints have been refactored to follow the unified interface design: - -- ✅ **Response Format:** All endpoints use `success_response()` or `error_response()` -- ✅ **Error Format:** All errors use unified format with `error` and `errors` fields -- ✅ **Permissions:** All endpoints use standardized permission classes -- ✅ **Throttle Scopes:** All endpoints have explicit `throttle_scope` assignments -- ✅ **Exception Handling:** All errors handled by centralized exception handler - -See [API-REFACTOR-CHECKLIST.md](../docs/API-REFACTOR-CHECKLIST.md) for detailed status of each endpoint. -``` - -#### Step 6.2: Link from API Endpoints Analysis - -**File:** `unified-api/API-ENDPOINTS-ANALYSIS.md` - -**Update Response Format Standards section:** -```markdown -## Response Format Standards - -**Status:** ✅ **Implemented** (See Section 1) - -All endpoints now return unified response format. See [API-IMPLEMENTATION-PLAN-SECTION1.md](./API-IMPLEMENTATION-PLAN-SECTION1.md) for details. - -For endpoint-specific status, see [API-REFACTOR-CHECKLIST.md](../docs/API-REFACTOR-CHECKLIST.md). -``` - -#### Step 6.3: Add Visual Diff (Optional) - -**File:** `docs/API-RESPONSE-FORMAT-DIFF.md` - -```markdown -# API Response Format: Old vs New - -## Before (Old Format) - -```json -{ - "id": 1, - "name": "Example", - "status": "active" -} -``` - -## After (New Format) - -```json -{ - "success": true, - "data": { - "id": 1, - "name": "Example", - "status": "active" - }, - "message": "Resource retrieved successfully" -} -``` - -## Error Response: Before vs After - -### Before -```json -{ - "error": "Validation failed" -} -``` - -### After -```json -{ - "success": false, - "error": "Validation failed", - "errors": { - "field_name": ["Error detail"] - }, - "request_id": "abc-123-def" -} -``` -``` - -**Estimated Time:** 6 hours - ---- - -## Task 7: Changelog Entry - -### Goal -Document the comprehensive refactoring in the changelog. - -### Implementation Steps - -#### Step 7.1: Update CHANGELOG.md - -**File:** `CHANGELOG.md` - -**Add Entry:** -```markdown -## [1.1.0] - 2025-01-XX - -### Changed -- **API Refactoring:** Refactored all major API endpoints to unified response, error, permission, and throttling design - - All endpoints now use `BaseTenantViewSet` with standardized permissions - - All responses use unified format: `{ success, data, message, error, errors }` - - All errors handled by centralized exception handler - - All endpoints have explicit `throttle_scope` assignments - - All custom actions have explicit permission checks - -### Affected Modules -- **System Module:** AIPromptViewSet, AuthorProfileViewSet, StrategyViewSet, IntegrationSettingsViewSet -- **Billing Module:** CreditBalanceViewSet, CreditUsageViewSet, CreditTransactionViewSet -- **Planner Module:** KeywordViewSet, ClusterViewSet, ContentIdeasViewSet -- **Writer Module:** TasksViewSet, ContentViewSet, ImagesViewSet -- **Auth Module:** RegisterView, LoginView, UsersViewSet, AccountsViewSet, SiteViewSet - -### Migration Guide -1. **Frontend Updates:** - - Update API client to handle unified response format - - Check `success` field before accessing `data` - - Use `error` field for top-level error messages - - Use `errors` field for field-specific validation errors - -2. **API Testing:** - - All endpoints now return unified format - - Rate limiting is enforced (see throttle headers) - - Authentication required for most endpoints (except public ones) - -3. **Backward Compatibility:** - - Legacy format available via `?legacy=1` query param (deprecated) - - Will be removed in version 2.0.0 - -### Documentation -- See [API-REFACTOR-CHECKLIST.md](../docs/API-REFACTOR-CHECKLIST.md) for endpoint status -- See [API-IMPLEMENTATION-PLAN-SECTION1.md](../unified-api/API-IMPLEMENTATION-PLAN-SECTION1.md) for response format details -``` - -**Estimated Time:** 1 hour - ---- - -## Testing Strategy - -### Unit Tests - -**Test Cases:** -- Each ViewSet returns unified format -- Error responses use unified format -- Permissions are enforced correctly -- Throttle scopes are assigned correctly -- Custom actions work correctly - -### Integration Tests - -**Test Cases:** -- End-to-end workflows (keyword → content → publish) -- Error scenarios (validation, 401, 403, 429, 500) -- Rate limiting enforcement -- Permission enforcement -- Frontend integration - -### Manual Testing - -**Checklist:** -- [ ] All modules refactored -- [ ] All endpoints return unified format -- [ ] All errors use unified format -- [ ] All endpoints have proper permissions -- [ ] All endpoints have throttle scopes -- [ ] Frontend works correctly -- [ ] No breaking changes -- [ ] Documentation updated -- [ ] Checklist completed - ---- - -## Rollout Plan - -### Phase 1: System & Billing (Week 1) -- ✅ Task 1: Create checklist -- ✅ Task 2.1: Refactor System module -- ✅ Task 2.2: Refactor Billing module -- ✅ Testing and validation - -### Phase 2: Planner & Writer (Week 2) -- ✅ Task 2.3: Refactor Planner module -- ✅ Task 2.4: Refactor Writer module -- ✅ Testing and validation - -### Phase 3: Auth & Compatibility (Week 3) -- ✅ Task 2.5: Refactor Auth module -- ✅ Task 4: Backward compatibility (if needed) -- ✅ Testing and validation - -### Phase 4: Final Testing (Week 4) -- ✅ Task 5: Comprehensive testing -- ✅ Frontend integration testing -- ✅ Regression testing -- ✅ Bug fixes - -### Phase 5: Documentation & Release (Week 5) -- ✅ Task 6: Documentation updates -- ✅ Task 7: Changelog entry -- ✅ Release to staging -- ✅ Production deployment - ---- - -## Success Criteria - -### Definition of Done - -1. ✅ All modules refactored -2. ✅ All endpoints return unified format -3. ✅ All errors use unified format -4. ✅ All endpoints have proper permissions -5. ✅ All endpoints have throttle scopes -6. ✅ Frontend integration tested and working -7. ✅ No breaking changes (or backward compatibility provided) -8. ✅ Documentation updated -9. ✅ Checklist completed -10. ✅ Changelog entry created - -### Metrics - -- **Coverage:** 100% of endpoints refactored -- **Frontend Compatibility:** All workflows functional -- **Documentation:** All endpoints documented -- **Testing:** All test scenarios pass - ---- - -## Risk Assessment - -### Risks - -1. **Breaking Changes:** Frontend may break with new format - - **Mitigation:** Comprehensive frontend testing, backward compatibility option - -2. **Incomplete Refactoring:** Some endpoints may be missed - - **Mitigation:** Comprehensive checklist, systematic approach - -3. **Performance Impact:** New format may add overhead - - **Mitigation:** Minimal overhead, test performance - -4. **Timeline:** Refactoring may take longer than estimated - - **Mitigation:** Prioritize critical endpoints, phase approach - -### Rollback Plan - -If issues arise: -1. Revert specific module refactoring if needed -2. Use legacy format support for backward compatibility -3. Gradually roll back changes -4. Document issues for future fixes - ---- - -**Document Status:** Implementation Plan -**Last Updated:** 2025-01-XX -**Next Review:** After Phase 1 completion - diff --git a/unified-api/API-IMPLEMENTATION-PLAN-SECTION6.md b/unified-api/API-IMPLEMENTATION-PLAN-SECTION6.md deleted file mode 100644 index 92dbdae8..00000000 --- a/unified-api/API-IMPLEMENTATION-PLAN-SECTION6.md +++ /dev/null @@ -1,1432 +0,0 @@ -# API Implementation Plan - Section 6: Testing and Documentation - -**Date:** 2025-01-XX -**Status:** Planning -**Priority:** High -**Related Documents:** All previous sections (1-5) - ---- - -## Executive Summary - -This document outlines the implementation plan for **Section 6: Testing and Documentation** - the final phase of the API standardization project. The goal is to fully test all standardized endpoints, validate frontend compatibility, and update internal and external API documentation to reflect the new structure. - -**Key Objectives:** -- Create comprehensive API test suite (manual + automated) -- Verify frontend integration works correctly -- Document unified API format for internal and external use -- Publish Swagger/OpenAPI documentation (optional) -- Add health check endpoint -- Finalize changelog and version bump -- Complete production readiness checklist - -**Note:** This section should be executed after Sections 1-5 are complete. - ---- - -## Table of Contents - -1. [Current State Analysis](#current-state-analysis) -2. [Implementation Tasks](#implementation-tasks) -3. [Task 1: Create API Test Suite](#task-1-create-api-test-suite) -4. [Task 2: Verify Frontend Integration](#task-2-verify-frontend-integration) -5. [Task 3: Document Unified API Format](#task-3-document-unified-api-format) -6. [Task 4: Publish Swagger/OpenAPI Docs (Optional)](#task-4-publish-swaggeropenapi-docs-optional) -7. [Task 5: Add Health Check Endpoint](#task-5-add-health-check-endpoint) -8. [Task 6: Finalize Changelog and Version Bump](#task-6-finalize-changelog-and-version-bump) -9. [Task 7: Production Readiness Checklist](#task-7-production-readiness-checklist) -10. [Testing Strategy](#testing-strategy) -11. [Rollout Plan](#rollout-plan) -12. [Success Criteria](#success-criteria) - ---- - -## Current State Analysis - -### Current Testing & Documentation Status - -1. **Testing:** - - Unit tests exist but may not cover all standardized endpoints - - Integration tests may need updates for new format - - Manual testing done ad-hoc, not systematic - -2. **Documentation:** - - `API-ENDPOINTS-ANALYSIS.md` exists but may be outdated - - No comprehensive API usage guide - - No Swagger/OpenAPI documentation - - Internal docs may not reflect new format - -3. **Frontend Integration:** - - Frontend uses API extensively - - Need to verify all workflows work with new format - - Error handling may need updates - ---- - -## Implementation Tasks - -### Overview - -| Task ID | Task Name | Priority | Estimated Effort | Dependencies | -|---------|-----------|----------|------------------|--------------| -| 1.1 | Create Postman collection | High | 4 hours | Sections 1-5 | -| 1.2 | Create automated test suite | High | 8 hours | Sections 1-5 | -| 2.1 | Frontend workflow testing | High | 8 hours | Sections 1-5 | -| 2.2 | Frontend error handling verification | High | 4 hours | Sections 1-5 | -| 3.1 | Update backend implementation docs | High | 4 hours | Sections 1-5 | -| 3.2 | Create API usage guide | High | 6 hours | Sections 1-5 | -| 4.1 | Set up Swagger/OpenAPI | Low | 6 hours | Sections 1-5 | -| 5.1 | Create health check endpoint | Medium | 2 hours | None | -| 6.1 | Finalize changelog | High | 2 hours | All sections | -| 6.2 | Version bump | High | 1 hour | 6.1 | -| 7.1 | Production readiness review | High | 4 hours | All tasks | - -**Total Estimated Effort:** ~49 hours (~1.2 weeks) - ---- - -## Task 1: Create API Test Suite - -### Goal -Create comprehensive test suite covering all standardized endpoints with manual (Postman) and automated (Python) tests. - -### Implementation Steps - -#### Step 1.1: Create Postman Collection - -**File:** `unified-api/postman/API-Standardized-Tests.postman_collection.json` - -**Collection Structure:** -```json -{ - "info": { - "name": "IGNY8 API - Standardized Tests", - "description": "Comprehensive test suite for standardized API endpoints", - "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json" - }, - "item": [ - { - "name": "Auth Module", - "item": [ - { - "name": "Register", - "request": { - "method": "POST", - "url": "{{base_url}}/api/v1/auth/register/", - "body": { - "mode": "raw", - "raw": "{\n \"email\": \"test@example.com\",\n \"password\": \"testpass123\",\n \"password_confirm\": \"testpass123\"\n}" - } - }, - "event": [ - { - "listen": "test", - "script": { - "exec": [ - "pm.test(\"Status code is 201\", function () {", - " pm.response.to.have.status(201);", - "});", - "pm.test(\"Response has success field\", function () {", - " var jsonData = pm.response.json();", - " pm.expect(jsonData).to.have.property('success');", - " pm.expect(jsonData.success).to.eql(true);", - "});", - "pm.test(\"Response has data field\", function () {", - " var jsonData = pm.response.json();", - " pm.expect(jsonData).to.have.property('data');", - "});" - ] - } - } - ] - }, - { - "name": "Login", - "request": { - "method": "POST", - "url": "{{base_url}}/api/v1/auth/login/", - "body": { - "mode": "raw", - "raw": "{\n \"email\": \"test@example.com\",\n \"password\": \"testpass123\"\n}" - } - }, - "event": [ - { - "listen": "test", - "script": { - "exec": [ - "pm.test(\"Status code is 200\", function () {", - " pm.response.to.have.status(200);", - "});", - "pm.test(\"Response has success field\", function () {", - " var jsonData = pm.response.json();", - " pm.expect(jsonData).to.have.property('success');", - " pm.expect(jsonData.success).to.eql(true);", - "});", - "pm.test(\"Response has tokens\", function () {", - " var jsonData = pm.response.json();", - " pm.expect(jsonData.data).to.have.property('tokens');", - " pm.environment.set(\"access_token\", jsonData.data.tokens.access);", - "});" - ] - } - } - ] - } - ] - }, - { - "name": "Planner Module", - "item": [ - { - "name": "List Keywords", - "request": { - "method": "GET", - "url": "{{base_url}}/api/v1/planner/keywords/", - "header": [ - { - "key": "Authorization", - "value": "Bearer {{access_token}}" - } - ] - }, - "event": [ - { - "listen": "test", - "script": { - "exec": [ - "pm.test(\"Status code is 200\", function () {", - " pm.response.to.have.status(200);", - "});", - "pm.test(\"Response has success field\", function () {", - " var jsonData = pm.response.json();", - " pm.expect(jsonData).to.have.property('success');", - " pm.expect(jsonData.success).to.eql(true);", - "});", - "pm.test(\"Response has pagination\", function () {", - " var jsonData = pm.response.json();", - " pm.expect(jsonData).to.have.property('count');", - " pm.expect(jsonData).to.have.property('results');", - "});" - ] - } - } - ] - } - ] - } - ] -} -``` - -**Test Scenarios to Include:** - -1. **Success Cases (200/201):** - - Valid requests return `success: true` - - Response has `data` field - - Optional `message` field present - -2. **Validation Errors (400):** - - Invalid input returns `success: false` - - Response has `error` field - - Response has `errors` field with field-specific errors - -3. **Authentication Errors (401):** - - Unauthenticated requests return 401 - - Response has `success: false` - - Response has `error: "Authentication required"` - -4. **Permission Errors (403):** - - Unauthorized requests return 403 - - Response has `success: false` - - Response has `error: "Permission denied"` - -5. **Not Found (404):** - - Non-existent resources return 404 - - Response has `success: false` - - Response has `error: "Resource not found"` - -6. **Rate Limiting (429):** - - Rate limit exceeded returns 429 - - Response has throttle headers - - Response has `Retry-After` header - -7. **Server Errors (500):** - - Server errors return 500 - - Response has `success: false` - - Response has `error: "Internal server error"` - -**Estimated Time:** 4 hours - -#### Step 1.2: Create Automated Test Suite - -**File:** `backend/igny8_core/api/tests/test_standardized_endpoints.py` - -**Implementation:** -```python -""" -Comprehensive Test Suite for Standardized API Endpoints - -Tests all endpoints to ensure they follow the unified interface design. -""" - -from django.test import TestCase -from rest_framework.test import APIClient -from rest_framework import status -from django.contrib.auth import get_user_model -from igny8_core.auth.models import Account - -User = get_user_model() - - -class StandardizedEndpointTestCase(TestCase): - """ - Base test case for standardized endpoint testing. - """ - - def setUp(self): - self.client = APIClient() - self.account = Account.objects.create(name="Test Account") - self.user = User.objects.create_user( - email="test@example.com", - password="testpass123", - account=self.account, - role="owner" - ) - self.client.force_authenticate(user=self.user) - - def assert_unified_success_response(self, response, expected_status=status.HTTP_200_OK): - """ - Assert that response follows unified success format. - """ - self.assertEqual(response.status_code, expected_status) - data = response.json() - self.assertIn('success', data) - self.assertTrue(data['success']) - self.assertIn('data', data) - - def assert_unified_error_response(self, response, expected_status=status.HTTP_400_BAD_REQUEST): - """ - Assert that response follows unified error format. - """ - self.assertEqual(response.status_code, expected_status) - data = response.json() - self.assertIn('success', data) - self.assertFalse(data['success']) - self.assertIn('error', data) - - def assert_paginated_response(self, response): - """ - Assert that response follows unified paginated format. - """ - self.assertEqual(response.status_code, status.HTTP_200_OK) - data = response.json() - self.assertIn('success', data) - self.assertTrue(data['success']) - self.assertIn('count', data) - self.assertIn('results', data) - self.assertIn('next', data) - self.assertIn('previous', data) - - -class AuthEndpointsTestCase(StandardizedEndpointTestCase): - """Test Auth module endpoints""" - - def test_register_endpoint(self): - """Test register endpoint returns unified format""" - self.client.logout() # Ensure not authenticated - response = self.client.post('/api/v1/auth/register/', { - 'email': 'newuser@example.com', - 'password': 'newpass123', - 'password_confirm': 'newpass123' - }) - self.assert_unified_success_response(response, status.HTTP_201_CREATED) - - def test_login_endpoint(self): - """Test login endpoint returns unified format""" - self.client.logout() - response = self.client.post('/api/v1/auth/login/', { - 'email': 'test@example.com', - 'password': 'testpass123' - }) - self.assert_unified_success_response(response) - data = response.json() - self.assertIn('tokens', data['data']) - - def test_me_endpoint(self): - """Test /me endpoint returns unified format""" - response = self.client.get('/api/v1/auth/me/') - self.assert_unified_success_response(response) - data = response.json() - self.assertIn('user', data['data']) - - -class PlannerEndpointsTestCase(StandardizedEndpointTestCase): - """Test Planner module endpoints""" - - def test_list_keywords(self): - """Test keywords list endpoint returns unified paginated format""" - response = self.client.get('/api/v1/planner/keywords/') - self.assert_paginated_response(response) - - def test_create_keyword(self): - """Test keyword creation returns unified format""" - response = self.client.post('/api/v1/planner/keywords/', { - 'keyword': 'test keyword', - 'site_id': 1, - 'sector_id': 1 - }) - self.assert_unified_success_response(response, status.HTTP_201_CREATED) - - def test_validation_error(self): - """Test validation error returns unified error format""" - response = self.client.post('/api/v1/planner/keywords/', {}) - self.assert_unified_error_response(response, status.HTTP_400_BAD_REQUEST) - data = response.json() - self.assertIn('errors', data) - - -class WriterEndpointsTestCase(StandardizedEndpointTestCase): - """Test Writer module endpoints""" - - def test_list_tasks(self): - """Test tasks list endpoint returns unified paginated format""" - response = self.client.get('/api/v1/writer/tasks/') - self.assert_paginated_response(response) - - def test_rate_limiting(self): - """Test that rate limiting works and returns 429""" - endpoint = '/api/v1/writer/tasks/auto_generate_content/' - - # Make 10 requests (should succeed) - for i in range(10): - response = self.client.post(endpoint, {}) - # May fail for other reasons, but not rate limiting - - # 11th request should be rate limited - response = self.client.post(endpoint, {}) - if response.status_code == status.HTTP_429_TOO_MANY_REQUESTS: - self.assert_unified_error_response(response, status.HTTP_429_TOO_MANY_REQUESTS) - self.assertIn('Retry-After', response.headers) - - -class ErrorHandlingTestCase(StandardizedEndpointTestCase): - """Test error handling across endpoints""" - - def test_401_unauthorized(self): - """Test 401 returns unified error format""" - self.client.logout() - response = self.client.get('/api/v1/auth/me/') - self.assert_unified_error_response(response, status.HTTP_401_UNAUTHORIZED) - data = response.json() - self.assertEqual(data['error'], 'Authentication required') - - def test_404_not_found(self): - """Test 404 returns unified error format""" - response = self.client.get('/api/v1/planner/keywords/99999/') - self.assert_unified_error_response(response, status.HTTP_404_NOT_FOUND) - data = response.json() - self.assertEqual(data['error'], 'Resource not found') - - def test_403_permission_denied(self): - """Test 403 returns unified error format""" - # Create user with viewer role (limited permissions) - viewer_user = User.objects.create_user( - email='viewer@example.com', - password='viewerpass123', - account=self.account, - role='viewer' - ) - self.client.force_authenticate(user=viewer_user) - - # Try to delete user (requires admin/owner) - response = self.client.delete(f'/api/v1/auth/users/{self.user.id}/') - if response.status_code == status.HTTP_403_FORBIDDEN: - self.assert_unified_error_response(response, status.HTTP_403_FORBIDDEN) - data = response.json() - self.assertEqual(data['error'], 'Permission denied') -``` - -**Run Tests:** -```bash -# Run all standardized endpoint tests -python manage.py test igny8_core.api.tests.test_standardized_endpoints - -# Run specific test class -python manage.py test igny8_core.api.tests.test_standardized_endpoints.AuthEndpointsTestCase - -# Run with coverage -coverage run --source='.' manage.py test igny8_core.api.tests.test_standardized_endpoints -coverage report -``` - -**Estimated Time:** 8 hours - ---- - -## Task 2: Verify Frontend Integration - -### Goal -Ensure frontend application works correctly with standardized API endpoints. - -### Implementation Steps - -#### Step 2.1: Frontend Workflow Testing - -**Test Major Workflows:** - -1. **Keyword → Cluster → Ideas → Tasks → Content → Images → Publish:** - - Open frontend: `http://localhost:5173` - - Navigate to Planner module - - Create keyword - - Create cluster - - Auto-generate ideas - - Queue ideas to writer (create tasks) - - Auto-generate content for task - - Generate images for content - - Verify all steps complete successfully - - Check browser console for errors - -2. **Settings Flow:** - - Navigate to Settings - - Update AI prompts - - Update integration settings - - Test integration connections - - Update author profiles - - Verify changes saved - -3. **Credits Flow:** - - Navigate to Billing - - View credit balance - - View usage logs - - View usage summary - - Verify data displays correctly - -4. **Auth Flow:** - - Logout - - Register new user - - Login - - Change password - - View current user profile - - (As admin) Manage users - -**Check Browser DevTools:** -- Network tab: Verify all API calls return expected format -- Console: Check for JavaScript errors -- Response preview: Verify `success`, `data`, `message` fields - -**Estimated Time:** 8 hours - -#### Step 2.2: Frontend Error Handling Verification - -**Test Error Scenarios:** - -1. **Validation Errors:** - - Submit form with invalid data - - Verify error message displays - - Verify field-specific errors show on form fields - -2. **Authentication Errors:** - - Try to access protected endpoint without token - - Verify redirect to login - - Verify error message displays - -3. **Permission Errors:** - - Try to access admin-only endpoint as regular user - - Verify error message displays - - Verify UI hides restricted actions - -4. **Rate Limiting:** - - Trigger rate limit (make many requests quickly) - - Verify rate limit warning displays - - Verify UI disables buttons when rate limited - -5. **Network Errors:** - - Simulate network failure - - Verify error message displays - - Verify retry mechanism works - -**Files to Check:** -- `frontend/src/services/api.ts` - API client error handling -- `frontend/src/stores/*.ts` - Store error handling -- `frontend/src/components/*.tsx` - Component error display - -**Estimated Time:** 4 hours - ---- - -## Task 3: Document Unified API Format - -### Goal -Create comprehensive documentation for internal and external API usage. - -### Implementation Steps - -#### Step 3.1: Update Backend Implementation Docs - -**File:** `docs/04-BACKEND-IMPLEMENTATION.md` - -**Add Section: "Standardized API Response Structure"** - -```markdown -## Standardized API Response Structure - -All API endpoints follow a unified response format for consistency and predictability. - -### Success Response Format - -```json -{ - "success": true, - "data": { - // Response data (object or array) - }, - "message": "Optional success message" -} -``` - -**Example:** -```json -{ - "success": true, - "data": { - "id": 1, - "name": "Example Keyword", - "status": "active" - }, - "message": "Keyword created successfully" -} -``` - -### Error Response Format - -```json -{ - "success": false, - "error": "Top-level error message", - "errors": { - "field_name": ["Field-specific error messages"] - }, - "request_id": "uuid-for-error-tracking" -} -``` - -**Example:** -```json -{ - "success": false, - "error": "Validation failed", - "errors": { - "email": ["Invalid email format"], - "password": ["Password must be at least 8 characters"] - }, - "request_id": "550e8400-e29b-41d4-a716-446655440000" -} -``` - -### Paginated Response Format - -```json -{ - "success": true, - "count": 100, - "next": "http://api.igny8.com/api/v1/endpoint/?page=2", - "previous": null, - "results": [ - // Array of results - ], - "message": "Optional message" -} -``` - -### Status Codes - -- `200 OK` - Success -- `201 Created` - Resource created successfully -- `400 Bad Request` - Validation error or bad request -- `401 Unauthorized` - Authentication required -- `403 Forbidden` - Permission denied -- `404 Not Found` - Resource not found -- `429 Too Many Requests` - Rate limit exceeded -- `500 Internal Server Error` - Server error - -### Rate Limiting - -Rate limits are enforced using scoped throttling. Check response headers: - -- `X-Throttle-Limit`: Maximum requests allowed -- `X-Throttle-Remaining`: Requests remaining in current window -- `Retry-After`: Seconds to wait before retrying (when throttled) - -**Rate Limit Scopes:** -- `ai_function`: 10 requests/minute -- `image_gen`: 15 requests/minute -- `content_write`: 30 requests/minute -- `auth`: 20 requests/minute -- `planner`: 60 requests/minute -- `writer`: 60 requests/minute -- `system`: 100 requests/minute -- `billing`: 30 requests/minute - -### Authentication - -Most endpoints require authentication via JWT token: - -``` -Authorization: Bearer -``` - -Public endpoints (no authentication required): -- `POST /api/v1/auth/register/` -- `POST /api/v1/auth/login/` -- `GET /api/v1/auth/plans/` -- `GET /api/v1/auth/industries/` -- `GET /api/v1/system/status/` - -### Global Exception Handling - -All exceptions are handled by a centralized exception handler that: -- Wraps errors in unified format -- Logs errors with request context -- Includes request ID for error tracking -- Provides debug information in development mode - -### Sample ViewSet Format - -```python -from igny8_core.api.viewsets import BaseTenantViewSet -from igny8_core.api.response import success_response, error_response -from igny8_core.api.permissions import IsAuthenticatedAndActive, HasTenantAccess - -class MyViewSet(BaseTenantViewSet): - throttle_scope = 'planner' - permission_classes = [IsAuthenticatedAndActive, HasTenantAccess] - queryset = MyModel.objects.all() - serializer_class = MySerializer - - @action(detail=False, methods=['post']) - def custom_action(self, request): - # Validation - if not request.data.get('field'): - return error_response( - error="Field is required", - errors={"field": ["This field is required"]}, - status_code=status.HTTP_400_BAD_REQUEST - ) - - # Success - return success_response( - data={"result": "..."}, - message="Action completed successfully" - ) -``` -``` - -#### Step 3.2: Create API Usage Guide - -**File:** `docs/API-USAGE-GUIDE.md` - -**Implementation:** -```markdown -# IGNY8 API Usage Guide - -Complete guide for using the IGNY8 API, including authentication, request/response formats, and examples. - -## Table of Contents - -1. [Getting Started](#getting-started) -2. [Authentication](#authentication) -3. [Request Format](#request-format) -4. [Response Format](#response-format) -5. [Error Handling](#error-handling) -6. [Rate Limiting](#rate-limiting) -7. [Examples](#examples) - -## Getting Started - -### Base URL - -``` -Production: https://api.igny8.com/api/v1/ -Development: http://localhost:8000/api/v1/ -``` - -### API Version - -Current version: `v1` - -## Authentication - -### Register New User - -```bash -curl -X POST https://api.igny8.com/api/v1/auth/register/ \ - -H "Content-Type: application/json" \ - -d '{ - "email": "user@example.com", - "password": "securepassword123", - "password_confirm": "securepassword123" - }' -``` - -**Response:** -```json -{ - "success": true, - "data": { - "user": { - "id": 1, - "email": "user@example.com", - "role": "owner" - } - }, - "message": "Registration successful" -} -``` - -### Login - -```bash -curl -X POST https://api.igny8.com/api/v1/auth/login/ \ - -H "Content-Type: application/json" \ - -d '{ - "email": "user@example.com", - "password": "securepassword123" - }' -``` - -**Response:** -```json -{ - "success": true, - "data": { - "user": { ... }, - "tokens": { - "access": "eyJ0eXAiOiJKV1QiLCJhbGc...", - "refresh": "eyJ0eXAiOiJKV1QiLCJhbGc...", - "access_expires_at": "2025-01-15T12:30:00Z", - "refresh_expires_at": "2025-01-22T12:00:00Z" - } - }, - "message": "Login successful" -} -``` - -### Using Access Token - -Include the access token in the Authorization header: - -```bash -curl -X GET https://api.igny8.com/api/v1/auth/me/ \ - -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGc..." -``` - -## Request Format - -### Content-Type - -All requests should use `Content-Type: application/json` for JSON payloads. - -### Pagination - -List endpoints support pagination: - -``` -GET /api/v1/planner/keywords/?page=1&page_size=25 -``` - -**Query Parameters:** -- `page`: Page number (default: 1) -- `page_size`: Items per page (default: 10, max: 100) - -## Response Format - -### Success Response - -```json -{ - "success": true, - "data": { - // Response data - }, - "message": "Optional success message" -} -``` - -### Error Response - -```json -{ - "success": false, - "error": "Top-level error message", - "errors": { - "field_name": ["Field-specific errors"] - }, - "request_id": "uuid-for-tracking" -} -``` - -### Paginated Response - -```json -{ - "success": true, - "count": 100, - "next": "http://api.igny8.com/api/v1/endpoint/?page=2", - "previous": null, - "results": [ ... ], - "message": "Optional message" -} -``` - -## Error Handling - -### Status Codes - -- `200 OK` - Success -- `201 Created` - Resource created -- `400 Bad Request` - Validation error -- `401 Unauthorized` - Authentication required -- `403 Forbidden` - Permission denied -- `404 Not Found` - Resource not found -- `429 Too Many Requests` - Rate limit exceeded -- `500 Internal Server Error` - Server error - -### Handling Errors - -Always check the `success` field first: - -```javascript -const response = await fetch('/api/v1/endpoint/', { - method: 'POST', - headers: { - 'Authorization': `Bearer ${token}`, - 'Content-Type': 'application/json' - }, - body: JSON.stringify(data) -}); - -const result = await response.json(); - -if (result.success) { - // Handle success - console.log(result.data); -} else { - // Handle error - console.error(result.error); - if (result.errors) { - // Handle field-specific errors - Object.keys(result.errors).forEach(field => { - console.error(`${field}: ${result.errors[field].join(', ')}`); - }); - } -} -``` - -## Rate Limiting - -Rate limits are enforced per endpoint scope. Check response headers: - -- `X-Throttle-Limit`: Maximum requests -- `X-Throttle-Remaining`: Remaining requests -- `Retry-After`: Wait time when throttled (seconds) - -**Example:** -```javascript -const throttleLimit = response.headers.get('X-Throttle-Limit'); -const throttleRemaining = response.headers.get('X-Throttle-Remaining'); - -if (parseInt(throttleRemaining) < 5) { - console.warn('Approaching rate limit'); -} -``` - -## Examples - -### JavaScript (Fetch API) - -```javascript -async function createKeyword(keywordData) { - const response = await fetch('https://api.igny8.com/api/v1/planner/keywords/', { - method: 'POST', - headers: { - 'Authorization': `Bearer ${accessToken}`, - 'Content-Type': 'application/json' - }, - body: JSON.stringify(keywordData) - }); - - const result = await response.json(); - - if (result.success) { - return result.data; - } else { - throw new Error(result.error); - } -} -``` - -### Python (Requests) - -```python -import requests - -def create_keyword(keyword_data, access_token): - url = 'https://api.igny8.com/api/v1/planner/keywords/' - headers = { - 'Authorization': f'Bearer {access_token}', - 'Content-Type': 'application/json' - } - - response = requests.post(url, json=keyword_data, headers=headers) - result = response.json() - - if result['success']: - return result['data'] - else: - raise Exception(result['error']) -``` - -### cURL - -```bash -# Create keyword -curl -X POST https://api.igny8.com/api/v1/planner/keywords/ \ - -H "Authorization: Bearer $ACCESS_TOKEN" \ - -H "Content-Type: application/json" \ - -d '{ - "keyword": "example keyword", - "site_id": 1, - "sector_id": 1 - }' - -# List keywords (paginated) -curl -X GET "https://api.igny8.com/api/v1/planner/keywords/?page=1&page_size=25" \ - -H "Authorization: Bearer $ACCESS_TOKEN" -``` -``` - -**Estimated Time:** 4 hours (backend docs) + 6 hours (usage guide) = 10 hours - ---- - -## Task 4: Publish Swagger/OpenAPI Docs (Optional) - -### Goal -Generate and publish interactive API documentation using Swagger/OpenAPI. - -### Implementation Steps - -#### Step 4.1: Set Up drf-spectacular - -**Install:** -```bash -pip install drf-spectacular -``` - -**File:** `backend/igny8_core/settings.py` - -**Add to INSTALLED_APPS:** -```python -INSTALLED_APPS = [ - # ... other apps ... - 'drf_spectacular', -] -``` - -**Add to REST_FRAMEWORK:** -```python -REST_FRAMEWORK = { - # ... existing settings ... - 'DEFAULT_SCHEMA_CLASS': 'drf_spectacular.openapi.AutoSchema', -} -``` - -**Add SPECTACULAR_SETTINGS:** -```python -SPECTACULAR_SETTINGS = { - 'TITLE': 'IGNY8 API', - 'DESCRIPTION': 'IGNY8 Content Planning and Generation API', - 'VERSION': '1.0.0', - 'SERVE_INCLUDE_SCHEMA': False, - 'COMPONENT_SPLIT_REQUEST': True, - 'SCHEMA_PATH_PREFIX': '/api/v1/', -} -``` - -#### Step 4.2: Add URL Routes - -**File:** `backend/igny8_core/urls.py` - -```python -from drf_spectacular.views import ( - SpectacularAPIView, - SpectacularRedocView, - SpectacularSwaggerView, -) - -urlpatterns = [ - # ... existing URLs ... - - # API Schema - path('api/schema/', SpectacularAPIView.as_view(), name='schema'), - - # Swagger UI - path('api/docs/', SpectacularSwaggerView.as_view(url_name='schema'), name='swagger-ui'), - - # ReDoc - path('api/redoc/', SpectacularRedocView.as_view(url_name='schema'), name='redoc'), -] -``` - -#### Step 4.3: Customize Schema - -**File:** `backend/igny8_core/api/schema.py` - -```python -""" -Custom OpenAPI Schema Configuration -""" - -from drf_spectacular.utils import extend_schema, OpenApiResponse -from drf_spectacular.types import OpenApiTypes - - -# Define unified response schemas -SUCCESS_RESPONSE_SCHEMA = { - "type": "object", - "properties": { - "success": {"type": "boolean", "example": True}, - "data": {"type": "object"}, - "message": {"type": "string", "nullable": True} - }, - "required": ["success", "data"] -} - -ERROR_RESPONSE_SCHEMA = { - "type": "object", - "properties": { - "success": {"type": "boolean", "example": False}, - "error": {"type": "string"}, - "errors": {"type": "object", "nullable": True}, - "request_id": {"type": "string", "nullable": True} - }, - "required": ["success", "error"] -} -``` - -**Estimated Time:** 6 hours (optional) - ---- - -## Task 5: Add Health Check Endpoint - -### Goal -Create a simple health check endpoint for API availability monitoring. - -### Implementation Steps - -#### Step 5.1: Create Health Check View - -**File:** `backend/igny8_core/api/views.py` - -```python -""" -Health Check Endpoint -""" - -from rest_framework.views import APIView -from rest_framework.response import Response -from rest_framework import status -from django.utils import timezone -from igny8_core.api.response import success_response - - -class PingView(APIView): - """ - Health check endpoint for API availability. - - Returns simple pong response to verify API is live. - """ - permission_classes = [] # Public endpoint - - def get(self, request): - """ - Return health check response. - """ - return success_response( - data={ - "pong": True, - "time": timezone.now().isoformat(), - "version": "1.0.0" - }, - message="API is live" - ) -``` - -#### Step 5.2: Add URL Route - -**File:** `backend/igny8_core/urls.py` - -```python -from igny8_core.api.views import PingView - -urlpatterns = [ - # ... existing URLs ... - path('api/ping/', PingView.as_view(), name='ping'), -] -``` - -**Test:** -```bash -curl http://localhost:8000/api/ping/ -``` - -**Expected Response:** -```json -{ - "success": true, - "data": { - "pong": true, - "time": "2025-01-15T12:30:00Z", - "version": "1.0.0" - }, - "message": "API is live" -} -``` - -**Estimated Time:** 2 hours - ---- - -## Task 6: Finalize Changelog and Version Bump - -### Goal -Complete changelog and update version numbers. - -### Implementation Steps - -#### Step 6.1: Finalize Changelog - -**File:** `CHANGELOG.md` - -**Add Complete Entry:** -```markdown -## [1.1.0] - 2025-01-XX - -### Changed -- **API Standardization:** Comprehensive API standardization across all modules - - Unified response format: All endpoints return `{ success, data, message, error, errors }` - - Centralized error handling with request ID tracking - - Standardized permission classes and authentication - - Scoped rate limiting for all endpoints - - Refactored all ViewSets to use BaseTenantViewSet - -### Added -- **Response Format:** Unified success/error response format -- **Exception Handling:** Centralized exception handler with logging -- **Rate Limiting:** Scoped rate limiting with throttle headers -- **Permissions:** Standardized permission classes (IsAuthenticatedAndActive, HasTenantAccess, etc.) -- **Health Check:** `/api/ping/` endpoint for API availability -- **Documentation:** Comprehensive API usage guide and Swagger/OpenAPI docs - -### Security -- All endpoints now require authentication (except public ones) -- Rate limiting prevents API abuse -- Tenant isolation enforced at permission level -- Request ID tracking for error correlation - -### Affected Areas -- All API modules: Auth, Planner, Writer, System, Billing -- Frontend API client -- API documentation - -### Migration Guide -See [API-USAGE-GUIDE.md](../docs/API-USAGE-GUIDE.md) for complete migration guide. - -### Breaking Changes -- ⚠️ **Response Format:** All API responses now follow unified format - - Frontend must check `success` field before accessing `data` - - Error responses use `error` field instead of `message` - - Paginated responses include `success: true` field - -- ⚠️ **Authentication:** Most endpoints now require authentication - - Public endpoints: register, login, plans, industries, status, ping - - All other endpoints require JWT token - -- ⚠️ **Rate Limiting:** Rate limits are now enforced - - Check `X-Throttle-Remaining` header - - Handle 429 responses gracefully - -### Documentation -- [API-ENDPOINTS-ANALYSIS.md](../unified-api/API-ENDPOINTS-ANALYSIS.md) -- [API-USAGE-GUIDE.md](../docs/API-USAGE-GUIDE.md) -- [API-REFACTOR-CHECKLIST.md](../docs/API-REFACTOR-CHECKLIST.md) -- Swagger UI: `/api/docs/` -- ReDoc: `/api/redoc/` -``` - -#### Step 6.2: Version Bump - -**File:** `backend/igny8_core/settings.py` - -```python -# API Version -API_VERSION = '1.1.0' -``` - -**File:** `package.json` (if exists) - -```json -{ - "version": "1.1.0" -} -``` - -**File:** `docker-compose.yml` (if exists) - -```yaml -services: - backend: - environment: - - API_VERSION=1.1.0 -``` - -**Estimated Time:** 2 hours (changelog) + 1 hour (version) = 3 hours - ---- - -## Task 7: Production Readiness Checklist - -### Goal -Complete final production readiness review. - -### Implementation Steps - -#### Step 7.1: Production Readiness Review - -**Checklist:** - -**API Response Structure:** -- [ ] All endpoints return unified format -- [ ] Success responses have `success: true` -- [ ] Error responses have `success: false` -- [ ] Paginated responses include `success: true` - -**Security:** -- [ ] All ViewSets secured with proper permissions -- [ ] Tenant isolation enforced -- [ ] Rate limits applied and tested -- [ ] Authentication required for protected endpoints - -**Error Handling:** -- [ ] All errors handled by centralized exception handler -- [ ] Error responses include request ID -- [ ] Error logging configured -- [ ] Debug information excluded in production - -**Rate Limiting:** -- [ ] Rate limits configured for all endpoint types -- [ ] Throttle headers exposed -- [ ] Abuse detection logging implemented -- [ ] Frontend handles rate limits gracefully - -**Testing:** -- [ ] Unit tests pass -- [ ] Integration tests pass -- [ ] Frontend integration tested -- [ ] Regression tests pass -- [ ] Rate limiting tests pass - -**Documentation:** -- [ ] API usage guide complete -- [ ] Backend implementation docs updated -- [ ] Swagger/OpenAPI docs published (if implemented) -- [ ] Changelog complete -- [ ] Version bumped - -**Frontend:** -- [ ] All workflows functional -- [ ] Error messages display correctly -- [ ] Rate limit warnings work -- [ ] No console errors - -**Deployment:** -- [ ] Staging environment tested -- [ ] Production deployment plan ready -- [ ] Rollback plan documented -- [ ] Monitoring configured - -**Estimated Time:** 4 hours - ---- - -## Testing Strategy - -### Unit Tests -- All standardized endpoints -- Response format validation -- Error format validation -- Permission enforcement -- Rate limiting - -### Integration Tests -- End-to-end workflows -- Frontend integration -- Error scenarios -- Rate limiting scenarios - -### Manual Testing -- Postman collection -- Frontend workflows -- Browser DevTools verification -- Production readiness checklist - ---- - -## Rollout Plan - -### Phase 1: Test Suite Creation (Week 1) -- ✅ Task 1: Create test suite -- ✅ Task 2: Frontend integration testing - -### Phase 2: Documentation (Week 2) -- ✅ Task 3: Document API format -- ✅ Task 4: Swagger/OpenAPI (optional) -- ✅ Task 5: Health check endpoint - -### Phase 3: Finalization (Week 3) -- ✅ Task 6: Changelog and version -- ✅ Task 7: Production readiness -- ✅ Final review and deployment - ---- - -## Success Criteria - -### Definition of Done - -1. ✅ Comprehensive test suite created -2. ✅ Frontend integration verified -3. ✅ API documentation complete -4. ✅ Swagger/OpenAPI published (if implemented) -5. ✅ Health check endpoint added -6. ✅ Changelog finalized -7. ✅ Version bumped -8. ✅ Production readiness checklist complete -9. ✅ All tests pass -10. ✅ Ready for production deployment - -### Metrics - -- **Test Coverage:** >90% for standardized endpoints -- **Documentation:** Complete API usage guide -- **Frontend Compatibility:** All workflows functional -- **Production Readiness:** All checklist items complete - ---- - -**Document Status:** Implementation Plan -**Last Updated:** 2025-01-XX -**Next Review:** After Phase 1 completion - diff --git a/unified-api/sample -singel-doc-for-6-sections.md b/unified-api/sample -singel-doc-for-6-sections.md deleted file mode 100644 index 36e840d0..00000000 --- a/unified-api/sample -singel-doc-for-6-sections.md +++ /dev/null @@ -1,419 +0,0 @@ -Understood. -Here is the **complete consolidated master document**, combining all 6 sections into a single, clean, unified API Standard for IGNY8. -No questions. No fragmentation. This becomes the **single source of truth** for backend, frontend, plugin, and 3rd-party integration. - ---- - -# **IGNY8 API STANDARDIZATION – MASTER CONSOLIDATED DOCUMENT** - -**Version:** 1.0 -**Purpose:** Establish a single unified API contract for the IGNY8 platform (app, plugin, and external API consumers). -**Applies To:** - -* Django backend -* React frontend -* WordPress plugin -* 3rd-party API consumers -* Internal automations / CRON / AI functions - ---- - -# **1. API Versioning & Structure** - -**Base URL:** - -``` -/api/v1/ -``` - -**Module Namespaces:** - -``` -auth/ -planner/ -writer/ -system/ -billing/ -``` - -All current and future endpoints must follow predictable REST patterns under these namespaces. - -**Versioning Rules:** - -* v1 remains stable long-term -* Breaking changes require v2 -* Deprecations allowed only with explicit timeline - ---- - -# **2. Authentication Standard (Unified)** - -**Primary method:** -`JWT Bearer token` - -``` -Authorization: Bearer -``` - -**Fallbacks:** - -* Session auth (admin panel) -* Basic auth (debug/testing) - -**Token Guarantees:** - -* Contains user_id and account_id -* Resolves account → tenant context automatically -* Sets `request.account` in all ViewSets - -**Public endpoints allowed:** - -* register -* login -* plans -* industries -* system/status - -Everything else requires JWT. - ---- - -# **3. Authorization & Permissions Standard** - -Every endpoint must use consistent layered authorization: - -### **Base layers:** - -1. User must be authenticated -2. User must belong to the tenant/account -3. User must have appropriate role -4. User must have access to requested site/sector - -### **Role hierarchy:** - -``` -owner > admin > editor > viewer > system_bot -``` - -### **Standard permission classes:** - -* IsAuthenticatedAndActive -* HasTenantAccess -* IsViewerOrAbove (read-only) -* IsEditorOrAbove (content operations) -* IsAdminOrOwner (settings, keys, billing) - -All ViewSets must explicitly declare the correct class. - ---- - -# **4. Unified Response Format (Mandatory)** - -This is **the global standard for all endpoints**, no exception. - -### **Success** - -``` -{ - "success": true, - "data": {}, - "message": "Optional readable message" -} -``` - -### **Error** - -``` -{ - "success": false, - "error": "Readable top-level message", - "errors": { "field": ["details"] }, - "request_id": "uuid" -} -``` - -### **Paginated** - -``` -{ - "success": true, - "count": 120, - "next": "...", - "previous": "...", - "results": [...], - "message": "Optional" -} -``` - -This single structure is used across: - -* App -* Plugin -* External API clients -* AI pipeline -* CRON jobs - ---- - -# **5. Unified Error Handling (Global Standard)** - -### **Centralized exception handler requirements:** - -* Wrap all errors in unified format -* Use proper status codes (400, 401, 403, 404, 409, 422, 500) -* Include sanitised validation errors under `errors` -* Always attach `request_id` -* Log full exception details -* In DEBUG mode: include traceback + request context - -### **Server-side Logging** - -* All 4xx logs as warning -* All 5xx logs as error -* Use structured format -* Store in rotating log files -* Provide hooks for Sentry in production - ---- - -# **6. Unified Request Payload Rules** - -All endpoints must follow: - -* snake_case field names -* predictable field groups -* standardized pagination params -* standardized filters -* standardized bulk update patterns -* standardized action payloads -* standardized error messages - -**Bulk actions:** - -``` -{ "ids": [1,2,3], ... } -``` - -**Filtering:** - -``` -?search= -?status= -?site_id= -?sector_id= -``` - ---- - -# **7. Tenant / Site / Sector Scoping Rules** - -Every resource created or fetched must be scoped by: - -1. **Account/Tenant** -2. **Site** -3. **Sector** - -### Enforcement: - -* `AccountModelViewSet` handles account isolation -* `SiteSectorModelViewSet` filters queries by site/sector -* All custom actions must also use `.get_queryset()` to avoid bypassing filters -* Any ID list must be verified to belong to the authenticated tenant - ---- - -# **8. Module-Wise API Rules (Planner / Writer / System / Billing)** - -## **Planner Module** - -Covers keywords → clusters → ideas. - -Rules: - -* All endpoints require authentication -* Auto-cluster limited to 20 keywords/request -* Ideas generation limited to 1 cluster/request -* Standardized payloads for imports/exports -* Bulk actions must validate tenant ownership - ---- - -## **Writer Module** - -Covers tasks → content → image prompts → images. - -Rules: - -* 1 AI content generation per task -* 1 image generation per request -* Bulk actions max 50 -* Image prompt extraction uses unified response -* Must align with progress tracking - ---- - -## **System Module** - -Covers prompts, strategies, profiles, integrations, task progress. - -Rules: - -* Saving prompts requires editor/admin -* Integration settings require admin/owner -* All tests (OpenAI/Runware) return unified format -* Progress endpoint returns stable JSON for progress modal - ---- - -## **Billing Module** - -Covers credit balance, usage logs, transactions. - -Rules: - -* Balance/usage require authenticated tenant -* Transactions require admin/owner -* Unified reporting structure -* No raw model data returned - ---- - -# **9. API Governance, Rate Limits, and Safety Controls** - -### **Soft limits** - -* 50 tasks per bulk action -* 20 keywords per cluster batch -* 1 cluster → ideas generation -* 1 content → image prompt extraction -* 1 image generation per job -* Per-tenant throttling (future) - -### **Safety checks** - -* Validate all IDs belong to tenant -* Validate dependent records exist -* Reject oversized payloads -* Reject invalid site/sector assignments -* Reject stale/wrong tenant tokens - ---- - -# **10. Logging, Monitoring, and Diagnostics** - -### **Backend logging** - -* timestamp -* request_id -* endpoint -* user_id -* account_id -* status_code -* error_message -* traceback for 5xx - -### **Frontend Debug Panel Integration** - -All responses must support: - -* request ID -* readable error messages -* consistent progress steps -* error visibility in real time - ---- - -# **11. Progress Modal Standard (AI Functions)** - -All AI functions must output uniform progress metadata: - -**Steps example for image prompt extraction** - -``` -Smart Image Prompts -Checking content and image slots -Mapping Content for X Image Prompts -Writing Featured Image Prompts -Writing X In-article Image Prompts -Assigning Prompts to Slots -``` - -Success message: - -``` -Featured Image and X In-article Image Prompts ready for image generation -``` - -All four AI functions must follow identical UI messaging structure. - ---- - -# **12. Standard Frontend/Plugin Integration Rules** - -### **Frontend** - -* All responses parsed through the unified envelope -* 401 triggers logout → redirect to login -* 403 triggers permission alert -* Errors read from `.error` or `.errors` -* Pagination reads `.results` - -### **WordPress Plugin** - -* Uses the same JWT flow -* No custom plugin-specific endpoints -* Uses the app’s `/api/v1/` fully -* WP only handles mapping + meta assignment - -### **3rd-Party APIs** - -* Full compatibility with unified format -* Full access where tenant/permissions allow -* No special handling needed - ---- - -# **13. Refactor & Migration Plan (One-Time Update)** - -1. Introduce unified response wrapper -2. Replace all custom responses in ViewSets -3. Inject consistent permission classes -4. Wrap all exceptions with centralized handler -5. Update serializers to rely on unified errors -6. Patch all custom actions to validate payloads -7. Update app’s frontend API client -8. Remove deprecated endpoint patterns -9. Regenerate Postman/OpenAPI schema -10. Update WordPress plugin integration -11. Run full-suite regression tests - -This whole migration becomes the Phase-4 API refactor inside the IGNY8 full migration pipeline. - ---- - -# **14. Success Criteria** - -* Every endpoint returns same envelope -* All errors fully unified -* No endpoint leaks raw DRF format -* No inconsistent auth behavior -* No inconsistent pagination -* No inconsistent validation errors -* Plugin works using same exact API -* Third-party clients can implement without special rules -* Frontend debug panel reads everything correctly -* CRON + automations structured identically - ---- - -# **15. Final Notes** - -This consolidated document **replaces all six API planning files**. -It is now the single standard for: - -* Backend implementation -* Frontend integration -* Plugin development -* External API docs -* AI function progress UI -* DevOps & monitoring