775 lines
29 KiB
Markdown
775 lines
29 KiB
Markdown
# IGNY8 Phase 2: GSC Integration (02C)
|
|
## Google Search Console — Indexing, Inspection & Analytics
|
|
|
|
**Document Version:** 1.0
|
|
**Date:** 2026-03-23
|
|
**Phase:** IGNY8 Phase 2 — Feature Expansion
|
|
**Status:** Build Ready
|
|
**Source of Truth:** Codebase at `/data/app/igny8/`
|
|
**Audience:** Claude Code, Backend Developers, Architects
|
|
|
|
---
|
|
|
|
## 1. CURRENT STATE
|
|
|
|
### Existing Integration Infrastructure
|
|
- `SiteIntegration` model (db_table=`igny8_site_integrations`) stores WordPress connections with `platform='wordpress'`
|
|
- `SyncEvent` model (db_table=`igny8_sync_events`) logs publish/sync operations
|
|
- Integration app registered at `/api/v1/integration/`
|
|
- No Google API connections of any kind exist
|
|
- No OAuth 2.0 infrastructure for third-party APIs
|
|
- `IntegrationProvider` model supports `provider_type`: ai/payment/email/storage — no `search_engine` type yet
|
|
|
|
### Content Publishing Flow
|
|
- When `Content.site_status` changes to `published`, a `PublishingRecord` is created
|
|
- Content gets `external_url` and `external_id` after WordPress publish
|
|
- No automatic indexing request after publish
|
|
- No tracking of whether published URLs are indexed by Google
|
|
|
|
### What Doesn't Exist
|
|
- Google Search Console OAuth connection
|
|
- URL Inspection API integration
|
|
- Indexing queue with priority and quota management
|
|
- Search analytics data collection/dashboard
|
|
- Re-inspection scheduling
|
|
- Plugin-side index status display
|
|
- Any GSC-related models, endpoints, or tasks
|
|
|
|
---
|
|
|
|
## 2. WHAT TO BUILD
|
|
|
|
### Overview
|
|
Full Google Search Console integration with four capabilities:
|
|
1. **OAuth Connection** — connect GSC property via Google OAuth 2.0
|
|
2. **URL Inspection** — inspect URLs via Google's URL Inspection API (2K/day quota), auto-inspect after publish
|
|
3. **Indexing Queue** — priority-based queue with quota management and re-inspection scheduling
|
|
4. **Search Analytics** — fetch and cache search performance data (clicks, impressions, CTR, position)
|
|
|
|
### OAuth 2.0 Connection Flow
|
|
|
|
```
|
|
User clicks "Connect GSC" →
|
|
IGNY8 backend generates Google OAuth URL →
|
|
User authorizes in Google consent screen →
|
|
Google redirects to /api/v1/integration/gsc/callback/ →
|
|
Backend stores encrypted access_token + refresh_token →
|
|
Backend fetches user's GSC properties →
|
|
User selects property → GSCConnection created
|
|
```
|
|
|
|
**Google Cloud project requirements:**
|
|
- Search Console API enabled
|
|
- URL Inspection API enabled (separate from Search Console API)
|
|
- OAuth 2.0 client ID (Web application type)
|
|
- Scopes: `https://www.googleapis.com/auth/webmasters.readonly`, `https://www.googleapis.com/auth/indexing`
|
|
- Redirect URI: `https://{domain}/api/v1/integration/gsc/callback/`
|
|
|
|
**Token management:**
|
|
- Access tokens expire after 1 hour → background refresh via Celery task
|
|
- Refresh tokens stored encrypted in `GSCConnection.refresh_token`
|
|
- If refresh fails → set `GSCConnection.status='expired'`, notify user
|
|
|
|
### URL Inspection API
|
|
|
|
**Google API endpoint:**
|
|
```
|
|
POST https://searchconsole.googleapis.com/v1/urlInspection/index:inspect
|
|
Body: {"inspectionUrl": "https://example.com/page", "siteUrl": "sc-domain:example.com"}
|
|
```
|
|
|
|
**Response fields tracked:**
|
|
| Field | Type | Stored In |
|
|
|-------|------|-----------|
|
|
| `verdict` | PASS/PARTIAL/FAIL/NEUTRAL | URLInspectionRecord.verdict |
|
|
| `coverageState` | e.g., "Submitted and indexed" | URLInspectionRecord.coverage_state |
|
|
| `indexingState` | e.g., "INDEXING_ALLOWED" | URLInspectionRecord.indexing_state |
|
|
| `robotsTxtState` | e.g., "ALLOWED" | URLInspectionRecord.last_inspection_result (JSON) |
|
|
| `lastCrawlTime` | ISO datetime | URLInspectionRecord.last_crawled |
|
|
| Full response | JSON | URLInspectionRecord.last_inspection_result |
|
|
|
|
**Quota:** 2,000 inspections per day per GSC property (resets midnight Pacific Time)
|
|
**Rate limit:** 1 request per 3 seconds (safe limit: 600 per 30 minutes)
|
|
|
|
### Indexing Queue System
|
|
|
|
Priority-based queue that respects daily quota:
|
|
|
|
| Priority | Trigger | Description |
|
|
|----------|---------|-------------|
|
|
| 100 | Content published (auto) | Newly published content auto-queued |
|
|
| 90 | Re-inspection (auto) | Scheduled follow-up check |
|
|
| 70 | Manual inspect request | User requests specific URL inspection |
|
|
| 50 | Information query | Check status only, no submit |
|
|
| 30 | Scheduled bulk re-inspect | Periodic re-check of all URLs |
|
|
|
|
**Queue processing:**
|
|
- Celery task runs every 5 minutes
|
|
- Checks `GSCDailyQuota` for remaining capacity
|
|
- Processes items in priority order (highest first)
|
|
- Respects 1 request/3 second rate limit
|
|
- Status flow: `queued → processing → completed/failed/quota_exceeded`
|
|
|
|
### Re-Inspection Schedule
|
|
|
|
After initial inspection, automatically schedule follow-up checks:
|
|
|
|
| Check | Timing | Purpose |
|
|
|-------|--------|---------|
|
|
| Check 1 | 24 hours after submission | Quick verification |
|
|
| Check 2 | 3 days after | Give Google time to crawl |
|
|
| Check 3 | 6 days after | Most URLs indexed by now |
|
|
| Check 4 | 13 days after | Final automatic check |
|
|
|
|
If still not indexed after Check 4 → mark `status='manual_review'`, stop auto-checking.
|
|
|
|
### Search Analytics
|
|
|
|
**Google API endpoint:**
|
|
```
|
|
POST https://searchconsole.googleapis.com/v3/sites/{siteUrl}/searchAnalytics/query
|
|
Body: {
|
|
"startDate": "2025-01-01",
|
|
"endDate": "2025-03-23",
|
|
"dimensions": ["page", "query", "date"],
|
|
"rowLimit": 25000
|
|
}
|
|
```
|
|
|
|
**Metrics collected:** clicks, impressions, ctr, position
|
|
**Dimensions:** page, query (keyword), country, device, date
|
|
**Date range:** up to 16 months historical
|
|
**Caching:** Results cached in `GSCMetricsCache` with 24-hour TTL, refreshed daily via Celery
|
|
|
|
### Auto-Indexing After Publish
|
|
|
|
When `Content.site_status` changes to `'published'` and the content has an `external_url`:
|
|
1. Check if `GSCConnection` exists for the site with status='active'
|
|
2. Create or update `URLInspectionRecord` for the URL
|
|
3. Add to `IndexingQueue` with priority=100
|
|
4. If SAG data available: hub pages get inspected before supporting articles (blueprint-aware priority)
|
|
|
|
### Plugin-Side Status Sync
|
|
|
|
IGNY8 pushes index statuses to the WordPress plugin:
|
|
- Endpoint: `POST /wp-json/igny8/v1/gsc/status-sync`
|
|
- Payload: `{urls: [{url, status, verdict, last_inspected}]}`
|
|
- Plugin displays status badges on WP post list table:
|
|
- ⏳ `pending_inspection`
|
|
- ✓ `indexed`
|
|
- ✗ `not_indexed`
|
|
- ➡ `indexing_requested`
|
|
- 🚫 `error_noindex`
|
|
|
|
---
|
|
|
|
## 3. DATA MODELS & APIs
|
|
|
|
### New Models (integration app)
|
|
|
|
```python
|
|
class GSCConnection(AccountBaseModel):
|
|
"""Google Search Console OAuth connection per site."""
|
|
site = models.ForeignKey(
|
|
'igny8_core_auth.Site', on_delete=models.CASCADE,
|
|
related_name='gsc_connections'
|
|
)
|
|
google_email = models.CharField(max_length=255)
|
|
access_token = models.TextField(help_text="Encrypted OAuth access token")
|
|
refresh_token = models.TextField(help_text="Encrypted OAuth refresh token")
|
|
token_expiry = models.DateTimeField()
|
|
gsc_property_url = models.CharField(
|
|
max_length=500,
|
|
help_text="GSC property URL, e.g., sc-domain:example.com"
|
|
)
|
|
status = models.CharField(
|
|
max_length=20, default='active',
|
|
choices=[
|
|
('active', 'Active'),
|
|
('expired', 'Token Expired'),
|
|
('revoked', 'Access Revoked'),
|
|
],
|
|
db_index=True
|
|
)
|
|
|
|
class Meta:
|
|
app_label = 'integration'
|
|
db_table = 'igny8_gsc_connections'
|
|
unique_together = [['site', 'gsc_property_url']]
|
|
|
|
|
|
class URLInspectionRecord(SiteSectorBaseModel):
|
|
"""Tracks URL inspection history and indexing status."""
|
|
url = models.URLField(max_length=2000, db_index=True)
|
|
content = models.ForeignKey(
|
|
'writer.Content', on_delete=models.SET_NULL,
|
|
null=True, blank=True, related_name='inspection_records',
|
|
help_text="Linked IGNY8 content (null for external/non-IGNY8 URLs)"
|
|
)
|
|
last_inspection_result = models.JSONField(
|
|
default=dict, help_text="Full Google API response"
|
|
)
|
|
verdict = models.CharField(
|
|
max_length=20, blank=True, default='',
|
|
help_text="PASS/PARTIAL/FAIL/NEUTRAL"
|
|
)
|
|
coverage_state = models.CharField(max_length=100, blank=True, default='')
|
|
indexing_state = models.CharField(max_length=100, blank=True, default='')
|
|
last_crawled = models.DateTimeField(null=True, blank=True)
|
|
last_inspected = models.DateTimeField(null=True, blank=True)
|
|
inspection_count = models.IntegerField(default=0)
|
|
next_inspection = models.DateTimeField(
|
|
null=True, blank=True,
|
|
help_text="Scheduled next re-inspection datetime"
|
|
)
|
|
status = models.CharField(
|
|
max_length=30, default='pending_inspection',
|
|
choices=[
|
|
('pending_inspection', 'Pending Inspection'),
|
|
('indexed', 'Indexed'),
|
|
('not_indexed', 'Not Indexed'),
|
|
('indexing_requested', 'Indexing Requested'),
|
|
('error_noindex', 'Error / No Index'),
|
|
('manual_review', 'Manual Review Needed'),
|
|
],
|
|
db_index=True
|
|
)
|
|
|
|
class Meta:
|
|
app_label = 'integration'
|
|
db_table = 'igny8_url_inspection_records'
|
|
unique_together = [['site', 'url']]
|
|
ordering = ['-last_inspected']
|
|
|
|
|
|
class IndexingQueue(SiteSectorBaseModel):
|
|
"""Priority queue for URL inspection API requests."""
|
|
url = models.URLField(max_length=2000)
|
|
url_inspection_record = models.ForeignKey(
|
|
URLInspectionRecord, on_delete=models.SET_NULL,
|
|
null=True, blank=True, related_name='queue_entries'
|
|
)
|
|
priority = models.IntegerField(
|
|
default=50, db_index=True,
|
|
help_text="100=auto-publish, 90=re-inspect, 70=manual, 50=info, 30=bulk"
|
|
)
|
|
status = models.CharField(
|
|
max_length=20, default='queued',
|
|
choices=[
|
|
('queued', 'Queued'),
|
|
('processing', 'Processing'),
|
|
('completed', 'Completed'),
|
|
('failed', 'Failed'),
|
|
('quota_exceeded', 'Quota Exceeded'),
|
|
],
|
|
db_index=True
|
|
)
|
|
date_added = models.DateTimeField(auto_now_add=True)
|
|
date_processed = models.DateTimeField(null=True, blank=True)
|
|
error_message = models.TextField(blank=True, default='')
|
|
|
|
class Meta:
|
|
app_label = 'integration'
|
|
db_table = 'igny8_indexing_queue'
|
|
ordering = ['-priority', 'date_added']
|
|
|
|
|
|
class GSCMetricsCache(SiteSectorBaseModel):
|
|
"""Cached search analytics data from GSC API."""
|
|
metric_type = models.CharField(
|
|
max_length=50, db_index=True,
|
|
choices=[
|
|
('search_analytics', 'Search Analytics'),
|
|
('page_performance', 'Page Performance'),
|
|
('keyword_performance', 'Keyword Performance'),
|
|
]
|
|
)
|
|
dimension_filters = models.JSONField(
|
|
default=dict,
|
|
help_text="Filters used for this query: {dimensions, filters}"
|
|
)
|
|
data = models.JSONField(
|
|
default=list,
|
|
help_text="Cached query results"
|
|
)
|
|
date_range_start = models.DateField()
|
|
date_range_end = models.DateField()
|
|
expires_at = models.DateTimeField(
|
|
help_text="Cache expiry — refresh after this time"
|
|
)
|
|
|
|
class Meta:
|
|
app_label = 'integration'
|
|
db_table = 'igny8_gsc_metrics_cache'
|
|
ordering = ['-date_range_end']
|
|
|
|
|
|
class GSCDailyQuota(SiteSectorBaseModel):
|
|
"""Tracks daily URL Inspection API usage per site/property."""
|
|
date = models.DateField(db_index=True)
|
|
inspections_used = models.IntegerField(default=0)
|
|
quota_limit = models.IntegerField(default=2000)
|
|
|
|
class Meta:
|
|
app_label = 'integration'
|
|
db_table = 'igny8_gsc_daily_quota'
|
|
unique_together = [['site', 'date']]
|
|
```
|
|
|
|
### Migration
|
|
|
|
```
|
|
igny8_core/migrations/XXXX_gsc_integration.py
|
|
```
|
|
|
|
New tables:
|
|
1. `igny8_gsc_connections`
|
|
2. `igny8_url_inspection_records`
|
|
3. `igny8_indexing_queue`
|
|
4. `igny8_gsc_metrics_cache`
|
|
5. `igny8_gsc_daily_quota`
|
|
|
|
### API Endpoints
|
|
|
|
```
|
|
# OAuth Connection
|
|
POST /api/v1/integration/gsc/connect/ # Initiate OAuth (returns redirect URL)
|
|
GET /api/v1/integration/gsc/callback/ # OAuth callback (stores tokens)
|
|
DELETE /api/v1/integration/gsc/disconnect/ # Revoke + delete connection
|
|
GET /api/v1/integration/gsc/properties/ # List connected GSC properties
|
|
GET /api/v1/integration/gsc/status/ # Connection status
|
|
|
|
# Quota
|
|
GET /api/v1/integration/gsc/quota/ # Today's quota usage (used/limit)
|
|
|
|
# URL Inspection
|
|
POST /api/v1/integration/gsc/inspect/ # Queue single URL for inspection
|
|
POST /api/v1/integration/gsc/inspect/bulk/ # Queue multiple URLs
|
|
GET /api/v1/integration/gsc/inspections/ # List inspection records (filterable)
|
|
GET /api/v1/integration/gsc/inspections/{id}/ # Single inspection detail
|
|
|
|
# Search Analytics
|
|
GET /api/v1/integration/gsc/analytics/ # Search analytics (cached)
|
|
GET /api/v1/integration/gsc/analytics/keywords/ # Keyword performance
|
|
GET /api/v1/integration/gsc/analytics/pages/ # Page performance
|
|
GET /api/v1/integration/gsc/analytics/export/ # CSV export
|
|
|
|
# Queue Management (admin)
|
|
GET /api/v1/integration/gsc/queue/ # View queue status
|
|
POST /api/v1/integration/gsc/queue/clear/ # Clear failed/quota_exceeded items
|
|
```
|
|
|
|
### Services
|
|
|
|
```python
|
|
# igny8_core/business/integration/gsc_service.py
|
|
|
|
class GSCService:
|
|
"""Google Search Console API client wrapper."""
|
|
|
|
def get_oauth_url(self, site_id: int, redirect_uri: str) -> str:
|
|
"""Generate Google OAuth consent URL."""
|
|
pass
|
|
|
|
def handle_oauth_callback(self, code: str, site_id: int, account_id: int) -> GSCConnection:
|
|
"""Exchange auth code for tokens, create GSCConnection."""
|
|
pass
|
|
|
|
def refresh_access_token(self, connection: GSCConnection) -> bool:
|
|
"""Refresh expired access token using refresh_token."""
|
|
pass
|
|
|
|
def inspect_url(self, connection: GSCConnection, url: str) -> Dict:
|
|
"""Call URL Inspection API. Returns parsed response."""
|
|
pass
|
|
|
|
def fetch_search_analytics(
|
|
self, connection: GSCConnection,
|
|
start_date: str, end_date: str,
|
|
dimensions: list, row_limit: int = 25000
|
|
) -> List[Dict]:
|
|
"""Fetch search analytics data from GSC API."""
|
|
pass
|
|
|
|
def list_properties(self, connection: GSCConnection) -> List[str]:
|
|
"""List all GSC properties accessible by the connected account."""
|
|
pass
|
|
|
|
|
|
class IndexingQueueProcessor:
|
|
"""Processes the indexing queue respecting quota and rate limits."""
|
|
|
|
RATE_LIMIT_SECONDS = 3 # 1 request per 3 seconds
|
|
QUOTA_BUFFER = 50 # Reserve 50 inspections for manual use
|
|
|
|
def process_queue(self, site_id: int):
|
|
"""Process queued items for a site, respecting daily quota."""
|
|
quota = GSCDailyQuota.objects.get_or_create(
|
|
site_id=site_id,
|
|
date=timezone.now().date(),
|
|
defaults={'quota_limit': 2000}
|
|
)[0]
|
|
|
|
remaining = quota.quota_limit - quota.inspections_used - self.QUOTA_BUFFER
|
|
if remaining <= 0:
|
|
return {'processed': 0, 'reason': 'quota_exceeded'}
|
|
|
|
items = IndexingQueue.objects.filter(
|
|
site_id=site_id,
|
|
status='queued'
|
|
).order_by('-priority', 'date_added')[:remaining]
|
|
|
|
processed = 0
|
|
for item in items:
|
|
item.status = 'processing'
|
|
item.save()
|
|
try:
|
|
result = self.gsc_service.inspect_url(connection, item.url)
|
|
self._update_inspection_record(item, result)
|
|
item.status = 'completed'
|
|
item.date_processed = timezone.now()
|
|
quota.inspections_used += 1
|
|
processed += 1
|
|
time.sleep(self.RATE_LIMIT_SECONDS)
|
|
except QuotaExceededException:
|
|
item.status = 'quota_exceeded'
|
|
break
|
|
except Exception as e:
|
|
item.status = 'failed'
|
|
item.error_message = str(e)
|
|
finally:
|
|
item.save()
|
|
quota.save()
|
|
|
|
return {'processed': processed}
|
|
|
|
def _update_inspection_record(self, queue_item, result):
|
|
"""Create/update URLInspectionRecord from API result."""
|
|
record, created = URLInspectionRecord.objects.update_or_create(
|
|
site=queue_item.site,
|
|
url=queue_item.url,
|
|
defaults={
|
|
'last_inspection_result': result,
|
|
'verdict': result.get('inspectionResult', {}).get('indexStatusResult', {}).get('verdict', ''),
|
|
'coverage_state': result.get('inspectionResult', {}).get('indexStatusResult', {}).get('coverageState', ''),
|
|
'indexing_state': result.get('inspectionResult', {}).get('indexStatusResult', {}).get('indexingState', ''),
|
|
'last_inspected': timezone.now(),
|
|
'inspection_count': models.F('inspection_count') + 1,
|
|
}
|
|
)
|
|
# Schedule re-inspection
|
|
self._schedule_next_inspection(record)
|
|
|
|
def _schedule_next_inspection(self, record):
|
|
"""Schedule follow-up inspection based on inspection count."""
|
|
delays = {1: 1, 2: 3, 3: 6, 4: 13} # days after inspection
|
|
if record.inspection_count in delays:
|
|
record.next_inspection = timezone.now() + timedelta(days=delays[record.inspection_count])
|
|
record.save()
|
|
elif record.inspection_count > 4 and record.verdict != 'PASS':
|
|
record.status = 'manual_review'
|
|
record.next_inspection = None
|
|
record.save()
|
|
```
|
|
|
|
### Celery Tasks
|
|
|
|
```python
|
|
# igny8_core/tasks/gsc_tasks.py
|
|
|
|
@shared_task(bind=True, max_retries=3, default_retry_delay=60)
|
|
def process_indexing_queue(self, site_id: int = None):
|
|
"""Process pending indexing queue items. Runs every 5 minutes."""
|
|
# If site_id provided, process that site only
|
|
# Otherwise, process all sites with active GSCConnection
|
|
pass
|
|
|
|
@shared_task(bind=True, max_retries=3, default_retry_delay=300)
|
|
def refresh_gsc_tokens(self):
|
|
"""Refresh expiring GSC OAuth tokens. Runs hourly."""
|
|
expiring = GSCConnection.objects.filter(
|
|
status='active',
|
|
token_expiry__lte=timezone.now() + timedelta(minutes=10)
|
|
)
|
|
for conn in expiring:
|
|
GSCService().refresh_access_token(conn)
|
|
|
|
@shared_task(bind=True, max_retries=3, default_retry_delay=300)
|
|
def fetch_search_analytics(self):
|
|
"""Fetch and cache search analytics for all connected sites. Runs daily."""
|
|
pass
|
|
|
|
@shared_task(bind=True, max_retries=3, default_retry_delay=60)
|
|
def schedule_reinspections(self):
|
|
"""Add due re-inspections to the queue. Runs daily."""
|
|
due = URLInspectionRecord.objects.filter(
|
|
next_inspection__lte=timezone.now(),
|
|
status__in=['not_indexed', 'indexing_requested']
|
|
)
|
|
for record in due:
|
|
IndexingQueue.objects.get_or_create(
|
|
site=record.site,
|
|
url=record.url,
|
|
status='queued',
|
|
defaults={'priority': 90, 'url_inspection_record': record}
|
|
)
|
|
|
|
@shared_task(bind=True)
|
|
def auto_queue_published_content(self, content_id: int):
|
|
"""Queue newly published content for GSC inspection. Triggered by publish signal."""
|
|
content = Content.objects.get(id=content_id)
|
|
if not content.external_url:
|
|
return
|
|
connection = GSCConnection.objects.filter(
|
|
site=content.site, status='active'
|
|
).first()
|
|
if not connection:
|
|
return
|
|
record, _ = URLInspectionRecord.objects.get_or_create(
|
|
site=content.site, url=content.external_url,
|
|
defaults={'content': content, 'sector': content.sector, 'account': content.account}
|
|
)
|
|
IndexingQueue.objects.create(
|
|
site=content.site, sector=content.sector, account=content.account,
|
|
url=content.external_url, url_inspection_record=record,
|
|
priority=100, status='queued'
|
|
)
|
|
```
|
|
|
|
**Beat schedule additions** (add to `igny8_core/celery.py`):
|
|
```python
|
|
'process-indexing-queue': {
|
|
'task': 'gsc.process_indexing_queue',
|
|
'schedule': crontab(minute='*/5'), # Every 5 minutes
|
|
},
|
|
'refresh-gsc-tokens': {
|
|
'task': 'gsc.refresh_gsc_tokens',
|
|
'schedule': crontab(minute=30), # Every hour at :30
|
|
},
|
|
'fetch-search-analytics': {
|
|
'task': 'gsc.fetch_search_analytics',
|
|
'schedule': crontab(hour=4, minute=0), # Daily at 4 AM
|
|
},
|
|
'schedule-reinspections': {
|
|
'task': 'gsc.schedule_reinspections',
|
|
'schedule': crontab(hour=5, minute=0), # Daily at 5 AM
|
|
},
|
|
```
|
|
|
|
### Auto-Indexing Signal
|
|
|
|
Connect to Content model's post-publish flow:
|
|
|
|
```python
|
|
# igny8_core/business/integration/signals.py
|
|
|
|
from django.db.models.signals import post_save
|
|
from django.dispatch import receiver
|
|
|
|
@receiver(post_save, sender='writer.Content')
|
|
def queue_for_gsc_inspection(sender, instance, **kwargs):
|
|
"""When content is published, auto-queue for GSC inspection."""
|
|
if instance.site_status == 'published' and instance.external_url:
|
|
auto_queue_published_content.delay(instance.id)
|
|
```
|
|
|
|
### Credit Costs
|
|
|
|
| Operation | Credits | Notes |
|
|
|-----------|---------|-------|
|
|
| GSC OAuth connection setup | 1 | One-time per connection |
|
|
| URL inspections (per 100) | 0.1 | Batch pricing |
|
|
| Indexing request (per URL) | 0.05 | Minimal cost |
|
|
| Analytics caching (per site/month) | 0.5 | Monthly recurring |
|
|
|
|
Add to `CreditCostConfig`:
|
|
```python
|
|
CreditCostConfig.objects.get_or_create(
|
|
operation_type='gsc_inspection',
|
|
defaults={'display_name': 'GSC URL Inspection', 'base_credits': 1}
|
|
)
|
|
CreditCostConfig.objects.get_or_create(
|
|
operation_type='gsc_analytics',
|
|
defaults={'display_name': 'GSC Analytics Sync', 'base_credits': 1}
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## 4. IMPLEMENTATION STEPS
|
|
|
|
### Step 1: Create GSC Models
|
|
File to create/modify:
|
|
- `backend/igny8_core/business/integration/gsc_models.py` (or add to existing `models.py`)
|
|
- 5 new models: GSCConnection, URLInspectionRecord, IndexingQueue, GSCMetricsCache, GSCDailyQuota
|
|
|
|
### Step 2: Create and Run Migration
|
|
```bash
|
|
cd /data/app/igny8/backend
|
|
python manage.py makemigrations --name gsc_integration
|
|
python manage.py migrate
|
|
```
|
|
|
|
### Step 3: Build GSCService
|
|
File to create:
|
|
- `backend/igny8_core/business/integration/gsc_service.py`
|
|
- Requires `google-auth`, `google-auth-oauthlib`, `google-api-python-client` packages
|
|
|
|
Add to `requirements.txt`:
|
|
```
|
|
google-auth>=2.0.0
|
|
google-auth-oauthlib>=1.0.0
|
|
google-api-python-client>=2.0.0
|
|
```
|
|
|
|
### Step 4: Build IndexingQueueProcessor
|
|
File to create:
|
|
- `backend/igny8_core/business/integration/indexing_queue_processor.py`
|
|
|
|
### Step 5: Build Celery Tasks
|
|
File to create:
|
|
- `backend/igny8_core/tasks/gsc_tasks.py`
|
|
|
|
Add beat schedule entries to:
|
|
- `backend/igny8_core/celery.py`
|
|
|
|
### Step 6: Build Auto-Indexing Signal
|
|
File to create:
|
|
- `backend/igny8_core/business/integration/signals.py`
|
|
|
|
Register in:
|
|
- `backend/igny8_core/business/integration/apps.py` — `ready()` method
|
|
|
|
### Step 7: Build Serializers
|
|
File to create:
|
|
- `backend/igny8_core/modules/integration/serializers/gsc_serializers.py`
|
|
|
|
### Step 8: Build ViewSets and URLs
|
|
Files to create:
|
|
- `backend/igny8_core/modules/integration/views/gsc_views.py`
|
|
- Modify `backend/igny8_core/modules/integration/urls.py` — register GSC endpoints
|
|
|
|
### Step 9: Add OAuth Settings
|
|
Add to `backend/igny8_core/settings.py`:
|
|
```python
|
|
# Google OAuth 2.0 (GSC Integration)
|
|
GOOGLE_CLIENT_ID = env('GOOGLE_CLIENT_ID', default='')
|
|
GOOGLE_CLIENT_SECRET = env('GOOGLE_CLIENT_SECRET', default='')
|
|
GOOGLE_REDIRECT_URI = env('GOOGLE_REDIRECT_URI', default='')
|
|
```
|
|
|
|
### Step 10: Frontend
|
|
Files to create in `frontend/src/`:
|
|
- `pages/Integration/GSCDashboard.tsx` — main GSC dashboard
|
|
- `pages/Integration/GSCAnalytics.tsx` — search analytics with charts
|
|
- `pages/Integration/GSCInspections.tsx` — URL inspection list with status badges
|
|
- `pages/Integration/GSCConnect.tsx` — OAuth connection flow
|
|
- `components/Integration/QuotaIndicator.tsx` — daily quota usage bar
|
|
- `components/Integration/InspectionStatusBadge.tsx` — status badges
|
|
- `stores/gscStore.ts` — Zustand store
|
|
- `api/gsc.ts` — API client
|
|
|
|
### Step 11: Tests
|
|
```bash
|
|
cd /data/app/igny8/backend
|
|
python manage.py test igny8_core.business.integration.tests.test_gsc_service
|
|
python manage.py test igny8_core.business.integration.tests.test_indexing_queue
|
|
python manage.py test igny8_core.modules.integration.tests.test_gsc_views
|
|
```
|
|
|
|
---
|
|
|
|
## 5. ACCEPTANCE CRITERIA
|
|
|
|
- [ ] 5 new database tables created and migrated successfully
|
|
- [ ] Google OAuth 2.0 flow works: connect → consent → callback → tokens stored
|
|
- [ ] GSC properties listed after successful OAuth connection
|
|
- [ ] Token refresh works automatically before expiry via Celery task
|
|
- [ ] URL Inspection API calls succeed and results stored in URLInspectionRecord
|
|
- [ ] Daily quota tracked in GSCDailyQuota, respects 2,000/day limit
|
|
- [ ] Rate limit of 1 request/3 seconds enforced in queue processor
|
|
- [ ] Re-inspection schedule runs: 1 day, 3 days, 6 days, 13 days after initial check
|
|
- [ ] URLs not indexed after Check 4 marked as 'manual_review'
|
|
- [ ] Content publish triggers auto-queue at priority 100
|
|
- [ ] Search analytics data fetched and cached with 24-hour TTL
|
|
- [ ] Analytics endpoints return cached data with date range filtering
|
|
- [ ] All endpoints require authentication and enforce account isolation
|
|
- [ ] Frontend GSC dashboard shows: connection status, quota usage, inspection list, analytics charts
|
|
- [ ] inspection status badges display correctly on URL list
|
|
- [ ] `google-auth`, `google-auth-oauthlib`, `google-api-python-client` added to requirements.txt
|
|
- [ ] Disconnecting GSC revokes token and deletes GSCConnection
|
|
|
|
---
|
|
|
|
## 6. CLAUDE CODE INSTRUCTIONS
|
|
|
|
### Execution Order
|
|
1. Read `backend/igny8_core/business/integration/models.py` — understand existing SiteIntegration, SyncEvent
|
|
2. Read `backend/igny8_core/modules/integration/urls.py` — understand existing URL patterns
|
|
3. Read `backend/igny8_core/celery.py` — understand beat schedule registration
|
|
4. Add new packages to requirements.txt
|
|
5. Create GSC models (5 models)
|
|
6. Create migration, run it
|
|
7. Build GSCService (OAuth + API client)
|
|
8. Build IndexingQueueProcessor
|
|
9. Build Celery tasks (4 tasks) + register in beat schedule
|
|
10. Build auto-indexing signal
|
|
11. Build serializers, ViewSets, URLs
|
|
12. Build frontend components
|
|
|
|
### Key Constraints
|
|
- ALL primary keys are `BigAutoField` (integer). No UUIDs.
|
|
- Model class names: GSCConnection, URLInspectionRecord, IndexingQueue, GSCMetricsCache, GSCDailyQuota (descriptive names, not plural)
|
|
- Frontend: `.tsx` files, Zustand stores, Vitest testing
|
|
- Celery app name: `igny8_core`
|
|
- All db_tables use `igny8_` prefix
|
|
- Tokens MUST be encrypted at rest (use same encryption as SiteIntegration.credentials_json)
|
|
- OAuth client_id/secret must be in environment variables, never in code
|
|
- Follow existing integration app patterns for URL structure
|
|
|
|
### File Tree (New/Modified)
|
|
```
|
|
backend/igny8_core/
|
|
├── business/integration/
|
|
│ ├── models.py # MODIFY or NEW gsc_models.py: 5 new models
|
|
│ ├── gsc_service.py # NEW: GSCService (OAuth + API)
|
|
│ ├── indexing_queue_processor.py # NEW: IndexingQueueProcessor
|
|
│ ├── signals.py # NEW: auto-indexing signal
|
|
│ └── apps.py # MODIFY: register signals in ready()
|
|
├── tasks/
|
|
│ └── gsc_tasks.py # NEW: 4 Celery tasks
|
|
├── celery.py # MODIFY: add 4 beat schedule entries
|
|
├── settings.py # MODIFY: add GOOGLE_* settings
|
|
├── modules/integration/
|
|
│ ├── serializers/
|
|
│ │ └── gsc_serializers.py # NEW
|
|
│ ├── views/
|
|
│ │ └── gsc_views.py # NEW
|
|
│ └── urls.py # MODIFY: register GSC endpoints
|
|
├── migrations/
|
|
│ └── XXXX_gsc_integration.py # NEW: auto-generated
|
|
├── requirements.txt # MODIFY: add google-auth packages
|
|
|
|
frontend/src/
|
|
├── pages/Integration/
|
|
│ ├── GSCDashboard.tsx # NEW
|
|
│ ├── GSCAnalytics.tsx # NEW
|
|
│ ├── GSCInspections.tsx # NEW
|
|
│ └── GSCConnect.tsx # NEW
|
|
├── components/Integration/
|
|
│ ├── QuotaIndicator.tsx # NEW
|
|
│ └── InspectionStatusBadge.tsx # NEW
|
|
├── stores/
|
|
│ └── gscStore.ts # NEW: Zustand store
|
|
├── api/
|
|
│ └── gsc.ts # NEW: API client
|
|
```
|
|
|
|
### Cross-References
|
|
- **01E** (blueprint-aware pipeline): triggers auto-indexing after publish, hub pages prioritized
|
|
- **02E** (backlinks): GSC impressions data feeds backlink KPI dashboard
|
|
- **02F** (optimizer): GSC position data identifies optimization candidates
|
|
- **03A** (WP plugin standalone): standalone plugin has GSC dashboard tab
|
|
- **03B** (WP plugin connected): connected mode syncs index statuses from IGNY8 to WP
|
|
- **04B** (reporting): GSC metrics (clicks, impressions, CTR) feed into service reports
|