igny8/docs/fixes/footer-widget-pagination-fix.md

# Footer Widget Pagination Fix - Summary

## Problem

All pages with `ThreeWidgetFooter` are calculating metrics using **page-filtered arrays** instead of **total counts** from the API. This causes incorrect metric values when users are viewing paginated results.

### Example:
- If there are 100 total keywords with 10 on the current page
- And 5 keywords on the current page don't have a `cluster_id`
- The footer shows "Unmapped: 5" instead of the actual total unmapped count

## Root Cause

The footer widgets use JavaScript `.filter()` methods on the local `items` array (which only contains the current page's data) instead of making separate API calls to get total counts for each status.

```typescript
// WRONG - Uses page-filtered array
{ label: 'Unmapped', value: keywords.filter(k => !k.cluster_id).length }

// CORRECT - Uses total count from API
{ label: 'Unmapped', value: totalUnmapped }
```

## Solution Pattern

For each affected page:

1. **Add state variables for total counts**
2. **Create a `loadTotalMetrics()` function** that makes lightweight API calls (page_size=1) filtered by status
3. **Call `loadTotalMetrics()` when site/sector changes**
4. **Update footer widget** to use the total count state instead of filtering local arrays

## Files Fixed

### ✅ 1. Keywords.tsx
- Added: `totalClustered`, `totalUnmapped`, `totalVolume` state
- Added: `loadTotalMetrics()` function
- Updated: Footer widget to use total counts

### ✅ 2. Clusters.tsx
- Added: `totalWithIdeas`, `totalReady` state
- Added: `loadTotalMetrics()` function
- Updated: Footer widget to use total counts

### ⏳ 3. Ideas.tsx
- **TODO**: Add `totalInTasks`, `totalPending` state
- **TODO**: Add `loadTotalMetrics()` function with calls to:
  - `fetchContentIdeas({ status: 'queued' })` → `totalInTasks`
  - `fetchContentIdeas({ status: 'new' })` → `totalPending`
- **TODO**: Update footer widget metrics

###  ⏳ 4. Tasks.tsx
- **TODO**: Add total count state variables
- **TODO**: Add `loadTotalMetrics()` function
- **TODO**: Update footer widget

### ⏳ 5. Content.tsx
- **TODO**: Add total count state variables for each status (draft, review, approved)
- **TODO**: Add `loadTotalMetrics()` function
- **TODO**: Update footer widget

### ⏳ 6. Images.tsx
- **TODO**: Add total count state variables
- **TODO**: Add `loadTotalMetrics()` function
- **TODO**: Update footer widget

### ⏳ 7. Review.tsx
- **TODO**: Add total count state variables
- **TODO**: Add `loadTotalMetrics()` function
- **TODO**: Update footer widget

### ⏳ 8. Approved.tsx
- **TODO**: Add total count state variables
- **TODO**: Add `loadTotalMetrics()` function
- **TODO**: Update footer widget

## Implementation Template

### Step 1: Add State Variables

```typescript
// Total counts for footer widget (not page-filtered)
const [totalWithStatus1, setTotalWithStatus1] = useState(0);
const [totalWithStatus2, setTotalWithStatus2] = useState(0);
```

### Step 2: Create loadTotalMetrics Function

```typescript
// Load total metrics for footer widget (not affected by pagination)
const loadTotalMetrics = useCallback(async () => {
  if (!activeSite) return;

  try {
    // Get items with status1
    const status1Res = await fetchItems({
      page_size: 1,
      site_id: activeSite.id,
      ...(activeSector?.id && { sector_id: activeSector.id }),
      status: 'status1',
    });
    setTotalWithStatus1(status1Res.count || 0);

    // Get items with status2
    const status2Res = await fetchItems({
      page_size: 1,
      site_id: activeSite.id,
      ...(activeSector?.id && { sector_id: activeSector.id }),
      status: 'status2',
    });
    setTotalWithStatus2(status2Res.count || 0);
  } catch (error) {
    console.error('Error loading total metrics:', error);
  }
}, [activeSite, activeSector]);
```

### Step 3: Call on Mount/Change

```typescript
// Load total metrics when site/sector changes
useEffect(() => {
  loadTotalMetrics();
}, [loadTotalMetrics]);
```

### Step 4: Update Footer Widget

```typescript
<ThreeWidgetFooter
  pageProgress={{
    metrics: [
      { label: 'Total', value: totalCount },
      { label: 'Status 1', value: totalWithStatus1, percentage: `${totalCount > 0 ? Math.round((totalWithStatus1 / totalCount) * 100) : 0}%` },
      { label: 'Status 2', value: totalWithStatus2 },
    ],
    progress: {
      value: totalCount > 0 ? Math.round((totalWithStatus1 / totalCount) * 100) : 0,
      label: 'Processed',
      color: 'blue',
    },
    hint: totalWithStatus2 > 0
      ? `${totalWithStatus2} items ready for processing`
      : 'All items processed!',
  }}
  // ... rest of props
/>
```

## Testing Checklist

For each fixed page, verify:

- [ ] Footer metrics show correct total counts (not page counts)
- [ ] Metrics update when changing sites
- [ ] Metrics update when changing sectors
- [ ] Metrics are consistent with automation page metrics
- [ ] Performance is acceptable (lightweight API calls with page_size=1)

## Related Files

- `/frontend/src/components/dashboard/ThreeWidgetFooter.tsx`
- `/frontend/src/pages/Automation/AutomationPage.tsx` (reference implementation)
- All planner and writer page files

## API Endpoints Used

All pages use their respective `fetch*` functions with filters:
- `fetchKeywords({ status, page_size: 1 })`
- `fetchClusters({ status, page_size: 1 })`
- `fetchContentIdeas({ status, page_size: 1 })`
- `fetchTasks({ status, page_size: 1 })`
- `fetchContent({ status, page_size: 1 })`
- `fetchContentImages({ status, page_size: 1 })`

The `page_size: 1` ensures minimal data transfer while still getting the count.

## Performance Considerations

- Each page makes 2-3 additional API calls on load
- Calls are lightweight (page_size=1, only count is used)
- Calls are cached until site/sector changes
- Total overhead: ~100-300ms per page load (acceptable)

## Automation Page Consistency

The AutomationPage already uses this pattern correctly:
- Lines 99-149: Fetches total counts for all metrics
- Uses `fetchKeywords({ status: 'new' })`, `fetchKeywords({ status: 'mapped' })`, etc.
- Sets metrics in state: `setMetrics({ keywords: { total, new, mapped } })`
- All stage cards and metric cards use these pre-fetched totals

This fix brings all other pages in line with the Automation page's correct implementation.