Add operational checks and visibility for background metadata/embedding processing #28

Closed
opened 2026-04-13 20:13:46 +00:00 by sgcommand · 0 comments
Member

Problem

Background enrichment only works if operators can tell whether metadata extraction and embedding generation are keeping up. Current visibility into pending, failed, and retried background work appears limited.

Impact

  • Silent backlog growth
  • Failed enrichment can go unnoticed
  • Hard to distinguish transient AI/provider issues from stuck records

Expected behavior

AMCS should expose operational checks for background metadata and embedding processing so failures and backlogs are visible before users notice degraded behavior.

Acceptance criteria

  • There is a recurring operational check or report for pending/failed metadata and embedding work
  • Logs or metrics expose backlog size, retry counts, and recent failure reasons
  • Missing embeddings continue to degrade gracefully in user-facing flows
  • Operators can tell whether background workers are healthy without manual database inspection
  • Documentation identifies how and where the background checks run
## Problem Background enrichment only works if operators can tell whether metadata extraction and embedding generation are keeping up. Current visibility into pending, failed, and retried background work appears limited. ## Impact - Silent backlog growth - Failed enrichment can go unnoticed - Hard to distinguish transient AI/provider issues from stuck records ## Expected behavior AMCS should expose operational checks for background metadata and embedding processing so failures and backlogs are visible before users notice degraded behavior. ## Acceptance criteria - There is a recurring operational check or report for pending/failed metadata and embedding work - Logs or metrics expose backlog size, retry counts, and recent failure reasons - Missing embeddings continue to degrade gracefully in user-facing flows - Operators can tell whether background workers are healthy without manual database inspection - Documentation identifies how and where the background checks run
Sign in to join this conversation.