The first dashboard I built for the invoice platform had twelve panels. CPU, memory, Node event-loop lag, HTTP p99, Mongo connections, Lambda duration, SQS visible messages, and a sparkline nobody could explain.

It looked serious. It missed the outage where uploads succeeded and nothing processed for forty minutes — because the queue was deep, workers were healthy, and a bad deploy had made every validation fail soft into a silent retry loop.

age queue wait dlq poison depth p99 model latency fail% validation
CPU graphs lie. Queue age and validation fail rate tell the truth.

Four signals that earned a page

We cut the board to four numbers that map to user pain:

  1. Queue age — oldest message waiting. If this climbs, work is late even if boxes are green.
  2. DLQ depth — poison or schema drift. Finance notices weird vendors before your CPU does.
  3. Validation fail rate — parser or model regression. Spikes after deploys are a smell, not "traffic."
  4. Model p99 — budget and timeout risk. When this blows, visibility timeouts and double spends follow.

Everything else is diagnostic. Useful when you are already investigating. Useless as a wake-up call.

SLOs that match the architecture

We stopped promising "API p99 under 200ms" as the only SLO. Ingest is fast by design. The promise that matters for customers is time from upload to extracted-or-review.

So we track:

  • Ingest availability and latency (the HTTP face)
  • End-to-end processing latency for successful jobs
  • Review-queue backlog age (humans are part of the system)

If you only measure the edge, you will optimize a 202 that hides a stuck worker.

Alerts that survive a quiet night

Good alerts are rare and actionable:

  • Queue age > N minutes for M minutes → page
  • DLQ depth rising and not draining → ticket, then page if still rising
  • Validation fail rate > baseline after a deploy → page the deployer
  • Model error rate or timeout share > threshold → page, and flip to OCR/degraded mode if you have it

Bad alerts are high CPU, single-host flaps, and "error log contains Error." Those train people to mute Slack.

We already log job_id, attempt, and stage. Observability is unfinished until an on-call engineer can answer:

  • Which account is affected?
  • Which object key?
  • Did we already send the webhook?
  • Is this a redelivery or a new job?

Without those fields, you have metrics and a prayer. With them, a replay is a decision, not an archaeology dig.

What I would delete from your dashboard

  • Per-instance CPU unless you are capacity planning
  • Generic "5xx rate" without route and job type
  • Model token counters as alerts (keep them as cost reports)
  • Anything you have never used in a real incident

Keep a deep dashboard for forensics. Keep a short one for pages. If a chart has never changed a decision, it is decoration — and decoration is how you miss the queue filling up while the CPU looks calm.