The first version of our invoice pipeline did the honest thing: a request came in, we validated the file, called a model, wrote to MongoDB, and fired a webhook. Then we returned 200.

It worked in staging. It failed in the only way production fails — slowly, then all at once. A 40-page scan and a slow model call sat on the Node event loop. Timeouts retried. Retries stampeded the same work. Operators saw a spinner. We saw event-loop delay and a queue of nothing, because there was no queue.

POST /upload auth · store S3 + SQS object · key 202 · job id workers extract validate persist webhook / DLQ
Ingest returns fast. The workflow lives on the queue.

The request is not a workflow

HTTP is a conversation. Invoice processing is a job. Those should not share a lifetime.

The ingest path now does four things:

  1. Authenticate and authorize.
  2. Store the raw bytes (S3, not the database).
  3. Enqueue a message that names the object and an idempotency key.
  4. Return 202 with a job id.

Extraction, totaling, reminder scheduling, and vendor webhooks happen in workers. If OpenAI is having a bad afternoon, users still uploaded successfully. The job is delayed, not lost.

SQS is enough until it is not

We used AWS SQS, not Kafka. That was not a lack of ambition. Invoice jobs are discrete, ordered per document, and bursty around month-end. We needed:

  • At-least-once delivery we could reason about
  • A dead-letter queue for poison PDFs
  • Visibility timeouts longer than the slowest model call
  • Zero extra cluster to babysit

Kafka would have given us replay and consumer groups. It would also have given us a second career as part-time brokers. Replay is useful; it is not free. If you cannot say which team owns the cluster at 2am, do not put your invoices on it.

When we needed fan-out (extract once, notify billing and search), we published a second message after the document was valid — not a topic topology that looked good on a whiteboard.

Idempotency is the real feature

At-least-once means your handler will run twice. If "twice" sends two reminder emails or double-writes a payment, you do not have a retry policy. You have a bug with retries.

Every job carries a key derived from accountId + sha256(object). The worker:

  • Writes a processing row unique on that key
  • Does the work
  • Marks the row complete

A redelivery sees the row and exits. Partial failure is the sharp edge: if you crash after the model call but before the write, you will call the model again. That costs money. It should not corrupt data. Spend the token budget; do not invent a second invoice number.

Visibility timeout must exceed the p99 of the worker, including the model. If it does not, SQS will hand the same message to another worker while the first is still talking to GPT. Unique keys still save you. Overlapping model spend does not.

Dead letters are a product surface

A DLQ is not a trash can. It is where the weird vendors live.

We inspect DLQ payloads weekly: encrypted zips, HTML pretending to be PDF, statements with no invoice number. Some get a parser. Some get a "needs human" state in the UI. Sweeping the DLQ into /dev/null trains you to ignore the only documents that hurt finance teams.

What changed in the numbers that matter

I will not cite a fictional 10 million events a day. What we actually needed was:

  • Ingest p95 in the tens of milliseconds, not seconds
  • Workers that scale on queue depth, not on web traffic
  • A way to pause processing without refusing uploads

Splitting accept from process gave us all three. The code is not exotic. The discipline is: no network you do not control inside the request that the user is waiting on.

If you are still extracting invoices inside POST /upload, move the file, enqueue the key, and return. The rest is a worker problem — which is a better class of problem.