The best complement to a queue is a worker that does not mind running twice. The worst is a worker that "mostly" does.

This is not philosophy. It is how you sleep when SQS redelivers, when you replay a DLQ after a bad deploy, and when a new teammate asks why production sent two webhooks on Tuesday.

message job · key object key attempt S3 bytes canonical input replayable ledger intent complete skip on retry idempotent
Small messages. Stored inputs. A ledger for every side effect.

Inputs live in the object store, not in the message

A queue message should be small and boring: job type, account id, object key, idempotency key, attempt. The PDF lives in S3. The email lives in the mailbox you already stored. If the message is a 2MB JSON blob of "the document we parsed so far," you will eventually hit size limits and you will never know which version was canonical.

Replay means: put the same key on the queue again. The worker loads bytes, runs the pipeline, writes results. No hidden state in the process.

Side effects need a ledger

The handler will call APIs. Those are the dangerous parts.

For each side effect — send reminder, POST webhook, charge a usage counter — we record an intent row before the call and a completion row after. On retry:

  • Completed effects are skipped
  • Intents without completion are the gray area: the call may have succeeded and the process died. That is why webhooks should be idempotent on the receiver, and why reminder emails include a unique id the mailer can de-dupe.

You cannot make every third-party API perfect. You can make your worker obvious: here is what we already did.

Logs are for a single job id

When extraction fails, nobody wants a grep across "error" at 14:07. They want job_id=... and the object key.

We structure logs with job id, attempt, and stage (fetch, extract, validate, persist). Traces are nice; consistent fields are mandatory. A replay should produce a new attempt number and the same job id so you can see the story in order.

Metrics that mattered for us were not "CPU on the node." They were: queue age, DLQ depth, validation fail rate, model latency, pool hit rate (on the FaaS side). If validation fail rate spikes, you shipped a parser bug or a vendor changed letterhead — not a "scale" event.

Postgres notes I actually use

The invoice product's system of record is MongoDB. Other systems I have run sit on Postgres. A few habits transfer:

  • Put a unique constraint on the idempotency key. Application "select then insert" will lose a race.
  • Index what you filter in the operator UI (account + status + created_at). Do not index every JSON path you might query someday.
  • Connection pooling belongs in front of the database if you have many workers. Hundreds of direct connections are a tax.

I do not have a tale about PgBouncer saving a Fortune 500. I have a tale about unique constraints saving us from double inserts when the queue got chatty.

A worker checklist

Before you call a job "done":

  • Can I replay it from the object key alone?
  • Will a second run skip completed side effects?
  • Does a poison payload leave the queue and enter a place humans look?
  • Can I find all logs for one job without knowing the timestamp?

If the answer is no, the queue is a delay line for a request handler you did not want to admit you still have. Fix the worker first. The fancy bus can wait.