A background job runner is one of those systems you only notice when it breaks. When it works, nobody thinks about it. When it doesn’t, you get duplicate emails, missing reports, and a dead-letter table that quietly grows until somebody notices.
Most of the design of a resilient runner is about being honest about the failure modes. Retries are inevitable; idempotency is the contract you make with your future self when retries fire.
Idempotency is the contract
A job is idempotent if running it twice produces the same observable result as running it once. Almost nothing in distributed systems is naturally idempotent — sending an email charges the recipient’s attention twice, charging a card twice is worse, posting to a webhook twice may or may not be safe depending on the receiver.
The fix is to make idempotency explicit. Every job declares an idempotency key, and the worker writes through a unique index keyed on that key. A duplicate execution then becomes a no-op at the storage layer rather than a duplicate at the side-effect layer.
async function executeJob(job: Job): Promise<void> {
const key = job.idempotencyKey();
const alreadyDone = await effects.exists(key);
if (alreadyDone) {
await markComplete(job);
return;
}
await runSideEffect(job);
await effects.record(key, job.result());
await markComplete(job);
}
The interesting case is the worker that crashes between the side-effect and the record. The idempotency key still helps: on retry, the worker checks the effects table, sees the row was written, and marks the job complete without re-running the side-effect.
Retries with backoff
A naive retry loop that re-executes a job every second is a denial-of-service attack on whatever the job depends on. Backoff — exponential, with jitter — gives the downstream system a chance to recover and spreads retries out so a thundering herd does not arrive at the moment the downstream comes back up.
A reasonable default is exponential backoff with full jitter: delay = random(0, base * multiplier^attempt), capped at a maximum. The exact numbers depend on the job, but the shape — small initial delay, exponential growth, hard cap, plenty of jitter — holds up across a wide range of workloads.
Different jobs need different policies. A webhook delivery to a third-party API wants short, aggressive retries with a few attempts. A nightly report job wants long delays between attempts because the input only changes once a day. Encoding the policy on the job rather than in the runner keeps the runner generic and the behavior legible.
Dead-lettering is part of the system
A job that retries forever is not a resilient system; it is a system that hides its bugs. After a configurable number of attempts the job should move to a dead-letter table with a clear record of what happened, when, and why. Anything else is a graveyard.
The dead-letter table should be small and well-labeled. The minimum useful fields are the job identifier, the last error, the attempt count, the timestamp of the last attempt, and enough of the job payload to reproduce the failure. Without those, the dead-letter table is just a place where jobs go to be forgotten.
A dead-letter table that is on a dashboard is part of the system. A dead-letter table that is not, is not.
At-least-once is the right default
Exactly-once delivery is not a real property of distributed systems. It is a marketing phrase. The honest version is at-least-once delivery combined with idempotent workers, which produces the same observable behavior that exactly-once promises, without the operational cost of distributed transactions.
Build for at-least-once. Make every side-effect idempotent. Trust the dead-letter table to catch the cases where the worker is honest about giving up. The system that does those three things well will outperform the one that pretends exactly-once is real.
Where to start
The minimum viable resilient runner is small: a queue, a worker pool, a lease/heartbeat protocol, retry with backoff, an idempotency contract, and a dead-letter table on a dashboard. Everything else is detail. The detail matters, but the shape is the shape.