Implementation record
Keep the evidence that makes a retry safe.
| Store | When | On repeat delivery |
|---|---|---|
| Stable event or idempotency key | Before starting the side effect | Return the stored result when the key already completed |
| Processing state | While the first request is in progress | Avoid two workers performing the same action together |
| Result and expiry | After the side effect succeeds | Preserve the answer for the retry window, then expire deliberately |
What you are looking at
The danger is not failure. It is a success whose receipt got lost.
A webhook error does not prove that nothing happened. The receiver may have completed an effect and then lost its acknowledgement through a timeout, dropped connection or restart. Repeating that action without duplicate protection can repeat the effect.
The simulator models sequential attempts and assumes that a completed effect and its stored result stay together. Its Hard fail scenario explicitly fails before charging. Real errors do not establish that boundary, and real concurrent requests or a crash between the effect and result storage need additional handling.
The parts, one by one
One technique makes it correct; three make it well-behaved.
Each control in the demo is one of these. Idempotency is the one that decides whether you double-charge. The other three decide how the retrying behaves while it gets there.
Idempotency key
The one that fixes the charge
A stable id identifies the logical operation across attempts. The receiving implementation must enforce it while work is in progress and after completion, including a crash before the result is saved. The sequential simulator assumes that enforcement.
Exponential backoff
Space the retries out
Retry after 1s, then 2s, 4s, 8s, doubling each time, instead of a tight fixed interval. It gives a struggling receiver room to recover and stops a retry storm from becoming a self-inflicted outage.
Jitter
Do not let everyone retry in lockstep
Add a random offset to each backoff. Without it, a thousand senders that all failed at the same moment retry at the same moments forever, a thundering herd. Jitter smears them across the interval so the load is smooth.
Dead-letter after a cap
Know when to stop
Retries should not run forever. After a handful of attempts, park the event in a dead-letter queue where a human can see it and replay it, rather than hammering an endpoint that is clearly not coming back.
// The receiver everyone writes first. Do not ship it.
app.post('/webhooks', async (req, res) => {
const event = req.body; // e.g. charge.succeeded
await chargeCard(event.customer, event.amount); // the side effect
res.sendStatus(200);
});
// If the ack is lost after chargeCard runs, the sender retries,
// this handler runs again, and the customer is charged twice.This fictional handler repeats the effect if the acknowledgement is lost and the sender retries. Repeated delivery must be anticipated; it is not guaranteed to occur for every event.
Why retrying alone is not enough
You cannot fix this by making delivery more reliable.
Every layer between the two systems can drop the acknowledgement without dropping the work. The receiver has to assume the event will arrive again and make that harmless.
Where the duplicate comes from
The at-least-once tax
Check the provider’s delivery contract. Repeated delivery is possible under at-least-once delivery; it is not a prediction that every event will be duplicated.
An acknowledgement can be lost after the effect, leaving the sender uncertain about completion.
Match a repeated request to the same logical operation and validate its parameters rather than inventing a new key per attempt.
Enforce duplicate safety at the effect boundary, including concurrent attempts and recovery after a crash; transport retries alone cannot provide it.
app.post('/webhooks', async (req, res) => {
const key = req.header('Idempotency-Key') ?? req.body.id; // stable per event
const seen = await store.get(key);
if (seen) return res.status(200).send(seen.result); // replay, do not act again
const result = await chargeCard(req.body.customer, req.body.amount);
await store.set(key, { result }); // record on completion
res.status(200).send(result);
});This sketch replays a result after it has been saved. It has a race between store.get and chargeCard, and a crash window before store.set. It is not safe to ship as shown. A real design must enforce the stable operation key at the effect boundary and reconcile uncertain outcomes.
async function deliver(url, body, attempt = 1) {
const res = await post(url, body).catch(() => null);
if (res && res.ok) return res;
if (attempt >= 5) return deadLetter(body); // give up cleanly
const base = 2 ** (attempt - 1) * 1000; // 1s, 2s, 4s, 8s
const wait = base / 2 + Math.random() * base; // random delay between half and 1.5 times base
await sleep(wait);
return deliver(url, body, attempt + 1);
}Illustrative bounded sender only. A real sender must classify the fault and follow provider retry guidance; it must not retry every rejection. The hero uses four capped attempts, while this separate sketch uses five.
Setting it up in an internal tool
Pick a stable key, give the store a TTL, and watch the dead-letter queue.
These decisions belong in a production design. The short code sketches do not implement authentication, request validation, atomic duplicate protection or recovery from every crash window.
Key
Derive it from the event, once
Choose a stable identifier for the logical effect and retain its scope and parameters. A provider event ID may identify a delivery without identifying every possible business operation. Do not generate a fresh key for each retry.
Store
Give the idempotency store a TTL
Choose retention from the provider contract and the longest permitted automatic or manual replay window. Twenty-four hours is not a universal safe limit. Once a key is removed, an old event may look new; keep the evidence required by the recovery policy.
Watch
A dead-letter queue you actually monitor
Exhausted retries have to land somewhere queryable, with an alert on depth. A dead-letter queue nobody looks at is just a slower way to lose events. This is the link back to monitoring your integrations.
// Idempotency store: key -> result, with a TTL so it cannot grow forever.
await store.set(key, { result }, { ttlHours: 24 });
// Dead-letter: when retries are exhausted, park the event for a human.
async function deadLetter(event) {
await dlq.add(event); // queryable queue, alert on depth
metrics.increment('webhook.dead_letter');
}Illustrative storage and dead-letter calls. The 24-hour value is a sample parameter, not a recommended retention window or an atomic implementation.
The rule of thumb
Make the receiver safe to run twice, then let it retry. The key does the safety; backoff does the manners.
A saved key is useful only if the effect and the replay rules actually enforce it. Review concurrent requests, crashes and expired keys before relying on the pattern. Backoff and dead-letter handling manage attempts; they do not repair an unsafe receiver.
