What you are looking at

The danger is not failure. It is a success whose receipt got lost.

If a webhook plainly errors, nothing happened and retrying is safe. The trap is the middle case: the server received the event, charged the card, and then the acknowledgement was lost on the way back, a dropped connection, a timeout, a restart. The sender never heard "done", so it does the only safe thing it knows and delivers again.

Now the same event runs a second time. Without a way to recognise that it has already been handled, the receiver charges again. At-least-once delivery guarantees this will happen eventually, so the fix cannot be "make delivery reliable". It has to be "make the receiver safe to run twice".

The parts, one by one

One technique makes it correct; three make it well-behaved.

Each control in the demo is one of these. Idempotency is the one that decides whether you double-charge. The other three decide how the retrying behaves while it gets there.

Idempotency key

The one that fixes the charge

A stable id for the operation, taken from the event, not made up per attempt. Recorded when the work completes. On a repeat the receiver sees the key, returns the first result, and does not act again. This is the difference between one charge and three.

Exponential backoff

Space the retries out

Retry after 1s, then 2s, 4s, 8s, doubling each time, instead of a tight fixed interval. It gives a struggling receiver room to recover and stops a retry storm from becoming a self-inflicted outage.

Jitter

Do not let everyone retry in lockstep

Add a random offset to each backoff. Without it, a thousand senders that all failed at the same moment retry at the same moments forever, a thundering herd. Jitter smears them across the interval so the load is smooth.

Dead-letter after a cap

Know when to stop

Retries should not run forever. After a handful of attempts, park the event in a dead-letter queue where a human can see it and replay it, rather than hammering an endpoint that is clearly not coming back.

the tempting versionJS
// The receiver everyone writes first. Do not ship it.
app.post('/webhooks', async (req, res) => {
  const event = req.body;                            // e.g. charge.succeeded
  await chargeCard(event.customer, event.amount);    // the side effect
  res.sendStatus(200);
});

// If the ack is lost after chargeCard runs, the sender retries,
// this handler runs again, and the customer is charged twice.

Nothing here is wrong on a good day. The bug only appears when the acknowledgement is lost after the charge has already happened, which at-least-once delivery guarantees will happen sooner or later.

Why retrying alone is not enough

You cannot fix this by making delivery more reliable.

Every layer between the two systems can drop the acknowledgement without dropping the work. The receiver has to assume the event will arrive again and make that harmless.

Where the duplicate comes from

The at-least-once tax

Delivery guarantees are almost always at-least-once, never exactly-once, because exactly-once across a network is effectively impossible.

The ack can be lost after the work is done: a timeout, a dropped connection, a receiver restart mid-response.

The sender cannot tell "did the work, lost the ack" apart from "never did the work", so it retries both.

Only the receiver knows whether it already acted, which is why the fix lives there, in an idempotency key, not in the transport.

the version to shipJS
app.post('/webhooks', async (req, res) => {
  const key = req.header('Idempotency-Key') ?? req.body.id; // stable per event

  const seen = await store.get(key);
  if (seen) return res.status(200).send(seen.result); // replay, do not act again

  const result = await chargeCard(req.body.customer, req.body.amount);
  await store.set(key, { result });                   // record on completion
  res.status(200).send(result);
});

The key is taken from the event, so every retry carries the same one. The receiver records it only after the charge succeeds, and a repeat returns the stored result instead of charging again. One event, one charge, no matter how many times it is delivered.

the sender sideJS
async function deliver(url, body, attempt = 1) {
  const res = await post(url, body).catch(() => null);
  if (res && res.ok) return res;
  if (attempt >= 5) return deadLetter(body);      // give up cleanly

  const base = 2 ** (attempt - 1) * 1000;         // 1s, 2s, 4s, 8s
  const wait = base / 2 + Math.random() * base;   // full jitter
  await sleep(wait);
  return deliver(url, body, attempt + 1);
}

The sender doubles the gap each attempt and adds a random offset so retries do not line up into a spike. After five tries it stops and dead-letters the event rather than hammering forever. This is the "manners" half; the idempotency key on the receiver is the "safety" half.

Setting it up in an internal tool

Pick a stable key, give the store a TTL, and watch the dead-letter queue.

The technique is only as good as the key you choose and the queue you watch. These are the three decisions that make it hold up in production rather than in a demo.

Key

Derive it from the event, once

Use the event id the sender already assigns, or a hash of the meaningful fields. Never generate the key on the receiver per request, that gives every retry a different key and defeats the whole thing.

Store

Give the idempotency store a TTL

Keys do not need to live forever, only long enough to cover the retry window, a day is generous. A TTL keeps the store from growing without bound while still catching every realistic duplicate.

Watch

A dead-letter queue you actually monitor

Exhausted retries have to land somewhere queryable, with an alert on depth. A dead-letter queue nobody looks at is just a slower way to lose events. This is the link back to monitoring your integrations.

the plumbingJS
// Idempotency store: key -> result, with a TTL so it cannot grow forever.
await store.set(key, { result }, { ttlHours: 24 });

// Dead-letter: when retries are exhausted, park the event for a human.
async function deadLetter(event) {
  await dlq.add(event);                 // queryable queue, alert on depth
  metrics.increment('webhook.dead_letter');
}

Two small pieces make the pattern operable: a TTL so the idempotency store stays bounded, and a dead-letter queue with a metric so a run of failures is visible instead of silent.

The rule of thumb

Make the receiver safe to run twice, then let it retry. The key does the safety; backoff does the manners.

You cannot stop duplicates from arriving, so stop them from mattering. An idempotency key recorded on completion turns a duplicate into a no-op, and exponential backoff with jitter and a dead-letter cap keep the retrying itself from becoming the next incident.