Why it matters in practice
Retries are not the exception. They are how reliable integrations work.
A system that retries on failure is more reliable than one that gives up, but only if the operation it retries is idempotent. Otherwise the mechanism designed to recover from failure becomes a source of it.
Why it runs twice
At-least-once is the normal guarantee
Networks time out after the work was done but before the acknowledgement arrived. Webhook providers re-send when they do not get a fast 200. Someone re-runs a batch that half-finished. None of these are bugs. They are how delivery is supposed to work. The message arriving twice is expected, not exceptional.
What breaks without it
The retry becomes a second failure
A non-idempotent write does its work again on every attempt: two invoices for one order, two contacts for one signup, a card charged twice. The damage is worse than the original failure because it is silent, the integration reports success both times, and the duplicate surfaces later as a data problem nobody can trace.
How to make it safe
Give the operation a stable identity
Derive an idempotency key from the event itself, an order id, an event id, not a fresh value per attempt. Record it when the operation completes. On a repeat, the system sees the key exists and returns the first result instead of acting again. For record writes, a natural unique key plus an upsert does the same job.
