Building Endpoints That Survive Being Retried

Ask a beginner what the hard case in a payment endpoint is & they'll say "what if the charge fails?" That one's easy — it failed, you show an error, everyone understands. The genuinely hard case is the opposite: the charge succeeded & then the response got lost on the way back. The client waited, saw a timeout & did the reasonable thing — it tried again. Now the customer has been charged twice for one click & both requests were, individually, completely correct.

This is the problem underneath a huge amount of backend reliability work & most tutorials skip it because the happy path never shows it to you. Networks lose responses. Clients retry. If your write isn't built for that, "it worked" and "it ran twice" become indistinguishable — and the second one costs real money and a real apology.

Why retries are not optional

The instinct is to say "just don't retry." That doesn't survive contact with reality. A request can fail after the work is done in a dozen ways — the connection drops mid-response, a load balancer times out, the client's network blips, a mobile app loses signal at the worst moment. From the client's side, all of these look identical to "the server never got my request." It has no way to tell "it didn't happen" apart from "it happened but I didn't hear back."

So every serious client retries & it should. The network guarantees you at best "at least once" — a message that arrives might arrive more than once. That's not a bug to eliminate; it's the environment you're building in. The job isn't to stop retries. It's to make a repeated request land harmlessly.

Idempotency keys: letting the client say "this is the same request"

The clean fix is to let the client tag each intended operation with a unique id — an idempotency key — and reuse that same key on every retry of that operation. The server remembers keys it has already processed & if it sees one twice, it returns the original result instead of doing the work again.

async function createCharge(req) {
    const key = req.headers["idempotency-key"];
 
    const existing = await store.get(key);
    if (existing) return existing; // already did this exact operation
 
    const result = await chargeCard(req.body);
    await store.set(key, result, { ttl: 60 * 60 * 24 });
    return result;
}

The key is per-operation, generated by the client — a UUID made when the user clicks "Pay," reused if that click has to be retried & thrown away for the next purchase. First request: no stored key, do the charge, remember the result. Retry with the same key: found it, return the same result, charge nothing new. A different purchase brings a different key and goes through normally. This is exactly the pattern payment providers like Stripe expose to their callers, for exactly this reason.

The race that quietly reopens the hole

The naive version above has a gap that only shows up under load — and duplicates love load, because a slow response is what triggered the retry in the first place. Picture two requests with the same key arriving nearly together. Both run store.get(key), both find nothing, both proceed to charge. The check-then-act isn't atomic, so the very thing you were preventing slips through the crack between the check and the write.

Closing it means making "claim this key" an atomic step, before doing the work. A conditional insert — succeed only if the key doesn't already exist — turns two racing requests into one winner and one that's told "already in progress":

async function createCharge(req) {
    const key = req.headers["idempotency-key"];
 
    // Atomic claim: only the first request to insert this key wins.
    const claimed = await store.insertIfAbsent(key, { status: "pending" });
    if (!claimed) {
        const existing = await store.get(key);
        // Still pending? A duplicate is mid-flight — tell the client to wait.
        if (existing.status === "pending")
            throw new ConflictError("in progress");
        return existing.result;
    }
 
    const result = await chargeCard(req.body);
    await store.set(key, { status: "done", result });
    return result;
}

The atomic insert (a unique constraint in a database, SET NX in Redis — the same primitive either way) is what actually makes this safe. Without it, you have an idempotency check that mostly works, which under retry pressure is the same as one that doesn't.

The better move: make the operation naturally repeatable

Idempotency keys are the tool for operations that are inherently one-shot — charging a card, sending an email. But a lot of writes can be designed so that repeating them simply doesn't matter & then you need no key-tracking machinery at all.

The clearest example is the difference between "create" and "set." POST /orders that appends a new row is dangerous to repeat — twice means two orders. But an operation phrased as "make this resource have this state" is safe to repeat by nature, because running it twice lands you in the same place as running it once:

-- Run this once or five times, the row ends up identical either way.
INSERT INTO user_settings (user_id, theme)
VALUES (42, 'dark')
ON CONFLICT (user_id) DO UPDATE SET theme = 'dark';

This is why PUT (set to this value) is considered idempotent and POST (create another) is not. Where you have the freedom to shape the operation — an upsert instead of an insert, "set status to shipped" instead of "add a shipment" — reach for the naturally-repeatable version first. The safest retry is the one that was never going to cause damage in the first place.

Not everything should be retried, either

One more piece, on the client side: retrying blindly is its own bug. A 500 or a timeout is worth retrying — the server had a bad moment, trying again might work. A 400 is not — the request is malformed & sending the exact same broken request again just wastes everyone's time and, if it's not idempotent, risks doubling up. Retry the failures that might be transient; give up on the ones that won't fix themselves.

And when you do retry, space the attempts out with backoff and a little randomness. If a service hiccups and a thousand clients all retry at the same instant, they arrive as one synchronized wave right when it's least able to cope — you've turned a blip into an outage. Increasing delays plus a bit of jitter spreads that wave into something survivable.

The mindset that prevents the whole class

The shift that makes all of this second nature is to stop assuming your write runs exactly once. It doesn't. In a networked system, any write worth caring about will eventually run twice — a retry, a duplicate delivery, an impatient user double-clicking "Submit." So for every write endpoint, ask the one question up front: what happens if this runs twice?

If the honest answer is "double charge," "duplicate order" or "two emails," you have work to do — an idempotency key, an atomic claim, or a redesign into something naturally repeatable. If the answer is "nothing, it's an upsert," you're already safe. Asking that question before you ship, instead of after the first duplicate-charge ticket, is most of what separates an endpoint that survives production from one that merely passed its tests.