Idempotency in Distributed Systems: Why Retries Don't Break Things

Every distributed system leaks failures somewhere: a request times out, a network partition hides a response, or a client gives up and hits “pay” again. Getting idempotency in distributed systems right is what separates a platform that shrugs off these hiccups from one that double-charges customers and ships orders twice. This guide walks through the design decisions — keys, storage, delivery semantics, and testing — that turn safe retry logic into something boring instead of catastrophic.

What Idempotency Really Means (and Why Retries Happen)

An operation is idempotent when performing it once produces the same result as performing it N times. Reading a row is naturally idempotent. Setting status = 'shipped' is idempotent. Appending to a ledger or incrementing a balance is not — and that gap is where money leaks.

Retries are not an edge case; they are the default behavior of every layer in a modern stack. Consider a single “create payment” call:

• The client SDK retries on connection reset.
• The load balancer retries on a 502.
• The message queue redelivers if the consumer doesn’t ACK in time.
• A human refreshes the checkout page because it “looks stuck.”

None of these actors know whether the previous attempt succeeded. A common failure mode is the timed-out success: the server processed the request in 3.2 seconds, but the client’s 3-second timeout already fired and triggered a retry. The work happened; the acknowledgment didn’t arrive. Achieving idempotency in distributed systems means the server can recognize “I have already done this exact thing” and return the original result instead of doing it again.

At AelfTech, the mental model we teach is simple: you cannot prevent retries, so you must make them safe. The network is allowed to lie about what completed. Your job is to make the truth recoverable.

The Duplicate-Processing Problem: Real Payment and Order Examples

Let’s make this concrete. A payments endpoint receives a charge request, calls the card processor, and writes a row:

# NOT idempotent — a retry charges the card twice
def create_charge(user_id, amount_cents):
    result = processor.charge(user_id, amount_cents)  # side effect!
    db.execute(
        "INSERT INTO charges (user_id, amount, provider_ref) VALUES (%s, %s, %s)",
        (user_id, amount_cents, result.reference),
    )
    return result

If the client retries this after a timeout, the card gets hit twice. For a $49.99 subscription, that is a chargeback, a support ticket, and a trust problem. Multiply by a Black-Friday retry storm and you have an incident.

Order creation has the same shape with a different blast radius. A retried “place order” can:

Symptom	Root cause	Customer impact
Two identical shipments	Duplicate order rows	Refund + return logistics cost
Double inventory decrement	Non-atomic stock update	Phantom stockouts
Two confirmation emails	Duplicate side-effect fan-out	Erodes trust, spam complaints
Doubled loyalty points	Non-idempotent increment	Revenue leakage

The whole discipline of idempotency in distributed systems comes down to one move: give each logical operation a stable identity that survives retries, and make the write conditional on that identity not already existing. That identity is the idempotency key.

Designing a Robust Idempotency Key

An idempotency key is a client-supplied token that uniquely identifies one logical operation. The server uses it to detect and collapse duplicates. Get its properties wrong and idempotency silently breaks.

A robust key is:

• Client-generated, not server-generated. The client must send the same key on the retry, so it has to create the key before the first attempt. A UUIDv4 generated once per checkout intent works well.
• Scoped to the operation, not the session. One key per “charge for cart #123,” not one key per user. Reusing a key across genuinely different operations makes the second one a silent no-op.
• Tied to the request payload. Store a hash of the request body alongside the key. If the same key arrives with a different body, that’s a client bug — reject it with 422 rather than returning a stale result.

// Client generates the key once, reuses it across retries
const idempotencyKey = crypto.randomUUID();

async function charge(body: ChargeRequest): Promise<ChargeResult> {
  return retryWithBackoff(() =>
    fetch("/v1/charges", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey, // identical on every retry
      },
      body: JSON.stringify(body),
    })
  );
}

A subtle but important rule: never derive the key from something that changes per attempt — a timestamp, a retry counter, or Date.now(). That defeats the entire mechanism. When teams ask us at teamaelftech.com why their “idempotent” endpoint still doubles up, a per-attempt key is the culprit roughly half the time.

For operations where the client genuinely can’t hold a key (e.g., webhook receivers), derive a deterministic key from stable fields in the payload, such as sha256(event_id + event_type).

Storing, Matching, and Expiring Idempotency Records

The server needs a durable record of which keys it has seen and what result each produced. The canonical approach is an idempotency_keys table with a unique constraint doing the heavy lifting.

CREATE TABLE idempotency_keys (
  key           TEXT PRIMARY KEY,
  request_hash  TEXT        NOT NULL,
  status        TEXT        NOT NULL DEFAULT 'in_progress', -- in_progress | done
  response_code INT,
  response_body JSONB,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at    TIMESTAMPTZ NOT NULL
);

The request lifecycle uses the database as the coordination point, so two concurrent retries can’t both proceed:

-- Claim the key atomically. If it already exists, we get a conflict.
INSERT INTO idempotency_keys (key, request_hash, expires_at)
VALUES ($1, $2, now() + interval '24 hours')
ON CONFLICT (key) DO NOTHING;

If the INSERT created a row, you are the first attempt — do the work, then flip status to done and store the response in the same transaction as the business write. If the INSERT did nothing, a record already exists, and you branch on its state:

• status = 'done' → return the stored response_code and response_body. This is the timed-out-success case, handled perfectly.
• status = 'in_progress' → a concurrent attempt is mid-flight. Return 409 Conflict and let the client back off. Do not start a second execution.
• request_hash mismatch → same key, different payload. Return 422.

Writing the business row and the idempotency record in one transaction is non-negotiable. If they can diverge, you can mark a key done while the charge rolled back, and the retry returns success for work that never happened. This atomicity is the load-bearing detail of idempotency in distributed systems — everything else is bookkeeping around it.

Expiry keeps the table bounded. Sweep expired rows with a scheduled job:

DELETE FROM idempotency_keys WHERE expires_at < now();

Pick the TTL deliberately: it must be longer than your longest client retry horizon but short enough to keep the table lean. For most APIs, 24 hours is the sweet spot; for slow batch systems, 72 hours.

Exactly-Once vs At-Least-Once Delivery

Here is the claim that saves the most confusion: exactly-once delivery does not exist over an unreliable network. What you actually build is at-least-once delivery plus idempotent processing, and the combination produces exactly-once effects. That distinction is the whole game.

Guarantee	What it promises	Cost	Reality
At-most-once	Never processed twice	Can lose messages	Unsafe for money
At-least-once	Never lost	May process duplicates	The practical default
Exactly-once delivery	One delivery, ever	Impossible end-to-end	A marketing term
Exactly-once effect	Duplicates are harmless	Requires idempotency	What you should target

So-called “exactly-once” features in systems like Kafka are really exactly-once processing within a bounded scope — the transactional producer plus idempotent consumer offsets — and they stop at the boundary of your own side effects. The moment you call an external payment processor or send an email, you are back to at-least-once, and only your idempotency layer keeps effects singular.

The practical takeaway: design every consumer to tolerate seeing the same message twice. Don’t chase a delivery guarantee the physics won’t give you; make duplicates a no-op instead.

Idempotency Across Distributed Transactions and Queues

Once an operation spans multiple services, a single database transaction can’t cover it. Idempotency in distributed systems that use distributed transactions is coordinated with a saga — a sequence of local transactions, each with a compensating action — and every step must be individually idempotent, because any step can be retried independently.

Consider an order saga: reserve inventory → charge payment → create shipment. If “charge payment” times out and the orchestrator retries, the payment service must recognize the idempotency key and not double-charge. If “create shipment” fails permanently, the compensations (refund payment, release inventory) must also be idempotent — a retried refund should refund once.

For queue consumers, combine deduplication with idempotent handlers:

def handle_message(msg):
    key = msg.headers["idempotency-key"]
    # Atomic claim: unique constraint rejects the duplicate.
    if not claim_key(key):
        ack(msg)          # already processed — ACK and move on
        return
    try:
        with db.transaction():
            process_business_logic(msg)   # the real work
            mark_key_done(key)            # same transaction
        ack(msg)
    except Exception:
        release_key(key)  # allow a legitimate retry
        nack(msg)

Two patterns make this robust. The inbox pattern records processed message IDs so redeliveries are dropped. The outbox pattern writes outgoing events to a table in the same transaction as the business change, and a relay publishes them — guaranteeing that if the business write commits, the event will eventually publish, exactly the at-least-once-plus-idempotency recipe again.

One caveat we stress in the TeamAelfTech knowledge hub: ACK only after the work and the key update commit. ACK-before-process is the classic way to silently drop messages under crash conditions.

Testing Your Retry Logic Before It Bites You

Idempotency bugs hide until production traffic and concurrency find them. Test your safe retry logic on purpose.

• Duplicate-request test. Fire the identical request (same key) twice sequentially. Assert exactly one side effect and identical responses.
• Concurrent-request test. Fire N identical requests in parallel. Only one should execute; the rest get the stored result or a 409. This catches missing unique constraints and race conditions that a sequential test misses.

def test_concurrent_charges_produce_one_effect():
    key = str(uuid4())
    body = {"user_id": 42, "amount_cents": 4999}

    with ThreadPoolExecutor(max_workers=10) as pool:
        results = list(pool.map(
            lambda _: post("/v1/charges", key, body), range(10)
        ))

    # Exactly one real charge in the ledger
    assert count_charges(user_id=42) == 1
    # Every caller sees the same provider reference
    refs = {r.json()["provider_ref"] for r in results if r.status == 200}
    assert len(refs) == 1

• Payload-mismatch test. Reuse a key with a changed body; expect 422.
• Fault injection. Kill the process between the business write and the key update, then replay. The retry must not double-apply. Tools like toxiproxy for latency and partition simulation are worth wiring into integration suites.
• Expiry test. Reuse a key after its TTL and confirm the operation runs fresh rather than returning a garbage-collected result.

Testing idempotency in distributed systems is where most teams discover their unique constraint was advisory, not enforced. Better to learn it in CI than in an incident channel.

The AelfTech Checklist for Safe Retries

Run every mutating endpoint against this list before it ships:

1. Client generates a stable idempotency key once per logical operation and reuses it on every retry.
2. The key is stored with a request-body hash so a reused key with a new payload is rejected, not silently collapsed.
3. A unique constraint enforces single execution — the database, not application code, is the source of truth for “already processed.”
4. The business write and the idempotency record commit in one transaction. They can never diverge.
5. Concurrent duplicates return 409 or the stored result, never a second execution.
6. Consumers assume at-least-once and treat duplicates as no-ops; ACK only after the commit.
7. Multi-service flows use sagas with idempotent steps and idempotent compensations.
8. Records expire on a deliberate TTL, swept by a scheduled job.
9. Concurrency and fault-injection tests exist and run in CI, not just happy-path duplicate checks.

We keep an expanded, code-heavy version of this checklist and related reliability patterns at aelftech.com for teams hardening their APIs.

Conclusion

Retries are not a bug to be eliminated — they are the honest behavior of an unreliable network, and they will happen whether you plan for them or not. Robust idempotency in distributed systems reframes the problem: instead of preventing duplicates, you make duplicates harmless. A client-generated key, a unique constraint, a single atomic transaction, and consumers that tolerate redelivery together turn “exactly-once” from an impossible delivery promise into an achievable effect. Get those four things right and retries stop being incidents and start being invisible.

For more engineering deep-dives on reliability, queues, and API design, browse the Engineering hub.