Idempotency in API Design: Prevent Duplicates

You usually discover you need idempotency the hard way. A customer insists they only clicked “Pay” once. Your logs show two requests. Your payment provider shows two charges. The load balancer retried after a timeout. The mobile app did what it was told.

Everyone is technically right. Your system is still wrong.

Idempotency in API design means this: if the same operation is requested multiple times, the outcome is the same as if it happened once. In HTTP terms, some methods are defined as idempotent, like PUT and DELETE. But in real production systems, retries happen at every layer: browsers, SDKs, reverse proxies, gateways, and background workers. If you do not explicitly design for safe retries, your system will eventually duplicate work.

What Experienced API Teams Emphasize

When you look at how high-scale systems approach this, the pattern is remarkably consistent.

Malcolm Featonby, Principal Engineer at Amazon, has written extensively about “making retries safe.” His core insight is simple but easy to miss: retries are normal behavior, not failure cases. Networks are unreliable. Clients will retry. Infrastructure will retry. You must design assuming at-least-once delivery.

He also points out something subtle and important. Two identical payloads are not always duplicates. Creating two identical EC2 instances is valid. Creating the same DynamoDB table twice is not. Idempotency is about intent, not byte equality.

From a distributed systems perspective, Pat Helland, long-time architect and author on distributed transactions, frames this as embracing replay. In systems that accept at-least-once messages, you survive by remembering what you already processed and making replay harmless.

Meanwhile, teams like Stripe take a brutally practical stance. For a given idempotency key, they store the first response, including status code and body, and return that exact result for all subsequent retries. Even if the first attempt returned an error. That single decision eliminates the most dangerous failure mode, charging a customer twice because a client retried after a timeout.

The common thread is clear. You need a stable operation identity plus a durable memory of what happened for that identity.

The Three Layers of Idempotency You Must Design For

You should think about idempotency across three layers.

First, protocol semantics. HTTP defines PUT and DELETE as idempotent. GET is safe and idempotent by design. That gives you a starting point, but it does not solve concurrency or duplicate side effects in downstream systems.

Second, application semantics. You must define what “same intent” means. “Create order” usually means once. “Send email” might mean once per event. “Provision server” might legitimately mean multiple times.

Third, storage semantics. You need database constraints or conditional writes to enforce uniqueness and state transitions. If your deduplication logic lives only in application memory, it will fail under concurrency.

Choose the Right Pattern for Each Endpoint

Not every endpoint needs the same mechanism. A quick mental model helps you avoid overengineering.

The key insight is this: HTTP semantics reduce risk, but storage-level guarantees prevent disasters.

Implement Idempotency-Key the Way It Actually Works in Production

For operations like payments, checkouts, refunds, or subscriptions, the Idempotency-Key pattern is the most battle-tested approach.

Step 1: Make the Client Generate the Key

The client generates a UUID per operation intent and sends it in an Idempotency-Key header.

One user action equals one key.

If the user clicks again because the UI froze, the client reuses the same key. That is how retries collapse into a single logical operation.

Step 2: Define What “Same Request” Means

If you only store the key, a buggy client could reuse the same key for a different payload and receive the wrong cached response.

Store a request hash alongside the key. Canonicalize the JSON body and hash it. On a repeat request:

• If the hash matches, return the stored result.
• If the hash differs, return a 409 Conflict or 422 error explaining that the key was reused with a different payload.

This enforces semantic equivalence, not just key reuse.

Step 3: Make the First Write Atomic

This is the critical step.

Insert a row like:

• idempotency_key
• request_hash
• status = IN_PROGRESS
• created_at
• ttl

Enforce a uniqueness constraint on idempotency_key.

If the insert succeeds, you are the first writer and can execute the operation.

If it fails due to uniqueness, another request already claimed this key. Fetch the existing record and return the appropriate response.

This single atomic insert is your safety barrier. Without it, concurrent requests will both pass “not seen” checks and double-execute.

Step 4: Store the Final Response

Do not just mark the operation as completed. Store:

• final status, COMPLETED or FAILED
• http_status
• response_body, or a pointer to it
• resource_id
• completed_at

On any retry with the same key and matching hash, return the same HTTP status and body as the original request.

If the first request failed due to validation, the retry should fail identically. If it succeeded but the client timed out, the retry should return success without re-running the side effect.

Step 5: Handle “In Progress” Retries

If a retry arrives while the original request is still processing, you have options.

You can return 202 Accepted and provide a status endpoint like /operations/{key}.

You can block briefly and then return the final cached result.

Or you can return a deterministic “still processing” response.

For long-running workflows, modeling operations as first-class resources is usually the cleanest approach.

Step 6: Set a Realistic Expiration Window

You do not want to store idempotency records forever. But you must store them long enough to cover real-world retry patterns.

Consider:

• Mobile clients on unstable networks.
• API gateways that retry automatically.
• Background queues with redelivery.
• Users are refreshing a page repeatedly.

Many payment systems use a 24-hour TTL as a starting point. Tune based on your retry horizon and business risk.

A Simple Worked Example