Idempotency in Distributed Systems: Patterns and Implementation

An operation is idempotent if performing it multiple times produces the same result as performing it once. GET /users/123 is naturally idempotent — calling it ten times returns the same user. POST /payments is not — calling it ten times might charge the user ten times.

In distributed systems, network failures, timeouts, and retries make duplicate requests inevitable. Without idempotency, every retry is a gamble.

Why Retries Create Duplicates

The client doesn't know if the first request succeeded. It retries — and the server processes the payment twice. This isn't theoretical. Payment systems, order processors, and any state-mutating API will encounter this in production.

Pattern 1: Idempotency Keys

The client generates a unique key for each logical operation and sends it with the request. The server stores the key with the result. On retry, the server recognizes the key and returns the stored result without re-executing.

Implementation

The Database Schema

Key Lifecycle

Cleanup is essential. Without it, the idempotency table grows unbounded. Run a periodic job:

Set the retention period longer than any reasonable retry window. If clients might retry after 24 hours, keep keys for 48.

Pattern 2: Natural Idempotency Keys

Some operations have natural identifiers that serve as idempotency keys without the client generating one.

This is simpler than the idempotency key pattern but only works when the operation has a natural unique identifier. For generic API endpoints, you still need explicit idempotency keys.

Pattern 3: Optimistic Locking

Prevent concurrent modifications by including a version number in every update. The update only succeeds if the version hasn't changed since the last read.

This doesn't prevent duplicates on its own — it prevents conflicting updates. Combine with an idempotency key for full duplicate protection.

Stripe's Approach

Stripe's idempotency implementation is the gold standard for API design. Key design choices:

1. Client-generated keys. The Idempotency-Key header accepts any string up to 255 characters. UUIDs are recommended but not required.

2. Request parameter matching. If a retry uses the same key but different parameters, Stripe returns a 400 error. This prevents accidental key reuse across different logical operations.

3. Atomic processing with ACID transactions. The idempotency key insertion, business logic, and response storage happen in a single database transaction. If any step fails, everything rolls back — including the key claim.

4. 24-hour key retention. Keys expire after 24 hours. Retries after that window are treated as new requests. This bounds storage growth while covering any reasonable retry scenario.

5. In-flight request handling. If a request with the same key is currently being processed, Stripe returns 409 Conflict. The client should wait and retry.

Handling Concurrent Requests with the Same Key

The trickiest edge case: two requests with the same idempotency key arrive simultaneously. Without careful handling, both might start processing.

The UNIQUE constraint on the key column handles this atomically. The first request to insert wins; subsequent ones get a constraint violation and can be rejected or asked to retry.

Alternative: advisory locks. For more complex workflows where you want the second request to wait (not immediately fail):

Testing Idempotency

Idempotency is one of those properties that's easy to claim and hard to verify. Test it explicitly:

Idempotency isn't optional in distributed systems — it's a requirement that's either handled explicitly (well) or implicitly (badly, via duplicate records, double charges, and data corruption). Start with idempotency keys for your external-facing APIs, use natural keys where they exist, and test the concurrent case. The cost of implementing it is a database table and some middleware. The cost of not implementing it is customer trust.