Network hiccups happen constantly in distributed systems. A TCP connection drops halfway through a response, a microservice times out waiting for a database write, or a mobile client loses signal right as the user hits "Complete Purchase."
When a client experiences a timeout, the standard resilience pattern is to retry the request. However, if the server already processed the original request, a naive retry using HTTP POST will execute the action a second time. In payment systems, this means duplicate credit card charges, corrupted database state, and urgent customer support calls.
Idempotent APIs solve this problem by guaranteeing that performing the same operation multiple times yields the exact same result as performing it once.
How Idempotency Keys Work
The standard architectural pattern for idempotent POST requests—used by payment providers like Stripe and Adyen—relies on client-generated idempotency keys.
Here is the step-by-step lifecycle of an idempotent request:
- Key Generation: The client generates a unique identifier (typically a UUID v4) representing the single intent to perform a transaction.
- Header Transmission: The client attaches this key to the HTTP request using a custom header, such as
Idempotency-Key. - Server Lookup: Before executing business logic, the server checks a fast datastore like Redis for the presence of the key.
- Cache Hit: If the key exists, the server immediately returns the stored HTTP status code and response payload without re-executing the payment process.
- Cache Miss: If the key does not exist, the server processes the transaction, saves the final response in Redis tagged with the key, and returns the result.
Implementing Idempotency Middleware in Node.js
Placing idempotency logic inside an API middleware layer keeps your core business logic clean and decoupled from caching logic.
Here is a practical Express.js middleware using Redis to cache API responses:
const redis = require('./redisClient'); async function idempotencyMiddleware(req, res, next) { const idempotencyKey = req.headers['idempotency-key']; // Skip if no key is provided (or return 400 depending on API strictness) if (!idempotencyKey) { return next(); } const cacheKey = `idempotency:${idempotencyKey}`; try { const cachedResponse = await redis.get(cacheKey); if (cachedResponse) { const { statusCode, body } = JSON.parse(cachedResponse); return res.status(statusCode).json(body); } // Intercept res.json to capture and cache the outgoing response body const originalJson = res.json.bind(res); res.json = (body) => { if (res.statusCode >= 200 && res.statusCode < 300) { // Cache successful response for 24 hours (86400 seconds) redis.set(cacheKey, JSON.stringify({ statusCode: res.statusCode, body }), 'EX', 86400); } return originalJson(body); }; next(); } catch (error) { console.error('Idempotency middleware error:', error); next(error); } } module.exports = idempotencyMiddleware; Handling Concurrent Retries
A common edge case occurs when a client fires a retry while the initial request is still actively processing on the server. If the second request hits the middleware before the first finishes caching its response, both requests will see a cache miss and run concurrently.
To handle this race condition, you can acquire an in-flight status lock in Redis before processing:
// Attempt to set a temporary lock key const acquiredLock = await redis.set( `lock:${idempotencyKey}`, 'processing', 'NX', 'EX', 30 ); if (!acquiredLock) { // Another request with the same key is currently being processed return res.status(409).json({ error: 'Concurrent request in progress. Please wait and retry.' }); } Setting a short lock expiration (such as 30 seconds) ensures that if the server crashes mid-flight, subsequent retries will not be locked out permanently.
Key Architectural Considerations
- Expiration TTL: Retain idempotency records in Redis for at least 24 hours. Most network retries occur within seconds, but a day-long window accounts for background worker retries and offline client syncing.
- Tenant Scoping: Scope idempotency keys to individual authenticated users (e.g.,
idempotency:user_123:key_abc). This prevents cross-tenant key collisions or unauthorized access to cached responses. - Payload Validation: Optionally hash the incoming request body alongside the key. If a client reuses an idempotency key with a different request payload, return a
400 Bad Requestto report the mismatch.
Building resilient APIs requires anticipating network failure. Idempotency keys shift the safety guarantee to the server level, protecting your state and your users against accidental retries.
About me
Tanmay builds AI agents and automation for businesses and is open to freelance projects. Feel free to connect on LinkedIn.