A cache stampede is one of those bugs nobody sees in a load test and everybody sees at the first real traffic peak. A popular entry expires, a hundred requests notice in the same second, and all hundred recompute it. With a database query that is annoying. With a language model behind the cache it becomes slow and expensive, and in the worst case Amazon Bedrock throttles you exactly when traffic is highest.

In my article on the semantic cache for Bedrock I deliberately left this topic out and wrote that I would only add a short lock under high load. This is the follow-up: I will show you how to prevent a cache stampede with Lambda, ElastiCache for Valkey and a few lines of TypeScript. At the end you have a runnable project with AWS CDK, tests that need no AWS account and a simple load test to measure the effect yourself.

In short

When a frequently read cache entry expires, exactly one request should recompute it, not all of them at once. You get there with a short lock via SET NX in Valkey, an old value that the other requests receive in the meantime, and spread-out expiry times. The biggest pitfall is releasing the lock: it may only be deleted while it still belongs to you.

What you will learn in this article

  • What a cache stampede is and why it hurts so much in front of Bedrock
  • Why in-memory tricks do not help in Lambda
  • Lock, stale value, XFetch and jitter in one small TypeScript function
  • A safe lock release with a token and a Lua script or DELIFEQ
  • How to verify with a load test that the value is really computed only once

What is a cache stampede?

The term describes a load spike that the cache itself triggers. While an entry is valid, the cache protects the system behind it. The moment it expires, that protection disappears for everyone at once. Every request arriving now sees a cache miss and starts the expensive computation. The pattern is also known as “dogpile”, and it is closely related to the “thundering herd”, where many waiting processes start running simultaneously in response to a single event.

My example for this article is the fictional bookshop “Seitenwind”. Its home page shows a “pick of the day” per genre, written by a language model on Amazon Bedrock. The recommendation stays in the cache for ten minutes. That sounds harmless until a newsletter goes out on Saturday morning and 300 people click “crime” at the same time. If the entry expires right then, in the worst case 300 identical model calls land at Bedrock.

Why this hurts so much in front of a language model

Three things come together. First, a model call takes seconds instead of milliseconds, so the window in which other requests see the empty cache is long. Second, every call costs tokens, and you pay for 300 identical answers 300 times. Third, Bedrock has quotas for requests and tokens per minute. A stampede can use them up, and then even requests that have nothing to do with the pick of the day start to fail.

Why in-memory tricks do not help in Lambda

In a classic Node.js server you would simply coalesce identical requests inside the process: the first one starts the computation, all others attach to the same promise. That does not work in Lambda. In the standard model, an execution environment handles only one request at a time. If 300 requests arrive in parallel, Lambda starts many environments, each with its own memory. Coordination therefore has to happen in a place everyone can see. In our case, that place is the cache itself.

Strategies against a cache stampede

A few building blocks help against a cache stampede, and they combine well. The table shows what I typically use in serverless projects and what for.

Building block What it does When I use it
Lock (SET NX) Only the request that gets the lock recomputes Whenever a computation is expensive
Stale value Other requests get the previous answer in the meantime When a slightly outdated answer is acceptable
Early refresh (XFetch) Single requests recompute shortly before expiry For very frequently read keys
Jitter on the TTL Expiry times are spread randomly Whenever many keys are written at the same time
External refresh A schedule recomputes values in advance For a few well-known keys

A lock with SET NX

At the core is a single command: SET lock:pick:v1:crime <token> NX PX 20000. NX means the key is only written if it does not exist yet. PX sets the lifetime in milliseconds. If Valkey answers OK, you hold the lock. If it answers with nothing, someone else already has it. The lifetime is your safety net: if the Lambda function crashes in the middle of the computation, the lock disappears on its own after 20 seconds.

The value of the lock is a random token. That sounds like a detail, yet it is the most important part. If your computation takes longer than the lifetime, the lock expires and a second request grabs a new one. If the first request then blindly deletes it with DEL, it removes the lock of the second one. That is why the key is only deleted when the stored value still matches your own token. Valkey 9.0 added the DELIFEQ command for this; on older versions you use a small Lua script.

Serving the stale value

A lock alone prevents duplicate computation, but the other requests then have to wait. It is better to keep the entry longer than it is fresh. I therefore store two times: freshUntil inside the entry and a longer TTL for the key in Valkey. After ten minutes the pick counts as stale, but it stays stored for another hour. Anyone who does not get the lock during that time immediately serves the old pick. Waiting only happens when there is really nothing in the cache, for example right after a deployment.

Early refresh with XFetch

In 2015, Vattani, Chierichetti and Lowenstein described a method known as XFetch. The idea: every request rolls the dice on whether to recompute the entry before it expires. Long before expiry the chance is tiny; shortly before, it rises sharply. The formula is surprisingly short: recompute if now − delta · beta · ln(random) ≥ expiry. delta is the duration of the last computation, beta a factor that moves the refresh earlier or later. According to the authors, beta = 1 works well in practice.

Its charm is that slow computations are automatically refreshed earlier. A model call taking three seconds gets a bigger head start than a query taking 50 milliseconds. Combined with the lock, XFetch makes sure the entry is usually refreshed before it ever becomes stale.

Jitter on the expiry time

The fourth building block is the simplest. If you write five genres into the cache at the same time after a deployment, a fixed TTL makes them expire at the same time too. A few percent of randomness on the lifetime spreads the expirations. You might know the principle from exponential backoff with jitter, where it is used for retries for the same reason.

🎬 Cache stampede with a lock: only one request asks the model – this animated diagram only runs on my site: watch it on sascha-ms.de

Prevent a cache stampede: step by step with Lambda and Valkey

Now it gets practical. We build an HTTP API with a Lambda function that reads the pick of the day from Valkey and rewrites it via Bedrock when needed. This time I describe the infrastructure with AWS CDK in TypeScript.

Preventing a cache stampede: HTTP API, Lambda, ElastiCache Valkey with SET NX and exactly one call to Amazon Bedrock

Architecture of the example: the Lambda function asks the cache first. Only the lock holder calls Bedrock.

Prerequisites

  • Node.js 22 or newer and an AWS account with CDK bootstrapped in your region
  • Docker, because the native Valkey GLIDE library is installed inside the Lambda build image
  • An ElastiCache Serverless cache for Valkey 9 with a public endpoint and IAM authentication. The semantic cache article shows step by step how to create it.
  • Model access in Bedrock, in this example Amazon Nova Lite via the EU inference profile

Step 1: Create the project

Create an empty folder and install the dependencies. The versions in the example repository are the ones I tested with.

mkdir bookshop-picks && cd bookshop-picks npm init -y npm pkg set type=module npm install @aws-sdk/client-bedrock-runtime @valkey/valkey-glide aws-cdk-lib constructs npm install -D typescript@5.9 tsx vitest aws-cdk esbuild @types/node@24 @types/aws-lambda mkdir -p src lib bin test 

Step 2: The logic against the cache stampede

The heart of it is the getOrCompute function. It does not know Valkey at all, only a small interface with four methods, so you can test it without a network. Read the code once from top to bottom: fresh hit, try the lock, compute and store on success, otherwise serve the stale value or wait briefly.

// src/stampede.ts import { randomUUID } from "node:crypto"; /** Minimal cache interface: Valkey in production, an in-memory fake in tests. */ export interface CacheStore { get(key: string): Promise<string | null>; /** SET key value PX ttlMs */ set(key: string, value: string, ttlMs: number): Promise<void>; /** SET key value NX PX ttlMs – true if this caller wrote the key */ setIfAbsent(key: string, value: string, ttlMs: number): Promise<boolean>; /** Deletes the key only if it still holds the given value (lock release). */ deleteIfEquals(key: string, value: string): Promise<boolean>; } /** What we store per key: the value plus the data XFetch needs. */ export interface CacheEntry { value: string; /** How long the last computation took (ms) – "delta" in XFetch */ computeMs: number; /** From this point on the entry counts as stale (ms since epoch) */ freshUntil: number; } export type Source = "hit" | "computed" | "stale" | "waited"; export interface Result { value: string; source: Source; } export interface StampedeOptions { /** How long an entry stays fresh */ freshMs: number; /** How long a stale entry may still be served while someone refreshes it */ staleMs: number; /** Lock lifetime: must be longer than the slowest computation */ lockMs: number; /** How long callers without the lock wait for the result at most */ maxWaitMs: number; /** Polling interval while waiting */ pollMs: number; /** XFetch factor: 1 is a good start, > 1 refreshes earlier */ beta: number; /** Random variation of freshMs (0.1 = ±10 %) */ jitter: number; now?: () => number; random?: () => number; sleep?: (ms: number) => Promise<void>; } export const defaultOptions: StampedeOptions = { freshMs: 10 * 60_000, staleMs: 60 * 60_000, lockMs: 20_000, maxWaitMs: 3_000, pollMs: 150, beta: 1, jitter: 0.1, }; /** Thrown when the value is not ready within maxWaitMs. */ export class CacheBusyError extends Error { constructor(public readonly key: string) { super(`Value for ${key} is being computed, please retry shortly`); this.name = "CacheBusyError"; } } /** * XFetch (Vattani et al. 2015): the closer the expiry, the more likely * a single caller refreshes early. random() lies in (0, 1]. */ export function shouldRefreshEarly( entry: CacheEntry, now: number, beta: number, random: number, ): boolean { const safeRandom = Math.min(Math.max(random, Number.MIN_VALUE), 1); return now - entry.computeMs * beta * Math.log(safeRandom) >= entry.freshUntil; } /** Spreads expiry times so that keys written together do not expire together. */ export function withJitter(ms: number, jitter: number, random: number): number { return Math.round(ms * (1 + jitter * (random * 2 - 1))); } export function parseEntry(raw: string | null): CacheEntry | undefined { if (!raw) return undefined; try { const entry = JSON.parse(raw) as CacheEntry; return typeof entry.value === "string" && typeof entry.freshUntil === "number" ? entry : undefined; } catch { return undefined; // broken entries count as a miss } } /** * Reads a value from the cache. Only the caller holding the lock computes it; * everyone else gets the stale value or waits briefly for the new one. */ export async function getOrCompute( store: CacheStore, key: string, compute: () => Promise<string>, options: Partial<StampedeOptions> = {}, ): Promise<Result> { const o = { ...defaultOptions, ...options }; const now = o.now ?? Date.now; const random = o.random ?? Math.random; const sleep = o.sleep ?? ((ms: number) => new Promise<void>((r) => setTimeout(r, ms))); const entry = parseEntry(await store.get(key)); if (entry && !shouldRefreshEarly(entry, now(), o.beta, random())) { return { value: entry.value, source: "hit" }; } const lockKey = `lock:${key}`; const token = randomUUID(); const gotLock = await store.setIfAbsent(lockKey, token, o.lockMs); if (gotLock) { try { const started = now(); const value = await compute(); const computeMs = Math.max(1, now() - started); const freshMs = withJitter(o.freshMs, o.jitter, random()); const fresh: CacheEntry = { value, computeMs, freshUntil: now() + freshMs }; await store.set(key, JSON.stringify(fresh), freshMs + o.staleMs); return { value, source: "computed" }; } finally { await store.deleteIfEquals(lockKey, token); } } // Someone else is computing. A stale value is better than waiting. if (entry) return { value: entry.value, source: entry.freshUntil > now() ? "hit" : "stale" }; const deadline = now() + o.maxWaitMs; while (now() < deadline) { await sleep(o.pollMs); const ready = parseEntry(await store.get(key)); if (ready) return { value: ready.value, source: "waited" }; } throw new CacheBusyError(key); } 

Two details are easy to miss. The finally block releases the lock even when Bedrock throws an error; without it, everyone else would wait until the lock expires. In addition, broken entries count as a miss instead of an error. A format change after a deployment therefore does not take the page down.

Step 3: The Valkey adapter with a safe release

This adapter translates the interface into commands for Valkey GLIDE, the official client of the Valkey project. Each execution environment creates the connection once and then reuses it. For the release I use a Lua script, because it runs on every Valkey version and I could also test it locally against Redis OSS.

// src/valkey-store.ts import { GlideClusterClient, Script, ServiceType, TimeUnit } from "@valkey/valkey-glide"; import type { CacheStore } from "./stampede.js"; // Deletes the lock only if it still holds our token. redis.call also works on // Valkey (and on Redis OSS for local tests). On Valkey 9 or later // the single command DELIFEQ does the same. const releaseScript = new Script(` if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("del", KEYS[1]) end return 0`); let pending: Promise<GlideClusterClient> | undefined; /** One client per Lambda execution environment; GLIDE renews the IAM token itself. */ export function getClient(): Promise<GlideClusterClient> { pending ??= GlideClusterClient.createClient({ addresses: [{ host: required("CACHE_HOST"), port: 6379 }], useTLS: true, requestTimeout: 500, credentials: { username: required("CACHE_USER"), iamConfig: { clusterName: required("CACHE_NAME").toLowerCase(), service: ServiceType.Elasticache, region: required("AWS_REGION"), }, }, }).catch((error: unknown) => { pending = undefined; // try again on the next call throw error; }); return pending; } export function valkeyStore(client: GlideClusterClient): CacheStore { return { async get(key) { const value = await client.get(key); return value === null ? null : value.toString(); }, async set(key, value, ttlMs) { await client.set(key, value, { expiry: { type: TimeUnit.Milliseconds, count: ttlMs } }); }, async setIfAbsent(key, value, ttlMs) { const result = await client.set(key, value, { conditionalSet: "onlyIfDoesNotExist", expiry: { type: TimeUnit.Milliseconds, count: ttlMs }, }); return result === "OK"; }, async deleteIfEquals(key, value) { const deleted = await client.invokeScript(releaseScript, { keys: [key], args: [value] }); return deleted === 1; }, }; } export function required(name: string): string { const value = process.env[name]; if (!value) throw new Error(`Environment variable ${name} is missing`); return value; } 

If your cache runs Valkey 9 or newer, you can also implement deleteIfEquals with client.customCommand(["DELIFEQ", key, value]). That saves the script but ties you to the new version.

Step 4: Bedrock call and handler

The model call is deliberately simple. What matters is that it is passed in as a function, so getOrCompute decides whether it happens at all.

// src/recommend.ts import { BedrockRuntimeClient, ConverseCommand } from "@aws-sdk/client-bedrock-runtime"; import { required } from "./valkey-store.js"; const bedrock = new BedrockRuntimeClient({}); const SYSTEM_PROMPT = `You write the "pick of the day" for the fictional bookshop "Seitenwind". Recommend one well-known, real book for the given genre in at most 80 words. Do not invent prices, stock levels or quotes.`; /** The expensive call we want to protect: takes seconds and costs tokens. */ export async function writeRecommendation(genre: string): Promise<string> { const response = await bedrock.send( new ConverseCommand({ modelId: required("MODEL_ID"), system: [{ text: SYSTEM_PROMPT }], messages: ...
Source:https://dev.to/sascha-ms/prevent-a-cache-stampede-lambda-valkey-and-bedrock-3p9a