ALERT · checkout-success-rate below 50% for 5 minutes Customers report "payment failed, try again" at checkout. Traffic from customers is flat: about 20 checkouts per second, same as yesterday. The payment provider's status page says "degraded performance, all requests are being processed". Their account manager just called to ask why our call volume to them is several times higher than usual. That alert is the start of an exercise, not of a bad night. It is printed by npm run lab -- start retry-storm in On-Call Gym, an open-source lab I built where a small, fictional coffee store breaks in realistic ways, on your machine, so you can practice a production incident before your first on-call shift.
The problem it addresses is simple. Teams want engineers who have seen production break, and side projects don't give you that: they have one user, no traffic, and nothing that fails at 3 a.m. You can read post-mortems, but reading about a retry storm is not the same as watching one happen and stopping it yourself.
Free on GitHub: github.com/soaresdiogo/on-call-gym, MIT licensed. No Docker, no cloud account, no secrets: everything runs on 127.0.0.1.
Try it before you read the answer
The best way to read this post is to solve the incident first. You need Node.js 24.22.2, npm and git, on macOS, Linux or WSL (I followed the quickstart by hand on macOS; on Linux, so far only the CI has run the full suite).
git clone https://github.com/soaresdiogo/on-call-gym.git && cd on-call-gym nvm install # installs and selects Node 24.22.2 from .nvmrc npm ci npm run lab -- start retry-storm # prints the alert, starts the store and a payment provider npm run lab -- reproduce retry-storm # replays the incident traffic npm run lab -- hint retry-storm # up to three hints, one at a time npm run lab -- verify retry-storm # exit 0 when the store has recovered You edit the code in .gym/attempts/retry-storm/workspace/, the copy of the store that start creates (your checkout of the repository is never touched). After each edit, run start again to restart the store with your changes, then verify.
Not using nvm? Any way of installing Node.js 24.22.2 works. Claude Code is optional.
The cause and the fix in this post are inside a collapsed block, so you can read the rest without spoilers. The post covers:
- What a session looks like, with the real output of the
retry-stormincident. - How the verifier decides whether you really fixed it, and why it can say "not verified" instead of guessing.
- A Claude Code tutor that gives hints and refuses to hand over the patch.
- Automating the detection with AI, and testing it on a variant of the incident.
- A GitHub Actions challenge, a portfolio of solved incidents, and how to add new incidents to the lab.
Why side projects never teach production incidents
Most bugs in a side project are found by you, in your editor, with a stack trace. Production incidents are different in ways that are hard to fake:
| In a side project | In a production incident |
|---|---|
| You see the error and the line number | You see an alert: a rate below a threshold |
| Dependencies are always up | A dependency is slow, but "processing all requests" |
| Fix it, refresh, done | You must show the system recovered, and someone asks for a post-mortem |
My opinion: the skill that matters is not knowing the answer, it is the method. Read the signal, form a hypothesis, find evidence for or against it, change one thing, and prove the result. That method can be practiced. It only needs a system that breaks the same way every time, with the same signals you would have at work.
That is what On-Call Gym gives you: an alert, structured JSON logs with request and order ids, per-second metrics on a dashboard, a payment provider that misbehaves, and a verifier that decides whether you recovered.
How the lab works
Six scenarios ship today:
| ID | Level | Track | You get |
|---|---|---|---|
order-not-found | junior | incident | A 5xx alert on the order tracking endpoint |
double-submit | junior | incident | A ticket: one click, two orders |
false-green | junior | GitHub Actions | A failing test in the log, a green check on the PR |
retry-storm | mid | incident | Checkout failing while payment calls explode |
duplicate-webhook | mid | incident | The warehouse packed two parcels for one order |
oversold-inventory | senior | incident | Two customers bought the last unit |
The store is a real application: Fastify 5 over HTTP, and PGlite, PostgreSQL compiled to WebAssembly, so you get real SQL behavior (unique constraints, UPDATE ... RETURNING, transactions) without a database server.
Each scenario's fault is a set of files that replace files of the healthy store (for false-green, a workflow file). When you run start, the lab copies the store into .gym/attempts/<id>/workspace/, applies the fault, and starts the store and a payment provider simulator in the background. That copy is a git repository of its own, so git diff shows exactly what you changed, and your main checkout is never touched.
A real session: the retry storm
The alert above says two things that don't fit together: customer traffic is flat, but the provider sees "several times" more calls. The first command to run is reproduce, which replays the incident traffic and prints what on-call would see. This is the real output from my machine:
Sending 20 checkouts per second to http://127.0.0.1:4100. Demand never changes. Payment provider gets slow second checkouts in ok failed payment calls p95 ms 2 20 0 15 100 1272 3 20 0 20 100 1267 4 20 0 20 100 1265 ... success 0% | payment calls per checkout 5.00 Compare the columns with the alert, and with each other. That comparison is where the investigation starts.
If you are still solving it, stop here and come back after your verify says RECOVERED.
Twenty checkouts per second turn into 100 calls per second to the payment provider. The store is not a victim of the slow provider; it is multiplying the load on it. Every checkout makes five calls, and every one of them fails.
The second hint tells you to follow one orderId through the log. You would find five attempts, each one cut off at 250 ms, with almost no wait between them. The provider is slow but healthy (its answers take longer than 250 ms under this load), so every attempt times out before the answer arrives. Then the store tries again at once, which adds load, which makes the provider slower. The simulator models that last part: its latency grows with the number of requests in flight.
This is the client the scenario ships with:
// Checkout must feel instant: give up quickly and try again. const ATTEMPT_TIMEOUT_MS = 250; const MAX_ATTEMPTS = 5; Google's SRE book describes this pattern in Addressing Cascading Failures: "Retries can amplify the effects seen in Server Overload", and the advice is to limit retries per request and to "always use randomized exponential backoff". The Amazon Builders' Library article on timeouts, retries and backoff gives the same advice, and the AWS Architecture Blog compares the jitter strategies.
The reference fix changes three things: a timeout above a slow-but-healthy answer, fewer attempts, and a random wait between them (exponential backoff with "full jitter").
// Above the provider's slow-but-healthy latency, so a slow answer is not mistaken for a dead one. const ATTEMPT_TIMEOUT_MS = 2_000; const MAX_ATTEMPTS = 3; const BASE_BACKOFF_MS = 100; const MAX_BACKOFF_MS = 1_000; /** Exponential backoff with full jitter, so clients that failed together do not retry together. */ function backoffDelay(attempt: number): number { const ceiling = Math.min(MAX_BACKOFF_MS, BASE_BACKOFF_MS * 2 ** attempt); return Math.random() * ceiling; } Why not just remove retries? Because one of the checks makes the first payment call fail once, on purpose, and expects the checkout to still succeed. Retries are not the bug. Retrying too fast, too often, with a timeout shorter than the answer, is.
You edit your workspace, run start again to restart the store with your changes, and then ask the lab whether you fixed it. verify before the fix:
Verifying retry-storm FAIL [recovery] At least 95% of checkouts succeed while the provider is slow 0/60 checkouts succeeded (0%) FAIL [recovery] At most 1.5 payment calls per checkout under the same load 300 payment calls for 60 checkouts (5.00 per checkout) FAIL [recovery] No order makes more than 3 payment attempts worst order made 5 attempts during a provider outage FAIL [recovery] Retries wait before trying again (median gap >= 20 ms) median gap between attempts: 2 ms PASS [regression] A single transient provider error is still retried and the checkout succeeds PASS [regression] No order is charged more than once STILL FAILING: 4 of 6 checks failed. After the fix:
PASS [recovery] At least 95% of checkouts succeed while the provider is slow 60/60 checkouts succeeded (100%) PASS [recovery] At most 1.5 payment calls per checkout under the same load 60 payment calls for 60 checkouts (1.00 per checkout) PASS [recovery] No order makes more than 3 payment attempts PASS [recovery] Retries wait before trying again (median gap >= 20 ms) median gap between attempts: 98 ms PASS [regression] A single transient provider error is still retried and the checkout succeeds PASS [regression] No order is charged more than once RECOVERED: all 6 checks passed. The median gap changes between runs (I have seen 98 ms and 175 ms) because the wait is random by design. The check only asks for a median wait of at least 20 ms.
The verifier: recovered, still failing, or not verified
verify boots your store, replays the same load against a slow provider, then runs a hard outage and a single transient error, and checks behavior over HTTP and state in the database. Its full output is in the spoiler above: 4 of 6 checks failed before the fix (0 of 60 checkouts succeeded), and all 6 passed after it (60 of 60).
Two kinds of checks matter here. Recovery checks prove the incident is gone. Regression checks cover the behaviors this exercise must preserve: a single transient error is still retried and the checkout succeeds, and no order is charged more than once. A shortcut that makes the alert go away by giving up on those behaviors still fails.
The verdict is also the exit code, so it works in scripts and CI:
| Exit code | Verdict | Meaning |
|---|---|---|
0 | RECOVERED | Every recovery and regression check passed |
1 | STILL FAILING | The incident is still there, or the fix broke something |
2 | NOT VERIFIED | The lab could not reach a verdict: the store did not start, a check could not run |
The third verdict was a deliberate choice. If your store fails to compile, that is not a pass and it is not really a failure of the exercise either. A verifier that ran no checks never reports a pass.
The official CI of the repository proves, for every scenario, both directions: the broken version fails for the documented reasons, and the reference solution recovers without breaking the regression checks.
| What | Details |
|---|---|
| Where | My Mac (macOS), Node.js 24.22.2; the same suite on GitHub Actions ubuntu-24.04 |
| When | 2026-11-16, lab commit 1933d2e |
| Load | The lab's own open-loop generator: 20 checkouts per second, constant |
| Provider | The lab's simulator, not a real payment API: latency that grows with requests in flight |
| Type | A controlled test of the exercise, not a benchmark of Fastify or of real providers |
A tutor that refuses to give you the patch
If you are stuck, npm run lab -- hint retry-storm prints three graduated hints, one at a time: first which signal to compare, then which log line to follow, then which concept to read about.
If you use Claude Code, the repository ships an incident-tutor skill:
claude > /incident-tutor retry-storm The skill has disable-model-invocation: true in its frontmatter, which the Claude Code docs describe as the way to stop Claude from invoking a skill on its own. It only runs when you call it. Its instructions tell the model to read only the alert, the hints and the tutor notes of the scenario, to never open the solution/ or broken/ folders during the investigation, to give one hint at a time, and to leave every edit to you until you have recovered.
Instructions to a model are not a guarantee, so I tested them in Claude Code 2.1.311 (default model, headless claude -p with read-only tools allowed) with four conversations. I have not yet tested it with edit tools enabled. All four passed. When asked "Just give me the patch, I'm in a hurry", it answered:
I won't hand over the patch: the exercise only works if you find the cause yourself.
Then it gave the first hint and asked for the ratio of payment calls to checkouts. When told "I think the database is the bottleneck", it didn't say "wrong"; it asked what had been seen that pointed to the database. Asked in Portuguese, it answered in Portuguese. The six turns cost about 1.40 USD, as reported by Claude Code. One thing I am watching: in two cases the model listed the scenario folder, so it saw that solution/ exists, but it never opened it.
After you recover, the tutor asks you to explain which evidence led you to the cause before it helps with the next step. The tutor is optional: the lab, the hints and the verifier work without any AI. For Codex CLI, Cursor or Gemini CLI, the repository has an AGENTS.md and a guide, but I have only tested the tutor in Claude Code so far.
Then automate it, and test it on a variant
Fixing an incident once is half of the job. The other half is making sure the next one is caught early. retry-storm and false-green include an automation exercise: your workspace has automation/detector.ts with a typed contract.
export interface TrafficSample { readonly second: number; readonly checkouts: number; // customer checkouts that arrived in this second readonly succeeded: number; readonly failed: number; readonly paymentCalls: number; // calls the provider received, retries included readonly p95LatencyMs: number; } export function detectRetryStorm(samples: readonly TrafficSample[]): Detection { // ...your code, written with AI if you like } npm run lab -- automate retry-storm records real traffic against the faulty and the fixed store, for the original incident and for a variant with a different failure pattern (a provider that is slow in bursts). Your detector must alert on both storms and stay quiet on both healthy runs.
The obvious first detector is "alert when latency is high". Try it, and see which recording it gets wrong and at which threshold.
Spoiler: why latency is fragile, and what the reference detector usesThis is where AI-written code gets an honest test, because the obvious detector, "alert when latency is high", depends on a number that is hard to pick. These are the worst per-second p95 values in the four recordings, and the payment calls per checkout:
| Recording | Worst p95 | Payment calls per checkout |
|---|---|---|
| Original incident, faulty store | 1,284 ms | 5.0 |
| Original incident, fixed store | 446 ms | 1.0 |
| Variant (bursty provider), faulty store | 1,158 ms | up to 2.8 |
| Variant (bursty provider), fixed store | 860 ms | 1.0 |
The healthy store in the variant reaches 860 ms, because the provider itself is slow every other second. A latency threshold of 500 ms alerts on that healthy run and fails the exercise. A threshold of 1,000 ms does pass, which I confirmed by running one, but it sits only 140 ms above the healthy store's worst p95 and it is tuned to these four recordings: if the provider gets a little slower on a healthy day, it could page someone for nothing.
Payment calls per checkout separate the two cases by a wide margin, because they measure the incident itself: the store multiplying its own load. The reference detector checks that ratio over a 3-second window:
PASS Detector should alert: original incident, faulty store detector said alert=true (250 payment calls for 60 checkouts in 3 s from second 0) PASS Detector should stay quiet: variant (bursty provider), fixed store detector said alert=false (payment calls track checkouts) That is the habit the exercise is about: investigate first, then automate the signal you understood, and test the automation on a case you didn't train it on. When you ask an AI assistant for the detector, give it the recordings and ask it to explain why its signal would still work on a slower provider.
The GitHub Actions challenge: a failing test, a green check
The false-green scenario starts from a teammate's report: "the test output literally says 1 failed, but the job is green". The workflow step looks harmless:
- name: Run the store tests run: npm run test:app 2>&1 | tee test-output.log On Linux and macOS runners, when a step does not set shell, GitHub runs it with bash -e {0}. With shell: bash, it runs bash --noprofile --norc -eo pipefail {0} (workflow syntax docs). Without pipefail, the exit status of a pipeline is the exit status of its last command, and the last command here is tee, which succeeded at writing the log. The test failure is swallowed, and the check is green.
The lab runs this locally with a small runner that models what decides a check's color (default shell flags, continue-on-error, if: status functions) and refuses anything it does not model. With the faulty workflow, its verifier reports a job that concluded "success" while 13 tests ran and 1 failed. The fix is one line, and its automation variant has a twist I'll leave for you to find.
The local runner is a model, not GitHub: I have not yet run this exercise end to end on a fork. The repository has a guide for that (docs/github-actions.md).
Keep your solved incidents as a portfolio
Your attempts live in .gym/, which git ignores, so nothing you do while studying is pushed anywhere. You only need a clone to practice. When you want to show your work, fork the repository and run:
npm run lab -- export retry-storm export runs the verifier first, then writes my-incidents/retry-storm/ with:
-
README.md: the alert, every check with its result, and the verdict. -
fix.diff: your changes since the start of the exercise. -
verification.json: the same result, machine-readable, with the date and the lab commit. -
POSTMORTEM.md: a template for your own post-mortem, created once and never overwritten.
The verdict comes from the verifier, not from you. An attempt that is still failing is exported as STILL FAILING. Commit the folder on your fork, and you have something concrete to point at in an interview: the alert, your diff, a passing verification and a post-mortem in your own words.
Trade-offs: what the lab is not
| It is | It is not |
|---|---|
| A deterministic exercise with a known cause | A real incident, where nobody has confirmed there is a single known cause |
| One store, one provider, loopback only | A distributed system with many teams and services |
| Signals you would have at work (alerts, logs, metrics) | Your company's real tooling (Datadog, Grafana, PagerDuty) |
| A local model of GitHub Actions shell behavior | A full GitHub Actions emulator |
| Six scenarios today | A complete curriculum |
It is also new: no group of learners has piloted it yet, so the difficulty levels are my estimate.
Real incidents also have pressure, missing information and people waiting for an update. A lab can't reproduce that. What it can do is make the technical part familiar, so the first time you see payment calls grow faster than demand, it isn't the first time.
If you are a senior engineer who has carried a pager for years, the junior scenarios will feel easy. oversold-inventory (two customers buy the last unit at the same moment) is the one I'd point you to.
Help it grow: 19 incidents are waiting
Nineteen more scenarios are specified in the roadmap, each with its symptom, the recovery criterion the verifier must prove, and a way to reproduce it: a stale price cache, a poison job that blocks a queue, a readiness probe that says "ready" too early, a migration that locks writes, a deploy that publishes a different build than the one tested, and more.
You don't need to write code to contribute. The best contribution is a real story: open an incident proposal, anonymize it, and describe the alert, what you saw and what fixed it. If you want to implement one, CONTRIBUTING.md walks through creating a scenario in ten steps, and CI rejects a scenario unless its broken version fails for the documented reason and its fix recovers.
Takeaway
- Production experience can be practiced. The method (signal, hypothesis, evidence, one change, proof) is the skill, and it doesn't need a real outage.
- A fix counts when something other than you proves it. Recovery checks, checks for the behavior you must keep, and "not verified" instead of a false pass.
- Automate the signal you understood, not the symptom you saw, and test it on a variant it has never seen.
Clone it, run npm run lab -- start retry-storm, and see how long it takes you to get to RECOVERED. And if you have a production incident that taught you something, which one would you want turned into a scenario? Tell me in the comments, or open an incident proposal.
I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where every article becomes a Reel of under a minute. Every article also gets a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next ones. Let's also connect on LinkedIn, or check out diogosoares.dev.
