The UX cost of a security patch
Janus Arena (November 2026). Every figure comes from real runs recorded in the project's event
log: a brute-force fix on OWASP Juice Shop, and five Launch Day runs on ShopLab, a demo shop. The
reports are in the repository (docs/reports),
and the console's demo build replays two of the runs in your browser. The users are synthetic:
language-model agents that play a person in a real browser. Nothing here is a conversion figure.
A security fix is judged by one question: does the attack still work? It is a good question and it
is not enough. A fix can stop the attack and also stop the people it was meant to protect, and the
second half only shows up if you measure it. This is what happened when we measured both, in two
experiments, and what we could not conclude from them.
The rule that keeps it honest
Attacks are generated by code and judged by code. An attack succeeds only when a deterministic
oracle says so (for brute force: every wrong password was refused fast, nothing throttled, and the
right password still signs in afterwards). A model proposes the patch; it never decides whether the
patch works. The shoppers are agents, but whether one bought what it was asked to is read from the
shop's orders, not from what the agent says. Rates carry 95 % Wilson intervals.
Experiment 1: two ways to stop a password guesser
The finding was a brute-force on POST /rest/user/login. The Blue Team wrote alternatives and
three independent verifiers judged each: the original proof of concept must stop working (V1), the
regression tests must pass (V2), and a paired run of synthetic shoppers must stay within tolerance
(V3). Two alternatives got to the end:
| Alternative | Still exploitable? | Shoppers who bought, original → fix | p95 of the login |
|---|---|---|---|
| Per-account limit, 10 tries in 5 minutes | no, throttled at guess 11 | 4/6 → 5/6 | 50 → 90 ms |
| Account lockout | no, throttled at guess 6 | 4/6 → 3/6 | 46 → 41 ms |
Both close the attack. They differ in who else they stop. One of the personas, Inés, is forgetful:
before the right password she tries five old ones, as someone who does not remember which password
this shop has. Five is the usual threshold of a lockout, and on the lockout variant Inés was refused
with a 429 in both of her sessions and never completed a purchase. The limit of ten tries is above
the five she tries.
What this shows is a mechanism, not a rate. Six shoppers per variant is too few for the
difference between 4/6 and 3/6 to mean anything: it is one session, and the intervals overlap
completely. What is solid is the cause, because it is in the events: a specific persona, a specific
threshold, a 429. A third alternative (a progressive delay) is not in the table because the API
credit ran out before it was written; we report it as not measured rather than as passed. The whole
comparison cost 0.98 €.
Experiment 2: Launch Day
The cost of a defense changes with how many people arrive at once and where they come from, so we
put a launch on ShopLab: 18 synthetic shoppers and two saboteurs per variant. One saboteur sends
requests at a steady rate; the other guesses at an account's password. Variant A runs without the
defense, B with it, and every shopper plays the same task on both sides. Each defense was one run.
| Defense in B | Attacks that got through, A → B | Shoppers who bought, A → B | Shoppers blocked, A → B |
|---|---|---|---|
| Rate limit per client, 20/min | 100 % → 50 % | 86 % → 88 % | 0 % → 0 % |
| Rate limit per account, 3/min | 100 % → 50 % | 80 % → 79 % | 0 % → 0 % |
| Checkout step-up (type a code) | 100 % → 100 % | 75 % → 73 % | 0 % → 0 % |
| Rate limit per client, 6/min, nine shoppers at once behind three shared addresses | 100 % → 0 % | 75 % → 73 % | 0 % → 18 % |
Four things stand out.
Each defense stops what it was built for, and nothing else. The per-client limit stopped the
saturation saboteur after 20 answered requests and left the password guesser alone, and the
per-account limit did the reverse. The two 50 % are two defenses, each covering one of the two
saboteurs.
The step-up stopped neither attack. It is a defense for another problem (an account that was
just created buying with a stolen card), and the run says so: attacks 100 % → 100 %. What it costs
shoppers shows in the median number of steps, 15 → 16, not in anyone being blocked.
A defense that costs nothing can become one that costs shoppers. At 20 requests per minute no
shopper was blocked. At six, with nine of them arriving together behind three shared addresses
(an office, a campus), both saboteurs were stopped and 2 of 11 shoppers were blocked too, where none
had been without the limit. The interval is wide (5 % to 48 %). The same limit with only three shoppers at a time (an earlier
run that was interrupted, so it is not in the report) blocked nobody, and neither did 20 per minute
with shared addresses and low concurrency: the burst is what makes the limit bite.
The same number written differently, "this limit is safe", would have been true in one run and
false in the other.
Contamination was zero. Every shopper marks its shipping address with an invented code, and
after the task we check that no order it can read carries another session's code. None did, so the
numbers are not an artefact of sessions leaking into each other.
What we do not claim
- Not that a limit of 20 is the right one. Those are the values we tried, on a demo shop, with one task.
- Not a measure of attack success with any precision. There are two saboteurs per variant, so "100 % → 0 %" is 2/2 against 0/2, an interval from 0 % to 66 %. It says the defense stops them in these runs, not how often.
- Not conversion, and not what real people do. Synthetic shoppers say where a flow is hard for an agent that behaves like a person; they do not predict revenue.
- Not that the defenses combine. Each was measured alone.
What to take from it
Measure three things before you merge a security fix: whether the attack still works, how many of
the people you protect it stops, and what it adds for those it does not stop. The second number is
the one nobody has by default, and in these runs it was zero for one setting and 18 % for another
that differed in a single parameter. A gate that shows both on the same pull request, and never
merges on its own, is what Janus Arena is for.
To reproduce the Launch Day numbers: pnpm run:launch-day <runId>... writes the report from the
recorded events, with no model involved; the five runs cost between 0.24 € and 0.29 € each.