I hash one file out of the Claude Code npm tarball on every release. It's the only part of the prompt surface you can still read, and it's been getting less stable all year.
The reason I started: a post about leaving Claude Code for Pi makes a checkable claim. It says there were "244 releases this year; over 3/4 of these changed the system prompt or tool descriptions," and uses that churn as the argument that the harness is an unstable thing to build a workflow on. I wanted to see if that holds up, because "the tool changed under me" is the kind of claim that's usually vibes.
Half of that claim is no longer testable
I can't check the system prompt half. Since 2.1.113 (2026-04-17) the readable code left the package. The published unpacked size dropped from 49,328,681 bytes at 2.1.112 to 132,292 bytes one release later, so there is no system prompt in the tarball to diff. Whatever ships now is a binary.
What still ships is package/sdk-tools.d.ts. Its JSDoc blocks are the tool descriptions sent to the model, which is the other half of the claim and the half that matters for how the agent behaves. So that's the file I hash.
Method, if you want to reproduce it:
# full packument, no Accept header (the abbreviated one returns a stale `time` map) pkg = json.loads(urlopen("registry.npmjs.org/@anthropic-ai/claude-code").read()) # per version: download dist.tarball, read package/sdk-tools.d.ts, sha256 it # a "change" is one version's hash differing from the previous version's A change is one release's hash differing from the release before it. Nothing fancier. One gotcha worth writing down: fetch the full packument. With Accept: application/vnd.npm.install-v1+json the time map comes back truncated and a date filter silently returns zero releases for a package that has hundreds. I lost an hour to that.
The churn rate is not flat, it climbs
Six windows, February through November. The two widest are 24 releases each, the rest are 12. I'm flagging the window size because a 12-release window has a coarser denominator, and I'd rather you see that than have me quietly mix them.
| window | releases | rate of change |
|---|---|---|
| late Feb (2026-02-07 to 02-28) | 24 | 21.7% |
| late Feb (02-20 to 02-28) | 12 | 9.1% |
| May (05-20 to 05-31) | 12 | 36.4% |
| Aug (08-21 to 08-31) | 12 | 45.5% |
| Sep 10 to Oct 6 | 24 | 47.8% |
| Oct 7 to Nov 6 | 24 | 43.5% |
Early in the year the tool schema mostly sat still. By autumn, roughly every other release rewrites at least one tool description. Same 24-release window size at both ends of the recent period, and the two independent windows agree with each other at 47.8% and 43.5%, which is the part that makes me believe the number rather than the trend line. Two windows landing within four points of each other is a result. A single window would have been a coincidence.
On the volume claim, the count holds up: 277 releases tagged in 2026 through 2026-11-06, and 239 through 2026-09-21, the post's date. Their "244" is in the right place. It's the "3/4 changed the prompt" part I can't confirm, because the artifact that would confirm it stopped shipping in April. That's the honest version of the result: half the claim reproduces, half is now unmeasurable, and the half that reproduces is worse than the post implied for the recent period.
A harness on the same registry barely moves
I ran the same idea against @google/gemini-cli for contrast. Its system prompt still ships readable inside the bundle, so I hashed a 12,000-character window starting at "You are Gemini CLI" across 0.64.0 (10-13), 0.65.0 (10-20), 0.66.0 (10-27) and 0.67.0 (11-04). Identical hash every time (sha256 prefix e011f0d58a), found in the two largest bundle chunks each release.
Four stable releases, zero prompt-surface changes. That's roughly a 10x difference in how much the harness rewrites its own instructions, measured the same way, on the same registry, in the same month. It's also a useful reminder that "all coding agents churn" is not a law. One of them doesn't.
Why this matters more than it sounds
If you run evals against a coding agent, the prompt surface is part of your test fixture. When a tool description changes, you've changed the input to the model, and instruction phrasing measurably moves model behaviour across model families. So a regression you measure in October might be a prompt rewrite, not a model update or your own code.
That failure mode has a tell. GitHub wrote up something adjacent in their own reviewer, where swapping in better exploration tools made review cost go up and produced fewer useful comments. Their tool instructions, written for a general coding assistant, made the agent browse the repo instead of investigating the diff. Same class of bug: the instructions are the behaviour, and they move on a schedule you don't control.
What I'd do about it
Three things, all cheap.
- Pin the harness version in the eval config, not just in your lockfile. If the agent is installed globally, the version is invisible.
- Record the harness version alongside every eval result. A number with no harness version attached is not a result you can compare next month.
- When a score moves, re-run the baseline on the pinned version before you blame the model. It takes ten minutes and it's the difference between a finding and a guess.
There's a fourth thing if you maintain the harness itself: the file is right there. Diffing sdk-tools.d.ts between two versions before you upgrade is a one-command check that tells you whether the agent's instructions changed underneath your repo conventions. If they did, the change is worth reading before it lands in your CI.
The narrow version
The thing you're evaluating is a moving target, and for at least one popular harness it moves about every other release. The broader version is that "we ran the eval last quarter" is not a statement about anything unless you know what the tool's instructions said when you ran it. Reproducibility in this space is not only about seeds and temperatures. It's also about whether the binary you tested is the binary you're shipping.
If you want the file to hash, it's package/sdk-tools.d.ts in the tarball on the npm registry. One file, one hash, a few minutes of work per release. It's the least glamorous benchmark in this space and the one I'd check before trusting any other.
