At one point in my generated StateBench example, g holds 99 and c holds 57. Three versions of the next update all leave g at 57:

g <- c; g <- (g + 58) mod 100; g <- 57;

The answer agrees. The work needed to get it does not. One line reads another variable, one reads its own previous value, and one supplies the answer directly.

That is a useful way into Ai2's new StateBench release. Rather than treating every long-context question as the same reading task, it separates different kinds of state updates. [1]

Three generated updates leave g at 57: copy c, add 58 modulo 100, or assign 57 directly. The immutable family instead retains the original value 99.

An actual update from the generated trace family. This is a program-semantics illustration, not a language-model score. Image by the author.

Start with a shared trace

I ran the released generator and inspected four matched samples. The repository calls their families periodic, solvable, aperiodic and r-trivial. The announcement describes the first two as general-finite and soluble problems respectively. [1][2]

For the specific preview I used seed 13, eight ordinary variables, and eight extra assignments, without filler. The generated program also initializes a temporary register. That produces 17 assignments in each version: nine initializations plus the eight extra steps.

To reproduce that preview, use Python 3.12 or later and uv. Check out the source revision first:

git clone --depth 1 https://github.com/allenai/state-bench.git cd state-bench git checkout 5b333b8d426638d8825f40efe8a508c20eb85730 uv sync --no-dev

Then generate the four JSON previews:

for family in periodic solvable aperiodic r-trivial; do uv run --no-dev state-bench sample \ configs/experiments/solvable.yaml \ "integer-code-modular--$family" \ --seed 13 --num-variables 8 --extra-assignments 8 \ --filler-proportion 0 --json > "$family.json" done

I used the repository's supported variable count. Asking this configuration for five variables failed validation; its configured counts are 2, 4, 8 and 16. A CLI flag selects a permitted condition here rather than silently rewriting the experiment configuration.

Follow what the right-hand side reads

In the aperiodic version, the update states the new value explicitly. To recover the final g, a reader can find the last assignment to g.

In the solvable version, the update needs the current g. For this step, (99 + 58) mod 100 is 57. Skipping an earlier update can change the input to a later one.

In the periodic version, g <- c needs the current value of another register. Earlier changes to c can now matter to g. These are copies at assignment time, not persistent aliases: changing c later does not retroactively change g. [2]

The r-trivial construction replaces later writes with the first value assigned to their target. At this position it writes 99, so that family keeps a different answer. It does not share the changing-state answer just because its assignments appear in the same places.

For this preview, the final values of a through h, followed by TMP, were:

periodic: [16, 75, 99, 1, 48, 28, 57, 23, 28] solvable: [16, 75, 99, 1, 48, 28, 57, 23, 28] aperiodic: [16, 75, 99, 1, 48, 28, 57, 23, 28] r-trivial: [28, 75, 57, 1, 48, 34, 99, 23, 81]

I independently replayed every canonical assignment with a small interpreter and checked those outputs. The first three families also matched after every assignment in this sample, rather than merely arriving at the same final vector. That is a check of the generator's transformations, not evidence about an LLM's ability to follow them.

Retrieval is only part of the question

When I read a memory benchmark result, I want to know whether it tests finding an explicit value or composing updates. A larger input window does not make those operations interchangeable.

He Fangsheng's DEV discussion of derived state and retrieval argues for treating the current state as a separate object. [3] I would keep the claim narrower than “retrieval cannot help”: retrieval can provide relevant updates, but the system still needs to put those updates in order and apply their semantics.

The matched traces help isolate that distinction. A system can be handed similar-looking programs, the same update positions, and even the same expected output, while the dependency structure changes. Looking only at final answers would hide how each question was constructed.

Ai2 also evaluates pretrained and frontier models in the release. [1] I did not rerun those evaluations, train a model, or measure accuracy here. The concrete result is smaller: the published generator makes the difference between lookup, self-update and cross-variable dependency visible in an executable example.

References and resources

  1. Ai2, StateBench: Testing how language models track state changes over long documents.
  2. Ai2 contributors, StateBench source at the checked revision. The repository documents family derivation and eager-copy semantics.
  3. He Fangsheng, Vector RAG can't fix long-context state tracking. Its model measurements were not reproduced for this article.

Asset credits

Original assignment diagram based on the generated synthetic traces. StateBench source is Apache-2.0; the figure contains no third-party screenshots or benchmark charts.

AI disclosure

AI was used to draft the article.