A bigger model will not connect your company's facts for you. On 600 Wikidata questions, the best model that answers without thinking connected two facts it knew in 55 % of cases, and in about 30 % for little-known facts. Asked to write both steps first, it reached 98 %.
Ask a language model two questions:
Who wrote The Ugly Duckling?
In which city was Hans Christian Andersen born?
All eleven models I tested get both right: Hans Christian Andersen, Odense.
Now ask one question that needs both facts, and ask for the answer only:
In which city was the author of the book The Ugly Duckling born?
Five of the eleven say:
Copenhagen.
Copenhagen is where Andersen lived and wrote. It is the city his name brings up first. It is not what I asked. Claude Haiku 4.5 and Qwen3 235B are among the five, and each of them answers "Odense" when asked about Andersen by name.
It is not one unlucky question. Ask where the director of the 1960 film La Dolce Vita was born. Ten models know that Federico Fellini directed it and that he was born in Rimini. Six of those ten answer Rome, where the film is set.
When the same models write the two steps before the answer, all eleven get Odense and all eleven get Rimini.
Those two questions sum up a few weeks of experiments. Here is the full story, with numbers.
The question I was actually trying to answer
I work on company knowledge systems: chatbots and search over a company's own documents. The recurring question from clients is:
"Can't we just train a model on our data, so it knows everything?"
Earlier I tested this with a small open model (3B parameters) trained on a fictional company whose facts I generated from a hidden graph, so I knew every true connection. The result was clear and a little sad:
- The model remembered single facts fine.
- It could not connect two of them. "Who leads the division that owns product X?" failed, even when it knew both "X belongs to division D" and "D is led by Y".
- It learned to connect them only when I trained it to write the intermediate step ("X belongs to D. D is led by Y. So: Y.").
This is known in research as the two-hop curse. My conclusion was: store facts in a graph or search index, and train skills, not facts.
Then a fair objection came up, from me as much as anyone:
You trained a small model on a few thousand facts. Frontier models are trained on trillions of tokens of diverse text. Maybe at that scale composition just emerges?
I couldn't test that by training a frontier model. But frontier models have already been trained on Wikipedia. So I could test them on Wikipedia.
The setup: 600 tiny detective cases
I built 600 two-hop chains from Wikidata. Each one is A → B → C:
- Gummo (film) → directed by Harmony Korine → born in Bolinas
- Buddenbrooks (book) → written by Thomas Mann → born in Lübeck
- Vivendi (company) → founded by... Napoleon III? (Wikidata is wild sometimes. More on noise later.)
Six templates: film → director → birth city or birth year; book → author → birth city; company → founder → birth year; album → artist → birth city; painting → painter → birth year.
Two difficulty groups:
- Head: famous starting points (the film or book has articles in 30+ Wikipedias).
- Tail: obscure ones (2–6 Wikipedias). Nobody writes "the director of Hoodman Blind (1923) was born in Cape Elizabeth" on the internet.
For every chain, each model got four questions:
- Hop 1: "Who directed the 1997 film Gummo?"
- Hop 2: "In which city was Harmony Korine born?"
- Answer only: "In which city was the director of the 1997 film Gummo born? Reply with the city name only. Do not name any person."
- Step by step: same question, "think step by step, then give the answer."
The key metric: among chains where the model gets both single hops right, how often does it get the composed question right without writing the step? If it knows both facts but can't combine them silently, that's the curse.
The questions came from fixed templates, not from an LLM. (Lesson from earlier work: LLM-written "multi-hop" questions are often answerable in one hop, which flatters every model.) The chain set was frozen and hashed before any model saw it.
Round 1: does scale help?
Nine models on Amazon Bedrock, from tiny to frontier:
| model | answer only | step by step |
|---|---|---|
| Amazon Nova Micro | 17 % | 83 % |
| Qwen3 32B | 25 % | 93 % |
| Amazon Nova Pro | 29 % | 90 % |
| Qwen3 235B | 33 % | 94 % |
| Claude Haiku 4.5 | 38 % | 93 % |
| Claude Haiku 5.5 | 45 % | 93 % |
| Claude Opus 4.8 | 51 % | 99 % |
| Claude Sonnet 5.5 | 55 % | 98 % |
| Claude Opus 5.5* | 99 % | 98 % |
*Opus 5.5 refuses to turn its thinking off on Amazon Bedrock. I tried every setting. Its "answer only" includes hidden reasoning, so it is the control, not a contestant.
I had predicted the gap would not shrink with scale. I was wrong. Latent composition goes from 17 % to 55 %. Bigger models do connect facts in their heads more often.
But split by head and tail:
| model | famous (head) | obscure (tail) |
|---|---|---|
| Qwen3 235B | 38 % | 23 % |
| Claude Sonnet 5.5 | 68 % | 31 % |
| Claude Opus 4.8 | 63 % | 30 % |
The improvement sits almost entirely in famous facts. On obscure ones, the best non-thinking models stay around 30 %. And step by step closes the gap almost completely at every size: 96–98 % on the tail for Sonnet and Opus.
Side discovery: on obscure chains, even the best non-thinking models knew the first hop only about half the time. The bigger problem with rare facts isn't connecting them. It's knowing them at all.
Company facts are tail facts. Nobody on the internet writes about who owns your internal billing service.
A fun failure mode: models that can't keep a secret
I told every model: "Reply with the city only. Do not name any person." Here is Claude Haiku 5.5:
Curtis Hanson was born in Los Angeles... wait, I need to be careful...
Sonnet 5.5 named the intermediate person in 18 % of "answer only" replies (41 % in a harder set later). It literally cannot help writing the step. That counts as wrong for the latent test, but it's also a hint about how newer models are tuned: think out loud, always.
Round 2: "but maybe they cheated"
Then a nagging thought:
Maybe providers did exactly what you did with your fictional company: extract a graph from Wikipedia, generate composed facts like "the director of Gummo was born in Bolinas", and put them in the training data. Then the model isn't composing, it's remembering.
Totally plausible. Synthetic data is standard practice now, and there are published methods that turn entity graphs into training text.
Honestly, though: no black-box test can prove a provider didn't do something. If they synthesized every combination, the result looks the same as real reasoning. But two tests can still tell us a lot.
Test A: weird questions nobody writes
If models recall composites written in text, they should do well on natural combos ("the director's birth year") and worse on odd ones ("the director's father", "the school the author attended"). I built 480 new chains on famous films and books and kept everything fixed except what was asked about the person:
| model | natural pairs | odd pairs | three hops |
|---|---|---|---|
| OLMo 2 13B | 18 % | 16 % | 5 % |
| OLMo 2 32B | 27 % | 31 % | 0 % |
| Qwen3 235B | 40 % | 32 % | 13 % |
| Claude Sonnet 5.5 | 64 % | 58 % | 23 % |
| Claude Opus 4.8 | 61 % | 57 % | 28 % |
| Claude Opus 5.5 (thinks) | 99 % | 92 % | 97 % |
Odd pairs come out close to natural ones, so it isn't just memorized popular phrases. But add one more hop ("the birth year of the spouse of the author of X") and everything without thinking collapses to 0–28 %. Thinking models stay above 90 %.
Test B: a model whose training data I can read
That's why OLMo shows up in the table. The Allen Institute publishes OLMo 2's entire training data, and the infini-gram project indexes it. So you can literally ask: "Does the title Buddenbrooks ever appear within 100 tokens of 'Lübeck' in everything this model was trained on?" (Yes: 18 times.)
So for every chain I counted how often the starting point and the final answer appear near each other in OLMo's training data. Then I split results by "the composite is written somewhere" vs "it never is".
This is where I nearly published a wrong result. Twice.
Wrong result #1: the counter that lies about zero
My first table said: where the composite is never written, OLMo 13B still got 15 % right. Something was composing!
I looked at the "hits" one by one. They were almost all years and big cities: Hitchcock died in 1980, Woody Allen born 1935, End of Days → New York City.
It turns out infini-gram has a max_clause_freq limit (500,000). If one side of your AND query is more frequent than that (and "1980" or "New York City" certainly is), it subsamples occurrences, and the co-occurrence count can drop to zero. My "never written" bucket was full of composites that were written, just next to very common words.
The fix: count each answer on its own and keep only chains where both sides are under the limit, so the count is exact. That leaves 232 of 1,080 chains. Fewer, but honest.
With the fix, OLMo 13B got 2 of 17 where the composite is provably absent. Both were guessable. One of them:
- Arthur and the Minimoys → Luc Besson → Paris. A guess any film fan would make.
The other was just as guessable.
And its typical failure is to stop halfway: asked for the father of a book's author, it names the author. One hop, then stop.
I had a draft that said "the one model whose data we can check shows no sign of silent composition." Then I re-read it and saw the hole.
Wrong result #2: "but OLMo 13B is a small model"
Right. OLMo 13B scores about like the tiniest model in my test. "A small model doesn't compose" is not news. That's where this whole thing started.
Luckily, infini-gram also indexes the training data of OLMo 2 32B. It doesn't fit on my laptop, so I rented a GPU: one Amazon EC2 Spot Instance with an NVIDIA L40S, llama.cpp, the official 32.7 GB checkpoint. The box got its prompts and returned answers through Amazon S3 pre-signed URLs, with no cloud credentials on it, and switched itself off. 4,440 answers took 16 minutes. Downloading the model took an hour. Total: about $2.50.
Here's the table, exact counts only:
| model | composite never in the data | written 1–9× | written ≥ 10× |
|---|---|---|---|
| OLMo 2 13B | 12 % (2 of 17) | 7 % | 15 % |
| OLMo 2 32B | 24 % (6 of 25) | 10 % | 36 % |
| Qwen3 235B | 22 % | 21 % | 42 % |
| Claude Sonnet 5.5 | 35 % | 58 % | 80 % |
| Claude Opus 4.8 | 38 % | 62 % | 78 % |
| Claude Opus 5.5 (thinks) | 93 % | 97 % | 98 % |
OLMo 32B's six answers where the composite is provably not in its training data:
- Besson → Paris, Camus → University of Algiers, and one more: guessable.
- Stefan Zweig → University of Vienna: half guessable.
- 25th Hour → Spike Lee → Tonya Lewis Lee.
- The Kreutzer Sonata → Tolstoy → Kazan University. Nobody guesses Kazan.
So my 13B conclusion was too strong. In a model whose data we can actually check, silent composition appears, and it grows with size (2 of 17 → 6 of 25). Small numbers, so treat it as a direction, not a measurement. The closed models show the same shape at a higher level: 35–38 % when the composite isn't in the open corpus, about double when it is.
Bonus: the model that anonymized its own answers
When I asked OLMo 32B to think step by step and finish with a line like Answer: <the person's name>, it did this. 104 times:
The director of the 1951 film Strangers on a Train was Alfred Hitchcock. Alfred Hitchcock's father was William Hitchcock. Answer:
<PRESIDIO_ANONYMIZED_PERSON>
<PRESIDIO_ANONYMIZED_PERSON> is the placeholder Microsoft's Presidio tool puts where it removes names. The Allen Institute scrubbed personal data from the training corpus that way, and the model learned that an angle bracket where a name should go means: write the placeholder. My prompt template asked for exactly that.
The reasoning was right, but the final line was censored by the model's own training data. I fixed the parser (if the answer is a placeholder, take the sentence before it). Lesson: no angle-bracket placeholders in prompts for models trained on scrubbed data.
What I take away
- Silent two-hop composition is real, partial, and grows with scale. You can see it even in a model whose training data you can audit, and it got better from 13B to 32B.
- A good part of it is still recall. When the composite is written somewhere, accuracy jumps (about ×1.5 for OLMo 32B, ×2 for Sonnet and Opus). Famous entities and natural questions benefit most, because the internet already did the work.
- Depth is the wall. Three hops without thinking: 0–28 %, even for frontier models.
- Writing the step fixes almost everything, at every size. Reasoning models do it automatically; that's why Opus 5.5 sits at 92–100 % across the board.
- For company knowledge, the advice doesn't change. Your facts are tail facts, they change every week, and you need to know where an answer came from and who may see it. Even the best non-thinking models knew only about half of the obscure single facts. Put the facts in a graph or a search index. Use (or train) the model for the procedure: find the first fact, then look up the next one.
Training a model on your company wiki gives you recall at best: no sources, no updates, no access control. Silent composition gets better with scale, but it won't get you to a reliable answer. Writing the steps does.
Numbers for the skeptics
- Data: 600 + 480 chains from Wikidata, frozen and hashed before any model ran. Exactly one value per relation; names that give the answer away were filtered out.
- Models: 11. Nine on Amazon Bedrock (Amazon Nova Micro and Nova Pro, Qwen3 32B/235B, Claude Haiku 4.5/5.5, Sonnet 5.5, Opus 4.8/5.5), OLMo 2 13B on a MacBook, OLMo 2 32B on an Amazon EC2 Spot Instance.
- Scoring: alias matching, then Claude Sonnet 4.6 as judge for unclear cases (not one of the tested models, blind to which model answered). I checked about 70 judge decisions by hand and overruled 2 (it accepted Kingston for Saint Mary Parish, Jamaica). A borough counts for its city (The Bronx ↔ New York City), because the question asks for a city and Wikidata often stores the borough.
- Cost: about $37 total: $34 on Amazon Bedrock (Claude Opus 5.5 was a third of it) and $2.50 of Amazon EC2 GPU time, plus about 3 hours of OLMo 13B on a 16 GB MacBook.
- Caveats:
- one run, temperature 0 where allowed;
- question templates not reworded by another model;
- the open-corpus counts only match main names within 100 tokens, so "never written" is a lower bound;
- co-occurrence counts come from the 13B training index; the 32B uses the same pretraining corpus with a newer mid-training set;
- Sonnet 5.5's thinking is switched off via a special mode, and Opus 5.5's can't be switched off at all;
- Wikidata has noise: yes, it lists Napoleon III as a founder of Vivendi. Chains like that drop out automatically, because I only score chains where the model gets every hop right.
How it ran on Amazon Web Services
- Nine models, one call. The Converse API of Amazon Bedrock in the Europe (Frankfurt) Region, eu-central-1. The same request goes to every model. Only the model ID changes, plus one field that switches thinking off where the model allows it. The Claude and Amazon Nova models run through EU cross-Region inference profiles, so requests stay in AWS Regions in Europe.
- Cost per model comes with the answer. Every response reports its input and output tokens. Multiplied by the Amazon Bedrock on-demand prices of October 2026, the 600 main chains cost from $0.01 (Amazon Nova Micro) to $10.11 (Claude Opus 5.5) per model.
- The judge also runs on Amazon Bedrock (Claude Sonnet 4.6) and is kept out of the tested set.
- The open 32B model ran on one Amazon EC2 Spot Instance (g6e.xlarge, one NVIDIA L40S GPU with 48 GB) with llama.cpp in a container. The instance has no AWS Identity and Access Management (IAM) role and no access keys. It fetches its prompts and uploads its answers through presigned URLs for Amazon S3. A watchdog shuts it down, and shutdown terminates it, so a forgotten box cannot keep billing.
Practical notes if you try this yourself
- Respect robots.txt. Wikidata's SPARQL endpoint and API are disallowed for crawlers. The per-entity JSON (
Special:EntityData/Q42.json) is allowed. - QLever (a fast public Wikidata index) sometimes returned empty results for valid queries: the same query gave 6,974 rows, then 0. Use it only for simple patterns and verify facts elsewhere.
- infini-gram AND counts are approximate above
max_clause_freq. Count each side separately before you trust a zero. - OLMo 2 13B on a 16 GB Mac: a 16k context runs out of GPU memory.
-np 2 -c 3072 -fa on -ctk q8_0 -ctv q8_0works. - OLMo 2 32B: there's no official Q4_K_M; Q8_0 (32.7 GB) fits a 48 GB L40S with a 16k context. Budget an hour for the download.
And if anyone at a model provider wants to tell me what's in the training data, I'm happy to rerun the tail. 🙂