If you ask an AI for a recommendation — which car, which tool, which plan — you are asking in exactly the category where models disagree most.
We sent the same 24 questions, unchanged, to five models — GPT, Claude, Gemini, Grok and Llama. Six were recommendations with tradeoffs. On factual questions the panel was near-unanimous (0.33 material divergences per question). On recommendations, they averaged 3.17 divergences per question — the highest of any category — and 5 of the 6 recommendation questions produced at least one material split.
Three recommendation examples from the study:
- Minivan vs 3-row SUV: most answers converged on sliding doors and step-in height. Only one model stated when its own recommendation flips — towing over about 3,500 lbs, real off-road or all-weather needs, and image. That condition is the part a buyer actually needs, and a single-model workflow would have missed it.
- Bakery commercial oven: every judged model said finance a commercial oven now, on capacity grounds. Only one raised the point that could void the whole comparison — some health inspectors and insurers will not accept domestic ovens for commercial production. It was also the only answer with implementation guidance: used or refurbished units, a lease-vs-loan comparison, and budgeting for hookups and ventilation.
- Remote team communication: three models picked async-first. One dissented and picked Slack-style chat as the main tool, arguing chat is already effectively asynchronous across time zones — a premise the other answers rejected. Same question, opposite operating model, presented as settled either way.
That is the pattern across the study: of 14 sole-source claims, hand-checked, 8 were the highest-value line in their set and 3 were wrong or invented. On a recommendation, the lone voice is sometimes the only answer that named the condition that changes the decision — and sometimes the one inventing a premise. You cannot tell which from a single answer, because a single model never flags that it is alone.
My practical rule now: one model is fine for a factual lookup; for a recommendation I will live with — a vehicle, a lease, a tool stack, a training plan — I look at where more than one model splits before I act. The disagreement is the information.
Full study, method and caveats (including that one judge was also a panel member, and a memory-bleed incident we found and disclosed): https://multio.ai/research/model-divergence
Disclosure: I build MultiO (multio.ai) — One subscription. Every top AI model… Auto sends each message to the most cost-efficient model that can handle it. Compare asks up to three models side by side, and every message shows its exact token charge on a receipt. Free trial: 3,000,000 tokens, no card required.
