GSM8K on Hugging Face: 4 of the Top 12 Rows Show a Sibling Checkpoint's Score, and 20 of 28 Rows Are Unmerged PRs

TL;DR

• On November 3, 2026 the Hugging Face GSM8K board lists 28 rows from 28 repos. 20 come from pull requests that are still open, 8 sit on the repo's main branch, and 0 are verified.
• Four of the top 12 rows show a number that the model card reports for a different checkpoint of the same family. Rank 1 (99.6) is the MiMo-V2.5-Pro Base score, registered on the post-trained repo. Rank 2 (96.8) is the Llama 3.1 405B Instruct score, registered on the base repo. Ranks 7 and 12 are DeepSeek base scores on the post-trained repos.
• The numbers are real self-reports. The labels are what is off, and the board has no column that would show it.
• GSM8K has 1,319 test problems, so one problem is 0.076 points. The rank 2 to rank 3 gap (0.74) is 10 problems, and a 95% interval at 95% accuracy is about 1.2 points wide on each side.
• A score of 99.6 means about 5 misses. The GSM8K-Platinum project relabeled 10 answers in the same test set. If those relabels are right, a score that high implies the model matched at least 5 of the 10 keys they call wrong.

In the chart, red marks rows where the model card gives this number for a sibling checkpoint, blue marks a row whose card table matches the repo (LLaDA2.0-flash), and grey marks rows I did not check against the card.

What does the GSM8K board on Hugging Face contain?

GSM8K is a set of grade-school math word problems with a train split of 7,473 and a test split of 1,319. Hugging Face lists openai/gsm8k as a benchmark dataset, so a model repo can publish a result in a .eval_results/*.yaml file and the dataset's board collects it.

I pulled the board on November 3, 2026 (KST) from the JSON behind the leaderboard view. It has 28 rows. Each row is from a different repo, verified is false on every row, and the top of the page reads like a clean ranking: 99.6, 96.8, 96.06, 94.16, 93.93.

The dataset's own eval.yaml describes a specific protocol: an inspect-ai task, a zero-shot prompt template that asks for a boxed final answer, four epochs reduced to pass@1, and a model-graded scorer. The rows do not follow one protocol. Of the 28 notes, 15 are empty. Where a note gives a shot count it ranges across 0, 4, 5 and 8. Two rows from one lab give a complete setup (8-shot, temperature 0, n=1319, lm-evaluation-harness), and one row reports a strict-match score over 1,075 of the 1,319 problems. So I treat each row as a claim with a source link, not as a measurement the board produced.

Why does rank 1 belong to a different checkpoint?

The top row is XiaomiMiMo/MiMo-V2.5-Pro at 99.6. Its model card has a section titled "Base Model Evaluation", and the GSM8K row there reads 8-shot with these values: 99.6 for MiMo-V2.5-Pro Base, 83.3 for MiMo-V2.5 Base, 92.6 for DeepSeek-V4-Pro Base. That is the only place GSM8K appears in the card. The card's own download table lists two repos: MiMo-V2.5-Pro and MiMo-V2.5-Pro-Base, and the second one exists on the Hub.

So the board shows the base-model score on the post-trained repo. The pull request that registered it says the results "were extracted from the model card". It was opened on April 27, 2026 and is still open. Nothing in the YAML says "base", and the 8-shot setting is not in the note, which is empty.

The card is not hiding anything. It labels its table as Base. The distinction is lost only in the step that turns a card into a board row.

Is that one slip, or does it repeat?

I checked the sources for the top rows. It repeats in four of the top 12, and in both directions.

Rank 2 is the mirror image. The Llama 3.1 model card covers every size and both variants. Its "Base pretrained models" table has no GSM8K row. Its "Instruction tuned models" table has one, and the 405B Instruct value is 96.8. The pull request is titled "Add GSM8K eval result (96.8, 8-shot CoT)" and attaches that number to the base repo meta-llama/Llama-3.1-405B.

For DeepSeek-V3 the card has a Base Model section with GSM8K at 89.3 and a Chat Model section with no GSM8K row. For DeepSeek-V4-Pro the table header names the three columns DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base and DeepSeek-V4-Pro-Base, and 92.6 is the last one.

Why does it matter if the number is the same? Because the board is read by repo name. Someone choosing a model for math word problems sees MiMo-V2.5-Pro at the top and downloads that repo. The card reports no GSM8K number for that checkpoint. If the four rows were moved to the sibling repos, the post-trained MiMo, DeepSeek-V4-Pro and DeepSeek-V3 would have no GSM8K row at all, and the Llama 405B base repo would lose its rank 2.

The correct placement would not change the ordering of the numbers: 99.6 would still lead, from the right repo. It would change which model each number tells you about.

Where do the rows come from, and what does "verified" mean here?

I fetched the discussion for every row that carries a pullRequest number, and listed .eval_results on the main branch of every repo.

All 20 pull requests are open, and for all 20 the YAML file is absent from main. For all 8 other rows it is present. Four of the 20 pull requests were opened on November 2, one day before I pulled the board, and one of them produced rank 3 (LLaDA2.0-flash, 96.06). The board moves when someone opens a pull request, with no action from the model's owner.

The Hugging Face documentation defines the badges. "Community" means a result submitted via an open PR that is not merged to main. "Verified" means a valid verify token, which requires the evaluation to have run in HF Jobs with inspect-ai. So 20 of 28 rows are community results by definition, and no row meets the verified bar. The documentation says community scores are visible while the PR is open. That is a reasonable design for bootstrapping a board. It also means a reader has to look at the badge to know which rows the model's owner stands behind, and a screenshot of the ranking does not show it.

The eight main-branch rows are not uniformly clean either. Two Mellum2 repos show the same 81.73. The YAML on the Base repo states that its numbers were measured on the pre-extension Base-Pretrain checkpoint, so it is one measurement displayed twice. This one is disclosed in the file, and it still occupies two ranks.

How much can 1,319 problems separate?

GSM8K is nearly saturated at the top, so the ordering is decided by a handful of problems.

The interval rows are a sampling calculation for a single run on a fixed test set. They ignore prompt variation, which is larger when rows use different shot counts. Even so, ranks 3 to 6 sit inside a band that a single run cannot separate, and the rows within it come from different protocols: 8-shot CoT for some, no note for others.

One other check is exact arithmetic. The two rows with a full setup report 84.9886 and 78.4685, and with n=1319 these are 1,121 and 1,035 correct answers. ProKope-7B's note says 1,075 of 1,319, which is 81.50. Those rows can be audited back to a count. Most rows cannot.

Can a 99.6 be right if some answer keys are wrong?

This part is arithmetic with one assumption, so I mark it as a question to check, not a finding.

The GSM8K-Platinum project (MIT, Gradient Science) reviewed the 1,319 test problems. Its dataset card reports 219 flagged by models, 110 rejected as poorly written, 10 relabeled, 99 verified, leaving 1,209 questions. The 10 relabeled items are answer keys that the authors found to be wrong.

Take a score of 99.6 on the original test set. 1,314 of 1,319 rounds to 99.6, so that is about 5 misses. Suppose the 10 relabels are correct and scoring is exact match against the original key. A model that solves an item correctly then loses it, because the key is wrong. A model that is right on all 10 would lose all 10 and top out at 99.24. To score 99.6 it must have matched the key on at least 5 of the 10, which means it gave the answer the key says, not the answer the authors say is right.

Three explanations fit: the relabels are wrong for some of those items, the scoring is more lenient than exact match, or the model has seen the original keys. I have not run the model, and a base model's 8-shot number with no note on answer extraction does not let me tell these apart. What I can say is that for a benchmark with known key errors, a score above the ceiling is a reason to read the setup, not a reason to rank it first. Platinum also reports that cleaning the set separates models that look tied on the original, which is the same point from the other side.

How do I pull the board and classify rows myself?

This script needs only the standard library. It counts rows by source and prints the sampling arithmetic used above.

import json, urllib.request, collections def get(url): req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"}) return json.load(urllib.request.urlopen(req, timeout=60)) DATASET = "openai/gsm8k" rows = get(f"https://huggingface.co/api/datasets/{DATASET}/leaderboard") by_source = collections.Counter() for r in rows: pr = r.get("pullRequest") if pr: disc = get(f"https://huggingface.co/api/models/{r['modelId']}/discussions/{pr}") by_source[("open PR" if disc["status"] == "open" else disc["status"])] += 1 else: by_source["main branch"] += 1 print("rows:", len(rows), "| verified:", sum(bool(r["verified"]) for r in rows)) print(dict(by_source)) n = 1319 print("one problem = %.3f points" % (100 / n)) for acc in (0.95, 0.99): se = (acc * (1 - acc) / n) ** 0.5 * 100 print("accuracy %.2f: 95%% half-width = %.2f points" % (acc, 1.96 * se)) print("ceiling if 10 keys are wrong: %.2f" % (100 * (n - 10) / n)) 

On November 3, 2026 it printed:

rows: 28 | verified: 0 {'open PR': 20, 'main branch': 8} one problem = 0.076 points accuracy 0.95: 95% half-width = 1.18 points accuracy 0.99: 95% half-width = 0.54 points ceiling if 10 keys are wrong: 99.24 

The board endpoint is undocumented, so the field names may change. To check a row, open its pullRequest discussion, read the YAML, then open the source card and find the table the number came from.

Checklist for reading a saturated board

1. Open the badge before the rank. Community means an open PR, and verified means a run in HF Jobs. Neither is a statement from the model's owner.
2. Match the checkpoint, not the family. Find the card table the number came from and read its column header. Base, instruct and chat checkpoints sit in different tables.
3. Ask how many problems separate two rows. On GSM8K, 10 problems is 0.76 points.
4. Prefer rows that state their setup. Shot count, temperature, n and the answer-extraction rule decide the score more than rank does.
5. Check for the same measurement on two repos. A copied number is one result, not two.
6. Distrust a score above the key ceiling. If a benchmark has known label errors, read the evaluation setup before you trust the lead.

FAQ

How many GSM8K results on Hugging Face are verified?
None. On November 3, 2026 all 28 rows on the board show verified as false. Verified requires a valid verify token from an inspect-ai run in HF Jobs.

Is the 99.6 at rank 1 of the GSM8K board wrong?
Nothing here shows the number is wrong. It is the MiMo-V2.5-Pro Base score from the model card's Base Model table, 8-shot. The problem is that it is registered on the post-trained repo, which has its own separate row in the card's download table.

What does "community" mean on a Hugging Face leaderboard row?
It means the result was submitted through an open pull request that has not been merged to the repo's main branch. In the GSM8K board, 20 of 28 rows are in that state.

Why is the Llama 3.1 405B base repo ranked second on GSM8K?
Its row carries 96.8, which the shared model card gives for Llama 3.1 405B Instruct, 8-shot CoT. The card has no GSM8K row in its base table. The instruct repo is a separate repo.

How many problems does one point of GSM8K accuracy represent?
GSM8K has 1,319 test problems, so one point is about 13 problems and one problem is 0.076 points.

What is GSM8K-Platinum?
It is a revised version of the GSM8K test set from Gradient Science. Of 1,319 test problems it rejected 110, relabeled 10 and kept 1,209. It exists because the original set has ambiguous questions and wrong keys.

Further reading

• GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank
• How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides

Sources

• GSM8K dataset and board: https://huggingface.co/datasets/openai/gsm8k
• GSM8K evaluation definition (eval.yaml): https://huggingface.co/datasets/openai/gsm8k/blob/main/eval.yaml
• Hugging Face documentation, Evaluation Results (badges, community PRs): https://huggingface.co/docs/hub/eval-results
• GSM8K-Platinum dataset card: https://huggingface.co/datasets/madrylab/gsm8k-platinum
• GSM8K-Platinum write-up: https://gradientscience.org/gsm8k-platinum/
• Rank 1 model card (Base Model Evaluation table): https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro
• Rank 1 registering pull request: https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro/discussions/1
• Llama 3.1 model card (base and instruction tuned tables): https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md
• Rank 2 registering pull request: https://huggingface.co/meta-llama/Llama-3.1-405B/discussions/37
• DeepSeek-V4-Pro model card: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro
• DeepSeek-V3 model card: https://huggingface.co/deepseek-ai/DeepSeek-V3

All counts in this article are my own measurements from the board JSON, pull request discussions, dataset cards and the cited model cards on November 3, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. Which table a number came from is my reading of each card, and the board endpoint is undocumented and may change.