Every dataset comparison you have read is a sales deck.

The author picks the rows they win. They bury the rows they lose. They call it a benchmark.

I did the opposite. I put my data next to four bigger datasets and measured all of us with the same ruler. I lost on size. I lost on repos. I lost on retry chains. And I lost on the one metric I invented myself.

Here is the whole table. Nothing hidden.

Benchmark results

First: what is one row?

Before any number, fix the unit.

My row is one full executed session. SWE-smith's row is a task with a bug put in it. SWE-Gym's row is a task. ToolACE's row is a dialogue. CommitPackFT's row is an edit pair, before and after, no process at all.

Those are not the same thing. Compare them raw and every number after is a lie by accident. So where a dataset has no process, I wrote N/A. Not zero. Zero would be an insult. N/A is the truth.

The ruler I built, and how it beat me

I care about one thing: does the data show a real error followed by a real fix? Agents do not fail at writing a function. They fail in the loop. They misread a traceback, repeat a mistake, never recover.

So I made a metric nobody else publishes: recovery density. Real error into real fix, per 10k tokens.

ToolACE and CommitPackFT cannot score it. They have no real errors. That is not fraud. ToolACE says its data is made up on purpose, on its own card. Different tool, different job.

SWE-smith and SWE-Gym can score it. So I scored them:

  • SWE-Gym: 10.72.
  • MegaBit-trajectories: 6.48.
  • SWE-smith: 4.03.

I lost on my own ruler with SWE-Gym. I printed it anyway.

Here is why that matters: if I only showed you the rows I win, you should not trust a single row I win either.

Where I lose. Read this first.

  1. Size: 66 sessions against 59,136 tasks and 702,062 edit pairs.
  2. Repos: Just 5 against 222.
  3. Longest error into retry chain: 4 against 64.
  4. Published training proof: they have it. I do not, yet.

A table where I win every row is marketing. A table where I lose these rows is measurement.

Where I win. The boring rows.

  1. Tool calls per row: 29. The next best is 26.
  2. Assistant turns per row: 27. Also the top.
  3. Tool calls per 1k tokens, the number that cannot cheat:
  • MegaBit-trajectories -> 1.485
  • SWE-smith -> 1.164
  • SWE-Gym -> 1.018
  • ToolACE -> 0.973

A long row can fake a big count. This one cannot. More real work in every token. Not bigger rows. Denser rows.

  1. Audit checks: 4 out of 4. The next best is 2.
  2. An offline verifier that passes 66 out of 66 on your machine.
  3. A hash per row. A license per row. A commit SHA per row.
  4. Split labels with reasons. Rejected edits flagged, not deleted.
  5. Rows with a real failing run and a retry: 89.4%, the highest of the five.

Nobody puts these in a sales deck. They are the rows that let you trust the other rows.

Why the boring rows matter more

You are not buying a score. You are buying food for a model. Food you cannot check is food you cannot trust.

I would rather hand you 66 sessions you can verify byte for byte than 700,000 rows you have to take on faith.

One tokenizer for all five, so the token counts mean the same thing. Versions and dates pinned for all five, including mine, which changed yesterday. If I did not pin mine, this table would be stale by tomorrow.

Same deal as always: MIT, free, no signup.

git clone https://github.com/Sumama-Jameel/megabit-trajectories.git cd megabit-trajectories python3 verify.py 

Run it on my data. Then run the same idea on theirs. Tell me where my ruler is wrong. I will fix it in the open, like last time.

One question

A dataset that publishes its losses looks more trustworthy than one that publishes a win.

But is that honesty? Or is it just better marketing? Pick a side. I will be down there.