I got tired of seeing the same model quoted at 39% and 12.8% with no explanation, so I recorded a full walkthrough breaking down exactly where those numbers come from. It started when I re-ran a coding eval and got a score nearly a third of what the leaderboard claimed — same weights, same prompts, different harness.

The video goes through the two evaluation setups side by side: one uses the model's native tool-calling format, the other wraps everything in a generic function-calling shim. That shim alone drops the score from 39% to 12.8% because the model keeps emitting parallel tool calls that the generic parser rejects, and every rejected call counts as a failed test case. I show the raw logs where you can see the parsing failures line up one-to-one with the score drop.

It cost me about six hours of GPU time re-running both configurations to confirm the delta wasn't noise, plus the editing time to make the logs readable on screen. The walkthrough is up at https://www.youtube.com/watch?v=gBSZXtzBSfE if you want to see the exact log lines that explain the gap.