I gave two Gemini Flash-Lite models the same 9 real bug reports from GitHub and let each one work in a loop: search, read a file, edit, run the tests, submit. Each model fixed one bug. Between them they made 964 model calls.

The tasks come from SWE-bench-Live (github.com/microsoft/SWE-bench-Live, MIT). I picked small pure-Python repos I could run on a Windows machine without Docker: attrs, faker and cfn-lint. A task only counts if the original tests fail before the fix and pass with the real fix applied. The agent never sees the new tests. When it submits, I apply them and run them. If they fail, it gets one more attempt and sees the tail of the failing output, like a CI log. Each attempt is capped at 30 steps.

The models were gemini-3.5-flash-lite and gemini-3.1-flash-lite, no thinking, on the free tier, so nothing was billed. The dollar figures below are token counts multiplied by the paid list prices ($0.30 / $0.03 cached / $2.50 per million tokens for 3.5, $0.25 / $0.025 / $1.50 for 3.1). Read them as what this trace would cost, not as a bill.

3.5 Flash-Lite 3.1 Flash-Lite
Bugs fixed 1 of 9 (first try) 1 of 9 (second try)
Attempts 17 18
Model calls 483 481
Input tokens 4.29M 3.00M
Output tokens 16K 24K
Input read from cache 66% 53%
Total at list price $0.561 $0.426
Per task $0.062 $0.047
Per fixed bug $0.561 $0.426
Spent on tasks that never passed 96% 88%
Spent on second attempts 49% 46%

Per task it is about 5 or 6 cents. Per fixed bug it is 43 to 56 cents, because one success has to carry all the failures. That gap is the whole idea of cost per successful task. It is also a single data point per model: the two models fixed different bugs, and a different task would have moved these numbers a lot.

Output was tiny, 16K and 24K tokens against millions of input. An agent loop sends the whole conversation back on every step. The first call of an attempt was about 1,060 tokens. The last call was 10,000 to 15,000. The cache helped, but fresh input was still 78% and 82% of the cost.

What did they do with their steps? 3.5 made 306 searches, 124 file reads, 33 test runs and 5 edits in 483 calls. 14 of its 17 attempts ran out of steps before it submitted anything. 3.1 searched less (160 times), edited more (50 edits) and submitted in 7 of 18 attempts, but 11 attempts still hit the cap, and only one of the submitted attempts passed the hidden tests. The only retry that paid off was 3.1 on attrs-1320: the second attempt saw the failing test output and fixed it.

I also put the measured numbers into three calculators on my site, to see which ones hold up on an agent loop.

  • Importing the trace into the team API cost report gave $0.5611 and $0.4256, the same as my per-call sums. The trace format has no field for thinking tokens, latency or tool calls, so I keep those in a side file.
  • The retry/fallback calculator, given about 2 attempts per task, 89% of tasks unresolved and the cached share, came to $0.5616 and $0.4258.
  • The workflow calculator, which takes calls per task and tokens per call, said $1.33 and $0.78. That is 2.4 and 1.8 times too high. It has no cache input, so it bills every re-sent token at the full input price. For a loop that re-reads its context 50 times per task, that was the biggest error I found, and it is a gap in my own tool.

Things that make this weaker than it looks. Nine tasks, one fix each, on different bugs, so I can't say one model is better. The issues date from July 2024 to March 2025, so both models may have seen them. Google's pricing page lists context caching as unavailable on the free tier for 3.5 Flash-Lite, yet cache tokens were reported, and paid-tier hit rates could differ. The 30-step cap and my prompt probably cost both models some fixes. 3.1 hit its free daily limit of 500 requests after 9 of 12 tasks, so I left the other three out for both models. And 3.1 needed 14 API retries where 3.5 needed 1.

I built this to compare GPT-6.2 Sol and Gemini 4 Neon on the same tasks. I don't have an API budget for those yet, and Neon has no free tier, so I ran what I could. The harness has adapters for both, but I have only tested them against fake servers so far. There is also a write-up of cost per successful task if you want the definition I used.