Public leaderboards will not tell you which model should run your highest-volume prompt. I'd start with one narrow step you can score, a labelled set built from real production misses, and a side-by-side run against a frontier model plus one small language model candidate. Only after that bake-off would I commit to post-training or knowledge distillation.

Which task is worth a bake-off?

Pick a single step with a stable target output, not the whole agent. The pattern from teams that prototype on the frontier and shrink later is to isolate high-volume ancillary work such as query summarization and canonicalization before retrieval. A classifier with a rubric you can apply consistently is another candidate I would test.

If requirements still change week to week, stay on the frontier until the bar stops moving. If one prompt dominates spend and the right answer format is stable, that is your bake-off candidate. Fergal Reid describes Intercom running query summarization and canonicalization before RAG at scale, then replacing a costly frontier-class setup with a post-trained open-weight model once back tests and production tests showed it was close enough to justify tuning.

Do not bake off open-ended reasoning or multi-step workflows and expect a small model to tie the frontier. Maxime Labonne is blunt that replacement depends on the task.

So it really depends on your task. It will not be able to replace cloud models quality if you need some kind of complex workflows.

Maxime Labonne, Liquid AI

How do I build a labelled set that actually bites?

Your set should hurt a little. Every time a customer reports a hallucination, Fergal Reid’s team adds it to their internal back tests, so the suite holds real hallucinations from the wild. Copy that shape: pull production failures, edge phrasings, ambiguous inputs, and cases where your current system barely passes.

Label for the exact output you need from this one step, not downstream resolution unless this prompt directly drives it. Keep inputs as users actually type them. Score with a mix of exact checks where you can (schema, required fields, forbidden phrases) and human or judge review on a fixed slice so you are not tuning to a contaminated public benchmark.

it's so easy for a researcher or a scientist trying to optimize a benchmark to get good at teaching the model how to do the kind of task that's in the benchmark and then suffer generalization failure.

Fergal Reid, Intercom

Run the frontier model first on the same set while you are still defining good enough. That ceiling tells you what pass rate a cheaper model must hit before you invest in fine-tuning.

What should I measure besides accuracy?

For each model variant, log quality on the labelled set, p50 and p95 latency for the prompt at production context sizes, and cost per successful task (tokens in and out times price, plus any fixed hosting if you self-host the candidate).

Compare at equal prompting effort: well-prompted frontier vs well-prompted small model before you declare the small one needs training. Fergal Reid says they like a production test of a well-prompted but untrained open-weight model before post-training, just to see if it is in the ballpark.

Production A/B still wins once you have a release candidate, but the bake-off keeps you from burning GPU cycles on the wrong base model. Back tests can drift from live behavior; plan to reconcile with in-production tests when volume allows.

What does a minimal bake-off script look like?

Below is a sketch, not production code. It runs the same JSONL fixtures through two named endpoints you already have, scores pass rate, and prints latency and rough cost.

# Sketch: side-by-side bake-off (no vendor SDKs) import json, time, statistics from dataclasses import dataclass @dataclass class Result: model: str ok: bool latency_s: float in_tok: int out_tok: int def call(model: str, prompt: str) -> tuple[str, int, int]: # Wire to your gateway: returns (text, input_tokens, output_tokens) raise NotImplementedError def score(gold: str, pred: str) -> bool: return gold.strip() == pred.strip() # replace with rubric / schema check def run(model: str, fixtures, price_per_1k): rows = [] for ex in fixtures: t0 = time.perf_counter() pred, ti, to = call(model, ex["input"]) lat = time.perf_counter() - t0 rows.append(Result(model, score(ex["gold"], pred), lat, ti, to)) ok = sum(r.ok for r in rows) / len(rows) cost = sum((r.in_tok + r.out_tok) / 1000 * price_per_1k for r in rows) / len(rows) p95 = statistics.quantiles([r.latency_s for r in rows], n=20)[-1] print(model, f"pass={ok:.1%}", f"cost/task=${cost:.4f}", f"p95={p95:.2f}s") fixtures = [json.loads(l) for l in open("prod_slice.jsonl")] run("frontier-default", fixtures, price_per_1k=0.015) run("slm-candidate", fixtures, price_per_1k=0.001)

Swap in your real scorer, token accounting, and the smallest model that clears the pass rate you measured on the frontier run. If the candidate clears the bar with headroom, that is your shrink signal. If it misses, try another base or stay on the frontier for that step.

Checklist

  • Pick one high-volume prompt with a stable target format.
  • Build the labelled set from production failures and colloquial inputs, not leaderboard tasks.
  • Compare the frontier model and the small candidate with matched prompting before any post-training.
  • Record pass rate, p95 latency, and cost per task in the same run.
  • Plan a production test for whichever model wins the bake-off.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.