Almost every RAG evaluation metric on offer needs a language model to produce it. Faithfulness, answer relevance, context precision: a model reads the answer and scores it. Those are good metrics. They measure things that are hard to measure otherwise, and for a research sweep or a quarterly quality review, use them. They cannot gate a build, and the reason people usually give is only half of it. The half everyone says A judged evaluation costs money per run. A hundred cases across a few metrics is a few thousand model calls, which is real money on every commit and every branch. The consequence is that you move the gate. It runs nightly instead of per commit, then weekly, then on a button someone remembers to press. A check that costs a dollar gets run less, and a check that runs less catches things later, which is the property you were buying. The half that matters more A judge is not deterministic. Run the same dataset against the same predictions twice and you get slightly different scores. Not wildly different, but different. That is workable for a report and disqualifying for a gate, because a gate exists to answer one question: did this change make things worse? Answering it means comparing two numbers. If both carry noise of unknown size, you cannot separate a regression from the measurement. You get a check that fails sometimes for no reason, and the standard response to that is to disable it, usually within a week, usually by someone under deadline pressure. So the judged metric fails twice: too expensive to run often, and untrustworthy on the difference when it does run. What is left when you remove the model More than you would think, and all of it deterministic. From labelled relevant documents: precision@k, recall@k, MRR, nDCG@k, hit rate. These need a golden dataset and nothing else. From expected answers: exact match after SQuAD style normalisation, token F1 for partial credit, required phrase presence for when an amount or a date must appear. Recall bounds everything downstream, so it is the one to be loudest about. The model cannot cite what retrieval never fetched. At recall@k of 0.6, forty percent of your questions were unanswerable before generation began, and no amount of prompt engineering touches that. Precision measures how much noise is in the context window, which is what predicts hallucination. The honest limit One measure in my own tool is a proxy, and the documentation says so. groundedness is the share of answer content words that appear in the retrieved context: const answerTokens = normalise(predicted).split(' ').filter((t) => t.length > 2); const contextTokens = new Set(normalise(contexts.join(' ')).split(' ')); return answerTokens.filter((t) => contextTokens.has(t)).length / answerTokens.length; That is lexical overlap. It will miss a fluent misreading of a passage that was correctly retrieved, which a judge would catch. It does catch an answer invented wholesale, which is the failure that gets shipped, and it costs nothing and returns the same number every time. The trade is a cheap proxy running on every commit against an accurate measure running monthly. Which one is the better metric and which one is still switched on in March are different questions. Missing inputs are not zeroes One detail decides whether people trust the output. If a case has no labelled relevant documents, the retrieval metrics for that case are omitted rather than scored zero. Aggregation skips missing values instead of averaging them in: if (typeof value !== 'number' || Number.isNaN(value)) continue; A partially labelled dataset should report what it can measure. Scoring the gaps as zero produces a column of failures that describes your labelling rather than your system, and nobody reads a report full of zeroes twice. Why I wrote one The platforms are priced for teams. Confident AI runs Free, Starter at $200 a month and Team at $2,000, with Enterprise above that. Braintrust Pro is $249. Galileo Pro is $100. Each has a free tier and each meters it: Confident AI's is two seats, one project and five test runs a week, which a per-commit gate exhausts by Tuesday. (Checked on the vendors' own pages, 9 September 2026.) The open source libraries, RAGAS and DeepEval, compute good metrics and leave you to build the storage, the comparison and the CI gate yourself. Most solo projects end up with three tools wired together and no gate at all. npx ragbench gate --baseline main --threshold recall@k=0.8 Local SQLite for history, zero dependencies, nothing leaves your machine. ragbench. The gate has a second half that matters more than the thresholds, and that is the next article.