Most write-ups about AI automation show what was built. Fewer show how the builder knows it can be trusted. This article is about the second part: a method for testing an LLM step inside an automated pipeline. The examples come from a proof of concept I am building. The point is the method. The numbers illustrate it and will change as the project moves.
Four questions come up with every LLM step:
- Selection. Which model, with which settings and which prompt, is safe enough for this task?
- Optimization. Among the configurations that are safe, which one costs least in review work, latency and hardware?
- Regression. When one of them changes, what exactly moves, on which of the criteria you set in advance, and is the trade worth it?
- Reproducibility. Does the same configuration give the same result twice, or is a difference between two runs just noise?
The answer this article proposes is a series of tests on generated, labelled data, with one change per run, and what those tests leave behind: results tied to the exact model, settings and prompt that produced them, measured against a set that does not move. Results like that can be compared, and a repeat of the same record shows how much of a difference is the change and how much is the model itself.
An LLM step differs from ordinary code in one way that matters for testing. Its behaviour is a distribution, and the distribution moves without a code change: a new model version, another quantisation, a different server, a shift in the input. So even a functional check stops being one run with pass or fail. It needs a sample and statistics. The object is new, but the tasks look a lot like performance testing, and after 11 years of test automation and performance work in banking and SaaS that is where I took the method from.
Part 1: the method
The bench at a glance
Nothing in the picture is specific to language models. A frozen input set, a system under test, a record of every run and a scorer are what any load test is made of. The model changes two things: the scorer has to weigh errors instead of counting them, and every result needs a repeat before it counts, because a difference smaller than the spread between two repeats is not a result.
The pipeline under test
The pipeline has to be built so that it can be tested, and that is a design requirement, not a testing one. Every step is a module that can be swapped on its own, and the routing between steps is plain code, not a model. Then the model, the prompt or a check changes in one place while everything else stays the same, and the same input can be run again. In my case the chain runs on a small workflow engine I am building, where each step is an external module. The concrete chain is in Part 2.
The same property keeps the method open. A second model as a judge, a second reader to compare with the first, a reasoning pass for hard cases: each is one more node, tested the same way. The task in Part 2 does not need a judge yet, because labels and code checks answer the question cheaper and without a second distribution to test. When a task needs one, the bench does not change. A step that returns a probability or a score, a classifier, a decision model, a judge, is the easiest node of all to test: its own number is a metric, and a threshold on it is a check of code that the bench tunes on the labelled set like any other.
The labelled set
The list of scenarios comes from the person who does the work by hand today, and the negative scenarios come first. They are what the gate protects: a case the automation must refuse is worth more to the set than ten cases it handles easily. Each hard variant gets an allowed range of outcomes before the first run. In this proof of concept a message with a typo may be released exactly or sent to a person, but never released wrong. Without that range a typo in the test is either an unfair failure or a quietly forgiven one. Two things are scored, the outcome by the gate and the values by the model's score, so the label carries both.
Rare scenarios are represented as often as common ones, because critical errors live in the rare forms and a set that mirrors the traffic barely contains them. The list is never complete on the first day: every production error is a scenario it did not have. So adding a scenario has to be cheap, or the set stops growing and the gate stops meaning anything.
Whatever feeds the templates, two things hold. Real data is the best source for the common cases and still only shows the past, so the hard variants are generated on top of it. And generated data is honest only with the two checks on the right of the picture: the owner's verdicts on a sample of every scenario, and a held-out part written from the scenario list without reading the templates, so that it tests the pipeline and not the generator.
Scoring: a gate for selection, a cost for optimization
A single accuracy number hides the only thing that matters. The order in the picture is the order of weight, and the weights are not traded against each other. Critical errors are a gate: a step that adds one is out, whatever it saves. A case sent to a person is the price of a critical error avoided, so cost is reported and never used to reject a step that removes a critical error. Selection is the gate. Optimization is the cost among the configurations that pass it. Load testing has the same shape: the error rate is a gate, latency and throughput are a cost.
Silent closures get their own line because every pipeline with a person in the loop has this class: the outcome that is neither done nor handed over, and that nobody checks afterwards. Folded into cost it disappears. Counted apart it rejected two configurations in this proof of concept that looked fine on every other number.
Performance belongs on the target hardware. The same model on a laptop and on a GPU server gives numbers that cannot be compared, so capacity, tokens per second and latency tails are measured where the system will run, with host and GPU metrics, the way any load test does it. In this proof of concept only quality was measured, since the target hardware is not known yet. The run record keeps the time of every call anyway, so the first performance run will have something to compare with.
How many cases are enough
Zero errors on N cases does not mean zero. With no failures observed, the upper bound of the 95% confidence interval for the error rate is about 3/N. Zero critical errors on 545 cases says the true rate is below about 0.55%. On 74 cases the bound is about 4%.
One caveat belongs next to that arithmetic. The bound assumes independent cases. Generated messages come from a catalogue of a few dozen forms, so 697 messages are not 697 independent draws, and the real bound is looser than 3/N suggests. The remedy is more forms and more hand-written cases, not more messages from the same forms.
So the question "is testing expensive" turns into "what error rate does the business accept, and how much does it cost to prove it". Showing fewer than 1% critical errors takes about 300 clean independent cases. Showing fewer than 0.1% takes about 3000. It is close to asking how many runs you need before you trust a 99th percentile.
The cheapest source of those cases already exists in most teams. Every edge case a client reports and every decision a person confirms is a labelled example. The missing step is to add it to the regression set instead of only fixing it.
Every run is recorded
Prompts and settings change often, and a result that cannot be traced back to its configuration is worthless a week later. So every run writes a manifest and the raw answers: the model with its digest and quantisation, the server, every setting, the prompt version and its hash, the harness version and commit, the hashes of the set, and the reply to every message with tokens and time.
A script builds a database from these files, and a report recomputes every verdict from the database. The expected difference from what the run recorded is zero. No number is copied from a console or a dashboard.
One change per run: regression
Two changes in one run leave nobody able to say which one moved the result. The split into readings and recounts is what makes the rule affordable: a recount needs no model, so a check or a rule can be tried in seconds on readings already recorded, and a check and a prompt are never changed in the same step. Everything the model can notice, including a setting, needs a reading. A quantisation is a model change: Q8 against Q4 of the same model gave different counts and a different silent closure.
Comparisons happen on one host. The same model with the same prompt gave the same reply text for only 167 of 222 messages on a laptop CPU and on a GPU. Numbers from different servers are set side by side only with that caveat.
Reproducibility is checked with a repeat. Readers run at temperature 0 with reasoning off, and a repeat on the same host is expected to give the same text, which a repeat run confirms. With a model that is not deterministic, the spread between repeats is the resolution of the bench, and a change smaller than it is not measured. A stable error is useful: it reproduces as reliably as a correct answer, so it can be found and fixed.
How the bench is used
The baseline is what runs in production today, so every accepted change becomes the next baseline and the comparison is always against what is live, not against a number from months ago.
A change starts on one part of the set, a few hundred messages. That is not a cheap acceptance, it is where a change is understood: which form of message moved, which step of the chain, why. In this proof of concept every critical error was traced and closed in code on a small part first, and only then came the decision to drop the change, fix the cause and rerun, or go to the full set. Acceptance is always the full set against the baseline on the same host with a repeat. A release also spends the held-out set.
Where a small change ends and the full set from the start begins is a line each project draws for itself. The line that does not move is the kind of run: a change to a check or a rule is a recount, a change to anything the model sees is a reading.
After release
This half of the cycle is designed and not yet running, because there is no production yet. Its point is that the evidence does not stop at release. Every decision a person confirms or corrects is a labelled case the set did not have, and the improvements it suggests go through the same bench as everything before release, one change at a time. The move that pays most is from the model into code: a case that has become well understood is cheaper and cannot be misread once a rule handles it. The bundle ships only after a full run of the whole chain, because components can pass while the system fails. The tests before release build the evidence. The loop after release spends it.
What the bench pays back
Decisions on data during the proof of concept. Model, settings, prompt and checks are chosen from recorded runs, and every choice can be shown.
A change process that can run on its own. When production turns up a problem and something has to change, a rule, a check, the model or the prompt, the change is compared with the recorded baseline before it ships. The runs are scripts, so a CI job can do this on every change. It is regression testing, applied to a model.
Data for optimization. The record holds, for every message, where it went and why, and what the call cost in time and tokens. So the cost has an address: which forms send the most work to a person, which step of the chain, and whether a rule for one form is cheaper than a larger model.
A history of models under known conditions. Over time the record holds how different models behaved on the same frozen set and the same host. Two limits apply. The numbers belong to this task and this hardware, and they do not rank models in general. And comparability lives on the frozen set: when the set grows, old runs compare with new ones on the common part only.
A head start on the next task. The bench, the way of building a set with the process owner, the typical failures of small models and the code checks that catch them all carry over. The labelled data does not. It belongs to the domain, and often to the client, and the next task starts its own set.
Part 2: what came out of it
These are observations from one proof of concept at one point in time. The numbers will move with the next model or the next prompt, and the criteria will move too: a new class of error, once seen, becomes a new count, and the recorded answers let it be applied to every earlier run. What stays is the way each decision was reached: a frozen set, a recorded run, one change, a gate set before the run. None of them was taken on a feeling.
The chain this came from. The proof of concept reads incoming payment emails for a small business and decides, for each message, whether to pay and against which reference numbers. Four steps: a rule set extracts what it can find by pattern, a local model reads the message and answers in JSON with a decision and a list of reference numbers, code checks run over that answer, and a message is released for payment only if the rules and the model agree and no check fires. Any signal sends it to a person. The checks run after the reading, never inside the prompt, and each is named and versioned. The pipeline itself, its checks and what they catch, is a separate article.
Critical errors went from 51 to 1 in nine steps. Each step was one change, a code check, a prompt version or a rule file, measured against the run before it on the same machine. The held-out set, run once at the end, gave 0 critical errors. One lesson: a message form that must not be paid moved with every prompt change, so it is guarded by a code check, not by the prompt alone.
Once the gate holds, choosing a model is a cost decision. On one GPU host, with the same chain and the same set, five models from 4B to 31B made no critical errors. The 12B and the 31B model released 452 and 460 of 545 payable messages, a difference this set cannot tell apart, so the choice between them is hardware and speed, not quality. A Q8 build of the 12B model released four more messages and closed one silently. Without a separate count of silent closures it would have looked like the better of the two. Two smaller models made critical errors and are out.
A stronger prompt was rejected. A prompt variant told the model to send a message to a person when the subject names a reference that the payment table in the body does not. It did what it was asked, and it also sent four more regular messages to a person, closed one silently, and released 448 messages against 452 for the current prompt. A code check for the same form removed all 3 critical errors of that variant on a recount, with no loss in automation. The check was taken, the prompt was not. When a check and a prompt both close a hole, the check wins: it costs nothing in automation and does not move with the next model.
Small decision models did not separate the numbers. Before the chain had its code checks, a second idea was tested as a safety net behind the main reader: a small model that answers a yes or no question with a probability, asked about every 4 or 5 digit number the code finds in a message. Three such models from 0.3B to 2B, eight wordings, 114 messages. None of them separated the references from the other numbers. The best combination caught 6 of 8 misses of the main reader and flagged 37 to 46 percent of the messages that were already read correctly. The idea was dropped for this task, and may be tried again with larger models. It still paid for itself: reading the false flags by hand showed where they came from, a number next to a word of another role, the whole part of an amount, the last digits of a card, a reference in a quoted thread, and that list became the code checks of the chain.
A stable error is found by repeats. In a second pass with a reasoning model at temperature 0.6, on a frozen list of 30 messages to release and 15 controls, one model closed the same message silently in all three repeats. Because it repeats, it became a test case and a candidate for a code check. The same test showed a limit of the design: the second pass only raises automation. A critical error is released without a flag and never reaches it, so the gate has to hold before it.
Limits. The data is generated from the wording of real messages, and real mail has not been run through the chain, so false flags on real input are unknown. The checks were written with knowledge of the generator's forms. The prompt was tuned on one small model, and the held-out set was run for that model only. Performance was not measured: call times were recorded, but capacity, tokens per second and latency tails wait for the target hardware.
The point
The model is the new part. Most of what it takes to trust it is ordinary test engineering: a frozen labelled set, a record of every run, one change at a time, a gate set before the run, and a repeat before anything counts.
