A prediction routine returned a valid result. The command then failed before it could save that result.
This happened in spec-lane, a CLI I maintain that can estimate token usage for AI-assisted development from previous observations.
The command ran two related paths: estimate/v2, which selects observations with matching conditions, followed by a v1 estimator over a broader population. The second path could throw even after the first had produced a prediction.
The surprising part was an assumption in the error handler: because v2's population was a subset of v1's, a v1 failure supposedly meant v2 must also have abstained.
That does not follow when the two paths select their nearest neighbors from different populations.
Eight observations worked. Twelve did not.
The regression fixture makes the problem small enough to inspect.
These are synthetic observations used to pin down the failure conditions, not a claim that the production store contained exactly twelve records.
The fixture supplies a configured cohort and an explicit novel_surface: established declaration, so an unknown novelty condition does not cause abstention. No reference-table values are supplied.
| Observations | Count | Distance from the target | Usable for prediction? |
|---|---|---|---|
| Matching cohort, eligible | 8 | Slightly farther away | Yes |
| No matching cohort, ineligible | 4 | Closer; distance zero in the fixture | No |
A cohort groups observations with matching model and execution-policy conditions. V2 filters for eligible, matching observations before estimating. Here it retains eight usable records and can predict, reusing the existing estimator over that subset.
The later global v1 run ranks the broader population. It selects the nearest seven observations, then requires at least five usable observations among them.
The four ineligible records are closer, so all four enter those seven neighbors. Only three usable observations remain. The global path throws ReferenceTableRequiredError.
| Path | Selected population and neighbors | Outcome |
|---|---|---|
| V2 | Filter to eight eligible, matching observations, then select neighbors | Predicts |
| Global v1 | Select the nearest seven from all twelve: four ineligible plus three usable | Fails the five-usable-neighbor gate |
There are enough usable observations in the store. There are not enough usable observations among the seven selected by the global run.
Population inclusion alone does not determine the outcome. Filtering and neighbor selection interact.
The error handler treated success as an impossible state
Before the fix, buildEstimateRevision calculated the v2 decision first and then ran v1.
When v1 threw ReferenceTableRequiredError, the code could save an abstained revision if v2 had also abstained. But if v2 had predicted, it rethrew the original exception:
if (!(err instanceof ReferenceTableRequiredError)) throw err; if (decisionV2.decision.status !== "abstained") { throw err; } The comment above this code justified the branch using the subset relationship between the populations.
That argument overlooks the nearest-neighbor gate. The broader run can select a different set of neighbors and fail, while the filtered run still has enough usable neighbors to predict.
The prediction was already available. The integration logic discarded the call because it regarded that combination of outcomes as unexpected.
Keep the result, and keep its provenance
PR #71 changed the internal v2 API to retain the exact estimator result behind a predicted decision.
That result includes the selected neighbors and population information. The code does not reconstruct the result from token quantiles or implement a second ranking algorithm.
When global v1 throws ReferenceTableRequiredError but v2 has predicted, the revision is built from the captured cohort-subset result. It also receives a distinct version label:
estimator_version: estimate/v2-cohort@lane-estimator/0.1.0 That label matters. Successfully saving v2's numbers under the global v1 label would leave a misleading record of where those numbers came from.
The exception handling remains narrow:
- Errors other than
ReferenceTableRequiredErrorstill propagate. - If v2 abstains, the existing abstained-revision behavior remains.
- An explicitly supplied reference table keeps its existing path.
- A successful global v1 run keeps its normal path.
The saved result also carries an estimated cost from the same captured computation. That is an auxiliary output, not a guarantee made by the token-prediction estimate/v2 contract.
Do not let the target train its own estimate
The same change excludes observations belonging to the target intent before either estimator sees the population.
Otherwise, a completed intent's own observation can appear at distance zero. Re-estimating it would not be an independent check of how well the system predicts another task.
This filter lives in one place, in buildEstimateRevision, and feeds both paths. The function now requires targetIntentId, which is a breaking change for external TypeScript callers.
The persisted local revision also gains optional neighbors[].record_id values. An intent can have several observation records, so an intent ID alone does not identify the exact record selected. The strict estimate/v2 decision contract remains unchanged.
Compatibility is backward-read only: new readers accept old revisions, while an older JSON-Schema consumer that rejects additional neighbor fields rejects revisions carrying record_id.
What the checks establish
The eight-plus-four fixture asserts that global v1 fails while the revised service returns a v2-cohort revision. It compares the stored prediction, neighbors, and population information with the result computed over the matching subset.
The PR records regression checks failing on the earlier 048715d revision. It separately identifies characterization checks—such as preserving abstention and unrelated-error behavior—that already passed before the fix. Not every added test is a red-before/green-after regression.
There is also a post-release verification report using real living-twin data and version 0.14.0. Those commands ran against a copy of the spec directory without the --reference-* workaround:
| Target | Reported outcome |
|---|---|
| G5-2a | Exit 0 through the v2-cohort path; eight eligible observations after self-exclusion; seven neighbor record IDs, none belonging to the target |
| G5-2b | Exit 0 through the ordinary global-v1 path, with v2 also predicting |
These are summaries of the execution report, separate from the synthetic fixture. G5-2b did not exercise the v2-cohort fallback path.
The checks establish that a usable v2 result can survive this particular later failure and be saved with its provenance. They do not establish the accuracy of future token predictions.
The fix changed how two estimators were connected, rather than their prediction formula. If related implementations consume different populations, the failure conditions of one are not automatically valid failure conditions for the other.
When keeping two paths in the same command, I would inspect their input selection and failure rules as closely as their return types.
AI assistance was used to draft and edit this article.
