How we made a 4.5B model think only when it changes the answer

There is a trap in building a decision model, and it is not the one you expect.

The obvious approach is: give the model a chain-of-thought budget and let it reason before answering. It works. Accuracy goes up on the hard cases. And then you ship it, and every trivial request — "which of these four teams owns this ticket" — now costs you a thousand tokens of deliberation that changes nothing.

Worse, on the cases where the model was already right, the reasoning can talk it out of the correct answer. A model that reasons itself from 96% correct into 51% wrong is not smarter. It is just slower and less reliable.

THX-02 is a 4.5B decision model we released under Apache-2.0, with a free API. This post is about the specific mechanism we built to avoid that trap: a gate that decides whether thinking is worth doing, and a mixing step that makes a bad thought unable to overturn a confident answer.

The numbers, up front, so you know what the mechanism buys: on math word problems the thought adds about 25 points. On natural language inference, code tracing and graduate-level science it adds 4 to 6. Median latency with the gate active stays around 80 ms. Direct answering alone scores 63.83 on Decision Index 0.3; with the gate, 65.06.

The shape of the problem

A decision model is not a chat model with a different prompt. The interface is different in a way that changes everything downstream:

from thx02 import Client, choice, noul, score client = Client() result = client.decide( state="Ticket: customer says the mobile app logs them out every few minutes " "after the latest update. Android 14, app version 4.2.1.", questions=[ choice("team", ["billing", "support", "shipping", "technical", "security"]), noul("escalate", "Should this go to a senior engineer?"), score("severity", "How severe is this for the customer?", 1, 5), ], ) 

Every answer is a full distribution over the options you supplied. That is the contract. The model cannot emit a label that is not on your list, because there is no generation step — it scores each option with a marker token and normalizes.

That constraint is what makes the gate possible in the first place. If the model were free-generating, "did thinking help?" would be a question about text similarity. Here it is a question about probability mass on a fixed set, which is measurable.

The gate

The gate is not a separate classifier bolted on the side. It is part of the model's forward pass, and it answers two questions:

1. Is the model unsure? Measured from the entropy and top-probability of the direct pass.
2. Does this question need step-by-step work? Arithmetic, multi-step policy rules, code tracing — the categories where a single forward pass structurally cannot do the job.

Both have to point the same way. Uncertainty alone is not enough: a model can be unsure about a recognition task and thinking will not help, because there is nothing to compute. Structural need alone is not enough either: if the direct pass is already at 97%, the arithmetic is not going to move it.

When the gate fires, a small adapter writes a thought capped at 96 tokens. Then the model re-reads the state and question with that thought in context and produces a second distribution.

result = client.decide( state="Applicant: monthly income 4,200 AZN, existing loan payments 1,800 AZN, " "requested new payment 600 AZN.", questions=[noul("dti", "Does this applicant pass the 50% debt-to-income policy?")], think=True, return_thoughts=True, ) print(result.answers[0].probabilities) print(result.thoughts[0]) 

fail: 75.5% pass: 24.5% thought: existing 1800 + new 600 = 2400. 2400 / 4200 = 57.1% > 50% policy limit. fail. 

The thought is short on purpose. 96 tokens is enough for arithmetic and a two-step rule check. It is not enough to ramble, and that matters — long reasoning traces are where models go to lose the plot.

Why a bad thought cannot overturn a confident answer

This is the part I would want to read if I were evaluating someone else's model.

The final answer is a weighted mixture of the direct distribution and the post-thought distribution. The weight is not fixed. It is conditioned on how confident the direct pass was:

• If the direct pass is confident, the thought's contribution is bounded. A thought that says "actually it's the other one" cannot drag a 96% answer below the decision threshold.
• If the direct pass is uncertain, the thought carries real weight, because that is exactly the case the gate fired for.

In practice this means the failure mode is asymmetric in the safe direction. The worst case is that thinking fails to help. The bad case — thinking actively hurts — is suppressed by construction.

We did not take this on faith. Candidate thoughts were tested on the model itself during training, and kept only if they raised the probability of the correct answer. A thought that did not move the answer was discarded, because we do not pay latency for decoration.

Calibration is the actual hard part

Anyone can make a model output numbers between 0 and 1. Making those numbers mean something is the work.

Three things break naive probability output:

Multi-token labels. If your option is "billing_and_refunds", the model tokenizes it into several pieces, and "the probability of the label" becomes ambiguous. THX-02 reads each option with a marker token, so every option is scored through the same path regardless of how many tokens its name contains.

Position bias. Options presented in different orders get different scores. We train with option-order averaging, and at inference you can enable tta=4 to average over four orderings. It costs latency; it buys stability.

Calibration drift. A model trained with cross-entropy on hard labels learns to be confident, not to be right about its confidence. We trained against full probability distributions with a strictly proper scoring rule plus a KL anchor to the base model. A proper scoring rule is the only family of losses where the optimal strategy is to report your true belief — anything else rewards overconfidence or hedging.

The practical test is not "is the top answer right." It is: of all the requests where the model said 90%, how many were actually right? If that number is not close to 90, the probabilities are decorative and your thresholds are meaningless.

What the gate actually buys, per category

That last row is the point. On pure recognition the gate stays off, and you pay nothing. The 25-point category is where a single forward pass is structurally incapable — you cannot compute 2400/4200 without doing the division somewhere.

If you are building a router, most of your traffic looks like the bottom row. That is why the median stays at 80 ms instead of climbing.

Evaluation, honestly

Decision Index 0.3: 65.06 with thinking, 63.83 direct. For context, the best 27B decision model we could find scores 62.25, Cloudflare clef 61.71, Jev 57.96.

The caveats, because a benchmark number without them is marketing:

• We ran the official kit on a stratified sample of 6,018 requests, not the full suite.
• Competitor numbers are their published public indexes, not our re-runs.
• We decontaminated against test sets: exact and n-gram matching for text, exact and perceptual hashing for all 10,742 evaluation images.
• No benchmark test item was used in training.
• We will submit for independent measurement and publish the result.

The honest summary: the mechanism works, the numbers are ours, and the sample is a sample.

Where it fails

• Very large bare-label sets with no descriptions. Hand it 200 category names and nothing else and it degrades. Add a one-line description per option and it recovers — the model needs something to score against.
• Pure recognition. Thinking does not help, correctly, but there is also no reasoning gain available.
• Long explanations. The thought trace is ≤96 tokens by design. If you need a paragraph of justification, this is not the tool.

Try it

pip install thx02 export HALX_API_KEY="your-key" 

The API is free. The weights are Apache-2.0 at HAL-X-AI/THX-02, and pip install "thx02[local]" runs the same model on a single 24 GB GPU if you would rather keep everything in your own perimeter.

If you build something with the gate, I would like to hear where it fires when it should not. That is the failure mode I am least sure we have fully closed.

HAL-X, Baku