A workbench at night: an open graphics card under a lamp. Made with NeuroBrix: Sana 1600M.

Made with NeuroBrix: every picture in this article was drawn by Sana 1600M, running as a .nbx container on one 16 GB V100.

People have run our last published release on machines we do not own. One of them opened issue 6 from an RTX 4070 on Windows. Our own checks found more. We owe you an account, so here it is: what went wrong, why, and what we did about it.

Two rules for reading it:

  • Every fix below is on the main branch of the engine (neurobrix) or of its tensor library (nbxtensor), and each one links to its commit. You can read the change and the test that holds it.
  • None of it is released yet. The next version is not tagged and not on PyPI. It ships when the list at the end of this article is closed, and we give no date before then.

1. The same question could get a different answer on another machine

What you could see. A served conversation could give a different fifth answer after a fresh install, with the same model, the same prompt and the same seed. The library's changelog records that the fault was already present in published nbxtensor 0.1.3 and 0.0.8.

Why it happened. Each of our GPU kernels can launch under several candidate shapes, such as the tile size or the width of a chunk. On first use, each machine measures the candidates and keeps the fastest one. That is the right way to get speed from any card. But floating-point addition is not associative: (a + b) + c and a + (b + c) can differ in the last bit. Two tile shapes add the same numbers in a different order, so two machines that picked two different tiles could produce two different last bits. Over a long generation, one different bit can become a different token, and the conversation drifts.

On a V100, 13 of the 24 candidates of a [320, 512] softmax gave different bits from the others.

Two identical hourglasses. Made with NeuroBrix: Sana 1600M.

Schema: two tile shapes add the same four numbers in two orders and get two last bits; after the fix, every candidate adds in the one fixed order.

What we changed. Every kernel now gives the same bits under every launch candidate. The tuner still picks the fastest shape, but the shape no longer decides the order of the sums. This covers softmax, RMS norm, layer norm, the vector GEMV, paged decode attention, matmul, addmm and baddbmm in float16 and float32, and conv2d (fixed on a 16-wide input-feature chunk).
Commits: 6871e24, 8349314, f5727cf, 8011b99, 7015963.

The fixed order is not free, so we measured it where it matters. The paged decode now steps its online softmax over one fixed tile of 64 positions. A free sub-tile would also have kept the order, but it doubled the time: 194.9 µs against 108.9 µs at 8,000 positions on a V100 (ffdf213).

The tuner itself had faults, and we fixed them in the same pass:

  • A zero that never came. Tuning a kernel that asks for its output to be reset to zero raised an AttributeError, because the tensor had no zero_. It now zeroes in place, and a strided view goes to the fill kernel (1fce065).
  • One configuration fed the next. When tuning the chunked gated-delta-rule state kernel, each configuration after the first read the previous one's output as its input. The first untuned call returned 2.96e+02 where the bound is 4e-06 (Mac M4 Pro). The input is now restored before each configuration (3a9e3fd), and the certification harness restores or zeroes its arguments and runs each configuration's pre_hook, as Triton's own sweep does (23acfec).
  • A keyword silently dropped. A tuning keyword that the installed Triton does not know used to be dropped without a word. A kernel that overwrites its input would then have been tuned on the previous configuration's output. It is now refused by name (65e30fc, fc23171).

Old results are never reused by mistake. Because the fix changes bits, every kernel certificate now carries a "bits epoch". A certificate from another epoch, or with none, is refused by name at load, and those keys are tuned again. The replay artifact is named by the epoch too, so a file written by an older library is never read back (e9e5e76, 5f4d5a6, 135670e).

The same rule now holds on Apple GPUs. Conv2d offers the tuner only the candidates whose float32 tile fits the device's threadgroup memory, because larger tiles gave other bits on Metal. An Apple GPU keeps 34 of the 36 candidates, and a V100 keeps all 36 (182f0dd).

2. A chat prompt is now the vendor's prompt, token for token

A language model answers best when it receives exactly the tokens it was trained on. We found four ways our prompts could differ:

  • The prompt's token ids. Over five served turns of a TinyLlama chat, the engine prefilled 25, 94, 165, 232 and 300 tokens where the vendor's tokenizer gives 23, 90, 159, 224 and 290. There were two causes. The rendered template was encoded with special tokens added, which put in an extra beginning-of-text token that the template never wrote; the vendor encodes a finished template without them. And a tokenizer that adds its word-start mark only at the start of the input got an extra one after each end-of-turn token. The prompt is now the vendor's ids, token for token (61cfb8f).
  • A decision that leaked between requests. Our own review of that fix found that it stored the choice of whether to add the prefix on tokenizer parts shared between encodes. With several serving threads, one request could leak it into another: 7 wrong prompts in 6,000 in a two-thread test. The choice is now passed down through each encode, and the same test gives 0 (59f95be).
  • An extra gap at the start of a reply. The chat template had already written that gap, and we wrote it again. It is now removed once, so the stored history is exactly the prompt the model saw (142a6d7).
  • Templates and tokenizer files we read differently from the vendor.
    • A chat template that marks assistant spans with the generation tag now renders as the vendor renders it (9abbcc8).
    • A tokenizer.json that uses the vendor's Unicode classes, possessive quantifiers and line anchors now loads and splits as the vendor does (e986780).
    • Two tokenizers now give the vendor's ids for the control characters U+001C to U+001F (0a6bc4c).

3. An RTX 40-series card was refused (issue 6)

What a user saw. On Windows with an RTX 4070, neurobrix serve stopped with No hardware profile for nvidia/ada.

Why. NeuroBrix reads every hardware-dependent value (block sizes, shared-memory limits, precision support) from a profile for each architecture, and it refuses to guess one. A guessed block size gives wrong output rather than an error, and we would rather stop than answer wrongly. The refusal was correct. The missing piece was the profile.

What we changed.

  • Cards of compute capability 7.5 (Turing: T4, RTX 20-series), 8.9 (Ada: L4, L40S, RTX 40-series, including the RTX 4070 of issue 6) and 10.0 (Blackwell: B200) now have a hardware profile. The profile is keyed on the capability, so every card of that capability uses it. It is read from NVIDIA's documentation and marked "unmeasured", because no card of these classes is on our machines yet. These cards run on the PyTorch branch; the Triton branch refuses them by name until their profile is measured. A card of capability 8.6 (an A10 or an RTX 30-series) is still refused by name (af59676).
  • The library reads the documented shared-memory figures for every NVIDIA capability that the compiler targets. A minor revision reads its major's file only where the documentation says it may; any other minor is refused by name (f19f865, 3cba8db).
  • Every NVIDIA architecture the detector names now either plans on its own profile or is refused by name, and a test holds this.

4. The first launch on a fresh install is long

What you could see. On a fresh install, the first launch took minutes before the first answer, and the more cards the machine had, the longer it took. We timed neurobrix serve from the command to ready, on a fresh home, on our development line of 19 November (after the last release), on a machine with four V100 cards (two of 16 GB, two of 32 GB):

Model Branch 1 card 2 cards 4 cards
TinyLlama-1.1B-Chat PyTorch 2.6 min 5.1 min 15.4 min
TinyLlama-1.1B-Chat Triton 3.0 min 5.0 min 16.3 min
Qwen2.5-Coder-7B-Instruct PyTorch 2.6 min 5.3 min 14.9 min
Qwen2.5-Coder-7B-Instruct Triton 12.9 min 14.5 min 16.6 min

The released version was not timed on its own.

Why. The first launch measures your machine instead of assuming it, and two of those measures cost most of the time. The first finds the size of the memory-pool chunk that suits each card model. It tries the candidate sizes one by one, each in its own child process that maps and unmaps card memory, and on a four-card machine that came to 48 child processes. The second finds the best number of parallel reads for your disk. Both ran before the first answer, and the chunk-size search ran for every card model in the machine. The table shows the effect: the more cards, the longer the first launch.

Schema: where 20.6 minutes of one four-card first launch went: 17.0 minutes in 48 chunk-size probes, 2.2 minutes measuring the disk, 1.4 minutes for everything else.

What is already fixed on main:

  • First serve now plans like run. On a fresh home it measures the card's runtime total and context before planning, through the same launch path as run and upscale (4b95b21). On a machine with mixed cards, each card model is measured on its own card (590105a).
  • The weight reader now judges itself honestly.
    • It finds its best number of parallel reads on the load it is serving, with no trial run first, and starts from the width an earlier load settled on (4058bc7, 806fd04).
    • It used to over-rate a wider width, because reads still in flight from the narrower width were counted in full. It now judges over a window of reads: a test store that saturates at 4 readers no longer grows to 8 (709ba0a, eb8e6cd).
    • A window closed by the end of the load could read up to 10% high, so such windows are no longer judged (d1af928, f87f53d).
  • A crash in the memory pool. A block taken with the pool switched on and freed with it switched off went to the driver alone. The driver refused it or released the whole segment, and the next write to a neighbouring segment crashed. The block now goes back to its pool whatever the switch says. In a child-process test, it crashed 3 times out of 3 before the fix and passed 3 times out of 3 after (a8d1134).

What is not fixed yet. A cure for the long first launch exists and is measured, but it is not on main yet. We will give its figures when it is landed and released, not before. One first-launch path on a fresh install also ends in an error (a file-path fault in a probe). We have found it, and it is not fixed yet.

5. The rest of the list

Vision-language models

  • A text-only question to a vision-language container now gets an answer from its language model alone, on both engines (6bca009, 7f41f85).
  • That request is now priced only for the towers it carries an input for (c9a2322).
  • The speech output of a vision-language model now receives the rows of the whole speech span, instead of the last row alone (37f5b0c, 164cc6f).

Video

  • Serving a video container used to stop with MissingRuntimeValue num_frames. It now plans with the frame count and temporal compression that the container declares (b528be9).
  • A container that declares no frame count, step count, guidance scale or frame rate is now refused by name, instead of running on a default from a family file (002125d, 8c138e7). The value belongs in the container, not in the engine.

Installing

  • Windows. A native-Windows install did not resolve. The engine now asks for Triton only where PyPI publishes a wheel for it (Linux x86_64 and aarch64) (531b700, dc42b1c). On Windows, CUDA_VISIBLE_DEVICES is read as on Linux (25bff62).
  • Missing requirements are named before anything loads.
    • Without Triton, run and serve in a Triton mode refuse by name: they either say that PyPI publishes no Triton for your platform, or give you the install line (19a1d45).
    • A torch older than 2.12.0 is refused before any weight loads (4ae33cf).
    • A Triton older than 3.5.0 is refused by name; before, it failed with an AttributeError at the first launch (b68d851, fde70b9).
  • No C compiler. A machine without a C compiler, whose Triton cache already holds the driver helper, is now accepted. A missing compiler is named before the first build (3fe56ef, e41f6ef).

6. How every model is judged now: from the command line, by its output

A test suite proves that a function does what its test says. It does not prove that a model answers well. So before a version ships, every model in the catalogue is judged the way you will use it, from the command line, and its output is checked by reading, looking or listening:

  • A language model is started with neurobrix serve and given a chat of three to four exchanges. The answers are read, and they must be coherent.
  • An image model is run, and the picture is looked at.
  • An audio model is run, and its output is listened to or transcribed.
  • Every other model is started with neurobrix run, and its output is read.

This check runs again on the exact engine and library pair that will ship. A model we have not judged this way is not called working. Three models have an open fault whose cause we have not found yet. If you run one of them, this is what you may see:

  • CogVideoX-5b-I2V: the video is not your scene. After the first frame, it becomes a blur with a grid pattern.
  • Allegro-TI2V on the Triton branch: the picture fills only the top-left part of the frame, and the rest is flat grey.
  • Ming-Lite-Omni-1.5: requests with an image, with audio, or for speech output end in a server error (HTTP 500) instead of an answer.

They are not listed as working until the cause is found and fixed. These are the results of our latest check, which is now being redone on the final pair.

7. Still on the bench before the next version

These points are open, and none of them is done:

  • Kernel certificates. The same-answer fix changed the bits of several kernels, so all 70 shipped certificates (NVIDIA and Apple) must be proven again on the new library.
  • Tuning after the upgrade. After upgrading, each machine tunes its kernels once more at its first launch. This is expected, and the release note will say so.
  • The first launch. The cure for the long first launch, and the fix for the probe's file-path fault, are not landed yet (section 4).
  • The test suites. Every test suite must be green on main. Today 14 engine tests are not.
  • Conversation context with run. Models that need conversation context outside chat will keep it with neurobrix run. Chat stays for language models.
  • The final check. The catalogue check from the command line (section 6) runs again on the final pair.

The next version ships when this list is closed.

If you ran into one of these faults, thank you: some of this list exists because you told us. If you hit something that is not here, open an issue with your card and the command you ran. We answer every one. And if you want to follow the work as it lands, a star on GitHub is the best way to keep NeuroBrix on your radar.

One engine for every model, on the hardware you already own. Let's build what comes next, together.