Two days ago this model wouldn't even export. Today its vision tower runs on a real RK3588 board: a 392×392 image encoded in 4,445.68 ms (median of three independent processes), model load about 7.3 seconds, zero RKNN errors at runtime. This is a field note from the conversion chain — what broke, what the fix actually was, and what the number does and doesn't mean.

Where it started: the official demo doesn't run on the transformers it targets

Rockchip's official RKLLM multimodal demo ships a Vision export script that assumes an older transformers contract: model.visual, and a rotary_pos_emb argument handed to each block. On transformers 5.8.0 the script dies with an AttributeError — .visual has moved to .model.visual. On 4.57.0 it dies differently — a TypeError, because VisionAttention unconditionally unpacks position_embeddings and gets None.

We bisected the versions by reading source, not by guessing. The combination that works is transformers 4.51.3: it still has self.visual, and when position_embeddings is None it derives it from rotary_pos_emb. That compatibility was removed in 4.54. The fix is version pinning — nothing more. We tried a .visual alias shim first; it got past the attribute error and died on the attention contract anyway. A shim that passes the first error and silently produces wrong embeddings would have poisoned every measurement downstream, so it went in the bin.

One more wrinkle worth naming: the same repository writes its quantization/embeddings script against the 5.x structure (model.model.language_model), while the Vision export needs the 4.x structure. Half the repo is new, half is old. That's the kind of thing you only find by running it.

The chain, step by step

  1. Vision → ONNX (transformers 4.51.3, torch 2.6.0): the exported graph is 53 MB, plus a 2.66 GB data file. onnx.checker passes.
  2. ONNX → RKNN (RKNN-Toolkit2): the built model is 1.44 GB. The build log carried 11 E RKNN lines — REGTASK bit-width warnings and two "Unknown op target: 0" — and still finished with rknn building done.
  3. Board (RK3588, 3 cores): demo.jpg, input [1, 392, 392, 3] NHWC FP16, output [196, 1536] FP16. Three independent processes: 4,442.44 / 4,445.68 / 4,488.95 ms — median 4,445.68 ms. Model load median about 7.3 s. And the part I cared about most: zero E RKNN lines at runtime. The 11 build warnings did not reproduce on the board.

What 4.45 seconds is — and isn't

It is the vision tower only. The language half of this model — the RKLLM export that turns the whole thing into a runnable .rkllm — is still waiting on a GPU machine that went offline mid-download. So this is not end-to-end question-answering speed, and I won't dress it up as that. The full 24-question run (the demo's own 20 MMBench questions plus 4 of our own in Chinese) is the next wave, once the language model lands.

What the number does say: the vision tower is real, on-device, and reproducible to within ~1% across processes. The conversion chain that looked broken two days ago is one missing half away from a working on-board VLM.

The rule to take home

Pin the version; don't patch upstream. When a vendor demo and a library disagree, the fix that survives is the boring one — a version that satisfies both contracts — not a shim that makes the error go away and leaves you wondering whether the numbers are real. And measure the tower before you claim the model works: a 4.45-second encode is a fact you can build on; a "VLM runs on RK3588" headline, at this stage, would be a wish.