Originally published on hexisteme notes. I had a 27B vision model running locally (IQ4_XS quantized, 15GB resident) and needed to decide whether it was worth using for OCR. The comparison was macOS's built-in Vision framework (VNRecognizeTextRequest) — a dedicated text-recognition engine, free, zero memory footprint. My expectation going in: the specialist wins on character accuracy, and the general-purpose model is reserved for when you need semantic understanding. Slow and expensive, use sparingly. That expectation was wrong, and it was wrong in a way that would have been invisible in production. Method: an image whose answer I already knew The usual mistake in an OCR comparison is measuring against real documents, where you don't have ground truth. Then you can't distinguish plausible output from correct output — and plausible output is exactly what both engines produce when they fail. So I rendered a 1100×720 test image with the answer fixed in advance: A title and date in Korean, a 4-column × 3-row table (model / memory / speed / status), four lines of prose One adversarial line: A0-1lO9-B8 — digit 1 next to lowercase l, capital O next to digit 0 Two empty table cells containing - Then I looked at it. The first render was wrong — a label came out as tofu boxes (□□), because the monospace font had no Korean glyphs. If ground truth is broken at the moment you fix it, every measurement afterwards is void. That check costs thirty seconds and it's the whole experiment. The results Local 27B VLM Apple Vision (dedicated) Character errors 2 8 Reading order preserved destroyed Table cells dropped 0 2 (the - cells) Wall clock 82.8s (cold) 0.27s Vision's eight errors: IQ4_XS→I04_XS, cloud→cLoud (twice), tok/s→tok/5, two characters inside a code string, em dash —→-, arrow →→->. 300× faster. On accuracy alone, four times the error rate on a document of this size is arguably a fine trade. Accuracy alone is not what decided it. The difference was structural, not lexical Vision returned the table decomposed by column. Three model names in a row, then three memory figures, then the speed and status columns appended at the end of the document. Which means: you cannot recover which speed belongs to which model from the output. The row associations are gone. Not garbled — gone. The characters are all there, correctly grouped, in a well-formed sequence, and the relation between them has evaporated. The VLM kept the rows. Next to that, 2 errors versus 8 is a rounding difference. What this actually says is narrower than "the general model is better": "A dedicated tool beats a general one" depends entirely on where you cut the task. Vision is dedicated to character recognition. It is not dedicated to document understanding. My task needed the second and I was picking tools by the first one's benchmark. The specialist was genuinely better at the thing it specializes in — I had just mislabeled what I needed. Why this breaks the cheap-first fallback The obvious architecture is: run the cheap engine, detect failure, escalate to the expensive one. Almost everyone reaches for this. It requires failure to be detectable. Column-shredded output is syntactically perfect. It has plausible text, plausible structure, no error signal of any kind. Downstream, it is indistinguishable from a correct read. The information didn't get corrupted — it got dropped, and dropped information leaves no residue to detect. This generalizes past OCR. Any escalation ladder — cheap model then expensive model, cache then origin, heuristic then solver — is only sound when the cheap tier's failure mode is observable at the boundary. If the cheap tier can fail by silently discarding a relation rather than producing a wrong value, "cheap first" isn't an optimization. It's an undetected data loss path with a cost saving attached. Both engines failed in exactly the same place The adversarial string A0-1lO9-B8: Attempt Output VLM, full image A0-1109-B8 — 2 misreads VLM, that line at 4× with an explicit "distinguish 0/O and 1/l" instruction A0-1l09-B8 — recovered l, still lost O Vision, on the enlarged crop A0-1109-B8 — 2 misreads, unchanged I inspected the enlarged image myself. The font renders digit 0 with a slash through it and capital O as a plain oval. The two glyphs are visibly different. This isn't image ambiguity that more pixels would resolve — it's both engines genuinely misreading a distinguishable character, and resolution doesn't touch it. So: No OCR engine can be trusted on strings where homoglyphs change the meaning — codes, IDs, hashes, addresses, license keys. That's not a tool-selection problem. It's a property of the entire tool class, which means the remedy isn't a better engine. It's human confirmation or a checksum. If you're about to build an OCR path for identifiers, build the checksum first. What I adopted Vision as the first pass; escalate to the VLM only for documents where reading order carries meaning — tables, forms. With one correction to that rule, from the paragraph above: since column-shredding isn't detectable downstream, "escalate on failure" doesn't work for tables. If the corpus is mostly tables, go to the VLM first and eat the 300×. Cheap-first is only valid when failure is visible. What would change my mind If the target documents are mostly prose, Vision alone is sufficient and the VLM is a 300× waste. If Vision gains layout analysis and starts preserving table structure, this verdict is dead. The homoglyph failure is common to both engines, so improving one doesn't touch that part of the conclusion. More notes at hexisteme.github.io/notes.