My sub-second sprite judge can't tell a profile from a 3/4 view

I run a pipeline that draws 2D fighting-game characters with image and video models, then checks every drawing before it ships. A check that fails means paying to redraw. For a while the checks went to a cloud vision model, which takes about 10–15 seconds per call. So I wrote a NumPy-only "pixel judge" that answers in under a second. Then I found the catch. On side-view takes, the expensive model still gets called almost every time, because the pixel judge can't tell a true profile from a three-quarter view.

This post is the research I did before fixing it. Nothing below is built yet, and I have no new benchmark numbers. It's a plan with sources, and I'd like you to poke holes in it.

The setup

Each fighter starts as a full-body front painting. From that front, the pipeline draws a side view, then 11 motions (idle, walk, run, attacks, block, hit, fall, and so on). Two kinds of judgement happen along the way:

• Side take: is this new drawing the same character, seen from the side, with whole limbs?
• Loop: do the walk and run cycles actually alternate legs and loop cleanly?

The worker supports three judge modes:

hybrid is meant to be the fast-and-safe default. It isn't fast for side takes.

Where the pixel judge goes blind

Reading the code made the problem obvious. The pixel side-check always returns side: unknown. Colour regions, silhouettes and frame differences can't separate "face pointing left" from "face turned 45° toward the camera". So in hybrid, every side take that passes the other checks escalates anyway, and the time saving there is close to zero.

In pixels mode it's worse, because unknown means pass. That lets through:

• three-quarter takes labelled as side views
• face changes
• parts that change size between frames (the gloves shrink mid-walk)
• wrong limb shapes

Two other judgements never touch pixels at all: the front check (exactly one character, whole from head to soles, facing the viewer) and the look judge. Both always go to the cloud model.

The thresholds that do exist came from 6 probe fighters and 20 verdicts (8 takes, 12 loops). Worse, the work folders are deleted after 7 days, so verdicts never pile up into anything I could measure against.

Research A: small local models for anime art

I wanted CPU-friendly models, ideally ONNX, with permissive licences.

• deepghs/imgutils (MIT): anime person, head and face detectors, plus CCIP, a contrastive model for "is this the same character?". The detectors are tiny (a few million parameters). Some of its models and weights carry OpenRAIL licences, so check each one.
• WD14 tagger v3, SwinV2 (Apache-2.0, ONNX). Its tag list has exactly the vocabulary I need: profile, from_side, facing_viewer, solo, full_body, multiple_views, feet_out_of_frame, head_out_of_frame. I didn't find a three-quarter-view tag, which matters below.
• rtmlib (Apache-2.0): RTMPose and DWPose on ONNX Runtime, no mmcv. It was trained on photos (COCO), so its accuracy on anime limbs is an open question.
• DINOv2-small (Apache-2.0, 22M params) for appearance similarity. The DreamBooth authors reported that DINO similarity tracks human judgement better than CLIP image similarity (about 0.32 vs 0.27 correlation).

The honest gap: I found no validated model that estimates the yaw (left-right turn) of an anime face. So profile-vs-3/4 has to come from several weak signals combined:

1. WD14 scores for profile, from_side and facing_viewer
2. where the face box sits inside the head box, and whether a face is detected at all
3. optionally, facial landmarks: is only one eye visible?

Research B: a small local VLM, used carefully

For machines with a GPU, a small VLM could sit between the pixel judge and the cloud. Qwen3-VL-4B Instruct (Apache-2.0) is about 3.3 GB on Ollama. It should fit an 8 GB card, but I found no published latency on an RTX 4060, so I'll have to measure it.

The bigger lesson was how to ask. Techniques that make VLM judging steadier:

• Atomic yes/no questions. Split "is this a valid side view?" into small checks (Davidsonian Scene Graph).
• Read the probability, not the word. Score with P("Yes") from logprobs, as VQAScore does, then put a threshold on a continuous number.
• Ask twice, swapped. When comparing two images, ask in both orders to cancel position bias.
• One grid image instead of many. Tile frames into a single image (IG-VLM).
• Cap image tokens with max_pixels so latency stays predictable.

Research C: a cascade with calibrated thresholds

My pixels → cloud VLM setup already is a judge cascade. It just has hand-picked thresholds. Trust or Escalate formalises it: a cheap judge decides when it's confident, and escalates to a stronger judge otherwise. The confidence threshold is calibrated so that agreement with humans on the non-escalated cases meets a level you choose.

Conformal risk control and Learn-then-Test can calibrate several thresholds at once. But with 20 calibration examples any guarantee is very loose. A realistic start is 50–100 labels per check, growing to hundreds. The labelling order is cheap: label the cases where the judges disagree first.

The plan, in priority order

1. Keep every verdict. Copy each pixels.json (with its measurements), each cloud verdict and a small thumbnail to an archive outside the 7-day cleanup. No user prompts included. Add an eval script that replays a labelled set and prints, per check, a confusion matrix (pixels vs VLM vs human), the escalation rate, and p50/p95 latency. Everything else depends on this.
2. Make side decidable. Add a models stage (WD14 + head/face boxes, maybe eye landmarks). It answers pass/fail only when confident and otherwise stays unknown. This has the biggest speed payoff.
3. Identity and part sizes. Add CCIP or DINOv2 cosine similarity next to the colour comparison, so a matching palette with a changed face or outfit gets caught. Track the variance of head-box height over body height across frames for the shrinking-glove case.
4. Pre-screen the front check locally. Exactly one person, solo + full_body, none of multiple_views / feet_out_of_frame / head_out_of_frame, and facing_viewer. Only obvious fails end locally. Anything that looks like a pass still goes to the cloud model, because this is a registration gate.
5. Optional GPU tier. Qwen3-VL-4B with atomic questions and P(yes), only after steps 1–4 and only if its measured latency and accuracy earn it.
6. Set thresholds from data. Recalibrate per check with the Trust-or-Escalate method as labels grow. Re-test the rule "a pixel fail is final", because a false fail costs a paid redraw. Until then, label current thresholds as uncalibrated.

Constraints I'm keeping: new models live in a separate host-side venv, away from the generation engine. Model versions and hashes go into the judge cache key, so changing a model marks old verdicts stale. MIT/Apache licences come first, OpenRAIL only after reading the terms, and no GPL/AGPL models.

Honest limits

• None of this is measured yet. The 10–15 s and under-1 s figures are what the current judges take. Every gain above is a hypothesis until step 1's eval set exists.
• Profile vs 3/4 will probably stay partly unsolved. Without a yaw model, the best case is "confident on the clear cases, escalate the rest".
• Photo-trained pose models may simply be wrong on chibi or stylised anatomy.
• A local VLM only helps machines with a GPU. CPU-only generation PCs skip that tier.

What I'd love to hear

Have you found anything that reliably estimates head yaw on anime or cel-shaded art? Or have you calibrated a cheap-judge → expensive-judge cascade in production? How many labels did it take before the thresholds stopped moving?