AI & ML
Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell
arXiv cs.IR0 views
arXiv:2611.21974v1 Announce Type: new Abstract: Low-bit LLM quantization is useful only if a model preserves service-relevant accuracy, executes through supported GPU kernels, and meets measured memory and latency limits. We study Qwen3-8B on NVIDIA B200 with vLLM, including INT2/INT3, GPTQ, NVFP4, and mixed-precision configurations. We verify native packed execution and precision on all 144 fused linear modules and evaluate Korean, English, numerical, and reasoning quality under preregistered gates. Packed INT2 and INT3 execute, but all seven tested INT2 grids degenerate on generation. Isolated module-restoration effects are strongly non-additive; Korean and reasoning sensitivity rankings disagree, and a promising INT4/INT8 development candidate fails independent final evaluation. A configuration with 16-bit activations, 98 NVFP4 weight-only modules, 46 INT8 modules, and an INT8 output head passes all gates on the untouched FINAL-81 set (5,168 items): 5.917 GiB of weight-loading memory (limit 5.98), Korean teacher-correct retention 0.904 (gate 0.90), and TPOT ratio 0.939 relative to BF16. It improves paired accuracy over NVFP4 W4A4 by +1.4 percentage points (Bonferroni-adjusted interval [+0.2, +2.6]), compared with +3.4 points on the selection set. This comparison changes both activation precision and INT8 protection; exploratory analyses suggest that the activation change contributes substantially, but the isolated allocation effect is not established. Final checkpoints were not retained, and bitwise rebuild reproducibility remains unverified.
Source:https://arxiv.org/abs/2611.21974PreviousWhat RL on Expert-Built Tasks Teaches Nemotron 3.5 SuperNextSIMQ: A Single Graph-based Index for Multi-Vector Query
Related posts

Preparing for a post-quantum future
0TechRadar

75+ best Black Friday deals 2026 in the UK — GTA 6, Kindles, Switch 2, Shark vacuums, LG OLED TVs, and more
0TechRadar

Why no 40fps mode for GTA 6? — Rockstar says the ‘complexity of simulating and rendering our world’ requires it to ‘carefully balance how we spend every millisecond of both the central processor and t
0TechRadar

'Enterprises need to connect AI to the right data': Report finds over half of firms have abandoned AI projects over unclear goals
0TechRadar

$3.26 million in SpaceX AI data center revenue will be pumped back into the City of Memphis
0Tom's Hardware

The Downfall of Jensen Huang — a timeline of allegations and scandals that brought down the king of AI infrastructure
0Tom's Hardware