AI & ML
SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding
arXiv cs.DC0 views
arXiv:2611.22296v1 Announce Type: cross Abstract: Speculative decoding (SD) accelerates large language model (LLM) inference, but its optimal configuration depends jointly on workload characteristics, GPU hardware, and available memory. For example, on the RTX~5090, the state-of-the-art speculative drafter DFlash achieves $3.50\times$ the throughput of autoregressive (AR) decoding at low concurrency, but only $0.76\times$ at high concurrency with short outputs. We present SpecSearch, a hardware-aware search engine that jointly optimizes decoding strategy, draft length, and precision across workload--GPU pairs. Its analytical cost model captures memory and compute constraints to identify feasible configurations and narrow the search space for throughput optimization. Guided by this model, calibration measurements are collected from multiple GPUs. Using the measurements, a Jev-based predictor learns to select high-throughput decoding configurations on held-out GPUs without requiring deployment-GPU calibration. The predictor is parameter-efficiently trained on calibration data, while adapting a frozen Qwen3-4B using LoRA adapters and a linear correction layer. Across NVIDIA RTX~5090, H100, L40 and AMD MI300X GPUs, SpecSearch with full calibration achieves an average of $2.01\times$ the throughput of AR and $1.36\times$ the throughput of SD, both served by vLLM. The proposed Jev-based predictor closely matches this search performance with $1.30\times$ throughput improvement over SD with vLLM, while eliminating the need for held-out GPU calibration. Additionally, it retains $97.36\%$ of the throughput achieved by SpecSearch that utilizes full calibration.
Source:https://arxiv.org/abs/2611.22296PreviousCommunicate the Change: Efficient Expert Parallelism via Activation CompressionNextJigSLO: Solving the Resource Puzzle for Efficient Multi-LLM Serving
Related posts
Hydra · Coeziune OM–IA — the complete DOI registry (2026-11-18): 76 published works
0dev.to OpenSource
Your Rust Code Sets an Environment Variable and the 2024 Edition Calls That Unsafe
0dev.to Rust
Your Go Function Returned a Nil Error and the Caller Saw an Error Anyway
0dev.to Go
Kengele: an alarm for when an activist goes missing, that nobody can switch off
0dev.to Go
How OpenAI scales PostgreSQL to 800 million users
0dev.to Database
Spec-Driven PBT: Deriving Semantic Invariants from Code Specifications for Agentic Property-based Testing
0arXiv cs.SE