Open Source
BatchGen: An Architecture for Scalable and Efficient Batch Inference
arXiv cs.DC0 views
arXiv:2606.21712v2 Announce Type: replace Abstract: Batch inference has become a central mode of AI computation, yet existing inference engines still rely on execution models designed for interactive serving. When scaled to millions of sequences, batch workloads reveal two fundamental requirements: the ability to handle extreme inter- and intra-sequence load variation that emerges only at runtime, and the ability to sustain high utilization across large fleets of GPUs. Existing systems fail to meet these requirements, losing substantial fractions of achievable throughput. We introduce a new architectural foundation for batch inference: the sequence coroutine compute model, which represents each sequence as a fine-grained, event-driven coroutine. This model exposes expressive primitives that allow the runtime to reorganize work dynamically, enabling larger expert-level batches, mitigating stragglers, reallocating work across devices, and maintaining utilization even on cost-effective or memory-constrained GPUs. Building on this abstraction, we implement BatchGen, a production-ready system that uses the coroutine model at cluster scale. On a 128-GPU cluster, BatchGen reduces batch completion time by up to $2.3\times$, and on memory-constrained accelerators it outperforms the strongest offloading baseline by up to $9.6\times$. We will open-source BatchGen at https://github.com/batchgen-project/batchgen
Source:https://arxiv.org/abs/2606.21712PreviousDemystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU CommunicationNextProgrammable Probabilistic Computer with 1,000,000 p-bits
Related posts
Extracting Japanese addresses and refusing to guess the municipality code (span F1 0.998)
0dev.to
ASUS Expands TUF Gaming Series in India With New F16 and A15 Laptops, T700 Mini Desktop
0FOSS Bytes

How to Set Up a Self-Service Support Portal for Your Community with SeaTicket
0dev.to Tutorial

Tantou (担当) v1.0.0 — Open-source (AGPL-3.0) Human Capacity OS & real-time request dispatch
0dev.to PHP
YUL: Bringing Software Supply Chain Security to Coding Agents
0arXiv cs.SE
Signature-Kernel Based Evaluation Metrics for Robust Probabilistic and Tail-Event Forecasting
0arXiv stat.ML