Training a model gets the glory. Serving it — answering millions of small requests every second — is where the bills pile up. Google has now open-sourced one of its sharpest answers to that problem: XSE, the XLA Serving Engine, a runtime that takes the Python interpreter out of the serving path completely.

The announcement landed on the Google Open Source Blog on November 5, from Oleg Shaldybin and Monica Song of Google's TPU software team. XSE is a general-purpose model export and inference runtime for the OpenXLA ecosystem, released under the Apache 2.0 license. And this is no research toy: inside Google, XSE already serves over six million queries per second, powering inference for Search, Google Ads, YouTube Shorts and parts of Gemini Omni.

Train in Python, serve in C++

Google sums up the workflow in one line: "Train in Python, Serve in C++." Engineers keep writing model code in Python, usually with JAX handling the maths. When a model is ready, XSE's export API packages it as an "XLA SavedModel" holding the model's function in StableHLO, an intermediate representation the XLA compiler can work with. The XLA Serving Runtime then loads that package and runs it as compiled native code, with no interpreter sitting between the request and the answer.

Why does cutting Python out matter so much? At Google-scale request volumes, the interpreter itself becomes overhead: extra memory, extra CPU, latency that wobbles when you least want it to. A compiled C++ runtime carries none of that baggage. Google says it began deploying core components of XSE inside its production serving infrastructure last year, so this release is less a debut than a handover — Google is giving away a tool it already trusts with some of the highest-volume inference pipelines on the internet.

The code and quick-start instructions live in the OpenXLA xla-serving repository on GitHub, and the runtime targets CPUs, GPUs and TPUs. PyTorch users are not left out for long, either: Google plans integration with TorchTPU, so PyTorch models can be exported for C++ serving on TPUs as well.

The benchmarks — with the usual caveat

How fast is it? In Google's own tests on short requests, XSE delivered two to five times the throughput of the best optimized Python server the company could build, while using a quarter of the memory. Read those numbers with the standard caution: they are vendor benchmarks, run on Google's workloads under Google's conditions, and nobody outside Google has replicated them yet. Still, removing an interpreter from a hot path almost always buys speed — and usually memory headroom too.

Nathan Beach, a Google director of product management, described the target workloads as short requests with very high queries per second and tight latency budgets: recommendation rankings refreshed on every swipe, ad scoring decided in milliseconds, vision models checking images as fast as users can upload them. Not glamorous jobs, but they never stop, and they never get to be slow.

Not a chatbot engine — and that is fine

Some headlines will skip this part: XSE is not built for large-scale LLM inference. If you are serving a giant language model, Google itself points you towards vLLM or SGLang instead. XSE is a lightweight, one-shot inference engine aimed at the other end of the spectrum: smaller models, sharply defined tasks, one prediction per request, relentless volume.

How much of the world's AI traffic actually looks like that? Most of it. Every Shorts swipe, every ad auction, every search ranking call is a small model answering a small question under a brutal deadline. The chatbots get the keynotes, but this quiet layer is where AI turns into infrastructure.

TPU Sync arrived the same week

XSE was not the only piece of Google's serving stack to go open. Google Cloud has also open-sourced TPU Sync, a native C++ library that moves model state around a TPU cluster — shuttling KV cache blocks and model weights between TPU high-bandwidth memory, host DRAM, shared storage and the data centre network, treating all four as a single memory hierarchy.

According to Google, TPU Sync cuts KV transfer latency by over 90 percent and reduces time to first token by an average of 90 percent for heavyweight open-weight MoE models in the 235B to 480B parameter range, in prefix-caching and disaggregated-serving workloads. It also plugs into vLLM and SGLang as a TPU KV cache connector. Those are Google's own claims from its own testing, not independently verified results — treat them as a claim to test rather than a fact to quote. The timing is not subtle, either: the library landed alongside the new hardware we covered in our look at Google's TPU 8tx and TPU 8i Max.

Will the bet pay off? Too early to say. But the code is public, the license is permissive, and the production pedigree is unusually concrete for an open-source launch. Six million queries a second is not a slide in a keynote. It is a Tuesday at Google — and now the engine behind it belongs to anyone who wants to read it, run it, or argue with it.

Read the full version of this story on Rohan Tech Official: Google Open-Sources XSE, the Serving Engine Behind Six Million Queries a Second