Google unveils inference engine handling 6 million requests per second... removes Python from servers

Google open-sourced its XLA Serving Engine (XSE) under the Apache 2.0 license on November 5, a C++ inference runtime that executes models exported from JAX or PyTorch as XLA SavedModels without a Python execution environment in the serving path. The engine, part of the OpenXLA ecosystem, separates model development in Python from production serving, handling user requests through the C++ runtime. Google claims XSE can process up to 6 million inferences per second, positioning it as a high-throughput alternative to Python-based serving stacks. The release matters because Python overhead in inference serving has been a persistent bottleneck for latency and scaling, and a permissively licensed Google-backed runtime could become a standard for OpenXLA deployments. What to watch next is adoption by major inference providers and framework maintainers, plus independent benchmarks verifying the 6 million-per-second throughput claim.