Ray Serve LLM on GKE: 5x Throughput, 8x Lower Latency

Developers love Ray Serve for its Python-native, developer-friendly APIs. But when you pair it with Google Kubernetes Engine (GKE), you get a powerhouse for LLM inference. The only catch? Performance used to take a hit. Not anymore. Google and Anyscale have teamed up to deliver up to 5x higher throughput and 8x lower latency—without sacrificing that sweet developer experience.

What’s New Under the Hood?

Three major optimizations make this possible:

• HAProxy integration: Ray Serve now uses HAProxy for internal routing and load balancing, slashing proxy overhead and keeping Python runtimes from choking under traffic.
• Direct token streaming: Tokens stream directly from model replicas back to the proxy, bypassing the ingress router for the streaming path. Less hops, less latency.
• v2 Ray executor for vLLM: A revamped backend moves Ray out of the data plane, enabling async scheduling and closing the performance gap with native vLLM executors.

Benchmarks That Speak Volumes

Testing on GKE clusters with NVIDIA HGX B200 hardware and Gemma 4 E2B models, the new Ray Serve LLM showed up to 5x higher throughput and 8x better latency compared to previous versions. It even rivals plain vLLM setups—but with all the flexibility Ray brings.

With more concurrent users, Ray now scales throughput while keeping 99th percentile time-to-first-token low. No more trade-offs between features and performance.

Why GKE?

GKE provides the foundation that makes these optimizations shine. The Ray Operator add-on gives you turnkey deployment across Google Cloud’s AI accelerators, with automated scaling, monitoring, and fault tolerance. It abstracts the messy orchestration so you can focus on your models.

If you’re already using Google Cloud, this is a no-brainer. And if you’re comparing options, check out our Vercel and Supabase reviews for other serverless paths.

Try It Yourself

These enhancements are available in Ray 2.56 and later. Dive into the resources below to get started:

• Anyscale blog post
• High Throughput on Ray Serve with KubeRay
• Multi-cluster Ray Serve with GKE Inference Gateway
• Serve Gemma on GKE with Ray

Why this matters for platform teams

Ray Serve’s historical trade-off was ergonomics versus raw throughput. It gave you a Python-native way to wire up multi-stage inference pipelines — tokenization, a retrieval step, the model call, then post-processing — without leaving the framework or bolting together separate services. The cost of that flexibility was overhead: requests passed through Ray’s serving layer rather than straight to the engine, and for single-model, high-volume LLM workloads that gap showed up against plain vLLM. Closing it is the real story here. You no longer have to choose between Ray’s orchestration and the throughput numbers your capacity planning depends on.

“Moving Ray out of the data plane” is the conceptual core. Picture two paths through your system: the control plane decides where a request goes and how the cluster is shaped, while the data plane is the hot path each token actually travels. When a general-purpose proxy sits in the data plane, every streamed token pays a tax on its way back to the caller. That tax is nearly invisible for a single batch response but compounds badly for token-by-token streaming, where it inflates time-to-first-token and tail latency. Routing internally through HAProxy and streaming tokens directly keeps Ray’s scheduling intelligence in the control plane and gets it off the per-token path.

When is Ray Serve the right tool? Reach for it when you’re serving compound or multi-model systems, need Python pre- and post-processing inline, or are juggling heterogeneous accelerators behind one endpoint. If you’re serving a single model with no surrounding logic, plain vLLM is still the simpler answer and you should not add an orchestration layer you won’t use.

A caution on the headline figures. Throughput, TTFT, and p99 latency pull against each other — aggressive batching lifts throughput while pushing tail latency up, so no single number tells you whether a setup fits your SLOs. Read them together. And benchmark conditions are not production: results shift with model size, hardware, batch configuration, and traffic shape. Treat the numbers as a ceiling worth chasing on your own workload, not a promise. Like a capybara picking its spot in the swamp, match the tool to the water you’re actually in — then measure.

Original announcement published on Google Cloud.