llm-d 1.0 Brings Agent-Aware Routing to Kubernetes AI Inference

llm-d, the open-source system for running distributed AI inference on Kubernetes, released version 1.0 on Tuesday at PyTorch Conference North America. The release stabilizes key routing APIs, adds controls for sharing accelerator fleets among competing workloads, and gives operators a coordinated way to update systems that split prompt processing from token generation.

The project is aimed at organizations serving open-weight models through engines such as vLLM and SGLang. Its central idea is that model replicas cannot be managed like ordinary stateless web servers: the cheapest and fastest place to send a request often depends on where its previous context is cached, which accelerators are saturated, and whether the request came from an interactive user or a bursty autonomous agent.

Google, IBM and Red Hat announced the milestone with a community that now spans more than 500 contributors at 25 organizations. llm-d was launched in 2025 by those companies alongside CoreWeave and Nvidia, and became a Cloud Native Computing Foundation Sandbox project in March. Version 1.0 is a software-maturity milestone, however, not a promotion beyond CNCF’s entry-level Sandbox stage.

Why agent traffic changes model serving

A conventional load balancer can send each request to the least busy replica or rotate through a pool. That approach misses an expensive property of large-model inference: a replica may already hold the key-value cache created while processing the system prompt, codebase, conversation history or other shared prefix for a session.

Sending the next turn elsewhere can force the new replica to process that context again. Long-running agents amplify the problem because they repeatedly return to the same instructions and working material, fan out into parallel tool calls, and then arrive back at the model in bursts.

llm-d puts inference-aware routing and flow control in front of model servers. Its router tracks cache locality, load and predicted latency; it can keep a multi-turn session near its existing state while still enforcing priority and fairness across tenants. When accelerator memory is tight, supported connectors can move cache data from high-bandwidth accelerator memory into host RAM or storage rather than immediately discarding it.

The stack can also separate prefill, the compute-heavy work of reading a prompt, from decode, the memory-bound process of generating tokens. Those phases can run in independently sized pools. For mixture-of-experts models, llm-d supports spreading experts across multiple accelerators, while batch and asynchronous paths handle work that does not need an immediate interactive response.

What is actually new in llm-d 1.0

Many of those inference techniques existed before this release. The 1.0 change is chiefly about making them work together under ordinary production pressures: several tenants competing for capacity, software upgrades, replica failures, traffic spikes and mixed service-level targets.

The project’s 1.0 release notes identify four main changes:

• Multi-tenant flow control: priority- and fairness-aware admission can hold back or shed lower-priority work when a fleet is overloaded, instead of allowing a burst of background agent calls to crowd out interactive requests.
• Coordinated, zero-downtime updates: router high availability adds hot standby behavior, while DisaggregatedSet manages prefill and decode replicas as one logical service so matching versions can roll together.
• Stable API surfaces: the router configuration API graduates to v1. The EPPConfig and InferenceObjective resources and core endpoint-picker plugins now have stable configuration surfaces.
• Agent-aware state: session affinity, turn-priority scheduling and support for previous_response_id in the Responses API are intended to keep repeated agent turns close to cached context.

The release also aligns most core llm-d components at version 1.0.0 after earlier releases used a collection of different version numbers. The router endpoint picker, disaggregation sidecar, coordinator, simulator, autoscaling service, batch gateway, latency predictor and benchmarking tools are among the components in the aligned matrix.

For observability, the router adds end-to-end W3C trace propagation, OpenTelemetry tracing in the coordinator, Prometheus trace exemplars and telemetry for tool calls. Those details matter because an aggregate latency graph cannot explain whether a slow response was queued by flow control, missed its cached prefix, crossed a storage tier or waited on the wrong phase of a disaggregated deployment.

The performance claims need local testing

Google’s release write-up includes several substantial results. On an agentic coding workload using Ironwood TPUs, the company reports that prefix-aware routing with host-memory offload cut median time to first token by more than 85% and increased throughput 2.4 times compared with a plain Kubernetes Service. It also reports that an internal production platform doubled its prefix-cache hit rate from 35% to 70% and reduced time to first token for coding agents by more than 35%.

Those figures demonstrate why inference-aware scheduling can matter, but they are not a general forecast for every cluster. The benefit depends on repeated prefixes, prompt length, concurrency, cache capacity, accelerator topology and the cost of moving cache blocks between memory tiers. A service dominated by unrelated short prompts may see far less cache reuse than a coding agent that revisits the same repository for dozens of turns.

The project provides an open benchmark harness and the Prism dashboard so teams can inspect configurations and compare their own throughput, latency and cost curves. That is the right place to begin: reproduce a real traffic mix before changing production routing or buying capacity around a headline multiplier.

1.0 still has migration edges

The version number should not be read as a claim that every path is settled. The router’s priority holdback policy is marked beta. Experimental SGLang rendering remains in parts of the API path. The release also removes or deprecates earlier behavior, including the old inference.networking.x-k8s.io pool group, InferenceObjective v1alpha2, deprecated cache metrics and the disaggregation sidecar’s --vllm-port flag.

Image provenance also requires attention. The project states that images produced from its main branch and formal releases are published through GitHub Container Registry and signed with Cosign. Pull-request builds use Quay as an unsigned staging registry. That distinction applies to images produced by the main llm-d repository, not automatically to every upstream model-server or ecosystem image in a deployment.

The broader governance status is another useful guardrail. CNCF Sandbox projects have neutral open-source stewardship, but Sandbox acceptance is not the same as the technical and governance due diligence associated with CNCF Incubating or Graduated projects. Teams still need to evaluate maintainer responsiveness, upgrade policy, security reporting, support ownership and compatibility with their chosen Kubernetes distribution and accelerator stack.

A practical llm-d 1.0 evaluation plan

1. Capture the current baseline. Record time to first token, inter-token latency, end-to-end latency, throughput, queue time, cache-hit rate, accelerator utilization and failure rate under a replay of real traffic.
2. Separate workload classes. Label interactive chat, coding agents, background agents, batch jobs and reinforcement-learning rollouts. Decide which classes may wait or be shed when capacity is tight.
3. Measure prefix reuse. Identify how much context is genuinely shared across turns and sessions. Cache-aware routing pays off only when reusable state exists and remains available long enough.
4. Test one capability at a time. Compare load-aware routing, prefix-aware routing, cache offload and prefill/decode disaggregation independently before combining them. Otherwise, a throughput gain can hide a latency or storage penalty.
5. Audit the upgrade removals. Search manifests, Helm values, dashboards and alerts for the deprecated API group, v1alpha2 resources, removed flags and old metric names documented in the router release notes.
6. Exercise failure and rollout behavior. Kill router and model-server pods, interrupt a cache tier, roll mismatched prefill and decode versions, and confirm that active sessions either continue or fail in a controlled, observable way.
7. Verify every image. Pin digests, check Cosign signatures where the project provides them, and document separately which upstream vLLM, SGLang, accelerator and storage images are inside the deployment.
8. Price the whole stack. Include idle accelerator time, CPU memory, shared storage, cross-node traffic and operator effort. Lower token-serving cost can be offset by an unnecessarily complex cache or disaggregation design.

IBM describes 1.0 as the point where llm-d moves from demonstrating individual inference optimizations to supporting “day 2” operations. The release supplies credible building blocks for that claim: stable configuration APIs, explicit overload policy, high-availability routing, coordinated rollouts and deeper tracing. For platform teams, the useful question is now narrower than whether distributed inference works. It is whether their traffic contains enough repeated state, concurrency and workload conflict to justify operating it.

Sources

• llm-d v1.0.0 release notes
• llm-d router v1.0.0 release notes
• Google Cloud: Announcing llm-d 1.0
• IBM Research: llm-d 1.0 and production inference
• CNCF: llm-d Sandbox project announcement