How LLM-D Enables Distributed AI Inference with Kubernetes

TL;DR

LLM-D distributes AI inference across a Kubernetes cluster to boost performance and cut costs. It uses an inference gateway and endpoint picker to route requests based on current load, latency, and cache likelihood. Prefill and decode phases run on different resources but share a common KV cache to maximize efficiency.

Transcript

Real quick, I want you to imagine your local airport. See, planes are flying in from all types of locations. Maybe it's small domestic flights, for example, or huge international planes. And thank goodness that we have something like an air traffic controller to really understand and be able to see the request. Or for this example, planes coming in... Read More

Key Insights

• LLM-D uses a gateway to intelligently route inference requests rather than simple round robin balancing.
• The prefill phase uses high memory GPUs to prepare inputs, while decode scales separately to generate responses.
• Routing decisions consider current load, predicted latency, and cache probabilities to optimize throughput and cost.
• Disaggregating workloads reduces bottlenecks by separating input processing from output generation.
• A shared KV cache enables efficient reuse of results for similar prompts across workloads.
• P90 latency improvements show LLM-D can significantly reduce slowest requests in practice.
• First token time to respond improves dramatically with the disaggregated approach.
• The approach supports HLPS (high latency prone systems) and QoS goals in hyperscale environments.

Questions & Answers

Q: How does LLM-D route incoming AI requests in a Kubernetes environment

LLM-D routes incoming requests using an inference gateway and an endpoint picker. The gateway evaluates metrics such as current load, predicted latency, and the likelihood that the data has been cached. Based on these factors, it assigns the request to either a prefill path that prepares the input with high memory GPUs or a decode path that generates the output on scalable resources. This two phase routing helps balance hardware utilization and reduces overall latency while maintaining a shared KV cache for similar prompts. The routing logic is designed to prevent congestion that would occur with simple round robin schemes and aims to optimize throughput and cost across the cluster.

Q: What are the prefill and decode phases in LLM-D

In LLM-D the prefill phase occurs first and focuses on evaluating and preparing the incoming prompt. It typically uses high memory GPUs to load models and maximize prompt context handling, enabling faster overall processing. The decode phase follows, where the actual generation of the response happens. Decoding can scale independently from prefill, allowing the system to adapt to demand while sharing the same KV cache for similar requests. This separation improves latency characteristics and resource efficiency.

Q: How does the two phase approach impact latency

The two phase approach reduces latency by separating input preparation from response generation. Prefill leverages high memory GPUs to minimize input processing time, while decode scales to meet demand without waiting for prefill to finish. This reduces inter token latency and tail latency because slower requests can be handled by additional decoding capacity, rather than stalling on a single processing path. Overall, it yields faster first tokens and better P90 performance in high traffic scenarios.

Q: Why is a KV cache important in LLM-D

The KV cache stores key computed representations across similar prompts, allowing the system to reuse work and avoid recomputing redundant tokens. By sharing the cache between prefill and decode phases, LLM-D can accelerate responses for repeated or similar requests, lowering compute requirements and energy use. This shared cache is central to achieving cost efficiency and faster response times in distributed inference.

Q: What benefits does LLM-D claim for P90 latency

LLM-D reports a threefold improvement in P90 latency, meaning the slowest 10 percent of requests are much faster. This improvement is achieved through intelligent routing that avoids congestion, the division of workloads into prefill and decode, and the shared KV cache enabling reuse of calculations. Lower tail latency helps meet strict service level objectives and improves user experience in demanding AI workloads.

Q: How does LLM-D affect first token response time

The system reports a substantial improvement in first token response time, up to fifty seven times faster in some configurations. This is driven by optimized routing that directs requests to appropriate resources, the prefill phase efficiently preparing inputs, and more flexible scaling during decode. Faster first tokens reduce interactive latency and improve perceived responsiveness for users.

Q: In what scenarios is LLM-D particularly useful

LLM-D is particularly useful in high demand AI workflows such as RAG applications and agent based coding assistants that require low latency and cost efficiency. The architecture distributes inference across a Kubernetes cluster, uses an intelligent gateway to route requests, and leverages disaggregated, scalable resources with a shared cache. This setup helps meet service level objectives and supports mission critical AI tasks.

Q: What role does caching play in LLM-D

Caching in LLM-D increases efficiency by reducing redundant computation for similar prompts. The endpoint picker evaluates the likelihood of cached data and routes accordingly, enabling reuse of previously computed tokens. This reduces processing time and hardware usage while maintaining responsive performance for recurring or similar requests across multiple workloads. The shared KV cache is essential for achieving cost savings and speed.

Summary & Key Takeaways

• LLM-D introduces a dedicated inference gateway that routes requests to appropriate workloads based on load, latency, and cache hit probability, enabling more efficient use of hardware.

• Disaggregation of tasks into prefill and decode allows high memory GPUs for input processing and scalable compute for output generation, while sharing a KV cache for similar prompts.

• The system aims to reduce inter token latency and improve tail latency (P90) to meet stringent service level objectives in large scale AI deployments.