Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM-D: Kubernetes-Native Distributed Inference

llm-d.ai

Post reply on HN
Print view View original

11–17 of 17 posts

Re: LLM-D: Kubernetes-Native Distributed Inference

#11

Post by smarterclayton »

qntty wrote: post #10
rdli wrote: post #9

This is really interesting. For SOTA inference systems, I've seen two general approaches: * The "stack-centric" approach such as vLLM production stack, AIBrix, etc. These set up an entire inference stack for you including KV cache, routing, etc. * The "pipeline-centric" approach such as NVidia Dynamo, Ray, BentoML. These give you more of an SDK so you can define inference pipelines that you can then deploy on your sp…

It sounds like you might be confusing different parts of the stack. NVIDIA Dynamo for example supports vLLM as the inference engine. I think you should think of something like vLLM as more akin to GUnicorn, and llm-d as an application load balancer. And I guess something like NVIDIA Dynamo would be like Django.

llm-d is intended to be three clean layers:

1. Balance / schedule incoming requests to the right backend

2. Model server replicas that can run on multiple hardware topologies

3. Prefix caching hierarchy with well-tested variants for different use cases

So it's a 3-tier architecture. The biggest difference with Dynamo is that llm-d is using the inference gateway extension - https://github.com/kubernetes-sigs/gateway-api-inference-ext... - which brings Kubernetes owned APIs for managing model routing, request priority and flow control, LoRA support etc.

Re: LLM-D: Kubernetes-Native Distributed Inference

#12

Post by rdli »

qntty wrote: post #10
rdli wrote: post #9

This is really interesting. For SOTA inference systems, I've seen two general approaches: * The "stack-centric" approach such as vLLM production stack, AIBrix, etc. These set up an entire inference stack for you including KV cache, routing, etc. * The "pipeline-centric" approach such as NVidia Dynamo, Ray, BentoML. These give you more of an SDK so you can define inference pipelines that you can then deploy on your sp…

It sounds like you might be confusing different parts of the stack. NVIDIA Dynamo for example supports vLLM as the inference engine. I think you should think of something like vLLM as more akin to GUnicorn, and llm-d as an application load balancer. And I guess something like NVIDIA Dynamo would be like Django.

In this analogy, Dynamo is most definitely not like Django. It includes inference aware routing, KV caching, etc. -- all the stuff you would need to run a modern SOTA inference stack.

Re: LLM-D: Kubernetes-Native Distributed Inference

#13

Post by rdli »

smarterclayton wrote: post #11
qntty wrote: post #10

Earlier quoted context omitted.

It sounds like you might be confusing different parts of the stack. NVIDIA Dynamo for example supports vLLM as the inference engine. I think you should think of something like vLLM as more akin to GUnicorn, and llm-d as an application load balancer. And I guess something like NVIDIA Dynamo would be like Django.

llm-d is intended to be three clean layers: 1. Balance / schedule incoming requests to the right backend 2. Model server replicas that can run on multiple hardware topologies 3. Prefix caching hierarchy with well-tested variants for different use cases So it's a 3-tier architecture. The biggest difference with Dynamo is that llm-d is using the inference gateway extension - https://github.com/kubernetes-sigs/gateway-a…

I would think that that the NVidia Dynamo SDK (pipelines) is a big difference as well (https://github.com/ai-dynamo/dynamo/tree/main/deploy/sdk/doc...), or am I missing something?

Re: LLM-D: Kubernetes-Native Distributed Inference

#15

Post by smarterclayton »

rdli wrote: post #13
smarterclayton wrote: post #11

Earlier quoted context omitted.

llm-d is intended to be three clean layers: 1. Balance / schedule incoming requests to the right backend 2. Model server replicas that can run on multiple hardware topologies 3. Prefix caching hierarchy with well-tested variants for different use cases So it's a 3-tier architecture. The biggest difference with Dynamo is that llm-d is using the inference gateway extension - https://github.com/kubernetes-sigs/gateway-a…

I would think that that the NVidia Dynamo SDK (pipelines) is a big difference as well ( https://github.com/ai-dynamo/dynamo/tree/main/deploy/sdk/doc... ), or am I missing something?

That's a good example - I can at least answer about why it's a difference: different target user.

As I understand the Dynamo SDK it is about simplifying and helping someone get started with Dynamo on Kubernetes.

From the user set we work with (large inference deployers) that is not a high priority - they already have mature deployment opinions or a set of tools that would not compose well with something like the Dynamo SDK. Their comfort level with Kubernetes is moderate to high - either they use Kubernetes for high scale training and batch, or they are deploying to many different providers in order to get enough capacity and need a standard orchestration solution.

llm-d focuses on helping achieve efficiency dynamically at runtime based on changing traffic or workload on Kubernetes - some of the things the Dynamo SDK encodes are static and upfront and would conflict with that objective. Also, large deployers with serving typically have significant batch and training and they are looking to maximize capacity use without impacting their prod serving. That requires the orchestrator to know about both workloads at some level - which Dynamo SDK would make more difficult.

Re: LLM-D: Kubernetes-Native Distributed Inference

#16

Post by qntty »

rdli wrote: post #12
qntty wrote: post #10

Earlier quoted context omitted.

It sounds like you might be confusing different parts of the stack. NVIDIA Dynamo for example supports vLLM as the inference engine. I think you should think of something like vLLM as more akin to GUnicorn, and llm-d as an application load balancer. And I guess something like NVIDIA Dynamo would be like Django.

In this analogy, Dynamo is most definitely not like Django. It includes inference aware routing, KV caching, etc. -- all the stuff you would need to run a modern SOTA inference stack.

You're right, I was confusing TensorRT with Dynamo. It looks like the relationship between Dynamo and vLLM is actually the opposite of what I was thinking -- Dynamo can use vLLM as a backend rather than vice versa.
Post reply on HN