Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
1–10 of 44 posts
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#2As someone who spent the better part of a year trying to get various Nvidia inference products to work _at all_ even with a direct line to their developers, I will simply say "beware".
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#3As someone who spent the better part of a year trying to get various Nvidia inference products to work _at all_ even with a direct line to their developers, I will simply say "beware".
Can you share some of your wisdom on setting up a scalable inference infrastructure?
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#4As someone who spent the better part of a year trying to get various Nvidia inference products to work _at all_ even with a direct line to their developers, I will simply say "beware".
Can you share some of your wisdom on setting up a scalable inference infrastructure?
Use Ray Serve. https://docs.ray.io/en/latest/serve/index.html
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#5Built in Rust for performance and in Python for extensibility
Omg, a team that knows how to selectively use tech as needed. Looking at the Rust web developers in corner.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#6Earlier quoted context omitted.
Can you share some of your wisdom on setting up a scalable inference infrastructure?
Use Ray Serve. https://docs.ray.io/en/latest/serve/index.html
As someone who has run LLMs in production, using Ray is probably the worst idea. It's not optimized for language models, and is extremely slow. There's no KV-caching, model parallelism, and other basic table stakes features that are offered by Dynamo or other open source inference frameworks. Useful only if you have Use SGLang, vLLM, or text-generation-inference instead.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#7Earlier quoted context omitted.
Use Ray Serve. https://docs.ray.io/en/latest/serve/index.html
As someone who has run LLMs in production, using Ray is probably the worst idea. It's not optimized for language models, and is extremely slow. There's no KV-caching, model parallelism, and other basic table stakes features that are offered by Dynamo or other open source inference frameworks. Useful only if you have Use SGLang, vLLM, or text-generation-inference instead.
This is probably true, but unlike every Nvidia product we tried, it did, you know, reply to inference requests with actual output. That said, you can serve vLLM with Ray Serve. https://docs.ray.io/en/latest/serve/tutorials/vllm-example.h...
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#8[deleted]
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#9So this replaces triton for LLMs or?
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#10Earlier quoted context omitted.
Use Ray Serve. https://docs.ray.io/en/latest/serve/index.html
As someone who has run LLMs in production, using Ray is probably the worst idea. It's not optimized for language models, and is extremely slow. There's no KV-caching, model parallelism, and other basic table stakes features that are offered by Dynamo or other open source inference frameworks. Useful only if you have Use SGLang, vLLM, or text-generation-inference instead.
It really depends on the task. If you have 1 massive job, Ray sucks and doesn't provide table stakes. If you have 50M tiny jobs, Ray and kuberay is great and serves as the backbone of several billion dollar products.
Good for the goose, good for the gander...