As someone who spent the better part of a year trying to get various Nvidia inference products to work _at all_ even with a direct line to their developers, I will simply say "beware".
is this in reference to Triton?
Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
11–20 of 44 posts
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#12Earlier quoted context omitted.
As someone who has run LLMs in production, using Ray is probably the worst idea. It's not optimized for language models, and is extremely slow. There's no KV-caching, model parallelism, and other basic table stakes features that are offered by Dynamo or other open source inference frameworks. Useful only if you have Use SGLang, vLLM, or text-generation-inference instead.
This is probably true, but unlike every Nvidia product we tried, it did, you know, reply to inference requests with actual output. That said, you can serve vLLM with Ray Serve. https://docs.ray.io/en/latest/serve/tutorials/vllm-example.h...
Ray doesn't offer anything if you use vLLM on top of Ray Serve though.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#13Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#14So this replaces triton for LLMs or?
This is very narrowly focused on LLMs, whereas triton is still useful for running all kinds of ML models. In practice, Triton is a very poor choice for LLMs specifically because it has none of the required non negotiable features like KV caching built in.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#15As someone who spent the better part of a year trying to get various Nvidia inference products to work _at all_ even with a direct line to their developers, I will simply say "beware".
Triton is not that bad at all, considering the wide scope of systems it has to support (tensorrt, onnx, multiple generations of pytorch, cuda, python). It was much nicer than the old Torchserve project which was JVM based.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#16[deleted]
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#17Earlier quoted context omitted.
This is probably true, but unlike every Nvidia product we tried, it did, you know, reply to inference requests with actual output. That said, you can serve vLLM with Ray Serve. https://docs.ray.io/en/latest/serve/tutorials/vllm-example.h...
Ray doesn't offer anything if you use vLLM on top of Ray Serve though.
It does if you need pipeline parallelism across multiple nodes.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#18As someone who spent the better part of a year trying to get various Nvidia inference products to work _at all_ even with a direct line to their developers, I will simply say "beware".
Just curious what your issues with Triton were. We've done OK with it using it to serve LLM models w/ a classifier head via HF Transformers pipeline & Flash Attention 2, as well as serving text generation models with the vLLM back-end.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#19Built in Rust for performance and in Python for extensibility Omg, a team that knows how to selectively use tech as needed. Looking at the Rust web developers in corner.
Unsure if the implication is that Rust is poorly suited for web development or what.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#20Built in Rust for performance and in Python for extensibility Omg, a team that knows how to selectively use tech as needed. Looking at the Rust web developers in corner.
Unsure if the implication is that Rust is poorly suited for web development or what.
It is, in my opinion (as an avid Rust user!). The type errors from most of the major web frameworks/ORMs (diesel, sqlx) are just awful, more often than not. Usually some inscrutable thing involving Send/Sync. Or some hilariously complicated type/trait hackery on the part of the library, attempting to save me from the former, that I'm never going to figure out.
Great language in many other settings, but not this one. At least not right now, but given my experience with async Rust in general, I'm not sure it ever will be.