Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

Post by richardliaw » Thu, Mar 20, 2025, 4:08 PM UTC

erulabs wrote: post #10

ipsum2 wrote: post #6

Earlier quoted context omitted.

As someone who has run LLMs in production, using Ray is probably the worst idea. It's not optimized for language models, and is extremely slow. There's no KV-caching, model parallelism, and other basic table stakes features that are offered by Dynamo or other open source inference frameworks. Useful only if you have Use SGLang, vLLM, or text-generation-inference instead.

It really depends on the task. If you have 1 massive job, Ray sucks and doesn't provide table stakes. If you have 50M tiny jobs, Ray and kuberay is great and serves as the backbone of several billion dollar products. Good for the goose, good for the gander...

Post by bytesandbits » Fri, Mar 21, 2025, 10:04 PM UTC

dlewis1788 wrote: post #18

Carrok wrote: post #2

As someone who spent the better part of a year trying to get various Nvidia inference products to work _at all_ even with a direct line to their developers, I will simply say "beware".

Just curious what your issues with Triton were. We've done OK with it using it to serve LLM models w/ a classifier head via HF Transformers pipeline & Flash Attention 2, as well as serving text generation models with the vLLM back-end.