Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

github.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

#31

Post by imtringued »

echelon wrote: post #30
m00dy wrote: post #26

Earlier quoted context omitted.

same here, Rust with actix can even replace nginx.

This. People are sleeping on one of the biggest developments to hit backend. They'll know soon enough. I'm too tired to respond to the two detractors, but it's hilarious that one of the arguments against Rust is pulling in packages. Some of the best packages, at that. I wonder if that's their argument against most other languages. Big standard libraries are a mistake , because the language is forever left with shitty…

The complaint about serde in particular... Everyone and their dog includes Jackson for webdev in Java.

Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

#32

Post by raffraffraff »

Carrok wrote: post #2

As someone who spent the better part of a year trying to get various Nvidia inference products to work _at all_ even with a direct line to their developers, I will simply say "beware".

I've done very little with Nvidia software, but what I have done puts me off ever doing it again. I quit a job partially because it involved trying to get their shit to work. (There were other factors, but that was definitely on the 'GTFO' side)

Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

#33

Post by tormeh »

imtringued wrote: post #31
echelon wrote: post #30

Earlier quoted context omitted.

This. People are sleeping on one of the biggest developments to hit backend. They'll know soon enough. I'm too tired to respond to the two detractors, but it's hilarious that one of the arguments against Rust is pulling in packages. Some of the best packages, at that. I wonder if that's their argument against most other languages. Big standard libraries are a mistake , because the language is forever left with shitty…

The complaint about serde in particular... Everyone and their dog includes Jackson for webdev in Java.

Jackson is awful. Serde on the other hand is the smoothest JSON library I've ever used.

Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

#34

Post by tormeh »

bloomingkales wrote: post #5

Built in Rust for performance and in Python for extensibility Omg, a team that knows how to selectively use tech as needed. Looking at the Rust web developers in corner.

I do think Rust would be better for web dev if it had GC, but it doesn't, and no other language comes close to having as good ergonomics otherwise. And the memory management is something you learn once and then it's a bit verbose but no big deal. If you feel like you absolutely have to write custom data structures with circular references for your web server then I tentatively suggest that maybe you're doing web dev wrong.

In my team we onboarded a data scientist used to working in Python who had never used Rust onto a Rust project, and it was just not a big deal. Maybe I'm just fortunate when it comes to colleagues.

Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

#35

Post by saagarjha »

Nvidia not name products after existing things in the ML space challenge: IMPOSSIBLE

More seriously, though:

> OpenAI Compatible Frontend – High performance OpenAI compatible http api server written in Rust.

Is this normal in this space? I know everyone has settled on copying the S3 API for block storage but I’m unsure if we’ve done the same for LLM serving.

Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

#36

Post by DeveloperErrata »

saagarjha wrote: post #35

Nvidia not name products after existing things in the ML space challenge: IMPOSSIBLE More seriously, though: > OpenAI Compatible Frontend – High performance OpenAI compatible http api server written in Rust. Is this normal in this space? I know everyone has settled on copying the S3 API for block storage but I’m unsure if we’ve done the same for LLM serving.

Increasingly so. Many other popular inference tools in this space also expose an OpenAI compatible API: VLLM, Llama.cpp, and LiteLLM all do.

Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

#37

Post by bloomingkales »

tormeh wrote: post #34
bloomingkales wrote: post #5

Built in Rust for performance and in Python for extensibility Omg, a team that knows how to selectively use tech as needed. Looking at the Rust web developers in corner.

I do think Rust would be better for web dev if it had GC, but it doesn't, and no other language comes close to having as good ergonomics otherwise. And the memory management is something you learn once and then it's a bit verbose but no big deal. If you feel like you absolutely have to write custom data structures with circular references for your web server then I tentatively suggest that maybe you're doing web dev…

Sounds like nonsense. Again, I asked in another comment for an example of some Rust “web” code that exemplifies what you are talking about. You mentioned “ergonomics”, and you mentioned “noob friendly”. I’d love to see some of this Rust code.

Show us, we’ll discuss.

I feel like some devs are so insecure that they really think , ugh, I can’t even fully explain the pathology of the Rust people without cursing them out.

You are not a better developer, that’s ALL I want to say to the Rust people. In fact, most of you are bad developers for doing what you have been doing with this language. You ALL must find a better way to show your intellectual prowess.

I heard you guys are even bugging the Linux people.

Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

#38

Post by pjmlp »

echelon wrote: post #23
seangrogg wrote: post #19

Earlier quoted context omitted.

Unsure if the implication is that Rust is poorly suited for web development or what.

Rust is emerging as one of the best web programming languages out there. Actix and Axum feel like Python's Flask. Rust has decent Redis and connection pool libraries, but the SQL space needs more work. Diesel SQL is too ORM-y (I've never liked ORMs). While SQLx allows you to write "typechecked" SQL, it still has really annoying edge cases (WHERE IN clauses can't be typechecked, type bindings can get hairy, etc.) I'm…

Nowehere close to the pleothora of tooling and frameworks available in Java and .NET ecosystem for all kinds of distributed computing scenarios.

And if one misses an advanced ML type system, Scala, Kotlin, F# are there.

Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

#39

Post by tempest_ »

impulser_ wrote: post #27
echelon wrote: post #23

Earlier quoted context omitted.

Rust is emerging as one of the best web programming languages out there. Actix and Axum feel like Python's Flask. Rust has decent Redis and connection pool libraries, but the SQL space needs more work. Diesel SQL is too ORM-y (I've never liked ORMs). While SQLx allows you to write "typechecked" SQL, it still has really annoying edge cases (WHERE IN clauses can't be typechecked, type bindings can get hairy, etc.) I'm…

This is obviously heavily biased, because there is no way any reasonable person would think Axum or Actix are like Flask. That's just not possible with a language like Rust. The Rust standard library is horrible compared to Python or Go. You need more dependencies to build a simple APi in Rust than you need in Python, and Go combined. Axum, tokio, serde, serde_json, anyhow, sqlx and probably 5 more to fix the bad sta…

The standard lib is not "bad" it just is modular to avoid C++ like pitfalls.

People doing web dev in python could use pythons native json but rarely do because there are other far more performant options for example.

Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework

#40

Post by pjmlp »

tempest_ wrote: post #39
impulser_ wrote: post #27

Earlier quoted context omitted.

This is obviously heavily biased, because there is no way any reasonable person would think Axum or Actix are like Flask. That's just not possible with a language like Rust. The Rust standard library is horrible compared to Python or Go. You need more dependencies to build a simple APi in Rust than you need in Python, and Go combined. Axum, tokio, serde, serde_json, anyhow, sqlx and probably 5 more to fix the bad sta…

The standard lib is not "bad" it just is modular to avoid C++ like pitfalls. People doing web dev in python could use pythons native json but rarely do because there are other far more performant options for example.

It is bad in the sense we need to reach out for external libraries to avoid manually writing all the boilerplate with handling errors, and any async runtime works, as long as it is tokio.
Post reply on HN