So this replaces triton for LLMs or?
same question here. Just asked Grok for a comparsion https://grok.com/share/bGVnYWN5_fa210574-f27b-45ae-9d95-19ed...
Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
21–30 of 44 posts
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#22Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#23Built in Rust for performance and in Python for extensibility Omg, a team that knows how to selectively use tech as needed. Looking at the Rust web developers in corner.
Unsure if the implication is that Rust is poorly suited for web development or what.
Rust is emerging as one of the best web programming languages out there.
Actix and Axum feel like Python's Flask.
Rust has decent Redis and connection pool libraries, but the SQL space needs more work. Diesel SQL is too ORM-y (I've never liked ORMs). While SQLx allows you to write "typechecked" SQL, it still has really annoying edge cases (WHERE IN clauses can't be typechecked, type bindings can get hairy, etc.)
I'm not very happy about the state of Rust's Elasticsearch libraries, either.
Rust probably needs a Rails/Django-like framework too for those that prefer a framework-oriented development lifecycle.
Rust also needs some observability frameworks. There are a few, but the choices are sparse.
I'd give Rust a 7.5/10 for web programming, and as far as the promise of the language goes, I'd give it an 11/10. Developing in Actix and Axum feels amazing. It's honestly better than Go and Python. The other pieces (database, API clients, etc.) will presumably get better in time.
And because of the way HTTP request flow logic is typically structured, 99.9% of the time you'll never hit Rust's borrow checker or have to worry about lifetimes. It's as if you've been given one of the best typed languages, best package managers, and nearly no tradeoffs. The server compiles down to a single static binary. It's multithreaded, and it's blindingly fast.
I'm picking Rust for every new web service I write these days.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#24Earlier quoted context omitted.
Unsure if the implication is that Rust is poorly suited for web development or what.
It is, in my opinion (as an avid Rust user!). The type errors from most of the major web frameworks/ORMs (diesel, sqlx) are just awful, more often than not. Usually some inscrutable thing involving Send/Sync. Or some hilariously complicated type/trait hackery on the part of the library, attempting to save me from the former, that I'm never going to figure out. Great language in many other settings, but not this one.…
> Usually some inscrutable thing involving Send/Sync.
What kind of queries are you writing? I never see SQLx emit anything like that. I always get back SQL errors.
Column mismatches are what I hit most in development, and they're pretty explicit.
The hairiest thing I see with SQLx is when I try to write custom type conversions for my own types to SQL fields. I sometimes have to delve into macros. But those errors are pretty self-explanatory too.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#25Earlier quoted context omitted.
Unsure if the implication is that Rust is poorly suited for web development or what.
Rust is emerging as one of the best web programming languages out there. Actix and Axum feel like Python's Flask. Rust has decent Redis and connection pool libraries, but the SQL space needs more work. Diesel SQL is too ORM-y (I've never liked ORMs). While SQLx allows you to write "typechecked" SQL, it still has really annoying edge cases (WHERE IN clauses can't be typechecked, type bindings can get hairy, etc.) I'm…
Show me some Rust web code that you think exemplifies what you are talking about. I think your example will speak for itself and close this argument. No reason to go back and forth.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#26Earlier quoted context omitted.
Unsure if the implication is that Rust is poorly suited for web development or what.
Rust is emerging as one of the best web programming languages out there. Actix and Axum feel like Python's Flask. Rust has decent Redis and connection pool libraries, but the SQL space needs more work. Diesel SQL is too ORM-y (I've never liked ORMs). While SQLx allows you to write "typechecked" SQL, it still has really annoying edge cases (WHERE IN clauses can't be typechecked, type bindings can get hairy, etc.) I'm…
same here, Rust with actix can even replace nginx.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#27Earlier quoted context omitted.
Unsure if the implication is that Rust is poorly suited for web development or what.
Rust is emerging as one of the best web programming languages out there. Actix and Axum feel like Python's Flask. Rust has decent Redis and connection pool libraries, but the SQL space needs more work. Diesel SQL is too ORM-y (I've never liked ORMs). While SQLx allows you to write "typechecked" SQL, it still has really annoying edge cases (WHERE IN clauses can't be typechecked, type bindings can get hairy, etc.) I'm…
This is obviously heavily biased, because there is no way any reasonable person would think Axum or Actix are like Flask. That's just not possible with a language like Rust. The Rust standard library is horrible compared to Python or Go.
You need more dependencies to build a simple APi in Rust than you need in Python, and Go combined.
Axum, tokio, serde, serde_json, anyhow, sqlx and probably 5 more to fix the bad standard library.
In Python and Go you can build web app with the standard library.
TBH after adding in a database, Rust is probably not that much faster than Go and Go has everything you need in the standard library, compiles to binary, and package manager doesn't matter cuz you don't need one using Go.
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#28Built in Rust for performance and in Python for extensibility Omg, a team that knows how to selectively use tech as needed. Looking at the Rust web developers in corner.
they actually implemented a decent amount of the HTTP stuff in rust. if you look at the docs
Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#29Re: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
#30Earlier quoted context omitted.
Rust is emerging as one of the best web programming languages out there. Actix and Axum feel like Python's Flask. Rust has decent Redis and connection pool libraries, but the SQL space needs more work. Diesel SQL is too ORM-y (I've never liked ORMs). While SQLx allows you to write "typechecked" SQL, it still has really annoying edge cases (WHERE IN clauses can't be typechecked, type bindings can get hairy, etc.) I'm…
same here, Rust with actix can even replace nginx.
This. People are sleeping on one of the biggest developments to hit backend. They'll know soon enough.
I'm too tired to respond to the two detractors, but it's hilarious that one of the arguments against Rust is pulling in packages. Some of the best packages, at that. I wonder if that's their argument against most other languages.
Big standard libraries are a mistake, because the language is forever left with shitty old design decisions. Python's standard library is full of crap.
I might just put together a side by side Flask / Go / Rust. It'll be so damning against Python and Go. Rust is the same LOC count, complexity, and yet is a nicer language, better type system, and is as fast as nginx.
People don't know how good Rust webdev is.