“human in the loop scheduler” is very funny
The Tokio/Rayon Trap and Why Async/Await Fails Concurrency
11–20 of 63 posts
Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency
#12There are some fundamental assumptions in the Rust async system: - The program is mostly I/O bound. - All tasks have equal priority. If your program isn't like that, the Tokio model is a bad match to the problem. Real time control is not like that. MMO and metaverse game programs are not like that. Most web stuff is, but that's a special case. A big special case, but a special case.
There's a few runtimes for IO bound work loads; monoio from bytedance comes to mind.
Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency
#13There are some fundamental assumptions in the Rust async system: - The program is mostly I/O bound. - All tasks have equal priority. If your program isn't like that, the Tokio model is a bad match to the problem. Real time control is not like that. MMO and metaverse game programs are not like that. Most web stuff is, but that's a special case. A big special case, but a special case.
To be fair, the Rust async model itself was intentionally designed not to be prescriptive in the way you describe. You can build, and there exists, different task executors that can handle things like priority and many other execution models.
Async is just a way to describe a tree of concurrent tasks that may depend on (wait on) each other at certain points. It is mostly declarative.
Tokio has taken over as the default choice, but there's a reason why it's not part of the standard library, it is not meant to be the only choice.
Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency
#14I am not really sure how much yet another post complaining about async/await that ends with “thread-per-core is the way to go” adds to this discussion. Granted I’m both an Erlang programmer and a big fan of Tokio and Rust’s async/await implementation in general and I think this post and many others like it betray a fundamental misunderstanding of these technologies so I am probably biased.
Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency
#15I've been thinking similar thoughts recently, in that I am not sure that a function call is the best way to model asynchronicity. Io-uring in particular feels much more suited to a req/res type model, which has the benefit that you can make a single threaded state machine the core of your app - very pleasant to test and reason about. I'd draw the analogy to RPC; it's a leaky abstraction because HTTP is fundamentally…
I agree that a request-response model can be quite ergonomic for concurrency. I've used this pattern in quite a few Rust projects in a somewhat ad-hoc manner.
It's a bit like having a system composed of microservices (nanoservices?) that communicate via function calls.
It sounds a lot like the actor model, but I always found the classic architecture too limiting: requiring every actor to be a single-threaded message processor, instead of being able to handle requests concurrently. It's not too different from classic object-oriented design either, with singleton services.
In some project I've called my concurrent services Gods just to have a bit of fun with it :)
Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency
#16I see two solid points here:
1. It's not reasonable to expect the application layer to carefully partition its work into "I/O heavy" and "CPU heavy" parts.
2. It's not reasonable to queue up an arbitrary amount of work without back-pressure.
I haven't used Tokio much, but if it falls prey to these pitfalls, it would make me pause before adopting it.
I think there are probably ways of using Rust async that don't fall prey to these. Maybe not so much with network servers (I haven't written that many of those), but models where you are evaluating a graph and have more control over how new work is added to the system.
Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency
#17There are some fundamental assumptions in the Rust async system: - The program is mostly I/O bound. - All tasks have equal priority. If your program isn't like that, the Tokio model is a bad match to the problem. Real time control is not like that. MMO and metaverse game programs are not like that. Most web stuff is, but that's a special case. A big special case, but a special case.
To be fair, the Rust async model itself was intentionally designed not to be prescriptive in the way you describe. You can build, and there exists, different task executors that can handle things like priority and many other execution models. Async is just a way to describe a tree of concurrent tasks that may depend on (wait on) each other at certain points. It is mostly declarative. Tokio has taken over as the defau…
Outside of Embassy in embedded, tokio is the only realistic choice though, because it is likely that any third party async crate has a dependency on it already. Yes, smol, monio, glommio etc exist, but they are marginalised (and as far as I can tell they don't really help that much with mixed IO / compute workloads).
In fact, async/await in Rust falls apart with a mixed IO / compute workload since scheduling is cooperative. As soon as you want preemption (most of the time for what I do), it is not the right choice.
Seeing how embassy (embedded async rust) handles preemption reinforces this: it uses a separate scheduler per preemption level. This works fine, but is a bit clunky. Basically you are at this point just using async to help write a state machine per preemptive thread, which can be useful for some code patterns (in particular those common in embedded, where you are often waiting for IO). But to talk between threads you are back to channels, mutexes etc.
Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency
#18I think it's important to understand how we got here and a lot of it has to do with serving network requests or RPCs. The first Web servers used CGI (Common Gateway Interface). This spawned an entire program (process) per request and had obvious overheads. This led to some optimizations (eg FastCGI, ISAPI/NSAPI) to reduce the overhead. This was the era of Perl scripts being popular. Then came the model of having a pe…
Your history is all valid, but I don't think it really hits on the main motivations for how we got here.
Thread per request works perfectly fine if your application is CPU constrained.
However the observation was made, that most web applications are IO constrained, the majority of the time spent serving a web request is spent waiting for a database or downstream API.
Since most of the threads are idle waiting, your application needs many threads to optimally utilize the servers resources.
There was a perception(valid or not) that OS threads have too much memory and scheduling overhead.
Nginx came out using async io, and it could handle much more concurrent requests than apache, which used a threaded model, it sparked a lot of interest in different kinds of application managed scheduling.
It inspired initiatives like the reactive manifesto[0], which spawned tools like RXJava.
Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency
#19The suggested alternative is interesting, I was expecting a Rust library though, but it's an Odin one. What's missing are any sort of benchmarks or proof of the claims that it will scale beyond the issues the article discuss.
Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency
#20I think it's important to understand how we got here and a lot of it has to do with serving network requests or RPCs. The first Web servers used CGI (Common Gateway Interface). This spawned an entire program (process) per request and had obvious overheads. This led to some optimizations (eg FastCGI, ISAPI/NSAPI) to reduce the overhead. This was the era of Perl scripts being popular. Then came the model of having a pe…
That's an informative overview. Somewhere in the story was Node.js and libuv, the callback style, promises, and the popularization of the async/await paradigm. Not sure if there was a direct influence on Rust's async libraries, but I imagine it affected how some people think what an intuitive async syntax might look like.