Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

pmbanugo.me

Post reply on HN
Print view View original

41–50 of 63 posts

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#41

Post by FridgeSeal »

elendilm wrote: post #34
FridgeSeal wrote: post #30

Earlier quoted context omitted.

> Increasing threads beyond system's hardware cores/threads resulted only in marginal gains of a couple of milliseconds worth of differences on huge workload with large increase in memory Careful, if you say that too loudly, the "get rid of async just spawn more threads!!!!!" people will come out of the woodwork to yell at you about how _all_ async is a lie and we should instead pretend none of it exists and just spa…

I am now trembling at the thought of warriors who will skin me alive :). Jokes aside, there are use cases for rayon, use cases for tokio async, use cases for "may" coroutines, use cases for a custom scheduling policy, or use cases for a combination of these. We went with "may" coroutines (with its thread pinning) + custom numa work pinning (to a may thread) due to "may's" lightweight nature and not having to have our…

It sounds like you’ve got a super interesting stack going on there, evidently with a large performance/latency focus.

May (haha) I ask what this is in service of? I’m somewhat a fan of the thread-per-core model, so I’m curious as to what you’re doing with it.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#42

Post by quietbritishjim »

random17 wrote: post #35
VorpalWay wrote: post #17

Earlier quoted context omitted.

Outside of Embassy in embedded, tokio is the only realistic choice though, because it is likely that any third party async crate has a dependency on it already. Yes, smol, monio, glommio etc exist, but they are marginalised (and as far as I can tell they don't really help that much with mixed IO / compute workloads). In fact, async/await in Rust falls apart with a mixed IO / compute workload since scheduling is coope…

How do you properly mix IO/compute (in any language)? In Rust what I’ve done in the past is have two Tokio runtimes, one for IO and one for compute. I know you can also use Rayon but the abstractions are not always flexible/convenient enough. But in either case the boundary between the two types of async work is never easy to cross.

In Python Trio, you would have a thread pool for each type of computation (potentially a "pool" of 1 thread shared between all tasks if that works for your application). You would await trio.to_thread.run_sync(...) [1] (pass it a normal non-async function, and a token representing the thread pool). This is a pretty simple wrapper that takes care of queuing the work to the thread pool, waiting for it to complete and for a message to be passed back to the Trio thread.

It works for simple situations. It certainly preserves backpressure, although a slow CPU task can impact other users of the same thread pool.

[1] https://trio.readthedocs.io/en/stable/reference-core.html#tr...

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#43

Post by ChrisMarshallNY »

I just had to rip out a lot of native Swift concurrency, and replace it with classic GCD.

Looks like tasks mess with the reference counter. My app was not releasing memory, and was constantly jetsam-crashing.

Also, the app was quite sluggish.

Reverting to GCD, fixed both these issues.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#45

Post by veber-alex »

diarrhea wrote: post #33
levkk wrote: post #24

This is nonsense. Tokio was built for I/O, not crunching numbers. Most programmers know to use spawn_blocking to crunch 10MB JSON, if needed. Additionally, the proposed workload per thread model will be orders of magnitude slower than Tokio for I/O-bound workloads, which most applications are.

Funnily enough, spawn_blocking is not the right tool here. It is meant for blocking I/O, such as DNS lookups, where your platform might not give you anything better. For genuine CPU-bound work, submitting to a Rayon worker pool is the way to go. It solved a runtime starvation issue for us at work, spawn_blocking did not work. The reason for all this is spawn_blocking having a very large underlying thread pool, in the…

You can easily limit the number of blocking threads tokio can spawn: https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.ht...

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#47

Post by adrian_b »

WatchDog wrote: post #18
jmyeet wrote: post #7

I think it's important to understand how we got here and a lot of it has to do with serving network requests or RPCs. The first Web servers used CGI (Common Gateway Interface). This spawned an entire program (process) per request and had obvious overheads. This led to some optimizations (eg FastCGI, ISAPI/NSAPI) to reduce the overhead. This was the era of Perl scripts being popular. Then came the model of having a pe…

Your history is all valid, but I don't think it really hits on the main motivations for how we got here. Thread per request works perfectly fine if your application is CPU constrained. However the observation was made, that most web applications are IO constrained, the majority of the time spent serving a web request is spent waiting for a database or downstream API. Since most of the threads are idle waiting, your a…

> Since most of the threads are idle waiting, your application needs many threads to optimally utilize the servers resources.

I do not think that "many" and "optimally" belong in the same sentence.

Instead of having a great number of threads that are idle waiting, it seems much more efficient to have a single thread or a small number of threads, which handle multiplex asynchronous I/O, using something like liburing on Linux or I/O completion ports on Windows, so that the threads are seldom idle, wasting resources.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#48

Post by tcfhgj »

veber-alex wrote: post #45
diarrhea wrote: post #33

Earlier quoted context omitted.

Funnily enough, spawn_blocking is not the right tool here. It is meant for blocking I/O, such as DNS lookups, where your platform might not give you anything better. For genuine CPU-bound work, submitting to a Rayon worker pool is the way to go. It solved a runtime starvation issue for us at work, spawn_blocking did not work. The reason for all this is spawn_blocking having a very large underlying thread pool, in the…

You can easily limit the number of blocking threads tokio can spawn: https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.ht...

Might not be a good option though if you also spawn a bunch of blocking io tasks, which benefit from a large number of threads.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#49

Post by oersted »

random17 wrote: post #35
VorpalWay wrote: post #17

Earlier quoted context omitted.

Outside of Embassy in embedded, tokio is the only realistic choice though, because it is likely that any third party async crate has a dependency on it already. Yes, smol, monio, glommio etc exist, but they are marginalised (and as far as I can tell they don't really help that much with mixed IO / compute workloads). In fact, async/await in Rust falls apart with a mixed IO / compute workload since scheduling is coope…

How do you properly mix IO/compute (in any language)? In Rust what I’ve done in the past is have two Tokio runtimes, one for IO and one for compute. I know you can also use Rayon but the abstractions are not always flexible/convenient enough. But in either case the boundary between the two types of async work is never easy to cross.

You can use tokio::spawn_blocking to separate compute-heavy stuff, it gets handled in a separate thread pool. You shouldn’t need two tokio runtimes, that sounds problematic.

> How do you properly mix IO/compute (in any language)?

Good question, it’s a really hard problem.

In principle you are meant to use threads, the OS will automatically switch to other work while you are waiting for IO, like with async. But there are difficulties with threads too: memory overhead, switching overhead, thread limits… Although I am getting the feeling that they have a much worse reputation than they deserve. More projects should try switching to threads and actually measuring the difference.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#50

Post by fatata123 »

herculity275 wrote: post #37
FridgeSeal wrote: post #27

This reads an awful lot like the prompter had a semi-confusing time with some async, and had their favourite model write an upset blog post about it. I don't think these systems are perfect, nor are they fit for every use-case, but some of the complaints ring a bit hollow. > fetching a database record over the network, then immediately crunching the data. But what happens when that data crunching involves parsing a 1…

This entire post was written/prompted as a pitch for the author's own concurrency framework. These types of "current state of the world - bad, my solution - good" posts have been annoying for far longer than LLMs have been around, but the LLMified over-embellishment coupled with the vagueness makes it so much worse.

[dead]
Post reply on HN