Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

pmbanugo.me

Post reply on HN
Print view View original

31–40 of 63 posts

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#31

Post by oersted »

didibus wrote: post #29
oersted wrote: post #23

I was looking forward to checking out Project Tina until I realised it was a completely different language. Classic story. Surely you can build a thread-per-core message passing concurrency framework in Rust, the language is designed to allow such alternatives. Is Project Tina a bit like the Actor Model, but having actors pinned to cores? And I don't understand how Tina deals better with the problem of compute-heavy…

I'm not 100% sure, but it looks like Tina forces your IO to be return values. You can't actually do IO inside the Isolate handlers, so your compute code runs and returns the next IO operation to run. That's what it meant by you need to be explicit about the state machine, you have to have a handler for before I make this IO, and a handler for after that IO has run. Isolates are like synchronous state machines. During…

That does clarify some things, but still. Say you have an Isolate pinned to a core and it's doing some long compute. In the meantime some background IO finished and an Isolate in the same core needs to handle the result. That will not happen until the long compute is done.

Isn't it just like in async and any other cooperative concurrency model? At least in multi-core async, that message can be handled in a different core so it's not completely stuck. But sure the author doesn't like that work-stealing cost happening automatically outside of their control, fair enough.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#32

Post by lunar_mycroft »

haberman wrote: post #16

I see two solid points here: 1. It's not reasonable to expect the application layer to carefully partition its work into "I/O heavy" and "CPU heavy" parts. 2. It's not reasonable to queue up an arbitrary amount of work without back-pressure. I haven't used Tokio much, but if it falls prey to these pitfalls, it would make me pause before adopting it. I think there are probably ways of using Rust async that don't fall…

I think 2 is a reasonable concern (and one which has solutions in rust async/await tokio, as FridgeSeal points out). I'm not sure about 1 though. I think having a rough idea what part of your programs are computationally expensive shouldn't be to much to ask of programmers.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#33

Post by diarrhea »

levkk wrote: post #24

This is nonsense. Tokio was built for I/O, not crunching numbers. Most programmers know to use spawn_blocking to crunch 10MB JSON, if needed. Additionally, the proposed workload per thread model will be orders of magnitude slower than Tokio for I/O-bound workloads, which most applications are.

Funnily enough, spawn_blocking is not the right tool here. It is meant for blocking I/O, such as DNS lookups, where your platform might not give you anything better.

For genuine CPU-bound work, submitting to a Rayon worker pool is the way to go. It solved a runtime starvation issue for us at work, spawn_blocking did not work.

The reason for all this is spawn_blocking having a very large underlying thread pool, in the hundreds. That is okay if you assume work will yield those threads and mostly sleep/wait. It is not okay if the work never yields, like pure data crunching. (Go solves this by forcefully preempting loops, no such thing in Rust without a language runtime)

Our solution shape was: multi-threaded Tokio (2 threads), then give the rest of available_concurrency to a Rayon thread pool. If you grant 6 vCPU you should see a thread pool of 4, and a maximum CPU consumption of about 400%, as the Tokio threads sit mostly idle (under low load single-threaded runtime should also suffice).

You inject the thread pool using an Arc.

Then, when work comes in, just spawn Tokio tasks liberally (cheap) and submit to the thread pool. Rayon will internally queue and limit concurrency and parallelism to 4 (this is the important bit compared to spawn_blocking: no way your system can hog all 6 threads with non-yielding work and starve Tokio runtime threads).

We use one-shot channels to submit results back, they are designed for exactly this. The tx aka sender end is sync, as there is never a wait (cannot block). The rx aka receiver side is async and can be awaited normally on the async side. This is a cheap operation, similar to Go.

Optionally you can reach for semaphores to also limit I/O concurrency. You probably want to do this for more control and avoiding resource exhaustion loudly (that is, not silently accidentally peg thousands of FDs, database connections, …).

It ended up working beautifully for our purposes and relatively simply. No lifetime woes, Arc solves those. Oneshot channels just transfer ownership etc.

Perhaps this is what TFA talks about, I have not read it.

One caveat: to reach all the above conclusions and designs, we had help from some genuine Rust experts. As much as I dislike Go, it "just works" there even if one writes naive code.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#34

Post by elendilm »

FridgeSeal wrote: post #30
elendilm wrote: post #22

A simple burst of memmap + soft fault with 100 or 1000 threads on a normal laptop would tell you that thread contention is real and cache locality gets destroyed. Couple that with pinned threads. You can see the latency increase by increasing thread count. Add to that the motherboard interconnect tax for numa systems. Work stealing is not the way for increasingly many workloads on modern hardware. Recently we built D…

> Increasing threads beyond system's hardware cores/threads resulted only in marginal gains of a couple of milliseconds worth of differences on huge workload with large increase in memory Careful, if you say that too loudly, the "get rid of async just spawn more threads!!!!!" people will come out of the woodwork to yell at you about how _all_ async is a lie and we should instead pretend none of it exists and just spa…

I am now trembling at the thought of warriors who will skin me alive :).

Jokes aside, there are use cases for rayon, use cases for tokio async, use cases for "may" coroutines, use cases for a custom scheduling policy, or use cases for a combination of these.

We went with "may" coroutines (with its thread pinning) + custom numa work pinning (to a may thread) due to "may's" lightweight nature and not having to have our functions colorized. We use rayon where cache locality penalty is minor compared to the millisecond latency gains due to work stealing - but this is only for an edge case.

Our log broker Monolog (akin to kafka) which sits atop Dip uses tokio async because it plays nicely with zmq while "may" doesn't.

So I am a big fan of using the right tools for the job. Each technique has its own pros and cons. Informed decisions based on use cases matter above any dogma.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#35

Post by random17 »

VorpalWay wrote: post #17
oersted wrote: post #13

Earlier quoted context omitted.

To be fair, the Rust async model itself was intentionally designed not to be prescriptive in the way you describe. You can build, and there exists, different task executors that can handle things like priority and many other execution models. Async is just a way to describe a tree of concurrent tasks that may depend on (wait on) each other at certain points. It is mostly declarative. Tokio has taken over as the defau…

Outside of Embassy in embedded, tokio is the only realistic choice though, because it is likely that any third party async crate has a dependency on it already. Yes, smol, monio, glommio etc exist, but they are marginalised (and as far as I can tell they don't really help that much with mixed IO / compute workloads). In fact, async/await in Rust falls apart with a mixed IO / compute workload since scheduling is coope…

How do you properly mix IO/compute (in any language)? In Rust what I’ve done in the past is have two Tokio runtimes, one for IO and one for compute. I know you can also use Rayon but the abstractions are not always flexible/convenient enough.

But in either case the boundary between the two types of async work is never easy to cross.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#36

Post by imtringued »

oersted wrote: post #26
VorpalWay wrote: post #17

Earlier quoted context omitted.

Outside of Embassy in embedded, tokio is the only realistic choice though, because it is likely that any third party async crate has a dependency on it already. Yes, smol, monio, glommio etc exist, but they are marginalised (and as far as I can tell they don't really help that much with mixed IO / compute workloads). In fact, async/await in Rust falls apart with a mixed IO / compute workload since scheduling is coope…

Yes I do agree. It's not that Tokio has taken over just due to community momentum. There's a certain subtle lock-in that happens due to how the Rust type system and dependency system work together. It's a hard problem to address. Regarding mixed IO/compute and preemptive scheduling, well that's what threads are. They are not as lightweight, but they are there and they are quite ergonomic to use in Rust. I could even…

I don't really know much about tokio ecosystem lock-in, but async is fundamentally an ecosystem split. In principle nothing stops you from running computationally expensive code inside an async context, it just needs to be designed to support self pre-emption. That means if you have an async JSON parser it needs to await inside long running loops or recursive functions.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#37

Post by herculity275 »

FridgeSeal wrote: post #27

This reads an awful lot like the prompter had a semi-confusing time with some async, and had their favourite model write an upset blog post about it. I don't think these systems are perfect, nor are they fit for every use-case, but some of the complaints ring a bit hollow. > fetching a database record over the network, then immediately crunching the data. But what happens when that data crunching involves parsing a 1…

This entire post was written/prompted as a pitch for the author's own concurrency framework. These types of "current state of the world - bad, my solution - good" posts have been annoying for far longer than LLMs have been around, but the LLMified over-embellishment coupled with the vagueness makes it so much worse.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#38

Post by herculity275 »

kev009 wrote: post #4

This is silly or just AI slop post? Because using a Go quote as an example of doing something right in the arena is laughable at best where it has the same problems, more magic, and worse observability. The punchline seems to be something like the LMAX disruptor style which is genuinely good for some things, but if you have I/O loops like the illustration shows you can easily block that loop with some long running fu…

It's clearly co-written by an AI

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#39

Post by rkangel »

elendilm wrote: post #22

A simple burst of memmap + soft fault with 100 or 1000 threads on a normal laptop would tell you that thread contention is real and cache locality gets destroyed. Couple that with pinned threads. You can see the latency increase by increasing thread count. Add to that the motherboard interconnect tax for numa systems. Work stealing is not the way for increasingly many workloads on modern hardware. Recently we built D…

The Erlang thing is different though - that's a userspace scheduler, scheduling userspace tasks onto one-thread-per-core. Classic M:N scheduling in a similar way to go.

It works extremely well at scale - you can handle 10k connections on a normal machine with very little thought, and WhatsApp has reported handling 1 million. Yes everything has some point at which it won't scale, and apparently that approach (or at least that implementation) struggles at 100 cores. That's not a normal machine.

Re: The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

#40

Post by elendilm »

rkangel wrote: post #39
elendilm wrote: post #22

A simple burst of memmap + soft fault with 100 or 1000 threads on a normal laptop would tell you that thread contention is real and cache locality gets destroyed. Couple that with pinned threads. You can see the latency increase by increasing thread count. Add to that the motherboard interconnect tax for numa systems. Work stealing is not the way for increasingly many workloads on modern hardware. Recently we built D…

The Erlang thing is different though - that's a userspace scheduler, scheduling userspace tasks onto one-thread-per-core. Classic M:N scheduling in a similar way to go. It works extremely well at scale - you can handle 10k connections on a normal machine with very little thought, and WhatsApp has reported handling 1 million. Yes everything has some point at which it won't scale, and apparently that approach (or at le…

It appears you are unfamiliar with "may" coroutines which is a userspace M:N scheduler.

Erlang phenomenon mentioned is the same and the behavior is easily reproducible on a normal laptop.

"may" defaults to the same number of OS threads as there are cores. Scale the set_pool_capacity() to 1000, 5000, or beyond for coroutine scaling. Also try set_workers() for OS threads.

Try it any which way and you see thread contention and cache locality penalty increasing latency.

Post reply on HN