Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Futurelock: A subtle risk in async Rust

rfd.shared.oxide.computer

Post reply on HN
Print view View original

121–130 of 251 posts

Re: Futurelock: A subtle risk in async Rust

#121

Post by csande17 »

hitekker wrote: post #62

Skimming through, this document feels thorough and transparent. Clearly, a hard lesson learned. The footnotes, in particular, caught my eye https://rfd.shared.oxide.computer/rfd/397#_external_referenc... > Why does this situation suck? It’s clear that many of us haven’t been aware of cancellation safety and it seems likely there are many cancellation issues all over Omicron. It’s awfully stressful to find out while w…

In case anyone else was confused: the link/quote in this comment are from the previous "async cancellation issue" write-up, which describes a situation where you "drop" a future: the code in the async function stops running, and all the destructors on its local variables are executed.

The new write-up from OP is that you can "forget" a future (or just hold onto it longer than you meant to), in which case the code in the async function stops running but the destructors are NOT executed.

Both of these behaviors are allowed by Rust's fairly narrow definition of "safety" (which allows memory leaks, deadlocks, infinite loops, and, obviously, logic bugs), but I can see why you'd be disappointed if you bought into the broader philosophy of Rust making it easier to write correct software. Even the Rust team themselves aren't immune -- see the "leakpocalypse" from before 1.0.

Re: Futurelock: A subtle risk in async Rust

#122

Post by clarkmcc »

tick_tock_tick wrote: post #107

It seems more and more clear every day that async was rushed out the door way to quickly in Rust.

I can’t say whether it was rushed out, but it’s clearly not everything it was advertised to be. Early on, the big talking point was that the async implementation was so modular you could swap runtimes like Lego bricks. In reality, that’s nowhere near true. Changing runtimes means changing every I/O dependency (mutexes, networking, fs), because everything is tightly coupled to the runtime. I raised this in a Reddit thread some time ago, and the feedback there reinforced that I'm not the only one with a sour Rust async taste in my mouth. https://www.reddit.com/r/rust/comments/1f4z84r/is_it_fair_to...

Re: Futurelock: A subtle risk in async Rust

#123

Post by clarkmcc »

mdasen wrote: post #104

I rewrote this in Go and it also deadlocks. It doesn't seem to be something that's Rust specific. I'm going to write down the order of events. 1. Background task takes the lock and holds it for 5 seconds. 2. Async Thing 1 tries to take the lock, but must wait for background task to release it. It is next in line to get the lock. 3. We fire off a goroutine that's just sleeping for a second. 4. Select wants to find a c…

I think the thing that rubs me the wrong way is that Rust was supposed to be "fearless" concurrency. Go doesn't claim that title so I'm not offended when it doesn't live up to it.

Re: Futurelock: A subtle risk in async Rust

#124

Post by mleonhard »

> // Start a background task that takes the lock and holds it for a few seconds.

Holding a lock while waiting for IO can destroy a system's performance. With async Rust, we can prevent this by making the MutexGuard !Send, so it cannot be held across an await. Specifically, because it is !Send, it cannot be stored in the Future [2], so it must be dropped immediately, freeing the lock. This also prevents Futurelock deadlock.

This is how I wrote safina::sync::Mutex [0]. I did try to make it Send, like Tokio's MutexGuard, but stopped when I realized that it would become very complicated or require unsafe.

> You could imagine an unfair Mutex that always woke up all waiters and let them race to grab the lock again. That would not suffer from risk of futurelock, but it would have the thundering herd problem plus all the liveness issues associated with unfair synchronization primitives.

Thundering herd is when clients overload servers. This simple Mutex has O(n^2) runtime: every task must acquire and release the mutex, which adds all waiting tasks to the scheduler queue. In practice, scheduling a task is very fast (~600ns). As long as polling the lock-mutex-future is fast and you have Performance is hard to predict. I wrote Safina using the simplest possible implementations and assumed they would be slow. Then I wrote some micro-benchmarks and found that some parts (like the async Mutex) actually outperform Tokio's complicated versions [1]. I spent days coding optimizations that did not improve performance (work stealing) or even reduced performance (thread affinity). Now I'm hesitant to believe assumptions and predictions about performance, even if they are based on profiling data.

[0] https://docs.rs/safina/latest/safina/sync/struct.MutexGuard....

[1] https://docs.rs/safina/latest/safina/index.html#benchmark

[2] Multi-threaded async executors require futures to be Send.

Re: Futurelock: A subtle risk in async Rust

#125

Post by foota »

forrestthewoods wrote: post #39

I feel like I’m pretty good at writing multithreaded code. I’ve done it a lot. As long as you use primitives like Rust Mutex that enforce correctness for data access (ie no accessing data without the lock) it’s pretty simple. Define a clean boundary API and you’re off to the races. async code is so so so much more complex. It’s so hard to read and rationalize. I could not follow this post. I tried. But it’s just a fu…

Async code isn't supposed to be simpler than sync code, it's supposed to be simpler than doing thing like continuation passing.

Re: Futurelock: A subtle risk in async Rust

#126

Post by danielheath »

mjevans wrote: post #112
kibwen wrote: post #108

Earlier quoted context omitted.

> So yeah, which of those concerns was async Rust really designed around? All of them I guess? Yes, all of them. Futures needed to work on embedded platforms (so no allocation), needed to be highly optimizable (so no virtual dispatch), and need to act reasonably in the presence of code that crosses FFI boundaries (so no stack shenanigans). Once you come to terms with these constraints--and then add on Rust's other pr…

In my view, the major design sin was not _forcing_ failure into the outcome list. .await(DEADLINE) (where deadline is any non 0 unit, and 0 is 'reference defined' but a real number) should have been the easy interface. Either it yields a value or it doesn't, then the programmer has to expressly handle failure. Deadline would only be the minimum duration after which the language, when evaluating the future / task, wou…

Does that imply a lot of syscalls to get the monotonic clock value? Or is there another way to do that?

Re: Futurelock: A subtle risk in async Rust

#128

Post by mjevans »

danielheath wrote: post #126
mjevans wrote: post #112

Earlier quoted context omitted.

In my view, the major design sin was not _forcing_ failure into the outcome list. .await(DEADLINE) (where deadline is any non 0 unit, and 0 is 'reference defined' but a real number) should have been the easy interface. Either it yields a value or it doesn't, then the programmer has to expressly handle failure. Deadline would only be the minimum duration after which the language, when evaluating the future / task, wou…

Does that imply a lot of syscalls to get the monotonic clock value? Or is there another way to do that?

If the scheduler is doing _any_ sort of accounting at all to figure out any remote sort of fairness balancing at all, then whatever resolution that is probably works.

At least for Linux, offhand, popular task scheduler frequencies used to be 100 and 1000hz.

Looks like the Kernel's tracking that for tasks:

https://www.kernel.org/doc/html/latest/scheduler/sched-desig...

"In CFS the virtual runtime is expressed and tracked via the per-task p->se.vruntime (nanosec-unit) value."

I imagine the .vruntime struct field is still maintained with the newer "EEVDF Scheduler".

...

A Userspace task scheduler could similarly compare the DEADLINE against that runtime value. It would still reach that deadline after the minimum wait has passed, and thus be 'background GCed' at a time of the language's choice.

Re: Futurelock: A subtle risk in async Rust

#129

Post by smallstepforman »

embedding-shape wrote: post #113
rtpg wrote: post #97

Earlier quoted context omitted.

I guess one big question here is whether there's a higher layer abstraction that is available to wrap around patterns to avoid this. It does feel like there's still generally possibilities of deadlocks in Rust concurrency right? I understand the feeling here that it feels like ... uhh... RAII-style _something_ should be preventing this, because it feels like statically we should be able to identify this issue in this…

> I guess one big question here is whether there's a higher layer abstraction that is available to wrap around patterns to avoid this. Something like Actors, on top of Tokio, would be one way: https://ryhl.io/blog/actors-with-tokio/

I love Actors and have used them professionally for over 6 years (C++). However to solve real world problems I have had to introduce “locks” to the Actor framework to support various scenarios. With my home-grown actor library, this was trivial to add, however for some 3rd party actor libraries, ideology is dominant and the devs refuse to add such a purity-breaking feature to their actor framework, and hence I cannot use their library for real-world code.
Post reply on HN