Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Futurelock: A subtle risk in async Rust

rfd.shared.oxide.computer

Post reply on HN
Print view View original

231–240 of 251 posts

Re: Futurelock: A subtle risk in async Rust

#231

Post by ufmace »

ufmace wrote: post #209
mleonhard wrote: post #124

> // Start a background task that takes the lock and holds it for a few seconds. Holding a lock while waiting for IO can destroy a system's performance. With async Rust, we can prevent this by making the MutexGuard !Send, so it cannot be held across an await. Specifically, because it is !Send, it cannot be stored in the Future [2], so it must be dropped immediately, freeing the lock. This also prevents Futurelock dea…

Considering this issue did also make me think - maybe the real footgun here is the async mutex. I think a better "rule" to avoid this issue might be something like, don't just use the tokio async mutex by default just because it's there and you're in an async function; instead default to a sync mutex that errors when held across awaits and think very hard about what you're really doing before you switch to the async…

Actually I think I might be a little misguided here - confusing a mutex with an awaitable lock method versus blocking, and a mutex whose LockGuard is Send and can be held across other await points.

To clarify, I do still think it's probably wise to prefer using a mutex whose LockGuard is not Send. If you're in an async context though, it seems clearly preferable to use a mutex that lets you await on lock instead of possibly blocking. Looks like that's what that Safina gives you.

It does bring to mind the point though - does it really make sense to call all of these things Mutexes? Most Mutexes, including the one in std, seem relatively simplistic, with no provision for exactly what happens if multiple threads/tasks are waiting to acquire the lock. As if they're designed for the case of, it's probably rare to never for multiple threads to actually need this thing at once, but we have to guard against it just to be certain. The case of this resource is in high demand by a bunch of threads, we expect there to be a lot of time spent by a lot of threads waiting to get the lock, so it's actually important which lock requesters actually get the lock in what order, seems different enough that it maybe ought to have a different name and more flexibility and selection as to what algorithm is being used to control the lock order.

Re: Futurelock: A subtle risk in async Rust

#232

Post by mechanical_berk »

Sytten wrote: post #3

I am wondering if there is a larger RFC for Rust to force users to not hold a variable across await points. In my mind futurelock is similar to keeping a sync lock across an await point. We have nothing right now to force a drop and I think the solution to that problem would help here.

I agree. It seems like this bug arises because one Future is awaited while another is ignored. I have seen this sort of bug a lot.

So maybe all that is needed is a lint that warns if you keep a Future (or a reference to one) across an await point? The Future you are awaiting wouldn't count of course. Is there some case where this doesn't work?

Re: Futurelock: A subtle risk in async Rust

#233

Post by levodelellis »

jstimpfle wrote: post #177
levodelellis wrote: post #146

Earlier quoted context omitted.

People mess up the order all the time. When you mess up locks you get a deadlock, when you mess up an atomic you have items in the queue drop or processed twice, or some other weird behavior (waking up the wrong thread) you didn't expect. You just get hard to understand race conditions which are always a pain to debug Just say no to atomics (unless they're hidden in a well written library)

People are messing up any number of things all the time. The more important question is, do you need to risk messing up in a particular situation? I.e. do you need multithreading? In many cases, for example HPC or GUI programming, the answer is yes, you need multithreading to avoid blocking and to get much higher performance. With a little bit of experience and a bit of care, multithreading isn't _that_ hard. You jus…

I completely disagree with you and I do write performance code (but not specifically HPC) and my current day job is highly async code with a GUI

Re: Futurelock: A subtle risk in async Rust

#234

Post by jacquesm »

yxhuvud wrote: post #175
jacquesm wrote: post #142

Earlier quoted context omitted.

Indeed, in Erlang the budget is counted in 'reductions'. Technically Erlang uses the BEAM as a CPU with some nifty extra features which allow you to pretend that you are pre-empting a process when in fact it is the interpreter of the bytecode that does the work and there are no interrupts involved. Erlang would not be able to do this if the Erlang input code was translated straight to machine instructions. But Go doe…

Fickle? Pray tell, when the OS switch your thread for another thread, in what way does that fickleness show?

I take it you've never actually interfaced directly with hardware?

Interrupts are at the most basic level an electrical signal to the CPU to tell it to load a new address into the next instruction pointer after pushing the current one and possibly some other registers onto the stack. That means you don't actually know when they will happen and they are transparent to the point that those two instructions that you put right after one another are possibly detoured to do an unknown amount of work in some other place.

Any kind of side effect from that detour (time spent, changes made to the state of the machine) has the potential to screw up the previously deterministic path that you were on.

To make matters worse, there are interrupts that can interrupt the detour in turn. There are ways in which you can tell the CPU 'not now' and there are ways in which those can be overridden. If you are lucky you can uniquely identify the device that caused the interrupt to be triggered. But this isn't always the case and given the sensitivity of the inputs involved it isn't rare at all that your interrupt will trigger without any ground to do so. If that happens and the ISR is not written with that particular idea in mind you may end up with a system in an undefined state.

Interrupts are a very practical mechanism. But they're also a nightmare to deal with in the otherwise orderly affairs of computing and troubleshooting interrupt related issues can eat up days, weeks or even months if you are really unlucky.

Re: Futurelock: A subtle risk in async Rust

#235

Post by jstimpfle »

levodelellis wrote: post #233
jstimpfle wrote: post #177

Earlier quoted context omitted.

People are messing up any number of things all the time. The more important question is, do you need to risk messing up in a particular situation? I.e. do you need multithreading? In many cases, for example HPC or GUI programming, the answer is yes, you need multithreading to avoid blocking and to get much higher performance. With a little bit of experience and a bit of care, multithreading isn't _that_ hard. You jus…

I completely disagree with you and I do write performance code (but not specifically HPC) and my current day job is highly async code with a GUI

Are you saying GUIs in general don't need multithreading or just that you think you haven't needed it so far? Or that you use some high-level async framework that hides just the synchronisation bits (at the cost of async type complexity)?

Re: Futurelock: A subtle risk in async Rust

#236

Post by jstimpfle »

levodelellis wrote: post #91
umvi wrote: post #87

Earlier quoted context omitted.

> I always said if your code locks or use atomics, it's wrong. Everyone says I'm wrong but you get things like what's described in the article. Ok so say you are simulating high energy photons (x-rays) flowing through a 3d patient volume. You need to simulate 2 billion particles propagating through the patient in order to get an accurate estimation of how the radiation is distributed. How do you accomplish this witho…

From my understanding this talk describes how he implemented a solution for a similar problem https://www.youtube.com/watch?v=Kvsvd67XUKw I'm saying if you're not writing multi-threaded code everyday, use a library. It can use atomics/locks but you shouldn't use it directly. If the library is designed well it'd be impossible to deadlock.

If you take programming serious, learn it.

With a library that encapsulates a low number of patterns (like message passing) you'll be very limited. If you never start learning about lower level multi-threading issues you'll never learn it. And it's not _that_ hard.

I'm not writing multi threaded every day (by far), but often enough that I can write useful things (using shared memory, atomics, mutexes, condition variables, etc). And I'm looking forward to learn more, better understand various issues, learn new patterns.

Re: Futurelock: A subtle risk in async Rust

#237

Post by vlovich123 »

Rusky wrote: post #204
vlovich123 wrote: post #135

Earlier quoted context omitted.

If it’s not inert, how do you use async in the kernel or microcontrollers? A non-inert implementation presumes a single runtime implementation within std+compiler and not usable in environments where you need to implement your own meaning of dispatch.

"Not inert" does not at all imply "a single runtime within std+compiler." You've jumped way too far in the opposite direction there. The problem is that the particular interface Rust chose for controlling dispatch is not granular enough. When you are doing your own dispatch, you only get access to separate tasks, but for individual futures you are at the mercy of combinators like `select!` or `FuturesUnordered` that…

If you’ve got eager dispatch I’m eager (pun intended) to learn how you have an executor that’s not baked into the std library and limited to a single runtime per process because at the time of construction you need the language to schedule dispatch of the created future. This is one of the main challenges behind the pluggable executor effort - the set of executors that could be written is so different (work stealing vs thread per core) that it’s impossible to unify without an effect system and even then you’ve got challenges of how to encode that in the language structure because the executor is a global thing determined at runtime but then it’s also local in the sense that you don’t know which executor a given piece of code will end up actually being dispatched into since you could have the same async function invoked on different executors.

For better or worse eager dispatch I think generally implies also not being able to cancel futures since ownership is transferred to the executor rather than being retained by your code.

Re: Futurelock: A subtle risk in async Rust

#238

Post by johnisgood »

jerf wrote: post #191
johnisgood wrote: post #150

Earlier quoted context omitted.

> It means you're paying for some relatively expensive full locks that the Rust async task management is trying to elide, for what can be quite significant performance gains if you're doing a lot of small tasks. The point of Erlang/Elixir is that it is as performant as possible, and Erlang's history is a testament to this. BEAM is wonderful, and really fast, along with the languages on it being ergonomic (OTP behavio…

This is a myth, from the old days when BEAM was the only thing that could juggle thousands of "processes" without losing performance, and even back then, people routinely missed that while BEAM could juggle those thousands of processes, each of them was individually not that fast. That is, BEAM's extremely high performance was only in one isolated thing, not high performance across the board. Now BEAM is far from the…

Go's runtime is indeed faster for CPU-bound and raw throughput scenarios, but it lacks the same fault-tolerance semantics, hot-code reloading, and actor-level isolation - things that make BEAM indispensable in telecoms, messaging systems, and fault-resilient distributed architectures. Go may run faster, but Erlang and Elixir recover faster and fail more gracefully, which in distributed systems often matters more than raw speed. Go might win on throughput, but the BEAM wins on fault-tolerant system design - and that's why systems built on it keep running reliably for decades.

Additionally, high-performance Erlang/Elixir systems often delegate compute-intensive work to NIFs (native implemented functions) in C or Rust, or use ports to external services.

Re: Futurelock: A subtle risk in async Rust

#239

Post by quadhome »

mycoliza wrote: post #34
dvt wrote: post #14

> &mut future1 is dropped, but this is just a reference and so has no effect. Importantly, the future itself (future1) is not dropped. There's a lot of talk about Rust's await implementation, but I don't really think that's the issue here. After all, Rust doesn't guarantee convergence. Tokio, on the other hand (being a library that handles multi-threading), should (at least when using its own constructs, e.g. the `se…

What could `tokio::select!` do differently here to prevent bugs like this? In the case of `select!`, it is a direct consequence of the ability to poll a `&mut` reference to a future in a `select!` arm, where the future is not dropped should another future win the "race" of the select. This is not really a choice Tokio made when designing `select!`, but is instead due to the existence of implementations of `Future` fo…

It's desirable to be able to express the idea that we want to continually poll drive one asynchronous operation to completion while periodically checking if some other thing has happened and taking action based on that, and then continue driving forward the ongoing operation.

This idea may be desirable; but, a deadlock is possible if there's a dependency between the two operations. The crux is the "and then continue," which I'm taking to mean that the first operation is meant to pause whilst the second operation occurs. The use of `&mut` in the code specifically enables that too.

If it's OK for the first operation to run concurrently with the other thing, then wrt. Tokio's APIs, have you seen LocalSet[1]? Specifically:

    let local = LocalSet::new();
    local.spawn_local(async move {
        sleep(Duration::from_millis(500)).await;
        do_async_thing("op2", lock.clone()).await;
    });
    local.run_until(&mut future1).await;
This code expresses your idea under a concurrent environment that resolves the deadlock. However, `op2` will still never acquire the lock because `op1` is first in the queue. I strongly suspect that isn't the intended behaviour; but, it's also what would have happened if the `select!` code had worked as imagined.

[1] https://docs.rs/tokio/latest/tokio/task/struct.LocalSet.html

Re: Futurelock: A subtle risk in async Rust

#240

Post by vacuity »

octoberfranklin wrote: post #52

For anybody who wants to cut to the chase, it's this: > The behavior of tokio::select! is to poll all branches' futures only until one of them returns `Ready`. At that point, it drops the other branches' futures and only runs the body of the branch that’s ready. This is, unfortunately, doing what it's supposed to do: acting as a footgun. The design of tokio::select!() implicitly assumes it can cancel tasks cleanly by…

> The state-machine transformation inside rustc is a thing of beauty, but the libraries and APIs layered on top of that should have been iterated many more times before being rolled out into widespread use.

Quite. The mistake was advertising the feature as production-ready (albeit not complete), which led the ecosystem to adopt async faster than it could work out the flaws and footguns. The nice thing about the language/library split is that there is a path to good async Rust without major backwards-compatibility-breaking language changes (probably...), because the core concepts are sound, but the library ecosystem will bear the necessary brunt of evolution.

Post reply on HN