Futurelock: A subtle risk in async Rust • Hacker News Board

Re: Futurelock: A subtle risk in async Rust

#191

Post by jerf » Sat, Nov 01, 2025, 1:52 PM UTC

johnisgood wrote: post #150

jerf wrote: post #79

Earlier quoted context omitted.

Erlang/Elixir and Go "solve" this problem by basically not giving you the rope to hang yourself in this particular way in the first place. This is a perfectly valid and sensible solution... but it is not the only solution. It means you're paying for some relatively expensive full locks that the Rust async task management is trying to elide, for what can be quite significant performance gains if you're doing a lot of…

> It means you're paying for some relatively expensive full locks that the Rust async task management is trying to elide, for what can be quite significant performance gains if you're doing a lot of small tasks. The point of Erlang/Elixir is that it is as performant as possible, and Erlang's history is a testament to this. BEAM is wonderful, and really fast, along with the languages on it being ergonomic (OTP behavio…

This is a myth, from the old days when BEAM was the only thing that could juggle thousands of "processes" without losing performance, and even back then, people routinely missed that while BEAM could juggle those thousands of processes, each of them was individually not that fast. That is, BEAM's extremely high performance was only in one isolated thing, not high performance across the board.

Now BEAM is far from the only runtime juggling that many processes, but it remains a relatively slow VM. I rule-of-thumb it at 10x slower than C, making it a medium performance VM at best, and you want to watch your abstraction layers in those nicer languages like Gleam because further multiplicative slow downs can really start to bite.

The first serious Go program I wrote was a replacement for something written in Erlang, there was no significant architectural improvement in the rewrite (it was already reasonably well-architected), and from the first deployment, we went from 4 systems, sometimes struggling with the load spikes, to where just one could handle it all, even with BEAM being over a decade more mature and the Go clustering code being something I wrote over a few weeks rather than battle tested, optimized code.

BEAM is good at managing concurrency, but it is slowish in other ways. It's better than the dynamic scripting languages like Python by a good amount but it is not performance-competitive with a modern compiled language.

Re: Futurelock: A subtle risk in async Rust

#192

Post by kibwen » Sat, Nov 01, 2025, 2:13 PM UTC

tux3 wrote: post #190

kibwen wrote: post #173

Earlier quoted context omitted.

> All the Tokio primitives involved were working precisely as they were supposed to. The only thing that would have prevented this without completely re-designing Rust's async from the ground up would be to ban the use of `&mut future`s in `select!`...but that eliminates a lot of correct code, too. But it still suggests that `tokio::select` is too powerful. You don't need to get rid of `tokio::select`, you just need…

I feel like select!() is a good case study because the common future timeout use-case maps pretty closely to a select!(), so there is only so much room to weaken it. The ways I can think of for making select!() safer all involve runtime checks and allocations (possibly this is just a failure of my imagination!). But if that's the case, I would find it bothersome if our basic async building blocks like select/timeout…

It feels to me like there's plenty of design space to explore. Sure, it's possible to view "selection" as a basic building block, but even that is insufficiently precise IMO. There's a reason that Javascript provides all of Promise.any and Promise.all and Promise.allSettled and Promise.race. Selection isn't just a single building block, it's an entire family of building blocks with distinct semantics.

Re: Futurelock: A subtle risk in async Rust

#193

Post by duped » Sat, Nov 01, 2025, 2:15 PM UTC

quietbritishjim wrote: post #90

duped wrote: post #86

Earlier quoted context omitted.

So there's a distinction between a task and a future. A future doesn't do anything until it's polled, and since there's nothing special about async runtimes (it's just user level code), it's always possible to create futures and never poll them, or stop polling them. A task is a different construct and usually tied to the runtime. If you look at the suggestions in the RFD they call out using a task explicitly instead…

Interesting, thanks. So is it fair to say that if tokio::select!() only accepted tasks (or implicitly turned any futures it receives into tasks, like Python's asyncio.gather() does) then it wouldn't have this problem? Or, even if the async runtime is careful, is it still possible to create and fail to poll a raw Future by accident?

It's always possible to create a future that is never polled, and this is a feature of Rusts zero cost abstraction for async/await. If tokio::select required tasks it would be a lot less useful.

This problem would have been avoided by taking the future by value instead of by reference.

Re: Futurelock: A subtle risk in async Rust

#194

Post by dap » Sat, Nov 01, 2025, 2:19 PM UTC

imtringued wrote: post #184

dvratil wrote: post #159

Earlier quoted context omitted.

I would guess this is just to make the explanation of the bug easier. In real world, the futurelock could occur even with very short locks, it just wouldn't be so deterministic. Having a minimal reproducer that you have to run a thousand times and it will maybe futurelock doesn't really make for a good example :)

>In real world, the futurelock could occur even with very short locks, it just wouldn't be so deterministic. You have to explain the problem properly then. The problem here has nothing to do with duration whatsoever so don't bring that up. The problem here is that if you acquire a lock, you're inside a critical section. Critical sections have a programming paradigm that is equivalent to writing unsafe Rust. You're no…

I think you misunderstand the problem. The only purpose of the sleep in this example is to control interleaving of execution to ensure the problem happens. Here's a version where the background task (the initial lock holder) only runs a bounded number of instructions with the lock held, just as you suggest:

https://play.rust-lang.org/?version=stable&mode=debug&editio...

It still futurelocks.

> After that process has finished, you release the lock. Then you return to the scheduler and execute the next future. The next future cannot be blocked because the lock has already been released. It's simply impossible.

This is true with threads and with tasks that only ever poll futures sequentially. It is not true in the various cases mentioned in this RFD (notably `tokio::select!`, but also others). Intuitively: when you have one task polling on multiple futures concurrently, you're essentially adding another layer to the scheduler (kernel thread scheduler, tokio task scheduler, now some task is acting as its own future scheduler). The problem is it's surprisingly easy to (1) not realize that and (2) accidentally have that "scheduler" not poll the next runnable future and then get stuck, just like if the kernel scheduler didn't wake up a runnable thread.

Re: Futurelock: A subtle risk in async Rust

#195

Post by dap » Sat, Nov 01, 2025, 2:24 PM UTC

jcalvinowens wrote: post #131

I have very little Rust experience... but I'm hung up on this: > The lock is given to future1 > future1 cannot run (and therefore cannot drop the Mutex) until the task starts running it. This seems like a contradiction to me. How can future1 acquire the Mutex in the first place, if it cannot run? The word "given" is really odd to me. Why would do_async_thing() not immediately run the prints, return, and drop the lock…

Your confusion is very natural:

> It's more typical in my experience that the act of granting the lock to a thread is what makes it runnable, and it runs right then.

This gets at why this felt like a big deal when we ran into this. This is how it would work with threads. Tasks and futures hook into our intuitive understanding of how things work with threads. (And for tasks, that's probably still a fair mental model, as far as I know.) But futures within a task are different because of the inversion of control: tasks must poll them for them to keep running. The problem here is that the task that's responsible for polling this future has essentially forgotten about it. The analogous thing with threads would seem to be something like if the kernel forgot to enqueue some runnable thread on a run queue.

Re: Futurelock: A subtle risk in async Rust

#196

Post by kibwen » Sat, Nov 01, 2025, 2:25 PM UTC

tick_tock_tick wrote: post #111

kibwen wrote: post #109

Earlier quoted context omitted.

There's a lot of improvements I could think of for async Rust, but there's basically nothing I would change about the fundamentals that underlie it (other than some tweaks to Pin, maybe, and I could quibble over some syntax). There's nothing rushed about it; it's a great foundation that demonstrably just needs someone to finish building the house on top of it (and, to continue the analogy, needs someone to finish bui…

A foundation full of warts belongs in experimental. I don't know how by your own confession of the house and the sub-basement not yet being finished doesn't instantly mean it should have stayed in experimental.

Your assertion is that it was "rushed". And yet here we are today, talking about how much we wish were implemented. That's not rushed--that's the polar opposite of rushed. Almost nothing about what we currently have on stable would have been better if it was still percolating on nightly, and would have the downside of having almost no feedback from real-world use. I remember the pre-async days, nesting callbacks by hand. What we have now is a great improvement, and just needs more niceties stacked on top of it, not any sort of fundamental overhaul.

Re: Futurelock: A subtle risk in async Rust

#197

Post by dap » Sat, Nov 01, 2025, 2:30 PM UTC

lowbloodsugar wrote: post #153

I know this is going to sound trite, but “don’t do that”. It’s no different than deciding to poll the win32 event queue inside an a method you executed in response to polling the event queue. Nested shit is always going to cause a bug. I guess each new generation just has to learn.

Don't do ... what, exactly? The RFD answers this more precisely and provides suggestions for alternatives. But it's not very simple because the things that can cause this are all common patterns individually and it's only the confluence (which can be spread across layers of the program) that introduces this problem. In our case, it wasn't a Mutex, but an mpsc channel (that was working correctly! it just got very briefly saturated) and it was 3-4 modules lower in the stack than the code with the `tokio::select!` that induced this.

Re: Futurelock: A subtle risk in async Rust

#198

Post by dap » Sat, Nov 01, 2025, 2:40 PM UTC

imtringued wrote: post #174

Based on the description: >This RFD describes futurelock: a type of deadlock where a resource owned by Future A is required for another Future B to proceed, while the Task responsible for both Futures is no longer polling A. Futurelock is a particularly subtle risk in writing asynchronous Rust. I was honestly wondering how you could possibly cause this in any sane code base. How can an async task hold a lock and keep…

> So how exactly did they bring about the impossible? They put an await call inside the critical section. The part of the code base that is not allowed to be subject to arbitrary delays. Massive facepalm.

I'm not sure where you got the impression that the example code was where we found the problem. That's a minimal reproducer trying to explain the problem from first principles because most people look at that code and think "that shouldn't deadlock". It uses a Mutex because people are familiar with Mutexes and `sleep` just to control the interleaving of execution. The RFD shows the problem in other examples without Mutexes. Here's a reproducer that futurelocks even though nobody uses `await` with the lock held: https://play.rust-lang.org/?version=stable&mode=debug&editio...

> I was honestly wondering how you could possibly cause this in any sane code base.

The actual issue is linked at the very top of the RFD. In our cases, we had a bounded mpsc channel used to send messages to an actor running in a separate task. That actor was working fine. But the channel did become briefly saturated (i.e., at capacity) at a point where someone tried to send on it via a `tokio::select!` similar to the one in the example.

Re: Futurelock: A subtle risk in async Rust

#199

Post by dboreham » Sat, Nov 01, 2025, 2:42 PM UTC

singron wrote: post #30

This sounds very similar to priority inversion. E.g. if you have Thread T_high running at high priority and thread T_low running at low priority, and T_low holds a lock that T_high wants to acquire, T_high won't get to run until T_low gets scheduled. The OS can detect this and make T_low "inherit" the priority of T_high. I wonder if there is a similar idea possible with tokio? E.g. if you are awaiting a Mutex held by…

Which is why "async" is a pox on our house. System threads can and do address these edge issues. User level concurrency generally doesn't (perhaps with the exceptions of golang and erlang).

Re: Futurelock: A subtle risk in async Rust

#200

Post by thenewwazoo » Sat, Nov 01, 2025, 2:47 PM UTC

IshKebab wrote: post #169

rtpg wrote: post #97

Earlier quoted context omitted.

I guess one big question here is whether there's a higher layer abstraction that is available to wrap around patterns to avoid this. It does feel like there's still generally possibilities of deadlocks in Rust concurrency right? I understand the feeling here that it feels like ... uhh... RAII-style _something_ should be preventing this, because it feels like statically we should be able to identify this issue in this…

The Fuchsia guys use the trait system to enforce a global mutex locking order, which can statically prevent deadlocks due to two threads locking mutexes that they are both waiting for. Doesn't help in this case, but it does suggest that we might be able to do better.

Any chance you could dig up a link to that code? I’m curious to learn more