Earlier quoted context omitted.
The Fuchsia guys use the trait system to enforce a global mutex locking order, which can statically prevent deadlocks due to two threads locking mutexes that they are both waiting for. Doesn't help in this case, but it does suggest that we might be able to do better.
Any chance you could dig up a link to that code? I’m curious to learn more
Futurelock: A subtle risk in async Rust
201–210 of 251 posts
Re: Futurelock: A subtle risk in async Rust
#202A masterclass in debugging
Re: Futurelock: A subtle risk in async Rust
#203Earlier quoted context omitted.
async rust continues to strike me as half-baked and too complex, if you’re developing an application (as opposed to some high performance utility like e.g. a data plane component) just use threads, they’re plenty cheap and not even half as messy.
Async Rust is as complex as it needs to be given its constraints. But I wholeheartedly agree with you that people need to treat threads (especially scoped ones) as the default concurrency primitive. My intuition is that experience with other languages has led people astray; in most languages threads are a nightmare and/or async is the default or only way to achieve concurrency, but threads in Rust are absolutely divi…
In the spirit of "every non-trivial program will expand until ...", I think preemptively choosing async for anything much more complex than a throwaway script might be justified. In this case, the relevant thing isn't performance or expected number of concurrent users/connections, but whether the program is likely to become or include a non-trivial state machine. My primary influence on this topic is this post from @sunshowers: https://sunshowers.io/posts/nextest-and-tokio/
Re: Futurelock: A subtle risk in async Rust
#204Earlier quoted context omitted.
The requirement is that the futures are not separate heap allocations, not that they are inert. It's not at all obvious that Rust's is the only possible design that would work here. I strongly suspect it is not. In fact, early Rust did some experimentation with exactly the sort of stack layout tricks you would need to approach this differently. For example, see Graydon's post here about the original implementation of…
If it’s not inert, how do you use async in the kernel or microcontrollers? A non-inert implementation presumes a single runtime implementation within std+compiler and not usable in environments where you need to implement your own meaning of dispatch.
"Not inert" does not at all imply "a single runtime within std+compiler." You've jumped way too far in the opposite direction there.
The problem is that the particular interface Rust chose for controlling dispatch is not granular enough. When you are doing your own dispatch, you only get access to separate tasks, but for individual futures you are at the mercy of combinators like `select!` or `FuturesUnordered` that only have a narrow view of the system.
A better design would continue to avoid heap allocations and allow you to do your own dispatch, but operate in terms of individual suspended leaf futures. Combinators like `join!`/`select!`/etc. would be implemented more like they are in thread-based systems, waiting for sub-tasks to complete, rather than being responsible for driving them.
Re: Futurelock: A subtle risk in async Rust
#205Earlier quoted context omitted.
This one is relevant because it avoids heap allocation while running the iterator and for loop body concurrently. Which is exactly the kind of thing that `async` does.
It avoids heap allocation in some situations. But in principle the exact same optimization could be done for stackful coroutines. Heck, right now in C I could stack-allocate an array and pass it to pthread_create as the stack for a new thread. To avoid an overlarge allocation I would need to know exactly how much stack is needed, but this is exactly the knowledge the Rust compiler already requires for async/await. Wh…
> in principle the exact same optimization could be done for stackful coroutines.
Yes, I totally agree, and this is sort of what I imagine a better design would look like.
> One of the reasons Rust does it the way it currently does is because the implementation avoids requiring support from, e.g., LLVM
This I would argue is simply a failure of imagination. All you need from the LLVM layer is tail calls, and then you can manage the stack layout yourself in essentially the same way Rust manages Future layout.
You don't even need arbitrary tail calls. The compiler can limit itself to the sorts of things LLVM asks for- specific calling convention, matching function signatures, etc. when transferring control between tasks, because it can store most of the state in the stack that it laid out itself.
Re: Futurelock: A subtle risk in async Rust
#206I have very little Rust experience... but I'm hung up on this: > The lock is given to future1 > future1 cannot run (and therefore cannot drop the Mutex) until the task starts running it. This seems like a contradiction to me. How can future1 acquire the Mutex in the first place, if it cannot run? The word "given" is really odd to me. Why would do_async_thing() not immediately run the prints, return, and drop the lock…
Your confusion is very natural: > It's more typical in my experience that the act of granting the lock to a thread is what makes it runnable, and it runs right then. This gets at why this felt like a big deal when we ran into this. This is how it would work with threads. Tasks and futures hook into our intuitive understanding of how things work with threads. (And for tasks, that's probably still a fair mental model,…
> tasks must poll them for them to keep running.
So async Rust introduces an entire novel class of subtle concurrent programming errors? Ugh, that's awful.
> The analogous thing with threads would seem to be something like if the kernel forgot to enqueue some runnable thread on a run queue.
Yes. But I've never written code in a preemptible protected mode environment like Linux userspace where it is possible to make that mistake. That's nuts to me.
From my POV this seems like a fundamental design flaw in async rust. Like, on a bare metal thing I expect to deal with stuff like this... but code running on a real OS shouldn't have to.
Re: Futurelock: A subtle risk in async Rust
#207I have very little Rust experience... but I'm hung up on this: > The lock is given to future1 > future1 cannot run (and therefore cannot drop the Mutex) until the task starts running it. This seems like a contradiction to me. How can future1 acquire the Mutex in the first place, if it cannot run? The word "given" is really odd to me. Why would do_async_thing() not immediately run the prints, return, and drop the lock…
> This seems like a contradiction to me. How can future1 acquire the Mutex in the first place, if it cannot run? The word "given" is really odd to me. `future1` did run for a bit, and it got far enough to acquire the mutex. (As the article mentioned, technically it took a position in a queue that means it will get the mutex, but that's morally the same thing here.) Then it was "paused". I put "paused" in scare quotes…
So my understanding was correct, it requires the programmer to deal with scheduling explicitly in userspace.
If I'm writing bare metal code for e.g. a little cortex M0, I can very much see the utility of this abstraction.
But it seems like an absolutely absurd exercise for code running in userspace on a "real" OS like Linux. There should be some simpler intermediate abstraction... this seems like a case of forcing a too-complex interface on users who don't really require it.
Re: Futurelock: A subtle risk in async Rust
#208Earlier quoted context omitted.
Interesting, thanks. So is it fair to say that if tokio::select!() only accepted tasks (or implicitly turned any futures it receives into tasks, like Python's asyncio.gather() does) then it wouldn't have this problem? Or, even if the async runtime is careful, is it still possible to create and fail to poll a raw Future by accident?
> if tokio::select!() only accepted tasks (or implicitly turned any futures it receives into tasks, like Python's asyncio.gather() does) then it wouldn't have this problem? Yes, this is correct. However, many of the use cases for select rely on the fact that it doesn't run all the tasks to completion. I've written many a select! statement to implements timeouts or other forms of intentionally preempting a task. Somet…
> However, many of the use cases for select rely on the fact that it doesn't run all the tasks to completion.
That seems like a different requirement than "all arguments are tasks". If I understand it right (and quite possibly I don't), making them all tasks means that they are all polled and therefore continue progressing until they are dropped. It doesn't mean that select! would have to run them all the way to completion.
Re: Futurelock: A subtle risk in async Rust
#209> // Start a background task that takes the lock and holds it for a few seconds. Holding a lock while waiting for IO can destroy a system's performance. With async Rust, we can prevent this by making the MutexGuard !Send, so it cannot be held across an await. Specifically, because it is !Send, it cannot be stored in the Future [2], so it must be dropped immediately, freeing the lock. This also prevents Futurelock dea…
Considering this issue did also make me think - maybe the real footgun here is the async mutex. I think a better "rule" to avoid this issue might be something like, don't just use the tokio async mutex by default just because it's there and you're in an async function; instead default to a sync mutex that errors when held across awaits and think very hard about what you're really doing before you switch to the async one.
Re: Futurelock: A subtle risk in async Rust
#210Earlier quoted context omitted.
> This seems like a contradiction to me. How can future1 acquire the Mutex in the first place, if it cannot run? The word "given" is really odd to me. `future1` did run for a bit, and it got far enough to acquire the mutex. (As the article mentioned, technically it took a position in a queue that means it will get the mutex, but that's morally the same thing here.) Then it was "paused". I put "paused" in scare quotes…
So my understanding was correct, it requires the programmer to deal with scheduling explicitly in userspace. If I'm writing bare metal code for e.g. a little cortex M0, I can very much see the utility of this abstraction. But it seems like an absolutely absurd exercise for code running in userspace on a "real" OS like Linux. There should be some simpler intermediate abstraction... this seems like a case of forcing a…
There is one: tasks. But having the lower level (futures) available makes it very tempting to use it, both for performance and because the code is simpler (at least, it looks simpler). Some things that are easy with select! would be clunky with tasks.
On the other hand, some direct uses of futures are reminiscent of the tendency to obsess over ownership and borrowing to maximize sharing, when you could just use .clone() and it wouldn’t make any practical difference. Because Rust is so explicit, you can see the overhead so you want to minimize it.