Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async hazard: MMAP is blocking IO

huonw.github.io

Post reply on HN
Print view View original

71–80 of 119 posts

Re: Async hazard: MMAP is blocking IO

#71

Post by dbaupp »

dataflow wrote: post #8
guerrilla wrote: post #6

Earlier quoted context omitted.

GP is aware. mmap makes files act like memory. Memory is always synchronous, thus blocking, so mmaped files are always blocking. I'm surprised OP even found this surprising. It should be completely obvious.

At first I thought the title meant the mmap() call itself blocks, which I figured could be slightly surprising. But it seems they're referring to I/O on the mapped file? I'm also baffled, how could it possibly not block?

Yes, it couldn't not block, it's obvious... but I've encountered non-trivial amounts of "magical thinking" around both async/await ("go-fast juice") and mmap ("go-fast juice") separately, so the intersection surely has a bunch of magical thinking too, where people haven't taken the time to properly think through what's going on.

Hence, my investigation to try to make the "obvious" conclusion obvious to more people.

(Author here)

Re: Async hazard: MMAP is blocking IO

#72

Post by dbaupp »

nemetroid wrote: post #37
danarmak wrote: post #23

Earlier quoted context omitted.

No, if the memory-mapped page you're accessing is in RAM, then you're just reading the RAM; there is no page fault and no syscall and nothing blocks. You could say that any non-register memory access "blocks" but I feel that's needlessly confusing. Normal async code doesn't "block" in any relevant sense when it accesses the heap.

So what is the definition of "blocking" here? That it takes more than 1 µs?

As the author, I don't think there's a clear definition of "blocking" in this space, other some vibes about an async task not switching back to the executor for too long, for some context-dependent definition of "too long".

It's all fuzzy and my understanding is that what one use-case considers being blocked for too long might be fine for another. For instance, a web server trying to juggle many requests might use async/await for performance and find 0.1ms of blocking too much, vs. a local app that uses async/await for its programming model might be fine with 10ms of "blocking"!

https://tokio.rs/blog/2020-04-preemption#a-note-on-blocking discusses this in more detail.

Re: Async hazard: MMAP is blocking IO

#73

Post by mjb »

cbsmith wrote: post #43
mjb wrote: post #9

I like this point - it's no secret that mmap can make memory access cost the same as an IO (swap can too) - but the interaction with async schedulers isn't immediately obvious. The cost can, sometimes, be even higher than this post says, because of write back behavior in Linux. Mmap is an interesting tool for system builders. It's super powerful, and super useful. But it's also kind of dangerous because the gap betwe…

I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads. This is the deal you make with the devil when you use cooperative scheduling without involving the kernel, so I'm surprised it is news to people working with cooperative schedulers. These faults can happen even if you never explicitly memory map files (particularly since executables and shared…

> I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads

Indeed. In practice, though, it's easier to write high performance servers and storage systems with async runtimes (like tokio) than with native threads, at least with the current state of the ecosystem. That's not for some fundamental reason - it's possible to get great threaded performance - just the current reality.

So, whoever's fault this is, it's useful to have good evidence of this downside of async runtimes (and worth thinking about ways that OSs could let runtimes know when they were about to block on IO).

Re: Async hazard: MMAP is blocking IO

#74

Post by dbaupp »

neonsunset wrote: post #61
correnos wrote: post #15

IMO this is a strong argument for proper threads over async: you can try and guess what will and won't block as an async framework dev, but you'll never fully match reality and you end up wasting resources when an executor blocks when you weren't expecting.

This kind of issue exists only in async executor implementations that cannot detect blocked workers and inject new ones to compensate for the starvation. I'm not aware if Rust has anything like this today (both Tokio and async-std are not like that) or in development for tomorrow, but there are implementations that demonstrate resilience to this in other language(s).

Do you have info about current (production) implementations that increase the number of workers?

In https://tokio.rs/blog/2020-04-preemption#a-note-on-blocking (2020), there's reference to .NET doing this, and an explicit suggestion that Go, Erlang and Java do not, as well as discussion of why Tokio did not.

Re: Async hazard: MMAP is blocking IO

#75

Post by dbaupp »

akira2501 wrote: post #18

> How do other mmap/madvise options influence this (for instance, MADV_SEQUENTIAL, MADV_WILLNEED, MADV_POPULATE, MADV_POPULATE_READ, mlock)? (Hypothesis: these options will make it more likely that data is pre-cached and thus fall into fast path more often, but without a guarantee.) That probably should have been the first thing to try. Too mad the mmap2 crate does not expose this. Also looking at the mmap2 crate, it…

> That probably should have been the first thing to try.

The point of the post is not "how to make mmap work best with async/await" or "how to optimise mmap with async/await", but exploring the consequence of incorrect code (and thus explaining why one might need potential remedies like those). Sorry if that didn't come across!

Re: Async hazard: MMAP is blocking IO

#76

Post by raggi »

cbsmith wrote: post #43
mjb wrote: post #9

I like this point - it's no secret that mmap can make memory access cost the same as an IO (swap can too) - but the interaction with async schedulers isn't immediately obvious. The cost can, sometimes, be even higher than this post says, because of write back behavior in Linux. Mmap is an interesting tool for system builders. It's super powerful, and super useful. But it's also kind of dangerous because the gap betwe…

I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads. This is the deal you make with the devil when you use cooperative scheduling without involving the kernel, so I'm surprised it is news to people working with cooperative schedulers. These faults can happen even if you never explicitly memory map files (particularly since executables and shared…

changes in the page table block the whole process, doesn't matter what combination of concurrency models you're using. we could do with a sub-process mapping API from the OS, but it's not something any major OS offers today, and requires designing for at a very fundamental level due to interaction with the hardware, and associated hardware constraints.

Re: Async hazard: MMAP is blocking IO

#77

Post by toast0 »

raggi wrote: post #76
cbsmith wrote: post #43

Earlier quoted context omitted.

I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads. This is the deal you make with the devil when you use cooperative scheduling without involving the kernel, so I'm surprised it is news to people working with cooperative schedulers. These faults can happen even if you never explicitly memory map files (particularly since executables and shared…

changes in the page table block the whole process, doesn't matter what combination of concurrency models you're using. we could do with a sub-process mapping API from the OS, but it's not something any major OS offers today, and requires designing for at a very fundamental level due to interaction with the hardware, and associated hardware constraints.

> changes in the page table block the whole process, doesn't matter what combination of concurrency models you're using.

I don't think that's necessarily true --- adding a mapping doesn't need to stop other threads that share a page table unless they're also modifying the page table. I don't think the TLB would cache an unmapped entry, but even if it did, the page fault handler will check, see that it's fine and resume execution.

For unmapping, it's different, in that you have to do IPI TLB shootdowns, but that interupts, not blocks.

Re: Async hazard: MMAP is blocking IO

#78

Post by raggi »

toast0 wrote: post #77
raggi wrote: post #76

Earlier quoted context omitted.

changes in the page table block the whole process, doesn't matter what combination of concurrency models you're using. we could do with a sub-process mapping API from the OS, but it's not something any major OS offers today, and requires designing for at a very fundamental level due to interaction with the hardware, and associated hardware constraints.

> changes in the page table block the whole process, doesn't matter what combination of concurrency models you're using. I don't think that's necessarily true --- adding a mapping doesn't need to stop other threads that share a page table unless they're also modifying the page table. I don't think the TLB would cache an unmapped entry, but even if it did, the page fault handler will check, see that it's fine and resu…

fair pushback, though s/unless they're also modifying the page table/unless they're also accessing the page table/, as it too needs to be synchronized. so yes, sometimes it has no effect, but given how often programs end up loading, crosstalk is super common

Re: Async hazard: MMAP is blocking IO

#79

Post by comex »

toast0 wrote: post #77
raggi wrote: post #76

Earlier quoted context omitted.

changes in the page table block the whole process, doesn't matter what combination of concurrency models you're using. we could do with a sub-process mapping API from the OS, but it's not something any major OS offers today, and requires designing for at a very fundamental level due to interaction with the hardware, and associated hardware constraints.

> changes in the page table block the whole process, doesn't matter what combination of concurrency models you're using. I don't think that's necessarily true --- adding a mapping doesn't need to stop other threads that share a page table unless they're also modifying the page table. I don't think the TLB would cache an unmapped entry, but even if it did, the page fault handler will check, see that it's fine and resu…

And even if other threads are contending for the page table lock, the kernel doesn’t hold that lock for the entire duration of the I/O. Only for the tiny fraction of that duration where the kernel is spending CPU time doing bookkeeping. For the rest of the time, during which the system is just waiting for the disk, the thread that triggered the page-in is still blocked, but other threads can do whatever they want, including page-table modifications.

From what I’ve read on LWN, contention on the page table lock (mmap_sem / mmap_lock) has been a real and perennial issue for Linux, especially on servers with huge numbers of CPUs; but it’s a far smaller effect than what this post is talking about.

Re: Async hazard: MMAP is blocking IO

#80

Post by krilovsky »

jcalvinowens wrote: post #64
krilovsky wrote: post #62

Earlier quoted context omitted.

> Generally the assumption is that userspace will treat -EIO as fatal A single bad disk doesn't make the situation fatal (unless that's the only disk in your system, in which case you're not even guaranteed to have your signal handler code in memory). > ENFILE is easy to deal with in a fatal path, by closing stdin so fd #0 can be reused That's assuming you have stdin open, which again, may work for 95% of your usecas…

> A single bad disk doesn't make the situation fatal (unless that's the only disk in your system, in which case you're not even guaranteed to have your signal handler code in memory). Yes it does. Your point about signal handlers is why I'm right, that's beyond the point where you can expect the machine to function in a sane way. Trying to recover is often actively harmful. > That's assuming you have stdin open, whic…

> Yes it does. Your point about signal handlers is why I'm right

If it's not a single disk system, not necessarily. To give you a concrete example: a process that writes logs to a disk dedicated for log collection can simply ignore an EIO/ENOSPC if logging isn't its main task. It can't easily recover from a SIGBUS in that scenario though.

> If you've hit EMFILE, you absolutely have some FD which you can sacrifice to collect debug info, is my point. If you don't you can reserve one a priori, this isn't that hard to deal with.

I'm not sure why you keep sticking to this example, when I already said that it was just an example of another detail that you need to take into account when implementing a SIGBUS handler. Sure, you can open /proc/self/maps a-priori and side-step the issue, but that's another detail that you need to take into account (and that you didn't mention until I brought it up). I never said that it was hard, only that writing a proper handler that deals with the edge cases isn't as trivial as you claim.

> you've invented an ambiguity that does not exist [...] you'll see that all the non-I/O cases you mention are independently identifiable via members of the siginfo_t struct

I'm not sure what's the ambiguity that you're claiming that I've invented. Yes, some of the specific examples that I gave (specifically CPU exceptions) are identifiable if you already know the details, but not all of them: non-disk faults can still result in SIGBUS with BUS_ADRERR, so that alone isn't enough to identify EIO errors or EOF coming from memory-mapped files, and I know that from personal experience debugging SIGBUS crashes.

> you're spewing unfounded FUD about the mmap() interface

I don't know where this is coming from. I never said that using mmap is bad or that it's impossible to write a SIGBUS handler to output debug info before crashing. I merely pointed out that it's not necessarily trivial, as there are details that should be taken care of, and that it may not in fact be suitable for 95% of usecases as you claimed.

You have a mental model of an ideal system which either can't recover from I/O errors, or doesn't get SIGBUS for reasons other than EIO or reading beyond EOF. I'm trying to tell you that not every system is like that, and that while mmap is useful, there are cases where explicit I/O is better suited for the task, and that your 95% might not be everyone's 95%. If you see FUD in simple facts, then I'm sorry, but I see no point in continuing this discussion.

Post reply on HN