Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async hazard: MMAP is blocking IO

huonw.github.io

Post reply on HN
Print view View original

81–90 of 119 posts

Re: Async hazard: MMAP is blocking IO

#81

Post by akira2501 »

dbaupp wrote: post #75
akira2501 wrote: post #18

> How do other mmap/madvise options influence this (for instance, MADV_SEQUENTIAL, MADV_WILLNEED, MADV_POPULATE, MADV_POPULATE_READ, mlock)? (Hypothesis: these options will make it more likely that data is pre-cached and thus fall into fast path more often, but without a guarantee.) That probably should have been the first thing to try. Too mad the mmap2 crate does not expose this. Also looking at the mmap2 crate, it…

> That probably should have been the first thing to try. The point of the post is not "how to make mmap work best with async/await" or "how to optimise mmap with async/await", but exploring the consequence of incorrect code (and thus explaining why one might need potential remedies like those). Sorry if that didn't come across!

I think it's harder for you to write "correct code" here because the crate is hiding most of the actual detail from you. I put that in quotes because there's absolutely nothing incorrect about the code, it's really just suboptimal, and most probably because it can't even use the full syscall interface.

Seriously, I hate to be a curmudgeon, but that crate looks like a particularly bad and naive wrapper around mmap. It works very hard to provide you things you don't need when the basic interface is much more flexible. Aside from having to put `unsafe` around the call and re-import the kernel header constants, there's almost no reason to even have this in a crate.

Re: Async hazard: MMAP is blocking IO

#82

Post by dbaupp »

akira2501 wrote: post #81
dbaupp wrote: post #75

Earlier quoted context omitted.

> That probably should have been the first thing to try. The point of the post is not "how to make mmap work best with async/await" or "how to optimise mmap with async/await", but exploring the consequence of incorrect code (and thus explaining why one might need potential remedies like those). Sorry if that didn't come across!

I think it's harder for you to write "correct code" here because the crate is hiding most of the actual detail from you. I put that in quotes because there's absolutely nothing incorrect about the code, it's really just suboptimal, and most probably because it can't even use the full syscall interface. Seriously, I hate to be a curmudgeon, but that crate looks like a particularly bad and naive wrapper around mmap. It…

I have a feeling we’re talking at cross purposes here: I was actively trying to write incorrect code. This post isn’t about memmap2 crate specifically at all, it just happens to be a convenient way to get the exact (“incorrect”) syscall I wanted from Rust.

I see where you’re coming from but… it feels like you’re trying to convince me of something about the post? If you feel like convincing a larger audience of the limitations of the memmap2 crate specifically, I suggest writing your own blog posts and/or getting involved with it. :)

Re: Async hazard: MMAP is blocking IO

#83

Post by jcalvinowens »

krilovsky wrote: post #80
jcalvinowens wrote: post #64

Earlier quoted context omitted.

> A single bad disk doesn't make the situation fatal (unless that's the only disk in your system, in which case you're not even guaranteed to have your signal handler code in memory). Yes it does. Your point about signal handlers is why I'm right, that's beyond the point where you can expect the machine to function in a sane way. Trying to recover is often actively harmful. > That's assuming you have stdin open, whic…

> Yes it does. Your point about signal handlers is why I'm right If it's not a single disk system, not necessarily. To give you a concrete example: a process that writes logs to a disk dedicated for log collection can simply ignore an EIO/ENOSPC if logging isn't its main task. It can't easily recover from a SIGBUS in that scenario though. > If you've hit EMFILE, you absolutely have some FD which you can sacrifice to…

> If it's not a single disk system, not necessarily.

Again, you miss the point. 95%+ of Linux systems are single disk. That's the expected case.

>> If you've hit EMFILE

> I'm not sure why you keep sticking to this example

You brought this up initially, saying it was difficult to handle. I'm demonstrating that you're wrong, it's actually quite trivial to handle. Handwaving about "edge cases" is FUD, if you have some specific point to make then make it

> I'm not sure what's the ambiguity that you're claiming that I've invented.

You claimed it wasn't possible to be sure SIGBUS is from an I/O error. That's wrong.

> non-disk faults can still result in SIGBUS with BUS_ADRERR, so that alone isn't enough to identify EIO errors or EOF coming from memory-mapped files

Wrong. You can resolve that ambiguity from the cited address and si_errno etc. Try it next time.

> I'm trying to tell you that not every system is like that.

The fact you think I need to be told that is amusing. You're completely missing the point.

Let me try one more time:

Don't make things hard when they don't have to be. 95% of the time, they don't have to be. Saying "no, this is actually really hard, and you need to care about these normally irrelevant things" without first acknowledging the simple case is FUD in my book.

Re: Async hazard: MMAP is blocking IO

#84

Post by jandrewrogers »

cbsmith wrote: post #43
mjb wrote: post #9

I like this point - it's no secret that mmap can make memory access cost the same as an IO (swap can too) - but the interaction with async schedulers isn't immediately obvious. The cost can, sometimes, be even higher than this post says, because of write back behavior in Linux. Mmap is an interesting tool for system builders. It's super powerful, and super useful. But it's also kind of dangerous because the gap betwe…

I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads. This is the deal you make with the devil when you use cooperative scheduling without involving the kernel, so I'm surprised it is news to people working with cooperative schedulers. These faults can happen even if you never explicitly memory map files (particularly since executables and shared…

I would describe it more as a limitation of mmap than a liability.

Modern async models have their origin in addressing serious shortcomings with the traditional POSIX APIs, particularly with respect to mmap and kernel schedulers. You can’t mix the models easily, many parts of POSIX don’t play nicely with async architectures. Traditional async I/O engines use direct I/O on locked memory, and if you use async you need to be cognizant of why this is. Half the point of async is to have explicit control and knowledge of when page faults are scheduled. Async is a power tool, not something that should be used lightly for casual tasks nor accidentally pulled in as a dependency.

The issue here appears to be that it is far too easy for someone to inadvertently mix models, not async per se. Async has substantial advantages versus native kernel threads, so getting rid of async is not a realistic solution. No one is going to give up a several-fold performance increase versus native kernel threads because some developers can’t figure out how to not mix models or the ecosystem doesn’t protect developers against inadvertently mixing models.

Async is used heavily in some C/C++ domains but it doesn’t seem to cause many issues there, perhaps because dependencies are much more explicit and intentional. Async has also been idiomatic for certain domains in C/C++ for decades so there is an element of maturity around working with it.

Re: Async hazard: MMAP is blocking IO

#86

Post by krilovsky »

jcalvinowens wrote: post #83
krilovsky wrote: post #80

Earlier quoted context omitted.

> Yes it does. Your point about signal handlers is why I'm right If it's not a single disk system, not necessarily. To give you a concrete example: a process that writes logs to a disk dedicated for log collection can simply ignore an EIO/ENOSPC if logging isn't its main task. It can't easily recover from a SIGBUS in that scenario though. > If you've hit EMFILE, you absolutely have some FD which you can sacrifice to…

> If it's not a single disk system, not necessarily. Again, you miss the point. 95%+ of Linux systems are single disk. That's the expected case. >> If you've hit EMFILE > I'm not sure why you keep sticking to this example You brought this up initially, saying it was difficult to handle. I'm demonstrating that you're wrong, it's actually quite trivial to handle. Handwaving about "edge cases" is FUD, if you have some s…

> Again, you miss the point. 95%+ of Linux systems are single disk. That's the expected case.

I specifically added ENOSPC as an example that's relevant on single disk systems as well.

Regardless, I thought we were talking about 95% of usecases in relation to implementations, not runtime systems, but even if we're talking about runtime systems, I'm not sure where you're pulling that 95% number from (or why you felt the need to add a plus sign this time around). That may be true for personal computers, but most Linux systems are servers, which generally aren't deployed in a single disk configuration.

> You brought this up initially, saying it was difficult to handle

I didn't say anything about difficulty. I only said that it wasn't trivial as you made it out to be, which isn't the same thing. Also, when I initially brought it up all I said was that in the FD exhaustion case it wouldn't work in the way you described in the comment that I responded to.

> You claimed it wasn't possible to be sure SIGBUS is from an I/O error. That's wrong.

I didn't. All I said in response to your claim of "It's either -EIO, or it's writing beyond EOF" was that there are other reasons for getting a SIGBUS. Moreover, I actually said (in the same paragraph), that if you know that a SIGBUS is caused by an I/O error, and that all of the code in your process is well-behaved (and by that I meant that terminating it with an abort() wouldn't cause side-effects due to e.g. atexit() handlers not running), using mmap with a SIGBUS handler might be reasonable.

> Wrong. You can resolve that ambiguity from the cited address. Try it next time.

First you claimed that I invented an ambiguity that doesn't exist, and that SIGBUS causes can be identifiable if I just read the sigaction(7) manpage. Now you say that there is an ambiguity, but that it can be resolved using the address, so which is it? [0]

I never said that using mmap is impossible, or even hard (and definitely not "this is actually really hard"). I actually agreed that in some cases it might be reasonable to do it with a SIGBUS handler. All I did say was that it isn't trivial to deal with errors, and that the 95% figure might be true for your usecases, but that it doesn't necessarily apply to other people's usecases.

The only one who said that something was "hard" during this discussion was you.

I get it, it's easier to attack the strawman rather than respond to my comments. I'm just not sure why you think it has anything to do with what I said.

[0] EDIT: I now see that you edited the sentence I quoted to say "from the cited address and si_errno etc.". It might surprise you to learn that si_errno is almost never set in Linux (the manpage is actually explicit about it with "si_errno is generally unused in Linux"), and definitely not in mmap-related SIGBUS coming from memory mapped files. I have no idea why you added this remark telling me that I should try it, when you clearly didn't.

Re: Async hazard: MMAP is blocking IO

#87

Post by cbsmith »

mjb wrote: post #73
cbsmith wrote: post #43

Earlier quoted context omitted.

I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads. This is the deal you make with the devil when you use cooperative scheduling without involving the kernel, so I'm surprised it is news to people working with cooperative schedulers. These faults can happen even if you never explicitly memory map files (particularly since executables and shared…

> I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads Indeed. In practice, though, it's easier to write high performance servers and storage systems with async runtimes (like tokio) than with native threads, at least with the current state of the ecosystem. That's not for some fundamental reason - it's possible to get great threaded performance…

> That's not for some fundamental reason - it's possible to get great threaded performance - just the current reality.

I would argue it is for a fundamental reason. Cooperative multitasking in user-space requires far less overhead than anything a kernel might do. It's just an explicit part of the trade-off: you get more efficient context switches and control when context can change, and in exchange you leave something on the table whenever the kernel is involved.

> So, whoever's fault this is, it's useful to have good evidence of this downside of async runtimes (and worth thinking about ways that OSs could let runtimes know when they were about to block on IO).

But it isn't specific to async runtimes (in fact, a kernel-based preemptively scheduled async runtime wouldn't have this problem). It's a problem specific to cooperative multitasking.

Re: Async hazard: MMAP is blocking IO

#88

Post by cbsmith »

raggi wrote: post #76
cbsmith wrote: post #43

Earlier quoted context omitted.

I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads. This is the deal you make with the devil when you use cooperative scheduling without involving the kernel, so I'm surprised it is news to people working with cooperative schedulers. These faults can happen even if you never explicitly memory map files (particularly since executables and shared…

changes in the page table block the whole process, doesn't matter what combination of concurrency models you're using. we could do with a sub-process mapping API from the OS, but it's not something any major OS offers today, and requires designing for at a very fundamental level due to interaction with the hardware, and associated hardware constraints.

Page faults block the thread, not the process, because the thread is trying to access memory that isn't available to it. Other threads run just fine so long as they too don't trigger page faults. The article specifically mentions this, and I've built entire architectures based around this reality. They work great.

...and of course there are also multi-process concurrency models, where even if a process were blocked, the other processes would not. So no, it does absolutely matter what combination of concurrency models you are using.

Re: Async hazard: MMAP is blocking IO

#89

Post by jcalvinowens »

krilovsky wrote: post #86
jcalvinowens wrote: post #83

Earlier quoted context omitted.

> If it's not a single disk system, not necessarily. Again, you miss the point. 95%+ of Linux systems are single disk. That's the expected case. >> If you've hit EMFILE > I'm not sure why you keep sticking to this example You brought this up initially, saying it was difficult to handle. I'm demonstrating that you're wrong, it's actually quite trivial to handle. Handwaving about "edge cases" is FUD, if you have some s…

> Again, you miss the point. 95%+ of Linux systems are single disk. That's the expected case. I specifically added ENOSPC as an example that's relevant on single disk systems as well. Regardless, I thought we were talking about 95% of usecases in relation to implementations, not runtime systems, but even if we're talking about runtime systems, I'm not sure where you're pulling that 95% number from (or why you felt th…

> but most Linux systems are servers, which generally aren't deployed in a single disk configuration.

You are incorrect about that: most Linux servers in the world have one disk. Most servers are not storage servers.

> I didn't say anything about difficulty. I only said that it wasn't trivial as you made it out to be

...and I demonstrated by counterexample that you're wrong, it is trivial. If you think I'm missing some detail, you are free to explain it. You're just handwaving.

> First you claimed that I invented an ambiguity that doesn't exist, and that SIGBUS causes can be identifiable if I just read the sigaction(7) manpage. Now you say that there is an ambiguity, but that it can be resolved using the address, so which is it?

Both, obviously? If you only look at signo there's an "ambiguity", but with the rest of siginfo_t the "ambiguity" ceases to exist. There is no case where you cannot unambiguously handle -EIO in a mmap via SIGBUS.

You claimed that you could only use SIGBUS with mmap if you were sure there were no other sources of SIGBUS. Quoting you directly:

> So yeah, if you know that apart from the disk (or filesystem, at any rate) your hardware is in order, and that the only reason for SIGBUS could be a failed I/O through a memory mapped file, and you know that all of the code in your process is well behaved, writing a SIGBUS handler that terminates the process with a message indicating an mmap I/O error might be reasonable

That statement is completely wrong: you can always tell whether it came from the mmap or something else, by looking at the siginfo_t fields.

> and by that I meant that terminating it with an abort() wouldn't cause side-effects due to e.g. atexit() handlers not running

Any system that breaks if atexit() handlers don't run is fundamentally broken by design. There are a dozen reasons the process can die without running those.

> All I did say was that it isn't trivial to deal with errors

Yes, and that statement is wrong. Most of the time it is trivial, because you just call abort(). There is no possibly simpler error handling than printing a message and calling abort(). For 95% of the workloads running across the world on Linux, that is entirely sufficient.

It is very unusual to try to recover from I/O error, and most programmers who try are really shooting themselves in the foot without realizing it.

You're free to disagree obviously, but I'm directly refuting the points you're making. Calling it a "strawman" make you look really really silly.

Re: Async hazard: MMAP is blocking IO

#90

Post by cbsmith »

raggi wrote: post #78
toast0 wrote: post #77

Earlier quoted context omitted.

> changes in the page table block the whole process, doesn't matter what combination of concurrency models you're using. I don't think that's necessarily true --- adding a mapping doesn't need to stop other threads that share a page table unless they're also modifying the page table. I don't think the TLB would cache an unmapped entry, but even if it did, the page fault handler will check, see that it's fine and resu…

fair pushback, though s/unless they're also modifying the page table/unless they're also accessing the page table/, as it too needs to be synchronized. so yes, sometimes it has no effect, but given how often programs end up loading, crosstalk is super common

Crosstalk is absolutely common for a number of operations, but that crosstalk is NOT the same as blocking until the page is loaded into RAM. That operation is blocking on IO.
Post reply on HN