Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async hazard: MMAP is blocking IO

huonw.github.io

Post reply on HN
Print view View original

51–60 of 119 posts

Re: Async hazard: MMAP is blocking IO

#51

Post by magicalhippo »

danarmak wrote: post #45
nemetroid wrote: post #37

Earlier quoted context omitted.

So what is the definition of "blocking" here? That it takes more than 1 µs?

That the process/thread enters kernel mode and then is suspended waiting for IO or for some other event. As long as the thread is running your code (or, is scheduleable) it's not blocked. And then the async implementation can ensure your code cooperatively gives up the CPU for other code.

If your memory is paged out and you then access it, using your definition, it would block.

So, in the context of async code, there's no difference from the application perspective between reading mmap'ed data and reading "regular" data (ie memory from the regular paged pool), as both could incur blocking IO.

If you're lucky and the mmap'ed data is in the system cache, then reading that data will not block and is fast. If you're unlucky and your process has been swapped out, then doing a regular memory read will block and is slow.

Re: Async hazard: MMAP is blocking IO

#52

Post by jcalvinowens »

loeg wrote: post #48
jcalvinowens wrote: post #40

Earlier quoted context omitted.

On Linux, if you get a SIGBUS from poking a memory map that generally means you'd have certainly gotten -ENOMEM or -EIO during an equivalent sequence of syscalls (or been oom-killed, if you overcommit). Those are treated as fatal in the vast majority of programs, so dying to SIGBUS isn't meaningfully different for most usecases. By your logic, passing a file descriptor to a library is also "unwieldy", because the lib…

> On Linux, if you get a SIGBUS from poking a memory map that generally means you'd have certainly gotten -ENOMEM or -EIO during an equivalent sequence of syscalls (or been oom-killed, if you overcommit). Those are treated as fatal in the vast majority of programs, so dying to SIGBUS isn't meaningfully different for most usecases. No. EIO is not instantly fatal in the same way that SIGBUS is. It allows for printing a…

> It allows for printing an error message, associating the error with some context of what failed.

Nope: -EIO is not synchronous unless you're using O_SYNC. More commonly, you're going to get -EIO from sync() or close(), not write(). All write() does a lot of the time is memcpy() into the pagecache and mark the page as dirty.

Once you've written enough to trigger writeback, write() will return -EIO. But the error is still asynchronous, probably having occurred during the writeback of data you passed to a successful write() call in the past.

A SIGBUS handler that inspects /proc/self/maps and tells you what file the failure occurred in before calling abort() is trivial. That's sufficient for 95% of usecases.

Re: Async hazard: MMAP is blocking IO

#53

Post by jcalvinowens »

malkia wrote: post #50
jcalvinowens wrote: post #40

Earlier quoted context omitted.

On Linux, if you get a SIGBUS from poking a memory map that generally means you'd have certainly gotten -ENOMEM or -EIO during an equivalent sequence of syscalls (or been oom-killed, if you overcommit). Those are treated as fatal in the vast majority of programs, so dying to SIGBUS isn't meaningfully different for most usecases. By your logic, passing a file descriptor to a library is also "unwieldy", because the lib…

Passing file handle is different - a well written library would check and handle errors, but pointer to buffer - and I don't expect a library to "try" and "catch" (even if it's possible) cases. Would strlen do this? No.

See below: this is wrong, the IO errors from the syscalls aren't synchronous. You might successfully call write() and then get -EIO from a sync() call with no indication of what actually failed. At least with fsync() you would know which file it was, but you have no way of knowing what portion of the data you wrote was successfully written back.

The failure is fatal either way, most of the time. It's very rare you actually want to handle -EIO, you just want to tell the user to replace their broken disk and call abort(). SIGBUS is fine, you can easily write a handler to add debugging context to your failure message.

Re: Async hazard: MMAP is blocking IO

#54

Post by loeg »

jcalvinowens wrote: post #52
loeg wrote: post #48

Earlier quoted context omitted.

> On Linux, if you get a SIGBUS from poking a memory map that generally means you'd have certainly gotten -ENOMEM or -EIO during an equivalent sequence of syscalls (or been oom-killed, if you overcommit). Those are treated as fatal in the vast majority of programs, so dying to SIGBUS isn't meaningfully different for most usecases. No. EIO is not instantly fatal in the same way that SIGBUS is. It allows for printing a…

> It allows for printing an error message, associating the error with some context of what failed. Nope: -EIO is not synchronous unless you're using O_SYNC. More commonly, you're going to get -EIO from sync() or close(), not write(). All write() does a lot of the time is memcpy() into the pagecache and mark the page as dirty. Once you've written enough to trigger writeback, write() will return -EIO. But the error is…

First, you're free to not use buffered IO. Second, EIO on fsync or close for buffered IO is still adjacent to the relevant file descriptor.

Re: Async hazard: MMAP is blocking IO

#55

Post by krilovsky »

jcalvinowens wrote: post #52
loeg wrote: post #48

Earlier quoted context omitted.

> On Linux, if you get a SIGBUS from poking a memory map that generally means you'd have certainly gotten -ENOMEM or -EIO during an equivalent sequence of syscalls (or been oom-killed, if you overcommit). Those are treated as fatal in the vast majority of programs, so dying to SIGBUS isn't meaningfully different for most usecases. No. EIO is not instantly fatal in the same way that SIGBUS is. It allows for printing a…

> It allows for printing an error message, associating the error with some context of what failed. Nope: -EIO is not synchronous unless you're using O_SYNC. More commonly, you're going to get -EIO from sync() or close(), not write(). All write() does a lot of the time is memcpy() into the pagecache and mark the page as dirty. Once you've written enough to trigger writeback, write() will return -EIO. But the error is…

You're assuming here that mmap is only used for writing, where TFA is actually describing a read-only scenario, in which case EIO is synchronous as the read can't be completed.

As for the triviality of writing a SIGBUS handler correctly, that is an oversimplification at best. I/O errors aren't always fatal, even in the write case, and handling SIGBUS in the way you describe wouldn't work when e.g. you're also out of file descriptors, or when the cause of SIGBUS isn't due to an I/O error. So what works for 95% of your usecases doesn't necessarily apply to the 95% of other people's usecases.

Re: Async hazard: MMAP is blocking IO

#56

Post by jcalvinowens »

krilovsky wrote: post #55
jcalvinowens wrote: post #52

Earlier quoted context omitted.

> It allows for printing an error message, associating the error with some context of what failed. Nope: -EIO is not synchronous unless you're using O_SYNC. More commonly, you're going to get -EIO from sync() or close(), not write(). All write() does a lot of the time is memcpy() into the pagecache and mark the page as dirty. Once you've written enough to trigger writeback, write() will return -EIO. But the error is…

You're assuming here that mmap is only used for writing, where TFA is actually describing a read-only scenario, in which case EIO is synchronous as the read can't be completed. As for the triviality of writing a SIGBUS handler correctly, that is an oversimplification at best. I/O errors aren't always fatal, even in the write case, and handling SIGBUS in the way you describe wouldn't work when e.g. you're also out of…

The point is the same for reads: the vast majority of usecases just immediately abort() when a read fails. Writing byzantine fault logic to deal with broken storage media is like trying to recover from SIGSEGV, it's almost never a good idea.

> I/O errors aren't always fatal, even in the write case

Linux will not return -EIO unless the disk is in an unrecoverable state. Generally the assumption is that userspace will treat -EIO as fatal, so the kernel won't return it unless it's truly hosed. Sometimes the error is specific to a file, but that's the far less common case in practice.

> e.g. you're also out of file descriptors,

ENFILE is easy to deal with in a fatal path, by closing stdin so fd #0 can be reused (you're about to call abort(), you don't need it anymore). Try again :)

> or when the cause of SIGBUS isn't due to an I/O error.

It's either -EIO, or it's I/O beyond EOF. The second thing is a bug equivalent to a buffer overrun. That's synchronous, you can handle it just like you handle SIGSEGV if you want to emit more debugging or even write byzantine recovery logic.

Re: Async hazard: MMAP is blocking IO

#57

Post by jcalvinowens »

loeg wrote: post #54
jcalvinowens wrote: post #52

Earlier quoted context omitted.

> It allows for printing an error message, associating the error with some context of what failed. Nope: -EIO is not synchronous unless you're using O_SYNC. More commonly, you're going to get -EIO from sync() or close(), not write(). All write() does a lot of the time is memcpy() into the pagecache and mark the page as dirty. Once you've written enough to trigger writeback, write() will return -EIO. But the error is…

First, you're free to not use buffered IO. Second, EIO on fsync or close for buffered IO is still adjacent to the relevant file descriptor.

> adjacent to the relevant file descriptor.

So is SIGBUS: you get the address in the handler. You probably have a data structure associating the two things somewhere anyway, and if you don't you can look it up in /proc.

Re: Async hazard: MMAP is blocking IO

#58

Post by Moto7451 »

cbsmith wrote: post #43
mjb wrote: post #9

I like this point - it's no secret that mmap can make memory access cost the same as an IO (swap can too) - but the interaction with async schedulers isn't immediately obvious. The cost can, sometimes, be even higher than this post says, because of write back behavior in Linux. Mmap is an interesting tool for system builders. It's super powerful, and super useful. But it's also kind of dangerous because the gap betwe…

I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads. This is the deal you make with the devil when you use cooperative scheduling without involving the kernel, so I'm surprised it is news to people working with cooperative schedulers. These faults can happen even if you never explicitly memory map files (particularly since executables and shared…

I think your point here can be more generalized. Why should someone expect reading memory to benefit from async code?

The fact that the memory in this case has an access layer with exploitable latency is where the chatter about this stems from, but it misses the fundamental issue at hand.

If this was a valid concept we’d have async memcpy interfaces.

Re: Async hazard: MMAP is blocking IO

#59

Post by dathinab »

correnos wrote: post #15

IMO this is a strong argument for proper threads over async: you can try and guess what will and won't block as an async framework dev, but you'll never fully match reality and you end up wasting resources when an executor blocks when you weren't expecting.

It isn't.

Async is for tasks dominated by waiting, e.g. http serving, not computations. This means it's extremely rare to run into mmap blocking related issues if you don't do something strange.

Furthermore async doesn't exclude multi threading:

- having multi threaded worker threads in addition to CPU threads is pretty normal

- having multiple async threads potentially with cross core work stealing is also the nrom

I.e. if you just follow basic advice the huge majority of task interacting in any potential performance problematic way will not be run in async task even if you write an async web server.

> but you'll never fully match reality and you end up wasting resources when an executor blocks when you weren't expecting

and you wast tons of resources always even without doing something unusual with non async IO _iff_ it's about waiting dominated tasks as you have way more management overhead

furthermore in more realistic cases it's quite common that some unplanned blocking is mainly casing latency issues (which in worst case could case timeouts) but due async engines still using multi threading it not leading relevant utilization issues. That is if it's just some unplanned blocking. If you do obviously wrong things like processing large files in async tasks things can be different.

An argument against async is that depending what you use it can add complexity and that a lot of use-cases don't benefit form it's benefits enough to make it a reasonable choice. Through that is also a bit language dependent. E.g. JS is already anyway coperative in your program and using async makes things simpler here (as the alternative are callbacks). Or in pythons with GIL the perf. gain of async are much higher compared to the gains in idk. C++.

Re: Async hazard: MMAP is blocking IO

#60

Post by malkia »

jcalvinowens wrote: post #53
malkia wrote: post #50

Earlier quoted context omitted.

Passing file handle is different - a well written library would check and handle errors, but pointer to buffer - and I don't expect a library to "try" and "catch" (even if it's possible) cases. Would strlen do this? No.

See below: this is wrong, the IO errors from the syscalls aren't synchronous. You might successfully call write() and then get -EIO from a sync() call with no indication of what actually failed. At least with fsync() you would know which file it was, but you have no way of knowing what portion of the data you wrote was successfully written back. The failure is fatal either way, most of the time. It's very rare you ac…

Are you saying that I can get somehow an error during "strlen" post "read()" - is this what are you saying? Because what I'm saying is that with "mmap" you can get that.
Post reply on HN