Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async hazard: MMAP is blocking IO

huonw.github.io

Post reply on HN
Print view View original

61–70 of 119 posts

Re: Async hazard: MMAP is blocking IO

#61

Post by neonsunset »

correnos wrote: post #15

IMO this is a strong argument for proper threads over async: you can try and guess what will and won't block as an async framework dev, but you'll never fully match reality and you end up wasting resources when an executor blocks when you weren't expecting.

This kind of issue exists only in async executor implementations that cannot detect blocked workers and inject new ones to compensate for the starvation. I'm not aware if Rust has anything like this today (both Tokio and async-std are not like that) or in development for tomorrow, but there are implementations that demonstrate resilience to this in other language(s).

Re: Async hazard: MMAP is blocking IO

#62

Post by krilovsky »

jcalvinowens wrote: post #56
krilovsky wrote: post #55

Earlier quoted context omitted.

You're assuming here that mmap is only used for writing, where TFA is actually describing a read-only scenario, in which case EIO is synchronous as the read can't be completed. As for the triviality of writing a SIGBUS handler correctly, that is an oversimplification at best. I/O errors aren't always fatal, even in the write case, and handling SIGBUS in the way you describe wouldn't work when e.g. you're also out of…

The point is the same for reads: the vast majority of usecases just immediately abort() when a read fails. Writing byzantine fault logic to deal with broken storage media is like trying to recover from SIGSEGV, it's almost never a good idea. > I/O errors aren't always fatal, even in the write case Linux will not return -EIO unless the disk is in an unrecoverable state. Generally the assumption is that userspace will…

> Generally the assumption is that userspace will treat -EIO as fatal

A single bad disk doesn't make the situation fatal (unless that's the only disk in your system, in which case you're not even guaranteed to have your signal handler code in memory).

> ENFILE is easy to deal with in a fatal path, by closing stdin so fd #0 can be reused

That's assuming you have stdin open, which again, may work for 95% of your usecases, but isn't universal.

> It's either -EIO, or it's writing beyond EOF

That's an unfounded statement. A quick search of the kernel code will show that there are other reasons for getting a SIGBUS, which are unrelated to mmap (non-disk hardware failures, certain CPU exceptions, to name a few). So yeah, if you know that apart from the disk (or filesystem, at any rate) your hardware is in order, and that the only reason for SIGBUS could be a failed I/O through a memory mapped file, and you know that all of the code in your process is well behaved, writing a SIGBUS handler that terminates the process with a message indicating an mmap I/O error might be reasonable, but that's not the reality for every process, and likely not even 95% of processes.

Regardless, my main point wasn't that lack of file descriptors makes your suggestion problematic, but that your description of it as trivial is an oversimplification at best. mmap has its uses (as does writing a SIGBUS handler to deal with errors), but that doesn't mean that it doesn't have issues. Highlighting them doesn't mean that plain read/write are perfect and free from issues either, and certainly code that isn't ready to deal with EIO will have a bad time when a VFS operation fails. But there are cases where making I/O explicit is better, and I'm not sure why you seem to be making blanket statements that trivialise the issues with mmap.

Re: Async hazard: MMAP is blocking IO

#63

Post by dathinab »

While the general point the article is making is correct there are some issues.

- (minor issue) async example is artificially limited to 1 thread (the article states that). The issue is comparing 8 OS threads no async to 1 thread async is fundamentally not very useful as long as you didn't pin all threads to the same physical core.. So in general you should compare something async with num_cpus threads vs. num_cpus*X OS threads. Through this wouldn't have been that useful in this example without pinning the tokio async threads to CPUs to forcefully highlight the page issue, and doing it is bothersome so I wouldn't have done so either.

- (bigger issue) The singled thread async "traditional IO" example is NOT single threaded. Async _file_ IO anything between not a thing or very bad in most OSes hence most async engines including tokio do file IO in worker threads. This means the "single threaded" conventional IO async example is running 8 threads for reading IO and one to "touch the buffer" (i.e. do hardly anything).

To be clear the single threaded not being single threaded issue isn't discrediting the article, the benchmarks still show the problem it's that the 8 threaded conventional and 1 threaded async conventional are accidentally basically both 8 thraded.

Re: Async hazard: MMAP is blocking IO

#64

Post by jcalvinowens »

krilovsky wrote: post #62
jcalvinowens wrote: post #56

Earlier quoted context omitted.

The point is the same for reads: the vast majority of usecases just immediately abort() when a read fails. Writing byzantine fault logic to deal with broken storage media is like trying to recover from SIGSEGV, it's almost never a good idea. > I/O errors aren't always fatal, even in the write case Linux will not return -EIO unless the disk is in an unrecoverable state. Generally the assumption is that userspace will…

> Generally the assumption is that userspace will treat -EIO as fatal A single bad disk doesn't make the situation fatal (unless that's the only disk in your system, in which case you're not even guaranteed to have your signal handler code in memory). > ENFILE is easy to deal with in a fatal path, by closing stdin so fd #0 can be reused That's assuming you have stdin open, which again, may work for 95% of your usecas…

> A single bad disk doesn't make the situation fatal (unless that's the only disk in your system, in which case you're not even guaranteed to have your signal handler code in memory).

Yes it does. Your point about signal handlers is why I'm right, that's beyond the point where you can expect the machine to function in a sane way. Trying to recover is often actively harmful.

> That's assuming you have stdin open, which again, may work for 95% of your usecases, but isn't universal.

If you've hit EMFILE, you absolutely have some FD which you can sacrifice to collect debug info, is my point. If you don't you can reserve one a priori, this isn't that hard to deal with.

> writing a SIGBUS handler that terminates the process with a message indicating an mmap I/O error might be reasonable, but that's not the reality for every process, and likely not even 95% of processes.

You're completely wrong here: you've invented an ambiguity that does not exist. Take a look at the manpage for sigaction(), and you'll see that all the non-I/O cases you mention are independently identifiable via members of the siginfo_t struct passed to your SIGBUS handler (just like the I/O cases).

> but that your description of it as trivial is an oversimplification at best.

I'm not oversimplifying: you're spewing unfounded FUD about the mmap() interface, and I'm telling you that none of these details matter for 95% of usecases.

Re: Async hazard: MMAP is blocking IO

#65

Post by jcalvinowens »

malkia wrote: post #60
jcalvinowens wrote: post #53

Earlier quoted context omitted.

See below: this is wrong, the IO errors from the syscalls aren't synchronous. You might successfully call write() and then get -EIO from a sync() call with no indication of what actually failed. At least with fsync() you would know which file it was, but you have no way of knowing what portion of the data you wrote was successfully written back. The failure is fatal either way, most of the time. It's very rare you ac…

Are you saying that I can get somehow an error during "strlen" post "read()" - is this what are you saying? Because what I'm saying is that with "mmap" you can get that.

Absolutely: if the user page you read into experiences an uncorrectable ECC error in strlen(), you get SIGBUS.

Friends don't let friends write code to attempt recovery from hardware failures.

Re: Async hazard: MMAP is blocking IO

#66

Post by juped »

icedchai wrote: post #28
magicalhippo wrote: post #25

Earlier quoted context omitted.

In the languages and platforms I use, absolutely yes. Do you have some examples where a normal memory read is async?

Your definition of blocking is a bit different from my own. Synchronous is not always blocking. If the data is there, ready to go, there is no "blocking." If you consider all memory reads to be "blocking", then everything must be "blocking". The executable code must, after all, be read by the processor. In an extreme case, the entire executable could be paged out to disk! This interpretation is not what most people m…

you don't yield to your cooperative-multitasking runtime during reading from it, which is obviously what everyone in this thread means, and it's not helpful to start telling them "you're using the word blocking wrong" apropos of nothing

Re: Async hazard: MMAP is blocking IO

#67

Post by dbaupp »

dathinab wrote: post #63

While the general point the article is making is correct there are some issues. - (minor issue) async example is artificially limited to 1 thread (the article states that). The issue is comparing 8 OS threads no async to 1 thread async is fundamentally not very useful as long as you didn't pin all threads to the same physical core.. So in general you should compare something async with num_cpus threads vs. num_cpus*X…

(Author here)

> The singled thread async "traditional IO" example is NOT single threaded

The threads backing the single-threaded IO are an implementation detail of fulfilling the `.read().await` calls. The key is that there's a single coordinator thread that's issuing all the work, with the user-space runtime multiplexing tasks on that thread. I thought the fact that the "start a request and come back when it is finished" behaviour happens to be implemented via user-space threads rather than kernel-level epoll (or similar) is unlikely to affect behaviour.

I considered scaling up the number of files and using a multi-threaded runtime, but I felt that'd make everything more complicated without fundamentally changing behaviour.

However, maybe my theory is incorrect, in which case someone else can do their own experiments to provide more concrete information.

(This is referenced in a footnote: https://huonw.github.io/blog/2024/08/async-hazard-mmap/#fn:t... )

> The issue is comparing 8 OS threads no async to 1 thread async is fundamentally not very useful as long as you didn't pin all threads to the same physical core

The point is not to benchmark async vs. non-async, but provide a general reference point for "mmap working well" for comparison. As you suggest you agree with the "minor" issue tag, I don't think the parallelism vs. concurrency distinction matters much here... but again, definitely happy to see some concrete data that suggests otherwise!

Re: Async hazard: MMAP is blocking IO

#68

Post by dbaupp »

Retr0id wrote: post #10

> This is thus a worst case, the impact on real code is unlikely to be quite this severe! I think the actual worst-case would be to read the pages in a (pseudo-)random order.

(Author here.)

Good point. Do you think the difference will be observable? Will it be observable on an SSD (vs. HDD)?

Re: Async hazard: MMAP is blocking IO

#69

Post by dbaupp »

guerrilla wrote: post #6
01HNNWZ0MV43FF wrote: post #4

Earlier quoted context omitted.

By blocking they mean that it can take ballpark non-volatile storage times instead of ballpark RAM times

GP is aware. mmap makes files act like memory. Memory is always synchronous, thus blocking, so mmaped files are always blocking. I'm surprised OP even found this surprising. It should be completely obvious.

Yeah, of course a synchronous call that might block the thread is blocking, I agree... but, if I didn't have the context of "we're in a comment thread about a blog post about mmap", I'm pretty sure I wouldn't flag `x[i]` on `x: &[u8]` (or any other access) as a synchronous call that is worth worrying about.

Hence the discussion of subtlety in https://huonw.github.io/blog/2024/08/async-hazard-mmap/#at-a...

It's obvious when pointed out, but I don't think it's obvious without the context of "there's an mmap nearby".

(Author here, this behaviour wasn't surprising to me, but felt subtle enough to be worth investigating, and get some concrete data.)

Re: Async hazard: MMAP is blocking IO

#70

Post by dbaupp »

mwcampbell wrote: post #30
pengaru wrote: post #20

water is wet

Nobody automatically knows everything, and we have limited energy for drawing inferences based on what we do know. So the material covered in this post isn't obvious to everyone in its target audience, especially since Rust has had some success in making systems programming more approachable to inexperienced programmers, which is a good thing.

(Author here)

Thank you, you've expressed one of my goals with doing this sort of investigation/getting this data far better than I have. :)

It's something that's feels obvious once the dots are connected, but I was pretty sure many people wouldn't connect these dots automatically.

Post reply on HN