Use mmap with care • Hacker News Board

Re: Use mmap with care

#81

Post by blattimwind » Thu, May 02, 2019, 2:42 PM UTC

_0w8t wrote: post #20

nneonneo wrote: post #5

Even without NFS, using mmap requires being real careful about signals - SIGBUS can be raised any time the underlying file operation fails, including because someone else truncated the file, or because the underlying storage had an error (disk error, removed media, network storage). And, as this post so eloquently illustrates (and through my personal experience), handling SIGBUS/SIGSEGV cleanly in a multithreaded pro…

Alternatively one can run a separated process that does mmap and runs the calculations or whatever that needs to access the file as quickly as possible and do the the straightforward recovery in the parent process when the child process dies. The drawback is the need to some form of RPC, but there a lot of libraries to do that without much hustle.

If you are already copying things around (between address spaces even), you might as well just use pread or lseek+read, which at that point is likely a much better choice overall.

Re: Use mmap with care

#82

Post by klodolph » Thu, May 02, 2019, 2:43 PM UTC

The mistake here is using longjmp / siglongjmp. This is a possible way to handle SIGBUS, but in practice it will be intractable in larger programs written in C or C++. The compiler is generally free to move loads and stores around, and you might be completely blindsided by how the compiler has reordered your memory operations once you add side effects to one of the operations. Theoretically, if accessing a memory location can siglongjmp out then at least the memory location should be volatile.

A better way to handle SIGBUS is to just map zeroes over the offending pages using MAP_FIXED and then setting a flag. After every operation that works on a file, you check the flag.

Re: Use mmap with care

#83

Post by [deleted] » Thu, May 02, 2019, 2:43 PM UTC

gpderetta wrote: post #65

_wmd wrote: post #58

Earlier quoted context omitted.

Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid mmap whenever possible -- it's clear almost nobody fully understands it

> Anonymous mappings are backed by swap and may be overcommitted, So is normal memory. Many allocators today even use mmap internally.

[deleted]

Re: Use mmap with care

#84

Post by rocqua » Thu, May 02, 2019, 2:45 PM UTC

Honestly, this reads like a thorough indictment of signals in user space.

* Signal handlers are process global * Signal handlers need to be re-entrant safe

Re-entrancy is painful but can be done, but process-global signal handlers means that pulling in a totally unrelated library can break your code. Moreover, it makes the combined use of certain libraries straight-up impossible. Similarly, it means that the use of library precludes you from using certain features.

Combined, this means that the use of signal handlers is simply toxic. Which makes them an anti-feature. It feels to me like having the restrictions of kernel-space, with all of the downsides of user-space.

Are there any plans in linux to replace signals?

Re: Use mmap with care

#85

Post by gubbrora » Thu, May 02, 2019, 2:48 PM UTC

quotemstr wrote: post #21

There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said , there's nothing wrong with mmap or SIGB…

Interesting but how will you determine not-null is the hot path? Getting it wrong coughs cost you

Re: Use mmap with care

#86

Post by rocqua » Thu, May 02, 2019, 2:49 PM UTC

snarfy wrote: post #62

Now I'm curious how Vim deals with large files. I'm assuming the went the pread route.

Vim needs to do a lot more special stuff, because it has to handle random insertions and deletions. Doing those things to a file (mmaped or otherwise) requires moving all data after the edit.

If I recall correctly, vim/vi uses a linked list of 'chunks' that is dynamically merged. For long files I would expect some form of lazy loading of chunks.

Re: Use mmap with care

#87

Post by _wmd » Thu, May 02, 2019, 2:51 PM UTC

klodolph wrote: post #80

_wmd wrote: post #58

Earlier quoted context omitted.

Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid mmap whenever possible -- it's clear almost nobody fully understands it

> Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances Anonymous mappings won't cause signals, they'll trigger the OOM killer. Remember that malloc() is just a fancy wrapper for mmap() (and sbrk()).

If a process was swapped out and a fault fails to bring a page back due to an IO error, you can at least catch (I think) SIGBUS. But this just reinforces the point: nobody really understands virtual memory, even people like us that think they do

Re: Use mmap with care

#88

Post by z3t4 » Thu, May 02, 2019, 2:56 PM UTC

Or you could stream parse the file. eg. parse it one chunk at a time.

Re: Use mmap with care

#89

Post by BeeOnRope » Thu, May 02, 2019, 2:57 PM UTC

TheCondor wrote: post #61

burntsushi wrote: post #46

Earlier quoted context omitted.

> This old myth that mmap is the fast and efficient way to do IO just won't die. Well... because it's not a myth in all cases? $ time rg zqzqzqzq OpenSubtitles2016.raw.en --mmap real 1.167 user 0.815 sys 0.349 maxmem 9473 MB faults 0 $ time rg zqzqzqzq OpenSubtitles2016.raw.en --no-mmap real 1.748 user 0.506 sys 1.239 maxmem 9 MB faults 0 The OP's adventures with mmap mirror my own, which is why ripgrep includes this…

It’s not a myth at all, mmap is faster, you save on straight copies of data and the sys-calls to do it. It should be faster in nearly all circumstances, faster by at least a copy. In exchange you pick up a lot of complexity dealing with faults and you potentially put stress on the VM system. If you are doing to ‘O’ part of I/O then mmap starts to be really complex, fast. rg is kind of a special case, it’s not writing…

I agree with you it's faster on the systems I've tried (modern x86 systems with local drives, mostly). The drive type or IO speed doesn't matter much because it's cached IO we are interested in: if actual IO needs to occur that is the the time that will dominate, not memcpy to/from user space (except perhaps with very, very fast devices, i.e., in the 5 GB/s range).

That said, the case isn't as obvious as you make it: you apparently save on copies and explicit system calls, but mmap replaces those with page table manipulation and "hidden" system calls (i.e., page faults).

These page faults have as much per-call overhead as regular system calls, and so if mmap actually faulted in every page, I'm pretty sure it would actually be slower than read(), since read with a 16K buffer (for example) would make only 25% as many syscalls as mmap bringing in every 4K page.

On modern Linux, by default, mmap doesn't fault in every page, due to "faultaround" which tries to map in additional nearby pages every time it faults (16 by default), so the number of faults is 1/16th what you'd expect if it faulted on every page. You can avoid additional mapping on access with MAP_POPULATE or madvise (? maybe) calls, but then this introduces the same kind of window management problem as read: you lose the abstraction of the entire file just mapped into memory.

Beyond that, mmap has to do "per page" work to map the file: adjusting VM and OS structures to map the page into the process address space, and then undoing that work on munmap (which is the more expensive half since it includes TLB invalidation, and possibly a cross-core shootdown). You'd thing that this work would be much faster than copying 4 KiB of memory, but it isn't - and on some systems with small pages sizes and/or slow TLB invalidations it can be slower overall.

Re: Use mmap with care

#90

Post by tntn » Thu, May 02, 2019, 3:00 PM UTC

_wmd wrote: post #87

klodolph wrote: post #80

Earlier quoted context omitted.

> Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances Anonymous mappings won't cause signals, they'll trigger the OOM killer. Remember that malloc() is just a fancy wrapper for mmap() (and sbrk()).

If a process was swapped out and a fault fails to bring a page back due to an IO error, you can at least catch (I think) SIGBUS. But this just reinforces the point: nobody really understands virtual memory, even people like us that think they do

So should we extend your conclusion above to the following?

"There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid virtual memory whenever possible -- it's clear almost nobody fully understands it"

I'd suggest that is ludicrous, and for the same reason your original conclusion is also excessive.