Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use mmap with care

sublimetext.com

Post reply on HN
Print view View original

21–30 of 218 posts

Re: Use mmap with care

#21

Post by quotemstr »

There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first.

That said, there's nothing wrong with mmap or SIGBUS error reporting in concept. I think I've said it before, but you can think of mmap of a disk file as basically adding a temporary dedicated swap file to the system, with all the pages backed by that "swap file" already "swapped" out unless already cached. Sometimes that's exactly what you want!

The author's signal handler registration difficulties come from the POSIX signal API being awful, not from the idea that catching CPU traps is somehow bad. It's possible to do much better than sigaction(2). I wrote up a detailed proposal for improvement in [1].

When I ran [1] by the glibc people, though, the response from them was pretty stark and unwarranted hostility toward any use of signals at all, even in perfectly legitimate scenarios, like mmap SIGBUS or certain kinds of important JVM-style pointer check [2] optimizations. It's this "we won't change anything anywhere despite our views being out of step with real user needs" attitude that's responsible for a lot of weird friction in the Unix API surface.

Practically every program mmaps disk files has trouble recovering from IO errors. If it were easier for multiple components in a process to share responsibility for handling a signal, we'd more often get subtleties like this right. A better signal API will improve the user experience. Tilting at signal-shaped windmills won't.

[1] https://www.facebook.com/notes/daniel-colascione/toward-shar...

[2] Relying on CPU traps lets you perform certain checks for free. Why write "if (myptr != nullptr) var = myptr" when you can just write "var = myptr" and handle the SIGSEGV when myptr ends up being nullptr? The latter option is zero-overhead in the non-nullptr case. It's also much more complicated, but if you're a VM, and you generate millions and millions of these "if (myptr != nullptr)" checks in JITed code, the size and speed win of eliminating the check starts to justify the complexity.

Re: Use mmap with care

#22

Post by ben-schaaf »

eeZah7Ux wrote: post #7
ben-schaaf wrote: post #2

Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.

"In hindsight it's difficult to justify using mmap over pread" This needs a stronger justification. mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.

Using pread would have been more work from the start but provides a more robust solution and doesn't have problems on Windows. I would argue that the incrementaly built mmap based solution is strictly worse, thus difficult to justify doing again.

Re: Use mmap with care

#23

Post by quotemstr »

eeZah7Ux wrote: post #7
ben-schaaf wrote: post #2

Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.

"In hindsight it's difficult to justify using mmap over pread" This needs a stronger justification. mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.

> without copying

While it's true that memory bandwidth is sometimes a limiting factor in performance, I've found that much more frequently people overestimate the cost of memory operations and don't check their estimates against benchmarks.

Re: Use mmap with care

#24

Post by ben-schaaf »

evozer wrote: post #3
ben-schaaf wrote: post #2

Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.

Did you consider emulating mmap yourselves? "Memory mapped files work by mapping the full file into a virtual address space and then using page faults to determine which chunks to load into physical memory. In essence it allows you to access the file as if you had read the whole thing into memory, without actually doing so." I feel like this could be done in c++ directly, by maintaining an internal cache for each fil…

This is essentially how databases like PostgreSQL work, but in essence it only avoids the sys-call overhead. The OS is already caching the file, regardless of mmap, so using pread would have likely been enough for us.

It totally would have been simpler overall, but each incremental step we made was significantly less work than the refactoring required for pread.

Re: Use mmap with care

#25

Post by quotemstr »

_0w8t wrote: post #20
nneonneo wrote: post #5

Even without NFS, using mmap requires being real careful about signals - SIGBUS can be raised any time the underlying file operation fails, including because someone else truncated the file, or because the underlying storage had an error (disk error, removed media, network storage). And, as this post so eloquently illustrates (and through my personal experience), handling SIGBUS/SIGSEGV cleanly in a multithreaded pro…

Alternatively one can run a separated process that does mmap and runs the calculations or whatever that needs to access the file as quickly as possible and do the the straightforward recovery in the parent process when the child process dies. The drawback is the need to some form of RPC, but there a lot of libraries to do that without much hustle.

You can do that with a MAP_ANONYMOUS | MAP_SHARED mapping too: that kind of mapping is writable by both parent and child, but isn't backed by a disk file and so can't be truncated or surprise-removed. The article's points about mmap infelicity applies mostly to mappings of disk files. Anonymous mappings don't have the same problems.

Re: Use mmap with care

#26

Post by quotemstr »

ben-schaaf wrote: post #24
evozer wrote: post #3

Earlier quoted context omitted.

Did you consider emulating mmap yourselves? "Memory mapped files work by mapping the full file into a virtual address space and then using page faults to determine which chunks to load into physical memory. In essence it allows you to access the file as if you had read the whole thing into memory, without actually doing so." I feel like this could be done in c++ directly, by maintaining an internal cache for each fil…

This is essentially how databases like PostgreSQL work, but in essence it only avoids the sys-call overhead. The OS is already caching the file, regardless of mmap, so using pread would have likely been enough for us. It totally would have been simpler overall, but each incremental step we made was significantly less work than the refactoring required for pread.

> The OS is already caching the file

Not necessarily. With O_DIRECT, pread() doesn't put pages into page cache: it just DMAs them directly into your process. Using O_DIRECT and the process-private caching we've been discussing, sophisticated programs (like databases) can (and do!) implement their own "page cache" systems. And because databases have access pattern information that the generic kernel VM subsystem doesn't, such a database can frequently do a better job doing this caching on its own.

Re: Use mmap with care

#28

Post by ben-schaaf »

tobyhinloopen wrote: post #11

I feel like you should use some kind of library for this instead of handling it all yourself.

A library for mmap would have not helped with any of the Windows or Breakpad issues. It would have also been more work up-front than mmap, since none of the problems of mmap were known to us at the time.

Re: Use mmap with care

#29

Post by tus87 »

quotemstr wrote: post #25
_0w8t wrote: post #20

Earlier quoted context omitted.

Alternatively one can run a separated process that does mmap and runs the calculations or whatever that needs to access the file as quickly as possible and do the the straightforward recovery in the parent process when the child process dies. The drawback is the need to some form of RPC, but there a lot of libraries to do that without much hustle.

You can do that with a MAP_ANONYMOUS | MAP_SHARED mapping too: that kind of mapping is writable by both parent and child, but isn't backed by a disk file and so can't be truncated or surprise-removed. The article's points about mmap infelicity applies mostly to mappings of disk files. Anonymous mappings don't have the same problems.

Yes, but if the file is on local disk its exactly the same situation as the page file.

OP seems to be trying to cope with NFS errors....but if you are using mmap on NFS you have bigger problems...

Re: Use mmap with care

#30

Post by Pahr3yah »

quotemstr wrote: post #21

There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said , there's nothing wrong with mmap or SIGB…

If you roughly know your access patterns in advance you can reduce the page fault costs with madvise.
Post reply on HN