Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use mmap with care

sublimetext.com

Post reply on HN
Print view View original

61–70 of 218 posts

Re: Use mmap with care

#61

Post by TheCondor »

burntsushi wrote: post #46
quotemstr wrote: post #21

There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said , there's nothing wrong with mmap or SIGB…

> This old myth that mmap is the fast and efficient way to do IO just won't die. Well... because it's not a myth in all cases? $ time rg zqzqzqzq OpenSubtitles2016.raw.en --mmap real 1.167 user 0.815 sys 0.349 maxmem 9473 MB faults 0 $ time rg zqzqzqzq OpenSubtitles2016.raw.en --no-mmap real 1.748 user 0.506 sys 1.239 maxmem 9 MB faults 0 The OP's adventures with mmap mirror my own, which is why ripgrep includes this…

It’s not a myth at all, mmap is faster, you save on straight copies of data and the sys-calls to do it. It should be faster in nearly all circumstances, faster by at least a copy. In exchange you pick up a lot of complexity dealing with faults and you potentially put stress on the VM system. If you are doing to ‘O’ part of I/O then mmap starts to be really complex, fast. rg is kind of a special case, it’s not writing, it’s going to do mostly (maybe only, I assume it backtracks on matches) sequential reads of mostly static files, it’s really the easy case, its not clear that madvise would help and your brand is speed so saving on those copies is worth it. What might be interesting, on certain memory constrained systems you can slide a smaller map space through the file rather than mapping the whole thing; it’s been a while since I looked at it all but mapping the smaller pieces gives huge hints to the vmm and it would probably slow down rg incrementally but speed up overall system performance.

Re: Use mmap with care

#63

Post by altfredd »

quotemstr wrote: post #21

There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said , there's nothing wrong with mmap or SIGB…

> It's possible to do much better than sigaction(2). I wrote up a detailed proposal for improvement in [1].

Thanks for interesting read. That said, I can see why glibc people didn't appreciate the proposal.

The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue. There are some dangerous-sounding paragraphs too:

> It’s occasionally useful to longjmp out of a signal handler. It’s reasonable to want to return non-locally from a shared signal handler too --- that is, to resume program execution after SIGNAL_CONTINUE_EXECUTION in a different state from the state the program had when we entered the shared signal handler. Since the signal system probably wants to maintain some kind of state to track its progress through its shared signal handler list, a plain longjmp out of a shared signal handler will likely leave the system in an unspecified state.

Are you sure, that we should worry about "signal system maintaining some kind of state"? Not about rest of application being in completely unspecified state?!!

The article proposes a primitive system for setting signal priorities, but stops at a half-baked solution. There is a mention of banning longjmp, but individual handlers still can bail via SIGNAL_CONTINUE_EXECUTION. What if I want my handler to always run regardless of registration order?

The proposal does not offer a way to retrieve a list of already installed handlers, which makes that part of it even worse than existing Posix signal API.

The proposed API does not address challenges of using signals in multi-threading programs.

The article mentions, that signal handlers can't be reliably unloaded, but proposed API does not address it.

Overall the proposed interface brings little to the table, does not work well alongside with existing sigaction() API and creates false illusion, that signal handlers are safe and ok to use. I imagine, that if it had more technical "meat" — more like robust mutexes or FD_CLOEXEC ­— it would have seen a lot more constructive discussion and less hostility from glibc maintainers.

Re: Use mmap with care

#64

Post by burntsushi »

TheCondor wrote: post #61
burntsushi wrote: post #46

Earlier quoted context omitted.

> This old myth that mmap is the fast and efficient way to do IO just won't die. Well... because it's not a myth in all cases? $ time rg zqzqzqzq OpenSubtitles2016.raw.en --mmap real 1.167 user 0.815 sys 0.349 maxmem 9473 MB faults 0 $ time rg zqzqzqzq OpenSubtitles2016.raw.en --no-mmap real 1.748 user 0.506 sys 1.239 maxmem 9 MB faults 0 The OP's adventures with mmap mirror my own, which is why ripgrep includes this…

It’s not a myth at all, mmap is faster, you save on straight copies of data and the sys-calls to do it. It should be faster in nearly all circumstances, faster by at least a copy. In exchange you pick up a lot of complexity dealing with faults and you potentially put stress on the VM system. If you are doing to ‘O’ part of I/O then mmap starts to be really complex, fast. rg is kind of a special case, it’s not writing…

Yes... I know it's not a myth. :-) I was responding to someone who was saying that it was a myth.

> It should be faster in nearly all circumstances

As my previous comment showed, that's definitely not true. If you're searching a whole bunch of small files in a short period of time, then it appears that the overhead of memory mapping leads to a significant performance regression when compared to standard `read` calls.

> it’s really the easy case

I know. :-) That's why ripgrep has both modes. It chooses between them based on the predicted workload. It uses memory maps automatically if it's searching a file or two, but otherwise falls back to standard read calls.

Moreover, if ripgrep aborts once in a while because of a SIGBUS, then it's usually not a big deal. It's fairly rare for it to happen. And if it does happen to you a lot or you never want it to happen, then you just need to `alias rg="rg --no-mmap"`.

Re: Use mmap with care

#65

Post by gpderetta »

_wmd wrote: post #58
quotemstr wrote: post #25

Earlier quoted context omitted.

You can do that with a MAP_ANONYMOUS | MAP_SHARED mapping too: that kind of mapping is writable by both parent and child, but isn't backed by a disk file and so can't be truncated or surprise-removed. The article's points about mmap infelicity applies mostly to mappings of disk files. Anonymous mappings don't have the same problems.

Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid mmap whenever possible -- it's clear almost nobody fully understands it

> Anonymous mappings are backed by swap and may be overcommitted,

So is normal memory. Many allocators today even use mmap internally.

Re: Use mmap with care

#66

Post by daemin »

ben-schaaf wrote: post #2

Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.

I was using Sublime Merge on the weekend and after doing a few changes with my remotes using the normal command line, it exited (well crashed I assume). When things like this happen do you automatically get an error report? Because you mentioned that you use the error reporting library from Google in that article.

Re: Use mmap with care

#67

Post by planteen »

hyc_symas wrote: post #59
planteen wrote: post #51

Earlier quoted context omitted.

You should not directly use structs for data serialization. If you end up having to support a big endian platform, your structure orders will change, not to mention potential packing issues as you said. https://commandcenter.blogspot.com/2012/04/byte-order-fallac...

Wasting a good performance optimization on the rare chance that you might one day have to support a different endian architecture is IMO a poor tradeoff. The number of big-endian machines in use today is continually shrinking, and the number that are active on a heterogeneous network is even smaller. In LMDB we simply document "don't use this with remote filesystems" and avoid the issue - if you're never sharing file…

I still think you shouldn't be directly sending structs over the wire or to disk. The alternatives are so much better - SQLite or Cap’n Proto.

I'm a bit shell shocked from supporting both big and little endian in structs from previous jobs. I've had nightmare situations with it twice. I do embedded systems and while little endian is winning there too, you still have legacy things like the LEON (SPARC) that is big endian. I've heard lots of network embedded hardware is big endian too for obvious reasons.

Re: Use mmap with care

#68

Post by jacobush »

CoolGuySteve wrote: post #47
atemerev wrote: post #33

Things are so much better if you are not writing apps for general public (mine are trading-related). You can tell your few clients — make sure that the access to mmapped file is exclusive — and get away with it. And yes, mmap is the awesomest thing out there.

I’ve moved away from mmap for trading in favor of a separate write thread. While mmap is fast, the combination of factors that can make it decide to stall your thread while it commits to disk is difficult to manage from an operational standpoint. A slight misconfiguration is all it takes to introduce a rare and hard to notice multi-millisecond delay. Whereas with a spinlocked sized-reserved vector, the fail state per…

Or better yet, mmap in a read-write-thread?

Re: Use mmap with care

#69

Post by CoolGuySteve »

jacobush wrote: post #68
CoolGuySteve wrote: post #47

Earlier quoted context omitted.

I’ve moved away from mmap for trading in favor of a separate write thread. While mmap is fast, the combination of factors that can make it decide to stall your thread while it commits to disk is difficult to manage from an operational standpoint. A slight misconfiguration is all it takes to introduce a rare and hard to notice multi-millisecond delay. Whereas with a spinlocked sized-reserved vector, the fail state per…

Or better yet, mmap in a read-write-thread?

Ya sure, go nuts on the other thread. It’s not critical to trading. I normally fprintf all sorts of time stamps, log level info, and handle formatting over there.

Just make sure you put it on another core to protect your cache.

But if you want mmap, you do you.

Re: Use mmap with care

#70

Post by TheCondor »

burntsushi wrote: post #64
TheCondor wrote: post #61

Earlier quoted context omitted.

It’s not a myth at all, mmap is faster, you save on straight copies of data and the sys-calls to do it. It should be faster in nearly all circumstances, faster by at least a copy. In exchange you pick up a lot of complexity dealing with faults and you potentially put stress on the VM system. If you are doing to ‘O’ part of I/O then mmap starts to be really complex, fast. rg is kind of a special case, it’s not writing…

Yes... I know it's not a myth. :-) I was responding to someone who was saying that it was a myth. > It should be faster in nearly all circumstances As my previous comment showed, that's definitely not true. If you're searching a whole bunch of small files in a short period of time, then it appears that the overhead of memory mapping leads to a significant performance regression when compared to standard `read` calls.…

I love ripgrep, btw, great work.

I was pondering this some more in the shower, the mmap for rg case is also sort of naturally cache oblivious, copies will consume hardware cache for the write and while there is a ton of hardware for cache on modern hardware, it’s a noticeable cost on some tests. If you’re searching through something big, then it’d be like doubling hardware cache which is probably really noticeable on smaller devices.

The small files case is interesting, copying the data is faster than patching up the page table tree, I bet there is a strong correlation to the hardware cache size vs the average file size in that case. The files probably need to be N pages in size for it to be worth it, might be an interesting heuristic to use.

Post reply on HN