Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use mmap with care

sublimetext.com

Post reply on HN
Print view View original

71–80 of 218 posts

Re: Use mmap with care

#71

Post by avar »

ben-schaaf wrote: post #2

Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.

Is there a post where it's covered why Sublime Merge implements things like packfile reading on its own, rather than using git's own plumbing? E.g. in this case presumably keeping a "git cat-file --batch" would do the trick.

I contribute to git.git, and it would be interesting to know if there's inherent issues stopping you from doing that, or if it's implementation problems in some cases (e.g. missing plumbing commands or features). There's definitely interest from upstream in reviewing patches / helping if there's missing or inadequate plumbing.

You'd get upstream features for free as they come along. E.g. presumably you haven't implemented the new MIDX format, but that speeds up pack file access by a lot for some use-cases, and presumably the boring bits of low-level git operations aren't much of a selling point in and of themselves.

Aside from whether you'd use "git" itself, such a trick of using a slave process you'd talk to over IPC of some sort would cover some of the issues you wrote about, e.g. issue with sharing global state with libraries like Breakpad.

Re: Use mmap with care

#72

Post by raphlinus »

The other big problem with mmap is what happens when your file changes out from under you. This seems to be mostly for git packfiles, which I think can be treated as immutable by convention, but that's not strongly enforced anywhere. For reading, eg, program source files, I think mmap is hugely problematic.

I've been arguing for a long time that operating systems should provide read only snapshots of files as a primitive, but that's a pretty big ask; it's especially hard to do when the file system is network-mounted. There are a couple of copy-on-write filesystems on Linux (btrfs and ZFS if memory serves) which can do this locally, but it's not mainstream.

Re: Use mmap with care

#73

Post by cerebellum42 »

planteen wrote: post #67
hyc_symas wrote: post #59

Earlier quoted context omitted.

Wasting a good performance optimization on the rare chance that you might one day have to support a different endian architecture is IMO a poor tradeoff. The number of big-endian machines in use today is continually shrinking, and the number that are active on a heterogeneous network is even smaller. In LMDB we simply document "don't use this with remote filesystems" and avoid the issue - if you're never sharing file…

I still think you shouldn't be directly sending structs over the wire or to disk. The alternatives are so much better - SQLite or Cap’n Proto. I'm a bit shell shocked from supporting both big and little endian in structs from previous jobs. I've had nightmare situations with it twice. I do embedded systems and while little endian is winning there too, you still have legacy things like the LEON (SPARC) that is big end…

To disk is probably a bad idea but over the wire has some legitimate applications. Being reliant on same endian systems can be alright if for example you're building a distributed computing system where the same binary will be executed on a bunch of systems and all you're doing is sharing computation results between those instances. You do get unrivaled serialization speed that way. Timely Dataflow [1] works that way by default, but it also has an option for "real" serialization if that is required. Admittedly that's a fairly specific application but it's real and sometimes it's a good tradeoff.

[1] https://github.com/TimelyDataflow/timely-dataflow

Re: Use mmap with care

#74

Post by majewsky »

arethuza wrote: post #38
_0w8t wrote: post #20

Earlier quoted context omitted.

Alternatively one can run a separated process that does mmap and runs the calculations or whatever that needs to access the file as quickly as possible and do the the straightforward recovery in the parent process when the child process dies. The drawback is the need to some form of RPC, but there a lot of libraries to do that without much hustle.

I guess if you are using RPC you could forget about NFS and just run your "editor server" on the server where the file is actually stored :-)

In many cases, the server where the file is actually stored will not allow you to execute random code.

Re: Use mmap with care

#75

Post by avar »

raphlinus wrote: post #72

The other big problem with mmap is what happens when your file changes out from under you. This seems to be mostly for git packfiles, which I think can be treated as immutable by convention, but that's not strongly enforced anywhere. For reading, eg, program source files, I think mmap is hugely problematic. I've been arguing for a long time that operating systems should provide read only snapshots of files as a primi…

You can assume that git packfiles are immutable. They may disappear from under you as a repack happens, but they will not be changed.

They even have a name like pack-.pack where that is a SHA-1 of the contents of the pack (minus the last 20 bytes, the checksum SHA-1 is also part of the pack itself).

Re: Use mmap with care

#76

Post by manwe150 »

burntsushi wrote: post #46
quotemstr wrote: post #21

There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said , there's nothing wrong with mmap or SIGB…

> This old myth that mmap is the fast and efficient way to do IO just won't die. Well... because it's not a myth in all cases? $ time rg zqzqzqzq OpenSubtitles2016.raw.en --mmap real 1.167 user 0.815 sys 0.349 maxmem 9473 MB faults 0 $ time rg zqzqzqzq OpenSubtitles2016.raw.en --no-mmap real 1.748 user 0.506 sys 1.239 maxmem 9 MB faults 0 The OP's adventures with mmap mirror my own, which is why ripgrep includes this…

Did you do each of these after a clean reboot, or are we looking at possible caching effects from the kernel? If any part was in cache, then we might be just comparing shared memory against IPC, and that's an obvious performance win, but not really what's intended to be examined here.

The first numbers seem to imply that it takes equally long for pread to copy bytes from memory as it does to fetch them from the disk. For a quick back-of-the-napkin attempt at checking this, lets assume that disk IO accounts for 100% of this workload, and that local memory is one order of magnitude faster. In that case, I would expect the difference for an optimized implementation to be at most 10%.

I do think it is true that there are scenarios where the file mmap is faster, or that certain operations on each kernel might fall off a cliff. I just find it hard to believe that `mmap` must be as much faster as shown here in a typical situation (e.g. after a clean reboot, doing about the same amount of work, issuing optimal syscalls, with the OS/kernel not doing anything foolish).

Re: Use mmap with care

#77

Post by BeeOnRope »

mcculley wrote: post #49
rwmj wrote: post #37

And that's just for reading. Writing, especially if you want to be sure when the writes hit the backing file, or in what order, or if you run out of disk space, is another kettle of problems. I wonder how Multics dealt with all this, since AIUI in that system everything was effectively an mmapped file.

Did Multics have anything like NFS? Everything is easier if your kernel has control of the underlying device.

Well the problematic example given in the article was NTFS where the whole filesystem can disappear, but the problem applied to local files too, eg if their size is changed by another process.

Re: Use mmap with care

#78

Post by jlokier »

ben-schaaf wrote: post #2

Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.

You've used thread_local for the sigjmp_buf.

Are you sure the mechanism used by thread_local is safe to use in a signal handler?

Relevant bug from Rust: "TLS accesses aren't async-signal-safe", https://github.com/rust-lang/rust/issues/43146

I'm thinking even if it works on certain OSes, it's not guaranteed, because a signal handler's context is not a thread context - or is it?

E.g. the thread_local mechanism might depend on compile-time options (affecting how thread_local, whether it allocates memory on demand, and how it relocates the memory block when loading a shared library), whether it's main program or an -fPIC shared library, and the type of thread library (different ways of implementing pthreads).

Re: Use mmap with care

#79

Post by zzzcpan »

TheCondor wrote: post #61
burntsushi wrote: post #46

Earlier quoted context omitted.

> This old myth that mmap is the fast and efficient way to do IO just won't die. Well... because it's not a myth in all cases? $ time rg zqzqzqzq OpenSubtitles2016.raw.en --mmap real 1.167 user 0.815 sys 0.349 maxmem 9473 MB faults 0 $ time rg zqzqzqzq OpenSubtitles2016.raw.en --no-mmap real 1.748 user 0.506 sys 1.239 maxmem 9 MB faults 0 The OP's adventures with mmap mirror my own, which is why ripgrep includes this…

It’s not a myth at all, mmap is faster, you save on straight copies of data and the sys-calls to do it. It should be faster in nearly all circumstances, faster by at least a copy. In exchange you pick up a lot of complexity dealing with faults and you potentially put stress on the VM system. If you are doing to ‘O’ part of I/O then mmap starts to be really complex, fast. rg is kind of a special case, it’s not writing…

You can surely win some laptop benchmarks by mmaping some files on certain close to the metal filesystems.

But for general production case mmap shouldn't even be considered a solution to the syscall and memory copy overhead problem. If that overhead is too big for you, other approaches work better, like buffering, application level caching, etc.

Re: Use mmap with care

#80

Post by klodolph »

_wmd wrote: post #58
quotemstr wrote: post #25

Earlier quoted context omitted.

You can do that with a MAP_ANONYMOUS | MAP_SHARED mapping too: that kind of mapping is writable by both parent and child, but isn't backed by a disk file and so can't be truncated or surprise-removed. The article's points about mmap infelicity applies mostly to mappings of disk files. Anonymous mappings don't have the same problems.

Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid mmap whenever possible -- it's clear almost nobody fully understands it

> Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances

Anonymous mappings won't cause signals, they'll trigger the OOM killer. Remember that malloc() is just a fancy wrapper for mmap() (and sbrk()).

Post reply on HN