Use mmap with care

It's not that mmap per se leads to double caching, but that combining the page cache with application-level caching leads to double caching. Say you're reading hugecactus.png into your image processing program. Whether you use mmap(2) or ordinary read(2), the first step in reading hugecactus.png is the kernel DMAing the bytes into the page cache. In the mmap case, the kernel maps the page cache into your application's address space. In the read case, the kernel copies from the page cache to the application read buffer. Now suppose your application PNG-decodes hugecactus.png into RGB raster data. Now, whether you used mmap or read, the kernel has both the decoded RGB data blob and the original PNG data in memory. That's usually wasteful.

(Yes, you can reduce the severity of this problem with MADV_DONTNEED and friends.)

I'd go a step further and broadly recommend not using mmap to access files, unless there's a really good overriding reason. (E.g., if the file is some sort of special virtual device/filesystem that by nature cannot have errors.)

Mmap is not good for writing to files — pages may be persisted to disk in arbitrary order, which makes it harder for filesystems to coalesce adjacent writes into fewer larger (and faster) IOs. (This is still an issue on SSDs, although not as bad as on HDDs.) This is a performance issue and can result in bad file layout (making future reads slower).

As this article describes, the mmap() model sort of assumes no errors happen. This breaks when files are truncated, even on local POSIX filesystems, and you get SIGBUS. It can also break if files disappear, such as a failing media or removed USB stick, or network filesystem. It doesn't mesh well with distributed filesystems either, for obvious reasons. If a page is to be writeable, you must take an exclusive data lock on that page's region across your distributed filesystem (and read access requires a shared data lock, to prevent corruption from concurrent writers). What if you lose quorum / availability?

TFA's trick to use thread-specific longjmps around specific virtual memory accesses probably works out ok on POSIX platforms[1] but it requires wrapping all of your mmap'd regions carefully. You can't just cast portions to a struct and access directly, except in small critical regions protected by the sigsetjmp. And as they point out, SIGBUS is global — it can conflict with error catchers (mentioned in TFA) but also can be raised for reasons other than mmap IO failure, such as attempting to access a non-canonical virtual address, and thus a long-lived global handler may mask other bugs. (Also, if you mmap many files and install a single long-lived handler in a multi-threaded program, it can become difficult to determine which file-access raised the signal.)

rtorrent, for example, used to have a ton of reports of SIGBUS due to mmap'd file access failure. I don't know if they've addressed that in some way (perhaps by simply masking SIBGUS) or continue to ignore it.

TFA claims pread was about 2/3 as fast as mmap'd access; some slightly clever application-specific use of caching, prefetch, or larger IOs might help eliminate that gap by reducing syscall overhead and/or disk wait. The best thing about pread/pwrite is they return have error reporting in the interface, and you can actually check that your IO did what you wanted.

[1]: http://man7.org/linux/man-pages/man7/signal-safety.7.html :

    If a signal handler interrupts the execution of an unsafe
    function, and the handler terminates via a call to longjmp(3) or
    siglongjmp(3) and the program subsequently calls an unsafe
    function, then the behavior of the program is undefined.

There's nothing wrong with signals per se. The problem is the signals API. Why do signals handlers need to be process-global? Why do we need to live with global signal handler registration clobbering any previous registration? We can change these things!

These problems are all fixable without "replac[ing] signals" as the mechanism. Ultimately, as long as processors have traps (which they will, as long as we have virtual memory) and as long as you want to give userspace to do something in response to these traps other than immediately die (an ability that's tremendously useful) you need some kind of stop-the-thread-that-trapped-and-call-a-callback mechanism, and whatever shape that mechanism takes, it's going to end up looking at least somewhat like signals.

Instead of just saying "signals are awful" and burying our heads in the sand, we should talk about what a better signals API should look like. I've already written a detailed proposal that I've linked elsewhere.

The real problem here is that the glibc people are completely uninterested in actually improving the signals API. Instead, they've taken the radical, unhelpful, and realistic stance that nobody should be using signals. As long as they think that way, people will keep using sigaction(2), and the world will remain in a half-broken and awful state.

True for this workload, iff you assume the filesystem and/or media protects you from corruption (it probably doesn't). I would guess OP is commenting on mmap IO in general, rather than TFA's specific git use case.

Really any networked or distributed filesystem will struggle to implement mmap semantics well.