Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use mmap with care

sublimetext.com

Post reply on HN
Print view View original

201–210 of 218 posts

Re: Use mmap with care

#201

Post by gmueckl »

quotemstr wrote: post #142
gmueckl wrote: post #50

Earlier quoted context omitted.

Why would mmap lead to double caching? I can't follow.

It's not that mmap per se leads to double caching, but that combining the page cache with application-level caching leads to double caching. Say you're reading hugecactus.png into your image processing program. Whether you use mmap(2) or ordinary read(2), the first step in reading hugecactus.png is the kernel DMAing the bytes into the page cache. In the mmap case, the kernel maps the page cache into your application'…

Well, the kernel side cache is not much of a problem. The kernel is free to evict those pages at any time to respond to memory pressure etc. Linux treats its file system cache almost like unused memory in that it is normally the biggest pool from wich memory allocations for processes are drawn. Essentially, keeping the pages around in the cache is an optimization, because explicitly overwriting them too aggressively is just unnecessary work.

Re: Use mmap with care

#202

Post by hyc_symas »

rocqua wrote: post #169
beat wrote: post #166

Earlier quoted context omitted.

Signals play nice with threads. It's sloppy thread programming that doesn't play nice with signals.

Consider the following scenario: Two threads both want to MMAP a file / catch SIGSEGV for different pieces of code. Moreover, these threads come from different modules of a system maintained by different people. All of a sudden, these modules become coupled because we need some system for delegation of signal handlers between them. Or, we need to introduce a custom signal-handler module to deal with our delegation. I…

You're really just talking about poor practice in writing signal handlers. The signal API always tells you the previous value of a handler. Your own handlers should remember the previous value and call it if your own handler doesn't fully handle a signal it receives.

Re: Use mmap with care

#203

Post by hyc_symas »

zzzcpan wrote: post #79
TheCondor wrote: post #61

Earlier quoted context omitted.

It’s not a myth at all, mmap is faster, you save on straight copies of data and the sys-calls to do it. It should be faster in nearly all circumstances, faster by at least a copy. In exchange you pick up a lot of complexity dealing with faults and you potentially put stress on the VM system. If you are doing to ‘O’ part of I/O then mmap starts to be really complex, fast. rg is kind of a special case, it’s not writing…

You can surely win some laptop benchmarks by mmaping some files on certain close to the metal filesystems. But for general production case mmap shouldn't even be considered a solution to the syscall and memory copy overhead problem. If that overhead is too big for you, other approaches work better, like buffering, application level caching, etc.

Wow, this is so wrong in so many ways.

Buffering and application level caching mean you're wasting memory, and also wasting code space and CPU time because you're duplicating work that the OS already does.

Re: Use mmap with care

#204

Post by hyc_symas »

planteen wrote: post #67
hyc_symas wrote: post #59

Earlier quoted context omitted.

Wasting a good performance optimization on the rare chance that you might one day have to support a different endian architecture is IMO a poor tradeoff. The number of big-endian machines in use today is continually shrinking, and the number that are active on a heterogeneous network is even smaller. In LMDB we simply document "don't use this with remote filesystems" and avoid the issue - if you're never sharing file…

I still think you shouldn't be directly sending structs over the wire or to disk. The alternatives are so much better - SQLite or Cap’n Proto. I'm a bit shell shocked from supporting both big and little endian in structs from previous jobs. I've had nightmare situations with it twice. I do embedded systems and while little endian is winning there too, you still have legacy things like the LEON (SPARC) that is big end…

LMDB is orders of magnitude faster than SQLite, partly because it doesn't need to do fancy ser/deserialization.

The alternatives are not better. In fact, SQLite with its B+tree engine replaced by LMDB is still far better than vanilla SQLite - smaller, faster, and more reliable (SQLightning).

In a read-only workload, complex deserialization will become your limiting bottleneck, after you've eliminted all other bottlenecks from your code. Our profiling runs of OpenLDAP demonstrated this already, which is why LMDB was written, to allow structs to be persisted in in-memory format and used on read with no deserialization.

Re: Use mmap with care

#205

Post by bogomipz »

user5994461 wrote: post #184
beat wrote: post #108

The first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a series of shared memory buffers for IPC. It was originally written with shmat(), and on AIX (at least in those days), shmat was limited to three shared segments, so we had a lot of performance-wrecking blocking going on while waiting for the buffers to be clea…

>>> One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. How come you give this example just now? This can't be a coincidence. select() just caused us a major production outage. FYI: select() is broken on pretty much all Linux kernels up to very recent ones. Doesn't work when there are more than 1024 opened file descriptors, not a lot for a serve…

>"FYI: select() is broken on pretty much all Linux kernels up to very recent ones."

Can you elaborate or might you have some links? What was the cause and resolution?

Re: Use mmap with care

#206

Post by zzzcpan »

bogomipz wrote: post #205
user5994461 wrote: post #184

Earlier quoted context omitted.

>>> One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. How come you give this example just now? This can't be a coincidence. select() just caused us a major production outage. FYI: select() is broken on pretty much all Linux kernels up to very recent ones. Doesn't work when there are more than 1024 opened file descriptors, not a lot for a serve…

>"FYI: select() is broken on pretty much all Linux kernels up to very recent ones." Can you elaborate or might you have some links? What was the cause and resolution?

No, it's the least broken event notification mechanism on all systems. And it doesn't have a 1024 FDs limit on most systems either, as long as it used directly via syscall or through an event loop library that does that.

But in general for all event notification mechanism on all systems, it's incorrect to assume that reported FDs are actually ready, because that could cause a busy waiting loop, and incorrect to assume that unreported FDs are not ready, because that could cause timeouts.

Re: Use mmap with care

#207

Post by cbsmith »

ben-schaaf wrote: post #192
cbsmith wrote: post #181

Earlier quoted context omitted.

I'm just going to leave this right here: http://man7.org/linux/man-pages/man2/signalfd.2.html

I know about signalfd, how would it help dealing with mmap? > The signalfd mechanism can't be used to receive signals that are synchronously generated, such as the SIGSEGV

So a common way to handle synchronous signals is via a mechanism much like signalfd, where your signal handler just appends to a pipe that can be read at a convenient time.

Re: Use mmap with care

#208

Post by avar »

loeg wrote: post #196
avar wrote: post #173

Earlier quoted context omitted.

There are no license issues with bundling the git command-line tools in a commercial product, there's multiple existing proprietary commercial applications built on top of git that do so. The libgit2 code is GPL with a linking exception, so you can use it (unlike "git" itself) as a C library in a proprietary commercial product. > IPC and fork+exec has overhead[...] The "git cat-file --batch" command is something you'…

Certainly some commercial products can make GPL2 components work, but in general, it is a barrier to commercial adoption. I don't think you can totally dismiss GPL licensing concerns as "there aren't any."

Oh yeah, there's definitely concerns in general. It's not as easy for shippers of proprietary software as say the BSD license, you've got to keep a clear interface separation between your proprietary code and the GPL code etc.

You might also have lawyers who are lazy about it and don't want to deal with the liability, "we heard Apple banned GPL code..." or "the FSF sued Cisco...".

But there's no license reason for why you can't use that GPL code in some way, and everyone from Google with Android to Oracle with Oracle Linux and their DB bundles GPL code that's directly used by some accompanying proprietary piece of software.

But what I was more going for is that there's also a non-legal aspects to it that go beyond the license, which is that some maintainers of free software are actively hostile to their software being used as a smaller component in some proprietary product.

The GCC project is probably the most famous example of this, I think this has changed somewhat in recent years with LLVM+Clang, but they used to jealously guard things like their AST format. So e.g. someone with a proprietary editor (or Emacs for that matter...) could never hope to use GCC for spewing out parsing information for some C code.

I think it's fair to say that the Git project isn't like that. If someone maintaining proprietary software needs some plumbing interface to hook their stuff up and is willing to submit patches it'll be received as well as any other change (subject to review, maintenance & backwards-compatibility concerns etc.). If they find it useful it's likely that other people will too...

Re: Use mmap with care

#209

Post by eeZah7Ux »

eeZah7Ux wrote: post #55
eeZah7Ux wrote: post #7

Earlier quoted context omitted.

"In hindsight it's difficult to justify using mmap over pread" This needs a stronger justification. mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.

...and why the downvotes?

...keep going...
Post reply on HN