Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use mmap with care

sublimetext.com

Post reply on HN
Print view View original

101–110 of 218 posts

Re: Use mmap with care

#101

Post by jschwartzi »

saagarjha wrote: post #93
blattimwind wrote: post #81

Earlier quoted context omitted.

If you are already copying things around (between address spaces even), you might as well just use pread or lseek+read, which at that point is likely a much better choice overall.

Couldn’t you mmap a shared, read-only page so that your OS doesn’t copy it?

It would be better to mmap the file into N single-threaded processes instead of mmapping the file into 1 N-threaded process. This is exactly why signals are process-global instead of thread-local. The intent is that they were used for inter-process communication. The whole notion of processes and signals predates the notion of a thread, and threads are essentially a performance optimization for 30 or 40 year old hardware. If you use single-threaded processes then all of the processes can MAP_SHARED map the file and then they all get access to the file contents as each process is updating the file. And they all also have their own address space and can crash separately from each other.

Re: Use mmap with care

#102

Post by planteen »

cerebellum42 wrote: post #73
planteen wrote: post #67

Earlier quoted context omitted.

I still think you shouldn't be directly sending structs over the wire or to disk. The alternatives are so much better - SQLite or Cap’n Proto. I'm a bit shell shocked from supporting both big and little endian in structs from previous jobs. I've had nightmare situations with it twice. I do embedded systems and while little endian is winning there too, you still have legacy things like the LEON (SPARC) that is big end…

To disk is probably a bad idea but over the wire has some legitimate applications. Being reliant on same endian systems can be alright if for example you're building a distributed computing system where the same binary will be executed on a bunch of systems and all you're doing is sharing computation results between those instances. You do get unrivaled serialization speed that way. Timely Dataflow [1] works that way…

Yeah that's true for the distributed application you mentioned with the wire. Certainly advantages there, especially when you control all the nodes.

I hate seeing the fields of a communication protocol listed in two structs - an ifdef BIG_ENDIAN and LITTLE_ENDIAN. They will invariably be inconsistent, someone will change something in little but not big, or even worse, update one incorrectly and not test it.

Re: Use mmap with care

#103

Post by theoh »

raphlinus wrote: post #72

The other big problem with mmap is what happens when your file changes out from under you. This seems to be mostly for git packfiles, which I think can be treated as immutable by convention, but that's not strongly enforced anywhere. For reading, eg, program source files, I think mmap is hugely problematic. I've been arguing for a long time that operating systems should provide read only snapshots of files as a primi…

Sounds like https://en.wikipedia.org/wiki/Snapshot_isolation.

Re: Use mmap with care

#104

Post by quotemstr »

ben-schaaf wrote: post #40
quotemstr wrote: post #26

Earlier quoted context omitted.

> The OS is already caching the file Not necessarily. With O_DIRECT, pread() doesn't put pages into page cache: it just DMAs them directly into your process. Using O_DIRECT and the process-private caching we've been discussing, sophisticated programs (like databases) can (and do!) implement their own "page cache" systems. And because databases have access pattern information that the generic kernel VM subsystem doesn…

I might have undersold the performance advantage of writing your own cache, but let me reiterate the point I was trying to make: The reason we didn't consider doing so was because we weren't having a performance issue. Writing our own cache would be strictly more work than just using pread and accomplished the same thing.

Yeah. For your application, you did the right thing. I was speaking more abstractly.

Re: Use mmap with care

#105

Post by jschwartzi »

rocqua wrote: post #84

Honestly, this reads like a thorough indictment of signals in user space. * Signal handlers are process global * Signal handlers need to be re-entrant safe Re-entrancy is painful but can be done, but process-global signal handlers means that pulling in a totally unrelated library can break your code. Moreover, it makes the combined use of certain libraries straight-up impossible. Similarly, it means that the use of l…

The real issue is that people are using threads for things that processes were originally intended for. Having one thread for the UI, one thread for network code, one thread for program logic, etc. is actually a mis-use of threads in POSIX-land. POSIX is designed for you to use multiple processes where in Windows you would use multiple threads. This gets you a lot of robust interprocess communication mechanisms and also memory isolation on systems with an MMU. It also had performance implications on the state of the art hardware at the time, which is why threads were invented. And finally it requires you to expand your usage of the operating system.

Being multi-process would solve the issues with a process-global signal handler because there would no longer be a question of which thread generated the signal.

Re: Use mmap with care

#106

Post by tntn »

_wmd wrote: post #97
tntn wrote: post #90

Earlier quoted context omitted.

So should we extend your conclusion above to the following? "There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid virtual memory whenever possible -- it's clear almost nobody fully understands it" I'd suggest that is ludicrous, and for the same reason your original conclusion is also excessive.

It is not constructive to form a sweeping generalization from a statement and then claim the sweeping generalization is ludicrous, implying the original statement is ludicrous. :) My first comment was in reply to one claiming anonymous memory did not have the same problems as file-backed memory, indicating the parent did not understand they are the same thing. The subsequent reply was to another comment continuing to…

As if "avoid virtual memory" is substantially more of a "sweeping generalization" than "avoid mmap."

If you apply the same reasoning that you've used to conclude that everyone should avoid mmap, than you are led directly to the conclusion that everyone should avoid virtual memory.

The "same problems" that you are pointing out are possible with anonymous memory aren't unique to memory you get directly from mmap, they also apply to all memory, period. mmap'ed anonymous memory might have th same problems as file-backed memory, but those are the same problems that .text and .bss have.

The "powerful (and consequently hazardous) OS feature" here isn't mmap, it's literally virtual memory. At the moment you concede that memory may be backed by something besides physical memory at any point in time, you get the possibility of all those "exotic error paths."

Re: Use mmap with care

#107

Post by burntsushi »

manwe150 wrote: post #76
burntsushi wrote: post #46

Earlier quoted context omitted.

> This old myth that mmap is the fast and efficient way to do IO just won't die. Well... because it's not a myth in all cases? $ time rg zqzqzqzq OpenSubtitles2016.raw.en --mmap real 1.167 user 0.815 sys 0.349 maxmem 9473 MB faults 0 $ time rg zqzqzqzq OpenSubtitles2016.raw.en --no-mmap real 1.748 user 0.506 sys 1.239 maxmem 9 MB faults 0 The OP's adventures with mmap mirror my own, which is why ripgrep includes this…

Did you do each of these after a clean reboot, or are we looking at possible caching effects from the kernel? If any part was in cache, then we might be just comparing shared memory against IPC, and that's an obvious performance win, but not really what's intended to be examined here. The first numbers seem to imply that it takes equally long for pread to copy bytes from memory as it does to fetch them from the disk.…

Yes, the file is in cache, and that was my intent. That's why my `time` output says `faults 0` for both runs. That is, no page faults occurred. Everything is in RAM.

That is indeed a very common case for ripgrep, where you might execute many searches against the same corpus repeatedly. Optimizing that use case is important.

For cases where the file isn't cached, then it's much less interesting, because you're going to just be bottlenecked on disk I/O for the most part.

> then we might be just comparing shared memory against IPC, and that's an obvious performance win, but not really what's intended to be examined here.

Please don't take my comment out of context. I was specifically responding to this fairly broad sweeping claim with actual data:

> This old myth that mmap is the fast and efficient way to do IO just won't die.

You might think the fact that this isn't a myth is "obvious," but clearly, not everyone does. The right remedy to that is data, not back of the napkin theorizing. :-)

If you want to try your own benchmarks in your own environment, then you can: https://github.com/BurntSushi/ripgrep/

On Linux at least, you do not need to do a clean reboot to measure something without cache. You can drop the file cache with `sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'`.

Re: Use mmap with care

#108

Post by beat »

The first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a series of shared memory buffers for IPC. It was originally written with shmat(), and on AIX (at least in those days), shmat was limited to three shared segments, so we had a lot of performance-wrecking blocking going on while waiting for the buffers to be cleared.

My first being original-idea development was rewriting with mmap so I could use an arbitrary (and programmable) number of buffers, tuning blocking against memory consumption, with logging to track performance for tuning. It was very cool. Worked great!

Until it went to production.

In production, it crashed every time it ran, shortly after starting up. Since we were doing seasonal production, backing out my change also backed out other necessary changes. It was very embarrassing and frustrating. Worse, I could not replicate the problem in testing! It only happened on the prod servers. And, as a wet behind the ears junior programmer, everyone assumed I'd just screwed up and was too dumb to understand how.

So I wrote a test program, divorced from our regular code, to test mmap() itself. Turned out that it ran fine on dev/test servers, but on the prod servers, it would randomly overwrite 1k memory pages with nulls. Yeah. Once i convinced the senior engineers and my manager, I got to report an OS bug to IBM. Who were like "What's wrong with your code, really?" I wound up sending them my C test code and the compiled executable, along with results.

It turned out the bug was caused by the order in which OS patches had been applied on the servers.

One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is.

And if a junior programmer came to me today reporting a bug in OS memory management, my first response would be "What's wrong with your code, really?"

Re: Use mmap with care

#109

Post by quotemstr »

altfredd wrote: post #63
quotemstr wrote: post #21

There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said , there's nothing wrong with mmap or SIGB…

> It's possible to do much better than sigaction(2). I wrote up a detailed proposal for improvement in [1]. Thanks for interesting read. That said, I can see why glibc people didn't appreciate the proposal. The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue. There are some dangerous-sounding paragraphs too: > It’s occasionally u…

> The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue.

If you're writing a signal handler, the signal handler needs to be async-signal-safe. You can't just wave your arms and make the problem of async signal safety disappear, because CPU traps themselves are async-signal-unsafe. Even userfaultfd has to deal with async signal safety issues, since a thread causing a fault can, in principle, be anywhere!

> Are you sure, that we should worry about "signal system maintaining some kind of state"? Not about rest of application being in completely unspecified state?!!

If your handlers play by the rules and are async-signal-safe, there's no problem.

> The article proposes a primitive system for setting signal priorities, but stops at a half-baked solution.

What's half-baked about it?

> What if I want my handler to always run regardless of registration order?

That's a logical nonsense request. What if two components want to their handlers to be the highest priority?

> The proposal does not offer a way to retrieve a list of already installed handlers, which makes that part of it even worse than existing Posix signal API.

The whole point of the facility is to let different components share a signal without stepping on each other. Why would you need to retrieve the list of handlers?

> The proposed API does not address challenges of using signals in multi-threading programs.

This claim is too vague to rebut. What specific "challenges" are you talking about?

> The article mentions, that signal handlers can't be reliably unloaded, but proposed API does not address it.

The proposed API works fine with library unloading: a library can unregister whatever handlers it's installed just before being unloaded (e.g., in a static destructor), and this unregister operation is guaranteed to be safe no matter what the order handlers are unloaded.

> Overall the proposed interface brings little to the table

It allows multiple components to safely share signals. The objections you've mentioned are based on your misunderstanding my proposal.

> does not work well alongside with existing sigaction()

Yes it does. The article talks about this interaction specifically.

> I imagine, that if it had more technical "meat"

The glibc people literally think that nobody should be using signals. That's their objection, not anything you've talked about.

Re: Use mmap with care

#110

Post by ben509 »

btilly wrote: post #100
ben-schaaf wrote: post #24

Earlier quoted context omitted.

This is essentially how databases like PostgreSQL work, but in essence it only avoids the sys-call overhead. The OS is already caching the file, regardless of mmap, so using pread would have likely been enough for us. It totally would have been simpler overall, but each incremental step we made was significantly less work than the refactoring required for pread.

It totally would have been simpler overall, but each incremental step we made was significantly less work than the refactoring required for pread. Question. In 10 years will you be saying this about the next incremental problem that you run into? If you think this likely, then the next incremental problem is an excuse to do it right.

Mmap is right, though. Pread would also be right. There's a tradeoff and the complexity argument would only win if they knew all this when they started.
Post reply on HN