Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use mmap with care

sublimetext.com

Post reply on HN
Print view View original

31–40 of 218 posts

Re: Use mmap with care

#31

Post by 0db532a0 »

quotemstr wrote: post #25
_0w8t wrote: post #20

Earlier quoted context omitted.

Alternatively one can run a separated process that does mmap and runs the calculations or whatever that needs to access the file as quickly as possible and do the the straightforward recovery in the parent process when the child process dies. The drawback is the need to some form of RPC, but there a lot of libraries to do that without much hustle.

You can do that with a MAP_ANONYMOUS | MAP_SHARED mapping too: that kind of mapping is writable by both parent and child, but isn't backed by a disk file and so can't be truncated or surprise-removed. The article's points about mmap infelicity applies mostly to mappings of disk files. Anonymous mappings don't have the same problems.

The article points out the infelicity as applied to mappings of files into multi-threaded programmes. What fpoling suggests is a solution to the POSIX signals problem where anonymous mmaps are not an option.

Re: Use mmap with care

#32

Post by quotemstr »

Pahr3yah wrote: post #30
quotemstr wrote: post #21

There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said , there's nothing wrong with mmap or SIGB…

If you roughly know your access patterns in advance you can reduce the page fault costs with madvise.

Sure, but you still end up with double-caching and frequent entry in the kernel. madvise will never give you as much control and flexibility as just getting the system out of the way and doing the work yourself. (DB systems can also use fun tricks like compressing their cached pages.)

Re: Use mmap with care

#33

Post by atemerev »

Things are so much better if you are not writing apps for general public (mine are trading-related). You can tell your few clients — make sure that the access to mmapped file is exclusive — and get away with it.

And yes, mmap is the awesomest thing out there.

Re: Use mmap with care

#34

Post by Hackbraten »

ben-schaaf wrote: post #2

Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.

Small side note for completeness:

The effects of oh-noes-my-file-is-gone can be somewhat mitigated by using the heuristics built into NSData (instead of using mmap directly).

For example, you call NSData’s `dataWithContentsOfFile:options:error:` with the `NSDataReadingMappedIfSafe` option [1]. The framework will then transparently mmap the file unless it believes there’s an elevated risk of the file going away.

Apple doesn’t disclose how NSData exactly makes that decision; however, I’ve found a few reports that say it uses mmap internally when the file is on the root filesystem, and fall back on an in-memory copy otherwise.

It’s a rather dumb heuristics though, and may not solve the issue entirely.

[1] https://developer.apple.com/documentation/foundation/nsdatar...

Re: Use mmap with care

#35

Post by ben-schaaf »

saagarjha wrote: post #6

> Using setjmp and longjmping from a signal handler is actually unsafe. It seems to cause undefined behaviour, especially on MacOS. Have you considered making a dispatch_source_t of type DISPATCH_SOURCE_TYPE_SIGNAL and handling all signals in a dispatch queue, instead of trying do figure out what kind of behavior is legal in a signal handler? > If a library such as Breakpad registers for Mach exception messages, and…

> Have you considered making a dispatch_source_t

I think that would have been considerably more work than finding the SO answer that says you need to use sigsetjmp, and would probably still conflict with Breakpad ;)

> Would it be possible to install your own handler before Breakpad does?

I may be wrong, but I think you can only register one exception handler per "task" (process), so Breakpad would override ours.

Re: Use mmap with care

#36

Post by sara17372 »

World's biggest cat Even you can't believe it's size See her video https://howto105.blogspot.com/2019/04/worlds-biggest-cat.htm...

They found world's biggest fish Even you can't believe how big fish is this That's unbelievable http://bit.ly/2FHl4pM

Even some goats are better than humans See that she did and they made her as mayor of city http://bit.ly/2UjTcS5 See the video a dog shopping and lots of other acts like humans http://bit.ly/2DCpo9I

Even you can't believe there is a hen in world's biggest animals See the video how they looks and what they do https://cutt.ly/1rOU7Q

Re: Use mmap with care

#37

Post by rwmj »

And that's just for reading. Writing, especially if you want to be sure when the writes hit the backing file, or in what order, or if you run out of disk space, is another kettle of problems.

I wonder how Multics dealt with all this, since AIUI in that system everything was effectively an mmapped file.

Re: Use mmap with care

#38

Post by arethuza »

_0w8t wrote: post #20
nneonneo wrote: post #5

Even without NFS, using mmap requires being real careful about signals - SIGBUS can be raised any time the underlying file operation fails, including because someone else truncated the file, or because the underlying storage had an error (disk error, removed media, network storage). And, as this post so eloquently illustrates (and through my personal experience), handling SIGBUS/SIGSEGV cleanly in a multithreaded pro…

Alternatively one can run a separated process that does mmap and runs the calculations or whatever that needs to access the file as quickly as possible and do the the straightforward recovery in the parent process when the child process dies. The drawback is the need to some form of RPC, but there a lot of libraries to do that without much hustle.

I guess if you are using RPC you could forget about NFS and just run your "editor server" on the server where the file is actually stored :-)

Re: Use mmap with care

#39

Post by CoolGuySteve »

I feel like this is kind of a dumb design.

You want to abstract two different kinds of file reader: an mmap reader and a regular reader. (And I would add a gz reader, personally).

Then by inspecting the properties of the file, you can determine if it is local when opening, and if so, mmap the file.

I say this because if the file is coming via the network or a FAT32 partition you’re not going to save much time with mmap relative to the read speed anyways.

Re: Use mmap with care

#40

Post by ben-schaaf »

quotemstr wrote: post #26
ben-schaaf wrote: post #24

Earlier quoted context omitted.

This is essentially how databases like PostgreSQL work, but in essence it only avoids the sys-call overhead. The OS is already caching the file, regardless of mmap, so using pread would have likely been enough for us. It totally would have been simpler overall, but each incremental step we made was significantly less work than the refactoring required for pread.

> The OS is already caching the file Not necessarily. With O_DIRECT, pread() doesn't put pages into page cache: it just DMAs them directly into your process. Using O_DIRECT and the process-private caching we've been discussing, sophisticated programs (like databases) can (and do!) implement their own "page cache" systems. And because databases have access pattern information that the generic kernel VM subsystem doesn…

I might have undersold the performance advantage of writing your own cache, but let me reiterate the point I was trying to make: The reason we didn't consider doing so was because we weren't having a performance issue. Writing our own cache would be strictly more work than just using pread and accomplished the same thing.
Post reply on HN