Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The mmap pattern

john.freml.in

Post reply on HN
Print view View original

31–40 of 48 posts

Re: The mmap pattern

#31

Post by edwintorok »

rdtsc wrote: post #9

This is basically used for this little gem: http://symas.com/mdb/ LMDB is at the heart of the ubiquitous LDAP ( OpenLDAP ) and is very well optimized ( look at his benchmarks ). Now they are optimized for reading, which is important. I would imagine mmap-ing with large amount of write will result in unpredictable performance....

Writes to mmap-ed regions aren't very reliable: besides the synchronization issue there's also the question of what happens on an I/O error or ENOSPC. Usually your program just gets a SIGBUS and crashes. IIRC LMDB uses regular syscalls to update the mapping, and use mmap only for read.

Re: The mmap pattern

#32

Post by mtdewcmu »

comex wrote: post #5

For a while I've wanted a nice C library for an mmapped heap with allocation and common data structures (and maybe locks?) - all the usual stuff you'd expect in a standard library, but with support for relative offsets instead of pointers, crash robustness, introspection, and other features required to work well with a persistent file. I do not know any library of this type that currently exists.

BerkeleyDB?

Re: The mmap pattern

#33

Post by mtdewcmu »

guard-of-terra wrote: post #21
IgorPartola wrote: post #18

Earlier quoted context omitted.

Different types of crashes and different memory. You can crash because you program did not handle a network error well, etc. Also, you do not need to use mmap'ed memory for everything. You can still use the stack and the heap for temporary structures, but store permanent data in the mmap'ed segments. Think about persistent data vs running queries in a database server.

How is it then different from sqlite?

SQLite does it for you; here you do it yourself. Presumably you'd have done a cost/benefit analysis before choosing this over SQLite, so your reasoning would be sound...

Re: The mmap pattern

#34

Post by mtdewcmu »

brianpgordon wrote: post #19

> One very key architectural decision for a system is the degree of reliability that it should possess. This is an explicit trade-off between the rapidity of development (in particular the level of indoctrination needed before new contributors are able to augment the feature set) and the operational stability. By preserving state explicitly to memory backed files, several classes of unexpected events causing the prog…

This sounds like it could be a good fit for mobile on iOS (I can't speak to android). If the app has significant startup costs, this lets you basically save a core dump of those structures that were slow to build. Then you could have an alternate fast startup mode that reads from a core dump.

Re: The mmap pattern

#35

Post by pm215 »

barrkel wrote: post #25

The act of serializing data structures and reloading them can be a guard against long-term corruption. If you only use mmap, there's a risk of some corner of the object graph getting subtly wrong owing to a bug in one version of the software, and never getting repaired. Versioning of data structures is also a problem. I'd leave this pattern for use cases for which copying of memory on load has a measurable impact on…

The "just mmap" pattern has also got portability issues if you might move the data file between systems which could have different endianness, pointer size, alignment requirements, etc. (Serialization done properly avoids this because you can swap to a consistent host-independent on-the-wire representation as part of the process.) This is also less of an issue for the cases you suggest where you can regenerate the data from a primary source -- you can just treat the host-dependent data file as a cache that can be blown away if moving to a different machine.

Re: The mmap pattern

#36

Post by chrisrohlf »

"or try to mmap each region to a well-known start address (and fail if it cannot obtain that address)."

Optimizations aside, beware this approach. ASLR is one of your two best friends (the other is DEP). When you purposely circumvent the protection it provides a security researcher somewhere will make you the topic of a very pointy blog post.

Re: The mmap pattern

#37

Post by icedchai »

brianpgordon wrote: post #19

> One very key architectural decision for a system is the degree of reliability that it should possess. This is an explicit trade-off between the rapidity of development (in particular the level of indoctrination needed before new contributors are able to augment the feature set) and the operational stability. By preserving state explicitly to memory backed files, several classes of unexpected events causing the prog…

You can plan ahead and leave "spare" space in the memory mapped structures for future expansion. Combined with versioning, you can take this quite far.

I've worked on financial transaction processing systems using memory mapped files as their primary means of data storage. It is very effective.

Re: The mmap pattern

#38

Post by mikeash »

barrkel wrote: post #25

The act of serializing data structures and reloading them can be a guard against long-term corruption. If you only use mmap, there's a risk of some corner of the object graph getting subtly wrong owing to a bug in one version of the software, and never getting repaired. Versioning of data structures is also a problem. I'd leave this pattern for use cases for which copying of memory on load has a measurable impact on…

Isn't the potential for long-term corruption purely related to not rewriting the entire file from scratch every time you make changes? Even without mmap, any data store where you make incremental changes is potentially vulnerable to this sort of thing.

One obvious example would be filesystems. They're basically specialized databases which treat your whole disk as a single gigantic file, and of course there's a long, proud history of programs used to repair corruption in them due to bugs or other problematic events.

To me, that's an argument for not using formats with incremental formats if you can get away with rewriting the file each time, but once you have enough data to where you can't afford a total rewrite each time, does mmap make the problem any worse?

Re: The mmap pattern

#39

Post by ninkendo »

tsukikage wrote: post #23

"By preserving state explicitly to memory backed files, several classes of unexpected events causing the program to crash can be recovered from with minimal disruption." The cure is surely worse than the disease! The program state in memory at the time of abnormal termination is likely to be inconsistent, leading to an unusable file. The subset of that that happens to have been committed to disk is likely to be worse…

Also, beyond only the state in the file being inconsistent, even if you manage to get a complete atomic write of your program state before you crash, the very same program state that caused the crash (if the crash was your own program's fault) is now being faithfully read back from disk! You're just asking to crash again immediately by blindly loading it all back.

Re: The mmap pattern

#40

Post by kentonv »

I had a G+ thread about mmap a couple months ago that surfaced a lot of really good points. I like mmap a lot, but there are certainly some down sides:

https://plus.google.com/u/0/+KentonVarda/posts/NKUUzx2nEsN

If you're looking for an easy way to exploit mmap in your code, Cap'n Proto is a serialization format that works similarly to Protocol Buffers but is designed to work well with mmap():

http://capnproto.org

(Disclosure: I am the author of Cap'n Proto... and also the former maintainer of protobufs.)

Post reply on HN