Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The mmap pattern

john.freml.in

Post reply on HN
Print view View original

21–30 of 48 posts

Re: The mmap pattern

#21

Post by guard-of-terra »

IgorPartola wrote: post #18
guard-of-terra wrote: post #17

"explicitly choosing a file and mapping that into the memory space of the program means it can persist across crashes" It's not that your program crashes, it's that your memory becomes corrupt and that causes the crash. Restarting program is essentially redoing all the memory. If you persist your memory your program will crash on run.

Different types of crashes and different memory. You can crash because you program did not handle a network error well, etc. Also, you do not need to use mmap'ed memory for everything. You can still use the stack and the heap for temporary structures, but store permanent data in the mmap'ed segments. Think about persistent data vs running queries in a database server.

How is it then different from sqlite?

Re: The mmap pattern

#22

Post by jadc »

comex wrote: post #5

For a while I've wanted a nice C library for an mmapped heap with allocation and common data structures (and maybe locks?) - all the usual stuff you'd expect in a standard library, but with support for relative offsets instead of pointers, crash robustness, introspection, and other features required to work well with a persistent file. I do not know any library of this type that currently exists.

You can also check out libgist at http://gist.cs.berkeley.edu/libgist-2.0/

Re: The mmap pattern

#23

Post by tsukikage »

"By preserving state explicitly to memory backed files, several classes of unexpected events causing the program to crash can be recovered from with minimal disruption."

The cure is surely worse than the disease!

The program state in memory at the time of abnormal termination is likely to be inconsistent, leading to an unusable file. The subset of that that happens to have been committed to disk is likely to be worse.

Contrast with the traditional approach of explicit serialisation / deserialisation for persistence:

* serialisation occurs when the program is in a known state. We can reason about what invariants hold at the point when state is sent to storage. We can take steps to confirm that data actually has been committed to persistent storage before we treat the operation as complete.

* deserialisation recreates program state in a clean environment from minimal data. This has the side effect of reinitialising state that may have caused the unexpected termination. Coupled with the ability to reason about program state at the point of serialisation it makes it feasible to attempt error recovery.

* Explicit serialisation and deserialisation stages make it possible to construct persistent state that is portable across environments and through development

mmap() is blindingly fast, but it is a tradeoff between performance, complexity and reliability - naive use across the board is unlikely to improve all three!

Re: The mmap pattern

#24

Post by Negitivefrags »

brianpgordon wrote: post #19

> One very key architectural decision for a system is the degree of reliability that it should possess. This is an explicit trade-off between the rapidity of development (in particular the level of indoctrination needed before new contributors are able to augment the feature set) and the operational stability. By preserving state explicitly to memory backed files, several classes of unexpected events causing the prog…

That depends on how they were created.

On our game project we use a data tool that allows you to create tables of data out of columns with specific data types as well as references to other table rows.

When you export from the tool it creates data files but also generates the C++ struct that represents the schema. The loader simply mmaps the data in to memory. Variable length data like strings gets packed at the end of the table with the fixed length records containing relative pointers in to it.

The data on disk is stored in the exact format that is needed at run time.

References to rows in other tables turn in to pointer like objects so you get a perfectly natural syntax as if the data had been loaded in to hand written classes.

It's perfect for rapid development because designers can add new columns or tables of data and then just tell the programmer what they are called.

It also loads lightning fast and has excellent run time properties due to cache coherency.

Re: The mmap pattern

#25

Post by barrkel »

The act of serializing data structures and reloading them can be a guard against long-term corruption.

If you only use mmap, there's a risk of some corner of the object graph getting subtly wrong owing to a bug in one version of the software, and never getting repaired.

Versioning of data structures is also a problem.

I'd leave this pattern for use cases for which copying of memory on load has a measurable impact on performance, and where required integrity of data isn't very high - ideally, the data should be able to be regenerated from primary sources.

One case I know of where this technique was used was the precompiled headers file for the Borland C++ compiler. There, it makes sense on both counts.

Re: The mmap pattern

#26

Post by stuaxo »

simcop2387 wrote: post #3
tlb wrote: post #2

I did this for a high-performance time sequence database of robot activity. It's pretty hard. You need custom data structures for everything (no STL), adding fields to the data structures requires a schema migration tool, and it's easy to have subtle bugs around things like hash table growth. However, it is insanely fast when it works.

This is one place where I'm wondering how Cap'n Proto[1] would help. Since the structures are already made for in memory usage I think it might work pretty well. And given that it's all versioned/tagged structure wise it should be easy to do the schema migration. [1] http://kentonv.github.io/capnproto/

Wow, that looks great !

Re: The mmap pattern

#27

Post by asveikau »

Is it me or does this article completely ignore two huge downsides with mmap - number one is that page faults are expensive, number two is that i/o failures become bad pointer dereferences.

Re: The mmap pattern

#28

Post by nly »

comex wrote: post #5

For a while I've wanted a nice C library for an mmapped heap with allocation and common data structures (and maybe locks?) - all the usual stuff you'd expect in a standard library, but with support for relative offsets instead of pointers, crash robustness, introspection, and other features required to work well with a persistent file. I do not know any library of this type that currently exists.

For C++:

http://www.boost.org/doc/libs/1_55_0/doc/html/interprocess.h...

"Boost.Interprocess offers useful tools to construct C++ objects, including STL-like containers, in shared memory and memory mapped files"

Re: The mmap pattern

#29

Post by nly »

> Modern CPUs have the ability to arbitrarily (generally at a 4kB granularity) map memory addresses as visible to a program to physical RAM addresses

While true and exceptionally powerful TLBs are limited in size and, being essentially a hash table put in to your CPU by either Intel and AMD, are likely optimised for common-case mapping patterns. Of course, these penalties still pale in comparison to disk I/O... but kind of a downer if you're dreaming of something crazy like a linked list where every node lives in a separate random page.

Re: The mmap pattern

#30

Post by IgorPartola »

guard-of-terra wrote: post #21
IgorPartola wrote: post #18

Earlier quoted context omitted.

Different types of crashes and different memory. You can crash because you program did not handle a network error well, etc. Also, you do not need to use mmap'ed memory for everything. You can still use the stack and the heap for temporary structures, but store permanent data in the mmap'ed segments. Think about persistent data vs running queries in a database server.

How is it then different from sqlite?

I don't know how sqlite3 works so I cannot tell you, but it sounds like a pattern that would fit it well.
Post reply on HN