The mmap pattern
john.freml.in
The mmap pattern
1–10 of 48 posts
Re: The mmap pattern
#2I did this for a high-performance time sequence database of robot activity. It's pretty hard. You need custom data structures for everything (no STL), adding fields to the data structures requires a schema migration tool, and it's easy to have subtle bugs around things like hash table growth.
However, it is insanely fast when it works.
Re: The mmap pattern
#3I did this for a high-performance time sequence database of robot activity. It's pretty hard. You need custom data structures for everything (no STL), adding fields to the data structures requires a schema migration tool, and it's easy to have subtle bugs around things like hash table growth. However, it is insanely fast when it works.
This is one place where I'm wondering how Cap'n Proto[1] would help. Since the structures are already made for in memory usage I think it might work pretty well. And given that it's all versioned/tagged structure wise it should be easy to do the schema migration.
Re: The mmap pattern
#4Google code search uses an mmapped index of trigrams, in Go:
https://code.google.com/p/codesearch/source/browse/index/mma...
Re: The mmap pattern
#5For a while I've wanted a nice C library for an mmapped heap with allocation and common data structures (and maybe locks?) - all the usual stuff you'd expect in a standard library, but with support for relative offsets instead of pointers, crash robustness, introspection, and other features required to work well with a persistent file. I do not know any library of this type that currently exists.
Re: The mmap pattern
#6I did this for a high-performance time sequence database of robot activity. It's pretty hard. You need custom data structures for everything (no STL), adding fields to the data structures requires a schema migration tool, and it's easy to have subtle bugs around things like hash table growth. However, it is insanely fast when it works.
kdb is mmapping db files - it's also a high-perf time series db, with exactly the same advantages and disadvantages what you mention.
Re: The mmap pattern
#7I did this for a high-performance time sequence database of robot activity. It's pretty hard. You need custom data structures for everything (no STL), adding fields to the data structures requires a schema migration tool, and it's easy to have subtle bugs around things like hash table growth. However, it is insanely fast when it works.
You can't use most STL containers by default, but Boost Interprocess solves most of that: http://www.boost.org/doc/libs/1_55_0/doc/html/interprocess/a...
Re: The mmap pattern
#8TFA says "By mmap'ing a region and then accessing into it, the overhead of creating multitudes of small interlinked items can be reduced hugely."
You do not need to call mmap directly, because malloc() will do it if your allocation exceeds MMAP_THRESHOLD (usually on the order of a megabyte). So you get this optimization "for free"--the only important part is that you do not call malloc() for each tiny object.
Re: The mmap pattern
#9This is basically used for this little gem:
LMDB is at the heart of the ubiquitous LDAP ( OpenLDAP ) and is very well optimized ( look at his benchmarks ). Now they are optimized for reading, which is important.
I would imagine mmap-ing with large amount of write will result in unpredictable performance....
Re: The mmap pattern
#10For a while I've wanted a nice C library for an mmapped heap with allocation and common data structures (and maybe locks?) - all the usual stuff you'd expect in a standard library, but with support for relative offsets instead of pointers, crash robustness, introspection, and other features required to work well with a persistent file. I do not know any library of this type that currently exists.
I believe this would qualify as a Java library https://github.com/peter-lawrey/Java-Chronicle
"This library also supports distributed, durable, observable collections (Map, List, Set)" "It uses almost no heap, trivial GC impact, can be much larger than your physical memory size (only limited by the size of your disk) and can be shared between processes with better than 1/10th latency of using Sockets over loopback."