I'm sorry, this is probably a newbish question, but why didn't he use mmap for writes?
The mmap pattern
41–48 of 48 posts
Re: The mmap pattern
#42"By preserving state explicitly to memory backed files, several classes of unexpected events causing the program to crash can be recovered from with minimal disruption." The cure is surely worse than the disease! The program state in memory at the time of abnormal termination is likely to be inconsistent, leading to an unusable file. The subset of that that happens to have been committed to disk is likely to be worse…
Also, beyond only the state in the file being inconsistent, even if you manage to get a complete atomic write of your program state before you crash, the very same program state that caused the crash (if the crash was your own program's fault) is now being faithfully read back from disk! You're just asking to crash again immediately by blindly loading it all back.
Indeed. Just like this:
http://thedailywtf.com/Articles/Serial-Port-Pit-Bull.aspx
Re: The mmap pattern
#43The act of serializing data structures and reloading them can be a guard against long-term corruption. If you only use mmap, there's a risk of some corner of the object graph getting subtly wrong owing to a bug in one version of the software, and never getting repaired. Versioning of data structures is also a problem. I'd leave this pattern for use cases for which copying of memory on load has a measurable impact on…
Isn't the potential for long-term corruption purely related to not rewriting the entire file from scratch every time you make changes? Even without mmap, any data store where you make incremental changes is potentially vulnerable to this sort of thing. One obvious example would be filesystems. They're basically specialized databases which treat your whole disk as a single gigantic file, and of course there's a long,…
Most app file formats do not have the same reliability testing level as filesystems. Filesystems need years of use by brave people before they're considered OK for production.
If you're updating files in place without a whole rewrite, then you are effectively writing a filesystem.
I prefer to use a different approach where possible: write a diff log, and occasionally do a rewrite of the whole file.
When you do have to treat the on-disk file as a filesystem, I'd explicitly treat it as such. Have a low-level structure that does not need modification for different versions, is portable and highly tested. Do application-level structure with normal read-modify-write, but layered on top of the virtual filesystem layer. Consider having two modes, one where the virtual filesystem just proxies the real filesystem, and instead of a "file", you have a directory structure on disk. Consider using an existing component for this (zip archive libraries are popular, but not particularly efficient for writing).
Mmap does make the problem worse. It encourages you to be lazy towards versioning and portability.
Re: The mmap pattern
#44This is basically used for this little gem: http://symas.com/mdb/ LMDB is at the heart of the ubiquitous LDAP ( OpenLDAP ) and is very well optimized ( look at his benchmarks ). Now they are optimized for reading, which is important. I would imagine mmap-ing with large amount of write will result in unpredictable performance....
Writes to mmap-ed regions aren't very reliable: besides the synchronization issue there's also the question of what happens on an I/O error or ENOSPC. Usually your program just gets a SIGBUS and crashes. IIRC LMDB uses regular syscalls to update the mapping, and use mmap only for read.
Correct, at least for the original design of LMDB. We later added an option to write thru the mmap as well, at the request of some other users. Since LMDB is quite careful about writing and use of fsync/msync, reliability generally isn't an issue, but yes, hard errors on the disk and such can crash your program when using a writable mmap. Also, in the default mode of read-only mmap, the DB is incorruptible, stray pointer writes will crash the program instead of corrupting the DB. With the writable mmap you obviously have no such protection. We recommend developing your app with the default read-only mode, and only enabling the writable mmap after your app code is well tested.
Re: The mmap pattern
#45> One very key architectural decision for a system is the degree of reliability that it should possess. This is an explicit trade-off between the rapidity of development (in particular the level of indoctrination needed before new contributors are able to augment the feature set) and the operational stability. By preserving state explicitly to memory backed files, several classes of unexpected events causing the prog…
Your data structure design needs serious help if this is a problem for you. In OpenLDAP we store LDAP entries which can contain variable numbers of attributes and variable numbers of values per attribute, all in a unified format. Redefining your data structures is as simple as adding or removing an attribute or value from an entry. No sweat. Rolling your own data model is probably a bad idea...
Re: The mmap pattern
#46Is it me or does this article completely ignore two huge downsides with mmap - number one is that page faults are expensive, number two is that i/o failures become bad pointer dereferences.
Page faults are no more expensive than any other physical I/O. If the data you want isn't already in cache, a seek()/read() is as expensive as a page fault. Moreso really, because the read has to do a buffer copy too.
Fair point about I/O failures though.
Re: The mmap pattern
#47"By preserving state explicitly to memory backed files, several classes of unexpected events causing the program to crash can be recovered from with minimal disruption." The cure is surely worse than the disease! The program state in memory at the time of abnormal termination is likely to be inconsistent, leading to an unusable file. The subset of that that happens to have been committed to disk is likely to be worse…
Portability of binary data structures across environments is IMO irrelevant. We export to plaintext for migrations. Writing in a binary format that is portable across both 32 and 64 bit CPU architectures will be a waste on one or the other of the archs, and the point of approaches like this is to optimize runtime performance.
Your point about serialisation occurring with a known state is not lost in an mmap approach. If you have transactional writes (as LMDB does), then writing to the mmap also only occurs with a known state.
In reality, e.g. in OpenLDAP, you wind up with a minimal serialisation step, but with zero deserialisation. E.g., if I want to store a record: typedef struct foo { int len; char *data; } foo;
The struct and the data of the char ptr may originally have been obtained from two separate mallocs and thus reside at very different locations in memory. My serializer would simply allocate a new block and store the character data contiguously with the end of the struct. If you mmap at a fixed address, then you can store the address of the character data directly in the struct's "data" field, and the next time you come around to use this record, no deserialization is needed. Explicit deserialization is truly just a waste.
Re: The mmap pattern
#48"By preserving state explicitly to memory backed files, several classes of unexpected events causing the program to crash can be recovered from with minimal disruption." The cure is surely worse than the disease! The program state in memory at the time of abnormal termination is likely to be inconsistent, leading to an unusable file. The subset of that that happens to have been committed to disk is likely to be worse…
Also, beyond only the state in the file being inconsistent, even if you manage to get a complete atomic write of your program state before you crash, the very same program state that caused the crash (if the crash was your own program's fault) is now being faithfully read back from disk! You're just asking to crash again immediately by blindly loading it all back.
This is why you must have ACID transactions to use this approach.
Even without mmap, if your program successfully writes an invalid data structure to disk, the next time you run and read it back, you will crash. That's orthogonal to the method used to write and read the program state.