Live data from Hacker News

Lightning Memory-Mapped Database Manager (LMDB) 1.0

lmdb.tech

71–73 of 73 posts

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#71
post #32

Earlier quoted context omitted.

IIRC: it is to leverage the OS page cache rather than having a separate buffer pool in user land. By default lmdb uses normal pwrite/fsync for the write path, but can optionally use a writable mapping and (presumably) msync. However, some people think there are problems with this usage: (pdf warning) https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf

How is pwrite/fsync any better than mmap/msync? Both go through the page cache and combine asynchronous writeback with forced flush. One theoretical advantage of pwrite might be that you can handle I/O errors, but I’d like to see a case where recovering from an I/O error makes sense (rather than just crashing the database, which SIGBUS would do anyway by default).

write/fsync can be faster in a large dataset because writes let the filesystem know an explicit list of dirty pages, so fsync only needs to deal with them.

mmap/msync gives no hints about which pages are dirty (unless the app tracks them itself and msyncs them individually, which would completely defeat any reduced syscall advantage of using a writable mmap in the first place) so the entire map must be scanned for dirty pages.

In practice, the expected performance advantages of using a writable mmap just aren't there, and coupled with the ease of silent corruption, it's best to never use that approach.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#72
post #70

Earlier quoted context omitted.

LMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.

Dropping the explicit P_DIRTY flag in 1.0 is a neat change. What tracks which pages still need to be written back at commit now that the flag is gone?

The txnID was added to the page header to enable support for incremental backup. As a consequence, it's sufficient to compare a page's txnID to the current txnID to know if it's dirty or not, and spilled pages don't need a cleanup pass to clear their dirty bit on commit so commits of large txns are faster now.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#73

Earlier quoted context omitted.

You're already depending on the OS for many other things. Depending on it for page caching is just one more thing.

This level of reasoning is insufficient when building reliable systems. The consequences of depending on the OS for page caching are different than the consequences of depending on it for device drivers, file systems, or scheduling.

Don't waste your energy. If you look at the guy's other responses, it will tell you exactly what kind of person you are dealing with.
Post reply on HN