LMDB – Lightning Memory-Mapped Database Manager
1–10 of 23 posts
Re: LMDB – Lightning Memory-Mapped Database Manager
#2I like LMDB, but why does ~most sql/nosql use LSM/rocksdb compared to it ? At least the ones going for read-speed ? Cause of missing WAL ?
There is also a fork? who claims is better/more-features than LMDB: https://github.com/leo-yuriev/libmdbx
Re: LMDB – Lightning Memory-Mapped Database Manager
#3The web page seems to suggest that robust POSIX semaphores are Linux-specific, while they've been available in FreeBSD for quite some time. I wonder if they detect it properly, or is there some actual problem in FreeBSD's implementation?
Re: LMDB – Lightning Memory-Mapped Database Manager
#4Is there a design doc or talk about the internals?
In particular are there any good resources about the details of using memory mapping?
I know how to implement persistent data structures (and it seems like lmdb is just a persistent b+-tree). But I don't know how to make it persist to disk. Is it as simple as using a memory mapped file for all memory allocations? Can all data structures be turned into a "database" in this way? If your workload fits in memory is there any performance difference between in-memory data structures? When do writes actually flush? What happens if multiple processes use the same file? etc
Re: LMDB – Lightning Memory-Mapped Database Manager
#5> Data pages use a copy-on- write strategy so no active data pages are ever overwritten, which also provides resistance to corruption and eliminates the need of any special recovery procedures after a system crash.
But I imagine this is somewhat slower than keeping a log (and rewinding it if necessary)?
Re: LMDB – Lightning Memory-Mapped Database Manager
#6Is there a design doc or talk about the internals? In particular are there any good resources about the details of using memory mapping? I know how to implement persistent data structures (and it seems like lmdb is just a persistent b+-tree). But I don't know how to make it persist to disk. Is it as simple as using a memory mapped file for all memory allocations? Can all data structures be turned into a "database" in…
See these two talks by 'hyc...
LMDB talk at DEVOXX (2013) [video] https://youtu.be/Rx1-in-a1Xc
LMDB CMU Databaseology Lecture (2015) [video] https://youtu.be/tEa5sAh-kVk
Re: LMDB – Lightning Memory-Mapped Database Manager
#7I like LMDB, but why does ~most sql/nosql use LSM/rocksdb compared to it ? At least the ones going for read-speed ? Cause of missing WAL ? There is also a fork? who claims is better/more-features than LMDB: https://github.com/leo-yuriev/libmdbx
Google started the trend of LSM with its release of leveldb. But leveldb hasn't been updated in a long time. Facebook forked leveldb and renamed it to rocksdb. Those are the only two LSM databases I know of, and IMO they are really the same thing. Meanwhile, lmdb vs. rocksdb/leveldb is a frequently asked question that seems to have no clear answer. Test on your hardware to find the best solution for your use case.
Re: LMDB – Lightning Memory-Mapped Database Manager
#8I’ve used LMDB as a simpler alternative to SQLite as “an alternative to fopen”. The goal was simply robust file writes in the face of unpredictable server reboots for a tiny Python program writing data to be processed later by a tiny C++ program.
That’s harder than it sounds to roll by hand with fopen. SQLite with write ahead logging is pretty much as good as it gets for reliablity, but SQL at all was overkill for the task. LMDB is a close second and it’s memory mapped key-value interface is much simpler. . Would write again.
Re: LMDB – Lightning Memory-Mapped Database Manager
#9I like LMDB, but why does ~most sql/nosql use LSM/rocksdb compared to it ? At least the ones going for read-speed ? Cause of missing WAL ? There is also a fork? who claims is better/more-features than LMDB: https://github.com/leo-yuriev/libmdbx
Google started the trend of LSM with its release of leveldb. But leveldb hasn't been updated in a long time. Facebook forked leveldb and renamed it to rocksdb. Those are the only two LSM databases I know of, and IMO they are really the same thing. Meanwhile, lmdb vs. rocksdb/leveldb is a frequently asked question that seems to have no clear answer. Test on your hardware to find the best solution for your use case.
It primarily depends on your requirements, as a rule of thumb:
* If your workload is random-writes heavy, choose lsm
* If your workload is serial-writes heavy, both are similar
* If your workload is read-heavy (random or not) go for lmdb
Re: LMDB – Lightning Memory-Mapped Database Manager
#10Is there a design doc or talk about the internals? In particular are there any good resources about the details of using memory mapping? I know how to implement persistent data structures (and it seems like lmdb is just a persistent b+-tree). But I don't know how to make it persist to disk. Is it as simple as using a memory mapped file for all memory allocations? Can all data structures be turned into a "database" in…
There is much more information on the symas website (https://symas.com/lmdb/technical/) (see all the talks links)