[flagged]
Lightning Memory-Mapped Database Manager (LMDB) 1.0
41–50 of 73 posts
Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0
#42Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…
There's a project to have LMDB as a backend for sqlite. It originates in the LMDB author's (Howard Chu) own work.
https://lumosql.org/src/lumosql/doc/trunk/README.md
Also https://www.actordb.com/ uses LMDB with a sql queries. (Not sure if sqlite or not)
Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0
#43I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just…
Obligatory: https://db.cs.cmu.edu/mmap-cidr2022/
Consensus says "don't do it" ...
That said, having written my own buffer pool and paging, etc... in pure naive benchmarks ... it's actually kinda hard to beat mmap. And LMDB is really fast for what it is.
In real world workflows I think the story is more complicated. Especially under higher concurrency.
Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0
#44Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…
> And if compiler authors disagree, they are morons
I remember arguing with Howard years ago on “C vs Rust”. He said that you don’t need Rust, you just have to be good at C programming, so I pointed out CVEs in LMDB attributed to his own bare hands… so there’s that.
Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0
#45Like 8 years ago I encountered this package, I recall reading that it was designed by someone who programmed components for the Space Shuttles.
I was very impressed.
Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0
#46Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…
> And if compiler authors disagree, they are morons I remember arguing with Howard years ago on “C vs Rust”. He said that you don’t need Rust, you just have to be good at C programming, so I pointed out CVEs in LMDB attributed to his own bare hands… so there’s that.
I recently talked to Howard [1] about lies he was saying about Sanakirja, an LMDB-inspired disk allocator. That's always the same arguments: C is better than Rust for X, Y or Z reasons. While I reported a segfault just two weeks earlier... [2].
I love LMDB, we use it in Meilisearch (second most stared search engine on GitHub) [3] for about 7 years now. The main issues were related to write speed but we do a compaction of the database and write performances are way better after that. We never had any major DB corruption... I mean... other than when using it on Azure. Azure never works, that's expected, I suppose.
[1]: https://mastodon.social/@hyc/116838499082046918 [2]: https://bugs.openldap.org/show_bug.cgi?id=10522 [3]: https://github.com/meilisearch/meilisearch
Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0
#47New features in LMDB 1.0 include: - support for incremental backup - support for page-level checksums and encryption - support for DB on raw block devices - support for 2-phase commit - support for page sizes up to 64KB plus other minor additions to the API.
Yeah, and I also added support for parallel read your own writes where you can write tons of entries and spawn multi children read-only transactions from your writes where transaction and read from them in parallel.
We use this in Meilisearch [1] to post-process cache for our most common prefixes i.e., "w" will match "work", "word"... and computing this requires doing large unions of the documents matching those words.
Being able to do it in parallel is necessary, especially when you have billions of entries to operate on.
Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0
#48Bummer. I'm the maintainer of the Python bindings. I have to figure out how to support both versions now...
Yup! Same, I'm the maintainer of the main LMDB Rust wrapper [1] and I was maintaining heed and heed3 (because 1.0 was available from the mdb.master3 branch).
But now that it's LMDB 1.0, I need to find a better way to make it be the official one but I can't really rename heed3 into heed and heed into heed-0.9...
Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0
#49Bummer. I'm the maintainer of the Python bindings. I have to figure out how to support both versions now...
Yup! Same, I'm the maintainer of the main LMDB Rust wrapper [1] and I was maintaining heed and heed3 (because 1.0 was available from the mdb.master3 branch). But now that it's LMDB 1.0, I need to find a better way to make it be the official one but I can't really rename heed3 into heed and heed into heed-0.9... [1]: https://github.com/meilisearch/heed
heed09 and heed10 ? :)
Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0
#50Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…
If you use the default mode MDB_SYNC then it's reliable but can be slow for writes. For max write performance you need MDB_NOSYNC (IIRC that's what the official benchmarks use) but then the whole database can be unrecoverable in case of crash. It has happened to me.
Sqlite in WAL mode will never lose all your data and performance can be configured vs durability by setting pragma synchronous to full or normal.