Lightning Memory-Mapped Database Manager (LMDB) 1.0 • Hacker News Board

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#61

Post by quotemstr » Fri, Jul 03, 2026, 5:35 PM UTC

hyc_symas wrote: post #56

bagxrvxpepzn wrote: post #21

Earlier quoted context omitted.

> I've never understood the fascination some people have with mmap. Uncommonly used system calls give user-space programmers the sensation of learning something. > Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program. Yes, you're opting into non-determ…

Nonsense. The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm. But when you do it yourself you have to juggle the fine details of which indices to prioritize, and you will never get it perfect. If you're not running a dedicated machine, as soon as any other processes run all your careful tuning goes out the window. Since LMDB manag…

> The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm

First of all, even the kernel can do better than simple LRU. We have MGLRU now for example. That said, the kernel is at a structural disadvantage.

A general purpose eviction and prefetch algorithm is like an automatic transmission on a car. It can react only to what it's seen.

When you drive stick, you can react to what you can see on the road ahead of you. A database has a query plan. It can see the future as well as remember the past. It has more information than the kernel.

> So a simpleminded LRU always makes optimal use of available cache, regardless of access pattern or other load on the system

That cannot be true. If I have a random access pattern, LRU will perform no better than random. If I have a future-oracle, I can just evict what's most distant in my set of future accesses.

Regardless of whether you're right about the suitability of LRU for this or that workload, it's simply false, mathematically, from a computer science POV, that LRU is optimal.

And if you go around making confidently wrong claims like this, one must wonder about what else you are wrong. If you want to be disagreeable in public, fine: just make sure you have math on your side first.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#62

Post by quotemstr » Fri, Jul 03, 2026, 5:40 PM UTC

hyc_symas wrote: post #55

cmrdporcupine wrote: post #43

Earlier quoted context omitted.

Obligatory: https://db.cs.cmu.edu/mmap-cidr2022/ Consensus says "don't do it" ... That said, having written my own buffer pool and paging, etc... in pure naive benchmarks ... it's actually kinda hard to beat mmap. And LMDB is really fast for what it is. In real world workflows I think the story is more complicated. Especially under higher concurrency.

Obligatory "that paper is garbage" https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...

I read the linked post. You're not making a good argument.

The authors aren't arguing that a mmap database is worse because it's "more complex". They are arguing it must work with less information. You haven't refuted the original paper, but you have made me more skeptical of LMDB.

For example, you claim that applications "never" have control of memory. That's simply, again, false. We have explicit memory eviction and pinning operations. We even have VA-batched TLB shootdown IPIs via process_madvise. On some systems (AMD, soon Intel) we can do TLB invalidation without an IPI.

So no, you're just wrong in making the claim that you might as well use mmap because you can't control the memory lifecycle anyway. You absolutely can, and anyone reading this message can look up the relevant APIs for himself.

And you point to LMDB's benchmarks repeatedly as evidence you're right. That's not saying what you think it is. LMDB is fast despite being hobbled by vanilla kernel mmap. Yes, that means other databases are probably doing stupid things, but reverse stupidity is not intelligence.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#63

Post by hyc_symas » Fri, Jul 03, 2026, 6:29 PM UTC

quotemstr wrote: post #61

hyc_symas wrote: post #56

Earlier quoted context omitted.

Nonsense. The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm. But when you do it yourself you have to juggle the fine details of which indices to prioritize, and you will never get it perfect. If you're not running a dedicated machine, as soon as any other processes run all your careful tuning goes out the window. Since LMDB manag…

> The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm First of all, even the kernel can do better than simple LRU. We have MGLRU now for example. That said, the kernel is at a structural disadvantage. A general purpose eviction and prefetch algorithm is like an automatic transmission on a car. It can react only to what it's seen. W…

In the time it takes for your query optimizer to dissect a query and "look ahead" LMDB would have already answered a million queries. You think your magical "future oracle" is zero cost? How many KLOCs is it? LMDB's hot paths fit entirely inside a CPU's L1 cache.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#64

Post by hyc_symas » Fri, Jul 03, 2026, 6:34 PM UTC

quotemstr wrote: post #62

hyc_symas wrote: post #55

Earlier quoted context omitted.

Obligatory "that paper is garbage" https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...

I read the linked post. You're not making a good argument. The authors aren't arguing that a mmap database is worse because it's "more complex". They are arguing it must work with less information. You haven't refuted the original paper, but you have made me more skeptical of LMDB. For example, you claim that applications "never" have control of memory. That's simply, again, false. We have explicit memory eviction an…

You're dreaming. None of your explicit memory control operations mean anything in practice, because today everything runs in VMs with no actual control of the underlying hardware. Probably co-resident with an unknown number of other tenants.

As for what you claim the paper's authors were saying - I quoted their text verbatim. Your interpretation is not what they said.

They claimed using mmap safely is impossible, and using it correctly requires more complexity than a traditional DB design. The safety claim was already disproven by multiple researchers. To prove their second claim they would have had to produce a DB that did traditional buffer management and was simpler and more performant than using mmap. They never did any such thing, nor could they.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#65

Post by hyc_symas » Sat, Jul 04, 2026, 9:48 AM UTC

uroni wrote: post #35

markasoftware wrote: post #16

Earlier quoted context omitted.

Not amazing. In certain workloads I ran, once the db reached several hundred gb, writes would hang for longer and longer periods of time, eventually hours, while the db grew drastically in the background. https://news.ycombinator.com/item?id=30023623 seems to be the same issue, and it was serious enough that Shopify decided not to use lmdb. And yes, I ensured there were no outstanding long lived readers, verified wit…

That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior. Idk, if it was your issue, but for long running write transactions it doesn't spill to di…

By the way, you're wrong on both points - the cache of page mallocs is not infinite, and it does spill dirty pages to disk when necessary. And the latter is what bounds the number of malloc'd pages.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#66

Post by Kerollmops » Sat, Jul 04, 2026, 10:03 AM UTC

alfiedotwtf wrote: post #51

Kerollmops wrote: post #46

Earlier quoted context omitted.

I recently talked to Howard [1] about lies he was saying about Sanakirja, an LMDB-inspired disk allocator. That's always the same arguments: C is better than Rust for X, Y or Z reasons. While I reported a segfault just two weeks earlier... [2]. I love LMDB, we use it in Meilisearch (second most stared search engine on GitHub) [3] for about 7 years now. The main issues were related to write speed but we do a compactio…

Unfortunately, yeah. > The main issues were related to write speed but we do a compaction of the database and write performances are way better after that I'll ping you if I ever get around to rewriting a faster kv-store in Rust :)

Actually, you should take a look at [1]. It's made in Rust, inspired by LMDB, and supports a cool feature that allows close to anything to be implemented: allocating any page you want to store anything you want. The BTree storage is optional and you can implement whatever storage system you want. When storing a value to disk, you can allocate pages and decide exactly how you plan to store the bytes, allowing you not to store the length of them or to split your data into multiple pages, etc.

[1]: https://pijul.org/posts/2021-02-06-rethinking-sanakirja

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#67

Post by senderista » Sat, Jul 04, 2026, 7:34 PM UTC

thombles wrote: post #10

hmry wrote: post #6

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…

Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.

Isn’t that why the mapping is read-only by default?

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#68

Post by senderista » Sat, Jul 04, 2026, 7:40 PM UTC

tynorf wrote: post #32

zbentley wrote: post #29

Earlier quoted context omitted.

Wow, really? Then what’s the point of memory mapping in the fist place? Or do they suggest manual flush/sync actions for persistence.

IIRC: it is to leverage the OS page cache rather than having a separate buffer pool in user land. By default lmdb uses normal pwrite/fsync for the write path, but can optionally use a writable mapping and (presumably) msync. However, some people think there are problems with this usage: (pdf warning) https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf

How is pwrite/fsync any better than mmap/msync? Both go through the page cache and combine asynchronous writeback with forced flush. One theoretical advantage of pwrite might be that you can handle I/O errors, but I’d like to see a case where recovering from an I/O error makes sense (rather than just crashing the database, which SIGBUS would do anyway by default).

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#69

Post by senderista » Sat, Jul 04, 2026, 8:01 PM UTC

quotemstr wrote: post #28

teravor wrote: post #27

Earlier quoted context omitted.

with mmap you also don't have to worry about committing too much system memory, if another application needs it it will start evicting your cache.

You're right about that. Linux needs a way for userspace processes to participate in the kernel's shrinker system for reclaiming memory under pressure. Watching memory PSI is too coarse. MADV_FREE is too complicated and indiscriminate. You could imagine a notification FD, but then you've just reinvented PSI. You could imagine a synchronous signal, but everyone hates signals and won't couple any new functionality to t…

That’s why you do your own memory accounting in the database. Of course, that assumes you own the machine; for an “embedded” DB like LMDB something like PSI may be necessary.

Another possibility for reclaiming physical memory beside unused page decommit with MADV_FREE/MADV_DONTNEED (there might not be unused pages to decommit) is to manually page out cold anonymous pages with MADV_COLD/MADV_PAGEOUT (thanks Android). You can combine this with low swappiness so anonymous pages are unlikely to be paged out automatically when there are clean file-backed pages that can be reclaimed.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#70

Post by keynha » Sun, Jul 05, 2026, 6:44 AM UTC

hyc_symas wrote: post #53

uroni wrote: post #35

Earlier quoted context omitted.

That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior. Idk, if it was your issue, but for long running write transactions it doesn't spill to di…

LMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.

Dropping the explicit P_DIRTY flag in 1.0 is a neat change. What tracks which pages still need to be written back at commit now that the flag is gone?