Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lightning Memory-Mapped Database Manager (LMDB) 1.0

lmdb.tech

Post reply on HN
Print view View original

31–40 of 73 posts

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#31

Post by jnwatson »

markasoftware wrote: post #16
hmry wrote: post #6

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…

Not amazing. In certain workloads I ran, once the db reached several hundred gb, writes would hang for longer and longer periods of time, eventually hours, while the db grew drastically in the background. https://news.ycombinator.com/item?id=30023623 seems to be the same issue, and it was serious enough that Shopify decided not to use lmdb. And yes, I ensured there were no outstanding long lived readers, verified wit…

I've used LMDB in production for multi-terabyte databases, and we encountered the long-write time but found a solution.

The important idea is that LMDB offloads cache management almost completely to the OS. You have to become intimately familiar with the way that the page cache works and how to configure it.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#32

Post by tynorf »

zbentley wrote: post #29
thombles wrote: post #10

Earlier quoted context omitted.

Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.

Wow, really? Then what’s the point of memory mapping in the fist place? Or do they suggest manual flush/sync actions for persistence.

IIRC: it is to leverage the OS page cache rather than having a separate buffer pool in user land. By default lmdb uses normal pwrite/fsync for the write path, but can optionally use a writable mapping and (presumably) msync.

However, some people think there are problems with this usage: (pdf warning) https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#34

Post by jnwatson »

bagxrvxpepzn wrote: post #21
quotemstr wrote: post #19

I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just…

> I've never understood the fascination some people have with mmap. Uncommonly used system calls give user-space programmers the sensation of learning something. > Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program. Yes, you're opting into non-determ…

You're already depending on the OS for many other things. Depending on it for page caching is just one more thing.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#35

Post by uroni »

markasoftware wrote: post #16
hmry wrote: post #6

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…

Not amazing. In certain workloads I ran, once the db reached several hundred gb, writes would hang for longer and longer periods of time, eventually hours, while the db grew drastically in the background. https://news.ycombinator.com/item?id=30023623 seems to be the same issue, and it was serious enough that Shopify decided not to use lmdb. And yes, I ensured there were no outstanding long lived readers, verified wit…

That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior.

Idk, if it was your issue, but for long running write transactions it doesn't spill to disk. So you have all the changes being written to disk at the end of the transaction. One would think enabling write mapping fixes this, but it needs to mark all the pages as clean before commit, so same effect there. I fixed this for 0.9 here https://github.com/uroni/hs5/tree/main/external/lmdb . Will have to investigate if it is improved with 1.0, or if I need to redo the changes.

Edit: Just noticed that the issue is about free list in the file. Never had a problem with that, but I also had to replace that MIDL structure with something more scalable for the spilling.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#36

Post by bagxrvxpepzn »

jnwatson wrote: post #34
bagxrvxpepzn wrote: post #21

Earlier quoted context omitted.

> I've never understood the fascination some people have with mmap. Uncommonly used system calls give user-space programmers the sensation of learning something. > Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program. Yes, you're opting into non-determ…

You're already depending on the OS for many other things. Depending on it for page caching is just one more thing.

This level of reasoning is insufficient when building reliable systems. The consequences of depending on the OS for page caching are different than the consequences of depending on it for device drivers, file systems, or scheduling.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#38

Post by markasoftware »

uroni wrote: post #35
markasoftware wrote: post #16

Earlier quoted context omitted.

Not amazing. In certain workloads I ran, once the db reached several hundred gb, writes would hang for longer and longer periods of time, eventually hours, while the db grew drastically in the background. https://news.ycombinator.com/item?id=30023623 seems to be the same issue, and it was serious enough that Shopify decided not to use lmdb. And yes, I ensured there were no outstanding long lived readers, verified wit…

That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior. Idk, if it was your issue, but for long running write transactions it doesn't spill to di…

FWIW I had this issue even with the MDB_NOSYNC flag so it shouldnt be force flushing to disk unless I'm out of ram or whatever

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#39

Post by nomel »

hmry wrote: post #6

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…

I think lmdb is mostly unusable, for many use cases. I switched to libmdbx, which fixes all the issues [2] I (and most sibling comments) ran into with lmdb.

[1] https://github.com/Mithril-mine/libmdbx#improvements-beyond-...

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#40

Post by bch »

hmry wrote: post #6

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…

> stuck to SQLite instead

Different level of abstraction. I don’t think it’s highlighted enough either - this latest (1.0) and the previous 0.9.x are mutually incompatible, requiring essentially a dump/restore. It is mentioned (I forget which file ottofmh), but should be a

*HIGHLIGHT* in the CHANGES.

Post reply on HN