Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lightning Memory-Mapped Database Manager (LMDB) 1.0

lmdb.tech

Post reply on HN
Print view View original

21–30 of 73 posts

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#21

Post by bagxrvxpepzn »

quotemstr wrote: post #19

I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just…

> I've never understood the fascination some people have with mmap.

Uncommonly used system calls give user-space programmers the sensation of learning something.

> Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program.

Yes, you're opting into non-determinism you don't control. When resources get constrained and everything can't be in memory and someone asks you why the database sucks, all you'll be able to do is shrug. Anyone who builds critical systems would never rely on the kernel making decisions like this. Don't use LMDB for anything that matters.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#22

Post by wmanley »

hilariously wrote: post #9

Maybe rephrase this part - "It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database." I generally do think read-write mode would offer higher write performance than read only as well :)

The context is in the sentence before your quote:

> The memory map can be used as a read-only or read-write map.

So presumably lmdb writes to the database using the `pwrite` syscall by default, but can optionally write via the mmap instead - if you are willing to accept the increased risk of accidental data corruption.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#24

Post by quotemstr »

ok123456 wrote: post #20
quotemstr wrote: post #19

I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just…

The OS handles all of that transparently, without requiring any additional code. I think that is the draw.

And that's adequate for casual programs. LMDB is big and serious enough to warrant the extra complexity (which, to be fair, is significant) of userspace buffer management. LMDB does the work once and all users benefit.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#25

Post by OrangeDelonge »

hmry wrote: post #6

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…

We evaluated it but chose RocksDB instead

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#26

Post by wmanley »

quotemstr wrote: post #19

I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just…

I use mmap with my SQLite database[1] because I have many concurrent SQLite connections (one per concurrent HTTP request) and I don't want each connection to have its own 2MB cache[2]. It's better that all the connections simply share the page cache.

[1]: https://sqlite.org/pragma.html#pragma_mmap_size

[2]: https://sqlite.org/pragma.html#pragma_cache_size

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#27

Post by teravor »

quotemstr wrote: post #19

I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just…

with mmap you also don't have to worry about committing too much system memory, if another application needs it it will start evicting your cache.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#28

Post by quotemstr »

teravor wrote: post #27
quotemstr wrote: post #19

I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just…

with mmap you also don't have to worry about committing too much system memory, if another application needs it it will start evicting your cache.

You're right about that.

Linux needs a way for userspace processes to participate in the kernel's shrinker system for reclaiming memory under pressure. Watching memory PSI is too coarse. MADV_FREE is too complicated and indiscriminate. You could imagine a notification FD, but then you've just reinvented PSI. You could imagine a synchronous signal, but everyone hates signals and won't couple any new functionality to them.

Shrinker-BPF attached to a memfd perhaps? A BPF shrinker could not only choose which pages to evict in a non-stupid way, but could notify userspace in some sane manner (e.g. setting a bitmask somewhere) that it's done so.

(Zero-fill as "notification" is insane and doesn't actually work because zero is a perfectly valid value in a lot of contexts.)

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#29

Post by zbentley »

thombles wrote: post #10
hmry wrote: post #6

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…

Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.

Wow, really?

Then what’s the point of memory mapping in the fist place? Or do they suggest manual flush/sync actions for persistence.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#30

Post by erikschoster »

hmry wrote: post #6

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…

I use it as a session store for a computer music system. It has worked well for me as a way to read mutable (by any client) parameters during synthesis, clients will often read dozens of parameters during a block of computation (a relatively short window of time in the low milliseconds typically) without adding any noticeable overhead to the render time for each block.

Edit: I also tried using it for larger blobs of data (like audio) but ended up only storing a reference to shared memory for larger blocks, anything larger than IIRC 4k that can't be stored in a single node kills performance, but for small values it seems pretty great.

Post reply on HN