Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lightning Memory-Mapped Database Manager (LMDB) 1.0

lmdb.tech

Post reply on HN
Print view View original

51–60 of 73 posts

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#51

Post by alfiedotwtf »

Kerollmops wrote: post #46
alfiedotwtf wrote: post #44

Earlier quoted context omitted.

> And if compiler authors disagree, they are morons I remember arguing with Howard years ago on “C vs Rust”. He said that you don’t need Rust, you just have to be good at C programming, so I pointed out CVEs in LMDB attributed to his own bare hands… so there’s that.

I recently talked to Howard [1] about lies he was saying about Sanakirja, an LMDB-inspired disk allocator. That's always the same arguments: C is better than Rust for X, Y or Z reasons. While I reported a segfault just two weeks earlier... [2]. I love LMDB, we use it in Meilisearch (second most stared search engine on GitHub) [3] for about 7 years now. The main issues were related to write speed but we do a compactio…

Unfortunately, yeah.

> The main issues were related to write speed but we do a compaction of the database and write performances are way better after that

I'll ping you if I ever get around to rewriting a faster kv-store in Rust :)

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#53

Post by hyc_symas »

uroni wrote: post #35
markasoftware wrote: post #16

Earlier quoted context omitted.

Not amazing. In certain workloads I ran, once the db reached several hundred gb, writes would hang for longer and longer periods of time, eventually hours, while the db grew drastically in the background. https://news.ycombinator.com/item?id=30023623 seems to be the same issue, and it was serious enough that Shopify decided not to use lmdb. And yes, I ensured there were no outstanding long lived readers, verified wit…

That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior. Idk, if it was your issue, but for long running write transactions it doesn't spill to di…

LMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#54

Post by hyc_symas »

thombles wrote: post #10
hmry wrote: post #6

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…

Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.

Irrelevant, since LMDB doesn't dirty the pages in the mmap. The mmap is read-only.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#55

Post by hyc_symas »

cmrdporcupine wrote: post #43
quotemstr wrote: post #19

I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just…

Obligatory: https://db.cs.cmu.edu/mmap-cidr2022/ Consensus says "don't do it" ... That said, having written my own buffer pool and paging, etc... in pure naive benchmarks ... it's actually kinda hard to beat mmap. And LMDB is really fast for what it is. In real world workflows I think the story is more complicated. Especially under higher concurrency.

Obligatory "that paper is garbage" https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#56

Post by hyc_symas »

bagxrvxpepzn wrote: post #21
quotemstr wrote: post #19

I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just…

> I've never understood the fascination some people have with mmap. Uncommonly used system calls give user-space programmers the sensation of learning something. > Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program. Yes, you're opting into non-determ…

Nonsense. The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm. But when you do it yourself you have to juggle the fine details of which indices to prioritize, and you will never get it perfect. If you're not running a dedicated machine, as soon as any other processes run all your careful tuning goes out the window.

Since LMDB manages multiple tables as a tree of trees, no fine tuning is needed. The internal paths to every hot page automatically take priority, regardless of which index or how large each index is. So a simpleminded LRU always makes optimal use of available cache, regardless of access pattern or other load on the system.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#57

Post by hyc_symas »

jayct wrote: post #18
radiator wrote: post #4

Earlier quoted context omitted.

it is just a link to documentation

that could easily be trojan-horsed with links to malware if you are viewing it in a poorly secured setting (like public wifi), because you can't verify the origin. so the best we can say about the author is that we are getting inconsistent signals on how seriously they understand and implement security concerns. so better review that code carefully before use, rather than assuming their expertise from release notes.

If you're downloading binaries from a plaintext documentation site, I think that's on you.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#58

Post by bagxrvxpepzn »

hyc_symas wrote: post #56
bagxrvxpepzn wrote: post #21

Earlier quoted context omitted.

> I've never understood the fascination some people have with mmap. Uncommonly used system calls give user-space programmers the sensation of learning something. > Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program. Yes, you're opting into non-determ…

Nonsense. The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm. But when you do it yourself you have to juggle the fine details of which indices to prioritize, and you will never get it perfect. If you're not running a dedicated machine, as soon as any other processes run all your careful tuning goes out the window. Since LMDB manag…

First let me just say that while it's possible to interpret my original comment as uniquely applying to LMDB (or databases with similar page cache designs), in practice it applies to all general purpose databases including PostgreSQL and SQLite. This is because all general purpose databases will eventually fall short when it comes to tweaking behavior to meet application specific requirements, customizations notwithstanding. So to the extent that one should not use LMDB for anything that matters, one should also not use PostgreSQL or SQLite for anything that matters. If that corollary appears false in your frame of reference, then my statement about LMDB should also be false.

For high-stakes applications, you will have to maintain your own database code (either original or derived from an existing database) and that database code will need its own page caching layer (or a patched kernel), a generic page caching system (whether in-kernel with mmap or out of kernel) will not do. I acknowledge most applications don't operate in this regime.

> The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm.

This is not true. Applications often have specific high-priority data which should always exist in memory. That may be a moot point because you can do mlock() with mmap(). If we focus only on general-purpose caching, then even in that case there are many alternatives to LRU. SIEVE and ARC are two notable alternatives that perform significantly better for certain data. An application developer should be able to experiment with different general purpose caching strategies for different types of data, mmap() does not afford this.

Thank you Mr. Chu for your contributions to the technology commons and humanity in general.

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#59

Post by radiator »

hyc_symas wrote: post #55
cmrdporcupine wrote: post #43

Earlier quoted context omitted.

Obligatory: https://db.cs.cmu.edu/mmap-cidr2022/ Consensus says "don't do it" ... That said, having written my own buffer pool and paging, etc... in pure naive benchmarks ... it's actually kinda hard to beat mmap. And LMDB is really fast for what it is. In real world workflows I think the story is more complicated. Especially under higher concurrency.

Obligatory "that paper is garbage" https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...

"we don't have to pay back any vulture capitalists"! a good one

Re: Lightning Memory-Mapped Database Manager (LMDB) 1.0

#60

Post by hyc_symas »

radiator wrote: post #13
Retr0id wrote: post #12

Earlier quoted context omitted.

Cryptography engineers are not excluded from being lazy sysadmins.

What do you mean "lazy"? I thought you said TLS certs were free. Do you mean they cost something after all? Time, for example? Anyway, of course in case you feel the website is a risk, you should refrain from using it. Safety comes first.

Indeed, there's no need to use the doc website. There's nothing there that isn't embedded in the LMDB source code. All of the docs are generated from doxygen comments in the source.
Post reply on HN