Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

db.cs.cmu.edu

Post reply on HN
Print view View original

101–110 of 182 posts

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#101

Post by pclmulqdq »

jandrewrogers wrote: post #88
LAC-Tech wrote: post #63

Earlier quoted context omitted.

This is a classic appeal to authority. Let's play the argument, not the man. (My understanding is that the GP wrote LMDB, works on openLDAP, and was a maintainer for BerkelyDB for a number of years. But even if he'd only written 'hello, world!' I'm much more interested in the specific arguments).

I think the real argument is more nuanced. Where you see mmap() fail badly on Linux, even for read-only workloads, is under a few specific conditions: very large storage volumes, highly concurrent access, non-trivial access patterns (e.g. high-dimensionality access methods). Most people do not operate data models under these conditions, but if you do then you can achieve large integer factor gains in throughput by no…

GP wrote a key-value store called LMDB that is constrained to a single writer, and often used for small databases that fit entirely in memory but need to persist to disk. There's a whole different world for more scalable databases.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#102

Post by pclmulqdq »

arter4 wrote: post #39
dist1ll wrote: post #33

Earlier quoted context omitted.

The most common example is DPDK [1]. It's a framework for building bespoke networking stacks that are usable from userspace, without involving the kernel. You'll find DPDK mentioned a lot in the networking/HPC/data center literature. An example of a backend framework that uses DPDK is the seastar framework [2]. Also, I recently stumbled upon a paper for efficient RPC networks in data centers [3]. If you want to learn…

Interesting. I hear a lot more about sendfile(), kTLS and general kernel space tricks than I do about DPDK and userspace networking, but maybe it's just me. I do wonder what trend is going to win: bypass the kernel or embrace the kernel for everything? The way I see it, latency decreases either way (as long as you don't have to switch back and forth between kernel and user space), but userspace seems better from a se…

The people who use DPDK and the like are a lot quieter about it. The nature of kernel development means that people tend to hear about what you're doing, while DPDK and userspace networking tends to happen in more proprietary settings.

That said, I'm not sure many people write webservers in DPDK, since the Kernel is pretty well suited to webservers (sendfile, etc.). Most applications that use kernel-bypass are more specialized.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#103

Post by pclmulqdq »

hyc_symas wrote: post #79
gavinray wrote: post #74

Earlier quoted context omitted.

The argument is that: - Queries can trigger blocking page faults when accessing (transparently) evicted pages, causing unexpected I/O stalls - mmap() complicates transactionality and error-handling - Page table contention, single-threaded page eviction, and TLB shootdowns become bottlenecks

1 - for reading any uncached data, the I/O stalls are unavoidable. Whatever client requested that data is going to have to wait regardless. 2 - complexity? this is simply false. LMDB's ACID txns using MVCC are much simpler than any "traditional" approach. 3 - contention is a red herring since this approach is already single-writer, as is common for most embedded k/v stores these days. You lose more perf by trying to…

It's kind of disingenuous to talk about how great your concurrency system is when you only allow a single writer. RCU (which I imagine your system is isomorphic to) is pretty simple compared to what many DB engines use to do ACID transactions that involve both reads and writes.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#104

Post by neerajsi »

jasonhansel wrote: post #4

I've become convinced that there are very few, if any, reasons to MMAP a file on disk. It seems to simplify things in the common case, but in the end it adds a massive amount of unnecessary complexity.

I've been thinking for the past few years about how to get a scenario like 'git clone' of a large repo to go fast. One thought is to memory map the destination files being written by git and then copy/unzip the data there. You'd save a copy versus the staging buffer that you'd currently be passing to write(). However, the overhead of managing the tlb shootdowns would likely be fatal except for the largest output files.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#105

Post by ori_b »

dist1ll wrote: post #33
arter4 wrote: post #21

Earlier quoted context omitted.

Web servers doing kernel bypass for zero-copy networking? Do you have a specific example in mind? I'm curious.

The most common example is DPDK [1]. It's a framework for building bespoke networking stacks that are usable from userspace, without involving the kernel. You'll find DPDK mentioned a lot in the networking/HPC/data center literature. An example of a backend framework that uses DPDK is the seastar framework [2]. Also, I recently stumbled upon a paper for efficient RPC networks in data centers [3]. If you want to learn…

The downside, of course, is that each program owns one instance of the hardware. Applications don't share the network card. This isn't a general purpose solution.

That may be acceptable for your purposes, or it may not.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#106

Post by ori_b »

kwohlfahrt wrote: post #29

It sounds like a lot of the performance issues are TLB-related. Am I right in thinking huge-pages would help here? If so, it's a bit unfortunate they didn't test this in the paper. Edit: Hm, it might not be possible to mmap files with huge-pages. This LWN article[1] from 5 years ago talks about the work that would be required, but I haven't seen any follow-ups. [1]: https://lwn.net/Articles/718102/

No, huge pages wouldn't help. They would change when the TLB gets flushed, but the flushes would still be there.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#107

Post by LAC-Tech »

jandrewrogers wrote: post #88
LAC-Tech wrote: post #63

Earlier quoted context omitted.

This is a classic appeal to authority. Let's play the argument, not the man. (My understanding is that the GP wrote LMDB, works on openLDAP, and was a maintainer for BerkelyDB for a number of years. But even if he'd only written 'hello, world!' I'm much more interested in the specific arguments).

I think the real argument is more nuanced. Where you see mmap() fail badly on Linux, even for read-only workloads, is under a few specific conditions: very large storage volumes, highly concurrent access, non-trivial access patterns (e.g. high-dimensionality access methods). Most people do not operate data models under these conditions, but if you do then you can achieve large integer factor gains in throughput by no…

Can you explain "high-dimensionality access methods" to me? (Or if it's too big for an HN comment, maybe recommend a paper).

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#108

Post by ori_b »

hyc_symas wrote: post #43

This is a pretty old argument and IMO it's far out of date/obsolete. Taking full control of your I/O and buffer management is great if (a) your developers are all smart and experienced enough to be kernel programmers and (b) your DBMS is the only process running on a machine. In practice, (a) is never true, and (b) is no longer true because everyone is running apps inside containers inside shared VMs. In the modern a…

Do you have benchmarks of lmdb when the working set is much larger than memory? I couldn't find any.

In my experience -- and in line with the article -- mmap works fine with small working sets. It seems that most benchmarks of lmdb have relatively small data sets.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#109

Post by glandium »

jandrewrogers wrote: post #22

Another interesting limitation of mmap() is that real-world storage volumes can exceed the virtual address space a CPU can address. A 64-bit CPU may have 64-bit pointers but typically cannot address anywhere close to 64 bits of memory, virtually or physically. A normal buffer pool does not have this limitation. You can get EC2 instances on AWS with more direct-attached storage than addressable virtual address space o…

To put concrete numbers: x86-64 is limited to 48 bits for virtual addresses, which is "only" 256TiB (281TB).

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#110

Post by [deleted] »

ori_b wrote: post #105
dist1ll wrote: post #33

Earlier quoted context omitted.

The most common example is DPDK [1]. It's a framework for building bespoke networking stacks that are usable from userspace, without involving the kernel. You'll find DPDK mentioned a lot in the networking/HPC/data center literature. An example of a backend framework that uses DPDK is the seastar framework [2]. Also, I recently stumbled upon a paper for efficient RPC networks in data centers [3]. If you want to learn…

The downside, of course, is that each program owns one instance of the hardware. Applications don't share the network card. This isn't a general purpose solution. That may be acceptable for your purposes, or it may not.

[deleted]
Post reply on HN