Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

db.cs.cmu.edu

Post reply on HN
Print view View original

111–120 of 182 posts

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#111

Post by Svetlitski »

glandium wrote: post #109
jandrewrogers wrote: post #22

Another interesting limitation of mmap() is that real-world storage volumes can exceed the virtual address space a CPU can address. A 64-bit CPU may have 64-bit pointers but typically cannot address anywhere close to 64 bits of memory, virtually or physically. A normal buffer pool does not have this limitation. You can get EC2 instances on AWS with more direct-attached storage than addressable virtual address space o…

To put concrete numbers: x86-64 is limited to 48 bits for virtual addresses, which is "only" 256TiB (281TB).

Starting with Ice Lake there’s support for 5-level paging, which increases this to 128 PiB. Can’t say that I’ve ever seen this used in the wild though.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#112

Post by hyc_symas »

pclmulqdq wrote: post #103
hyc_symas wrote: post #79

Earlier quoted context omitted.

1 - for reading any uncached data, the I/O stalls are unavoidable. Whatever client requested that data is going to have to wait regardless. 2 - complexity? this is simply false. LMDB's ACID txns using MVCC are much simpler than any "traditional" approach. 3 - contention is a red herring since this approach is already single-writer, as is common for most embedded k/v stores these days. You lose more perf by trying to…

It's kind of disingenuous to talk about how great your concurrency system is when you only allow a single writer. RCU (which I imagine your system is isomorphic to) is pretty simple compared to what many DB engines use to do ACID transactions that involve both reads and writes.

You don't need more than single-writer concurrency if your write txns are fast enough.

Our experience with OpenLDAP was that multi-writer concurrency cost too much overhead. Even though you may be writing primary records to independent regions of the DB, if you're indexing any of that data (which all real DBs do, for query perf) you wind up getting a lot of contention in the indices. That leads to row locking conflicts, txn rollbacks, and retries. With a single writer txn model, you never get conflicts, never need rollbacks.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#113

Post by pclmulqdq »

hyc_symas wrote: post #112
pclmulqdq wrote: post #103

Earlier quoted context omitted.

It's kind of disingenuous to talk about how great your concurrency system is when you only allow a single writer. RCU (which I imagine your system is isomorphic to) is pretty simple compared to what many DB engines use to do ACID transactions that involve both reads and writes.

You don't need more than single-writer concurrency if your write txns are fast enough. Our experience with OpenLDAP was that multi-writer concurrency cost too much overhead. Even though you may be writing primary records to independent regions of the DB, if you're indexing any of that data (which all real DBs do, for query perf) you wind up getting a lot of contention in the indices. That leads to row locking conflic…

That's probably because your OpenLDAP benchmarks used a tiny database. If you have multi-terabyte databases, you will start to see huge gains from a multi-writer setup because you will be regularly be loading pages from disk, rather than keeping almost all of your btree/LSM tree in RAM.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#114

Post by hyc_symas »

ori_b wrote: post #108
hyc_symas wrote: post #43

This is a pretty old argument and IMO it's far out of date/obsolete. Taking full control of your I/O and buffer management is great if (a) your developers are all smart and experienced enough to be kernel programmers and (b) your DBMS is the only process running on a machine. In practice, (a) is never true, and (b) is no longer true because everyone is running apps inside containers inside shared VMs. In the modern a…

Do you have benchmarks of lmdb when the working set is much larger than memory? I couldn't find any. In my experience -- and in line with the article -- mmap works fine with small working sets. It seems that most benchmarks of lmdb have relatively small data sets.

> Do you have benchmarks of lmdb when the working set is much larger than memory? I couldn't find any.

Where did you look? This is a sample using DB 5x and 50x larger than RAM http://www.lmdb.tech/bench/hyperdex/

There are plenty of other larger-than-RAM benchmarks there.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#115

Post by hyc_symas »

pclmulqdq wrote: post #113
hyc_symas wrote: post #112

Earlier quoted context omitted.

You don't need more than single-writer concurrency if your write txns are fast enough. Our experience with OpenLDAP was that multi-writer concurrency cost too much overhead. Even though you may be writing primary records to independent regions of the DB, if you're indexing any of that data (which all real DBs do, for query perf) you wind up getting a lot of contention in the indices. That leads to row locking conflic…

That's probably because your OpenLDAP benchmarks used a tiny database. If you have multi-terabyte databases, you will start to see huge gains from a multi-writer setup because you will be regularly be loading pages from disk, rather than keeping almost all of your btree/LSM tree in RAM.

Yeah, no. Not with a DB 50x larger than RAM, anyway.

http://www.lmdb.tech/bench/hyperdex/

RAM is relatively cheap too, there's no real reason to be running multi-TB databases at greater than a 50x ratio.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#116

Post by Johnny555 »

hyc_symas wrote: post #70
crabbone wrote: post #65

Earlier quoted context omitted.

> your DBMS is the only process running on a machine. In practice, (a) is never true, and (b) is no longer true because everyone is running apps inside containers inside shared VMs. There's nothing special about kernel programmers. In fact, if I had to compare, I'd go with storage people being the more experienced / knowledgeable ones. They have a highly competitive environment, which requires a lot more understandin…

> There's nothing special about kernel programmers. Yes, that was a shorthand generalization for "people who've studied computer architecture" - which most application developers never have. > no DBA worth their salt would put database in the environment where it has to share resources with applications. Most applications today are running on smartphones/mobile devices. That means they're running with local embedded…

mmap doesn't allow their users to precisely control the persistence aspect

It's been a while since I've dealt with mmap(), but isn't this what msync() does? You can synchronously or asynchronously force dirty pages to be flushed to disk without waiting until munmap().

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#117

Post by hyc_symas »

pclmulqdq wrote: post #101
jandrewrogers wrote: post #88

Earlier quoted context omitted.

I think the real argument is more nuanced. Where you see mmap() fail badly on Linux, even for read-only workloads, is under a few specific conditions: very large storage volumes, highly concurrent access, non-trivial access patterns (e.g. high-dimensionality access methods). Most people do not operate data models under these conditions, but if you do then you can achieve large integer factor gains in throughput by no…

GP wrote a key-value store called LMDB that is constrained to a single writer, and often used for small databases that fit entirely in memory but need to persist to disk. There's a whole different world for more scalable databases.

"fit entirely in memory" is not a requirement. LMDB is not a main-memory database, it is an on-disk database that uses memory mapping.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#118

Post by otterley »

FridgeSeal wrote: post #87
crabbone wrote: post #69

Earlier quoted context omitted.

The problem is kubelet likes to spike in memory / CPU / network usage. It's not a well-behaved program to put alongside a database. It's not written with an eye for resource utilization. Also, it brings nothing of value to the table, but requires a lot of dance around it to keep it going. I.e. if you are a decent DBA, you don't have a problem setting up a node to run your database of choice, you would be probably opp…

I run DB’s on K8s, not because I don’t know what I’m doing, but because most of the trade offs are worth it. If I run a db workload in K8s, it’s a tiny fraction of the operational overhead, and not a massively noticeable performance loss. I would absolutely love a way to deploy and manage db’s as easily as K8s with fewer of the quite significant issues that have mentioned, so if you know of something that is better b…

If you run in the cloud, any of the major cloud providers can take that undifferentiated heavy lifting off your hands (Amazon RDS etc.).

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#119

Post by charcircuit »

danappelxx wrote: post #60
charcircuit wrote: post #57

Earlier quoted context omitted.

How would containers even hurt performance? How does the database no longer having the ability to see other processes on the machine somehow make it slower?

I’ll assume the worst case: - lots of containers running on a single host - containers are each isolated in a VM (aka virtualized) - workloads are not homogenous and change often (your neighbor today may not be your neighbor tomorrow) I believe these are fair assumptions if you’re running on generic infrastructure with kubernetes. In this setup, my concerns are pretty much noisy neighbors + throttling. You may get la…

None of those are the fault of containers. You can do all of what you said without containers.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#120

Post by hyc_symas »

kwohlfahrt wrote: post #29

It sounds like a lot of the performance issues are TLB-related. Am I right in thinking huge-pages would help here? If so, it's a bit unfortunate they didn't test this in the paper. Edit: Hm, it might not be possible to mmap files with huge-pages. This LWN article[1] from 5 years ago talks about the work that would be required, but I haven't seen any follow-ups. [1]: https://lwn.net/Articles/718102/

Huge pages aren't pageable, so they wouldn't be particularly advantageous for a mmap DB anyway, you'd have to do traditional I/O & buffer management for everything.
Post reply on HN