Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

db.cs.cmu.edu

Post reply on HN
Print view View original

141–150 of 182 posts

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#141

Post by ori_b »

hyc_symas wrote: post #114
ori_b wrote: post #108

Earlier quoted context omitted.

Do you have benchmarks of lmdb when the working set is much larger than memory? I couldn't find any. In my experience -- and in line with the article -- mmap works fine with small working sets. It seems that most benchmarks of lmdb have relatively small data sets.

> Do you have benchmarks of lmdb when the working set is much larger than memory? I couldn't find any. Where did you look? This is a sample using DB 5x and 50x larger than RAM http://www.lmdb.tech/bench/hyperdex/ There are plenty of other larger-than-RAM benchmarks there.

Hm. That seems to be comparing against a 2013 era leveldb, which at the time also used mmap. (It's since switched the default for performance reasons)

It's also strange to me that there's no transition in performance when the data set size grows beyond cache.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#142

Post by hyc_symas »

moonchild wrote: post #133
hyc_symas wrote: post #124

Earlier quoted context omitted.

The jump in address sizes starts to get too unwieldy. 32 bit addresses were ok, 64 bit addresses start to get clunky, 128 bit would be exorbitant for CPU real estate. There's a reason AMD64 still only supported 40 physical address bits when it was introduced, and later only expanded to 48 bits. The reality is there will always be a hierarchy for storage, and paging will always be the best mechanism to deal with it. B…

I don't really see what those two things have to do with each other. When you don't use mmap, you manage the disc ram storage virtualisation yourself. Hardware paging, then, is pure overhead. The parent doesn't argue against layering of storage media, nor against chunking in general. Only against mmus as a mechanism for implementing it.

The mention of a large shared flat address space implied no paging, to me. Maybe I just read something into it that wasn't there.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#143

Post by xyzzy_plugh »

crabbone wrote: post #77
charcircuit wrote: post #57

Earlier quoted context omitted.

How would containers even hurt performance? How does the database no longer having the ability to see other processes on the machine somehow make it slower?

There are many "holes" in these containers. 1. fsync. You cannot "divide" it between containers. Whoever does it, stalls I/O for everyone else. 2. Context switches. Unless you do a lot of configurations outside of container runtime, you cannot ensure exclusive access to the number of CPU cores you need. 3. Networking has the same problem. You would either have to dedicate a whole NIC or SRI-OV-style virtual NIC to yo…

This is extremely misinformed. No matter how you choose to manage workloads, ultimately you are responsible for tuning and optimization.

If you're not in control of the system, and thus kubelet, obviously your hands are tied. I'm not sure anyone is suggesting that for a serious workload.

Now to dispell your myths:

1. You can assign dedicated storage devices to your database. Outside of mount operations you're not going to see much alien fsync activity. This is paranoid.

2. You can pin kubelet CPU cores. You can ensure exclusive access to the remaining ones. There are a number of advanced techniques that are not at all necessary if you want to be a control freak, such as creating your own cgroups. This isn't "outside" of the runtime. Kubernetes is designed to conform to your managed cgroups. That's the whole point. RTFM.

3. The general theme of your complaint has nothing to do with kubernetes. There's no beating a dedicated NIC and even network fabric. Some cloud providers even allow you to multi-NIC out of the box so this is pretty solvable. Also, like, the dumbest QoS rules can drastically minimize this problem generally. Who cares.

4. Nah. RTFM. This is total FUD.

5.a. I don't understand. Are you sharing resources on the node or not? If you're not, then swap works fine. If you are, then this smells like cognitive dissonance and maybe listen to your own advice, but also swap is still very doable. It's just disk. swapon to your heart's content. But also swap is almost entirely dumb these days. Are you suggesting swapping to your primary IO device? Come on. More FUD.

5.b. OOM killer does what it wants. What's a better alternative that integrates "well" with the OOM killer? Do you even understand how resource limits work? The OOM killer is only ever a problem if you either do not configure your workload properly (true regardless of execution environment) or you run out of actual memory.

Bottom line: come down off your high horse and acknowledge that dedicated resources and kernel tuning is the secret to extreme high performance. I don't care how you're orchestrating your workloads, the best practices are essentially universal.

And to be clear, I'm not recommending using Kubernetes to run a high performance database but it's not really any worse (today) than alternatives.

> It's written for Web developers. To make things appear simpler for them, while sacrificing a lot of resources and hiding a lot of actual complexity... which is impossible to hide, and which, in an even of failure will come to bite you.

What planet are you currently on? This makes no sense. It's a set of abstractions and patterns, the intent isn't to hide the complexity but to make it manageable at scale. I'd argue it succeeds at that.

Seriously, what is the alternative runtime you'd prefer here? systemd? hand rolled bash scripts? puppet and ansible? All of the above??

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#144

Post by jandrewrogers »

hyc_symas wrote: post #112
pclmulqdq wrote: post #103

Earlier quoted context omitted.

It's kind of disingenuous to talk about how great your concurrency system is when you only allow a single writer. RCU (which I imagine your system is isomorphic to) is pretty simple compared to what many DB engines use to do ACID transactions that involve both reads and writes.

You don't need more than single-writer concurrency if your write txns are fast enough. Our experience with OpenLDAP was that multi-writer concurrency cost too much overhead. Even though you may be writing primary records to independent regions of the DB, if you're indexing any of that data (which all real DBs do, for query perf) you wind up getting a lot of contention in the indices. That leads to row locking conflic…

> You don't need more than single-writer concurrency if your write txns are fast enough.

This only works on systems with sufficiently slow storage. If your server has a bunch of NVMe, which is a pretty normal database config these days, you will be hard-pressed to get anywhere close to the theoretical throughput of the storage with a single writer. That requires 10+ GB/s sustained. It is a piece of cake with multiple writers and a good architecture.

Writes through indexing can be sustained at this rate (assuming appropriate data structures), most of the technical challenge is driving the network at the necessary rate in my experience.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#145

Post by xyzzy_plugh »

lokar wrote: post #97
crabbone wrote: post #77

Earlier quoted context omitted.

There are many "holes" in these containers. 1. fsync. You cannot "divide" it between containers. Whoever does it, stalls I/O for everyone else. 2. Context switches. Unless you do a lot of configurations outside of container runtime, you cannot ensure exclusive access to the number of CPU cores you need. 3. Networking has the same problem. You would either have to dedicate a whole NIC or SRI-OV-style virtual NIC to yo…

My background is more borg then k8s, but… Alway allocate whole cores, just mask them off Dedicate physical IO devices for sensitive workloads You can have per cgroup swap if you want, but imo swap is not useful I think all of this is possible in k8s

Whole core masking is not quite as easy as it should be, predominantly because the API is designed to hand wave away actual cores. The way you typically solve this is to go the other way and claim exclusive cores for the orchestrator and other overhead.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#146

Post by moonchild »

hyc_symas wrote: post #142
moonchild wrote: post #133

Earlier quoted context omitted.

I don't really see what those two things have to do with each other. When you don't use mmap, you manage the disc ram storage virtualisation yourself. Hardware paging, then, is pure overhead. The parent doesn't argue against layering of storage media, nor against chunking in general. Only against mmus as a mechanism for implementing it.

The mention of a large shared flat address space implied no paging, to me. Maybe I just read something into it that wasn't there.

The 'paging' is implemented in software, not in hardware. This is how databases not using mmap already work, so mmus are already pure overhead for them.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#147

Post by pclmulqdq »

kentonv wrote: post #96
crabbone wrote: post #82

Earlier quoted context omitted.

> This is not specific to mmap -- regular old write() calls have the same behavior. This is not true. This depends on how the file was opened. You may request DIRECT | SYNC when opening and the writes are acknowledged when they are actually written. This is obviously a lot slower than writing to cache, but this is the way for "simple" user-space applications to implement their own cache. In the world of today, you ar…

> This is not true. This depends on how the file was opened. You may request DIRECT | SYNC Well sure, but 99.9% of people don't do that (and shouldn't, unless they really know what they are doing). > In the world of today, you are very rarely writing to something that's not network attached, and depending on your appliance, the meaning of acknowledgement from write() differs. What network-attached storage actually us…

100% of people writing a database know about filesystem options like DIRECT and SYNC, and that is the subject of this paper.

Also, most of the network-attached storage we people use is in the form of things like EBS, which is very careful to imitate the behavior of a real disk, but with different performance and some different (albeit very rare) failure modes.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#148

Post by jandrewrogers »

hyc_symas wrote: post #130
glandium wrote: post #109

Earlier quoted context omitted.

To put concrete numbers: x86-64 is limited to 48 bits for virtual addresses, which is "only" 256TiB (281TB).

All of that is true, but I don't think it's a realistic concern. You're going to be sharding your data across multiple nodes before it gets that large. Nobody wants to sit around backing up or restoring a monolithic 256 TiB database.

Technically you get quite a bit less than the 256 TB theoretical in practice.

It is a realistic concern, I’ve lived it for more than a decade across many orgs, though I shared your opinion at one point. Storage density is massively important for both workload scalability and economic efficiency. Low storage density means buying a ton of server hardware that sits idle under max load and vastly larger clusters than would otherwise be necessary, which have their own costs.

When your database is sufficiently large, backup and restore often isn’t even a technical possibility so that requirement is a red herring. The kinds of workloads that can be recovered from backup at that scale on a single server, and some can, benefit massively from the economics of running it on a single server. A solution that has 10x the AWS bill for the same workload performance doesn’t get chosen.

At scale, hardware footprint economics is one of the central business decision drivers. Data isn’t getting smaller. It is increasingly ordinary for innocuous organizations to have a single table with a trillion records in it.

For better or worse, the market increasingly drives my technical design decisions to optimize for hardware/cloud costs above all else, and dense storage is a huge win for that.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#149

Post by jandrewrogers »

Svetlitski wrote: post #111
glandium wrote: post #109

Earlier quoted context omitted.

To put concrete numbers: x86-64 is limited to 48 bits for virtual addresses, which is "only" 256TiB (281TB).

Starting with Ice Lake there’s support for 5-level paging, which increases this to 128 PiB. Can’t say that I’ve ever seen this used in the wild though.

Yeah, there mostly isn’t a use case for it in databases. If you have that much storage you’ll need to bypass the kernel cache and scheduler anyway for other reasons. That was true even at the 48-bit limit.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#150

Post by sakras »

hyc_symas wrote: post #43

This is a pretty old argument and IMO it's far out of date/obsolete. Taking full control of your I/O and buffer management is great if (a) your developers are all smart and experienced enough to be kernel programmers and (b) your DBMS is the only process running on a machine. In practice, (a) is never true, and (b) is no longer true because everyone is running apps inside containers inside shared VMs. In the modern a…

Can you comment on what the paper gets wrong? It says that scalability with mmap is poor due to page table contention and others. How does LMDB manage to scale well with mmap? Is page table contention just not an issue in practice?
Post reply on HN