Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

db.cs.cmu.edu

Post reply on HN
Print view View original

171–180 of 182 posts

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#171

Post by crabbone »

kentonv wrote: post #166
pclmulqdq wrote: post #147

Earlier quoted context omitted.

100% of people writing a database know about filesystem options like DIRECT and SYNC, and that is the subject of this paper. Also, most of the network-attached storage we people use is in the form of things like EBS, which is very careful to imitate the behavior of a real disk, but with different performance and some different (albeit very rare ) failure modes.

100% of people writing databases also know how fsync() and msync() work. I interpreted this thread as being targeted at a wider audience.

It's literally for people writing their own database. Why would you interpret it differently?

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#172

Post by crabbone »

xyzzy_plugh wrote: post #143
crabbone wrote: post #77

Earlier quoted context omitted.

There are many "holes" in these containers. 1. fsync. You cannot "divide" it between containers. Whoever does it, stalls I/O for everyone else. 2. Context switches. Unless you do a lot of configurations outside of container runtime, you cannot ensure exclusive access to the number of CPU cores you need. 3. Networking has the same problem. You would either have to dedicate a whole NIC or SRI-OV-style virtual NIC to yo…

This is extremely misinformed. No matter how you choose to manage workloads, ultimately you are responsible for tuning and optimization. If you're not in control of the system, and thus kubelet, obviously your hands are tied. I'm not sure anyone is suggesting that for a serious workload. Now to dispell your myths: 1. You can assign dedicated storage devices to your database. Outside of mount operations you're not goi…

> You can assign dedicated storage devices to your database. Outside of mount operations you're not going to see much alien fsync activity. This is paranoid.

This is word salad. Do you even know what fsync is for? I'm not even asking if you know how it works... What is "alien" fsync activity? Mount is perhaps the one system call that has nothing to do with fsync... so, I wouldn't expect any fsync activity when calling mount...

Finally, I didn't say that you cannot allocate a dedicated storage device -- what I said is that Kubernetes or Docker or Singularity or containerd or... well, none of container (management) runtimes that I've ever used know how to do it. You need external tools to do it. The point isn't that you cannot, the point is that a container runtime will only stand in your way when you try to do it.

> You can pin kubelet CPU cores. You can ensure exclusive access to the remaining ones.

No you cannot. Not through Kubernetes. You need to do this on the node that hosts kubelet.

And... I don't have the time or the patience necessary to answer to the rest of the nonsense. Bottom line: you don't understand what you are replying to, and arguing with something I either didn't say, or just stringing meaningless words together.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#173

Post by crabbone »

lokar wrote: post #94
crabbone wrote: post #69

Earlier quoted context omitted.

The problem is kubelet likes to spike in memory / CPU / network usage. It's not a well-behaved program to put alongside a database. It's not written with an eye for resource utilization. Also, it brings nothing of value to the table, but requires a lot of dance around it to keep it going. I.e. if you are a decent DBA, you don't have a problem setting up a node to run your database of choice, you would be probably opp…

If you care about perf you would pin the kubelet and all other overhead workload to one core, and mask that off for your workload.

> If you care about perf you would pin the kubelet

Wrong. I wouldn't use kubelet at all. Kubernetes and good performance are not compatible. The goal of Kubernetes is to make it easier to deploy Web sites. Web is a very popular technology, so Kubernetes was adopted in many places where it's irrelevant / harmful because Web developers are plentiful and will help to power through the nonsense of this program. It's there because it makes trivial things even easier for less qualified personnel. It's not meant as a way to make things go faster, or to use less memory, or to use less persistent storage, or less network etc... it's the wheelchair of ops, not a highly-optimized professional-grade equipment.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#174

Post by xyzzy_plugh »

crabbone wrote: post #172
xyzzy_plugh wrote: post #143

Earlier quoted context omitted.

This is extremely misinformed. No matter how you choose to manage workloads, ultimately you are responsible for tuning and optimization. If you're not in control of the system, and thus kubelet, obviously your hands are tied. I'm not sure anyone is suggesting that for a serious workload. Now to dispell your myths: 1. You can assign dedicated storage devices to your database. Outside of mount operations you're not goi…

> You can assign dedicated storage devices to your database. Outside of mount operations you're not going to see much alien fsync activity. This is paranoid. This is word salad. Do you even know what fsync is for? I'm not even asking if you know how it works... What is "alien" fsync activity? Mount is perhaps the one system call that has nothing to do with fsync... so, I wouldn't expect any fsync activity when callin…

> Do you even know what fsync is for?

I do, though perhaps an ignorant life would be simpler. "Alien" is a word with a definition. Perhaps "foreign" is a better word. Forgive me for attempting to wield the English language.

No one well will use your fucking disk if you mount it exclusively in a pod. Does that make sense? You must be a joy to work with.

> The point isn't that you cannot, the point is that a container runtime will only stand in your way when you try to do it.

I have no idea what this means. How does kubernetes stand in your way?

> No you cannot. Not through Kubernetes. You need to do this on the node that hosts kubelet.

This is incorrect. You can absolutely configure the kubelet to reserve cores and offer exclusive cores to pods by setting a CPU management policy. I know because I was waiting for this for a very long time for all of the reason in the discussion here. It works fine.

You clearly have an axe to grind and it seems pretty obvious you're not willing to do the work to understand what you're complaining about. It might help to start by googling what a container runtime even is, but I'm not optimistic.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#175

Post by benlivengood »

justin_ wrote: post #152
benlivengood wrote: post #59

For all of its usefulness in the good old days of rusty disks I wonder if virtual memory is worth having for dedicated databases, caches, and storage heads. Avoiding TLB flushes entirely sounds like a huge win for massively multithreaded software and memory management in a large shared flat address space doesn't sound impossibly hard.

This is the kind of debate that has been going on surrounding virtual memory forever[0][1]. If you can keep everything in memory, then you're golden. But eventually you won't, and you'll need to rely on secondary storage. Is there a performance benefit to be had by managing the memory and paging yourself? Yes. But eventually you will also consider running processes next to your database, for logging, auditing, ingest…

I guess things like mshare could be extended to the entire process address spaces and the kernel could avoid TLB invalidation on context switches between them. Core affinity could be used to keep other programs from scheduling on the cores intended for processes sharing the whole address space.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#176

Post by heisjustsosmart »

LAC-Tech wrote: post #107
jandrewrogers wrote: post #88

Earlier quoted context omitted.

I think the real argument is more nuanced. Where you see mmap() fail badly on Linux, even for read-only workloads, is under a few specific conditions: very large storage volumes, highly concurrent access, non-trivial access patterns (e.g. high-dimensionality access methods). Most people do not operate data models under these conditions, but if you do then you can achieve large integer factor gains in throughput by no…

Can you explain "high-dimensionality access methods" to me? (Or if it's too big for an HN comment, maybe recommend a paper).

This guy talks a lot of crap. See his website for examples, and don't waste your time with him

>>

http://www.jandrewrogers.com/2015/10/08/spacecurve/

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#177

Post by heisjustsosmart »

hyc_symas wrote: post #162
jandrewrogers wrote: post #144

Earlier quoted context omitted.

> You don't need more than single-writer concurrency if your write txns are fast enough. This only works on systems with sufficiently slow storage. If your server has a bunch of NVMe, which is a pretty normal database config these days, you will be hard-pressed to get anywhere close to the theoretical throughput of the storage with a single writer. That requires 10+ GB/s sustained. It is a piece of cake with multiple…

That's all just false. Just because you're single-writer at the application level doesn't mean the OS isn't queueing enough writes to saturate storage at the device level. We've benchmarked plenty of high speed NVMe devices, like Intel Optane SSDs, etc. showing this. http://www.lmdb.tech/bench/optanessd/

this guy is a fool. Ignore him. Or see his website.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#178

Post by ayende »

hyc_symas wrote: post #161
ayende wrote: post #158

Earlier quoted context omitted.

That is absolutely not what you actually get from the hardware. For fun, there is no guarantee in terms of writing a page in what order it is written. SQLite documents that they assume (but cannot verify) that _sector_ writes are linear, but not atomic. https://www.sqlite.org/atomiccommit.html > If a power failure occurs in the middle of a sector write it might be that part of the sector was modified and another part…

Every HDD since the 1980s has guaranteed atomic sector writes: > Currently all hard drive/SSD manufacturers guarantee that 512 byte sector writes are atomic. As such, failure to write the 106 byte header is not something we account for in current LMDB releases. Also, failures of this type should result in ECC errors in the disk sector - it should be impossible to successfully read a sector that was written incorrectl…

Doesn't help when you work with pages :-)

Assume 512 sectors ( I know those are rare ), but I don't think that there is any guarantees that 4KB page would be:

* Written atomically * Written in a particular order

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#179

Post by ayende »

hyc_symas wrote: post #161
ayende wrote: post #158

Earlier quoted context omitted.

That is absolutely not what you actually get from the hardware. For fun, there is no guarantee in terms of writing a page in what order it is written. SQLite documents that they assume (but cannot verify) that _sector_ writes are linear, but not atomic. https://www.sqlite.org/atomiccommit.html > If a power failure occurs in the middle of a sector write it might be that part of the sector was modified and another part…

Every HDD since the 1980s has guaranteed atomic sector writes: > Currently all hard drive/SSD manufacturers guarantee that 512 byte sector writes are atomic. As such, failure to write the 106 byte header is not something we account for in current LMDB releases. Also, failures of this type should result in ECC errors in the disk sector - it should be impossible to successfully read a sector that was written incorrectl…

Also doesn't help when you are running on virtual / networked hardware. Nothing ensure that what you think is a sector write would actually align properly with the hardware.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#180

Post by colanderman »

ayende wrote: post #179
hyc_symas wrote: post #161

Earlier quoted context omitted.

Every HDD since the 1980s has guaranteed atomic sector writes: > Currently all hard drive/SSD manufacturers guarantee that 512 byte sector writes are atomic. As such, failure to write the 106 byte header is not something we account for in current LMDB releases. Also, failures of this type should result in ECC errors in the disk sector - it should be impossible to successfully read a sector that was written incorrectl…

Also doesn't help when you are running on virtual / networked hardware. Nothing ensure that what you think is a sector write would actually align properly with the hardware.

The design and guarantees of the virtualized hardware provide that guarantee. I've worked on several such products. They all guarantee atomic sector writes (typically via copy-on-write).
Post reply on HN