Live data from Hacker News

Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

db.cs.cmu.edu

131–140 of 182 posts

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#131

Earlier quoted context omitted.

It's kind of disingenuous to talk about how great your concurrency system is when you only allow a single writer. RCU (which I imagine your system is isomorphic to) is pretty simple compared to what many DB engines use to do ACID transactions that involve both reads and writes.

take a look at http://nms.csail.mit.edu/~stavros/pubs/OLTP_sigmod08.pdf - the overhead of coordinating multiple writers often makes multi-writer databases slower than single-writer databases. remember, everything has to be serialized when it goes to the write ahead log, so as long as you can do the database updates as fast as you can write to the log then concurrent writers are of no benefit.

This is another cool example of a toy database that is again very small:

> The database size for one warehouse is approximately 100 MB (we experiment with five warehouses for a total size of 500MB).

It is not surprising that when your database basically fits in RAM, serializing on one writer is worth doing, because it just plainly reduces contention. You basically gain nothing in a DB engine from multi-writer transactions when this is the case. A large part of a write (the vast majority of write latency) in many systems with a large database comes from reading the index up to the point where you plan to write. If that tree is in RAM, there is no work here, and you instead incur overhead on consistency of that tree by having multiple writers.

I'm not suggesting that these results are useless. They are useful for people whose databases are small because they are meaningfully better than RocksDB/LevelDB which implicitly assume that your database is a *lot* bigger than RAM.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#132

Earlier quoted context omitted.

Sorry, but "50x larger than RAM" is a pretty small DB - that's an 800 GB database on a machine with 16 GB of RAM. I usually have seen machines with 500-1000x ratios of flash to RAM. "RAM is relatively cheap" is also false when you're storing truly huge amounts of data, which is how the systems you compare yourself to (LevelDB, etc) are usually deployed. Note that RAM is now the single greatest cost when buying server…

> I am also a little surprised at the lack of a random writes benchmark. Eh? This was 20% random writes, 80% random reads. LMDB is for read-heavy workloads. > That is an insanely high cache hit rate, which should have probably set off your "unrepresentative benchmark" detector. No, that is normal for a B+tree; the root page and most of the branch pages will always be in cache. This is why you can get excellent effici…

> Eh? This was 20% random writes, 80% random reads. LMDB is for read-heavy workloads.

The page says "updates," not "writes." Updates are a constrained form of write where you are writing to an existing key. Updates, importantly, do not affect your index structure, while writes do.

> No, that is normal for a B+tree; the root page and most of the branch pages will always be in cache. This is why you can get excellent efficiency and performance from a DB without tuning to a specific workload.

It is normal for a small B+tree relative to the memory size available on the machine. The "small" was the unrepresentative part of the benchmark, not the "B+tree."

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#133

For all of its usefulness in the good old days of rusty disks I wonder if virtual memory is worth having for dedicated databases, caches, and storage heads. Avoiding TLB flushes entirely sounds like a huge win for massively multithreaded software and memory management in a large shared flat address space doesn't sound impossibly hard.

The jump in address sizes starts to get too unwieldy. 32 bit addresses were ok, 64 bit addresses start to get clunky, 128 bit would be exorbitant for CPU real estate. There's a reason AMD64 still only supported 40 physical address bits when it was introduced, and later only expanded to 48 bits. The reality is there will always be a hierarchy for storage, and paging will always be the best mechanism to deal with it. B…

I don't really see what those two things have to do with each other. When you don't use mmap, you manage the discram storage virtualisation yourself. Hardware paging, then, is pure overhead. The parent doesn't argue against layering of storage media, nor against chunking in general. Only against mmus as a mechanism for implementing it.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#134

Earlier quoted context omitted.

take a look at http://nms.csail.mit.edu/~stavros/pubs/OLTP_sigmod08.pdf - the overhead of coordinating multiple writers often makes multi-writer databases slower than single-writer databases. remember, everything has to be serialized when it goes to the write ahead log, so as long as you can do the database updates as fast as you can write to the log then concurrent writers are of no benefit.

This is another cool example of a toy database that is again very small: > The database size for one warehouse is approximately 100 MB (we experiment with five warehouses for a total size of 500MB). It is not surprising that when your database basically fits in RAM, serializing on one writer is worth doing, because it just plainly reduces contention. You basically gain nothing in a DB engine from multi-writer transac…

> RocksDB/LevelDB which implicitly assume that your database is a lot bigger than RAM.

Where are you getting that assumption from? LevelDB was built to be used in Google Chrome, not for multi-TB DBs. RocksDB was optimized specifically for in-memory workloads.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#135

Earlier quoted context omitted.

It’s becoming standard as a security measure. See: Kata containers, Firecracker VM

Not so; neither Kata containers nor Firecracker are in widespread public use today. (Source: I work for AWS and consult regularly with container services customers, who both use AWS and run on premise.)

Ah, good to know!

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#136

Earlier quoted context omitted.

This is another cool example of a toy database that is again very small: > The database size for one warehouse is approximately 100 MB (we experiment with five warehouses for a total size of 500MB). It is not surprising that when your database basically fits in RAM, serializing on one writer is worth doing, because it just plainly reduces contention. You basically gain nothing in a DB engine from multi-writer transac…

> RocksDB/LevelDB which implicitly assume that your database is a lot bigger than RAM. Where are you getting that assumption from? LevelDB was built to be used in Google Chrome, not for multi-TB DBs. RocksDB was optimized specifically for in-memory workloads.

I worked with the Bigtable folks at Google. LevelDB's design is ripped straight from BigTable, which was designed with that assumption in mind. I'm also pretty sure it was not designed specifically for Google Chrome's use case - it was written to be a general key-value storage engine based on BigTable, and Google Chrome was the first customer.

RocksDB is Facebook's offshoot of LevelDB, basically keeping the core architecture of the storage engine (but multithreading it), and is used internally at Facebook as the backing store for many of their database systems. I have never heard from anyone that RocksDB was optimized for in-memory workloads at all, and I think most benchmarks can conclusively say the opposite: both of those DB engines are pretty bad for workloads that fit in memory.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#137

Earlier quoted context omitted.

> I am also a little surprised at the lack of a random writes benchmark. Eh? This was 20% random writes, 80% random reads. LMDB is for read-heavy workloads. > That is an insanely high cache hit rate, which should have probably set off your "unrepresentative benchmark" detector. No, that is normal for a B+tree; the root page and most of the branch pages will always be in cache. This is why you can get excellent effici…

> Eh? This was 20% random writes, 80% random reads. LMDB is for read-heavy workloads. The page says "updates," not "writes." Updates are a constrained form of write where you are writing to an existing key. Updates, importantly, do not affect your index structure, while writes do. > No, that is normal for a B+tree; the root page and most of the branch pages will always be in cache. This is why you can get excellent e…

> The page says "updates," not "writes." Updates are a constrained form of write where you are writing to an existing key. Updates, importantly, do not affect your index structure, while writes do.

OK, I see your point. It would only have made things even worse for LevelDB here to do an Add/Delete workload because its garbage compaction passes would have had to do a lot more work.

> It is normal for a small B+tree relative to the memory size available on the machine. The "small" was the unrepresentative part of the benchmark, not the "B+tree."

This was 100 million records, and a 5-level deep tree. To get to 6 levels deep it would be about 10 billion records. Most of the branch pages would still fit in RAM; most queries would require at most 1 more I/O than the 5-level case. The cost is still better than any other approach.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#138

Earlier quoted context omitted.

> RocksDB/LevelDB which implicitly assume that your database is a lot bigger than RAM. Where are you getting that assumption from? LevelDB was built to be used in Google Chrome, not for multi-TB DBs. RocksDB was optimized specifically for in-memory workloads.

I worked with the Bigtable folks at Google. LevelDB's design is ripped straight from BigTable, which was designed with that assumption in mind. I'm also pretty sure it was not designed specifically for Google Chrome's use case - it was written to be a general key-value storage engine based on BigTable, and Google Chrome was the first customer. RocksDB is Facebook's offshoot of LevelDB, basically keeping the core arch…

I think we've gone off on a tangent. At any rate, both LevelDB and RocksDB are still single-writer so whatever point seems to have been lost along the way.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#139

This is a pretty old argument and IMO it's far out of date/obsolete. Taking full control of your I/O and buffer management is great if (a) your developers are all smart and experienced enough to be kernel programmers and (b) your DBMS is the only process running on a machine. In practice, (a) is never true, and (b) is no longer true because everyone is running apps inside containers inside shared VMs. In the modern a…

Who is deploying databases in containers?

My group and a bunch of my peer groups.

And we are running them at the scale that most people can’t even imagine.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#140

Earlier quoted context omitted.

mmap doesn't allow their users to precisely control the persistence aspect It's been a while since I've dealt with mmap(), but isn't this what msync() does? You can synchronously or asynchronously force dirty pages to be flushed to disk without waiting until munmap().

msync lets you force a flush so you can control the latest possible moment for a writeout. But the OS can flush before that, and you have no way to detect or control that. So you can only control the late side of the timing, not the early side. And in databases, you usually need writes to be persisted in a specific order; early writes are just as harmful as late writes.

I'd even take a memory ordering guarantee, something like, within each page, data is read out sequentially as atomic aligned 64-bit reads with acquire ordering. (Though this probably is what you get on AMD64.) As-is, there's not even a guarantee against an atomic aligned write being torn when written out.
Post reply on HN