Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

db.cs.cmu.edu

Post reply on HN
Print view View original

121–130 of 182 posts

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#121

Post by otterley »

danappelxx wrote: post #80
gcoakes wrote: post #76

Earlier quoted context omitted.

> - containers are each isolated in a VM (aka virtualized) Why are you assuming containers are virtualized? Is there some container runtime that does that as an added security measure? I thought they all use namespaces on Linux.

It’s becoming standard as a security measure. See: Kata containers, Firecracker VM

Not so; neither Kata containers nor Firecracker are in widespread public use today. (Source: I work for AWS and consult regularly with container services customers, who both use AWS and run on premise.)

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#122

Post by jeffffff »

pclmulqdq wrote: post #103
hyc_symas wrote: post #79

Earlier quoted context omitted.

1 - for reading any uncached data, the I/O stalls are unavoidable. Whatever client requested that data is going to have to wait regardless. 2 - complexity? this is simply false. LMDB's ACID txns using MVCC are much simpler than any "traditional" approach. 3 - contention is a red herring since this approach is already single-writer, as is common for most embedded k/v stores these days. You lose more perf by trying to…

It's kind of disingenuous to talk about how great your concurrency system is when you only allow a single writer. RCU (which I imagine your system is isomorphic to) is pretty simple compared to what many DB engines use to do ACID transactions that involve both reads and writes.

take a look at http://nms.csail.mit.edu/~stavros/pubs/OLTP_sigmod08.pdf - the overhead of coordinating multiple writers often makes multi-writer databases slower than single-writer databases. remember, everything has to be serialized when it goes to the write ahead log, so as long as you can do the database updates as fast as you can write to the log then concurrent writers are of no benefit.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#123

Post by otterley »

josephg wrote: post #20
formerly_proven wrote: post #14

Earlier quoted context omitted.

mmap can be handy but usually is not a good idea when you care about ACID properties. So it tends to be most useful outside databases.

Can you give some examples where mmap is useful?

glibc itself uses mmap under the covers when doing malloc in certain situations. Granted, it's anonymous and not file-backed, but it's still proven to be performant. See, e.g, mallopt(3).

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#124

Post by hyc_symas »

benlivengood wrote: post #59

For all of its usefulness in the good old days of rusty disks I wonder if virtual memory is worth having for dedicated databases, caches, and storage heads. Avoiding TLB flushes entirely sounds like a huge win for massively multithreaded software and memory management in a large shared flat address space doesn't sound impossibly hard.

The jump in address sizes starts to get too unwieldy. 32 bit addresses were ok, 64 bit addresses start to get clunky, 128 bit would be exorbitant for CPU real estate. There's a reason AMD64 still only supported 40 physical address bits when it was introduced, and later only expanded to 48 bits.

The reality is there will always be a hierarchy for storage, and paging will always be the best mechanism to deal with it. Because primary memory will always be most expensive, no matter what technology it's based on. There will always be something slower, cheaper, and denser that will be used for secondary storage. There will always be cheaper storage. And its capacity will exceed primary, and it will always be most efficient to reference secondary storage in chunks - pages - and not at individual byte addresses.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#125

Post by pclmulqdq »

hyc_symas wrote: post #115
pclmulqdq wrote: post #113

Earlier quoted context omitted.

That's probably because your OpenLDAP benchmarks used a tiny database. If you have multi-terabyte databases, you will start to see huge gains from a multi-writer setup because you will be regularly be loading pages from disk, rather than keeping almost all of your btree/LSM tree in RAM.

Yeah, no. Not with a DB 50x larger than RAM, anyway. http://www.lmdb.tech/bench/hyperdex/ RAM is relatively cheap too, there's no real reason to be running multi-TB databases at greater than a 50x ratio.

Sorry, but "50x larger than RAM" is a pretty small DB - that's an 800 GB database on a machine with 16 GB of RAM. I usually have seen machines with 500-1000x ratios of flash to RAM. "RAM is relatively cheap" is also false when you're storing truly huge amounts of data, which is how the systems you compare yourself to (LevelDB, etc) are usually deployed. Note that RAM is now the single greatest cost when buying servers.

> Now that the total database is 50 times larger than RAM, around half of the key lookups will require a disk I/O.

That is an insanely high cache hit rate, which should have probably set off your "unrepresentative benchmark" detector. I am also a little surprised at the lack of a random writes benchmark. I get that this is marketing material, though.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#126

Post by ilyt »

gavinray wrote: post #61
hyc_symas wrote: post #43

This is a pretty old argument and IMO it's far out of date/obsolete. Taking full control of your I/O and buffer management is great if (a) your developers are all smart and experienced enough to be kernel programmers and (b) your DBMS is the only process running on a machine. In practice, (a) is never true, and (b) is no longer true because everyone is running apps inside containers inside shared VMs. In the modern a…

Out of curiosity, how many databases have you written? This is co-authored by Pavlo, Viktor Leiss, with feedback from Neumann. I'm sorry, but if someone on the internet claims to know better than those 3, you're going to need some monumental evidence of your credibility. Additionally, what you link here: > ... (See the slide at 21:25 into the video). Modern DBMSs spend 96% of their time managing buffers and locks, an…

Out of curiosity, do you have anything actually useful to add or are just throwing appeals to authority because you don't ?

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#127

Post by ilyt »

jandrewrogers wrote: post #88
LAC-Tech wrote: post #63

Earlier quoted context omitted.

This is a classic appeal to authority. Let's play the argument, not the man. (My understanding is that the GP wrote LMDB, works on openLDAP, and was a maintainer for BerkelyDB for a number of years. But even if he'd only written 'hello, world!' I'm much more interested in the specific arguments).

I think the real argument is more nuanced. Where you see mmap() fail badly on Linux, even for read-only workloads, is under a few specific conditions: very large storage volumes, highly concurrent access, non-trivial access patterns (e.g. high-dimensionality access methods). Most people do not operate data models under these conditions, but if you do then you can achieve large integer factor gains in throughput by no…

I'd imagine same kind of worst case access would also be a problem doing IO the "classical" way

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#128

Post by hyc_symas »

Johnny555 wrote: post #116
hyc_symas wrote: post #70

Earlier quoted context omitted.

> There's nothing special about kernel programmers. Yes, that was a shorthand generalization for "people who've studied computer architecture" - which most application developers never have. > no DBA worth their salt would put database in the environment where it has to share resources with applications. Most applications today are running on smartphones/mobile devices. That means they're running with local embedded…

mmap doesn't allow their users to precisely control the persistence aspect It's been a while since I've dealt with mmap(), but isn't this what msync() does? You can synchronously or asynchronously force dirty pages to be flushed to disk without waiting until munmap().

msync lets you force a flush so you can control the latest possible moment for a writeout. But the OS can flush before that, and you have no way to detect or control that. So you can only control the late side of the timing, not the early side. And in databases, you usually need writes to be persisted in a specific order; early writes are just as harmful as late writes.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#129

Post by hyc_symas »

pclmulqdq wrote: post #125
hyc_symas wrote: post #115

Earlier quoted context omitted.

Yeah, no. Not with a DB 50x larger than RAM, anyway. http://www.lmdb.tech/bench/hyperdex/ RAM is relatively cheap too, there's no real reason to be running multi-TB databases at greater than a 50x ratio.

Sorry, but "50x larger than RAM" is a pretty small DB - that's an 800 GB database on a machine with 16 GB of RAM. I usually have seen machines with 500-1000x ratios of flash to RAM. "RAM is relatively cheap" is also false when you're storing truly huge amounts of data, which is how the systems you compare yourself to (LevelDB, etc) are usually deployed. Note that RAM is now the single greatest cost when buying server…

> I am also a little surprised at the lack of a random writes benchmark.

Eh? This was 20% random writes, 80% random reads. LMDB is for read-heavy workloads.

> That is an insanely high cache hit rate, which should have probably set off your "unrepresentative benchmark" detector.

No, that is normal for a B+tree; the root page and most of the branch pages will always be in cache. This is why you can get excellent efficiency and performance from a DB without tuning to a specific workload.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#130

Post by hyc_symas »

glandium wrote: post #109
jandrewrogers wrote: post #22

Another interesting limitation of mmap() is that real-world storage volumes can exceed the virtual address space a CPU can address. A 64-bit CPU may have 64-bit pointers but typically cannot address anywhere close to 64 bits of memory, virtually or physically. A normal buffer pool does not have this limitation. You can get EC2 instances on AWS with more direct-attached storage than addressable virtual address space o…

To put concrete numbers: x86-64 is limited to 48 bits for virtual addresses, which is "only" 256TiB (281TB).

All of that is true, but I don't think it's a realistic concern. You're going to be sharding your data across multiple nodes before it gets that large. Nobody wants to sit around backing up or restoring a monolithic 256 TiB database.
Post reply on HN