Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

db.cs.cmu.edu

Post reply on HN
Print view View original

151–160 of 182 posts

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#151

Post by AdamProut »

pclmulqdq wrote: post #103
hyc_symas wrote: post #79

Earlier quoted context omitted.

1 - for reading any uncached data, the I/O stalls are unavoidable. Whatever client requested that data is going to have to wait regardless. 2 - complexity? this is simply false. LMDB's ACID txns using MVCC are much simpler than any "traditional" approach. 3 - contention is a red herring since this approach is already single-writer, as is common for most embedded k/v stores these days. You lose more perf by trying to…

It's kind of disingenuous to talk about how great your concurrency system is when you only allow a single writer. RCU (which I imagine your system is isomorphic to) is pretty simple compared to what many DB engines use to do ACID transactions that involve both reads and writes.

Yeah for workloads with any long running write transactions a single writer design is a pretty big limitation. Say some long running data load (or a big bulk deletion) running along with some faster high throughput key value writes - the big data load would block all the faster key-value writes when it runs.

No "mainstream" database I'm aware of has a global single writer design.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#152

Post by justin_ »

benlivengood wrote: post #59

For all of its usefulness in the good old days of rusty disks I wonder if virtual memory is worth having for dedicated databases, caches, and storage heads. Avoiding TLB flushes entirely sounds like a huge win for massively multithreaded software and memory management in a large shared flat address space doesn't sound impossibly hard.

This is the kind of debate that has been going on surrounding virtual memory forever[0][1]. If you can keep everything in memory, then you're golden. But eventually you won't, and you'll need to rely on secondary storage.

Is there a performance benefit to be had by managing the memory and paging yourself? Yes. But eventually you will also consider running processes next to your database, for logging, auditing, ingesting data, running backups, etc. Virtual memory across the whole system helps with that, especially if other people will be using your database in ways you can't predict. As for the efficiency of MMUs and the OS, seems like for almost all cases it's "satisfactory" enough[1].

[0] http://denninginstitute.com/pjd/PUBS/bvm.pdf

[1] From 1969! https://dl.acm.org/doi/pdf/10.1145/363626.363629

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#153

Post by jrockway »

crabbone wrote: post #90
FridgeSeal wrote: post #87

Earlier quoted context omitted.

I run DB’s on K8s, not because I don’t know what I’m doing, but because most of the trade offs are worth it. If I run a db workload in K8s, it’s a tiny fraction of the operational overhead, and not a massively noticeable performance loss. I would absolutely love a way to deploy and manage db’s as easily as K8s with fewer of the quite significant issues that have mentioned, so if you know of something that is better b…

If you think that deploying anything with Kubernetes is simple... well, I have bad news for you. It's simple, until you hit a problem. And then it becomes a lot worse than if you had never touched it. You are now in the stage of a person who'd never made backups and never had a failure that required them to restore from backups, and you are wondering why would anyone do it. Adverse events are rare, and you may go lik…

Everyone running databases in production knows how to take backups and restore from them. K8s or not, even using your cloud provider's database's built-in backups is hardly safe. One click of the "delete instance" button (or nowadays, an exciting fuck up in IaC code), and your backups are gone! Not to mention the usual cloud provider problems of "oops your credit card bounced" or "the algorithm decided we don't like your line of business". You have to have backups, they have to be "off site", and you have to try restoring them every few months. There is pretty much no platform that gives you that for free.

I am not sure what complexity Kubernetes adds in this situation. Anything Kubernetes can do to you, your cloud provider (or a poorly aimed fire extinguisher) can do to you. You have to be ready for a disaster no matter the platform.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#154

Post by ricardo81 »

crabbone wrote: post #91
hyc_symas wrote: post #70

Earlier quoted context omitted.

> There's nothing special about kernel programmers. Yes, that was a shorthand generalization for "people who've studied computer architecture" - which most application developers never have. > no DBA worth their salt would put database in the environment where it has to share resources with applications. Most applications today are running on smartphones/mobile devices. That means they're running with local embedded…

> Most applications today are running on smartphones/mobile devices. That's patently false. There are about 8 bn. people. Even if everyone has a smartphone or two, it's nothing compared to the total of all devices that can be called "computer". I think that "smart TV" alone will beat the number of smartphones. But even that is a drop in a bucket when it comes to the total of running programs on Earth / its orbit. But…

>irrelevant for databases in general

It's one of the databases compared in the paper

OP is one of the authors of LMDB

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#155

Post by tsimionescu »

kentonv wrote: post #8
Dwedit wrote: post #2

Memory-Mapped Files = access violations when a disk read fails. If you're not prepared to handle those, don't use memory-mapped files. (Access violation exceptions are the same thing that happens when you attempt to read a null pointer) Then there's the part with writes being delayed. Be prepared to deal with blocks not necessarily updating to disk in the order they were written to, and 10 seconds after the fact. Thi…

> Be prepared to deal with blocks not necessarily updating to disk in the order they were written to, and 10 seconds after the fact. This can make power failures cause inconsistencies. This is not specific to mmap -- regular old write() calls have the same behavior. You need to fsync() (or, with mmap, msync()) to guarantee data is on disk.

It's fun to remember that fsync() on Linux on ext4 at least offers no real guarantee that the data was successfully written to disk. This happens when write errors from background buffered writes are handled internally by the kernel, and they cleanup the error situation (mark dirty pages clean etc). Since the kernel can't know if a later call to fsync() will ever happen, it can't just keep the error around. So, when the call does happen, it will not return any error code. I don't know for sure, but msync() may well have the same behavior.

Here is an LWN article discussing the whole problem as the Postgres team found out about it.

https://lwn.net/Articles/752063/

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#156

Post by kalleboo »

crabbone wrote: post #91
hyc_symas wrote: post #70

Earlier quoted context omitted.

> There's nothing special about kernel programmers. Yes, that was a shorthand generalization for "people who've studied computer architecture" - which most application developers never have. > no DBA worth their salt would put database in the environment where it has to share resources with applications. Most applications today are running on smartphones/mobile devices. That means they're running with local embedded…

> Most applications today are running on smartphones/mobile devices. That's patently false. There are about 8 bn. people. Even if everyone has a smartphone or two, it's nothing compared to the total of all devices that can be called "computer". I think that "smart TV" alone will beat the number of smartphones. But even that is a drop in a bucket when it comes to the total of running programs on Earth / its orbit. But…

Smart TVs are also all running SQLite

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#157

Post by stevefan1999 »

glandium wrote: post #109
jandrewrogers wrote: post #22

Another interesting limitation of mmap() is that real-world storage volumes can exceed the virtual address space a CPU can address. A 64-bit CPU may have 64-bit pointers but typically cannot address anywhere close to 64 bits of memory, virtually or physically. A normal buffer pool does not have this limitation. You can get EC2 instances on AWS with more direct-attached storage than addressable virtual address space o…

To put concrete numbers: x86-64 is limited to 48 bits for virtual addresses, which is "only" 256TiB (281TB).

Intel now extended the page table level to 5-level making this number not so valid. Granted, PL5 creates more TLB pressure and longer memory access time due to that.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#158

Post by ayende »

colanderman wrote: post #140
hyc_symas wrote: post #128

Earlier quoted context omitted.

msync lets you force a flush so you can control the latest possible moment for a writeout. But the OS can flush before that, and you have no way to detect or control that. So you can only control the late side of the timing, not the early side. And in databases, you usually need writes to be persisted in a specific order; early writes are just as harmful as late writes.

I'd even take a memory ordering guarantee, something like, within each page, data is read out sequentially as atomic aligned 64-bit reads with acquire ordering. (Though this probably is what you get on AMD64.) As-is, there's not even a guarantee against an atomic aligned write being torn when written out.

That is absolutely not what you actually get from the hardware.

For fun, there is no guarantee in terms of writing a page in what order it is written. SQLite documents that they assume (but cannot verify) that _sector_ writes are linear, but not atomic. https://www.sqlite.org/atomiccommit.html

> If a power failure occurs in the middle of a sector write it might be that part of the sector was modified and another part was left unchanged. The key assumption by SQLite is that if any part of the sector gets changed, then either the first or the last bytes will be changed. So the hardware will never start writing a sector in the middle and work towards the ends. We do not know if this assumption is always true but it seems reasonable.

You are talking several levels higher than that, at the page level (composed of multiple sectors).

Assume that they reside in _different_ physical locations, and are written at different times. That's fun.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#159

Post by ayende »

pclmulqdq wrote: post #131
jeffffff wrote: post #122

Earlier quoted context omitted.

take a look at http://nms.csail.mit.edu/~stavros/pubs/OLTP_sigmod08.pdf - the overhead of coordinating multiple writers often makes multi-writer databases slower than single-writer databases. remember, everything has to be serialized when it goes to the write ahead log, so as long as you can do the database updates as fast as you can write to the log then concurrent writers are of no benefit.

This is another cool example of a toy database that is again very small: > The database size for one warehouse is approximately 100 MB (we experiment with five warehouses for a total size of 500MB). It is not surprising that when your database basically fits in RAM, serializing on one writer is worth doing, because it just plainly reduces contention. You basically gain nothing in a DB engine from multi-writer transac…

I am using the same rough model, and I'm using that on a 1.5 TB db running on Raspberry PI very successfully.

Pretty much all storage libraries written in the past couple of decades are using single writer. Note that single writer doesn't mean single transaction. Merging transactions is easy and highly profitable, after all.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#160

Post by earthnail »

pclmulqdq wrote: post #136
hyc_symas wrote: post #134

Earlier quoted context omitted.

> RocksDB/LevelDB which implicitly assume that your database is a lot bigger than RAM. Where are you getting that assumption from? LevelDB was built to be used in Google Chrome, not for multi-TB DBs. RocksDB was optimized specifically for in-memory workloads.

I worked with the Bigtable folks at Google. LevelDB's design is ripped straight from BigTable, which was designed with that assumption in mind. I'm also pretty sure it was not designed specifically for Google Chrome's use case - it was written to be a general key-value storage engine based on BigTable, and Google Chrome was the first customer. RocksDB is Facebook's offshoot of LevelDB, basically keeping the core arch…

I've used RocksDB for an in-memory K/V store of ~600GB in size and it worked really well. Not saying it's the best choice out there but it did the job very well for us. And in particular because our dataset was always growing and we needed the option to fallback to disk if needed, RocksDB worked very well.

Was a PITA to optimise though; tons of options and little insight into which ones work.

Post reply on HN