Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

db.cs.cmu.edu

Post reply on HN
Print view View original

161–170 of 182 posts

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#161

Post by hyc_symas »

ayende wrote: post #158
colanderman wrote: post #140

Earlier quoted context omitted.

I'd even take a memory ordering guarantee, something like, within each page, data is read out sequentially as atomic aligned 64-bit reads with acquire ordering. (Though this probably is what you get on AMD64.) As-is, there's not even a guarantee against an atomic aligned write being torn when written out.

That is absolutely not what you actually get from the hardware. For fun, there is no guarantee in terms of writing a page in what order it is written. SQLite documents that they assume (but cannot verify) that _sector_ writes are linear, but not atomic. https://www.sqlite.org/atomiccommit.html > If a power failure occurs in the middle of a sector write it might be that part of the sector was modified and another part…

Every HDD since the 1980s has guaranteed atomic sector writes:

> Currently all hard drive/SSD manufacturers guarantee that 512 byte sector writes are atomic. As such, failure to write the 106 byte header is not something we account for in current LMDB releases. Also, failures of this type should result in ECC errors in the disk sector - it should be impossible to successfully read a sector that was written incorrectly in the ways you describe.

Even in extreme cases, the probability of failure to write the leading 128 out of 512 bytes of a sector is nearly nil - even on very old hard drives, before 512-byte sector write guarantees. We would have to go back nearly 30 years to find such a device, e.g.

https://archive.org/details/bitsavers_quantumQuaroductManual...

Page 23, Section 2.1 "No damage or loss of data will occur if power is applied or removed during drive operation, except that data may be lost in the sector being written at the time of power loss."

  From the specs on page 15, the data transfer rate to/from the platters is
 1.25MB/sec, so the time to write one full sector is 0.4096ms; the time to
 write the leading 128 bytes of the sector is thus 1/4 of that: 0.10ms. You
 would have to be very very unlucky to have a power failure hit the drive
 within this .1ms window of time. Fast-forward to present day and it's simply
 not an issue.
^ above quoted from https://lists.openldap.org/hyperkitty/list/openldap-devel@op...

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#162

Post by hyc_symas »

jandrewrogers wrote: post #144
hyc_symas wrote: post #112

Earlier quoted context omitted.

You don't need more than single-writer concurrency if your write txns are fast enough. Our experience with OpenLDAP was that multi-writer concurrency cost too much overhead. Even though you may be writing primary records to independent regions of the DB, if you're indexing any of that data (which all real DBs do, for query perf) you wind up getting a lot of contention in the indices. That leads to row locking conflic…

> You don't need more than single-writer concurrency if your write txns are fast enough. This only works on systems with sufficiently slow storage. If your server has a bunch of NVMe, which is a pretty normal database config these days, you will be hard-pressed to get anywhere close to the theoretical throughput of the storage with a single writer. That requires 10+ GB/s sustained. It is a piece of cake with multiple…

That's all just false. Just because you're single-writer at the application level doesn't mean the OS isn't queueing enough writes to saturate storage at the device level. We've benchmarked plenty of high speed NVMe devices, like Intel Optane SSDs, etc. showing this. http://www.lmdb.tech/bench/optanessd/

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#163

Post by athrowaway3z »

jandrewrogers wrote: post #148
hyc_symas wrote: post #130

Earlier quoted context omitted.

All of that is true, but I don't think it's a realistic concern. You're going to be sharding your data across multiple nodes before it gets that large. Nobody wants to sit around backing up or restoring a monolithic 256 TiB database.

Technically you get quite a bit less than the 256 TB theoretical in practice. It is a realistic concern, I’ve lived it for more than a decade across many orgs, though I shared your opinion at one point. Storage density is massively important for both workload scalability and economic efficiency. Low storage density means buying a ton of server hardware that sits idle under max load and vastly larger clusters than wou…

This entire comment section is a bit of a dumpster-fire. I'm convinced the word database has outlived its usefulness for any serious discussion. It has the informational density of saying : "I work in IT"

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#164

Post by xorcist »

hyc_symas wrote: post #70
crabbone wrote: post #65

Earlier quoted context omitted.

> your DBMS is the only process running on a machine. In practice, (a) is never true, and (b) is no longer true because everyone is running apps inside containers inside shared VMs. There's nothing special about kernel programmers. In fact, if I had to compare, I'd go with storage people being the more experienced / knowledgeable ones. They have a highly competitive environment, which requires a lot more understandin…

> There's nothing special about kernel programmers. Yes, that was a shorthand generalization for "people who've studied computer architecture" - which most application developers never have. > no DBA worth their salt would put database in the environment where it has to share resources with applications. Most applications today are running on smartphones/mobile devices. That means they're running with local embedded…

> "people who've studied computer architecture" - which most application developers never have

If you are developing an DBMS and haven't studied computer architecture, the best idea is probably to ask more experienced people to help out with your ideas.

From my limited knowledge, I don't think the article is old enough to be obsolete, just that there's a lot more to it.

Not to be gatekeeping or anything, but it is a pretty well studied field with lots of very knowledgeable people around, who are probably more than keep to help. There aren't too many qualified jobs around and you probably have a budget if you are developing a database commercially.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#165

Post by RaisingSpear »

hyc_symas wrote: post #79
gavinray wrote: post #74

Earlier quoted context omitted.

The argument is that: - Queries can trigger blocking page faults when accessing (transparently) evicted pages, causing unexpected I/O stalls - mmap() complicates transactionality and error-handling - Page table contention, single-threaded page eviction, and TLB shootdowns become bottlenecks

1 - for reading any uncached data, the I/O stalls are unavoidable. Whatever client requested that data is going to have to wait regardless. 2 - complexity? this is simply false. LMDB's ACID txns using MVCC are much simpler than any "traditional" approach. 3 - contention is a red herring since this approach is already single-writer, as is common for most embedded k/v stores these days. You lose more perf by trying to…

> for reading any uncached data, the I/O stalls are unavoidable.

Excuse me for a silly question, but whilst an I/O stall may be unavoidable, wouldn't a thread stall be avoidable if you're not using mmap?

Assuming that you're not swapping, you'll generally know if you've loaded something into memory or not, whilst mmap doesn't help you know if the relevant page is cached. If the data isn't in memory, you can send the I/O request to a thread to retrieve it, and the initiating thread can then move onto the next connection. I suspect this isn't doable under mmap based access?

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#166

Post by kentonv »

pclmulqdq wrote: post #147
kentonv wrote: post #96

Earlier quoted context omitted.

> This is not true. This depends on how the file was opened. You may request DIRECT | SYNC Well sure, but 99.9% of people don't do that (and shouldn't, unless they really know what they are doing). > In the world of today, you are very rarely writing to something that's not network attached, and depending on your appliance, the meaning of acknowledgement from write() differs. What network-attached storage actually us…

100% of people writing a database know about filesystem options like DIRECT and SYNC, and that is the subject of this paper. Also, most of the network-attached storage we people use is in the form of things like EBS, which is very careful to imitate the behavior of a real disk, but with different performance and some different (albeit very rare ) failure modes.

100% of people writing databases also know how fsync() and msync() work. I interpreted this thread as being targeted at a wider audience.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#167

Post by pjdesno »

tayo42 wrote: post #89
pjdesno wrote: post #42

Not just databases - we ran into the same issues when we needed a high-performance caching HTTP reverse proxy for a research project. We were just going to drop in Varnish, which is mmap-based, but performance sucked and we had to write our own. Note that Varnish dates to 2006, in the days of hard disk drives, SCSI, and 2-core server CPUs. Mmap might well have been as good or even better than I/O back then - a lot of…

Varnish has a file system backed cache that depends on the page cache to keep it fast. What did you differently in your custom one that was faster then varnish?

Simple multithreaded read/write. On a 20-core 40-thread machine with a couple of fast NVMe drives it was way faster.

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#168

Post by Mikhail_Edoshin »

gavinray wrote: post #61
hyc_symas wrote: post #43

This is a pretty old argument and IMO it's far out of date/obsolete. Taking full control of your I/O and buffer management is great if (a) your developers are all smart and experienced enough to be kernel programmers and (b) your DBMS is the only process running on a machine. In practice, (a) is never true, and (b) is no longer true because everyone is running apps inside containers inside shared VMs. In the modern a…

Out of curiosity, how many databases have you written? This is co-authored by Pavlo, Viktor Leiss, with feedback from Neumann. I'm sorry, but if someone on the internet claims to know better than those 3, you're going to need some monumental evidence of your credibility. Additionally, what you link here: > ... (See the slide at 21:25 into the video). Modern DBMSs spend 96% of their time managing buffers and locks, an…

Even thought the data resides mostly in-memory they still have to write transactions to disk to preserve them, don't they?

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#169

Post by afr0ck »

crabbone wrote: post #83
afr0ck wrote: post #24

Earlier quoted context omitted.

Linux throws a SIGBUS. A process should anticipate such I/O failures by implementing a SIGBUS handler, especially a database server. For the second part of your comment, on Linux systems, there is the msync() system call that can be used to flush the page cache on demand.

> msync() system call that can be used to flush the page cache on demand. for everyone , not just the file you mapped to memory. I.e. the guarantee is that your file will be written, but there's no way to do that w/o affecting others. This is not such a hot idea in an environment where multiple threads / processes are doing I/O.

msync() affects only the pages that part of the mmap area you ask for in the arguments. From the man pages:

> int msync(void addr[.length], size_t length, int flags);

> msync() flushes changes made to the in-core copy of a file that was mapped into memory using mmap(2) back to the filesystem

Re: Are You Sure You Want to Use MMAP in Your Database Management System? (2022)

#170

Post by crabbone »

afr0ck wrote: post #169
crabbone wrote: post #83

Earlier quoted context omitted.

> msync() system call that can be used to flush the page cache on demand. for everyone , not just the file you mapped to memory. I.e. the guarantee is that your file will be written, but there's no way to do that w/o affecting others. This is not such a hot idea in an environment where multiple threads / processes are doing I/O.

msync() affects only the pages that part of the mmap area you ask for in the arguments. From the man pages: > int msync(void addr[.length], size_t length, int flags); > msync() flushes changes made to the in-core copy of a file that was mapped into memory using mmap(2) back to the filesystem

No it doesn't. That's physically impossible. Read what you quoted -- it never says that it's going to do it only for the file in question.

If you don't know why it's not possible, here's a simplified version of it: hardware protocols (s.a. SCSI) must have fixed size messages to fit them through the pipeline. I.e. you cannot have a message larger than the memory segment used for communication with the device, because that will cause fragmentation and will lead to a possibility of message being corrupted (the "tail" being lost or arriving out of order).

On the other hand, to "flush" a file to persistent storage you'd have to specify all blocks associated with the file that need to be written. If you try to do this, it will create a message of arbitrary size, possibly larger than the memory you can store it in. So, the only way to "flush" all blocks associated with a file is to "flush" everything on a particular disk / disks used by the filesystem. And this is what happens in reality when you do any of the sync family commands. The difference is only in what portion of the not-yet synced data the OS will send to the disk before requesting a sync, but the sync itself is for the entire disk, there aren't any other syncs.

Post reply on HN