11–20 of 137 posts
why settle for errno when you can have a segfault.
Been down that primrose path, have the road rash to prove it. mmap() is great until you realize that pretty much all you've avoided is some buffer management that you probably need to do anyway. The OS just doesn't have the information it needs to do a great (or even correct) job of caching database pages.
RavenDB's response to this paper: https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to...
The really key part seems to be this: "If you aren’t using mmap, on the other hand, you still need to handle of all those issues" Which seems like a reasonable statement. Is it less work to make your own top-to-bottom buffer pool, and would that necessarily avoid similar issues? Or is it less work to use mmap(), but address the issues?
RavenDB's response to this paper: https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to...
Sure, it might be more work than using mmap. But it's also more correct, forces you to handle edge cases, and much more amenable to platform-specific improvements a la kqueue/io_uring.
Earlier quoted context omitted.
The really key part seems to be this: "If you aren’t using mmap, on the other hand, you still need to handle of all those issues" Which seems like a reasonable statement. Is it less work to make your own top-to-bottom buffer pool, and would that necessarily avoid similar issues? Or is it less work to use mmap(), but address the issues?
I suppose. Some problems with mmap() are a bit hard to fix from user land though. You will hit contention on locks inside the kernel (mmap_sem) if the database does concurrent high throughput mmap()/unmap(). I don't follow linux kernel development closely to know if this has been improved recently, but it was easy to reproduce it 4-5 years ago.
Url changed from https://db.cs.cmu.edu/mmap-cidr2022/ , which has the abstract and a link to this video: https://www.youtube.com/watch?v=1BRGU_AS25c and this code: https://github.com/viktorleis/mmapbench
As a consequence, early attempts to replace mmap() are often quite poor. You don't know what you don't know, and details of the implementation that are often glossed over turn out to be critical in practice. For example, most people eventually figure out that LRU cache replacement is a bad idea, but many of the academic alternatives cause CPU cache thrashing in real systems, replacing one problem with another. There are clever and non-obvious design elements that can greatly mitigate this but they are treated as implementation details in most discussions of cache replacement and largely not discoverable if you are writing one for the first time.
While mmap() is a mediocre facility for a database, I think we also have to be cognizant that replacing it competently is not a trivial ask for most software engineers. If their learning curve is anything like mine, I went from mmap() to designing obvious alternatives with many poorly handled edge cases, and eventually figuring out how to design non-obvious alternatives that could smoothly handled very diverse workloads. That period of "poor alternatives" in the middle doesn't produce great databases but it almost feels necessary to properly grok the design problem. Most people would rather spend their time working on other parts of a database.
RavenDB's response to this paper: https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to...
You don't want the OS to take care of reading from disk and page caching/eviction. You want the DB itself to have explicit control over that, because the DB has information on access patterns and table format that the OS is not aware of. It is better equipped than the OS to anticipate what portions of tables/indices need to be cached in memory. It is better equipped to calculate when/where/what/how much to prefetch f…
Aren't system calls such as madvise supposed to allow user space to let the kernel know precisely that information?
Url changed from https://db.cs.cmu.edu/mmap-cidr2022/ , which has the abstract and a link to this video: https://www.youtube.com/watch?v=1BRGU_AS25c and this code: https://github.com/viktorleis/mmapbench
I personally prefer the abstract to jumping straight into a full paper, especially since it's quite rich (not one of those two line entries like some arXiv paper abstracts). After reading the abstract I did end up opening the PDF.. but I'm hesitant to pay the PDF tax early. Is this one of those "original source" type decisions?
Makes me wonder if there is an alternative universe in which there is a syscall with semantics similar to mmap that avoids these pitfalls. It's not like mmap's semantics are the only semantics that we could have for memory-mapped IO.
RavenDB's response to this paper: https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to...
The really key part seems to be this: "If you aren’t using mmap, on the other hand, you still need to handle of all those issues" Which seems like a reasonable statement. Is it less work to make your own top-to-bottom buffer pool, and would that necessarily avoid similar issues? Or is it less work to use mmap(), but address the issues?
- single-threaded calls to 'fallocate' will help avoiding sparse files and SIGBUS during memory write - over-allocating, caching memory addresses and minimizing OS calls - transactional safety can be implemented via shared memory model - hugetlb can minimize TLB shootdowns
I personally do not have any regrets using mmap because of all the benefits they provide