61–70 of 137 posts
I have a great deal of experience in running very large memory-mapped databases using LMDB. The default Linux settings dealing with memory mapped files are pretty horrible. The observed poor performance is directly related to not configuring several very important kernel parameters.
Earlier quoted context omitted.
From that article: the whole fsyncgate thing seems like a pretty strong counterargument to "mmap adds more complexity than it removes": https://danluu.com/fsyncgate/
That actually doesn't matter This is orthogonal to mmap
Interesting parallels in this work to Tanenbaum's "RPC Considered Harmful"†; in both cases, you've got an abstraction that papers over a huge amount of complexity, and it ends up burning you because a lot of that complexity turns out to be pretty important and the abstraction has cost you control over it. † https://www.cs.vu.nl/~ast/Publications/Papers/euteco-1988.pd...
Yes. Whenever you need better performance and reliability, identify an abstraction beloved of CS professors, and bypass it. When I last checked, libtorrent was utterly failing to use O_DIRECT semantics. I started making a patch, but there are several places that do file ops, and the main one was more complicated than I could afford to dive into at the time.
We have abstractions for a reason. We have lower-level primitives for a reason. Understanding the differences, reasoning about all trade-off angles, and making the right choice in each project is a majority of the software engineering job.
One possible advantage of using mmap over a buffer pool can be programmer ergonomics. Reading data into a buffer pool in process RAM takes time to warm up, and the pool can only be accessed by a single process. In contrast, for an mmap-backed data structure, assuming that files are static once written (which can be the case for an multi-version concurrency control (MVCC) architecture), you open an mmap read-only conn…
Earlier quoted context omitted.
When I worked on/with BerkeleyDB in the late 90s we came to the conclusion that the various OS mmap() implementations had been tweaked/fixed to the point where they worked for the popular high profile applications (in those days: Oracle). So it can appear like everything is fine, but that probably means your code behaves the same way as .
Um... Oracle (and other enterprise databases like DB2) don't use mmap. They use Direct I/O. Oracle does have anonymous (non-file-backed) memory which is mmap'ed and shared across various Oracle processes, called the Shared Global Area (SGA), but it's not used for I/O.
Earlier quoted context omitted.
I personally prefer the abstract to jumping straight into a full paper, especially since it's quite rich (not one of those two line entries like some arXiv paper abstracts). After reading the abstract I did end up opening the PDF.. but I'm hesitant to pay the PDF tax early. Is this one of those "original source" type decisions?
Yes. I hear you about the downside, but the downside of the more superficial-accessible 'home page' is that people will not read any further, and instead simply respond generically.
But I agree with the other person; if people really won't read any further from the homepage, then I highly doubt they'd read any further than the headline and maybe abstract of the original paper anyway, so there ain't really much upside to linking directly to the paper - whereas there's quite a bit of downside for anyone who might feel inclined to watch the video instead (which essentially covers the same information as the paper, just at a higher level / without the same level of detail) or review the benchmark code - neither of which are accessible from the paper.
Earlier quoted context omitted.
Yes. Whenever you need better performance and reliability, identify an abstraction beloved of CS professors, and bypass it. When I last checked, libtorrent was utterly failing to use O_DIRECT semantics. I started making a patch, but there are several places that do file ops, and the main one was more complicated than I could afford to dive into at the time.
So... don't bypass abstractions, unless you actually have time to do a better job, no? We have abstractions for a reason. We have lower-level primitives for a reason. Understanding the differences, reasoning about all trade-off angles, and making the right choice in each project is a majority of the software engineering job.
And unless there's a clear benefit to it. If you haven't identified the abstraction as a significant bottleneck, then is it really worthwhile to go through the trouble of bypassing it?
Earlier quoted context omitted.
Generally for perf critical use cases you dedicate the machine to the database. This simplifies many things (avoiding having to reason about sharing, etc etc).
This makes me wonder whether there would be value in an OS that is also a DBMS (or vice versa). In other words, if the DBMS has total control over the hardware, perhaps performance can be maximized without too much additional complexity.
How do you implement lockless atomic updates for multiple writers across multiple threads & processes without mmap? With mmap it is straight forward for processes to open persistent arrays of atomics as a file, and use compare and exchange operations to prevent data races when multiple threads or processes update the same page without any file locks, advisory locks, or mutexes. With manual read() and write() calls, t…
Why do you need lockless atomic updates to a file-backed memory area? Genuinely curious.
Choosing mmap() gets you something that works sooner than later. But then you have a pile of blocking-style synchronous code likely exploiting problematic assumptions to rewrite when you realize you want something that doesn't just work, but works well .