Are you sure you want to use MMAP in your database management system? [pdf]

Post by CyberDildonics » Sat, Jan 15, 2022, 4:16 AM UTC

jstimpfle wrote: post #42

randbox wrote: post #40

Earlier quoted context omitted.

A user process doesn't have the information it needs to do a good job of coordinating updates from multiple writers to database pages and indices. With MMAP, writers have access to shared atomics which they can update using compare-exchange operations to prevent data races which would be common when using read() and write() without locks.

Are you saying that without mmap() there will be data races??

Post by bogomipz » Sat, Jan 15, 2022, 4:45 AM UTC

tmountain wrote: post #48

jandrewrogers wrote: post #16

The pragmatic consideration that usually influences the decision to use mmap() is the large discontinuity in skill and expertise required to replace it. Writing your own alternative to mmap() can be significantly superior in terms of performance and functionality, and often lends itself to a cleaner database architecture. However, this presumes a sufficiently sophisticated design for an mmap() replacement. The learni…

The original version of MongoDB used mmap, and I worked at a company that had a ton of issues with cache warmup and the cache getting trashed by competing processes. Granted this was a long time ago, but the main issue was the operating system's willingness to reallocate large swaths of memory from the address space to whatever process was asking for memory right now. Once the working set got trashed, performance wou…

Post by jfindley » Sat, Jan 15, 2022, 4:55 AM UTC

ayende wrote: post #30

10000truths wrote: post #13

Earlier quoted context omitted.

You don't want the OS to take care of reading from disk and page caching/eviction. You want the DB itself to have explicit control over that, because the DB has information on access patterns and table format that the OS is not aware of. It is better equipped than the OS to anticipate what portions of tables/indices need to be cached in memory. It is better equipped to calculate when/where/what/how much to prefetch f…

I'm the author (well, one of) RavenDB You are correct to an extent, but there are a few things yo noted. * you can design your system so the access pattern that the OS is optimized for matches your needs * you can use madvise() to give some useful hints * the amount of complexity you don't have to deal with is staggering

Post by dang » Sat, Jan 15, 2022, 5:13 AM UTC

yellowapple wrote: post #66

dang wrote: post #18

Earlier quoted context omitted.

Yes. I hear you about the downside, but the downside of the more superficial-accessible 'home page' is that people will not read any further, and instead simply respond generically.

The current URL just redirects back to that "superficial-accessible 'home page'" anyway (probably as a substitute for 404 handling, I'd guess); if the intent is to link directly to the paper/PDF, you probably want https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf But I agree with the other person; if people really won't read any further from the homepage, then I highly doubt they'd read any further than the h…

Post by pclmulqdq » Sat, Jan 15, 2022, 6:08 AM UTC

randbox wrote: post #29

How do you implement lockless atomic updates for multiple writers across multiple threads & processes without mmap? With mmap it is straight forward for processes to open persistent arrays of atomics as a file, and use compare and exchange operations to prevent data races when multiple threads or processes update the same page without any file locks, advisory locks, or mutexes. With manual read() and write() calls, t…