21–30 of 182 posts
Many general-purpose OS abstractions start leaking when you're working on systems-like software. You notice it when web servers are doing kernel bypass to for zero-copy, low-latency networking, or database engines throw away the kernel's page cache to implement their own file buffer.
Memory-Mapped Files = access violations when a disk read fails. If you're not prepared to handle those, don't use memory-mapped files. (Access violation exceptions are the same thing that happens when you attempt to read a null pointer) Then there's the part with writes being delayed. Be prepared to deal with blocks not necessarily updating to disk in the order they were written to, and 10 seconds after the fact. Thi…
For the second part of your comment, on Linux systems, there is the msync() system call that can be used to flush the page cache on demand.
Old timers will recall when using mmap was a prominently promoted selling point for the “no sql” dbms.
For documents it made access fast since there’s no joins, etc. that require paging from all over. The problem ended up being updates and compaction issues.
And there was a very well known cartoon video discussion about it with “web scale” and “just write to dev null” and other classics that became memes :)
Earlier quoted context omitted.
mmap can be handy but usually is not a good idea when you care about ACID properties. So it tends to be most useful outside databases.
Can you give some examples where mmap is useful?
Also, I've never tested this, but I believe mapped files will get flushed as long as the system stays running. So if you only need resilience against abnormal termination rather than system crashes, it seems like a good option?
Many general-purpose OS abstractions start leaking when you're working on systems-like software. You notice it when web servers are doing kernel bypass to for zero-copy, low-latency networking, or database engines throw away the kernel's page cache to implement their own file buffer.
Web servers doing kernel bypass for zero-copy networking? Do you have a specific example in mind? I'm curious.
Edit: Hm, it might not be possible to mmap files with huge-pages. This LWN article[1] from 5 years ago talks about the work that would be required, but I haven't seen any follow-ups.
Old timers will recall when using mmap was a prominently promoted selling point for the “no sql” dbms.
reminds me of how industries typically start out dominated by vertically integrated companies, move to specialized horizontal companies, then generally move back to vertical integration due to efficiency. Car industry started this way with Ford, went away from it, and now Tesla is doing it again. Lots of other examples in other industries