111–120 of 143 posts
Earlier quoted context omitted.
> MAP_HUGETLB can't be used for mmaping files on disk False. I've successfully used it to memory-map networked files.
Share your code?
See a quick example I whipped up here: https://github.com/inetknght/mmap-hugetlb
Shouldn't this be "io_uring is faster than mmap"? I guess that would not get much engagement though! That said, cool write up and experiment.
No. "io_uring faster than mmap" is sort of a truism: sequential page faults are slower than carefully orchestrated async I/O. The point of the article is that reading directly from a PCIe device, such as an NVMe flash, can actually be faster than caching things in RAM first.
That's not true though, because the PCIe device DMAs into RAM anyway.
The real difference is that with io_uring and O_DIRECT you manage the cache yourself (and can't share with other processes, and the OS can't reclaim the cache automatically if under memory pressure), and with mmap this is managed by the OS. If Linux had an API to say "manage this buffer you handled me from io_uring as if it were a VFS page cache (and as such it can be shared with other processes, like mmap), if you w…
Yes Linus has been ranting for decades against O_DIRECT saying similar things (aka better hints on pages and cache usage). The notorious archive of Linus rants on [0] starts with "The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances". It gets better afterwards, though I'm not clear w…
As a database example, there are major classes of optimization that require perfect visibility into the state of the entire page cache with virtually no overhead and strict control over every change of state that occurs. O_DIRECT allows you to achieve this. The optimizations are predicated on the impossibility of an external process modifying state. It requires perfect control of the schedule which is invalidated if the kernel borrows part of the page cache. Whether or not the kernel asks nicely doesn't matter, it breaks a design invariant.
The Linus rant is from a long time ago. Given the existence of things like io_uring which explicitly enables this type of behavior almost to the point of encouraging it, Linus may understand the use cases better now.
Cool. Original author here. AMA.
Like people mention, hugetlb,etc could be an improvement, but the core issue holding it it down probably has to do with mmap, 4k pages and paging behaviours, mmap will cause faults for each "small" 4k page not in memory, causing a kernel jump and then whatever machinery to fill in the page-cache (and bring up data from disk with the associated latency). This in contrast with the io_uring worker method where you keep…
Earlier quoted context omitted.
Yes Linus has been ranting for decades against O_DIRECT saying similar things (aka better hints on pages and cache usage). The notorious archive of Linus rants on [0] starts with "The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances". It gets better afterwards, though I'm not clear w…
I know people like to post this rant but in this case Linus simply doesn't understand the problem domain. O_DIRECT is commonly used in contexts where the fundamental mechanisms of the kernel cache are inappropriate. It can't be fixed with hints. As a database example, there are major classes of optimization that require perfect visibility into the state of the entire page cache with virtually no overhead and strict c…
Earlier quoted context omitted.
> MAP_HUGETLB can't be used for mmaping files on disk False. I've successfully used it to memory-map networked files.
My bad, don't use `MAP_HUGETLB`, just use `MAP_HUGE_1GB`. See a quick example I whipped up here: https://github.com/inetknght/mmap-hugetlb
Earlier quoted context omitted.
Lol. Thanks.
Its not even about clickbait for me, but I really dont want to go parse an article to figure out what is meant by "Memory is slow, Disk is fast". You want "clickbait" to make people click and think, we want descriptive tittles to know what the article is about before we read it. That used to be original purpose of tittles, we like it that way. Its like as if youd label your food product "you wont believe this", and f…
"Accessing memory is slower in some circumstances than direct disk access"
Earlier quoted context omitted.
My bad, don't use `MAP_HUGETLB`, just use `MAP_HUGE_1GB`. See a quick example I whipped up here: https://github.com/inetknght/mmap-hugetlb
Adding MAP_HUGE_1GB and not MAP_HUGETLB does compile and run for me. Not convinced that its' actually doing anything. Performance is the same.
This is pretty great. I only learned to use perf_events to see annotated disassembly a few weeks ago, although I don't know how to interpret what I see there yet. I suspect the slowness identified with mmap() here is somewhat fixable, for example by mapping already-in-RAM pages somewhat more eagerly. So it wouldn't be surprising to me (though see above for how much I'm not an expert) if next year mmap were faster tha…
Cool. Original author here. AMA.
Like people mention, hugetlb,etc could be an improvement, but the core issue holding it it down probably has to do with mmap, 4k pages and paging behaviours, mmap will cause faults for each "small" 4k page not in memory, causing a kernel jump and then whatever machinery to fill in the page-cache (and bring up data from disk with the associated latency). This in contrast with the io_uring worker method where you keep…
I think I'm crossing the numa boundary which means some percentage of the accesses are higher latency.