121–130 of 143 posts
Earlier quoted context omitted.
> MAP_HUGETLB can't be used for mmaping files on disk False. I've successfully used it to memory-map networked files.
My bad, don't use `MAP_HUGETLB`, just use `MAP_HUGE_1GB`. See a quick example I whipped up here: https://github.com/inetknght/mmap-hugetlb
This is wrong, because your mmap code is being stalled for page faults (including soft page faults that you have when the data is in memory, but not mapped to your process). The io_uring code looks like it is doing all the fetch work in the background (with 6 threads), then just handing the completed buffers to the counter. Do the same with 6 threads that would first read the first byte on each page and then hand tha…
Yes, it doesn't take a benchmark to find out that storage can not be faster than memory. Even if you had a million SSDs and somehow were able to connect them to a single machine somehow, you would not outperform memory, because the data needs to be read into memory first, and can only then be processed by the CPU. Basic `perf stat` and minor/major faults should be a first-line diagnostic.
Also, while we’re being annoyingly technical, a lot of server CPUs can DMA straight to the L3 cache so your proof of impossibility is not correct.
Earlier quoted context omitted.
Yes, it doesn't take a benchmark to find out that storage can not be faster than memory. Even if you had a million SSDs and somehow were able to connect them to a single machine somehow, you would not outperform memory, because the data needs to be read into memory first, and can only then be processed by the CPU. Basic `perf stat` and minor/major faults should be a first-line diagnostic.
> storage can not be faster than memory This is an oversimplification. It depends what you mean by memory. It may be true when using NVMe on modern architectures in a consumer use case, but it's not true about computer architecture in general. External devices can have their memory mapped to virtual memory addresses. There are some network cards that do this for example. The CPU can load from these virtual addresses…
Also, direct access of device memory is quite slow. High throughput usecases like storage or network have relied entirely on DMA to system RAM from the device for decades.
This is pretty great. I only learned to use perf_events to see annotated disassembly a few weeks ago, although I don't know how to interpret what I see there yet. I suspect the slowness identified with mmap() here is somewhat fixable, for example by mapping already-in-RAM pages somewhat more eagerly. So it wouldn't be surprising to me (though see above for how much I'm not an expert) if next year mmap were faster tha…
The io_uring solution avoids this whole effort of mapping. It doesn't have to map the already-in-RAM pages at all. It reuses a small set of buffers. So there is a lot of random cache-miss prone work that mmap() has to do that the io_uring solution avoids. If mmap() does this in the background it would cache up with io_uring. I'd then have to get a couple more drives to get io_uring to catch up. With enough drives I'd…
This is wrong, because your mmap code is being stalled for page faults (including soft page faults that you have when the data is in memory, but not mapped to your process). The io_uring code looks like it is doing all the fetch work in the background (with 6 threads), then just handing the completed buffers to the counter. Do the same with 6 threads that would first read the first byte on each page and then hand tha…
Indeed. Use with mmap with MAP_POPULATE which will pre populate.
Earlier quoted context omitted.
Its not even about clickbait for me, but I really dont want to go parse an article to figure out what is meant by "Memory is slow, Disk is fast". You want "clickbait" to make people click and think, we want descriptive tittles to know what the article is about before we read it. That used to be original purpose of tittles, we like it that way. Its like as if youd label your food product "you wont believe this", and f…
> Its like as if youd label your food product "you wont believe this", and forced customers to figure what it is from ingredients list. Indeed[0]. [0] https://en.wikipedia.org/wiki/I_Can't_Believe_It's_Not_Butte... !
Earlier quoted context omitted.
From the application perspective, it's not truly async. On a deference, your app may be blocked indefinitely as data is paged into memory. In the early 2000's I worked on systems that made heavy use of mmap. In constrained ("dev") environments with slow disks, you could be blocked for several seconds...
This branch of the discussion is is about dereferencing on multiple threads concurrently. That doesn't block the application, each mmap'd dereference only blocks its own thread (same as doing read()). In my own measurements with NVMe RAID, doing this works very well on Linux for storage I/O. I was getting similar performance to io_uring with O_DIRECT, and faster performance when the data is likely to be in the page c…
Earlier quoted context omitted.
I mean even with their memory mapped, the physical access to the memory ( NVME ) will always be slower than the physical access to the RAM, right? Which external device has memory access as fast or faster than generic RAM?
I have heard that some Intel NICs can put received data directly into L3 cache. That would definitely make it faster to access than if it were in main RAM. If a NIC can do that over PCI, probably other PCI devices could do the same, at least in theory.
> When a 100G NIC is fully utilized with 64B packets and 20B Ethernet overhead, a new packet arrives every 6.72 nanoseconds on average. If any component on the packet path takes longer than this time to process the individual packet, a packet loss occurs. For a core running at 3GHz, 6.72 nanoseconds only accounts for 20 clock cycles, while the DRAM latency is 5-10 times higher, on average. This is the main bottleneck of the traditional DMA approach.
> The Intel® DDIO technology in Intel® Xeon® processors eliminates this bottleneck. Intel® DDIO technology allows PCIe devices to perform read and write operations directly to and from the L3 cache, or the last level cache (LLC).
https://www.intel.com/content/www/us/en/docs/vtune-profiler/...
Earlier quoted context omitted.
OK, so we need a comparison between a multi threaded mmap approach and io_uring. Which would be faster?
If the memory access pattern is the same, there are no significant differences.
Earlier quoted context omitted.
No. "io_uring faster than mmap" is sort of a truism: sequential page faults are slower than carefully orchestrated async I/O. The point of the article is that reading directly from a PCIe device, such as an NVMe flash, can actually be faster than caching things in RAM first.
reading directly from a PCIe device, such as an NVMe flash, can actually be faster than caching things in RAM first. That's not true though, because the PCIe device DMAs into RAM anyway.
See https://www.intel.com/content/www/us/en/io/data-direct-i-o-t...