131–140 of 143 posts
Earlier quoted context omitted.
I don't work there any more (it was a decade ago) and I'm pretty busy right now with a new job coming up (offered today). Do you have kernel documentation that says that hugetlb doesn't work for files? I don't see that stated anywhere.
It's filesystem-dependent. In particular, tmpfs will work. To the best of my knowledge, no “normal” filesystems (e.g., ext4, xfs) will.
Earlier quoted context omitted.
reading directly from a PCIe device, such as an NVMe flash, can actually be faster than caching things in RAM first. That's not true though, because the PCIe device DMAs into RAM anyway.
No, it can DMA straight into L3 cache, as mentioned in the article. See https://www.intel.com/content/www/us/en/io/data-direct-i-o-t...
Earlier quoted context omitted.
This is quite interesting since I, too, was under the impression that mmap cannot be used on disk-backed files with huge pages. I tried and failed to find any official kernel documentation around this, but I clearly remember trying to do this at work (on a regular ECS machine with Ubuntu) and getting errors. Based on this SO discussion [1], it is possibly a limitation with popular filesystems like ext4? If anyone kno…
My bad, don't use `MAP_HUGETLB`, just use `MAP_HUGE_1GB`. See a quick example I whipped up here: https://github.com/inetknght/mmap-hugetlb
This is wrong, because your mmap code is being stalled for page faults (including soft page faults that you have when the data is in memory, but not mapped to your process). The io_uring code looks like it is doing all the fetch work in the background (with 6 threads), then just handing the completed buffers to the counter. Do the same with 6 threads that would first read the first byte on each page and then hand tha…
Earlier quoted context omitted.
Yes, it doesn't take a benchmark to find out that storage can not be faster than memory. Even if you had a million SSDs and somehow were able to connect them to a single machine somehow, you would not outperform memory, because the data needs to be read into memory first, and can only then be processed by the CPU. Basic `perf stat` and minor/major faults should be a first-line diagnostic.
This was a comparison of two methods of moving data from the VFS to application memory. Depending on cache status this would run the whole gambit of mapping existing memory pages, kernel to userspace memory copies, and actual disk access. Also, while we’re being annoyingly technical, a lot of server CPUs can DMA straight to the L3 cache so your proof of impossibility is not correct.
Yes, I think maybe a reasonable statement is that a benchmark is supposed to isolate a meaningful effect. This benchmark was not set up correctly to isolate a meaningful effect IMO.
> Also, while we’re being annoyingly technical, a lot of server CPUs can DMA straight to the L3 cache so your proof of impossibility is not correct.
Interesting, didn't know that, thanks!
I think this does not invalidate the point though. You can temporarily stream directly to the L3 cache with DDIO, but as it fills out the cache will get flushed back to the main memory anyway and you will ultimately be memory-bound. I don't think there is some way to do some non-temporal magic here that circumvents main memory entirely.
Earlier quoted context omitted.
I have heard that some Intel NICs can put received data directly into L3 cache. That would definitely make it faster to access than if it were in main RAM. If a NIC can do that over PCI, probably other PCI devices could do the same, at least in theory.
For the curious, > When a 100G NIC is fully utilized with 64B packets and 20B Ethernet overhead, a new packet arrives every 6.72 nanoseconds on average. If any component on the packet path takes longer than this time to process the individual packet, a packet loss occurs. For a core running at 3GHz, 6.72 nanoseconds only accounts for 20 clock cycles, while the DRAM latency is 5-10 times higher, on average. This is th…
> You can temporarily stream directly to the L3 cache with DDIO, but as it fills out the cache will get flushed back to the main memory anyway and you will ultimately be memory-bound. I don't think there is some way to do some non-temporal magic here that circumvents main memory entirely.
Earlier quoted context omitted.
People should just ignore such low-quality material and stop feeding the troll. Information found in both the first and second part of "Memory is slow, disk is fast" series is wrong on so many levels that it isn't worth correcting or commenting. It is obviously written with the help of the AI without actually fact-checking at all and all under the impression that the author is worthwhile which he isn't. Just look at…
wait, P5 is not modern any more ? :D
Earlier quoted context omitted.
Yes, it doesn't take a benchmark to find out that storage can not be faster than memory. Even if you had a million SSDs and somehow were able to connect them to a single machine somehow, you would not outperform memory, because the data needs to be read into memory first, and can only then be processed by the CPU. Basic `perf stat` and minor/major faults should be a first-line diagnostic.
> storage can not be faster than memory This is an oversimplification. It depends what you mean by memory. It may be true when using NVMe on modern architectures in a consumer use case, but it's not true about computer architecture in general. External devices can have their memory mapped to virtual memory addresses. There are some network cards that do this for example. The CPU can load from these virtual addresses…
This requires that device to bring meaningful amounts of its own memory. GPUs do that with VRAM. A storage device does not come with its own RAM, but interesting point!
Earlier quoted context omitted.
For the curious, > When a 100G NIC is fully utilized with 64B packets and 20B Ethernet overhead, a new packet arrives every 6.72 nanoseconds on average. If any component on the packet path takes longer than this time to process the individual packet, a packet loss occurs. For a core running at 3GHz, 6.72 nanoseconds only accounts for 20 clock cycles, while the DRAM latency is 5-10 times higher, on average. This is th…
From a response elsewhere in this thread (my current understanding, could be wrong): > You can temporarily stream directly to the L3 cache with DDIO, but as it fills out the cache will get flushed back to the main memory anyway and you will ultimately be memory-bound. I don't think there is some way to do some non-temporal magic here that circumvents main memory entirely.
EDIT: Here's an interesting writeup about trying to make use of it with FreeBSD+netmap+ipfw: https://adrianchadd.blogspot.com/2015/04/intel-ddio-llc-cach... So it can work as advertised, it's just very constraining, requiring careful tuning if not outright rearchitecting your processing pipeline with the constraints in mind.