Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches • Hacker News Board

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#21

Post by sumtechguy » Mon, Oct 18, 2021, 5:46 PM UTC

wtallis wrote: post #18

reacharavindh wrote: post #15

Does anyone know whether these optimisations in block layer and iouring benefit major IO hitters like PostgreSQL, ZFS, NFS writes etc? Will it take several years and monumental effort before they adopt iouring first?

io_uring is an improved interface between userspace and the kernel, so it doesn't provide any direct benefits to in-kernel filesystem operations, and the performance benefits io_uring does provide should be largely filesystem-agnostic. That said, some of these recent optimizations may be low enough in the kernel's io stack to also benefit io originating within the kernel itself. Userspace applications that already ha…

From an earlier article https://www.phoronix.com/scan.php?page=news_item&px=Linux-Ap...

"His patches pushing the greater performance have been changes to the block code, NVMe, multi-queue blk-mq, and IO_uring." https://git.kernel.dk/cgit/linux-block/log/?h=perf-wip

So it looks like he is playing with a good portion of the IO block stack with a very recent concentration on io_uring. So maybe some of it?...

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#22

Post by anarazel » Mon, Oct 18, 2021, 6:11 PM UTC

benlwalker wrote: post #17

anarazel wrote: post #14

Earlier quoted context omitted.

It's worth noting that there's some batching involved, so it's not 125ns for processing one io, then again 125ns for the next. Both combining some of the work, and superscalar execution is necessary to reach this high numbers...

To expand on the batching details, I haven't seen exactly what he's doing here, but historically the numbers quoted are from benchmarks that submit something like 128 total queue depth in alternating batches of 64. The disks hit max performance with a much lower queue depth than 64. So it's very unrealistic perfect batching that you'll never see in a real application. The workload is only a useful tool for optimizing…

I think you need batching for the nvme doorbell alone at anything close to these kinds of rates...

For some workloads it's not that hard to have that deep queues. What's harder is to know when to use them and when not. There's really not enough information available to make any of this self-tuning.

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#23

Post by the8472 » Mon, Oct 18, 2021, 6:43 PM UTC

egberts1 wrote: post #20

I look forward to a faster `cp` command.

io_uring certainly could help there, but (conditional on using SSDs) even switching to something parallel like xcp or fcp would speed things up compared to single-threaded gnu cp.

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#24

Post by tomc1985 » Mon, Oct 18, 2021, 7:56 PM UTC

I keep seeing Axboe as a typo'd version of Adobe and it's really bugging me

edit- It's someone's name, I thought it was a company or product

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#25

Post by terafo » Mon, Oct 18, 2021, 8:06 PM UTC

guerrilla wrote: post #5

Aissen wrote: post #3

That's 125 ns per IO, on a single CPU core, with two devices. Mind boggling.

Faster than DRAM from the 90's.

Modern SSDs generally have more bandwidth than DDR2.

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#26

Post by noir_lord » Mon, Oct 18, 2021, 8:06 PM UTC

tomc1985 wrote: post #24

I keep seeing Axboe as a typo'd version of Adobe and it's really bugging me edit- It's someone's name, I thought it was a company or product

It is a wlel kwnon pohnnomeen taht you can sawp all the ltteers in a wrod and as lnog as the fsrit and lsat is ccrreot it wlil sitll be cibemreplohnse.

https://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#27

Post by terafo » Mon, Oct 18, 2021, 8:09 PM UTC

He just hit 9M https://twitter.com/axboe/status/1450188650852065291

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#28

Post by kerneltrap » Mon, Oct 18, 2021, 8:32 PM UTC

egberts1 wrote: post #20

I look forward to a faster `cp` command.

The fastest `cp` is actually doing no data copies at all (relying on copy-on-write), on filesystems with reflink support. Incidentally, coreutils v9.0 cp switched to doing reflinks by default [1], so there's already a faster cp.

[1] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=2...

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#29

Post by junon » Mon, Oct 18, 2021, 8:43 PM UTC

pengaru wrote: post #16

junon wrote: post #7

I know this will be down voted but I've learned to be a bit skeptical of Axboe's monumental claims. There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories. Take these with a grain of salt - his results are notoriously very difficult to reproduce, even on identical hardware. Nevertheless, io_uring is certainly the better design and I'm…

> There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories. Could you provide any links to these discussions?

Sure. https://github.com/axboe/liburing/issues/189

Original claims were in the 90% and above performance increase over epoll. Then issues were found, and the figure was adjusted to 60% over epoll. Then more issues were found, and now real-world performance tests are showing minimal speedups if any.

Unfortunately the sibling commentors don't see "computer science" as a science but instead as a "feel good hobby", it seems. My point wasn't to hurt feelings, it was to provide a word of caution with these sorts of groundbreaking claims with respect specifically to the io_uring efforts, as they have been disingenuous quite a few times historically.

I don't doubt Jens does fantastic work. I don't doubt that he's seen these speedups in very specific cases. But people are celebrating this as a win where they'd be skeptical of e.g. "breakthrough" treatments of cancer (footnote: in mice). It's the same thing.

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#30

Post by junon » Mon, Oct 18, 2021, 8:46 PM UTC

yuffaduffa wrote: post #13

junon wrote: post #10

Earlier quoted context omitted.

This wasn't nitpicking. It was claiming a nontrivial speedup over epoll for identical test cases. What was the point of your comment?

What was the point of yours? “Take benchmarks with a grain of salt” is like saying refrigerating food is important or, more appropriately, that it’s important to verify surprising claims. (You made it a personal observation for some reason, but your whole point is still about as insightful as both of those.) The person you’re going after here probably felt compelled to counter the needless personal nature of your rem…

Comparing very verifiable, applied science to very theoretical science is a strawman. Science doesn't care about feelings or tenure, so while his experience is relevant, it does not excuse the provably inflated performance figures that have been boasted historically.