Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

phoronix.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#11

Post by marcodiego »

junon wrote: post #7

I know this will be down voted but I've learned to be a bit skeptical of Axboe's monumental claims. There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories. Take these with a grain of salt - his results are notoriously very difficult to reproduce, even on identical hardware. Nevertheless, io_uring is certainly the better design and I'm…

Not disagreeing with you, but even if benchmarks are unfair, these reports are still a good illustration of progress. Unless there are some drawbacks to these optimizations, there should be no negative side-effects.

AFAIK as I know, these changes shouldn't negatively impact tasks that are not making any use of these features. So, even if improvements are not directly proportional to what is being reported, they're still real.

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#12

Post by devit »

Aissen wrote: post #3

That's 125 ns per IO, on a single CPU core, with two devices. Mind boggling.

At around 4 GHz and 2 IPC, it's about 1000 CPU instructions per I/O.

Assuming you have an hardware DMA ring buffer for I/O that is directly mapped in userspace, the only thing that is really needed is to write the operation type, size, disk position and memory position to the ring buffer, update the buffer position and check for flush, doable in around 8 CISC instructions (plus the slowpath), so around 100x inefficient.

Without the direct mapped ring buffer and with a filesystem, you need a kernel to translate from uring to the hardware ring buffer, and here it still seems around 10x inefficient as around 100 instructions should be enough to do the translation (assuming pages already mapped in the IOMMU, that you have the file block map in cache, and that the whole system is architected to maximize the efficiency of this operation).

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#13

Post by yuffaduffa »

junon wrote: post #10
Datagenerator wrote: post #8

Earlier quoted context omitted.

Nitpicking aside he has twenty years of Linux kernel development experience and does great things for everyones benefit. Congratulations Jens Axboe!

This wasn't nitpicking. It was claiming a nontrivial speedup over epoll for identical test cases. What was the point of your comment?

What was the point of yours? “Take benchmarks with a grain of salt” is like saying refrigerating food is important or, more appropriately, that it’s important to verify surprising claims. (You made it a personal observation for some reason, but your whole point is still about as insightful as both of those.)

The person you’re going after here probably felt compelled to counter the needless personal nature of your remarks. It’s difficult to experimentally verify relativity but we don’t criticize Einstein as a result.

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#14

Post by anarazel »

Aissen wrote: post #3

That's 125 ns per IO, on a single CPU core, with two devices. Mind boggling.

It's worth noting that there's some batching involved, so it's not 125ns for processing one io, then again 125ns for the next. Both combining some of the work, and superscalar execution is necessary to reach this high numbers...

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#16

Post by pengaru »

junon wrote: post #7

I know this will be down voted but I've learned to be a bit skeptical of Axboe's monumental claims. There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories. Take these with a grain of salt - his results are notoriously very difficult to reproduce, even on identical hardware. Nevertheless, io_uring is certainly the better design and I'm…

> There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories.

Could you provide any links to these discussions?

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#17

Post by benlwalker »

anarazel wrote: post #14
Aissen wrote: post #3

That's 125 ns per IO, on a single CPU core, with two devices. Mind boggling.

It's worth noting that there's some batching involved, so it's not 125ns for processing one io, then again 125ns for the next. Both combining some of the work, and superscalar execution is necessary to reach this high numbers...

To expand on the batching details, I haven't seen exactly what he's doing here, but historically the numbers quoted are from benchmarks that submit something like 128 total queue depth in alternating batches of 64. The disks hit max performance with a much lower queue depth than 64. So it's very unrealistic perfect batching that you'll never see in a real application. The workload is only a useful tool for optimizing the code, which is exactly what he's using it for.

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#18

Post by wtallis »

reacharavindh wrote: post #15

Does anyone know whether these optimisations in block layer and iouring benefit major IO hitters like PostgreSQL, ZFS, NFS writes etc? Will it take several years and monumental effort before they adopt iouring first?

io_uring is an improved interface between userspace and the kernel, so it doesn't provide any direct benefits to in-kernel filesystem operations, and the performance benefits io_uring does provide should be largely filesystem-agnostic. That said, some of these recent optimizations may be low enough in the kernel's io stack to also benefit io originating within the kernel itself.

Userspace applications that already have some support for asynchronous disk IO (either through the old libaio APIs or as a cleanly-abstracted thread pool) should be able to switch to using io_uring as their backend without too much trouble, and reap the benefits of async that actually works reliably (if switching from libaio) and with vastly lower overhead (if switching from a thread pool). Databases like PostgreSQL were some of the few applications that attempted to deal with the limitations of libaio, but I'm not sure how close they are to having a production-quality io_uring backend.

Re: Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

#19

Post by wtallis »

benlwalker wrote: post #17
anarazel wrote: post #14

Earlier quoted context omitted.

It's worth noting that there's some batching involved, so it's not 125ns for processing one io, then again 125ns for the next. Both combining some of the work, and superscalar execution is necessary to reach this high numbers...

To expand on the batching details, I haven't seen exactly what he's doing here, but historically the numbers quoted are from benchmarks that submit something like 128 total queue depth in alternating batches of 64. The disks hit max performance with a much lower queue depth than 64. So it's very unrealistic perfect batching that you'll never see in a real application. The workload is only a useful tool for optimizing…

The Intel Optane drives he's testing with hit max performance at pretty low queue depths, but flash-based SSDs with similar throughput will require those high queue depths.
Post reply on HN