Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Epoll vs. io_uring in Linux

sibexi.co

Post reply on HN
Print view View original

21–30 of 79 posts

Re: Epoll vs. io_uring in Linux

#21

Post by insanitybit »

csdreamer7 wrote: post #13
Uptrenda wrote: post #3

Yes, io_uring is significantly faster than epoll (I think I had like 20% faster req/s with io_uring.) The catch is that its kernel opt-in and disabled just about everywhere for security reasons. I think that it has direct memory sharing between the kernel and user-land which is kind of yikes. There's been multiple exploits that hit io_uring in recent times. It's because of this that even engineering projects that try…

RHEL 9 and 10 now fully support io_uring by default. It is very recent, but this covers a lot of corporate Linux installs. Gemini 'said' Ubuntu and SuSE support it as well, but did not provide any links to prove it. https://access.redhat.com/solutions/4723221 Go should reconsider support. They should have a 'go' at it.

It's still seccomp'd off in most environments because io-uring is still a seccomp bypass that doesn't play well with kernel security systems (audit subsystem), even if it weren't also like the #1 or #2 exploit vector for privesc.

Re: Epoll vs. io_uring in Linux

#22

Post by LoganDark »

mrlonglong wrote: post #2

Boost asio if you love C++ and asynchronous networking.

Boost is so inconvenient, they're huge dynamic libraries that are a pain to build and use. Even when I was already using CMake, getting Boost installed in a way where it could be discovered was super annoying. (I was on Mac, though)

Re: Epoll vs. io_uring in Linux

#23

Post by Cloudef »

Uptrenda wrote: post #3

Yes, io_uring is significantly faster than epoll (I think I had like 20% faster req/s with io_uring.) The catch is that its kernel opt-in and disabled just about everywhere for security reasons. I think that it has direct memory sharing between the kernel and user-land which is kind of yikes. There's been multiple exploits that hit io_uring in recent times. It's because of this that even engineering projects that try…

Quite depends, I had times when my posix emulation of io_uring (with poll, not epoll) was faster than io_uring. For large zero-copy buffers, io_uring is king however. Also io_uring is useful even for non asynchronous IO as it can implement chain of operations as single atomic operation (mkdir + open it for example).

For something like networking, if you are maximizing packets per second, you'll hit kernel limits[1] very quickly and instead have to start leveraging features like GSO/GRO or completely bypass the network stack.

1: https://github.com/axboe/liburing/discussions/1346

Re: Epoll vs. io_uring in Linux

#24

Post by toast0 »

MathMonkeyMan wrote: post #4
mrlonglong wrote: post #2

Boost asio if you love C++ and asynchronous networking.

I switched out asio's epoll backend for its io_uring in a database server and CPU utilization shot up. Probably depends on usage and the specifics of how it's integrated into the event code.

In addition to the other discussion. It's important to measure outcomes and not just look at the cpu meter...

At the same load, how did latency look for A vs B.

What was throughput and latency at maximum load like for A vs B. For whichever one had the smaller max throughput, what did latency look like for the other option.

For bonus points while testing: is there another observable metric to indicate available capacity, if cpu % free is less useful.

Re: Epoll vs. io_uring in Linux

#26

Post by lukeh »

Cloudef wrote: post #23
Uptrenda wrote: post #3

Yes, io_uring is significantly faster than epoll (I think I had like 20% faster req/s with io_uring.) The catch is that its kernel opt-in and disabled just about everywhere for security reasons. I think that it has direct memory sharing between the kernel and user-land which is kind of yikes. There's been multiple exploits that hit io_uring in recent times. It's because of this that even engineering projects that try…

Quite depends, I had times when my posix emulation of io_uring (with poll, not epoll) was faster than io_uring. For large zero-copy buffers, io_uring is king however. Also io_uring is useful even for non asynchronous IO as it can implement chain of operations as single atomic operation (mkdir + open it for example). For something like networking, if you are maximizing packets per second, you'll hit kernel limits[1] v…

Also it’s nice for things like SPI which have no user space non-blocking API.

Re: Epoll vs. io_uring in Linux

#28

Post by toast0 »

Sibexico wrote: post #18
toast0 wrote: post #11

> But my students weren’t as happy as I was - they wanted to build something genuinely useful, and they were really disappointed that our “product” had strong architectural limits and couldn’t outperform titans like nginx and haproxy. I took a (very brief) look at the github repo [1], it doesn't look like you're doing anything with cpu pinning. You can probably eke (thanks) out a bit more performance if you cpu pin y…

Basically, v0 and v1 of the repo is completely different implementations, written almost from scratch. Now working on the 3rd one implementation, I believe the last one. :) Completely different architectural choices was made.

If it's still running on more than a single core, and your students want it to go faster, aligning the work to cpus will almost certainly be useful.

I saw you mentioned windows development elsewhere. You might be interested to know that Microsoft pionered Receive Side Scaling and Send Side Scaling. If you try your proxy out on Windows, be sure to hook into those systems there.

The less work your proxy does, the more important avoiding cross core communication is.

Re: Epoll vs. io_uring in Linux

#29

Post by Asmod4n »

MathMonkeyMan wrote: post #4
mrlonglong wrote: post #2

Boost asio if you love C++ and asynchronous networking.

I switched out asio's epoll backend for its io_uring in a database server and CPU utilization shot up. Probably depends on usage and the specifics of how it's integrated into the event code.

No async io framework exists which utilizes everything io_uring can, they are all build around the poll model. As such io_uring will always be worse than the poll like abstractions.

The two things that make io_uring fast are chaining of operations and zero syscall mode, the former would require that all async io frameworks/libs would need to be rewritten to make use of that and then all user facing apps would also need to be rewritten since all you’d get now are completions to operations instead of waiting if you can run a operation.

Re: Epoll vs. io_uring in Linux

#30

Post by saghm »

vlovich123 wrote: post #20
saghm wrote: post #8

Earlier quoted context omitted.

Yeah, the explanation that I've usually heard for this sort of thing is that it's intended to get back CPU time that's lost when too many system threads are blocking to keep something on every core even during I/O (or pay for it in terms of the context switching overhead if you compensate for this with an extremely large number of system threads). The theory is that you'll avoid idle CPU compared to the common "one t…

That is an imprecise explanation being conveyed to you - thread per core is still valuable in an io_uring world. The reason for that is how computers are built - its inherent in the kinds of operations that are cheap and what happens otherwise, the I/O model doesn’t matter. Specifically, not thread per core code has the following issues: * you have to use atomics/locks to synchronize data access. This involves expens…

To clarify, I'm not talking specifically about io_uring but (multithreaded) async concurrency models in general. The explanation needs to be imprecise because not all of them work the same way, so it's impossible to say anything correct about all of them without intentionally leaving out some specifics.
Post reply on HN