An io_uring-based user-space block driver • Hacker News Board

Re: An io_uring-based user-space block driver

#41

Post by wmf » Thu, Aug 18, 2022, 8:50 PM UTC

bullen wrote: post #40

dundarious wrote: post #12

Earlier quoted context omitted.

If I'm understanding ublk and your question correctly, then yes, there are a lot of kernel-bypass networking options out there, such as openonload, dpdk, mellanox (though they seem to have been absorbed into nvidia). You'll likely need a special/particular network card, an external kernel module, and at least an LD_PRELOAD to use them though.

Is there no way to avoid kernel copying all network data? I understand the frustration of having the network driver crash but could it not be run in a way that it doesn't bring down the OS? It seems to me Java would have a no-brainer advantage of a user-space networking option since you're already in a VM!? When I saturate my HTTP server the kernel takes 30% of the CPU just copying data for no good reason?!

It's happening. https://lwn.net/Articles/879724/

Re: An io_uring-based user-space block driver

#42

Post by rwmj » Thu, Aug 18, 2022, 8:58 PM UTC

jmillikin wrote: post #7

I wonder if this could replace most uses of NBD (network block devices), and/or help get iSCSI into userspace where more flexible load-balancing policy can be implemented. It also reminds me of attempts to define BUSE[0][1][2], which would have been a block device equivalent of FUSE. IIRC attempts to get BUSE into the Linux kernel have been blocked for performance reasons -- the FUSE protocol isn't well designed and…

I'd be careful about giving up the network capabilities. With NBD it's very useful to move the client and server apart, either having them both run in userspace on the same machine over a Unix domain socket or talking remotely over the network. For our case this is by far the most common use of NBD, we hardly use nbd.ko at all.

Re: An io_uring-based user-space block driver

#43

Post by dundarious » Fri, Aug 19, 2022, 12:11 AM UTC

wmf wrote: post #41

bullen wrote: post #40

Earlier quoted context omitted.

Is there no way to avoid kernel copying all network data? I understand the frustration of having the network driver crash but could it not be run in a way that it doesn't bring down the OS? It seems to me Java would have a no-brainer advantage of a user-space networking option since you're already in a VM!? When I saturate my HTTP server the kernel takes 30% of the CPU just copying data for no good reason?!

It's happening. https://lwn.net/Articles/879724/

Yes, and also just to note, zero-copy and kernel-bypass are independent. Traditional Berkeley socket syscalls are copy+kernel, io_uring has/will have zerocopy+kernel, openonload provides both APIs for copy+kernelbypass and zerocopy+kernelbypass.