Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An io_uring-based user-space block driver

lwn.net

Post reply on HN
Print view View original

21–30 of 43 posts

Re: An io_uring-based user-space block driver

#21

Post by Joker_vD »

jmillikin wrote: post #10
Joker_vD wrote: post #8

That reminds me quite strongly of VirtIO (block) devices... and yet the actual command format is, of course, different. Why can't we stop re-inveting things over and over?

Because different use cases require different designs? If you try to create a protocol that can work for all purposes, it'll be a poor fit for any of them and will be out-competed by more specialized alternatives. There's a reason emulators design their virtual devices to resemble real hardware (PCI, SCSI, USB) -- there's already going to be a bunch of code in the hypervisor to create fake hardware. It's also more pr…

The basis of virtio is literally just a ring queue, with its request descriptors looking almost exactly like structs suitable for passing to readv(2) or writev(2); the PCI shim is built on top of that and is completely optional (you can have a purely MMIO virtio device, after all). It was built this way so that KVM would not have to mimic the idiosyncrasies of real hardware: passing data as-is to the physical devices can't work for obvious reasons (even if you disregard security completely); instead in can, after minimal processing, shove it into the Linux kernel and let it take care of the rest.

Re: An io_uring-based user-space block driver

#22

Post by loeg »

jmillikin wrote: post #20
loeg wrote: post #18

Earlier quoted context omitted.

> Block devices operate on blocks of data identified by offset. Sure, although CUSE read and write operations take offsets, too. The kernel could just send block-sized IOs to a CUSE driver and it wouldn't be all that different. > You can in principle implement a block device-ish API in FUSE by disabling open/close and requiring all reads/writes to be at given offsets Right, ok. I think the historical distinction betw…

There may be kernels that have simplified their device model to unify character and block devices, but Linux has not. FUSE/CUSE (and now ublk) are Linux-oriented protocols from the beginning, with relatively little thought given to cross-platform compatibility. If you use FreeBSD then you're likely familiar with the challenges they've faced adapting FUSE to their VFS, and last time I checked they don't have plans to…

FreeBSD does have CUSE, for what it’s worth.

Re: An io_uring-based user-space block driver

#23

Post by jmillikin »

trasz wrote: post #19
notacoward wrote: post #15

Earlier quoted context omitted.

Not proprietary, but not iSCSI-specific either. The whole idea is that you can use any protocol you like. Could be iSCSI, could be NBD, could be AoE, could be something proprietary but that's less likely than open/standard alternatives.

It’s obviously proprietary: it’s non-standard and specific to a single vendor. What is the whole idea, though? Serving things to kernel from userland is decades old and commonly used with both NFS and iSCSI. The fact that this particular implementation uses io_uring instead of something non-proprietary like RDMA, is just an implementation detail.

  > It’s obviously proprietary: it’s non-standard and specific to a
  > single vendor.
That's not how people typically use "proprietary" when referring to open-source code developed collectively by multiple vendors, universities, and thousands of independent contributors.

  > What is the whole idea, though? [...] this particular implementation
  > uses io_uring instead of something non-proprietary like RDMA, is just
  > an implementation detail.
When performance matters, sometimes implementation details are the whole idea.

According to the patch's author at https://lwn.net/Articles/904638/>, ublk has about twice the throughput of NBD.

Re: An io_uring-based user-space block driver

#24

Post by notacoward »

trasz wrote: post #19
notacoward wrote: post #15

Earlier quoted context omitted.

Not proprietary, but not iSCSI-specific either. The whole idea is that you can use any protocol you like. Could be iSCSI, could be NBD, could be AoE, could be something proprietary but that's less likely than open/standard alternatives.

It’s obviously proprietary: it’s non-standard and specific to a single vendor. What is the whole idea, though? Serving things to kernel from userland is decades old and commonly used with both NFS and iSCSI. The fact that this particular implementation uses io_uring instead of something non-proprietary like RDMA, is just an implementation detail.

> It’s obviously proprietary: it’s non-standard and specific to a single vendor.

I just re-read the LWN article and there's no suggestion of any such thing. Where are you seeing it?

> Serving things to kernel from userland is decades old and commonly used with both NFS and iSCSI.

...and regular FC/SCSI too. I worked on that for four years. Yawn. What's your point?

> The fact that this particular implementation uses io_uring instead of something non-proprietary like RDMA

RDMA implementations are even more likely to be bound up with proprietary bits than vanilla-TCP implementations. RDMA over IB or other even lesser known interconnects (both open/standard and proprietary) existed long before ROCE. Do you even know what "open" vs. "proprietary" mean? There were open network-disk protocols before iSCSI. I worked on such as early as 1989, and developed my own (along with my own user-space block device for Linux and Windows) in 2000. It seems like you've only been exposed to a small set of open/standards based technologies, and deride anything else as "proprietary" even though that's far from accurate. Is there some undisclosed interest at play here?

> is just an implementation detail

It's a very important detail, considering the performance difference.

Re: An io_uring-based user-space block driver

#25

Post by benlwalker »

jmillikin wrote: post #7

I wonder if this could replace most uses of NBD (network block devices), and/or help get iSCSI into userspace where more flexible load-balancing policy can be implemented. It also reminds me of attempts to define BUSE[0][1][2], which would have been a block device equivalent of FUSE. IIRC attempts to get BUSE into the Linux kernel have been blocked for performance reasons -- the FUSE protocol isn't well designed and…

The SPDK project is certainly looking to use this to replace our limited use of NBD, as well as present SPDK block devices as kernel block devices, including devices backed by userspace implementations of iSCSI, NVMe-oF, and various other network protocols.

Re: An io_uring-based user-space block driver

#26

Post by benlwalker »

For me, the killer use case for this is presenting logical volumes to containers. There just has not been an efficient mechanism for a local storage service in one container to serve logical volumes to another container on the same system until this. For VMs there is virtio/vfio-user, but for containers the highest performing option until this was NVMe-oF/TCP loopback.

Basically, you can implement a virtual SAN for containers efficiently with this.

Re: An io_uring-based user-space block driver

#27

Post by jmillikin »

loeg wrote: post #22
jmillikin wrote: post #20

Earlier quoted context omitted.

There may be kernels that have simplified their device model to unify character and block devices, but Linux has not. FUSE/CUSE (and now ublk) are Linux-oriented protocols from the beginning, with relatively little thought given to cross-platform compatibility. If you use FreeBSD then you're likely familiar with the challenges they've faced adapting FUSE to their VFS, and last time I checked they don't have plans to…

FreeBSD does have CUSE, for what it’s worth.

FreeBSD has a /dev/cuse device, and a libcuse reimplemented on top of it, but it uses a different protocol from Linux's CUSE. You can see the FreeBSD implementation at https://github.com/freebsd/freebsd-src/blob/release/13.1.0/s... -- note how cuse_server_read() and cuse_server_write() are stubs.

I am somewhat familiar with this because I wrote a FUSE/CUSE server library in Rust, and tried porting it to FreeBSD. The FUSE bits worked with only minor issues[0][1], but the CUSE bits were completely different so I had to turn off that part of the library for FreeBSD targets.

[0] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253411

[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253500

Re: An io_uring-based user-space block driver

#28

Post by lifty »

benlwalker wrote: post #26

For me, the killer use case for this is presenting logical volumes to containers. There just has not been an efficient mechanism for a local storage service in one container to serve logical volumes to another container on the same system until this. For VMs there is virtio/vfio-user, but for containers the highest performing option until this was NVMe-oF/TCP loopback. Basically, you can implement a virtual SAN for c…

Very appealing. Do you think this solution would be comparable in performance with an in-kernel storage driver?

Re: An io_uring-based user-space block driver

#29

Post by touisteur »

tptacek wrote: post #17
sanxiyn wrote: post #6

Earlier quoted context omitted.

There is XDP.

XDP is kind of the opposite of this, right? It's moving userland code into the kernel.

XDP is a lot of stuff, but I think I have someone around using af_xdp to bypass the kernel network stack and for some (and the filtering and decision of which streams, is done through some ebpf iirc) packets deliver them directly into userland buffer-queues? DPDK also has an AF_XDP backend to bridge your classical DPDK app and AF_XDP sockets.

Re: An io_uring-based user-space block driver

#30

Post by tptacek »

touisteur wrote: post #29
tptacek wrote: post #17

Earlier quoted context omitted.

XDP is kind of the opposite of this, right? It's moving userland code into the kernel.

XDP is a lot of stuff, but I think I have someone around using af_xdp to bypass the kernel network stack and for some (and the filtering and decision of which streams, is done through some ebpf iirc) packets deliver them directly into userland buffer-queues? DPDK also has an AF_XDP backend to bridge your classical DPDK app and AF_XDP sockets.

Ah, that's true. AF_XDP is definitely similar to userland block device offload.
Post reply on HN