The Case for a High-Level Kernel-Bypass I/O Abstraction
irenezhang.net
The Case for a High-Level Kernel-Bypass I/O Abstraction
1–10 of 27 posts
Re: The Case for a High-Level Kernel-Bypass I/O Abstraction
#2What does it even mean by high level?
Once the API is high level enough it gets unusable by major users who are high end networking, GPU and graphics libraries and low latency sound.
Nobody else truly needs to bypass the kernel. Even low latency can work with good real time task handling, making the users exactly two cases, who have special DMA handling in hardware already. If it means introducing special case kernel bypasses for high scale computing, it's already done, and the low level APIs just get wrapped.
And the Achilles's foot is security.
If it's arguing for making all hardware a kernel free fabric, it's essentially a move of everything to firmware. Worst case, we get zero memory protection and unfixable bugs.
Re: The Case for a High-Level Kernel-Bypass I/O Abstraction
#3What about UIO?
Re: The Case for a High-Level Kernel-Bypass I/O Abstraction
#4> we found that 30% of the cost of the Linux kernel comes from its interface. This overhead is just too much to carry around while using kernel-bypass devices.
One third of the cost is actually expensive !
Also, ScyllaDB NoSQL database(C++ clone of Cassandra) uses Seastar framework to achieve high IO throughput.
Re: The Case for a High-Level Kernel-Bypass I/O Abstraction
#5What does it even mean by high level? Once the API is high level enough it gets unusable by major users who are high end networking, GPU and graphics libraries and low latency sound. Nobody else truly needs to bypass the kernel. Even low latency can work with good real time task handling, making the users exactly two cases, who have special DMA handling in hardware already. If it means introducing special case kernel…
Well, that is the whole point of unikernels.
Memory safe languages with rich runtimes, only need a mini kernel to run bare metal.
Windows has been pushing for user space drivers for a while now, including GPUs.
Android is following the same path with Project Treble, and who knows what will happen with Fuchsia.
Likewise on many high integrity OSes for embedded deployment.
Re: The Case for a High-Level Kernel-Bypass I/O Abstraction
#6What does it even mean by high level? Once the API is high level enough it gets unusable by major users who are high end networking, GPU and graphics libraries and low latency sound. Nobody else truly needs to bypass the kernel. Even low latency can work with good real time task handling, making the users exactly two cases, who have special DMA handling in hardware already. If it means introducing special case kernel…
High-level means not exposing hardware limitations to the application. The primary target applications are datacenter services, which spend much of their time processing network I/O. As network latencies lower to a few microseconds, datacenter applications like Redis will need kernel-bypass because the kernel will become too expensive for them. In our experiments with a 25Gb network, the Linux kernel and POSIX interface costs Redis 60% of its latency.
Re: The Case for a High-Level Kernel-Bypass I/O Abstraction
#7> we found that 30% of the cost of the Linux kernel comes from its interface. This overhead is just too much to carry around while using kernel-bypass devices. One third of the cost is actually expensive ! Also, ScyllaDB NoSQL database(C++ clone of Cassandra) uses Seastar framework to achieve high IO throughput. http://seastar.io/networking/
I've updated the blog post with our experimental results from the Redis benchmark. Here is a link to the graph: http://irenezhang.net/img/demikernel-redis-exp.jpg
Re: The Case for a High-Level Kernel-Bypass I/O Abstraction
#8What about UIO?
It depends on the interface for the drivers to the application. However, UIO doesn't seem to support DMA, which is a non-starter.
RDMA and DPDK both use user-space drivers, which is necessary for kernel-bypass. I'm not advocating for a particular kernel-bypass solution. I'm arguing that if we use kernel-bypass for I/O, we should have a common, efficient, high-level interface for it.
Re: The Case for a High-Level Kernel-Bypass I/O Abstraction
#9BTW, the Demikernel will be open-sourced shortly .. as soon as I return from giving a talk at KubeCon Europe.
Re: The Case for a High-Level Kernel-Bypass I/O Abstraction
#10What does it even mean by high level? Once the API is high level enough it gets unusable by major users who are high end networking, GPU and graphics libraries and low latency sound. Nobody else truly needs to bypass the kernel. Even low latency can work with good real time task handling, making the users exactly two cases, who have special DMA handling in hardware already. If it means introducing special case kernel…
High-level means not exposing hardware limitations to the application. The primary target applications are datacenter services, which spend much of their time processing network I/O. As network latencies lower to a few microseconds, datacenter applications like Redis will need kernel-bypass because the kernel will become too expensive for them. In our experiments with a 25Gb network, the Linux kernel and POSIX interf…
So what's the API look like?
You said you don't want to make users deal with flow control and hardware details.. does that imply a userspace bypass library which does that stuff for us? Does it look posixy?