Learning containers from the bottom up • Hacker News Board

Re: Learning containers from the bottom up

#21

Post by yencabulator » Sat, Nov 27, 2021, 12:50 AM UTC

jandrewrogers wrote: post #19

yencabulator wrote: post #17

Earlier quoted context omitted.

More likely you're running your containers using overlay or some FUSE thing, and that's causing the I/O slowdown. What syscalls do you think are intercepted, how? Speaking as someone who can write kernel code, I'm not aware of any such thing specific to containers . (As far as the linux kernel is concerned, there's no such thing as a container.) If you're talking about BPF, that can be used outside of containers, e.g…

I have decades of experience writing this type of low-level high-performance data infrastructure code directly against the Linux kernel, deployed in diverse environments. I’ve seen almost every rare edge case in practice. You can find many examples in the wild of reputable software that loses significant performance once containerized no matter how configured. Literally no one has demonstrated state-of-the-art data i…

Lots of words, very little said. How do you think e.g. docker "intercepts syscalls"?

Re: Learning containers from the bottom up

#22

Post by jandrewrogers » Sat, Nov 27, 2021, 8:34 AM UTC

yencabulator wrote: post #21

jandrewrogers wrote: post #19

Earlier quoted context omitted.

I have decades of experience writing this type of low-level high-performance data infrastructure code directly against the Linux kernel, deployed in diverse environments. I’ve seen almost every rare edge case in practice. You can find many examples in the wild of reputable software that loses significant performance once containerized no matter how configured. Literally no one has demonstrated state-of-the-art data i…

Lots of words, very little said. How do you think e.g. docker "intercepts syscalls"?

No idea, don’t really care. That these performance disparities exist isn’t controversial, several well-known companies have written about it e.g. nginx[0]. I deploy on Kubernetes, but all of our I/O intensive infrastructure is deployed on virtual machines for performance reasons — we still lose ~10% on a virtual machine compared to bare metal but a container on bare metal is significantly worse.

All high-performance data infrastructure bypasses the operating system for most things, taking total control of the physical hardware it uses. Cores, memory, storage, network. Linux happily grants that control, with caveats. It doesn’t work the same inside Kubernetes and Docker, and no one can figure out how to turn it off. Maybe you don’t work on I/O intensive applications, but for people that do this is a well-documented phenomenon. My teams have wasted far too many hours trying to coax good performance out of code in containers that works just fine on bare metal and virtual machines.

[0] https://www.nginx.com/blog/comparing-nginx-performance-bare-...

Re: Learning containers from the bottom up

#23

Post by rajmann » Sun, Nov 28, 2021, 2:06 AM UTC

jandrewrogers wrote: post #19

yencabulator wrote: post #17

Earlier quoted context omitted.

More likely you're running your containers using overlay or some FUSE thing, and that's causing the I/O slowdown. What syscalls do you think are intercepted, how? Speaking as someone who can write kernel code, I'm not aware of any such thing specific to containers . (As far as the linux kernel is concerned, there's no such thing as a container.) If you're talking about BPF, that can be used outside of containers, e.g…

I have decades of experience writing this type of low-level high-performance data infrastructure code directly against the Linux kernel, deployed in diverse environments. I’ve seen almost every rare edge case in practice. You can find many examples in the wild of reputable software that loses significant performance once containerized no matter how configured. Literally no one has demonstrated state-of-the-art data i…

>Then clever people learned how to trick the VM into letting them punch a hole through the hypervisor, and we’ve been using that trick ever since

Can you say more on this trick. Is it available by default on the cloud VMs, or is it something that has to done in the user's code. I have seen tweets from DirectIO developers saying AWS Nitro machines are better for kernel-bypass-IO compared the Google cloud VMs. but my understanding of it was something done by Amazon Nitro card engineers/developers, and Google was working on something similar to improve the performance.