Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning containers from the bottom up

iximiuz.com

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Learning containers from the bottom up

#11

Post by jandrewrogers »

kodah wrote: post #2

This is a great article. I disagree with this: > Now, when you have a decent understanding of containers - from both the implementation and usage standpoints - it's time to tell you the truth. Containers aren't Linux processes! This is a bit of wordplay, I'm assuming, in absence of a word that defines the operating system features that power the concept of containers . To Linux, there is no (to my knowledge) concept…

The container runtime intercepts some syscalls, altering the observable behaviors of the kernel in ways that can adversely impact software that is otherwise perfectly designed to operate outside the container runtime. Normal processes don’t have their syscalls intercepted and this is material difference to the extent it is not transparent.

If running the same properly designed software exhibits material differences in behavior between a bare metal process and a containerized process, then they aren’t the same as a matter of practical semantics. Ironically, virtualized processes have much closer equivalence to a bare metal process than containerized processes in practice.

Saying a container is “just a process” is like saying a virtual machine is “just a process”, both are true in some sense depending on how you define “process”. But as a matter of practical engineering, they are different kinds of things.

Re: Learning containers from the bottom up

#12

Post by kodah »

jandrewrogers wrote: post #11
kodah wrote: post #2

This is a great article. I disagree with this: > Now, when you have a decent understanding of containers - from both the implementation and usage standpoints - it's time to tell you the truth. Containers aren't Linux processes! This is a bit of wordplay, I'm assuming, in absence of a word that defines the operating system features that power the concept of containers . To Linux, there is no (to my knowledge) concept…

The container runtime intercepts some syscalls, altering the observable behaviors of the kernel in ways that can adversely impact software that is otherwise perfectly designed to operate outside the container runtime. Normal processes don’t have their syscalls intercepted and this is material difference to the extent it is not transparent. If running the same properly designed software exhibits material differences i…

Are you talking about SecComp and namespacing?

Re: Learning containers from the bottom up

#13

Post by stevebmark »

kodah wrote: post #2

This is a great article. I disagree with this: > Now, when you have a decent understanding of containers - from both the implementation and usage standpoints - it's time to tell you the truth. Containers aren't Linux processes! This is a bit of wordplay, I'm assuming, in absence of a word that defines the operating system features that power the concept of containers . To Linux, there is no (to my knowledge) concept…

I think it’s important to understand that containers aren’t Linux processes. Containers can run more than one process. Containers can be stopped and restarted even though the initial process is gone forever. And containers have their own isolated writable layer.

Re: Learning containers from the bottom up

#16

Post by jandrewrogers »

kodah wrote: post #12
jandrewrogers wrote: post #11

Earlier quoted context omitted.

The container runtime intercepts some syscalls, altering the observable behaviors of the kernel in ways that can adversely impact software that is otherwise perfectly designed to operate outside the container runtime. Normal processes don’t have their syscalls intercepted and this is material difference to the extent it is not transparent. If running the same properly designed software exhibits material differences i…

Are you talking about SecComp and namespacing?

The root cause is likely SecComp. The notoriously poor I/O performance of containerized code, regardless of configuration, is largely a side effect of syscall interception.

In particular it breaks software that does I/O scheduling in user space, which is idiomatic and explicitly supported by the Linux kernel, even on virtual machines, but this use case conflicts with the container abstraction so runtimes offer an ersatz version that allows the code to run albeit poorly.

Re: Learning containers from the bottom up

#17

Post by yencabulator »

jandrewrogers wrote: post #16
kodah wrote: post #12

Earlier quoted context omitted.

Are you talking about SecComp and namespacing?

The root cause is likely SecComp. The notoriously poor I/O performance of containerized code, regardless of configuration, is largely a side effect of syscall interception. In particular it breaks software that does I/O scheduling in user space, which is idiomatic and explicitly supported by the Linux kernel, even on virtual machines, but this use case conflicts with the container abstraction so runtimes offer an ers…

More likely you're running your containers using overlay or some FUSE thing, and that's causing the I/O slowdown.

What syscalls do you think are intercepted, how? Speaking as someone who can write kernel code, I'm not aware of any such thing specific to containers. (As far as the linux kernel is concerned, there's no such thing as a container.)

If you're talking about BPF, that can be used outside of containers, e.g. systemd can limit any unit, and using it is not part of a definition of what a container is.

Re: Learning containers from the bottom up

#19

Post by jandrewrogers »

yencabulator wrote: post #17
jandrewrogers wrote: post #16

Earlier quoted context omitted.

The root cause is likely SecComp. The notoriously poor I/O performance of containerized code, regardless of configuration, is largely a side effect of syscall interception. In particular it breaks software that does I/O scheduling in user space, which is idiomatic and explicitly supported by the Linux kernel, even on virtual machines, but this use case conflicts with the container abstraction so runtimes offer an ers…

More likely you're running your containers using overlay or some FUSE thing, and that's causing the I/O slowdown. What syscalls do you think are intercepted, how? Speaking as someone who can write kernel code, I'm not aware of any such thing specific to containers . (As far as the linux kernel is concerned, there's no such thing as a container.) If you're talking about BPF, that can be used outside of containers, e.g…

I have decades of experience writing this type of low-level high-performance data infrastructure code directly against the Linux kernel, deployed in diverse environments. I’ve seen almost every rare edge case in practice.

You can find many examples in the wild of reputable software that loses significant performance once containerized no matter how configured. Literally no one has demonstrated state-of-the-art data infrastructure software that works around this phenomenon, and at this point you’d think someone would be able to if it was trivially possible. I test database kernels in a diverse set of environments and currently popular containers aren’t remotely competitive with VMs, never mind bare metal. The reasons for the performance loss are actually pretty well understood at a technical level, albeit esoteric.

Every popular container system has runtimes that intercept syscalls. Whether or not Linux requires containers to intercept syscalls is immaterial because in practice they all do in a manner destructive to I/O performance.

There used to be a similar phenomenon with virtual machines for many years, such that no one deployed databases on them. Then clever people learned how to trick the VM into letting them punch a hole through the hypervisor, and we’ve been using that trick ever since. It isn’t as fast as bare metal, but it is usually within 10%. No such trick exists for containers and as a consequence performance in containers is quite poor.

Re: Learning containers from the bottom up

#20

Post by kodah »

jandrewrogers wrote: post #19
yencabulator wrote: post #17

Earlier quoted context omitted.

More likely you're running your containers using overlay or some FUSE thing, and that's causing the I/O slowdown. What syscalls do you think are intercepted, how? Speaking as someone who can write kernel code, I'm not aware of any such thing specific to containers . (As far as the linux kernel is concerned, there's no such thing as a container.) If you're talking about BPF, that can be used outside of containers, e.g…

I have decades of experience writing this type of low-level high-performance data infrastructure code directly against the Linux kernel, deployed in diverse environments. I’ve seen almost every rare edge case in practice. You can find many examples in the wild of reputable software that loses significant performance once containerized no matter how configured. Literally no one has demonstrated state-of-the-art data i…

How can an unprivileged runtime intercept syscalls of an application talking directly to a kernel? I'll go browse through the containerd code to see if I can find such a thing because I know Go pretty well, but I have never heard of a runtime intercepting syscalls. That's why application kernels like gvisor exist.
Post reply on HN