Making containers safer
lwn.net
Making containers safer
1–10 of 52 posts
Re: Making containers safer
#2Note to anyone confused: The docker concept of '--privileged' is separate from what the LXD folks are refering to as 'privileged containers'. The LXD folks are talking about mapping UID 0 into the container, whereas (IIRC) the docker flag disables dropping capabilities and the seccomp syscall filters (and maybe some other things? I can't remember off the top of my head).
The equivalent docker functionality is userns-remap or sometimes just "user namespaces".
Re: Making containers safer
#3This was a pretty good read. I use containers quite a lot on my server at home at maintain a bunch of utilities. Mostly I'm using systemd-nspawn.
> User namespaces have been around since the 3.12 kernel, but few other container management systems use the feature to isolate their containers. Part of the reason for that is the difficulty in sharing files between containers because of the UID mapping. LXD is currently using shiftfs on Ubuntu systems to translate UIDs between containers and the parts of the host filesystem that are shared with it. Shiftfs is not upstream, however; there are hopes to add similar functionality to the new mount API before long.
This is exactly the problem I've run into. If you're trying to share files between containers or between the host and a container (e.g. with systemd-nspawn's "--bind" option), it's much harder to have your permissions set properly and still access them from in the container if you're using user namespacing.
There's also the issue of creating the container in the first place. If you follow the instructions for creating a container on the Arch Wiki[1], the files will end up owned by the host's root (mostly), which is a problem when you then try to boot the container as a different (namespaced) user. I don't know of a straightforward way to create a namespaced container with systemd-nspawn, and I don't think there's any way to convert an existing container to a namespaced container.
Yet another problem that sometimes arises is that various distributions have user namespacing disabled (it has to be enabled when the kernel is built) - notably Arch [2], though this may have recently changed. This is apparently due to concerns that the namespacing code is buggy and can itself lead to privilege escalation vulnerabilities.
[1] https://wiki.archlinux.org/index.php/Systemd-nspawn#Examples
Re: Making containers safer
#4This was a pretty good read. I use containers quite a lot on my server at home at maintain a bunch of utilities. Mostly I'm using systemd-nspawn. > User namespaces have been around since the 3.12 kernel, but few other container management systems use the feature to isolate their containers. Part of the reason for that is the difficulty in sharing files between containers because of the UID mapping. LXD is currently u…
While it still needs work for performance and other things, Podman already supports rootless containers using user namespaces. It's actually pretty easy to setup too, especially on Arch:
https://github.com/containers/libpod/blob/master/rootless.md
Re: Making containers safer
#5The whole container safety story has been a mess/cluster/bag-of-tricks since the very beginning. Unlike BSD jails, security was never number one priority for containers. Just take a look at what GCE/AWS use respectively. The former built an entire syscall proxy with Gvisor while the latter uses a hypervisor based solution with Firecracker (think mini-VMs). Anything in production that touches foriegn code can't really use generic out-of-the-box container systems like Docker. There is a whole bunch of tech such as Hyper Container, Kata, Runq etc. all so that you can run them on a hypervisor and get proper sandboxing. All in all quite disappointing to be honest.
(The main downside of hypervisors is that they are difficult to run on low-cost commodity cloud like Digital Ocean. You are forced to use bare metal stuff like Hetzner, AWS, Paket Cloud etc. The whole point of cloud is that the hardware/infrastructure is mostly abstracted away. If I have to run my own hypervisor just to ensure the container doesn't get broken out of, what's the point even calling it cloud?)
Re: Making containers safer
#6I have been a user of LXC and LXD since 2013. Earlier when docker was based on LXC I tried it. Then Docker went its own path and build its own libcontainer library. During that time LXC project added support for unprivileged container and since than I didn't use docker. Still today when majority of container runtime including kubernetes is based on docker container (some on OCI), I continue to use LXD and LXC.
LXD by default has been more secure given it allowed unprivileged containers very early that works very nicely. Recently when there was a security problem with Kubernetes, I was still ok given we only used unprivileged containers.
I love LXD container being lightweight compared to kubernetes and same ansible or other platform orchestration like puppet, chef can work with baremetla, VM and containers and no need to fiddle with shell scripts, Dockerfile and learning container orchestration specific domain specific language (DSL). Hopefully LXD gets more popular.
So far OpenStack, OpenNebula and Proxmox support native LXD containers besides KVM and other virtualization. For most of the small website with thousands or million users LXD itself can work pretty well without relying on any cloud orchestration platform.
Re: Making containers safer
#7I used containers in production since 2002. We knew back then that containers are escapable and nothing changed since. Containers are for ease of management - package and configuration sepparation. One role per container. In absolutely no case are they meant for multi tenant cases.
Containers are fine the way they are. Just use them for the right job.
Re: Making containers safer
#8The whole container safety story has been a mess/cluster /bag-of-tricks since the very beginning. Unlike BSD jails, security was never number one priority for containers. Just take a look at what GCE/AWS use respectively. The former built an entire syscall proxy with Gvisor while the latter uses a hypervisor based solution with Firecracker (think mini-VMs). Anything in production that touches foriegn code can't reall…
> (The main downside of hypervisors is that they are difficult to run on low-cost commodity cloud like Digital Ocean. You are forced to use bare metal stuff like Hetzner, AWS, Paket Cloud etc. The whole point of cloud is that the hardware/infrastructure is mostly abstracted away. If I have to run my own hypervisor just to ensure the container doesn't get broken out of, what's the point even calling it cloud?)
Sorry if I'm misunderstanding, but the hypervisor in the cloud (i.e. the VM running under your "instance") is to protect the cloud provider from you. The Hypervisor you're running (firecracker, kata containers, gvisor, nabla, etc) are to protect your dangerous workloads from your other workloads.
It is totally within your power to choose to run 1 cloud "instance" per workload so you don't have to worry about cross-workload contamination, but if you want to effectively multiplex your workload on constantly available machines, you're going to have to mix your workloads, and that almost certainly requires protection of some kind.
In an ideal world we might all be getting VMs that were unikernels that are a bit safer to use but in general it feels like you need to isolate these workloads somehow if you're going to run them on the same "instance". What does your perfect/preferred world look like?
Re: Making containers safer
#9LXC+LXD is one of the most undervalued container technologies out there. It can do a lot of cool things (like live migration via criu), and is IMO more production ready than Docker ever was.
For those wondering about the distinction between "system" containers and "app" containers, the difference is user namespacing -- I think we should stop using the naming difference but instead go with specifying that the containers "full user name spacing". User namespacing is the "magic sauce" for projects like Podman and a related project Buildah which builds images without root privileges -- the combination of user namespaces and FUSE.
BTW for those wondering if container orchestrators like Kubernetes will be shaken by this -- they very likely won't. Kubernetes has sidestepped this problem by introducing and relying on RuntimeClass[0], which allows you to swap out the runtime that's running underneath kubernetes. In fact, there is already a shim being worked on[1].
[EDIT] - While I'm here talking about container tech, please check out containerd[2]. It's been my runtime of choice for a very long time, has been very well maintained, and received many of the really advanced features first (ex. untrusted runtimes). IMO it should be the default container runtime of k8s.
[0]: https://kubernetes.io/docs/concepts/containers/runtime-class...
Re: Making containers safer
#10I have been a user of LXC and LXD since 2013. Earlier when docker was based on LXC I tried it. Then Docker went its own path and build its own libcontainer library. During that time LXC project added support for unprivileged container and since than I didn't use docker. Still today when majority of container runtime including kubernetes is based on docker container (some on OCI), I continue to use LXD and LXC. LXD by…
How secure do you feel LXD containers are, given the defaults? No worse than any bare-metal? No worse than any Qemu/KVM VM?
I do use LXD containers too, but mainly to create a bunch of testing nodes.