summary: containers have no security story without SELinux.
Making containers safer
11–20 of 52 posts
Re: Making containers safer
#12I have been a user of LXC and LXD since 2013. Earlier when docker was based on LXC I tried it. Then Docker went its own path and build its own libcontainer library. During that time LXC project added support for unprivileged container and since than I didn't use docker. Still today when majority of container runtime including kubernetes is based on docker container (some on OCI), I continue to use LXD and LXC. LXD by…
How secure do you feel LXD containers are, given the defaults? No worse than any bare-metal? No worse than any Qemu/KVM VM? I do use LXD containers too, but mainly to create a bunch of testing nodes.
Ok to answer your question, Qemu/KVM is more secure than lxd as they run kennel code for each VM. Here on containers there are just two choices use Dockerfile with Docker style containers or use lxd with.lxc. There are kata containers, but not as user friendly as LXD. Most of other container runtime run as privileged root user. In case of LXD, each of the container runs is userspace. So your security is like managing multi-user Linux. We understand management of multi-user Linux very well compared to other esoteric schemes. So I feel lxd offer better security than docker style containers. This is one of the reason most of the big cloud providers like gcp, aws, azure do not offer bare metal containers. Most run on top of their VM which are more secure in multi-tenant systems.
LXD we use for production and they are very lightweight. Running hundreds or thousands of them on a single baremetal will be ok.
Re: Making containers safer
#13The more I use Linux Containers (big fan of LXD) the more I'm convinced Solaris got it right with Zones, Crossbow, ZFS, SMF also for Zone services - Linux is still not there but instead we have lot's of vendor glue-code in go and many asteriks what is not possible. IMHO Linux should implement something like the zone concept that conceals cgroups, network, mount-namespaces - quota is still broken in btrfs, you can't delegate ZFS commands into a ZFS LXD-Container (where quota actually works) and if you attempt to run something like Docker Swarm in LXD you realize only parts of the kernel are network-namespace aware - it probably will converge to either some vendor solution or it will be resolved in the kernel (I doubt the ZFS integration will happen, as it looks like the current devs are activly fighting ZFS) but conceptionally the concept of a zone as a security boundary feels more sane to me than the glued-together mess in ther kernel at the moment. With kernel zones there was even a concept for stonger security.
Re: Making containers safer
#14I used containers in production since 2002. We knew back then that containers are escapable and nothing changed since. Containers are for ease of management - package and configuration sepparation. One role per container. In absolutely no case are they meant for multi tenant cases. Containers are fine the way they are. Just use them for the right job.
So instead of trying to improve their security and keep the startup/overhead advantage over VMs in the orders of magnitude, you simply suggest doing nothing?
FWIW, I like it that both camps are working on their shortcomings: LXD/container camp on isolation, and the VM camp on overhead (eg. firecracker).
Re: Making containers safer
#15LXC+LXD is one of the most undervalued container technologies out there. It can do a lot of cool things (like live migration via criu), and is IMO more production ready than Docker ever was. For those wondering about the distinction between "system" containers and "app" containers, the difference is user namespacing -- I think we should stop using the naming difference but instead go with specifying that the containe…
I'm curious as to how you get your images built. I'd love to remove the over reliance on docker in my team, but the sticking point seems to be building and storing images. I know that Kubernetes can run with different runtimes, and that docker images can be oci compliant, but it's a hard sell to ask everyone to try something different in production when they do everything else with docker. Do you have a workflow that could help with that sort of thing?
Re: Making containers safer
#16The whole container safety story has been a mess/cluster /bag-of-tricks since the very beginning. Unlike BSD jails, security was never number one priority for containers. Just take a look at what GCE/AWS use respectively. The former built an entire syscall proxy with Gvisor while the latter uses a hypervisor based solution with Firecracker (think mini-VMs). Anything in production that touches foriegn code can't reall…
> (The main downside of hypervisors is that they are difficult to run on low-cost commodity cloud like Digital Ocean. You are forced to use bare metal stuff like Hetzner, AWS, Paket Cloud etc. The whole point of cloud is that the hardware/infrastructure is mostly abstracted away. If I have to run my own hypervisor just to ensure the container doesn't get broken out of, what's the point even calling it cloud?) Sorry i…
Kata works pretty well. You set it as a runtime for docker and it gives a fantastic illusion that the container isn't different from any other docker container but uses a KVM VM with minimal hardware to run the container. It starts in under 200ms (and faster if you make it use firecracker), etc.
The problem your parent is pointing out is that AWS instances don't support nested virtualization (and it sounds like neither does Digital Ocean) so on both of these cloud providers you can't use something like kata containers or any nested VMs. GCP does support nested virtualization in any VM and aws `.metal` instance types do as well, but are rather expensive. It's really a shame that even with the new EC2 kvm-based hypervisor they still didn't enable nested virtualization for most instance types, otherwise we'd definitely be making heavy use of kata containers.
I wish Oracle hadn't bought Ravello Systems and that Ravello open sourced their binary translation stuff that made nested VMs possible in EC2 without the full overhead of software virtualization. Unfortunately, there are no open source implementations of similar software that I know of. Their blog is now hosted on oracle's site: https://blogs.oracle.com/ravello/nested-virtualization-with-...
Re: Making containers safer
#17Note to anyone confused: The docker concept of '--privileged' is separate from what the LXD folks are refering to as 'privileged containers'. The LXD folks are talking about mapping UID 0 into the container, whereas (IIRC) the docker flag disables dropping capabilities and the seccomp syscall filters (and maybe some other things? I can't remember off the top of my head). The equivalent docker functionality is userns-…
I hate unnecessary abstractions. All this "docker functionality" is actually just based on namespaces and cgroups. I get what you're trying to say though.
Re: Making containers safer
#18LXC+LXD is one of the most undervalued container technologies out there. It can do a lot of cool things (like live migration via criu), and is IMO more production ready than Docker ever was. For those wondering about the distinction between "system" containers and "app" containers, the difference is user namespacing -- I think we should stop using the naming difference but instead go with specifying that the containe…
I believe ContainerD is the default on Google Kubernetes Engine (GKE)
Re: Making containers safer
#19LXC+LXD is one of the most undervalued container technologies out there. It can do a lot of cool things (like live migration via criu), and is IMO more production ready than Docker ever was. For those wondering about the distinction between "system" containers and "app" containers, the difference is user namespacing -- I think we should stop using the naming difference but instead go with specifying that the containe…
I'm curious as to how you get your images built. I'd love to remove the over reliance on docker in my team, but the sticking point seems to be building and storing images. I know that Kubernetes can run with different runtimes, and that docker images can be oci compliant, but it's a hard sell to ask everyone to try something different in production when they do everything else with docker. Do you have a workflow that…
Bazel can natively build images without Docker.
Bazel has many other benefits too like creating a full dependency build graph and fully reproductible builds.
Re: Making containers safer
#20The more I use Linux Containers (big fan of LXD) the more I'm convinced Solaris got it right with Zones, Crossbow, ZFS, SMF also for Zone services - Linux is still not there but instead we have lot's of vendor glue-code in go and many asteriks what is not possible. IMHO Linux should implement something like the zone concept that conceals cgroups, network, mount-namespaces - quota is still broken in btrfs, you can't d…
We have something like kernel zones in the form of OpenVZ. I think we'll eventually have them again in mainline, but it'll be built from those pieces, just like the current vendor glue-code.