Making containers safer • Hacker News Board

Re: Making containers safer

#41

Post by laurieodgers » Thu, Aug 22, 2019, 4:53 PM UTC

mav3rick wrote: post #17

awirth wrote: post #2

Note to anyone confused: The docker concept of '--privileged' is separate from what the LXD folks are refering to as 'privileged containers'. The LXD folks are talking about mapping UID 0 into the container, whereas (IIRC) the docker flag disables dropping capabilities and the seccomp syscall filters (and maybe some other things? I can't remember off the top of my head). The equivalent docker functionality is userns-…

I hate unnecessary abstractions. All this "docker functionality" is actually just based on namespaces and cgroups. I get what you're trying to say though.

You nailed it. If you can take a hard to grasp or orchestrate concept and make it easy for the layperson to use, then you've built a linux tool

Re: Making containers safer

#42

Post by stgraber » Thu, Aug 22, 2019, 4:53 PM UTC

e12e wrote: post #36

bscphil wrote: post #3

This was a pretty good read. I use containers quite a lot on my server at home at maintain a bunch of utilities. Mostly I'm using systemd-nspawn. > User namespaces have been around since the 3.12 kernel, but few other container management systems use the feature to isolate their containers. Part of the reason for that is the difficulty in sharing files between containers because of the UID mapping. LXD is currently u…

I'm a little surprised that lxd doesn't cater more to nfs rather than bind mounts etc for file sharing. I mean if you already have private/secure networking - just use a network filesystem for your... Network filesystem needs?

NFS currently cannot be used inside a user namespace.

There are patches floating around (similar to the work we did to allow FUSE) but they haven't made it upstream yet and my understanding is that there is some tricky corner cases on NFSv3 which still need to be sorted (NFSv4 was easier due to already having uid mapping capabilities).

Re: Making containers safer

#43

Post by blibble » Thu, Aug 22, 2019, 5:44 PM UTC

sansnomme wrote: post #5

The whole container safety story has been a mess/cluster /bag-of-tricks since the very beginning. Unlike BSD jails, security was never number one priority for containers. Just take a look at what GCE/AWS use respectively. The former built an entire syscall proxy with Gvisor while the latter uses a hypervisor based solution with Firecracker (think mini-VMs). Anything in production that touches foriegn code can't reall…

cloud providers are starting to expose the virtualisation cpu feature these days

I've seen it on gcp and ovh (not tested aws)

Re: Making containers safer

#44

Post by eximius » Thu, Aug 22, 2019, 6:00 PM UTC

eyberg wrote: post #37

hardwaresofton wrote: post #8

Earlier quoted context omitted.

> (The main downside of hypervisors is that they are difficult to run on low-cost commodity cloud like Digital Ocean. You are forced to use bare metal stuff like Hetzner, AWS, Paket Cloud etc. The whole point of cloud is that the hardware/infrastructure is mostly abstracted away. If I have to run my own hypervisor just to ensure the container doesn't get broken out of, what's the point even calling it cloud?) Sorry i…

In the public cloud (aws/gce) we provision unikernels as vms themselves - that is there is no underlying linux instance. It's just the unikernel riding on top of the hypervisor (in the case of google that's kvm, for the t2s on aws it's xen but their c5s are kvm). For small services you use t2-micros (aws) and f1-micros (gce). Both will give you a thread. If you need more you scale more. This works great if you have a…

What domain is the problem you're solving in that you can effectively run unikernels? i.e., you don't need something that ultimately doesn't run well in unikernels. For example, our application is a web app - that by itself can run reasonably well in a unikernel (not with how we've written it, but it can be done). But we call out to imagemagick for some processing, which would pretty much preclude a unikernel approach (barring breaking it out into its own service, which is totally doable - its hard to provide a perfect example).

Also, what are you using to create the unikernels?

Re: Making containers safer

#45

Post by sansnomme » Thu, Aug 22, 2019, 7:03 PM UTC

hardwaresofton wrote: post #8

sansnomme wrote: post #5

The whole container safety story has been a mess/cluster /bag-of-tricks since the very beginning. Unlike BSD jails, security was never number one priority for containers. Just take a look at what GCE/AWS use respectively. The former built an entire syscall proxy with Gvisor while the latter uses a hypervisor based solution with Firecracker (think mini-VMs). Anything in production that touches foriegn code can't reall…

> (The main downside of hypervisors is that they are difficult to run on low-cost commodity cloud like Digital Ocean. You are forced to use bare metal stuff like Hetzner, AWS, Paket Cloud etc. The whole point of cloud is that the hardware/infrastructure is mostly abstracted away. If I have to run my own hypervisor just to ensure the container doesn't get broken out of, what's the point even calling it cloud?) Sorry i…

We don't need unikernels. The technology for running containers securely has existed for years. The only problem is that few mainstream/major cloud providers support them, it requires nested virtualization among other things. And your traditional companies like AWS and GCE make you pay out of the nose for it because if it becomes widely available, then the whole "Serverless", elastic container service etc. industry may be eroded. Right now you have two choices: run your own co-location bare metal server with all the complexity that comes with it, or pay for a VPS with nested virtualization support that is a magnitude more expensive.

Re: Making containers safer

#46

Post by _seemethere » Thu, Aug 22, 2019, 9:06 PM UTC

hardwaresofton wrote: post #9

LXC+LXD is one of the most undervalued container technologies out there. It can do a lot of cool things (like live migration via criu), and is IMO more production ready than Docker ever was. For those wondering about the distinction between "system" containers and "app" containers, the difference is user namespacing -- I think we should stop using the naming difference but instead go with specifying that the containe…

containerd is also the default container runtime for docker as well.

when you do a `docker run` it just calls out to containerd now to launch your container.

Re: Making containers safer

#47

Post by chmaynard » Thu, Aug 22, 2019, 9:41 PM UTC

laurieodgers wrote: post #38

Honestly its horses for courses. If you want to run your workload in the most secure manner using BSD jails then go for it, but you will soon find that anyone with expertise to maintain it is hard to find. Almost all DevOps/SRE/Systems Engineers want to run their workloads (containers) in Docker. The same goes for LXD. Lets get one thing right - none of these systems are a Virtual Machine with their own hardware and…

> This article is 2 years old.

Really? You just shot your credibility in the foot.

Re: Making containers safer

#48

Post by segmondy » Fri, Aug 23, 2019, 2:53 AM UTC

dragonsh wrote: post #6

I have been a user of LXC and LXD since 2013. Earlier when docker was based on LXC I tried it. Then Docker went its own path and build its own libcontainer library. During that time LXC project added support for unprivileged container and since than I didn't use docker. Still today when majority of container runtime including kubernetes is based on docker container (some on OCI), I continue to use LXD and LXC. LXD by…

I too have been using LXC/LXD since before Docker but it's not the same thing.

LXC is a system container. When you want a full system, instead of a VM, you can use LXC, have ssh, give people accounts. It has the same issues as a VM or a regular server, it's easy to leave snowflakes on it, unless you're very disciplined and automate everything.

Docker is an application container, it's not for hosting a user, but just an app. Easy to reproduce, you can share the images with other's or have them rebuild the image to run the app. So two very different things. When Docker first came out, I was very hesitant to use it, and thought it was very stupid to have just one container for one application. But as I thought about it and played with it, it made sense and I have come around. I still use both.

You compared LXD to kubernetes, they are not comparable. k8s is a container OS, it's for orchestrating tons of containers, so you don't have to manually deploy and network them, and restart them.

Re: Making containers safer

#49

Post by awirth » Fri, Aug 23, 2019, 4:38 AM UTC

mav3rick wrote: post #17

awirth wrote: post #2

Note to anyone confused: The docker concept of '--privileged' is separate from what the LXD folks are refering to as 'privileged containers'. The LXD folks are talking about mapping UID 0 into the container, whereas (IIRC) the docker flag disables dropping capabilities and the seccomp syscall filters (and maybe some other things? I can't remember off the top of my head). The equivalent docker functionality is userns-…

I hate unnecessary abstractions. All this "docker functionality" is actually just based on namespaces and cgroups. I get what you're trying to say though.

As I was trying to understand this space coming from a 'docker user' background I was incredibly confused by the two different definitions of "privileged containers" ("what do you mean if I don't add --privileged it's still privileged?") -- so I figured others might appreciate the pointer as well.

Now that I understand what's going on better, I definitely agree that docker does not abstract the kernel APIs in a way that makes it easy to understand what's going on underneath the hood. Is that a good thing? I honestly don't know.

I'd encourage anyone interested in learning more to check out the codebase for JessFraz's contained.af

Re: Making containers safer

#50

Post by auspex » Fri, Aug 23, 2019, 2:12 PM UTC

awirth wrote: post #2

Note to anyone confused: The docker concept of '--privileged' is separate from what the LXD folks are refering to as 'privileged containers'. The LXD folks are talking about mapping UID 0 into the container, whereas (IIRC) the docker flag disables dropping capabilities and the seccomp syscall filters (and maybe some other things? I can't remember off the top of my head). The equivalent docker functionality is userns-…

Any container not running as normal user is considered privileged. A root container is privileged bit isn't --privileged.

An important distinction. From a security point of view running --privileged is just lazy. If you need things like kernel permissions etc, run as root and then request kernel permissions in the deployment yaml... and if running something like k8s make sure to apply a pod security spec limiting permissions and the apply the right seccomp profile.