CVE-2019-5736: runc container breakout • Hacker News Board

Re: CVE-2019-5736: runc container breakout

#91

Post by geofft » Tue, Feb 12, 2019, 4:18 AM UTC

cyphar wrote: post #86

geofft wrote: post #64

Earlier quoted context omitted.

I think it's subtler than that. It is mostly safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. Yes, it's wrong to no…

> I think it's subtler than that. It is mostly safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. I disagree. It is d…

Can you use user namespaces with a released version of Docker or Kubernetes? I think the answer is no? (Or can you do it with Kubernetes + some other container runtime?)

We run Kubernetes + Docker with a policy of no root inside containers (we map you to your normal UID inside the container), but most of the rabbit hole we got into was trying to figure out the implications of setuid binaries inside the container. It seems like on a normal system, setuid binaries inside a container do in fact get host root unless you tell Docker to drop those capabilites, and also fscaps aren't really usable in a container image, I think.

Re: CVE-2019-5736: runc container breakout

#92

Post by cyphar » Tue, Feb 12, 2019, 4:45 AM UTC

geofft wrote: post #91

cyphar wrote: post #86

Earlier quoted context omitted.

> I think it's subtler than that. It is mostly safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. I disagree. It is d…

Can you use user namespaces with a released version of Docker or Kubernetes? I think the answer is no? (Or can you do it with Kubernetes + some other container runtime?) We run Kubernetes + Docker with a policy of no root inside containers (we map you to your normal UID inside the container), but most of the rabbit hole we got into was trying to figure out the implications of setuid binaries inside the container. It…

User namespaces have been supported in Docker since 1.10. I don't think that it's necessarily "supported" in Kubernetes -- there was a KEP to add support last year but it's still a while away.

As for setuid and fscaps, they both work in containers (container images can contain them but some filesystems don't support xattrs such as AUFS). And yes, without user namespaces, they escalate to host root. You can use no_new_privs which blocks things like setuid or fscap but it also can cause problems (though I think you can enable no_new_privs in Kubernetes).

Re: CVE-2019-5736: runc container breakout

#93

Post by ec109685 » Tue, Feb 12, 2019, 5:32 AM UTC

geggam wrote: post #78

Next year this exploit will still be in thousands if not millions of containers all over There is a distinct lack of knowledge on how to manage a system in the container ecosystem

What do you mean _in_ containers?

Re: CVE-2019-5736: runc container breakout

#94

Post by yjftsjthsd-h » Tue, Feb 12, 2019, 5:57 AM UTC

miguelmota wrote: post #13

For better isolation check out KataContainers: https://github.com/kata-containers/runtime It's a drop-in replacement for runc. With KataContainers it runs docker containers in a lightweight VM so you get all the security benefits of a VM. The downside is slightly slower container start up times and might not work in nested virtualized environments.

Did AWS's firecracker ever get to the point of being drop-in compatible? They were also doing containers-in-VMs.

Re: CVE-2019-5736: runc container breakout

#95

Post by miguelmota » Tue, Feb 12, 2019, 7:33 AM UTC

bpye wrote: post #83

miguelmota wrote: post #13

For better isolation check out KataContainers: https://github.com/kata-containers/runtime It's a drop-in replacement for runc. With KataContainers it runs docker containers in a lightweight VM so you get all the security benefits of a VM. The downside is slightly slower container start up times and might not work in nested virtualized environments.

gVisor is also pretty neat, they say KVM support is experiemental though: https://github.com/google/gvisor gVisor is used behind Go 1.11 on App Engine so Google must be fairly confident that it's a sufficient security boundary though I'm fairly sure they don't use the public KVM isolation so YMMV.

gVisor is a kernel implemented in userspace. The one downside of gVisor is that not all syscalls are implemented and they're relying on the community to implement them. This is what was holding me back from adopting it for a project.

Re: CVE-2019-5736: runc container breakout

#96

Post by wodny » Tue, Feb 12, 2019, 12:18 PM UTC

The vulnerability description seems to be lacking an explanation why the /proc/$PID/exe symlink is so special and why using the #!/proc/self/exe hashbang will work while using #!/usr/sbin/runc probably won't. Am I right that the proc filesystem in proc_exe_link() fills the file_operations struct in a way that causes open() not to go through a dereferencing procedure using the filesystem but just open the file used to run the executable?

Re: CVE-2019-5736: runc container breakout

#97

Post by olemartinorg » Tue, Feb 12, 2019, 1:31 PM UTC

justincormack wrote: post #24

darren0 wrote: post #9

The best fix is to upgrade to 18.09.2. For those that can't do that immediately, backported versions of runc for Docker releases going back to 1.12.6 are available from Rancher at https://github.com/rancher/runc-cve . But please only do that as a temporary workaround until you can properly upgrade to 18.09.2. Please patch if you don't 100% trust all users on your host.

We also released Docker 18.06.2 with the fix, as a lot of Kubernetes users are on this release.

The new package in Ubuntu Trusty seems to be broken. Not that Trusty is supported for much longer. See https://github.com/docker/for-linux/issues/591

Re: CVE-2019-5736: runc container breakout

#98

Post by el_duderino » Tue, Feb 12, 2019, 1:46 PM UTC

gr2020 wrote: post #3

Looks like Docker 18.09.2 was released a few minutes ago to address this: https://github.com/docker/docker-ce/releases

PoC here:

https://github.com/feexd/pocs/blob/master/CVE-2019-5736/expl...

Re: CVE-2019-5736: runc container breakout

#99

Post by ofrzeta » Tue, Feb 12, 2019, 3:09 PM UTC

geofft wrote: post #91

cyphar wrote: post #86

Earlier quoted context omitted.

> I think it's subtler than that. It is mostly safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. I disagree. It is d…

Can you use user namespaces with a released version of Docker or Kubernetes? I think the answer is no? (Or can you do it with Kubernetes + some other container runtime?) We run Kubernetes + Docker with a policy of no root inside containers (we map you to your normal UID inside the container), but most of the rabbit hole we got into was trying to figure out the implications of setuid binaries inside the container. It…

OpenShift (Kubernetes distro) is running all pods with random UIDs, and running a pod/container as root requires special privileges that only a cluster admin can grant. Also OpenShift is not vulnerable to this attack due the use of SELinux which is mandatory for OpenShift installations.

Re: CVE-2019-5736: runc container breakout

#100

Post by NathanKP » Tue, Feb 12, 2019, 4:47 PM UTC

tristanz wrote: post #89

NathanKP wrote: post #28

Amazon employee here: we have released a security bulletin covering how to update to the latest patched Docker on Amazon Linux, Amazon ECS, Amazon EKS, AWS Fargate, AWS IoT Greengrass, AWS Batch, AWS Elastic Beanstalk, AWS Cloud9, AWS SageMaker, AWS RoboMaker, and AWS Deep Learning AMI. Please check out the bulletin and update if you are using one of these services. https://aws.amazon.com/security/security-bulletins/…

As far as I understand, EKS doesn't support PodSecurityPolicy yet so any user that can launch a pod can trivially root the host via host mounts already. This surprisingly isn't clearly documented.

ECS doesn't have a top level resource called "PodSecurityPolicy" but we do provide task level configuration options for all the major settings that you would normally put in your pod security policy, including including adding and dropping capabilities, privileged or unprivileged mode, docker security options for controlling SELinux or AppArmor, ulimits, sysctl settings, among others. You can find all these configuration options and more documented here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

It is definitely possible to prevent a task running in ECS from getting root access to the host. If there is something missing that you feel we need to add to ECS to better enable this, definitely reach out and let me know!