The silent CVE in the heart of Kubernetes apiserver • Hacker News Board

Re: The silent CVE in the heart of Kubernetes apiserver

#11

Post by spiffxp » Thu, Dec 06, 2018, 4:58 AM UTC

colemickens wrote: post #6

Has there been any word of how long this has been known about? I recall a certain group of people talking confidently, open but vaguely about a serious k8s security bug at least a few months ago. Given the number of clusters that undoubtedly had to be upgraded for this, it wouldn't surprise me if this was found a while ago. Also, my heart goes out to the [big company] engineering manager asking in the GitHub thread i…

I was part of the Kubernetes v1.13 release team, though not part of the team that produced this fix.

The project's Product Security Team adhered to the timeline indicated in the project's security release process: https://github.com/kubernetes/sig-release/blob/master/securi...

tl;dr a fix is (edit: optionally) sent out to a private distributors list under embargo within 2 weeks of disclosure, and public disclosure (with new releases) happens within 3 weeks of disclosure (with some discretion for timing to make sure it's not buried in a weekend or off-hours)

I can't speak to who knew about it when outside of the project, but I know the project acted expediently once the vulnerability was disclosed.

Re: The silent CVE in the heart of Kubernetes apiserver

#12

Post by kevin_nisbet » Thu, Dec 06, 2018, 5:48 AM UTC

darren0 wrote: post #9

The PoC is mostly correct, but basically is testing for a side effect of the fix, not the presence of the vulnerability. Either which way it's probably still useful, but since we are all geeks here might as well talk about the minutia. The PoC basically tests that if you create a websocket request that results in non-101 response the socket is closed by the server. That is a side effect of the fix. The way the proxy…

You're analysis is correct.

I did briefly consider this when originally throwing together the PoC. It seemed reasonable at the time to test for the behaviour of the kubernetes apiserver as implemented, not how it could theoretically behave if implemented differently. In the ALB/L7 type use case, I'm not sure a reliable PoC can ever exist, because even if you do successfully exploit it, the second request can be load balanced elsewhere.

Also, I don't believe you would necessarily need a separate pod, the PoC could work by using different endpoints, such as exec vs logs, or even better, try and hit a non pod related API endpoint within kubelet. Unfortunately I tried to get the PoC working quickly, and didn't have a chance to revisit this today to see if it can be enhanced.

Re: The silent CVE in the heart of Kubernetes apiserver

#13

Post by colemickens » Thu, Dec 06, 2018, 8:14 AM UTC

spiffxp wrote: post #11

colemickens wrote: post #6

Has there been any word of how long this has been known about? I recall a certain group of people talking confidently, open but vaguely about a serious k8s security bug at least a few months ago. Given the number of clusters that undoubtedly had to be upgraded for this, it wouldn't surprise me if this was found a while ago. Also, my heart goes out to the [big company] engineering manager asking in the GitHub thread i…

I was part of the Kubernetes v1.13 release team, though not part of the team that produced this fix. The project's Product Security Team adhered to the timeline indicated in the project's security release process: https://github.com/kubernetes/sig-release/blob/master/securi... tl;dr a fix is (edit: optionally) sent out to a private distributors list under embargo within 2 weeks of disclosure, and public disclosure (w…

I should've known that there was an official timeline as part of the process, but forgot. I wouldn't have asked about that otherwise; I didn't mean to cast any aspersions. Whatever I'm thinking of was from folks I respect, presumably about another issue or I'm simply mis-remembering things. Thank you.

(Also, aside, props to everyone that got this rolled out so fast at the major providers.)

Re: The silent CVE in the heart of Kubernetes apiserver

#14

Post by colemickens » Thu, Dec 06, 2018, 8:17 AM UTC

darren0 wrote: post #10

colemickens wrote: post #6

Has there been any word of how long this has been known about? I recall a certain group of people talking confidently, open but vaguely about a serious k8s security bug at least a few months ago. Given the number of clusters that undoubtedly had to be upgraded for this, it wouldn't surprise me if this was found a while ago. Also, my heart goes out to the [big company] engineering manager asking in the GitHub thread i…

I'm fairly confident this issue has existed since pretty much the beginning of k8s. It is possible it doesn't go back that far because SPDY used to be used and not websockets. The basic Upgrade logic still applies so I assume it was there. This flaw is actually in quite a few open source projects, it's a common approach and mistake. For most projects it results in a functional bug, not a security issue. I personally…

Thanks Darren, I wish I'd read your post earlier to have realized the timeline. I agree with your conclusions re: the sky. Thank you for the write-up and replying here.