Kubernetes clusters being hijacked to mine cryptocurrencies • Hacker News Board

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#41

Post by raesene9 » Fri, Dec 07, 2018, 8:18 AM UTC

chrissnell wrote: post #31

whalesalad wrote: post #13

Got in a pretty heated debate with a colleague once about this. We had a really great infrastructure setup with a VPN bastion host that would get you into our VPC. You couldn't reach any of our kube nodes externally. Your Google account was your VPN account. It was pretty solid. When this engineer redid things they opted to go the public internet route where the master runs a public api and auth is done via a certifi…

No, a Kube cluster with client certificate authentication enabled is not going to be vulnerable to the specific issue discussed in OP's blog post: those are Kube cluster exposed publicly with no authentication whatsoever. I generally think it's no more risky to expose a Go app with cert-based auth than it is to expose OpenVPN so long as both are set up correctly.

Actually the CVE mentioned at the top of the blog could be exploited over the API server port in quite a few default configurations (thus the CVSS 9.8 score)

Many Kubernetes distributions enable anonymous authentication to allow for health checking, so there is some risk there.

As to the general point, the only thing I'd say is that Kubernetes is a massive 1.5 million Line code base which is relatively new code, where Openvpn has been around and attacked for a long time. I wouldn't be surprised if the recent CVE isn't the only issue we see in k8s over the next year.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#42

Post by raesene9 » Fri, Dec 07, 2018, 8:19 AM UTC

TheDong wrote: post #37

honkycat wrote: post #16

Earlier quoted context omitted.

I can't imagine a good reason to expose ANY of my services to the public internet. Aside from a rest-api that drives our application, where that is the feature of course. With software like google IAP, and many similar products, it just seems silly.

May I recommend reading up on beyondcorp [0]? Google has moved its internal stuff to the beyondcorp model, and it honestly seems like a better approach if you really care about security and have a big enough security team to make it work. [0]: https://www.beyondcorp.com/

Beyondcorp is a great model IF you can afford to manage it correctly.

Google have a) huge resources and b) a threat model which means they're subject to a lot of high-end attacks all the time.

for many corp's the idea of exposing all their services and endpoints to the general internet without firewalls or VPNs would ... end poorly...

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#43

Post by raesene9 » Fri, Dec 07, 2018, 8:21 AM UTC

ryacko wrote: post #23

kevin_nisbet wrote: post #10

Earlier quoted context omitted.

To be fair, kubernetes itself and most distributions are quite secure by default. So with kubernetes it's not the same as it was NoSQL databases that didn't have authentication that were bound to the internet. I'm not familiar with enough distributions to know if there is a popular distribution that totally disabled authentication by default, but in my companies distribution, kubeadm clusters, and I suspect all manag…

You will have to patch a critical vulnerability every year on production systems, no matter what language or who develops it. Secure defaults are irrevalent if you pay attention to the news.

On this point, worth noting that the Kubernetes support lifecycle is currently 9 months, so you indeed need to plan not only for bugfixes but a full upgrade, fairly regularly

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#44

Post by subway » Fri, Dec 07, 2018, 8:36 AM UTC

closeparen wrote: post #25

whalesalad wrote: post #13

Got in a pretty heated debate with a colleague once about this. We had a really great infrastructure setup with a VPN bastion host that would get you into our VPC. You couldn't reach any of our kube nodes externally. Your Google account was your VPN account. It was pretty solid. When this engineer redid things they opted to go the public internet route where the master runs a public api and auth is done via a certifi…

Why is a web server's demand for a certificate different from a VPN server's demand for a certificate?

The VPN server offers frivolous features like session tracking, and certificate revocation. Things k8s continues to punt down the road or outright ignore.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#45

Post by tetha » Fri, Dec 07, 2018, 8:45 AM UTC

TheDong wrote: post #35

tetha wrote: post #11

Earlier quoted context omitted.

How do you provide your initial credentials, though? Providing decently secure default initial credentials is possible, but tricky. And that's where I'll turn around 180 degrees and say: If you can't give me a hard reason why you'll be a hard target on the internet, you shouldn't have a public address. Default authentication isn't enough. I dislike trusting my edge firewall, but it gives me time to handle weak intern…

Kubernetes only accepts very limited forms of auth by default. Typically, it's limited to client certificates that have been signed by the private key the apiserver has access to. Client cert auth over tls is pretty damn secure. I expose my kubernetes cluster's apiserver to the internet and have, to my knowledge, had no issues yet.

Yeah, client certs are a good, hard reason. In fact, now that I think of it, the services we expose to the internet all deploy mutual TLS authentication.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#46

Post by igama » Fri, Dec 07, 2018, 8:48 AM UTC

roguecoder wrote: post #18

clubm8 wrote: post #17

Is anyone else a little tired of "X used to mine crypto" stories? Yes - if it has a CPU and access to the public internet, someone will hack it and make it mine "cypto". Let's stop pretending we aren't aware that the internet of things exists and writing breathless stories every time a toaster, router, or adult toy starts churning out Monero.

This is an important vulnerability in widely-used software. Crypto is relevant because the inherent design of crypto makes hacks like this more profitable, but it's not the main thing about the article.

Exactly, the main story is Kubernetes being exploited in the wild and in large numbers, Crypto mining is just one of the "attacks" tacking place.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#47

Post by balibebas » Fri, Dec 07, 2018, 8:49 AM UTC

raesene9 wrote: post #39

kevin_nisbet wrote: post #10

Earlier quoted context omitted.

To be fair, kubernetes itself and most distributions are quite secure by default. So with kubernetes it's not the same as it was NoSQL databases that didn't have authentication that were bound to the internet. I'm not familiar with enough distributions to know if there is a popular distribution that totally disabled authentication by default, but in my companies distribution, kubeadm clusters, and I suspect all manag…

Indeed modern versions of Kubernetes are fairly good for network security against unauthenticated attackers. That said it's not that long ago that a lot of distros were shipping unauthenticated kubelets, and I think that's where a lot of this will come from. From cluster reviews I've done, problems like this tend to arise where people are using older versions (so early adopters) or have hand-rolled their clusters, no…

Kubernetes is similar to WordPress in that outdated versions will always have a larger attack surface. And while k8s may be free if it's not EASY to upgrade and monitor or if the box rogue web miners won't be the real concern.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#48

Post by eecc » Fri, Dec 07, 2018, 9:05 AM UTC

whalesalad wrote: post #13

Got in a pretty heated debate with a colleague once about this. We had a really great infrastructure setup with a VPN bastion host that would get you into our VPC. You couldn't reach any of our kube nodes externally. Your Google account was your VPN account. It was pretty solid. When this engineer redid things they opted to go the public internet route where the master runs a public api and auth is done via a certifi…

Bah I hope they at the very least have firewall whitelist rules to limit API access to your CI vendor ranges.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#49

Post by igama » Fri, Dec 07, 2018, 9:25 AM UTC

CTO Binaryedge here. For those wondering, We have detected more than 15k Kubernetes APIs with Auth. This post focuses on ~1.5k found without Auth, that are fully open.

It's not just a Kubernetes Problem. Like many have posted, many databases, other types of clusters, shares, are accessible without Auth for those that know how to look for them (not that hard now days), mainly malicious actors.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#50

Post by ownagefool » Fri, Dec 07, 2018, 10:18 AM UTC

ram_rar wrote: post #34

nineteen999 wrote: post #4

This is one of the side-effects of products having enormous hype in this industry. Far too many people are adopting Docker/Kubernetes as they have been the hot new product for the last couple of years, often regardless of whether they are actually the best or most appropriate tool for the job. A lot of the people who get sucked into the hype are often inexperienced programmers, devops or admin types who are in positi…

>A lot of the people who get sucked into the hype are often inexperienced programmers, devops or admin types who are in positions of power or influence in companies that they probably shouldn't be, IMHO. I cannot agree more. Many times, I feel you da easily do away with ansible and terraform to setup VMs / docker. you dont quite need k8s. Just cuz K8s are cool.. people feel the need to use it.

It's more complicated that that. Whilst some people are probably jumping on kubernetes for the hype, there's a lot of things it makes really easy, especially for less experience teams.

For example:

- You want to spin up ephemeral environments to test PRs end2end. Sure, create a namespace, deploy your charts and run your tests. You want to do that with ansible, sure you can, but it's harder.

- You org is running apps via a multi-cloud and on-prem strategy? Okay, lets just write lots of tooling per cloud and another for on-prem, or we could abstract that away via kubernetes and only worry about tooling for kube itself.

- You want to do have rolling-upgrades. Sure, you build them with ansible then, or you could just use kubes.

Further to that, kubernetes is guiding reasonble abstractions, seperating infrastructure from code. Sure, it comes with complexity, but so does most things when you start throwing in scaling and auto-recovery.

For example, deploy terraform from your laptop? The device you probably browse porn on has becomes an attack surface. Move this to Jenkins, the CI is the attack surface. Put your code on Bitbucket? Bitbucket and the Jenkinsfile becomes the attack surface. Pretty much everything we do has complexity and attack surface _problems_ and using a managed k8s service will allow you some easy wins so you can actually think about those other problems, and those solutions will work on all platforms you can run k8s on.