Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes clusters being hijacked to mine cryptocurrencies

blog.binaryedge.io

Post reply on HN
Print view View original

31–40 of 67 posts

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#31

Post by chrissnell »

whalesalad wrote: post #13

Got in a pretty heated debate with a colleague once about this. We had a really great infrastructure setup with a VPN bastion host that would get you into our VPC. You couldn't reach any of our kube nodes externally. Your Google account was your VPN account. It was pretty solid. When this engineer redid things they opted to go the public internet route where the master runs a public api and auth is done via a certifi…

No, a Kube cluster with client certificate authentication enabled is not going to be vulnerable to the specific issue discussed in OP's blog post: those are Kube cluster exposed publicly with no authentication whatsoever.

I generally think it's no more risky to expose a Go app with cert-based auth than it is to expose OpenVPN so long as both are set up correctly.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#32

Post by blazespin »

voltagex_ wrote: post #22

At least cryptocurrency has removed most of the creativity from script kiddies - there's so many more interesting things you could do than just mine coins.

Yeah, exactly. It’s almost like a bounty for find a vuln. It seems to be a mostly harmless attack that doesn’t cause global internet grief like a DDoS or something.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#34

Post by ram_rar »

nineteen999 wrote: post #4

This is one of the side-effects of products having enormous hype in this industry. Far too many people are adopting Docker/Kubernetes as they have been the hot new product for the last couple of years, often regardless of whether they are actually the best or most appropriate tool for the job. A lot of the people who get sucked into the hype are often inexperienced programmers, devops or admin types who are in positi…

>A lot of the people who get sucked into the hype are often inexperienced programmers, devops or admin types who are in positions of power or influence in companies that they probably shouldn't be, IMHO.

I cannot agree more. Many times, I feel you da easily do away with ansible and terraform to setup VMs / docker. you dont quite need k8s. Just cuz K8s are cool.. people feel the need to use it.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#35

Post by TheDong »

tetha wrote: post #11
kevin_nisbet wrote: post #10

Earlier quoted context omitted.

To be fair, kubernetes itself and most distributions are quite secure by default. So with kubernetes it's not the same as it was NoSQL databases that didn't have authentication that were bound to the internet. I'm not familiar with enough distributions to know if there is a popular distribution that totally disabled authentication by default, but in my companies distribution, kubeadm clusters, and I suspect all manag…

How do you provide your initial credentials, though? Providing decently secure default initial credentials is possible, but tricky. And that's where I'll turn around 180 degrees and say: If you can't give me a hard reason why you'll be a hard target on the internet, you shouldn't have a public address. Default authentication isn't enough. I dislike trusting my edge firewall, but it gives me time to handle weak intern…

Kubernetes only accepts very limited forms of auth by default.

Typically, it's limited to client certificates that have been signed by the private key the apiserver has access to.

Client cert auth over tls is pretty damn secure. I expose my kubernetes cluster's apiserver to the internet and have, to my knowledge, had no issues yet.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#36

Post by TheDong »

ryacko wrote: post #23
kevin_nisbet wrote: post #10

Earlier quoted context omitted.

To be fair, kubernetes itself and most distributions are quite secure by default. So with kubernetes it's not the same as it was NoSQL databases that didn't have authentication that were bound to the internet. I'm not familiar with enough distributions to know if there is a popular distribution that totally disabled authentication by default, but in my companies distribution, kubeadm clusters, and I suspect all manag…

You will have to patch a critical vulnerability every year on production systems, no matter what language or who develops it. Secure defaults are irrevalent if you pay attention to the news.

> You will have to patch a critical vulnerability every year on production systems, no matter what language or who develops it.

Interesting. I've got a few openbsd boxes that do not have vulnerabilities that impact them nearly so often.

It turns out that if you practice defence in depth, the majority of security vulnerabilities in the news have no impact on you.

For example, on my openbsd boxes I have only a single user. I do not run any untrusted code. That means spectre/meltdown doesn't actually impact me because no one can run code which will perform such a timing attack.

There was a recent openbsd/Xorg security issue. I didn't have X installed, and even if I did since it's only a single-user server, it again wouldn't have impacted me (privilege escalation means nothing when everyone is effectively root in my threat model).

All vulnerabilities are not created equal, and with enough good practices it's possible to have boxes that are secure for years and years with no need for patches.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#37

Post by TheDong »

honkycat wrote: post #16
whalesalad wrote: post #13

Got in a pretty heated debate with a colleague once about this. We had a really great infrastructure setup with a VPN bastion host that would get you into our VPC. You couldn't reach any of our kube nodes externally. Your Google account was your VPN account. It was pretty solid. When this engineer redid things they opted to go the public internet route where the master runs a public api and auth is done via a certifi…

I can't imagine a good reason to expose ANY of my services to the public internet. Aside from a rest-api that drives our application, where that is the feature of course. With software like google IAP, and many similar products, it just seems silly.

May I recommend reading up on beyondcorp [0]?

Google has moved its internal stuff to the beyondcorp model, and it honestly seems like a better approach if you really care about security and have a big enough security team to make it work.

[0]: https://www.beyondcorp.com/

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#38

Post by TheDong »

raverbashing wrote: post #30
kevin_nisbet wrote: post #10

Earlier quoted context omitted.

To be fair, kubernetes itself and most distributions are quite secure by default. So with kubernetes it's not the same as it was NoSQL databases that didn't have authentication that were bound to the internet. I'm not familiar with enough distributions to know if there is a popular distribution that totally disabled authentication by default, but in my companies distribution, kubeadm clusters, and I suspect all manag…

To be fair, you're playing with fire if you rely on product security to be perfect. For every SSHd out there there are multiple SystemD or worse security products.

I think you meant "OpenSSH" not SSHd, and "systemd" is always spelled lowercase. The 'd' doesn't stand for something and is never capitalized.

"sshd" is an ambiguous term as there are many ssh daemons, from the libssh server to dropbear to OpenSSH, and OpenSSH is likely the one that you use and the most secure one.

systemd has had a few security incidents, but very few of them are actually a big deal. People have overblown each and every one since there has become a cult of systemd hate, which has muddied the waters significantly.

> you're playing with fire if you rely on product security to be perfect

That's a vacuous statement; of course you can't rely on everything being perfect, so you must practice defence in depth. Everyone already knows software sometimes has bugs.

The point of the parent post is that Kubernetes does intend to allow secure use while being exposed publicly (unlike e.g. the default redis configuration). The parent post does not claim it is perfect and that you must never patch it, merely that it is reasonable and can be hardened.

In the end, there are tradeoffs. You must decide that the convenience of developers being able to ssh into machines is worth the risk of running OpenSSH. You must decide that using Google Apps is worth the risk that Google will have a data breach exposing all of your confidential information. You must decide that Slack can be trusted to write secure enough php that your messages aren't being read by others.

Just because something isn't perfect doesn't mean that it can't still be a good tradeoff based on the expected risk.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#39

Post by raesene9 »

kevin_nisbet wrote: post #10
tetha wrote: post #8

Ugh. I mean, I recently got in an argument if anything but a hard firewall could or should be exposed to a WAN interface on the internet and we kinda agreed to not agree for now. But, popular services, on default ports, with default APIs enabled, without hard authentication on a WAN interface? That should be a paddling. That doesn't fly. Or, well it does, except not for the guy paying the power.

To be fair, kubernetes itself and most distributions are quite secure by default. So with kubernetes it's not the same as it was NoSQL databases that didn't have authentication that were bound to the internet. I'm not familiar with enough distributions to know if there is a popular distribution that totally disabled authentication by default, but in my companies distribution, kubeadm clusters, and I suspect all manag…

Indeed modern versions of Kubernetes are fairly good for network security against unauthenticated attackers.

That said it's not that long ago that a lot of distros were shipping unauthenticated kubelets, and I think that's where a lot of this will come from.

From cluster reviews I've done, problems like this tend to arise where people are using older versions (so early adopters) or have hand-rolled their clusters, not realising all the areas that require hardening.

Re: Kubernetes clusters being hijacked to mine cryptocurrencies

#40

Post by raesene9 »

TheDong wrote: post #35
tetha wrote: post #11

Earlier quoted context omitted.

How do you provide your initial credentials, though? Providing decently secure default initial credentials is possible, but tricky. And that's where I'll turn around 180 degrees and say: If you can't give me a hard reason why you'll be a hard target on the internet, you shouldn't have a public address. Default authentication isn't enough. I dislike trusting my edge firewall, but it gives me time to handle weak intern…

Kubernetes only accepts very limited forms of auth by default. Typically, it's limited to client certificates that have been signed by the private key the apiserver has access to. Client cert auth over tls is pretty damn secure. I expose my kubernetes cluster's apiserver to the internet and have, to my knowledge, had no issues yet.

Client cert auth is quite good against unauthenticated attacks but has its downsides.

At the moment Kubernetes has no certificate revocation process at all, so if one of your users has their cert stolen for an Internet facing cluster, you'll have to rebuild the entire CA and re-issue all certs to get round the problem.

Post reply on HN