Why the hell is your Kubernetes API public?

What the hell is a “hipster VPN?”

This article is working backwards to solving a problem that most development teams shouldn't have. I will first walk along with the article and try out their backwards solution before getting to my point:

If you're using Kubernetes managed by a cloud provider, e.g., Amazon EKS, authentication is already handled through IAM. This is a service that's designed to be exposed to Internet traffic and that AWS is essentially managing for you. If you're able to get past AWS IAM then the fact that your cluster API is public is moot because gaining access to an AWS account that has significant kubectl privileges is likely going to also expose a number of other privileges to make changes in AWS.

Okay, perhaps that's not enough for your liking. We want more defense in depth. AWS already tells us how to use a private API endpoint via their documentation: https://docs.aws.amazon.com/eks/latest/userguide/cluster-end...

But of course, those suggestions bring us here to this article: Amazon suggests using a bastion or a cloud IDE, but we might find that too clunky.

This is where the article offers "a better alternative: Tailscale is basically one of those fancy easy-to-use VPN replacements that promises to get rid of VPNs and bastion hosts in favor of protocol-aware proxying. And, that's fine, but it's not cheap compared to a traditional open source VPN if you're using it for a company: https://tailscale.com/pricing

Besides not being very cost-effective, we're still working backwards to the actual issue here. Who needs kubectl access and why are they using it so much in production that a bastion or VPN is annoying? Why don't we just make it so that developers don't have to use kubectl in production anymore? Isn't their code being deployed with an automated pipeline?

I think something doesn't quite add up in OP's claim. Having full access to the control plane effectively let's you do anything you want with the cluster. If there were really millions of completely unsecured Kubernetes instances lying around, we should see all kinds of worms and botnets already taking advantage of them - and we should also see lots of attempted probes/attacks to infect new instances as soon as they come online (dark forest, etc). Yet I haven't heard about anything like this so far. This leads me to believe that the "public" instances actually are protected in some way, just not by a private network.

Not a Kubernetes expert, but doesn't the Kubeconfig contain some sort of secret that the client needs for authentication? You might be able to get some basic health check without that, but I'd be very surprised if you could do much more without authentication.

At least if that's how it works, this would be an example of "zero-trust" networking, where you just assume anything can be reached from the internet anyway and secure your services based on that premise. If done well, it shouldn't matter whether or not your services are actually exposed on the internet or not.

Of course in the interest of defense-in-depth, I'd still try to make sure my services are not actually exposed - bugs happen all the time and additional security layer can't do harm.

Thought about it for a few seconds, and yeah, I'm sticking with meh on this one.

I do have a managed K8s cluster running on one of the major hosting services. It seems I can indeed hit the /healthz endpoint without auth. But I don't see any reason to care. First off, the control plane URL is a UUID tacked onto the hosting service's DNS. Good luck figuring out what it is starting from the URLs of the sites hosted on it. Even if you're my ISP or somehow otherwise in a position to observe traffic to the CP, good luck correlating it to what's actually hosted there. Yeah you can see that the cluster is healthy, but not much of anything else without auth AFAIK.

Second, there's a tension between "network restrictions don't help that much, if your control system is actually secure, it should be secure enough to expose to the clearnet" and "defense in depth always, another layer never hurts anything". I'm leaning towards the first. You can always come up with an excuse to add another layer, but how much does it actually protect you from any reasonably plausible attacker? Lots of stuff far more sensitive and valuable than anything I've got has control interfaces exposed to the net like that, relying solely on the security of OpenSSH, K8s TLS cert system, etc for their protection. If somebody figured out how to bypass that, and chose to use the capability against my stuff, who's to say that any extra layers of security I could dream up would actually slow them down? I think it's not likely, and those extra layers are much more likely to trip me up and let me accidentally lock myself out than to stop any plausible attacker.

[dead]