Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The future of Kubernetes is virtual machines

tech.paulcz.net

Post reply on HN
Print view View original

61–70 of 128 posts

Re: The future of Kubernetes is virtual machines

#61

Post by ec109685 »

state_less wrote: post #37

While the use of namespaces seemed a bit discounted in the article, I think part of the problem is that it's an underutilized and poorly conveyed feature, but maybe that's just me? I'd like to see a better out-of-the-box vanilla K8S user management interface in the dashboard. Maybe the default UX on cluster creation is a user/service account that isn't cluster admin that is limited to 1 or more namespaces. There shou…

Agree, I'm not sure what the author is getting at here:

"Compounding this is the fact that most Kubernetes components are not Tenant aware. Sure you have Namespaces and Pod Security Policies but the API itself is not. Nor are the internal components like the kubelet or kube-proxy. This leads to Kubernetes having a “Soft Tenancy” model."

That's like saying the Hypervisor isn't "tenant" aware.

Re: The future of Kubernetes is virtual machines

#62

Post by georgebarnett »

davidopp__ wrote: post #54
georgebarnett wrote: post #28

Earlier quoted context omitted.

> It should be OK to have one big cluster Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale. What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team. There are oper…

A pattern we're seeing a lot of recently is one cluster per "stage" per region, where a "stage" is something like dev/test, canary, and prod. (In some cases only prod is replicated across multiple regions.) I think this may end up being the "sweet spot" for Kubernetes multi-tenancy architecture. The number of clusters isn't quite at the "Kubesprawl" level (I love that phrase and am absolutely going to steal it) -- yo…

Your experience mirrors what I've seen.

Many teams use clusters for stages because they work on underlying cluster components and need to ensure they work together and upgrade processes work (e.g. terraform configs comes to mind). Theres no reason to separate accounts because the cluster constructs aren't there for security.

Considering it deeper (I haven't had to think about this for a while), I think multi tenancy would cover almost all of the use cases I've seen except for the platform dev where people use clusters for separation when testing cluster config-as-code changes.

Re: The future of Kubernetes is virtual machines

#63

Post by lykr0n »

halbritt wrote: post #50
lykr0n wrote: post #31

Earlier quoted context omitted.

True. My only experience is with VMware OpenStack, and my quick googling didn't turn up much info. I think Kubernetes will fall to the same fate as OpenStack is sliding into. Growing complexity with promises of the world. Time will tell. You seem to know more than I do, so I got to ask. Why does openstack-helm exist? Why would anyone want to deploy OpenStack on top of kubernetes? Is it so you can have the OpenStack A…

One of the issues with Openstack is managing the services required to run it. I imagine that helm could be used to make it easier to run these services in kubernetes.

True, but that adds another layer of complexity to an already complex system. If I'm using the ansible module, that's complex enough. Throwing in management of Kubernetes and the additional cruft containers adds- it seems like it's a lot of hassle for little gain.

Re: The future of Kubernetes is virtual machines

#64

Post by smarterclayton »

trjordan wrote: post #43
georgebarnett wrote: post #28

Earlier quoted context omitted.

> It should be OK to have one big cluster Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale. What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team. There are oper…

Config blast radius, hardware sizing, networking partitions, disaster recovery, dev/prod separation -- lots of good reasons to have multiple clusters! As a gut check, most of these reasons apply to AWS AZs as well. If your Kubernetes strategy calls for more than one per region per AWS account, it means that you, organizationally, don't trust containers as much as you trust VMs on AWS. And right now, you're probably r…

It’s not only about trusting containers. It’s also what data that cluster has access to (anything on a PV attached to a cluster is readable and writable by someone with privileges). The control plane can program the infra to some degree (increasingly so over time).

Treating a single cloud account and the clusters it hosts as a single security cell is more than reasonable, especially if you need to handle sensitive data, be able to audit the state of the cluster (hope your cluster can’t overwrite the contents of the S3 bucket that you’re using for cloudtrail logs, etc), or reason about the risk of compromise.

We run lots of multitenant kube clusters, and I would still recommend anyone who expects to grow to create hard walls between clusters and cloud infra as soon as reasonably possible.

Re: The future of Kubernetes is virtual machines

#65

Post by jacques_chester »

davidopp__ wrote: post #60
jacques_chester wrote: post #15

Earlier quoted context omitted.

> Personally, I think the value proposition is tenuous Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable. Relying on the cluster as the security boundary is very effective ... and very wasteful. > Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but…

I don't think multi-tenancy has been "retrofitted" onto Kubernetes. Kubernetes was designed with multi-tenancy in mind from the very early releases -- namespaces, authn/authz (initially ABAC, later RBAC), ResourceQuota, PodSecurityPolicy, etc. New features are added over time, such as NetworkPolicy (which has been in Kubernetes for a year and a half, so perhaps not "new" anymore!), EventRateLimit, and others, but alw…

> I don't think multi-tenancy has been "retrofitted" onto Kubernetes. Kubernetes was designed with multi-tenancy in mind from the very early releases -- namespaces, authn/authz (initially ABAC, later RBAC), ResourceQuota, PodSecurityPolicy, etc.

My complaint is that these require assembly and are in many cases opt-in (making RBAC opt-out was a massive leap forward).

Namespaces are the lynchpin, but are globally visible. In fact an enormous amount of stuff tends to wind up visible in some fashion. And I have to go through all the different mechanisms and set them up correctly, align them correctly, to create a firmer multi-tenancy than the baseline.

Put another way, I am having to construct multi-tenancy inside multiple resources at the root level, rather than having tenancy as the root level under which those multiple resources fall.

> there are a lot of advantages to minimizing the number of clusters.

The biggest is going to be utilisation. Combining workloads pools variance, meaning you can safely run at a higher baseline load. But I think that can be achieved more effectively with virtual kubelet .

Re: The future of Kubernetes is virtual machines

#66

Post by jacques_chester »

cyphar wrote: post #27
jacques_chester wrote: post #16

Earlier quoted context omitted.

> It's still containers. The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.

The attack surface of a container can be massively reduced with seccomp profiles -- there was a paper a few years ago which found that the effective attack surface of a hypervisor was about the same as the attack surface of a locked-down seccomp profile of a container (and LXC/Docker/etc already have a default whitelist profile which has in practice mitigated something like 90% of kernel 0days). And let's not forget…

> The attack surface of a container can be massively reduced with seccomp profiles

Yes, though as capabilities are added to the kernel, the profiles have to be updated.

That said, VM or no VM, this should be done no matter what.

> And let's not forget the recent CPU exploits which found that VMs aren't very separated after all.

This is a nil-all draw in terms of the respective security postures, though.

Re: The future of Kubernetes is virtual machines

#67

Post by jacques_chester »

davidopp__ wrote: post #54
georgebarnett wrote: post #28

Earlier quoted context omitted.

> It should be OK to have one big cluster Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale. What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team. There are oper…

A pattern we're seeing a lot of recently is one cluster per "stage" per region, where a "stage" is something like dev/test, canary, and prod. (In some cases only prod is replicated across multiple regions.) I think this may end up being the "sweet spot" for Kubernetes multi-tenancy architecture. The number of clusters isn't quite at the "Kubesprawl" level (I love that phrase and am absolutely going to steal it) -- yo…

The distinction is very helpful and gets at something I was struggling to articulate.

Edit: looking more in the thread, you clearly know this much better than I do. I'd like to get the chance to talk and improve my understanding, if you ever find some spare time.

Re: The future of Kubernetes is virtual machines

#68

Post by hardwaresofton »

trjordan wrote: post #22

This is, like all good HN articles, technically correct and practically incorrect. It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors. It is incorrect because users (e.g. non-in…

> Kubernetes doesn't win by forcing users to think about VMs. Kubernetes wins by adopting a VM standard that can be built by Dockerfiles. Infra engineers will love it.

This is what the CRI does/is, basically. Various projects sprung up to make it possible @ the runtime/kubelet level (kata-containers, frakti, containerd untrusted workloads), but support for runtimeClass[0] is what's going to tie it all together and it's already in Alpha.

Personally I think we should be moving away from docker files -- docker's superior ergonomics pushed the industry forward at the outset, but they lagged in features and compliance with any standards for a long time and were basically usurped. The Dockerfile is a decent format but it lacks a lot of good qualities, like being trivially machine-editable, and the docker client itself has some unfavorable tradeoffs when compared with tools like rkt and podman. I think it's a mistake to standardize on Dockerfiles, but it's definitely a good idea to standardize on the CRI (which Docker now adheres to via a containerd shim by default).

The best thing by far about kubernetes is the CRI[1], CNI[2] (Container Networking Interface), and CSI[3] (Container Storage Interface) standards that are coming out of it. I don't think any one realizes it yet, but they are inadvertently building secure/sandboxed computing for everyone. Containers are just sandboxed processes, and before this, most people were running basically completely unsandboxed processes (both in the cloud and on their personal computers) -- once all this stuff lands, linux is going to have some amazing features for running applications more safely -- production-grade safety for any application you run on your own machine available with standardized tooling.

[0]: https://github.com/kubernetes/enhancements/blob/master/keps/...

[1]: https://github.com/kubernetes/community/blob/master/contribu...

[2]: https://github.com/containernetworking/cni/

[3]: https://kubernetes-csi.github.io/

Re: The future of Kubernetes is virtual machines

#69

Post by ec109685 »

georgebarnett wrote: post #58
ec109685 wrote: post #52

Earlier quoted context omitted.

You have the blast radius reduction by deploying to multiple AWS regions, similar to what AWS does with their infrastructure.

This is true at the machine and network level, however once you overlay a Kubernetes cluster over those constructs you create a new failure domain.

Right, but you have to be able to survive a region going down, so while K8s might increase the chance marginally that your region will go down, you need to be able to handle that eventuality anyway.

Re: The future of Kubernetes is virtual machines

#70

Post by shaklee3 »

mcguire wrote: post #33

As someone who has been a sysadmin and system programmer for 25 years and a crusty rat bastard for almost that long, I have to wonder how long it is going to be until someone realizes that a piece of hardware doing a task is more efficient than 27 layers of virtual machines and a long pair of tongs.

That's kind of the point of k8s. Are you saying containers have too much overhead?
Post reply on HN