Live data from Hacker News

The future of Kubernetes is virtual machines

tech.paulcz.net

11–20 of 128 posts

Re: The future of Kubernetes is virtual machines

#11
post #7

Earlier quoted context omitted.

That is one big difference, with big implications.

Just wanted to clarify that Kubernetes was still scheduling containers. Even if VM's are being used to isolate them. It's not all or nothing either. Containerd will support running a mix of containers and kata-containers across workers. For anyone interested in this topic I wrote about some other container runtimes here: https://kubedex.com/kubernetes-container-runtimes/

Note that this is not the case for virtual kubelet-based implementations, and your point here and above are specific to how Kata works (the article is talking more generally).

Re: The future of Kubernetes is virtual machines

#12
post #6

An important element of Kubernetes is that it standardizes the infrastructure control plane, and allows different pieces to be plugged in (for networking, storage, etc.). The "virtual kubelet" essentially throws that all that away and keeps Kubernetes "in API only". For example, with virtual kubelets, scheduling is meaningless and networking and storage are restricted to whatever the virtual kubelet target supports (…

I agree, I'm not sure the virtual kubelet concept is a great idea overall. It sounds good on the surface, but most of the time these kind of abstractions do more harm then good, as far as I'm concerned. But, I could be wrong. :)

Re: The future of Kubernetes is virtual machines

#13

Kubernetes has a real chance to succeed where OpenStack failed. Good people at AWS have good reasons to be worried and will push us to a proprietary form of "serverless".

I think they are quite different solutions, and each benefits from the other. Further, they also tend to run into the same concerns at the about the same time in their life time, eg. concerns around complexity.

Re: The future of Kubernetes is virtual machines

#14
post #7

Earlier quoted context omitted.

That is one big difference, with big implications.

Just wanted to clarify that Kubernetes was still scheduling containers. Even if VM's are being used to isolate them. It's not all or nothing either. Containerd will support running a mix of containers and kata-containers across workers. For anyone interested in this topic I wrote about some other container runtimes here: https://kubedex.com/kubernetes-container-runtimes/

Yeah.. I think I see what you say, I mean, the end user interface is the quite the same but this has big implications anyway for the systems design POV, so is not a small thing :)

Re: The future of Kubernetes is virtual machines

#15
post #6

An important element of Kubernetes is that it standardizes the infrastructure control plane, and allows different pieces to be plugged in (for networking, storage, etc.). The "virtual kubelet" essentially throws that all that away and keeps Kubernetes "in API only". For example, with virtual kubelets, scheduling is meaningless and networking and storage are restricted to whatever the virtual kubelet target supports (…

> Personally, I think the value proposition is tenuous

Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable.

Relying on the cluster as the security boundary is very effective ... and very wasteful.

> Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but I don't think the answer is to reduce the standard to what we have today (a bunch of disparate VMs).

I think the argument is that rather than the painful (and it will be very painful) and probably incomplete quest to retrofit multi-tenancy into a single-tenancy design, we can introduce multi-tenancy where it basically actually matters: at the worker node.

At first glance it's confusing to go from "one master, many nodes" to "one node pool, many masters". But it actually works better on every front. Workload efficiency goes up. Security surface area between masters becomes close to nil.

Very cheap VMs are the means to that end.

Disclosure: I work for Pivotal and this argument fits our basic doctrine of how Kubernetes ought to be used.

Re: The future of Kubernetes is virtual machines

#16

It's still containers. On a cloud provider the Kubernetes workers are VM's which orchestrate containers. With Kata Containers you're just spawning containers inside micro-vm's.

> It's still containers.

The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.

Re: The future of Kubernetes is virtual machines

#17
post #6

An important element of Kubernetes is that it standardizes the infrastructure control plane, and allows different pieces to be plugged in (for networking, storage, etc.). The "virtual kubelet" essentially throws that all that away and keeps Kubernetes "in API only". For example, with virtual kubelets, scheduling is meaningless and networking and storage are restricted to whatever the virtual kubelet target supports (…

> Personally, I think the value proposition is tenuous Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable. Relying on the cluster as the security boundary is very effective ... and very wasteful. > Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but…

I'm saying that I think the value proposition for the virtual kubelet is tenuous, not multi-tenancy as a whole.

For a single cluster, "very cheap" VMs solve some of the problems, but leave others unsolved (e.g. they prevent some hardware and kernel exploits, but lots of security issues can still hit you -- like the last two big K8s CVEs). They also sacrifice a lot of the things that make containers compelling on the floor (high efficiency and density), so I don't think they should be spun as a panecea.

You seem to be arguing that one shouldn't bother with multi-tenancy on a single cluster, which is a fine approach, but I do think that the technologies and tools to support the single cluster model are evolving. Calling it a "multi-tenancy retrofit" seems a bit FUD-y to me. Just because there are challenges doesn't mean it's not worth doing.

Re: The future of Kubernetes is virtual machines

#18

Kubernetes has a real chance to succeed where OpenStack failed. Good people at AWS have good reasons to be worried and will push us to a proprietary form of "serverless".

OpenStack's failure is that it's an extremely complex system that looks to be nearly impossible to deploy by yourself. You have to use a Distro that does everything for you- their way.

Kubernetes is simple enough to setup yourself. They have well documented tooling, and a solid do it yourself guide. OpenStack has none of that (that I can find). You select RedHat Openstack (RDO) or Canonical OpenStack (MAAS), and you have to use their all-in-one system to have a deployment- and that requires a narrow set of variables which every environment might not have. Which is insane- and will hinder adoption.

EDIT. Not 100% correct, see below comments.

Re: The future of Kubernetes is virtual machines

#19

Earlier quoted context omitted.

> Personally, I think the value proposition is tenuous Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable. Relying on the cluster as the security boundary is very effective ... and very wasteful. > Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but…

I'm saying that I think the value proposition for the virtual kubelet is tenuous, not multi-tenancy as a whole. For a single cluster, "very cheap" VMs solve some of the problems, but leave others unsolved (e.g. they prevent some hardware and kernel exploits, but lots of security issues can still hit you -- like the last two big K8s CVEs). They also sacrifice a lot of the things that make containers compelling on the…

> I'm saying that I think the value proposition for the virtual kubelet is tenuous, not multi-tenancy as a whole.

I was tying them together because I see the former as an effective strategy to achieve the latter.

> Calling it a "multi-tenancy retrofit" seems a bit FUD-y to me. Just because there are challenges doesn't mean it's not worth doing.

What should I call it? It's being added retrospectively to a single-tenant design. The changes have to be correctly threaded through everything, through codebases managed by dozens of working groups, without breaking thousands of existing extensions, tools and applications.

What I expect will happen instead is that it will be better than it is now -- which is a win -- but that no complete, mandatorily-secure, top-to-bottom security boundaries will be created inside single clusters. We will still be left with lots of leaks.

Our industry is replete with folks trying to wedge the business of hypervisors and supervisors into applications and services. It's possible but always leaks and breaks and diverts enormous development bandwidth away from the core thing that is meant to be achieved. Kernels and hypervisors have privileged hardware access and decades of hardening that can't be truly replicated at the application or service level and which when imitated need to be designed in from the beginning.

I don't see that as FUD. I think it just is what it is. But I appreciate that my thinking is line with the doctrine Pivotal advances to its customers, which differs from the doctrine Red Hat and others advance (One Cluster To Rule Them All).

Re: The future of Kubernetes is virtual machines

#20

It's still containers. On a cloud provider the Kubernetes workers are VM's which orchestrate containers. With Kata Containers you're just spawning containers inside micro-vm's.

> It's still containers. The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.

The attack surface argument is debatable, depending on how the system is designers, since virtualization introduces the hypervisor surface.
Post reply on HN