Earlier quoted context omitted.
That is one big difference, with big implications.
Just wanted to clarify that Kubernetes was still scheduling containers. Even if VM's are being used to isolate them. It's not all or nothing either. Containerd will support running a mix of containers and kata-containers across workers. For anyone interested in this topic I wrote about some other container runtimes here: https://kubedex.com/kubernetes-container-runtimes/
The future of Kubernetes is virtual machines
11–20 of 128 posts
Re: The future of Kubernetes is virtual machines
#12An important element of Kubernetes is that it standardizes the infrastructure control plane, and allows different pieces to be plugged in (for networking, storage, etc.). The "virtual kubelet" essentially throws that all that away and keeps Kubernetes "in API only". For example, with virtual kubelets, scheduling is meaningless and networking and storage are restricted to whatever the virtual kubelet target supports (…
Re: The future of Kubernetes is virtual machines
#13Kubernetes has a real chance to succeed where OpenStack failed. Good people at AWS have good reasons to be worried and will push us to a proprietary form of "serverless".
Re: The future of Kubernetes is virtual machines
#14Earlier quoted context omitted.
That is one big difference, with big implications.
Just wanted to clarify that Kubernetes was still scheduling containers. Even if VM's are being used to isolate them. It's not all or nothing either. Containerd will support running a mix of containers and kata-containers across workers. For anyone interested in this topic I wrote about some other container runtimes here: https://kubedex.com/kubernetes-container-runtimes/
Re: The future of Kubernetes is virtual machines
#15An important element of Kubernetes is that it standardizes the infrastructure control plane, and allows different pieces to be plugged in (for networking, storage, etc.). The "virtual kubelet" essentially throws that all that away and keeps Kubernetes "in API only". For example, with virtual kubelets, scheduling is meaningless and networking and storage are restricted to whatever the virtual kubelet target supports (…
Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable.
Relying on the cluster as the security boundary is very effective ... and very wasteful.
> Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but I don't think the answer is to reduce the standard to what we have today (a bunch of disparate VMs).
I think the argument is that rather than the painful (and it will be very painful) and probably incomplete quest to retrofit multi-tenancy into a single-tenancy design, we can introduce multi-tenancy where it basically actually matters: at the worker node.
At first glance it's confusing to go from "one master, many nodes" to "one node pool, many masters". But it actually works better on every front. Workload efficiency goes up. Security surface area between masters becomes close to nil.
Very cheap VMs are the means to that end.
Disclosure: I work for Pivotal and this argument fits our basic doctrine of how Kubernetes ought to be used.
Re: The future of Kubernetes is virtual machines
#16It's still containers. On a cloud provider the Kubernetes workers are VM's which orchestrate containers. With Kata Containers you're just spawning containers inside micro-vm's.
The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.
Re: The future of Kubernetes is virtual machines
#17An important element of Kubernetes is that it standardizes the infrastructure control plane, and allows different pieces to be plugged in (for networking, storage, etc.). The "virtual kubelet" essentially throws that all that away and keeps Kubernetes "in API only". For example, with virtual kubelets, scheduling is meaningless and networking and storage are restricted to whatever the virtual kubelet target supports (…
> Personally, I think the value proposition is tenuous Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable. Relying on the cluster as the security boundary is very effective ... and very wasteful. > Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but…
For a single cluster, "very cheap" VMs solve some of the problems, but leave others unsolved (e.g. they prevent some hardware and kernel exploits, but lots of security issues can still hit you -- like the last two big K8s CVEs). They also sacrifice a lot of the things that make containers compelling on the floor (high efficiency and density), so I don't think they should be spun as a panecea.
You seem to be arguing that one shouldn't bother with multi-tenancy on a single cluster, which is a fine approach, but I do think that the technologies and tools to support the single cluster model are evolving. Calling it a "multi-tenancy retrofit" seems a bit FUD-y to me. Just because there are challenges doesn't mean it's not worth doing.
Re: The future of Kubernetes is virtual machines
#18Kubernetes has a real chance to succeed where OpenStack failed. Good people at AWS have good reasons to be worried and will push us to a proprietary form of "serverless".
Kubernetes is simple enough to setup yourself. They have well documented tooling, and a solid do it yourself guide. OpenStack has none of that (that I can find). You select RedHat Openstack (RDO) or Canonical OpenStack (MAAS), and you have to use their all-in-one system to have a deployment- and that requires a narrow set of variables which every environment might not have. Which is insane- and will hinder adoption.
EDIT. Not 100% correct, see below comments.
Re: The future of Kubernetes is virtual machines
#19Earlier quoted context omitted.
> Personally, I think the value proposition is tenuous Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable. Relying on the cluster as the security boundary is very effective ... and very wasteful. > Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but…
I'm saying that I think the value proposition for the virtual kubelet is tenuous, not multi-tenancy as a whole. For a single cluster, "very cheap" VMs solve some of the problems, but leave others unsolved (e.g. they prevent some hardware and kernel exploits, but lots of security issues can still hit you -- like the last two big K8s CVEs). They also sacrifice a lot of the things that make containers compelling on the…
I was tying them together because I see the former as an effective strategy to achieve the latter.
> Calling it a "multi-tenancy retrofit" seems a bit FUD-y to me. Just because there are challenges doesn't mean it's not worth doing.
What should I call it? It's being added retrospectively to a single-tenant design. The changes have to be correctly threaded through everything, through codebases managed by dozens of working groups, without breaking thousands of existing extensions, tools and applications.
What I expect will happen instead is that it will be better than it is now -- which is a win -- but that no complete, mandatorily-secure, top-to-bottom security boundaries will be created inside single clusters. We will still be left with lots of leaks.
Our industry is replete with folks trying to wedge the business of hypervisors and supervisors into applications and services. It's possible but always leaks and breaks and diverts enormous development bandwidth away from the core thing that is meant to be achieved. Kernels and hypervisors have privileged hardware access and decades of hardening that can't be truly replicated at the application or service level and which when imitated need to be designed in from the beginning.
I don't see that as FUD. I think it just is what it is. But I appreciate that my thinking is line with the doctrine Pivotal advances to its customers, which differs from the doctrine Red Hat and others advance (One Cluster To Rule Them All).
Re: The future of Kubernetes is virtual machines
#20It's still containers. On a cloud provider the Kubernetes workers are VM's which orchestrate containers. With Kata Containers you're just spawning containers inside micro-vm's.
> It's still containers. The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.