Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The future of Kubernetes is virtual machines

tech.paulcz.net

Post reply on HN
Print view View original

21–30 of 128 posts

Re: The future of Kubernetes is virtual machines

#21

Post by jacques_chester »

tptacek wrote: post #20
jacques_chester wrote: post #16

Earlier quoted context omitted.

> It's still containers. The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.

The attack surface argument is debatable, depending on how the system is designers, since virtualization introduces the hypervisor surface.

The hypervisor surface can be made smaller, since its major goal is to manage hardware resources. A kernel has the same mission, but also has a mission to provide a rich API for applications.

Re: The future of Kubernetes is virtual machines

#22

Post by trjordan »

This is, like all good HN articles, technically correct and practically incorrect.

It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors.

It is incorrect because users (e.g. non-infrastructure engineers) don't know or care about the precise definition of containers and VMs are. The point of "containers" is that I can define something that acts like an operating system from the ground up, and it builds quickly and runs quickly in production.

Kubernetes doesn't win by forcing users to think about VMs. Kubernetes wins by adopting a VM standard that can be built by Dockerfiles. Infra engineers will love it.

But besides them? Nobody will care, because Docker for Mac will look the same.

[1] Maybe 1 cluster per region? There's a whole fascinating topic that starts with the question "when building a PaaS, do you expose region placement to devs?" The answer implies a ton of stuff about what exactly it's reasonable to expect from a PaaS and how much infrastructure your average dev has to know.

Re: The future of Kubernetes is virtual machines

#24

Post by jchw »

"virtual machines" may be the wrong word. Containers is probably still the right word. What we probably want, is actually isolated containers.

And hell, you can nearly get that today. Combining Docker with gVisor is a potential solution to the soft tenancy problem as far as I can tell, and Kubernetes supports using it.

(And gVisor is by no stretch of the imagination a 'VM' - it is, at best, a tiny hypervisor, and maybe less than that.)

Re: The future of Kubernetes is virtual machines

#25

Post by bamboozled »

lykr0n wrote: post #18
old-gregg wrote: post #10

Kubernetes has a real chance to succeed where OpenStack failed. Good people at AWS have good reasons to be worried and will push us to a proprietary form of "serverless".

OpenStack's failure is that it's an extremely complex system that looks to be nearly impossible to deploy by yourself. You have to use a Distro that does everything for you- their way. Kubernetes is simple enough to setup yourself. They have well documented tooling, and a solid do it yourself guide. OpenStack has none of that (that I can find). You select RedHat Openstack (RDO) or Canonical OpenStack (MAAS), and you…

The issue with Kubernetes isn't that it's hard to setup, but it's difficult to keep operate long term. Simple things like balancing workloads across availability-zones is still not trivial.

I've heard many stories of people who tried to run Kubernetes themselves in production and didn't have a great experience for it.

Re: The future of Kubernetes is virtual machines

#26

Post by [deleted] »

trjordan wrote: post #22

This is, like all good HN articles, technically correct and practically incorrect. It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors. It is incorrect because users (e.g. non-in…

[deleted]

Re: The future of Kubernetes is virtual machines

#27

Post by cyphar »

jacques_chester wrote: post #16
stevenacreman wrote: post #5

It's still containers. On a cloud provider the Kubernetes workers are VM's which orchestrate containers. With Kata Containers you're just spawning containers inside micro-vm's.

> It's still containers. The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.

The attack surface of a container can be massively reduced with seccomp profiles -- there was a paper a few years ago which found that the effective attack surface of a hypervisor was about the same as the attack surface of a locked-down seccomp profile of a container (and LXC/Docker/etc already have a default whitelist profile which has in practice mitigated something like 90% of kernel 0days).

And let's not forget the recent CPU exploits which found that VMs aren't very separated after all.

The fact that Kubernetes disables this (and other) security features by default should be seen as a flaw in Kubernetes. (Just as some of the flaws of Docker should be seen as Docker flaws not containers-in-general flaws.)

Re: The future of Kubernetes is virtual machines

#28

Post by georgebarnett »

trjordan wrote: post #22

This is, like all good HN articles, technically correct and practically incorrect. It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors. It is incorrect because users (e.g. non-in…

> It should be OK to have one big cluster

Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale.

What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team.

There are operational reasons why having multiple clusters is a good idea. At the simplest level, making a config change and only risking a portion of the infrastructure is an example.

Re: The future of Kubernetes is virtual machines

#29

Post by serverascode »

lykr0n wrote: post #18
old-gregg wrote: post #10

Kubernetes has a real chance to succeed where OpenStack failed. Good people at AWS have good reasons to be worried and will push us to a proprietary form of "serverless".

OpenStack's failure is that it's an extremely complex system that looks to be nearly impossible to deploy by yourself. You have to use a Distro that does everything for you- their way. Kubernetes is simple enough to setup yourself. They have well documented tooling, and a solid do it yourself guide. OpenStack has none of that (that I can find). You select RedHat Openstack (RDO) or Canonical OpenStack (MAAS), and you…

You don't have to use a distro. There are well tested puppet modules, which you could make into your own "distro", as well as openstack-ansible and kolla, and openstack-helm which uses helm to deploy openstack. There is also StarlingX. There is no kubeadm like system however, though I'm not sure how many people will really use kubeadm in prod.

Is it complex? Yes. Is it more complex than k8s? Probably. However, are there multiple open source distros outside of RDO (which is not actually a distro and is instead packaging--see tripleo for a distro like solution based on RDO). MaaS is not an OpenStack distro; it's a way to manage baremetal nodes that OpenStack is then deployed onto using other Canonical related tools. That said, selecting a way to deploy and manage openstack is complex, but the same with k8s.

Re: The future of Kubernetes is virtual machines

#30

Post by justicezyx »

trjordan wrote: post #22

This is, like all good HN articles, technically correct and practically incorrect. It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors. It is incorrect because users (e.g. non-in…

Glad to see people believe there should be one cluster per region, and k8s can be fixed to make that happen.

I am a firm believer of Borg style cluster OS, what Borg lacks is a modern API, but the fundamentals are already there.

Disclaimer: I am with Google's Borg team, with a focus on its client side.

Post reply on HN