Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The future of Kubernetes is virtual machines

tech.paulcz.net

Post reply on HN
Print view View original

71–80 of 128 posts

Re: The future of Kubernetes is virtual machines

#71

Post by smarterclayton »

jacques_chester wrote: post #19
amscanne wrote: post #17

Earlier quoted context omitted.

I'm saying that I think the value proposition for the virtual kubelet is tenuous, not multi-tenancy as a whole. For a single cluster, "very cheap" VMs solve some of the problems, but leave others unsolved (e.g. they prevent some hardware and kernel exploits, but lots of security issues can still hit you -- like the last two big K8s CVEs). They also sacrifice a lot of the things that make containers compelling on the…

> I'm saying that I think the value proposition for the virtual kubelet is tenuous, not multi-tenancy as a whole. I was tying them together because I see the former as an effective strategy to achieve the latter. > Calling it a "multi-tenancy retrofit" seems a bit FUD-y to me. Just because there are challenges doesn't mean it's not worth doing. What should I call it? It's being added retrospectively to a single-tenan…

I’m not sure who at Red Hat is advocating one cluster to rule them all, but it’s just one point on the spectrum. There are lots of places where one cluster makes sense and two would be overkill - if you want to run lots of simple workloads, or have one very large scale app. But it’s equally smart to separate clusters by security domain or regulatory zone, or to create partitions to force your teams to treat clusters as fungible.

If there’s Red Hat documentation advising silly absolutes please let me know and I’ll make sure it gets fixed.

Re: The future of Kubernetes is virtual machines

#72

Post by smarterclayton »

halbritt wrote: post #55
geerlingguy wrote: post #51

Earlier quoted context omitted.

Namespaces++. Unless you’re just running one app in a Kubernetes cluster (where default == that app), namespaces solve a plethora of problems.

Resource limits, network policy boundaries, etc.

That’s what we designed them for :)

I do think we’ve not explored enough of the per namespace policy stuff though - i’d like both podpreset and a reasonably simple scheduling policy (toleration + node selector control to replace the annotation based system) to make it in, as well as a simpler namespace initialization path so you can more easily lock down the contents of a namespace without having to proxy the create namespace API call.

Re: The future of Kubernetes is virtual machines

#73

Post by halbritt »

lykr0n wrote: post #63
halbritt wrote: post #50

Earlier quoted context omitted.

One of the issues with Openstack is managing the services required to run it. I imagine that helm could be used to make it easier to run these services in kubernetes.

True, but that adds another layer of complexity to an already complex system. If I'm using the ansible module, that's complex enough. Throwing in management of Kubernetes and the additional cruft containers adds- it seems like it's a lot of hassle for little gain.

I see this argument relatively frequently. "Why use Kubernetes when I can accomplish X with [puppet|chef|ansible]".

The answer is that k8s offers something fundamentally different and until the person posing the question gets that distinction, the argument is relatively pointless.

I'm not just bowing behind the argument that "you just don't get it... man". Let me point out that you're right. You can manage the openstack control plane perfectly well with your configuration management tool of choice and if you have that process really dialed, then you'll have a difficult time improving upon it with something like k8s.

Re: The future of Kubernetes is virtual machines

#74

Post by halbritt »

smarterclayton wrote: post #72
halbritt wrote: post #55

Earlier quoted context omitted.

Resource limits, network policy boundaries, etc.

That’s what we designed them for :) I do think we’ve not explored enough of the per namespace policy stuff though - i’d like both podpreset and a reasonably simple scheduling policy (toleration + node selector control to replace the annotation based system) to make it in, as well as a simpler namespace initialization path so you can more easily lock down the contents of a namespace without having to proxy the create…

By podreset are you referring to something like poddisruptionbudget?

Because it'd be neat to define that on a per namespace basis.

Re: The future of Kubernetes is virtual machines

#75

Post by smarterclayton »

halbritt wrote: post #74
smarterclayton wrote: post #72

Earlier quoted context omitted.

That’s what we designed them for :) I do think we’ve not explored enough of the per namespace policy stuff though - i’d like both podpreset and a reasonably simple scheduling policy (toleration + node selector control to replace the annotation based system) to make it in, as well as a simpler namespace initialization path so you can more easily lock down the contents of a namespace without having to proxy the create…

By podreset are you referring to something like poddisruptionbudget? Because it'd be neat to define that on a per namespace basis.

PodPreset went alpha with service catalog but hasn’t made it out to beta yet. It makes certain forms of injection / rules easier (you must use a standard log dir, you should use the provided HTTP_PROXY vars, etc). https://kubernetes.io/docs/concepts/workloads/pods/podpreset...

Being able to limit a user to only being able to edit one pod preset or scheduling policy (via rbac name access) would provide some useful flexibility for splitting control between admin and namespace user.

Re: The future of Kubernetes is virtual machines

#76

Post by markbnj »

I get the multi-tenancy argument, for certain use cases. I'm not sure I understand the point about greater resource utilization. Presumably they mean workloads can be scheduled more densely given a set of hardware resources, vs. containers... but I'd think that containers would score better on that metric than VMs. Can someone expound?

The best part of the piece for me was "kubesprawl," my new favorite word for the week. We've seen it ourselves to some extent, but we are at least aware of it and try to exert some pressure in the other direction. Beyond that I am not particularly bothered by the idea of running lots of clusters for different purposes.

Re: The future of Kubernetes is virtual machines

#77

Post by halbritt »

smarterclayton wrote: post #75
halbritt wrote: post #74

Earlier quoted context omitted.

By podreset are you referring to something like poddisruptionbudget? Because it'd be neat to define that on a per namespace basis.

PodPreset went alpha with service catalog but hasn’t made it out to beta yet. It makes certain forms of injection / rules easier (you must use a standard log dir, you should use the provided HTTP_PROXY vars, etc). https://kubernetes.io/docs/concepts/workloads/pods/podpreset... Being able to limit a user to only being able to edit one pod preset or scheduling policy (via rbac name access) would provide some useful fle…

Okay, I get what your point.

I'm still living in the world where operating a platform for the benefit of a set of developers entails building and operating a set of services that abstracts the details of the infrastructure sufficiently that these things don't matter.

Re: The future of Kubernetes is virtual machines

#78

Post by jacques_chester »

smarterclayton wrote: post #71
jacques_chester wrote: post #19

Earlier quoted context omitted.

> I'm saying that I think the value proposition for the virtual kubelet is tenuous, not multi-tenancy as a whole. I was tying them together because I see the former as an effective strategy to achieve the latter. > Calling it a "multi-tenancy retrofit" seems a bit FUD-y to me. Just because there are challenges doesn't mean it's not worth doing. What should I call it? It's being added retrospectively to a single-tenan…

I’m not sure who at Red Hat is advocating one cluster to rule them all, but it’s just one point on the spectrum. There are lots of places where one cluster makes sense and two would be overkill - if you want to run lots of simple workloads, or have one very large scale app. But it’s equally smart to separate clusters by security domain or regulatory zone, or to create partitions to force your teams to treat clusters…

I don't have an example to hand, so it's obvious I went on second-hand accounts. Do you have something you'd normally point customers to when describing the tradeoffs?

For myself I see the argument for fewer clusters as about utilisation, the argument for more clusters about isolation. It's the oldest tug-of-war in computing. I think that shared node pools for multiple masters is going to be the combination that for most workloads will increase utilisation without greatly weakening isolation. I don't think multi-tenancy in the master will be as easily achieved or as effective.

Re: The future of Kubernetes is virtual machines

#79

Post by stubish »

mcguire wrote: post #33

As someone who has been a sysadmin and system programmer for 25 years and a crusty rat bastard for almost that long, I have to wonder how long it is going to be until someone realizes that a piece of hardware doing a task is more efficient than 27 layers of virtual machines and a long pair of tongs.

This is why things like MAAS exist, which is a 'cloud provider' that lets you provision bare metal machines. Plug in a new rack of servers, let them netboot, and let the clients spin up a machine matching their requirements in the same way they would spin up an AWS or Azure instance.

We mix bare metal spun up this way with Open Stack VMs, depending on the requirements. And slice up into containers using LXD when that makes sense.

Re: The future of Kubernetes is virtual machines

#80

Post by davidopp__ »

jacques_chester wrote: post #65
davidopp__ wrote: post #60

Earlier quoted context omitted.

I don't think multi-tenancy has been "retrofitted" onto Kubernetes. Kubernetes was designed with multi-tenancy in mind from the very early releases -- namespaces, authn/authz (initially ABAC, later RBAC), ResourceQuota, PodSecurityPolicy, etc. New features are added over time, such as NetworkPolicy (which has been in Kubernetes for a year and a half, so perhaps not "new" anymore!), EventRateLimit, and others, but alw…

> I don't think multi-tenancy has been "retrofitted" onto Kubernetes. Kubernetes was designed with multi-tenancy in mind from the very early releases -- namespaces, authn/authz (initially ABAC, later RBAC), ResourceQuota, PodSecurityPolicy, etc. My complaint is that these require assembly and are in many cases opt-in (making RBAC opt-out was a massive leap forward). Namespaces are the lynchpin, but are globally visib…

> The biggest is going to be utilisation. Combining workloads pools variance, meaning you can safely run at a higher baseline load.

Utilization is arguably the biggest benefit (fewer nodes if you can share nodes among users/workloads, fewer masters if you can share the control plane among users/workloads), but I wouldn't under-estimate the manageability benefit of having fewer clusters to run. Also, for applications (or application instances, e.g. in the case of a SaaS) that are short-lived, the amount of time it takes to spin up a new cluster to serve that application (instance) can cause a poor user experience; spinning up a new namespace and pod(s) in an existing multi-tenant cluster is much faster.

> But I think that can be achieved more effectively with virtual kubelet .

I think it's hard to compare virtual kubelet to something like Kata Containers, gVisor, or Firecracker. You can put almost anything at the other end of a virtual kubelet, and as others have pointed out in this thread virtual kubelet doesn't provide the full Kubelet API (and thus you can't use the full Kubernetes API against it). At a minimum I think it's important to specify what is backing the virtual kubelet, and what Kubernetes features you need, in order to compare it with isolation technologies like the others I mentioned.

Disclosure: I work at Google on Kubernetes and GKE.

Post reply on HN