Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The future of Kubernetes is virtual machines

tech.paulcz.net

Post reply on HN
Print view View original

51–60 of 128 posts

Re: The future of Kubernetes is virtual machines

#51

Post by geerlingguy »

halbritt wrote: post #48
state_less wrote: post #37

While the use of namespaces seemed a bit discounted in the article, I think part of the problem is that it's an underutilized and poorly conveyed feature, but maybe that's just me? I'd like to see a better out-of-the-box vanilla K8S user management interface in the dashboard. Maybe the default UX on cluster creation is a user/service account that isn't cluster admin that is limited to 1 or more namespaces. There shou…

My org is using namespaces for tenant isolation. Works quite well.

Namespaces++. Unless you’re just running one app in a Kubernetes cluster (where default == that app), namespaces solve a plethora of problems.

Re: The future of Kubernetes is virtual machines

#52

Post by ec109685 »

georgebarnett wrote: post #28
trjordan wrote: post #22

This is, like all good HN articles, technically correct and practically incorrect. It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors. It is incorrect because users (e.g. non-in…

> It should be OK to have one big cluster Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale. What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team. There are oper…

You have the blast radius reduction by deploying to multiple AWS regions, similar to what AWS does with their infrastructure.

Re: The future of Kubernetes is virtual machines

#53

Post by halbritt »

bamboozled wrote: post #25
lykr0n wrote: post #18

Earlier quoted context omitted.

OpenStack's failure is that it's an extremely complex system that looks to be nearly impossible to deploy by yourself. You have to use a Distro that does everything for you- their way. Kubernetes is simple enough to setup yourself. They have well documented tooling, and a solid do it yourself guide. OpenStack has none of that (that I can find). You select RedHat Openstack (RDO) or Canonical OpenStack (MAAS), and you…

The issue with Kubernetes isn't that it's hard to setup, but it's difficult to keep operate long term. Simple things like balancing workloads across availability-zones is still not trivial. I've heard many stories of people who tried to run Kubernetes themselves in production and didn't have a great experience for it.

I've been running it in production for stateful workloads for over a year and it's been going swimmingly. I've never had better uptime or efficiency.

Re: The future of Kubernetes is virtual machines

#54

Post by davidopp__ »

georgebarnett wrote: post #28
trjordan wrote: post #22

This is, like all good HN articles, technically correct and practically incorrect. It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors. It is incorrect because users (e.g. non-in…

> It should be OK to have one big cluster Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale. What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team. There are oper…

A pattern we're seeing a lot of recently is one cluster per "stage" per region, where a "stage" is something like dev/test, canary, and prod. (In some cases only prod is replicated across multiple regions.) I think this may end up being the "sweet spot" for Kubernetes multi-tenancy architecture. The number of clusters isn't quite at the "Kubesprawl" level (I love that phrase and am absolutely going to steal it) -- you can still treat them as pets. But you get good isolation; you can limit access to the prod clusters to only the small set of folks (and perhaps the CD system) authorized to push code there, you can canary Kubernetes upgrades on the canary cluster(s), etc.

As an aside, something that's useful when thinking about Kubernetes multi-tenancy is to understand the distinction between "control plane" multi-tenancy and "data plane" multi-tenancy. Data plane multi-tenancy is about making it safe to share a node (or network) among multiple untrusting users and/or workloads. Examples of existing features for data plane multi-tenancy are gVisor/Kata, PodSecurityPolicy, and NetworkPolicy. Control plane multi-tenancy is about making it safe to share the cluster control plane among multiple untrusting users and/or workloads. Examples of existing features for control plane multi-tenancy are RBAC, ResourceQuota (particularly quota on number of objects; quota on things like cpu and memory are arguably data plane), and the EventRateLimit admission controller.

There's active work in the Kubernetes community in both of these areas; if you'd like to participate (or lurk), please join the kubernetes-wg-multi-tenancy mailing list: http://groups.google.com/forum/#!forum/kubernetes-wg-multite...

Also, I gave a talk at KubeCon EU earlier this year that gives a rough overview of Kubernetes multi-tenancy, that might be of interest to some folks: https://kccnceu18.sched.com/event/Dqvb?iframe=no (links to the slides and YouTube video are near the bottom of the page)

Disclosure: I work at Google on Kubernetes and GKE.

Re: The future of Kubernetes is virtual machines

#55

Post by halbritt »

geerlingguy wrote: post #51
halbritt wrote: post #48

Earlier quoted context omitted.

My org is using namespaces for tenant isolation. Works quite well.

Namespaces++. Unless you’re just running one app in a Kubernetes cluster (where default == that app), namespaces solve a plethora of problems.

Resource limits, network policy boundaries, etc.

Re: The future of Kubernetes is virtual machines

#56

Post by georgebarnett »

trjordan wrote: post #43
georgebarnett wrote: post #28

Earlier quoted context omitted.

> It should be OK to have one big cluster Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale. What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team. There are oper…

Config blast radius, hardware sizing, networking partitions, disaster recovery, dev/prod separation -- lots of good reasons to have multiple clusters! As a gut check, most of these reasons apply to AWS AZs as well. If your Kubernetes strategy calls for more than one per region per AWS account, it means that you, organizationally, don't trust containers as much as you trust VMs on AWS. And right now, you're probably r…

I don't understand your argument suggesting I don't trust containers because I would have more than one Kube cluster per region/account.

Multiple Kubernetes clusters for workload separation and AWS accounts for workload separation aren't quiet the same and bring different levels of complexity depending on what your internal processes look like.

Re: The future of Kubernetes is virtual machines

#57

Post by halbritt »

tptacek wrote: post #20
jacques_chester wrote: post #16

Earlier quoted context omitted.

> It's still containers. The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.

The attack surface argument is debatable, depending on how the system is designers, since virtualization introduces the hypervisor surface.

The attack surface argument certainly is debatable.

I wonder how many multi-tenant workloads are actually at risk of an escape vulnerability. I wager that the multi-tenancy described in the article in the OP is actually disparate workloads across disparate teams in a particular enterprise where it seems (to me) fairly unlikely for someone with access to run a workload to also have the willingness to compile and run malicious code to take advantage of an escape vulnerability.

On the other hand, publicly available compute, i.e. AWS, GCP, Azure seems way more likely to be the subject of attacks from random malicious individuals seek to take advantage of an escape vulnerability if one existed.

Re: The future of Kubernetes is virtual machines

#58

Post by georgebarnett »

ec109685 wrote: post #52
georgebarnett wrote: post #28

Earlier quoted context omitted.

> It should be OK to have one big cluster Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale. What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team. There are oper…

You have the blast radius reduction by deploying to multiple AWS regions, similar to what AWS does with their infrastructure.

This is true at the machine and network level, however once you overlay a Kubernetes cluster over those constructs you create a new failure domain.

Re: The future of Kubernetes is virtual machines

#59

Post by 013a »

I really worry about cost in this future of leaning on something like ACI/Fargate to actually run the containers.

An m5.large instance (2vcpu/8gb) costs $70/mo on-demand ($44/mo with a 1 year reservation). A similar Fargate runtime costs $146/mo.

A b2ms Azure instance (2vcpu/8gb) costs $60/mo on-demand ($39/mo 1 year reservation). Azure Container Instances at a similar provisioning level costs $176/mo by my calculations.

That's not a small difference. That's, like, 3x.

Point being, I love virtual-kublet from the perspective of a scale-up just trying to get a product out the door. But for established companies, I still think the core idea of a container on a VM you control is going to rule. Fortunately Kubernetes allows amazingly easy flexibility to switch, and that's a reason why it might be the most important technology created in recent history.

Re: The future of Kubernetes is virtual machines

#60

Post by davidopp__ »

jacques_chester wrote: post #15
amscanne wrote: post #6

An important element of Kubernetes is that it standardizes the infrastructure control plane, and allows different pieces to be plugged in (for networking, storage, etc.). The "virtual kubelet" essentially throws that all that away and keeps Kubernetes "in API only". For example, with virtual kubelets, scheduling is meaningless and networking and storage are restricted to whatever the virtual kubelet target supports (…

> Personally, I think the value proposition is tenuous Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable. Relying on the cluster as the security boundary is very effective ... and very wasteful. > Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but…

I don't think multi-tenancy has been "retrofitted" onto Kubernetes. Kubernetes was designed with multi-tenancy in mind from the very early releases -- namespaces, authn/authz (initially ABAC, later RBAC), ResourceQuota, PodSecurityPolicy, etc. New features are added over time, such as NetworkPolicy (which has been in Kubernetes for a year and a half, so perhaps not "new" anymore!), EventRateLimit, and others, but always in a principled way. And the integration of container isolation technologies like gVisor and Kata are using a standard Kubernetes extension point (the Container Runtime Interface) so I do not view this work as retrofitting.

Moreover, even today there are real public PaaSes that expose the Kubernetes API served by a multi-tenant Kubernetes cluster to mutually untrusting end-users, e.g. OpenShift Online and one of the Huawei cloud products (I forget which one). Obviously Kubernetes multi-tenancy isn't going to be secure enough today for everyone, especially folks who want an additional layer of isolation on top of cgroups/namespaces/seccomp/AppArmor/etc., but there are a lot of advantages to minimizing the number of clusters. (See my other comment in this thread about the pattern we frequently see of separate clusters for dev/test vs. staging vs. prod, possibly per region, but sharing each of those among multiple users and/or applications.)

Disclosure: I work at Google on Kubernetes and GKE.

Post reply on HN