Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

blog.stackademic.com

Post reply on HN
Print view View original

131–140 of 216 posts

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#131

Post by andix »

figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

Isn't one of the strategies also to run one or two backup clusters for any production cluster? Which can take over the workloads if the primary cluster fails for some reason?

In a cloud environment the backup cluster can be scaled up quickly if it has to take over, so while it's idling it only requires a few smaller nodes.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#133

Post by nwatson »

anthonybsd wrote: post #129
mrweasel wrote: post #96

Earlier quoted context omitted.

> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long wa…

>Often you'd need production, staging, test, development, something like that. Normally in K8s, segregating environments is done via namespaces, not clusters (unless there are some very specific resource constraints).

The constraint often would be regulatory. Even if technically isolation is possible, management won't risk SOC2 or GDPR non-compliance.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#134

Post by Elidrake24 »

anthonybsd wrote: post #129
mrweasel wrote: post #96

Earlier quoted context omitted.

> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long wa…

>Often you'd need production, staging, test, development, something like that. Normally in K8s, segregating environments is done via namespaces, not clusters (unless there are some very specific resource constraints).

Which in many cases would break SOC2 compliance (co-mingling of development and customer resources), and even goes against the basic advice offered in the K8s manual. Beyond that, this limits your ability to test Control Plane upgrades against your stack, though that has generally been very stable in my experience.

To be clear I'm not defending the 47 Cluster setup of the OP, just the practice of separating Development/Production.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#135

Post by wbl »

AtlasBarfed wrote: post #127
mrweasel wrote: post #96

Earlier quoted context omitted.

> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long wa…

K8s requires a flat plane addressability model across all containers, meaning anyone can see and call anyone else? I can see security teams getting uppity about that. Also budgetary and org boundaries, cloud providers, disaster recovery/hot spares/redundancy/AB hotswap, avoid single tank point of failure.

Addressability is not accessibility . It's easy to control how services talk to each other through NetworkPolicy.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#136

Post by AstroJetson »

astura wrote: post #111
torginus wrote: post #70

Earlier quoted context omitted.

Because they have k8s engineers each of whom wants to put it on their resume that they designed and implemented a working cluster in prod.

Resume Driven Development.

Wouldn't that give you a prod mix of Kubernetes, Docker, Fargate, Rancher, etc?

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#138

Post by teeray »

this_user wrote: post #130
figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

It's just a matter of time before someone releases an orchestration layer for k8s clusters so the absurd Rube Goldberg machine that is modern devops stacks can grow even more complex.

We can call it k8tamari

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#139

Post by tinco »

karmarepellent wrote: post #85
tinco wrote: post #73

Earlier quoted context omitted.

You need a separate cluster for production because there are operations you'd do your staging/QA environments that might accidentally knock out your cluster, I did that once and it was not fun. I completely agree with keeping everything as simple as possible though. No extra clusters if not absolutely necessary, and also no extra namespaces if not absolutely necessary. The thing with Kubernetes is that it was designe…

Do you mind sharing what these operations were? I can think of a few things that may very well brick your control plane. But at the very least existing workloads continue to function in this case as far as I know. Same with e.g. misconfigured network policies. Those might cause downtimes, but at least you can roll them back easily. This was some time ago though. There may be more footguns now. Curious to know how you…

In my case, it was the ingress running out of subdomains because each staging environment would get its own subdomain, and our system had a bug that caused them to not be cleaned up. So the CI/CD was leaking subdomains, eventually the list became too long and it bumped the production domain off the list.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#140

Post by bdndndndbve »

anthonybsd wrote: post #129
mrweasel wrote: post #96

Earlier quoted context omitted.

> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long wa…

>Often you'd need production, staging, test, development, something like that. Normally in K8s, segregating environments is done via namespaces, not clusters (unless there are some very specific resource constraints).

I would want to have at least dev + prod clusters, sometimes people want to test controllers or they have badly behaved workloads that k8s doesn't isolate well (making lots of giant etcd objects). You can also test k8s version upgrades in non-prod.

That said it sounds like these people just made a cluster per service which adds a ton of complexity and loses all the benefits of k8s.

Post reply on HN