Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

blog.stackademic.com

Post reply on HN
Print view View original

51–60 of 216 posts

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#51

Post by karmarepellent »

figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

We ran only two (very small) clusters for some time in the past and even then it introduced some unnecessary overhead on the ops side and some headaches on the dev side. Maybe they were just growing pains, but if I have to run Kubernetes again I will definitely opt for a single large cluster.

After all Kubernetes provides all the primitives you need to enforce separation. You wouldn't create separate VMWare production and test clusters either unless you have a good reason.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#52

Post by raverbashing »

pm90 wrote: post #27
figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

There are genuine reasons for running multiple clusters. It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation so they get their own cluster (although somehow its ok that the nodes are still VMs that are probably running on shared nodes… these requirements can be arbitrary sometimes).

This doesn't make sense to me, and I feel like this is "holding it wrong"

But this is also a caveat with "managing it yourself", you get a lot of people having ideas and shooting themselves in the foot with it

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#53

Post by tonfreed »

figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

I work for a large corp and we have three for apps (dev, integrated testing, prod) plus I think two or three more for the platform team that I don't interact with. 47 seems horrendously excessive

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#55

Post by karmarepellent »

adamtulinius wrote: post #50
mrkeen wrote: post #47

> DevOps Team Is Happier Than Ever Or course they are. The original value proposition of cloud providers managing your infra (and moreso with k8s) was that you could fire your ops team (now called "DevOps" because the whole idea didn't pan out) and the developers could manage their services directly. In any case, your DevOps team has job security now.

It doesn't take any more competent people to self-host a modern stack, than it does to babysit how a company uses something like Azure. The original value proposition is false, and more and more are realising this.

I think the value proposition holds when you are just getting started with your company and you happen to employ people that know their way around the hyperscaler cloud ecosystems.

But I agree that moving your own infra or outsourcing operations when you have managed to do it on your own for a while is most likely misguided. Speaking from experience it introduces costs that cannot possibly be calculdated before the fact and thus always end up more complicated and costlier than the suits imagined.

In the past, when similar decicions were made, I always thought to myself: You could have just hired one more person bringing their own, fresh perspective on what we are doing in order to improve our ops game.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#56

Post by timhigins »

If you have 200 YAML files for a single service and 46 clusters I think you're using k8s wrong. And 5 + 3 different monitoring and logging tools could be a symptom of chaos in the organization.

k8s, and the go runtime and network stack have been heavily optimized by armies of engineers at Google and big tech, so I am very suspicious of these claims without evidence. Show me the resource usage from k8s component overhead, and the 15 minute to 3 minute deploys and then I'll believe you. And the 200 file YAML or Helm charts so I can understand why in gods name you're doing it that way.

This post just needs a lot more details. What are the typical services/workloads running on k8s? What's the end user application?

I taught myself k8s in the first month of my first job, and it felt like having super powers. The core concepts are very beautiful, like processes on Linux or JSON APIs over HTTP. And its not too hard to build a CustomResourceDefinition or dive into the various high performance disk and network IO components if you need to.

I do hate Helm to some degree but there are alternatives like Kustomize, Jsonnet/Tanka, or Cue https://github.com/cue-labs/cue-by-example/tree/main/003_kub.... You can even manage k8s resources via Terraform or Pulumi

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#57

Post by [deleted] »

timhigins wrote: post #56

If you have 200 YAML files for a single service and 46 clusters I think you're using k8s wrong. And 5 + 3 different monitoring and logging tools could be a symptom of chaos in the organization. k8s, and the go runtime and network stack have been heavily optimized by armies of engineers at Google and big tech, so I am very suspicious of these claims without evidence. Show me the resource usage from k8s component overh…

[deleted]

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#59

Post by oblio »

karmarepellent wrote: post #51
figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

We ran only two (very small) clusters for some time in the past and even then it introduced some unnecessary overhead on the ops side and some headaches on the dev side. Maybe they were just growing pains, but if I have to run Kubernetes again I will definitely opt for a single large cluster. After all Kubernetes provides all the primitives you need to enforce separation. You wouldn't create separate VMWare productio…

Kubernetes upgrades? Don't those risk bricking everything with just 1 environment?

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#60

Post by namaria »

timhigins wrote: post #56

If you have 200 YAML files for a single service and 46 clusters I think you're using k8s wrong. And 5 + 3 different monitoring and logging tools could be a symptom of chaos in the organization. k8s, and the go runtime and network stack have been heavily optimized by armies of engineers at Google and big tech, so I am very suspicious of these claims without evidence. Show me the resource usage from k8s component overh…

> I do hate Helm to some degree

I feel you. I learned K8s with an employer where some well intentioned but misguided back end developers decided that their YAML deployments should ALL be templated and moved into Helm charts. It was bittersweet to say the least, learning all the power of K8s but having to argue and feel like an alien for saying that templating everything was definitely not going to make everything easier on the long term.

Then again they had like 6 developers and 30 services deployed and wanted to go "micro front end" on top of it. So they clearly had misunderstood the whole thing. CTO had a whole spiel on how "microservices" were a silver bullet and all.

I didn't last long there but they paid me to learn some amazing stuff. Which, in retrospect, they also taught me a bunch of lessons on how not to do things.

Post reply on HN