Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

blog.stackademic.com

Post reply on HN
Print view View original

151–160 of 216 posts

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#151

Post by marcinzm »

m1keil wrote: post #16

How to save 1M off your cloud infra? Start from a 2M bill. That's how I see most of these projects. You create a massively expensive infra because webscale, then 3 years down the road you (or someone else) gets to rebuild it 10x cheaper. You get to write two blog posts, one for using $tech and one for migrating off $tech. A line in the cv and a promotion. But kudos for them for managing to stop the snowball and actua…

The article says they spend around $150k/year on infra. Given they have 8 DevOps engineers I assume a team of 50+ engineers. Assuming $100k/engineer that's $5 million/year in salary. That's all low end estimates.

They saved $100k in the move or 2% of their engineering costs. And they're still very much on the cloud.

If you tell most organizations that they need to change everything to save 2% they'll tell you to go away. This place benefited because their previous system was badly designed and not because it's cloud.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#152

Post by mad_vill »

this_user wrote: post #130
figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

It's just a matter of time before someone releases an orchestration layer for k8s clusters so the absurd Rube Goldberg machine that is modern devops stacks can grow even more complex.

cluster-api in a nutshell

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#153

Post by Zambyte »

nwatson wrote: post #133
anthonybsd wrote: post #129

Earlier quoted context omitted.

>Often you'd need production, staging, test, development, something like that. Normally in K8s, segregating environments is done via namespaces, not clusters (unless there are some very specific resource constraints).

The constraint often would be regulatory. Even if technically isolation is possible, management won't risk SOC2 or GDPR non-compliance.

SOC2 is voluntary, not regulatory.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#154

Post by lenkite »

pm90 wrote: post #27
figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

There are genuine reasons for running multiple clusters. It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation so they get their own cluster (although somehow its ok that the nodes are still VMs that are probably running on shared nodes… these requirements can be arbitrary sometimes).

Most important reason is the ridiculous etcd limit of 8Gb. That alone is the reason for most k8s cluster splits.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#155

Post by whstl »

mrweasel wrote: post #96
figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long wa…

Indeed. My previous company did this due to regulatory concerns.

One cluster per country in prod, one cluster per team in staging, plus individual clusters for some important customers.

A DevOps engineer famously pointed that it was stupid since they could access everything with the same SSO user anyway, and the CISO demanded individual accounts with separate passwords and separate SSO keys.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#156

Post by karmajunkie »

rdsubhas wrote: post #128

I wouldn't trust the management of this team for anything. They appear totally incompetent in both management and basic half-brain analytical skills. Who in the heck creates a cluster per service per cloud provider, duplicate all the supporting services around it, burn money and sanity in a pit, and blame the tool. Literally every single decision they listed was to use any of the given tools in the absolute worst, in…

that tends to be the take on most “k8s is too complex” articles, at least the ones i’ve seen.

yes, it’s complex, but it’s simpler than running true high availability setups without something like it to standardize the processes and components needed. what i want to see is a before and after postmortem on teams that dropped it and compare their numbers like outages to get at the whole truth of their experience.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#159

Post by mdaniel »

lenkite wrote: post #154
pm90 wrote: post #27

Earlier quoted context omitted.

There are genuine reasons for running multiple clusters. It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation so they get their own cluster (although somehow its ok that the nodes are still VMs that are probably running on shared nodes… these requirements can be arbitrary sometimes).

Most important reason is the ridiculous etcd limit of 8Gb. That alone is the reason for most k8s cluster splits.

I hate etcd probably more than most, but that 8Gb seems to just be a warning, unless you have information otherwise https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi...

I'll take this opportunity to once again bitch and moan that Kubernetes just fucking refuses to allow the KV store to be pluggable, unlike damn near everything else in their world, because they think that's funny or something

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#160

Post by edude03 »

mst wrote: post #117
edude03 wrote: post #75

Like most tech stories this had pretty much nothing to do the tool itself but with the people/organization. The entire article can be summarized with this one quote > In short, organizational decisions and an overly cautious approach to resource isolation led to an unsustainable number of clusters. And while I emphasize with how they could end up in this situation, it feels like a lot of words were spent blaming the…

The article reads to me as pretty explicitly saying that the only real takeaway wrt k8s itself is "it was the wrong choice for us and then we compounded that wrong choice by making more wrong choices in how we implemented it."

Maybe I'm reading it with rose coloured glasses - but I feel like the only thing kubernetes "did wrong" is allowing them to host multiple control planes. Yes, you need 3+ CP instances for HA, but the expectation is you'd have 3 CP instances for X (say 10) workers for Y (say 100) apps. Their implied ratio was insane in comparison.

Since you can't run the Fargate control plane that indirectly solved that problem for them

Post reply on HN