Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

blog.stackademic.com

Post reply on HN
Print view View original

121–130 of 216 posts

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#121

Post by mrweasel »

mschuster91 wrote: post #106
mrweasel wrote: post #96

Earlier quoted context omitted.

> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long wa…

> Out in the real world I've frequently seen companies build a cluster per service, or group of services, to better control load and scaling and again to control network access. Network Policies have solved that at least for ingress traffic. Egress traffic is another beast, you can't allow egress traffic to a service, only to pods or IP ranges.

I was thinking egress, but you're correct on ingress.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#122

Post by karmarepellent »

mst wrote: post #116
karmarepellent wrote: post #62

Earlier quoted context omitted.

In theory: absolutely. This is just anecdata and you are welcome to challenge me on it, but I have never had a problem upgrading Kubernetes itself. As long as you trail one version behind the latest to ensure critical bugs are fixed before you risk to run into them yourself, I think you are good. Edit: To expand on it a little bit. I think there is always a real, theoretical risk that must be taken into account when…

"I am cutting this corner because I absolutely cannot make a business case I believe in for doing it the hard (but more correct) way but believe me I am still going to be low key paranoid about it indefinitely" is an experience that I think a lot of us can relate to. I've actually asked for a task to be reassigned to somebody else before now on the grounds that I knew it deserved to be done the simple way but could n…

Actually I think the trick is to change ones own perspective on these things. Regardless of how many redundancies and how many 9's of availability your system theoretically achieves, there is always stuff that can go wrong for a variety of reasons. If things go wrong, I am faster at fixing a not-so-complex system than the more complex system that should, in theory, be more robust.

Also I have yet to experience that an outage of any kind had any negative consequences for me personally. As long as you stand by the decisions you made in the past and show a path forward, people (even the higher-ups) are going to respect that.

Anticipating every possible issue that might or might not occur during the lifetime of an application just leads to over-engineering.

I think rationalizing it a little bit may also help with the paranoia.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#124

Post by phendrenad2 »

Too bad the author and company are anonymous. I'd like to confirm my assumption that the author has zero business using k8s at all.

Infrastructure is a lost art. Nobody knows what they're doing. We've entered an evolutionary spandrel where "more tools = better" meaning the candidate for an IT role who swears by 10 k8s tools is always better than the one who can fix your infra, but will also remove k8s because it's not helping you at all.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#125

Post by phendrenad2 »

threePointFive wrote: post #123

Does anyone have access to the full article? I'm curious what their alternative was. Terraform? Directly using cloud apis? VMs and Ansible?

> We selected tools that matched specific workloads:

> Stateless services → Moved to AWS ECS/Fargate.

> Stateful services → Deployed on EC2 instances with Docker, reducing the need for complex orchestration.

> Batch jobs → Handled with AWS Batch.

> Event-driven workflows → Shifted to AWS Lambda.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#126

Post by _bare_metal »

m1keil wrote: post #16

How to save 1M off your cloud infra? Start from a 2M bill. That's how I see most of these projects. You create a massively expensive infra because webscale, then 3 years down the road you (or someone else) gets to rebuild it 10x cheaper. You get to write two blog posts, one for using $tech and one for migrating off $tech. A line in the cv and a promotion. But kudos for them for managing to stop the snowball and actua…

To expand on this, I run BareMetalSavings.com[0] and the most common cause of people staying with the cloud is it's very hard for them to maintain their own K8S cluster(s), which they want to keep because they're great for any non-ops developer.

So those savings are possible only if your devs are willing to leave the lock in of comfort

[0]: https://BareMetalSavings.com

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#127

Post by AtlasBarfed »

mrweasel wrote: post #96
figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long wa…

K8s requires a flat plane addressability model across all containers, meaning anyone can see and call anyone else?

I can see security teams getting uppity about that.

Also budgetary and org boundaries, cloud providers, disaster recovery/hot spares/redundancy/AB hotswap, avoid single tank point of failure.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#128

Post by rdsubhas »

I wouldn't trust the management of this team for anything. They appear totally incompetent in both management and basic half-brain analytical skills. Who in the heck creates a cluster per service per cloud provider, duplicate all the supporting services around it, burn money and sanity in a pit, and blame the tool.

Literally every single decision they listed was to use any of the given tools in the absolute worst, incompetent way possible. I wouldn't trust them with a Lego toy set with this record.

The people who quit didn't quit merely out of burnout. They quit the stupidity of the managers running this s##tshow.

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#129

Post by anthonybsd »

mrweasel wrote: post #96
figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long wa…

>Often you'd need production, staging, test, development, something like that.

Normally in K8s, segregating environments is done via namespaces, not clusters (unless there are some very specific resource constraints).

Re: I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever

#130

Post by this_user »

figassis wrote: post #18

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…

It's just a matter of time before someone releases an orchestration layer for k8s clusters so the absurd Rube Goldberg machine that is modern devops stacks can grow even more complex.
Post reply on HN