Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

141–150 of 203 posts

Re: Kubernetes Failure Stories

#141

Post by C4stor »

jrockway wrote: post #43
yongjik wrote: post #36

Earlier quoted context omitted.

Maybe I should mutter[1] "That's Kubernetes as fuck" next time I see another shenanigans involving multiple overcomplicated layers with confusing documentation interacting with each other in a way nobody can figure out. [1] Just to myself, of course.

Is Kubernetes really overcomplicated, though? Say you wanted to do a release of a new version of your app. You'd probably boot up a VM for the new version, provision it an IP, copy your code to it, and check if it's healthy. Then you'd edit your load balancer configuration to send traffic to that new IP, and drain traffic from the old IP. Then you'd shut the old instance down. That's basically what a Deployment in Ku…

So, as as a serious answer, yes.

The problem isn't deployments, it's that k8s doesn't only do deployments. It does a ton of stuff, and deployments are a small part of it. If you only wanted to use k8s for deploying your code, well, you can't.

That's what's was great with the UNIX programs philosophy, and what is lost in kubernetes. You can't pick the feature you need, and the feature you want may well break because of a problem in another part of the system you don't even know existed (hello persistent volumes flagged for deletions, resource allocation failing, ...)

No single Kubernetes feature may be overcomplicated, but k8s as a whole is for sure extremely complex.

Re: Kubernetes Failure Stories

#142

Post by TruthWillHurt »

Here's mine - We were running on Cloud Foundry, had one DevOps person that mostly dealt with Jenkins, payed for 32-64GB RAM. Decided to move to K8s (Azure AKS), Three months later we have 4-6 DevOps people dealing with networking, cross-az replication, cluster size and autoscaling, And we're paying thousands of $$$ pm for a minimum of 6 64GB VMs. FAIL

Re: Kubernetes Failure Stories

#143

Post by ahoka »

emptysongglass wrote: post #115
fremanator wrote: post #87

Earlier quoted context omitted.

Amazon SDE here. I'm not sure what internal infra you're talking about. Amazon's certainly isn't one of them. And AWS, with its design by accretion, makes that an impossibility. Kubernetes by comparison is a paragon of clarity.

Why does AWS insist on building this way? Has anyone tried to get the teams together and say, "stop, let's evaluate and simplify where we can?" As an AWS customer, I find AWS a huge pile of overcomplicated offerings that is dense with its own jargon and way of doing things. IAM is a trash fire. Everything built on IAM like IRSA is also a trash fire. Why are there managed worker nodes, spot managed worker nodes, Farga…

As if they are trying to tie them to their proprietary platform to make migration to other cloud providers hard.

Re: Kubernetes Failure Stories

#144

Post by amatix »

nyellin wrote: post #118
WJW wrote: post #37

Earlier quoted context omitted.

Not to knock on Tolstoy but there are many ways in which unhappy families can be grouped together. You have the alcoholic parents, the living-vicariously-through-their-children folks, the abusive parents, etc etc etc. To tie it back to Kubernetes, you have the scheduler-does-not-schedule lessons, the ingress-not-consistent lessons, the autoscaler-killed-all-my-pods lessons, the moving-stateful-services-with-60TB-stor…

The problem is that everyone needs to learn those lessons on their own. (But I'm biased: I'm working on a troubleshooting tool for K8s which has remediation rules for all those common cases.)

The tool sounds interesting — can you share any more details?

Re: Kubernetes Failure Stories

#145

Post by corty »

teach wrote: post #50
WJW wrote: post #37

Earlier quoted context omitted.

Not to knock on Tolstoy but there are many ways in which unhappy families can be grouped together. You have the alcoholic parents, the living-vicariously-through-their-children folks, the abusive parents, etc etc etc. To tie it back to Kubernetes, you have the scheduler-does-not-schedule lessons, the ingress-not-consistent lessons, the autoscaler-killed-all-my-pods lessons, the moving-stateful-services-with-60TB-stor…

I guess it depends on your definition of "at scale" but IMO with Hashicorp's stuff and maybe something like Backstage you get all the benefits k8s but in a package that's much simpler to reason about and manage.

By building a neat box with few buttons on the outside and still the same or more complexity on the inside. If something goes wrong, you still have to deal with all the inside parts. The only hope is that the few outside buttons only trigger happy paths without complications. "At scale" this seems unlikely, simplifications usually only work in smaller, simpler use cases.

Re: Kubernetes Failure Stories

#146

Post by crispyambulance »

goatinaboat wrote: post #140
Gene_Parmesan wrote: post #135

Earlier quoted context omitted.

I'm with you for most of this, but I do think one element you are missing here is that the ever-increasing scale is at least in part to blame as well. Yes, software is vastly more complicated today, and perhaps suffers more errors, although I would want to see data on that. But, like, YouTube in 2021 is a vastly more difficult engineering challenge than YouTube circa 2008. The same can be said for any site with users…

I really don't understand why. There seems to be a bit of a keeping-up-with-the-Jones effect The industry strongly incentivises individual engineers to make decisions that will ensure the latest buzzwords appear on their CVs - far more strongly than it incentivises making sound decisions for their current organisation.

It's true, nobody wants to be left behind with a bunch of skills that aren't attractive to desirable employers.

There's also very much a tendency to make things complicated in many workplaces as a form of job security and gatekeeping. But it doesn't last forever. Today's hot-shit is tomorrow's ball-and-chain.

10+ years from now, many k8s monsters will still be running and kept alive by stressed-out teams in India. Much like the "Oracle Enterprise Business Suite" giant-shit-ball from the late 90's is still working in many corporations doing absolute critical stuff with tentacles in every part of the org. Changes in these systems are nearly impossible because of the cost and nightmarish complexity, and _everyone_ is _forced_ to use it.

Re: Kubernetes Failure Stories

#147

Post by durnygbur »

bcrosby95 wrote: post #14
superbcarrot wrote: post #10

Earlier quoted context omitted.

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

A Kubernetes for Kubernetes! But what about when we want to upgrade that?

To ugrade it you summon Scrum of Scrums, the Master Scrum Master has all the answers you might need.

Re: Kubernetes Failure Stories

#148

Post by waynesonfire »

avereveard wrote: post #48
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

What are the best practices for transactional data storage in multi cluster environments, just ditch databases and go for distributed, raft based, nosql variants?

Am i not understanding the assumptions in your question or is it underspecified?

But, whats wrong with postgres here?

Re: Kubernetes Failure Stories

#149

Post by ThePhysicist »

It's too bad we don't have more UNIX-style tools in the DevOps space (in the sense of "do one thing and do it well"). Some features of K8s like software-defined networking would be really useful for our infrastructure, but it seems you'll have to fully commit to the rest of the software to use a single feature. Personally I'd prefer to have a loose set of tools for containerization, networking, orchestration, load balancing etc. that I can combine as I see fit and gradually adopt, and not a single monolith that ties me into a given paradigm for all these things.
Post reply on HN