Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

131–140 of 203 posts

Re: Kubernetes Failure Stories

#131

Post by FridgeSeal »

nyellin wrote: post #119
cbushko wrote: post #25

The Istio one hits home. It is the single scariest thing to work with in our kubernetes clusters. We've caused several outages by changing the smallest things.

What do you use Istio for? 90% of the usecases I hear about could be solved with simpler tooling.

I don’t use it yet, but I’m probably going to have to because my team is looking at using Kubeflow and it leverages Istio to do things like traffic-splitting and stuff so you can A/B test models without needing to handle that at your model code level.

KNative can also do really cool things like per-request routing, scale-to-zero deployments that it can bring back up when it gets a request, it’s pretty rad.

Re: Kubernetes Failure Stories

#132

Post by politelemon »

alexellisuk wrote: post #127

Disclaimer, I am a CNCF Ambassador (voluntary) - so it's in my interest to promote CNCF projects like Kubernetes. It seems like a good time to mention my blog post from last year "Then he asked me “Is Kubernetes right for us?" -> https://alexellisuk.medium.com/then-he-asked-me-is-kubernete... Some of the feedback I've had so far is that it was refreshing to get "permission" to consider alternatives vs. the current hy…

> I'll keep doing my bit to promote solutions that make K8s easier to understand like K3s (see also https://k3sup.dev) and to look into alternatives. But as you will see in my blog post - I don't think it's right to assume Kubernetes is the right solution for every team, and every project, without first talking about the problem being solved.

Thanks for doing your part. Frankly I don't believe the message is being spread enough. At least in the circles I circle, the unspoken expectation to use k8s for "everything" is persistent. The energy that I save by not dealing with it is sometimes spent on explaining why I'm not dealing with it. Yes, entirely an outcome of my circles. I will however share your links in the future, so thanks.

Re: Kubernetes Failure Stories

#133

Post by xorcist »

bacongobbler wrote: post #121
timeinput wrote: post #113

Earlier quoted context omitted.

My complaint comes from the at scale part. I don't do anything at scale. I don't have to scale. My core competency is as a library or tool builder, and often my primary deliverable is a tool or library. In the past year I got a new JR engineer on my team who was all hot and bothered with docker and k8s and he spent a month changing out CI process to be docker based. It went from a 20 line shell script to a pile of ga…

Sounds more like a people problem than a tech problem.

Everything is a people problem at the end of the day.

Re: Kubernetes Failure Stories

#134

Post by xorcist »

jrockway wrote: post #43
yongjik wrote: post #36

Earlier quoted context omitted.

Maybe I should mutter[1] "That's Kubernetes as fuck" next time I see another shenanigans involving multiple overcomplicated layers with confusing documentation interacting with each other in a way nobody can figure out. [1] Just to myself, of course.

Is Kubernetes really overcomplicated, though? Say you wanted to do a release of a new version of your app. You'd probably boot up a VM for the new version, provision it an IP, copy your code to it, and check if it's healthy. Then you'd edit your load balancer configuration to send traffic to that new IP, and drain traffic from the old IP. Then you'd shut the old instance down. That's basically what a Deployment in Ku…

> That's basically what a Deployment in Kubernetes would do

If that what was Kubernetes did, then we wouldn't be having this discussion.

What you describe is more like what something like Ansible does. Provision, health check, flip load balancer.

Re: Kubernetes Failure Stories

#135

Post by Gene_Parmesan »

colllectorof wrote: post #42
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

I'm with you for most of this, but I do think one element you are missing here is that the ever-increasing scale is at least in part to blame as well. Yes, software is vastly more complicated today, and perhaps suffers more errors, although I would want to see data on that. But, like, YouTube in 2021 is a vastly more difficult engineering challenge than YouTube circa 2008. The same can be said for any site with users numbering now in the millions. E.g., not only have Netflix's streaming user counts exploded since 2007 when it launched, the quality expected now of on-demand streaming has strongly increased as well.

Having said that, in reality that scale applies only to a small minority of products. I think an additional part of the real problem is the ever-present Cargo-Cult Oriented Programming model we've had for a while. I am sure container orchestration is important and maybe even required at places at Google scale. But I hear far too much about its use; it seems that nearly everyone is using it, and I really don't understand why. There seems to be a bit of a keeping-up-with-the-Jones effect, like people would be embarrassed to say that their startup relies on platforms that aren't the latest and greatest. Picking technologies because they sound cool on a resume or at a tech conference, instead of their use being appropriate for the circumstances, seems like a common issue.

Re: Kubernetes Failure Stories

#136

Post by grafelic »

AmericanChopper wrote: post #98
squibbles wrote: post #80

Definitely a good one: "Experiences with running PostgreSQL on Kubernetes - Gravitational - blog post 2018" For anyone who thinks running a database in a container environment is a neat idea, think again. I am guilty of using containers for temporary test databases, but the thought of running production databases in containers sends shivers down my spine.

I don’t know why anybody would presume that a technology focused on ephemeral resource provisioning would be a suitable place to put your persistence layer... That said, I don’t think it’s a sin at all to use it for testing. My default local dev setup is to use a Postgres container. But persistence is very much not required in that situation.

I strongly support this advice having felt the pain.

Inherited a setup using a semi-well-known vendors Patroni/Postgres HA operator implementation on OpenShift and it was extremely fragile to any kind of network latency/downtime (due to its strong tie to the master api) or worker node outage/drainage/maintenance. These events would mean hours of recovery work hacking around the operator.

It was not my decision to place Postgres on OpenShift and I will strongly discourage anyone planning to do this for production (or even testing). Please do not do it if you value your time and sanity. Spin up a replica set on VMs using one of the already production ready and battlehardened solutions or if in cloud use a managed Postgresql service.

Re: Kubernetes Failure Stories

#137

Post by trabant00 »

colllectorof wrote: post #42
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

That's exactly what happens. But not out of ignorance and not only in IT. It's recipe for marketing success in any domain. Customers only know the problems of current solutions. Solve those and you get instant influx of customers. Sure, your solution brings it's own problems but it will take a very long time for them to surface and become common knowledge. First people will only check that your solution does fix the old problems in testing environments. And it does, so they move on. When they encounter implementation/production issues that's actually an opportunity for you to sell consulting. People who took the bait early on will mob on any naysayer: "Agile doesn't suck. it's just most implementations, but real Agile rocks!"

Re: Kubernetes Failure Stories

#138

Post by tutfbhuf »

WJW wrote: post #38
YawningAngel wrote: post #20

Earlier quoted context omitted.

You end up needing this anyway, because Kubernetes clusters are confined to single geographic regions and you need some way of managing the separate cluster(s) you have in each region

Not saying you should, but it's entirely possible to have nodes all over the world managed by a single set of master nodes. The system is not really designed with that use case in mind though, the added latencies will hurt a lot in getting etcd consensus.

While it is certainly possible to have mixed region clusters, from my experience it is better to have one cluster per region. Do your synchronous communication within the cluster / within the datacenter with sub millisecond latencies and do asynchronous communication across cluster boundaries with up to triple digit millisecond latencies.

Re: Kubernetes Failure Stories

#139

Post by tutfbhuf »

bcrosby95 wrote: post #14
superbcarrot wrote: post #10

Earlier quoted context omitted.

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

A Kubernetes for Kubernetes! But what about when we want to upgrade that?

I think at a certain point up in the stack you don't want to have another orchestrator for the orchestrator. You would just tell each team to keep their cluster up-to-date without creating a (dangerous) master button to update them all at once.

Re: Kubernetes Failure Stories

#140

Post by goatinaboat »

Gene_Parmesan wrote: post #135
colllectorof wrote: post #42

Earlier quoted context omitted.

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

I'm with you for most of this, but I do think one element you are missing here is that the ever-increasing scale is at least in part to blame as well. Yes, software is vastly more complicated today, and perhaps suffers more errors, although I would want to see data on that. But, like, YouTube in 2021 is a vastly more difficult engineering challenge than YouTube circa 2008. The same can be said for any site with users…

I really don't understand why. There seems to be a bit of a keeping-up-with-the-Jones effect

The industry strongly incentivises individual engineers to make decisions that will ensure the latest buzzwords appear on their CVs - far more strongly than it incentivises making sound decisions for their current organisation.

Post reply on HN