Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

71–80 of 203 posts

Re: Kubernetes Failure Stories

#71

Post by seneca »

superbcarrot wrote: post #10
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

That's what Federation in Kubernetes was for, sort of, but it has never really come to fruition. The closest I've seen is Admiralty, but I'm not sure how it actually works out in reality.

Re: Kubernetes Failure Stories

#72

Post by seneca »

cbushko wrote: post #25

The Istio one hits home. It is the single scariest thing to work with in our kubernetes clusters. We've caused several outages by changing the smallest things.

Indeed. All the flak k8s gets about being overly complex and difficult to manage (maybe rightfully) ought to leveled 5x at Istio.

Re: Kubernetes Failure Stories

#73

Post by edoceo »

tharkun__ wrote: post #68
Karunamon wrote: post #54

Earlier quoted context omitted.

I've got a different read on this. It's always been complicated, it's just that each.. I don't want to say "generation", but roughly the same concept, grew up with and internalized and knew about the complexities of the tech stacks they learned, and so when something comes about that moves the abstraction one level higher than what people are used to, it's seen as unstable crap. This isn't some ageist kids-these-days…

Wow you must be old. Like probably retiring now? VM/370 came out 49 years ago. So assuming you did a master's at university (I guess math at the time) I'd say you are definite retired by now. https://en.m.wikipedia.org/wiki/VM_(operating_system) (yes this is tongue in cheek sort of since you said average but you also said 'enterprise' which my cheek reads as large company :))

Bruv. You're catching downvotes for two reasons. A: ageism and B: also not realising that tools like the "old ones" used lasted decades. Release dates are a poor metric here.

Re: Kubernetes Failure Stories

#74

Post by kungfuscious »

Karunamon wrote: post #54
colllectorof wrote: post #42

Earlier quoted context omitted.

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

I've got a different read on this. It's always been complicated, it's just that each.. I don't want to say "generation", but roughly the same concept, grew up with and internalized and knew about the complexities of the tech stacks they learned, and so when something comes about that moves the abstraction one level higher than what people are used to, it's seen as unstable crap. This isn't some ageist kids-these-days…

Imagine k8s but all the core features you need to run MMO-level apps are the equivalent of simple Unity Engine functions like Instantiate(gameObject). I want. Maybe I help create?

Re: Kubernetes Failure Stories

#75

Post by p_l »

edoceo wrote: post #73
tharkun__ wrote: post #68

Earlier quoted context omitted.

Wow you must be old. Like probably retiring now? VM/370 came out 49 years ago. So assuming you did a master's at university (I guess math at the time) I'd say you are definite retired by now. https://en.m.wikipedia.org/wiki/VM_(operating_system) (yes this is tongue in cheek sort of since you said average but you also said 'enterprise' which my cheek reads as large company :))

Bruv. You're catching downvotes for two reasons. A: ageism and B: also not realising that tools like the "old ones" used lasted decades. Release dates are a poor metric here.

to make this comparison funnier...

... the ancient equivalent of Kubernetes was released in 1973 (JES3)

Re: Kubernetes Failure Stories

#76

Post by p_l »

seneca wrote: post #71
superbcarrot wrote: post #10

Earlier quoted context omitted.

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

That's what Federation in Kubernetes was for, sort of, but it has never really come to fruition. The closest I've seen is Admiralty, but I'm not sure how it actually works out in reality.

Federation v2 is nice, but not many places use it.

Unfortunately, all my clusters are either confined locally anyway, or are deliberately supposed to be separated in all ways :(

Re: Kubernetes Failure Stories

#77

Post by JMTQp8lwXL »

cbushko wrote: post #25

The Istio one hits home. It is the single scariest thing to work with in our kubernetes clusters. We've caused several outages by changing the smallest things.

Shouldn't even small changes be tested in a staging cluster before promoting to the production cluster?

Re: Kubernetes Failure Stories

#79

Post by legohead »

Surprised I don't see something about "context deadline exceeded" in there. This error has been plaguing me from Kubernetes 1.5 to 1.18, and nobody has a solid answer for it other than maybe it's Docker related. Solution? Restart your node(s).

Re: Kubernetes Failure Stories

#80

Post by squibbles »

Definitely a good one: "Experiences with running PostgreSQL on Kubernetes - Gravitational - blog post 2018"

For anyone who thinks running a database in a container environment is a neat idea, think again. I am guilty of using containers for temporary test databases, but the thought of running production databases in containers sends shivers down my spine.

Post reply on HN