Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

31–40 of 203 posts

Re: Kubernetes Failure Stories

#31

Post by barrkel »

tempest_ wrote: post #7
gen220 wrote: post #2

This is a compilation of gotcha-discovery-reports, distributed across the surface area of K8s (which, since K8s is huge, covers many nooks and crannies). This is not a compilation of "K8s sucks, here are 10 reasons why", which is what my kneejerk expectation was. (maybe I am too cynical...). Overall, this is a fantastic index to some very interesting resources. There's an old saying that you learn more from failure t…

Which is why postmortem blog posts are always infinity more enlightening than the "How we build X" posts

"All happy families are alike; each unhappy family is unhappy in its own way."

https://en.wikipedia.org/wiki/Anna_Karenina_principle

Re: Kubernetes Failure Stories

#36

Post by yongjik »

pc86 wrote: post #22
fuzzythinker wrote: post #21

Meta. Is this a common usage of .af domains?

I only came here to comment that the domain name was fantastic for this usage.

Maybe I should mutter[1] "That's Kubernetes as fuck" next time I see another shenanigans involving multiple overcomplicated layers with confusing documentation interacting with each other in a way nobody can figure out.

[1] Just to myself, of course.

Re: Kubernetes Failure Stories

#37

Post by WJW »

barrkel wrote: post #31
tempest_ wrote: post #7

Earlier quoted context omitted.

Which is why postmortem blog posts are always infinity more enlightening than the "How we build X" posts

"All happy families are alike; each unhappy family is unhappy in its own way." https://en.wikipedia.org/wiki/Anna_Karenina_principle

Not to knock on Tolstoy but there are many ways in which unhappy families can be grouped together. You have the alcoholic parents, the living-vicariously-through-their-children folks, the abusive parents, etc etc etc.

To tie it back to Kubernetes, you have the scheduler-does-not-schedule lessons, the ingress-not-consistent lessons, the autoscaler-killed-all-my-pods lessons, the moving-stateful-services-with-60TB-storage-size-took-20-minutes lessons and probably many more. It's like Churchills explanation of democracy: terrible, but better than all the alternatives. (at scale, at least)

Re: Kubernetes Failure Stories

#38

Post by WJW »

YawningAngel wrote: post #20
superbcarrot wrote: post #10

Earlier quoted context omitted.

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

You end up needing this anyway, because Kubernetes clusters are confined to single geographic regions and you need some way of managing the separate cluster(s) you have in each region

Not saying you should, but it's entirely possible to have nodes all over the world managed by a single set of master nodes. The system is not really designed with that use case in mind though, the added latencies will hurt a lot in getting etcd consensus.

Re: Kubernetes Failure Stories

#39

Post by UweSchmidt »

It takes a high level of skill and maturity to articulate such failure stories. The thing you missed is obvious to others, the "bug" you found is actually documented somewhere, and you can mitigate and move on from disaster without completely drilling down and reconstructing the problem. So thanks for posting.

Re: Kubernetes Failure Stories

#40

Post by jrockway »

mitjak wrote: post #3

really cool resource for learning but a lot of these have nothing to do with k8s, beyond the company in question having k8s as part of their stack (i'm addressing the possible perception of the post title suggesting k8s horror stories).

A lot of them do have things to do with k8s, though. Admission webhooks, Istio sidecar injection, etc.

The CPU limits = weird latency spikes also shows up a lot there, but it's technically a cgroups problem. (Set GOMAXPROCS=16, set cpu limit to 1, wonder why your program is asleep 15/16th of every cgroups throttling interval. I see that happen to people a lot, the key point being that GOMAXPROCS and the throttling interval are not something they ever manually configured, hence it's surprising how they interact. I ship https://github.com/uber-go/automaxprocs in all of my open source stuff to avoid bug reports about this particular issue. Fun stuff! :)

DNS also makes a regular appearance, and I agree it's not Kubernetes' fault, but on the other hand, people probably just hard-coded service IPs for service discovery before Kubernetes, so DNS issues are a surprise to them. When they type "google.com" into their browser, it works every time, so why wouldn't "service.namespace.svc.cluster.local" work just as well? (I also love the cloud providers' approach to this rough spot -- GKE has a service that exists to scale up kube-dns if you manually scale it down!)

Anyway, it's all good reading. If you don't read this, you are bound to have these things happen to you. Many of these things will happen to you even if you don't use Kubernetes!

Post reply on HN