Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

61–70 of 203 posts

Re: Kubernetes Failure Stories

#61

Post by hinkley »

jrockway wrote: post #40
mitjak wrote: post #3

really cool resource for learning but a lot of these have nothing to do with k8s, beyond the company in question having k8s as part of their stack (i'm addressing the possible perception of the post title suggesting k8s horror stories).

A lot of them do have things to do with k8s, though. Admission webhooks, Istio sidecar injection, etc. The CPU limits = weird latency spikes also shows up a lot there, but it's technically a cgroups problem. (Set GOMAXPROCS=16, set cpu limit to 1, wonder why your program is asleep 15/16th of every cgroups throttling interval. I see that happen to people a lot, the key point being that GOMAXPROCS and the throttling in…

I have an environment variable to set process count for a nodejs service because everyone is finger pointing at who should be the one to sort out the fact that cgroup cpu limits and nprocs disagree most of the time. Our stuff is fine, those other guys should fix their stuff.

Thanks, everybody. Good effort.

Re: Kubernetes Failure Stories

#62

Post by [deleted] »

superbcarrot wrote: post #10
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

[deleted]

Re: Kubernetes Failure Stories

#65

Post by hansvm »

gen220 wrote: post #2

This is a compilation of gotcha-discovery-reports, distributed across the surface area of K8s (which, since K8s is huge, covers many nooks and crannies). This is not a compilation of "K8s sucks, here are 10 reasons why", which is what my kneejerk expectation was. (maybe I am too cynical...). Overall, this is a fantastic index to some very interesting resources. There's an old saying that you learn more from failure t…

> There's an old saying that you learn more from failure than from success

OT: That was easily the hardest lesson to instill in my math students, and the most impactful once internalized. Being comfortable experimenting with ideas you don't yet fully understand is critical to the learning process.

Re: Kubernetes Failure Stories

#66

Post by Lucasoato »

Karunamon wrote: post #54
colllectorof wrote: post #42

Earlier quoted context omitted.

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

I've got a different read on this. It's always been complicated, it's just that each.. I don't want to say "generation", but roughly the same concept, grew up with and internalized and knew about the complexities of the tech stacks they learned, and so when something comes about that moves the abstraction one level higher than what people are used to, it's seen as unstable crap. This isn't some ageist kids-these-days…

The fact that this is similar to something already happened reassures me a little. May Lambda and other FAAS frameworks be considered the next layer of abstraction? (even if cloud providers are using containers underneath)

Re: Kubernetes Failure Stories

#67

Post by MichaelMoser123 »

WJW wrote: post #37
barrkel wrote: post #31

Earlier quoted context omitted.

"All happy families are alike; each unhappy family is unhappy in its own way." https://en.wikipedia.org/wiki/Anna_Karenina_principle

Not to knock on Tolstoy but there are many ways in which unhappy families can be grouped together. You have the alcoholic parents, the living-vicariously-through-their-children folks, the abusive parents, etc etc etc. To tie it back to Kubernetes, you have the scheduler-does-not-schedule lessons, the ingress-not-consistent lessons, the autoscaler-killed-all-my-pods lessons, the moving-stateful-services-with-60TB-stor…

it's the too many moving parts and black boxes blues.

Re: Kubernetes Failure Stories

#68

Post by tharkun__ »

Karunamon wrote: post #54
colllectorof wrote: post #42

Earlier quoted context omitted.

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

I've got a different read on this. It's always been complicated, it's just that each.. I don't want to say "generation", but roughly the same concept, grew up with and internalized and knew about the complexities of the tech stacks they learned, and so when something comes about that moves the abstraction one level higher than what people are used to, it's seen as unstable crap. This isn't some ageist kids-these-days…

Wow you must be old. Like probably retiring now?

VM/370 came out 49 years ago. So assuming you did a master's at university (I guess math at the time) I'd say you are definite retired by now.

https://en.m.wikipedia.org/wiki/VM_(operating_system)

(yes this is tongue in cheek sort of since you said average but you also said 'enterprise' which my cheek reads as large company :))

Re: Kubernetes Failure Stories

#69

Post by seneca »

bo0tzz wrote: post #13
superbcarrot wrote: post #10

Earlier quoted context omitted.

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

https://github.com/kubernetes-sigs/cluster-api

CAPI is for creating clusters only. It doesn't offer any orchestration.

Re: Kubernetes Failure Stories

#70

Post by walrus01 »

fuzzythinker wrote: post #21

Meta. Is this a common usage of .af domains?

Registering a .af domain is a lot easier and less costly if you're an Afghan citizen and speak Dari or Pashto.

There's a way to do it and pay in person in Kabul.

Right now, the entire NIC website only answers at http://nic.af , if you prefix it with www it appears to be down.

It's not very common yet because the .af registry is not really set up to work with high volume, fully automated third party domain registrars.

Post reply on HN