Kubernetes Failure Stories
1–10 of 203 posts
Re: Kubernetes Failure Stories
#2This is a compilation of gotcha-discovery-reports, distributed across the surface area of K8s (which, since K8s is huge, covers many nooks and crannies). This is not a compilation of "K8s sucks, here are 10 reasons why", which is what my kneejerk expectation was. (maybe I am too cynical...).
Overall, this is a fantastic index to some very interesting resources.
There's an old saying that you learn more from failure than from success (whose analogue is a blogpost with a title "look how easy it is to set up k8s in 15 minutes at home!"). If you really want to think about how to deploy nuanced technology like this at scale, operational crash reports such as these are invaluable.
Re: Kubernetes Failure Stories
#3really cool resource for learning but a lot of these have nothing to do with k8s, beyond the company in question having k8s as part of their stack (i'm addressing the possible perception of the post title suggesting k8s horror stories).
Re: Kubernetes Failure Stories
#4The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance.
For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to apply cluster wide updates, one cluster at a time.
Re: Kubernetes Failure Stories
#5They walked so we could run our k8s services.
Anyone with their failures developing, deploying or running software (or anything!) put it out there. It's a great resource.
Re: Kubernetes Failure Stories
#6Nice to see HN submissions that I also just added to reading list based on another HN submission, Litestream in this case, right?
Re: Kubernetes Failure Stories
#7This is a compilation of gotcha-discovery-reports, distributed across the surface area of K8s (which, since K8s is huge, covers many nooks and crannies). This is not a compilation of "K8s sucks, here are 10 reasons why", which is what my kneejerk expectation was. (maybe I am too cynical...). Overall, this is a fantastic index to some very interesting resources. There's an old saying that you learn more from failure t…
Which is why postmortem blog posts are always infinity more enlightening than the "How we build X" posts
Re: Kubernetes Failure Stories
#8The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…
> For instance, we had several outages when upgrading the kubernetes version in our clusters.
Yes, this was a big problem in 2018-2019. Are things smoother now in your experience?
What magnified the problems where I was working at was that the India team would silently do the k8s upgrade overnight, and when the US office opened in the morning we'd find problems several hours later. So several hours of downtime.
Re: Kubernetes Failure Stories
#9[deleted]
Re: Kubernetes Failure Stories
#10The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…
Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?