Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

121–130 of 203 posts

Re: Kubernetes Failure Stories

#121

Post by bacongobbler »

timeinput wrote: post #113
Karunamon wrote: post #54

Earlier quoted context omitted.

I've got a different read on this. It's always been complicated, it's just that each.. I don't want to say "generation", but roughly the same concept, grew up with and internalized and knew about the complexities of the tech stacks they learned, and so when something comes about that moves the abstraction one level higher than what people are used to, it's seen as unstable crap. This isn't some ageist kids-these-days…

My complaint comes from the at scale part. I don't do anything at scale. I don't have to scale. My core competency is as a library or tool builder, and often my primary deliverable is a tool or library. In the past year I got a new JR engineer on my team who was all hot and bothered with docker and k8s and he spent a month changing out CI process to be docker based. It went from a 20 line shell script to a pile of ga…

Sounds more like a people problem than a tech problem.

Re: Kubernetes Failure Stories

#122

Post by dijit »

deathanatos wrote: post #111
AmericanChopper wrote: post #98

Earlier quoted context omitted.

I don’t know why anybody would presume that a technology focused on ephemeral resource provisioning would be a suitable place to put your persistence layer... That said, I don’t think it’s a sin at all to use it for testing. My default local dev setup is to use a Postgres container. But persistence is very much not required in that situation.

> I don’t know why anybody would presume that a technology focused on ephemeral resource provisioning would be a suitable place to put your persistence layer... Kubernetes does more than that, and has features like PVCs + Statefulsets are basically intended for, designed for exactly this use case. If you see the HN comments[1], the top comment mentions this, and that the article waves it away for reasons not related…

Kubernetes was not initially designed with persistence in mind. If it was then the etcd of the master nodes would also be in containers.

There are (good) attempts at shoving it in, but the general advice I would give is that if you care about your data you should give it every possible chance for to not be corrupted or disrupted; and that means keeping the number of abstractions and indirections low.

Re: Kubernetes Failure Stories

#123

Post by dilyevsky »

dijit wrote: post #122
deathanatos wrote: post #111

Earlier quoted context omitted.

> I don’t know why anybody would presume that a technology focused on ephemeral resource provisioning would be a suitable place to put your persistence layer... Kubernetes does more than that, and has features like PVCs + Statefulsets are basically intended for, designed for exactly this use case. If you see the HN comments[1], the top comment mentions this, and that the article waves it away for reasons not related…

Kubernetes was not initially designed with persistence in mind. If it was then the etcd of the master nodes would also be in containers. There are (good) attempts at shoving it in, but the general advice I would give is that if you care about your data you should give it every possible chance for to not be corrupted or disrupted; and that means keeping the number of abstractions and indirections low.

Everything comprising control plane runs in a container including etcd, apiserver, kube-controller and scheduler

Re: Kubernetes Failure Stories

#124

Post by xiwenc »

deathanatos wrote: post #108

Oh good lord, I'm in the middle of one of these right now. My team runs a fair number of K8s clusters, mostly on Azure AKS. Of the pods in our clusters, a few talk to the AKS API server for their cluster. Those pods that do, will, occasionally, lose contact with the API server. API calls will start timing out. It'll usually resolve on its own after some time, but then come back later. It's somewhat affected by load.…

This is the downside of “cloud services”. It’s all sunshine and rainbows when it works. When it breaks, start praying and pull the wallet hoping that helps. You are just at their mercy because you cant look under the hood even if you want to. And it takes a long time before they decide to assign competent people on your case.

My experience with Azure is pretty bad. (Premier) Support is next to useless. AWS is somewhat better, specially quality of their software/services/documentation. In my opinion, aws services feel more battle tested than azure.

Sorry for the rant, back on topic.

Do you have monitoring on the cluster? I would start to collect more low level metrics like open files, network traffic, pings

Also what could help is to run daemonset of a pod that can send you diagnostics at regular intervals. Perhaps even capability for remote shell so that you can troubleshoot from within the node when the problem surfaces again.

I would definitely be interested to know more how this turns out. I also run a fair share of AKS.

Re: Kubernetes Failure Stories

#125

Post by xiwenc »

tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Are you implying we need a manager1 (k8s) to manage containers? This manager1 cannot be trusted so introduce manager2 to manage manager1? What’s next? We need to look forward to manager3?

I dont like where this is going

Re: Kubernetes Failure Stories

#126

Post by Lutzb »

AmericanChopper wrote: post #100
colllectorof wrote: post #42

Earlier quoted context omitted.

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

> Meanwhile, the complexity of our technology goes up and its reliability in practice goes down. I don’t agree with this at all. Reliability in practice is improving at a phenomenal pace. 10 years ago maintenance outages were a normal feature of every service, and unplanned outages were perfectly ordinary occurrences. Consumers today expect a much higher level of availability and reliability, which they receive rathe…

Depends on what you compare. If you compare mainframes vs k8s I would say reliability is not on par yet. If you compare commodity systems with simple monolithic apps vs k8s system maybe we just reached parity.

Re: Kubernetes Failure Stories

#127

Post by alexellisuk »

Disclaimer, I am a CNCF Ambassador (voluntary) - so it's in my interest to promote CNCF projects like Kubernetes.

It seems like a good time to mention my blog post from last year "Then he asked me “Is Kubernetes right for us?" -> https://alexellisuk.medium.com/then-he-asked-me-is-kubernete...

Some of the feedback I've had so far is that it was refreshing to get "permission" to consider alternatives vs. the current hype. I use K8s and K3s quite broadly myself, but increasingly see consulting prospects and customers who are not comfortable to make the leap, but are very happy on managed services with their chosen vendor - Azure / AWS / GCP.

GitLab recently released a bunch of terraform to show you how to run side projects in the free tier of a cloud - https://gitlab.com/gitlab-org/5-minute-production-app/deploy...

The OpenFaaS project again is very coupled to Kubernetes, making it easier to use and more reliable is important to the community and project's future. However, we created a version called faasd that works more like docker-compose. It's received much more traction than we expected and companies and individuals are putting it into production. It does't have clustering, and supports only 1 replica per function, so it's surprising. https://github.com/openfaas/faasd

I'll keep doing my bit to promote solutions that make K8s easier to understand like K3s (see also https://k3sup.dev) and to look into alternatives. But as you will see in my blog post - I don't think it's right to assume Kubernetes is the right solution for every team, and every project, without first talking about the problem being solved.

Re: Kubernetes Failure Stories

#130

Post by FridgeSeal »

AmericanChopper wrote: post #114
deathanatos wrote: post #111

Earlier quoted context omitted.

> I don’t know why anybody would presume that a technology focused on ephemeral resource provisioning would be a suitable place to put your persistence layer... Kubernetes does more than that, and has features like PVCs + Statefulsets are basically intended for, designed for exactly this use case. If you see the HN comments[1], the top comment mentions this, and that the article waves it away for reasons not related…

You can make it work, but why would you want to? Databases aren’t generally something that benefits from using container orchestration. They’re not usually highly dynamic, horizontally scaling systems. Generally you’d optimize that part of your system to maximize stability and consistency. For most typical use cases I can’t see the intuitive leap required to decide that all that additional complexity is necessary to…

It that I advocate running your own Postgres setup in your own cluster instead of just renting a managed version, but I’ve run a few databases on K8s and found it pretty fine: useful for when your hosting provider doesn’t support the database you want to run (Clickhouse managed AWS service when?) or for application-specific KV-stores: EBS volumes and PVC’s are great, solid performance, kubernetes takes care of the networking, will resurrect it if the worst happens and it does go down.

I probably could have those things on their own instance but then I’d need to have to go through the hassle of networking, failover/recreation, deployments, etc and for the vast majority of cases that’s 100% more effort than deploy a stateful-set.

Post reply on HN