Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

111–120 of 203 posts

Re: Kubernetes Failure Stories

#111

Post by deathanatos »

AmericanChopper wrote: post #98
squibbles wrote: post #80

Definitely a good one: "Experiences with running PostgreSQL on Kubernetes - Gravitational - blog post 2018" For anyone who thinks running a database in a container environment is a neat idea, think again. I am guilty of using containers for temporary test databases, but the thought of running production databases in containers sends shivers down my spine.

I don’t know why anybody would presume that a technology focused on ephemeral resource provisioning would be a suitable place to put your persistence layer... That said, I don’t think it’s a sin at all to use it for testing. My default local dev setup is to use a Postgres container. But persistence is very much not required in that situation.

> I don’t know why anybody would presume that a technology focused on ephemeral resource provisioning would be a suitable place to put your persistence layer...

Kubernetes does more than that, and has features like PVCs + Statefulsets are basically intended for, designed for exactly this use case. If you see the HN comments[1], the top comment mentions this, and that the article waves it away for reasons not related to k8s, but to "well, if the underlying storage is slow or not durable, then…" … yeah, then it doesn't matter if you're running k8s in the middle of it or not.

[1]: https://news.ycombinator.com/item?id=16207430

Re: Kubernetes Failure Stories

#112

Post by ec109685 »

squibbles wrote: post #80

Definitely a good one: "Experiences with running PostgreSQL on Kubernetes - Gravitational - blog post 2018" For anyone who thinks running a database in a container environment is a neat idea, think again. I am guilty of using containers for temporary test databases, but the thought of running production databases in containers sends shivers down my spine.

This is exactly what AWS does with something like Aurora: https://awsmedia.awsstatic-china.com/blog/2017/aurora-design...

Not sure why you're so fearful. Bare metal machines can crash in weird ways and K8s containers can be just as reliable as the underlying host.

Re: Kubernetes Failure Stories

#113

Post by timeinput »

Karunamon wrote: post #54
colllectorof wrote: post #42

Earlier quoted context omitted.

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

I've got a different read on this. It's always been complicated, it's just that each.. I don't want to say "generation", but roughly the same concept, grew up with and internalized and knew about the complexities of the tech stacks they learned, and so when something comes about that moves the abstraction one level higher than what people are used to, it's seen as unstable crap. This isn't some ageist kids-these-days…

My complaint comes from the at scale part.

I don't do anything at scale. I don't have to scale. My core competency is as a library or tool builder, and often my primary deliverable is a tool or library.

In the past year I got a new JR engineer on my team who was all hot and bothered with docker and k8s and he spent a month changing out CI process to be docker based. It went from a 20 line shell script to a pile of garbage.

I disagreed with the decision at the time, and while a subject matter expert I'm not the team lead so I couldn't say no stop that's bad.

While I'm sure k8s solves problems for Google I'm not google. My company isn't google, and my team isn't solving that class of problems so docker is useless crap for me.

The person is no longer on the team. They left their docker mark, and ran off to dockerify some other project leaving a team with no container expertise and a CI pipeline that is hacks around docker bugs.

Re: Kubernetes Failure Stories

#114

Post by AmericanChopper »

deathanatos wrote: post #111
AmericanChopper wrote: post #98

Earlier quoted context omitted.

I don’t know why anybody would presume that a technology focused on ephemeral resource provisioning would be a suitable place to put your persistence layer... That said, I don’t think it’s a sin at all to use it for testing. My default local dev setup is to use a Postgres container. But persistence is very much not required in that situation.

> I don’t know why anybody would presume that a technology focused on ephemeral resource provisioning would be a suitable place to put your persistence layer... Kubernetes does more than that, and has features like PVCs + Statefulsets are basically intended for, designed for exactly this use case. If you see the HN comments[1], the top comment mentions this, and that the article waves it away for reasons not related…

You can make it work, but why would you want to? Databases aren’t generally something that benefits from using container orchestration. They’re not usually highly dynamic, horizontally scaling systems. Generally you’d optimize that part of your system to maximize stability and consistency. For most typical use cases I can’t see the intuitive leap required to decide that all that additional complexity is necessary to attempt to replicate what you’d get from a few VPS. Unless you have a specialized use case, to me it just seems like very obviously the wrong tool for the job.

Re: Kubernetes Failure Stories

#115

Post by emptysongglass »

fremanator wrote: post #87
ex_amazon_sde wrote: post #51

Earlier quoted context omitted.

Ex Amazon SDE here. I would pick the first method a hundred times. People would be surprised at how simple the internal infra is, given the fleet size, compared to stuff like k8s. (I'm talking about the infra that runs on bare metal, not AWS)

Amazon SDE here. I'm not sure what internal infra you're talking about. Amazon's certainly isn't one of them. And AWS, with its design by accretion, makes that an impossibility. Kubernetes by comparison is a paragon of clarity.

Why does AWS insist on building this way? Has anyone tried to get the teams together and say, "stop, let's evaluate and simplify where we can?"

As an AWS customer, I find AWS a huge pile of overcomplicated offerings that is dense with its own jargon and way of doing things. IAM is a trash fire. Everything built on IAM like IRSA is also a trash fire. Why are there managed worker nodes, spot managed worker nodes, Fargate worker nodes, self-managed worker nodes, and spot self-managed worker nodes? Why are there circular dependencies for every piece of infra I stand up making it almost impossible to delete any resources? Why can't I click on a service and see how much it costs me in one or two clicks?

This is a completely insane way of building software to me and I have to eat it.

Re: Kubernetes Failure Stories

#116

Post by nyellin »

deathanatos wrote: post #108

Oh good lord, I'm in the middle of one of these right now. My team runs a fair number of K8s clusters, mostly on Azure AKS. Of the pods in our clusters, a few talk to the AKS API server for their cluster. Those pods that do, will, occasionally, lose contact with the API server. API calls will start timing out. It'll usually resolve on its own after some time, but then come back later. It's somewhat affected by load.…

Would love to discuss in detail and see if I can help. At my last employer I wrote low level solutions for K8s related to security and at the moment I'm working on a stealth mode startup for K8s troubleshooting. Feel free to shoot me an email at aantny at gmail.

Re: Kubernetes Failure Stories

#117

Post by nyellin »

arthur_pryor wrote: post #90
hansvm wrote: post #65

Earlier quoted context omitted.

> There's an old saying that you learn more from failure than from success OT: That was easily the hardest lesson to instill in my math students, and the most impactful once internalized. Being comfortable experimenting with ideas you don't yet fully understand is critical to the learning process.

it's a hard thing to keep in mind and execute on, even if you know it consciously. i think people in general can often have too much of a move fast and break things attitude, and i tend to be the opposite, but my default tendency toward risk aversion can definitely go to far. balance is important. i think explicitly reminding myself to think about the realistic cost of failure can be helpful. e.g. 15 minutes or an ho…

I'm working on a project that lets you script those tedious responses really easily. Would you use something like that?

Re: Kubernetes Failure Stories

#118

Post by nyellin »

WJW wrote: post #37
barrkel wrote: post #31

Earlier quoted context omitted.

"All happy families are alike; each unhappy family is unhappy in its own way." https://en.wikipedia.org/wiki/Anna_Karenina_principle

Not to knock on Tolstoy but there are many ways in which unhappy families can be grouped together. You have the alcoholic parents, the living-vicariously-through-their-children folks, the abusive parents, etc etc etc. To tie it back to Kubernetes, you have the scheduler-does-not-schedule lessons, the ingress-not-consistent lessons, the autoscaler-killed-all-my-pods lessons, the moving-stateful-services-with-60TB-stor…

The problem is that everyone needs to learn those lessons on their own. (But I'm biased: I'm working on a troubleshooting tool for K8s which has remediation rules for all those common cases.)

Re: Kubernetes Failure Stories

#119

Post by nyellin »

cbushko wrote: post #25

The Istio one hits home. It is the single scariest thing to work with in our kubernetes clusters. We've caused several outages by changing the smallest things.

What do you use Istio for? 90% of the usecases I hear about could be solved with simpler tooling.

Re: Kubernetes Failure Stories

#120

Post by nyellin »

superbcarrot wrote: post #10
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

Kubernetes solves a problem. It isn't appropriate for all use cases, but the real problem isn't kubernetes but rather the knowledge gap around new technology.
Post reply on HN