Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

161–170 of 203 posts

Re: Kubernetes Failure Stories

#161

Post by wildermuthn »

WJW wrote: post #37
barrkel wrote: post #31

Earlier quoted context omitted.

"All happy families are alike; each unhappy family is unhappy in its own way." https://en.wikipedia.org/wiki/Anna_Karenina_principle

Not to knock on Tolstoy but there are many ways in which unhappy families can be grouped together. You have the alcoholic parents, the living-vicariously-through-their-children folks, the abusive parents, etc etc etc. To tie it back to Kubernetes, you have the scheduler-does-not-schedule lessons, the ingress-not-consistent lessons, the autoscaler-killed-all-my-pods lessons, the moving-stateful-services-with-60TB-stor…

I think you inadvertently explained why Tolstoy is right: while there are many categories of failure, within those categories each failure is it’s own big, and even for the same bug there are often contextual reasons why the bug manifests in one scenario but not another. Yes, there are alcoholics, but every alcoholic has their own lonely story of descent into addiction.

There are infinite ways to fail, and only finite ways to succeed. That relative difference between infinite and finite is what Tolstoy is getting at. And this is a helpful perspective, because then success becomes less about trying to prevent infinite failure-points and more about doing a few finite things very well.

This is why you see advice from YC like — “don’t worry about competition”, and “talk to users, talk to users, talk to users.”

Re: Kubernetes Failure Stories

#162

Post by cbushko »

nyellin wrote: post #119
cbushko wrote: post #25

The Istio one hits home. It is the single scariest thing to work with in our kubernetes clusters. We've caused several outages by changing the smallest things.

What do you use Istio for? 90% of the usecases I hear about could be solved with simpler tooling.

We are using it for code tracing, hostname ingress routing to the correct namespace, and egress blocking.

This year we should also be using it for canary deployments also.

Re: Kubernetes Failure Stories

#163

Post by rwoerz »

bcrosby95 wrote: post #14
superbcarrot wrote: post #10

Earlier quoted context omitted.

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

A Kubernetes for Kubernetes! But what about when we want to upgrade that?

Wake me up when I can finally run a bare-metal Kubernetes cluster in Kubernetes to get rid of all my hardware.

Re: Kubernetes Failure Stories

#164

Post by jrockway »

C4stor wrote: post #141
jrockway wrote: post #43

Earlier quoted context omitted.

Is Kubernetes really overcomplicated, though? Say you wanted to do a release of a new version of your app. You'd probably boot up a VM for the new version, provision it an IP, copy your code to it, and check if it's healthy. Then you'd edit your load balancer configuration to send traffic to that new IP, and drain traffic from the old IP. Then you'd shut the old instance down. That's basically what a Deployment in Ku…

So, as as a serious answer, yes. The problem isn't deployments, it's that k8s doesn't only do deployments. It does a ton of stuff, and deployments are a small part of it. If you only wanted to use k8s for deploying your code, well, you can't. That's what's was great with the UNIX programs philosophy, and what is lost in kubernetes. You can't pick the feature you need, and the feature you want may well break because o…

What is the unnecessary part, in your opinion?

I think the core is very good. You need a pool of machines, you need to be able to schedule workloads, you need the workloads to be able to talk to each other over the network, you need the cluster to reach a consensus about what state it's in, etc. That is complicated, but basically mandatory.

Intrinsically, I think you take on a lot of problems by increasing the number of computers you control. Once you have decided that you need more than 0 computers, you are in for a world of hurt -- that's the entire field of software engineering. Once you have decided to make the jump from 1 computer to 2 computers, you now have to contend with the entire field of distributed systems, just so that you can serve your website when one machine fails, or you want to upgrade Linux, or whatever. At some point, even "small" companies have 10 engineers and 10 computers, and that is right about when "some guy will handle this for me" fails and you start needing software to manage things like scheduling or deployments. You can certainly make it work for longer than that without using Kubernetes, but the returns start diminishing very quickly.

There are absolutely quirks of Kubernetes that have caused users a lot of unnecessary hurt -- mandating a 5 node etcd cluster when they could have bought an RDS instance for $70/month or something. That seems to be being fixed, though.

The way I see it is that people run into trouble with the parts that are underspecified. Ingress is my classic example -- it only provides API surface for the simplest of HTTP routing operations, and it's too simple for any real-world application. The result is that people hack in their own ingress controllers in horrifying ways ("just put an nginx.conf snippet in an annotation!") and that bites a lot of users. It goes further than things like Ingress; who is providing monitoring, tracing, and authentication? Bring your own! Who gives you a container registry? Bring your own. How do you code review changes to your cluster? Bring your own gitops! (This is all in addition to the things I mentioned in my original comment.)

There are too many things that Kubernetes should probably do because a lot of people need them, but those people have to cobble together a bunch of unrelated components to get what they need. That is nice! But one way should win, and it would save people a lot of time.

Re: Kubernetes Failure Stories

#165

Post by hodgesrm »

FridgeSeal wrote: post #130
AmericanChopper wrote: post #114

Earlier quoted context omitted.

You can make it work, but why would you want to? Databases aren’t generally something that benefits from using container orchestration. They’re not usually highly dynamic, horizontally scaling systems. Generally you’d optimize that part of your system to maximize stability and consistency. For most typical use cases I can’t see the intuitive leap required to decide that all that additional complexity is necessary to…

It that I advocate running your own Postgres setup in your own cluster instead of just renting a managed version, but I’ve run a few databases on K8s and found it pretty fine: useful for when your hosting provider doesn’t support the database you want to run (Clickhouse managed AWS service when?) or for application-specific KV-stores: EBS volumes and PVC’s are great, solid performance, kubernetes takes care of the ne…

> (Clickhouse managed AWS service when?)

Now! Altinity runs Altinity.Cloud now in AWS. Feel free to drop by.

There are also services in other clouds. Yandex runs one in their cloud and there are at least 3 in China. ClickHouse has a big and active community of providers.

Disclaimer: I work for Altinity.

Re: Kubernetes Failure Stories

#166

Post by ex_amazon_sde »

paulryanrogers wrote: post #85
ex_amazon_sde wrote: post #51

Earlier quoted context omitted.

Ex Amazon SDE here. I would pick the first method a hundred times. People would be surprised at how simple the internal infra is, given the fleet size, compared to stuff like k8s. (I'm talking about the infra that runs on bare metal, not AWS)

You mean the infra behind non-AWS services like retail Amazon.com? Or the infra underlying AWS itself?

The first one, but I'm trying not to divulge too many details on the architecture of Amazon - as you can imagine.

Re: Kubernetes Failure Stories

#167

Post by flurdy »

tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Which is why GCP's decision last year [1] to encourage people to only use 1 cluster with GKE for all of your minor, internal, exploring, testing, side and hobby projects was so frustrating.

For your main production load at work, paying management fees for many clusters is not that relevant costs. So not an issue with my clients' projects.

But for my many hobby-projects, dev and few self-employed business workloads balancing them all on as few clusters as possible is a major pain.

Before that I/you scaled it out to separation of concerns, maybe only grouping a few workloads. And then quite happily scrapping whole clusters all the time.

Now any lifecycle issues with them affects lots of independent applications. They went from cattle to pets over night.

[1] https://www.reddit.com/r/kubernetes/comments/fdgblk/google_g...

Re: Kubernetes Failure Stories

#168

Post by ex_amazon_sde »

emptysongglass wrote: post #115
fremanator wrote: post #87

Earlier quoted context omitted.

Amazon SDE here. I'm not sure what internal infra you're talking about. Amazon's certainly isn't one of them. And AWS, with its design by accretion, makes that an impossibility. Kubernetes by comparison is a paragon of clarity.

Why does AWS insist on building this way? Has anyone tried to get the teams together and say, "stop, let's evaluate and simplify where we can?" As an AWS customer, I find AWS a huge pile of overcomplicated offerings that is dense with its own jargon and way of doing things. IAM is a trash fire. Everything built on IAM like IRSA is also a trash fire. Why are there managed worker nodes, spot managed worker nodes, Farga…

> "let's evaluate and simplify where we can?"

Amazon has "Invent and Simplify" as one of the leadership principles. However, each team has its own hiring bar and culture, so some will take it seriously and some will not.

> I find AWS a huge pile of overcomplicated offerings

I fully agree. I'm speaking exclusively about the internal infrastructure, not about AWS.

Re: Kubernetes Failure Stories

#169

Post by tharkun__ »

edoceo wrote: post #73
tharkun__ wrote: post #68

Earlier quoted context omitted.

Wow you must be old. Like probably retiring now? VM/370 came out 49 years ago. So assuming you did a master's at university (I guess math at the time) I'd say you are definite retired by now. https://en.m.wikipedia.org/wiki/VM_(operating_system) (yes this is tongue in cheek sort of since you said average but you also said 'enterprise' which my cheek reads as large company :))

Bruv. You're catching downvotes for two reasons. A: ageism and B: also not realising that tools like the "old ones" used lasted decades. Release dates are a poor metric here.

I get it, nobody actually got the joke. Should've put the () first lol.

What I'm playing off of is that the poster probably meant when VMware and such took off in the enterprise. As one would see in my post history I know about what came before and that these things are not new in fact. And in many many cases still in use today which is your second point. What started 49 years ago is still in use on the IBM z series and all of that stuff has awesome backwards compatibility. My dad started his career with 360 assembler (as can also be seen in my post history).

So I guess it's ageism in the direction of _young_ people. I hope it's them down voting. Otherwise it's misjudging peoples sense of humour ;)

Re: Kubernetes Failure Stories

#170

Post by hinkley »

klohto wrote: post #153
TruthWillHurt wrote: post #142

Here's mine - We were running on Cloud Foundry, had one DevOps person that mostly dealt with Jenkins, payed for 32-64GB RAM. Decided to move to K8s (Azure AKS), Three months later we have 4-6 DevOps people dealing with networking, cross-az replication, cluster size and autoscaling, And we're paying thousands of $$$ pm for a minimum of 6 64GB VMs. FAIL

How is that K8s fault?

By that reasoning, how is anything ever anybody's fault?

Caveat emptor, my product might not be fit for purpose, and it's your own damned fault if you think it is. Consumer protection? What's that?

Post reply on HN