Live data from Hacker News

Kubernetes Failure Stories

k8s.af

91–100 of 203 posts

Re: Kubernetes Failure Stories

#91

Definitely a good one: "Experiences with running PostgreSQL on Kubernetes - Gravitational - blog post 2018" For anyone who thinks running a database in a container environment is a neat idea, think again. I am guilty of using containers for temporary test databases, but the thought of running production databases in containers sends shivers down my spine.

What about single-host container (i.e. regular docker host, not k8s) with data partition mounted from host?

And using delegated, for local development, multiple versions with a few commands is really great.

Re: Kubernetes Failure Stories

#92

Earlier quoted context omitted.

Ex Amazon SDE here. I would pick the first method a hundred times. People would be surprised at how simple the internal infra is, given the fleet size, compared to stuff like k8s. (I'm talking about the infra that runs on bare metal, not AWS)

Amazon SDE here. I'm not sure what internal infra you're talking about. Amazon's certainly isn't one of them. And AWS, with its design by accretion, makes that an impossibility. Kubernetes by comparison is a paragon of clarity.

I can't even imagine, sometimes I feel like AWS offers this pretty facade of "scale it up easy" and in reality its like "wait, I can't even see why this thing is failing because its a layer below, and AWS support costs mega bucks"

Re: Kubernetes Failure Stories

#94
post #93

I will die on the hill that almost nobody actually needs kubernetes. You are not Netflix or Google.

I had a disastrous experience with this.

Three years ago I joined a startup to help with infrastructure setup. They wanted to use Kubernetes to install their monolithic django/celery AI/web single app using the same script both for their cloud environment and customers servers with single node installation. I didn't last more than a month.

Since then, and for every company I worked with, no single time Kubernetes was proposed as a solution for the problem it really solves. It was always though of as the magical solution for whatever problem we had (or didn't even have).

Re: Kubernetes Failure Stories

#95
post #93

I will die on the hill that almost nobody actually needs kubernetes. You are not Netflix or Google.

Feel free, but it's a bad hill.

I can write my own config management, secret management, volume mounting, deployments, replica scaling, hardware affinity, antiaffinity for resilience, rollover kicking out e.g. staging if there's not enough room to schedule production workloads, and the list goes on.

I can also figure out how to install and upgrade each piece of software in my universe, across a few different languages.

Or I can just use kubernetes.

From the perspective of my developers, they tell me a few things like: "I need 4GB RAM, 2 cores, I need access to the postgres secret to talk to it, and here is my hostname" and while they can tell me many, many more things, with that little information, they can have a full application in one of several different languages in minutes.

If I were to hand roll this, it'd be held together by prayers and duct tape. And I lose out on a huge active community building tools around this. I now have istio and get cross service telemetry. For free. I can set networking rules up setting up QOS between arbitrary services, both inside my network and outside. All applied behind a single interface. There's a lot of stuff I get for free, and as my team discovers things it needs to do, I have a consistent layer to do that all behind.

I'm one guy. I don't work for Google. But my team still manages _a lot_ of services, both on application end, and things like queues, databases, etc. You can say we don't need all of that, but you'd be wrong. We arguably could use more. I can reason about my infrastructure in both the small and the large.

Then you get into the idea of transferring to other companies. If someone comes into my org, they can look at what I have and see more or less everything there is in any part of my infrastructure. It's all right there, laid out in YAML. If I were to switch companies, or even teams, it'd be the same. Having a lingua franca of common terms and ideas is super critical there. To say nothing of all the work happening in the ecosystem allowing me to help my users ship features faster to our stakeholders. Doing this in VMs would be a literal nightmare.

So, respectfully, I believe your opinion is not correct here. You personally may not need it, but it's not very long before you get to reasonable amounts of services for any project that's sufficiently complex, in my experience. And even when I have just 1 or 2 I still need many of the primitives of kubernetes.

Re: Kubernetes Failure Stories

#96

Earlier quoted context omitted.

Is Kubernetes really overcomplicated, though? Say you wanted to do a release of a new version of your app. You'd probably boot up a VM for the new version, provision it an IP, copy your code to it, and check if it's healthy. Then you'd edit your load balancer configuration to send traffic to that new IP, and drain traffic from the old IP. Then you'd shut the old instance down. That's basically what a Deployment in Ku…

> Is Kubernetes really overcomplicated, though? I imagine that a million voices read that and silently, immediately, think "yes, it is." and then giggle without commenting.

how is this even a question in 2021

Re: Kubernetes Failure Stories

#97
post #2

This is a compilation of gotcha-discovery-reports, distributed across the surface area of K8s (which, since K8s is huge, covers many nooks and crannies). This is not a compilation of "K8s sucks, here are 10 reasons why", which is what my kneejerk expectation was. (maybe I am too cynical...). Overall, this is a fantastic index to some very interesting resources. There's an old saying that you learn more from failure t…

At the same time your initial reaction must at the very least say something about how k8 is different from other technologies where you won't have this reaction.

Re: Kubernetes Failure Stories

#98

Definitely a good one: "Experiences with running PostgreSQL on Kubernetes - Gravitational - blog post 2018" For anyone who thinks running a database in a container environment is a neat idea, think again. I am guilty of using containers for temporary test databases, but the thought of running production databases in containers sends shivers down my spine.

I don’t know why anybody would presume that a technology focused on ephemeral resource provisioning would be a suitable place to put your persistence layer...

That said, I don’t think it’s a sin at all to use it for testing. My default local dev setup is to use a Postgres container. But persistence is very much not required in that situation.

Re: Kubernetes Failure Stories

#99

Earlier quoted context omitted.

Is Kubernetes really overcomplicated, though? Say you wanted to do a release of a new version of your app. You'd probably boot up a VM for the new version, provision it an IP, copy your code to it, and check if it's healthy. Then you'd edit your load balancer configuration to send traffic to that new IP, and drain traffic from the old IP. Then you'd shut the old instance down. That's basically what a Deployment in Ku…

> Is Kubernetes really overcomplicated, though? I imagine that a million voices read that and silently, immediately, think "yes, it is." and then giggle without commenting.

Those people clearly haven’t tried to setup vm live migration or done anything with openstack. Or deployed things at scale on Borg or Tupperware. I can go on...

Re: Kubernetes Failure Stories

#100
post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

> Meanwhile, the complexity of our technology goes up and its reliability in practice goes down.

I don’t agree with this at all. Reliability in practice is improving at a phenomenal pace. 10 years ago maintenance outages were a normal feature of every service, and unplanned outages were perfectly ordinary occurrences. Consumers today expect a much higher level of availability and reliability, which they receive rather consistently. Today it takes much less resource to produce a much more reliable system then you would have been able to produce in the rather recent past.

Post reply on HN