Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

51–60 of 203 posts

Re: Kubernetes Failure Stories

#51

Post by ex_amazon_sde »

yongjik wrote: post #30
superbcarrot wrote: post #10

Earlier quoted context omitted.

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

> There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. - C. A. R. Hoare

Ex Amazon SDE here. I would pick the first method a hundred times.

People would be surprised at how simple the internal infra is, given the fleet size, compared to stuff like k8s.

(I'm talking about the infra that runs on bare metal, not AWS)

Re: Kubernetes Failure Stories

#53

Post by Aeolun »

superbcarrot wrote: post #10
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

I think we’re already there? Maybe we’re better off looking for a cluster orchestrator orchestrator.

Re: Kubernetes Failure Stories

#54

Post by Karunamon »

colllectorof wrote: post #42
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

I've got a different read on this. It's always been complicated, it's just that each.. I don't want to say "generation", but roughly the same concept, grew up with and internalized and knew about the complexities of the tech stacks they learned, and so when something comes about that moves the abstraction one level higher than what people are used to, it's seen as unstable crap.

This isn't some ageist kids-these-days thing, I mean that the complexity has always existed, and it's always abstracted one level higher over time. Kubernetes doesn't exist because Google and Google-adjacent engineers wanted to foist off "ooh, shiny" on the world, it exists because it solves a problem with containers at scale. Containers solve a problem with resource usage at scale, which are really just an evolution of VMs and a response to their inefficient resource usage and brittleness at scale, and even they were once seen as the new hot fancy overly-complicated thing.

For context, I entered the IT workforce just as virtual machines were being introduced into the average enterprise, and a distressingly high amount of the complaints I hear about containers and k8s were being used against VMs as well.

I'm curious to see where we go from k8s. What the next level of abstraction up will be.

Re: Kubernetes Failure Stories

#55

Post by Aeolun »

jjtheblunt wrote: post #44
jrockway wrote: post #43

Earlier quoted context omitted.

Is Kubernetes really overcomplicated, though? Say you wanted to do a release of a new version of your app. You'd probably boot up a VM for the new version, provision it an IP, copy your code to it, and check if it's healthy. Then you'd edit your load balancer configuration to send traffic to that new IP, and drain traffic from the old IP. Then you'd shut the old instance down. That's basically what a Deployment in Ku…

> Is Kubernetes really overcomplicated, though? I imagine that a million voices read that and silently, immediately, think "yes, it is." and then giggle without commenting.

Now we have your comment to upvote instead :)

Re: Kubernetes Failure Stories

#56

Post by yongjik »

jrockway wrote: post #43
yongjik wrote: post #36

Earlier quoted context omitted.

Maybe I should mutter[1] "That's Kubernetes as fuck" next time I see another shenanigans involving multiple overcomplicated layers with confusing documentation interacting with each other in a way nobody can figure out. [1] Just to myself, of course.

Is Kubernetes really overcomplicated, though? Say you wanted to do a release of a new version of your app. You'd probably boot up a VM for the new version, provision it an IP, copy your code to it, and check if it's healthy. Then you'd edit your load balancer configuration to send traffic to that new IP, and drain traffic from the old IP. Then you'd shut the old instance down. That's basically what a Deployment in Ku…

I think we can draw parallels between C and Kubernetes - C can be an ideal tool for some stuff, and it's theoretically possible to write a secure program in C.

However, there's a reason why people keep saying that it's practically impossible to write C securely. C provides so many gotchas and footguns that it's not even funny. Unless your problem space really demands it, there's not much reason to use C other than "I love to live dangerously."

Similarly, Kubernetes can be an ideal choice for some people. It may even be an OK choice for some others. But there are many for whom there's no reason to use k8s other than "another shiny item on my resume."

Re: Kubernetes Failure Stories

#58

Post by Turbots »

colllectorof wrote: post #42
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start…

Or:

- we need to bring value into production faster - our monolithic app deployed manually and running on a bare metal server has served us well, but needs to be split up into some finer grained services to allow for more agile software delivery - we don't want to manually deploy and manage those, since it would multiply our work. Hey, let's use virtualized compute, network and storage that can be provisioned and managed more automated - rinse and repeat for VM -> container. More frequent, smaller deployables get rolled out on a container platform in an automated way. You need orchestration when you're building 10+ services continuously, regular VMs with a Docker installation won't cut it. - Setup good CI/CD pipelines that can continuously build and deploy your services to production, without downtime.

But the most important thing is: - as a company, stop building your own platform - look at the value line and find out what piece of the software is above it (aka: what software is necessary, but not valuable for my business by itself) - typically, only the actual applications are valuable, all the other shit like a datacenter, infra, virtualization, networking, storage and yes, Kubernetes are all below the value line. - buy or consume vendored solutions for anything that is below your value line, let them handle the complexity and integrations

Re: Kubernetes Failure Stories

#59

Post by hinkley »

tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

There's a fun one I hadn't thought about that was mentioned here sometime last summer:

What do you do if your Raft data is in zookeeper and you want to move it to etcd or consul? We need 'migrations' for our non-db data too.

Re: Kubernetes Failure Stories

#60

Post by Turbots »

superbcarrot wrote: post #10
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

What, like the [Cluster API](https://cluster-api.sigs.k8s.io/), [RedHat Advanced Cluster Mgmt](https://www.redhat.com/en/resources/advanced-cluster-managem...), [Rancher](https://rancher.com/), [VMware Tanzu Mission Control](https://tanzu.vmware.com/mission-control)?

Most large enterprises that have been using Kubernetes for a while, have many dozens or hundreds of Kubernetes clusters running.

Clusters for different Business Groups, environments or clouds.

Something needs to manage User Access, Network policies, storage policies/quotas, Security Policies across those clusters.

Something needs to provision, scale, patch and upgrade those clusters in an automated way. There are 4 major releases of Kubernetes a year, probably 20-30 minor releases and patch versions. Enterprises need to keep those up to date.

Terraform, Ansible or Chef/Puppet won't solve that problem without spending hundreds of mandays per year on it.

Post reply on HN