Kubernetes Failure Stories • Hacker News Board

Re: Kubernetes Failure Stories

#21

Post by dvnguyen » Sun, Jan 20, 2019, 6:32 PM UTC

Having used Docker Compose/Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn't really understand why, but overall I was relatively happy with them. Since Kubernetes seems to have won, I decided to learn it but got some disappointments.

The first disappointment is setting up a local development environment. I failed to get minikube running on a Macbook Air 2013 and a Ubuntu Thinkpad. Both have VTx enabled and Docker and VirtualBox running flawlessly. Their online interactive tutorial was good though, enough for the learning purpose.

Production setup is a bigger disappointment. The only easy and reliable ways to have a production grade Kubernetes cluster are to lock yourself into either a big player cloud provider, or an enterprise OS (Redhat/Ubuntu), or introduce a new layer on top of Kubernetes [1]. Locking myself into enterprise Ubuntu/Redhad is expensive, and I'm not comfortable with adding a new, moving, unreliable layer on top of Kubernestes which is built on top of Docker. One thing I like about the Docker movement is that they commoditize infrastructure and reduce lock-ins. I can design my infrastructure so it can utilize an open source based cloud product first and easily move to others or self-host if needed. With Kubernetes, things are going the other way. Even if I never moved out of the big 3 (AWS/Azure/GCloud), the migration process could be painful since their Kubernetes may introduce further lock-ins for logging, monitoring, and so on.

[1]: https://kubernetes.io/docs/setup/pick-right-solution/

Re: Kubernetes Failure Stories

#22

Post by sabareesh » Sun, Jan 20, 2019, 6:34 PM UTC

For our team size kubernetes was overwhelming so we stayed with docker swarm but we were afraid docker might drop swarm in favor of kubernetes but that didnt happen yet.

Re: Kubernetes Failure Stories

#23

Post by hjacobs » Sun, Jan 20, 2019, 6:35 PM UTC

peterwwillis wrote: post #19

Dang. I wish I had my SRE Wiki up and running already, or I'd add a "public postmortems" section.

Looking forward to your public postmortems.. (either yours or whatever you find in the wild)

Re: Kubernetes Failure Stories

#24

Post by awinter-py » Sun, Jan 20, 2019, 6:40 PM UTC

hjacobs wrote: post #20

awinter-py wrote: post #8

Beyond strictly runtime failures, 2018 feels like the year that most of my friends tried kube but not everybody stayed on. The adoption failures are mostly networking issues specific to their cloud. Performance and box limits vary widely depending on cloud vendor and I still don't quite understand the performance penalty of the different overlay networks / adapters.

> adoption failures are mostly networking issues specific to their cloud Do you have any pointers/write-ups with more information or plans in this direction? I would be interested to learn more.

this article is about feature completeness of the different managed kubes

https://kubedex.com/google-gke-vs-microsoft-aks-vs-amazon-ek...

It's pretty easy to dig up speed tests of the overlay networks, but a lot of these are just rating userspace overlay networks. The new hotness is the plugins provided by the cloud vendor which integrate with their SDN, and I haven't seen a good benchmark for those yet.

Most interesting reading will be to look up managed kube networking plugins on github and look for open/closed issues with lots of stars.

Re: Kubernetes Failure Stories

#25

Post by hjacobs » Sun, Jan 20, 2019, 6:42 PM UTC

dvnguyen wrote: post #21

Having used Docker Compose/Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn't really understand why, but overall I was relatively happy with them. Since Kubernetes seems to have won, I decided to learn it but got some disappointments. The first disappointment is setting up a local development environment. I failed to get minikube running on a Macbook A…

I never used Docker Swarm (so can't compare), but I don't fully understand your point about Kubernetes cloud lock-in. Certainly there are important differences in networking, load balancing, persistent volumes, and other cloud features, but that's not something any platform can just hide/eliminate (e.g. think about AWS ELB/ALB/NLB vs Google Load Balancer). The Kubernetes concepts (Deployment, Ingress, Service) still work mostly the same for the user across clouds. Some other details like non-standardized Ingress annotations are obviously due to not having them agreed in Kubernetes core API (nginx ingress supports other annotations than say Google LB or Skipper).

Re: Kubernetes Failure Stories

#26

Post by bonesss » Sun, Jan 20, 2019, 6:46 PM UTC

hjacobs wrote: post #25

dvnguyen wrote: post #21

Having used Docker Compose/Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn't really understand why, but overall I was relatively happy with them. Since Kubernetes seems to have won, I decided to learn it but got some disappointments. The first disappointment is setting up a local development environment. I failed to get minikube running on a Macbook A…

I never used Docker Swarm (so can't compare), but I don't fully understand your point about Kubernetes cloud lock-in. Certainly there are important differences in networking, load balancing, persistent volumes, and other cloud features, but that's not something any platform can just hide/eliminate (e.g. think about AWS ELB/ALB/NLB vs Google Load Balancer). The Kubernetes concepts (Deployment, Ingress, Service) still…

Most of the kubernetes toolchain provides nice support for delineating the requirements from separate cloud providers, too. Compared to most alternatives, a little HELM magic to support hybrid cloud installations is a piece of cake.

Re: Kubernetes Failure Stories

#27

Post by pepemon » Sun, Jan 20, 2019, 7:03 PM UTC

hjacobs wrote: post #17

gspetr wrote: post #15

Earlier quoted context omitted.

Is there also a list for Docker failure stories?

IMHO this would be less interesting, some people already run other container runtimes such as containerd with Kubernetes (e.g. Datadog: https://www.youtube.com/watch?v=2dsCwp_j0yQ ) --- so Docker might stay as some user interface for local development, but I would not know what "Docker failures" would be in the future.

Docker is using containerd under the hood as its container runtime component.

Re: Kubernetes Failure Stories

#28

Post by stingraycharles » Sun, Jan 20, 2019, 7:05 PM UTC

lucidone wrote: post #18

coredog64 wrote: post #5

Earlier quoted context omitted.

Microservices failure stories? “All of them. The End”

I'm consulting on a micro services back end right now with mostly prior experience with monoliths. What is the selling point that drives companies down this direction? It's insane, and my client keeps trying to hire new developers and bring on more consultants to build this thing, but the amount of knowledge required is more than any one person can handle. I have similar issues with their choice of db (nosql) and its…

The biggest reason for choosing microservices _should_ be scaling development teams: microservices allow multiple teams to work on different code bases, without stepping on each other's toes.

What actually happens, tough, is that (uninformed) people choose it because they think it brings them scalability (wrong), it's more cloud compatible (wrong) or the worst offender, it's more modern.

Re: Kubernetes Failure Stories

#29

Post by pepemon » Sun, Jan 20, 2019, 7:07 PM UTC

dvnguyen wrote: post #21

Having used Docker Compose/Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn't really understand why, but overall I was relatively happy with them. Since Kubernetes seems to have won, I decided to learn it but got some disappointments. The first disappointment is setting up a local development environment. I failed to get minikube running on a Macbook A…

Out of curiosity, why do you need RHEL or subscription from Canonical for the production Kubernetes setup? What's wrong with plain Ubuntu or CentOS?

Re: Kubernetes Failure Stories

#30

Post by fhrow4484 » Sun, Jan 20, 2019, 7:08 PM UTC

hjacobs wrote: post #25

dvnguyen wrote: post #21

Having used Docker Compose/Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn't really understand why, but overall I was relatively happy with them. Since Kubernetes seems to have won, I decided to learn it but got some disappointments. The first disappointment is setting up a local development environment. I failed to get minikube running on a Macbook A…

I never used Docker Swarm (so can't compare), but I don't fully understand your point about Kubernetes cloud lock-in. Certainly there are important differences in networking, load balancing, persistent volumes, and other cloud features, but that's not something any platform can just hide/eliminate (e.g. think about AWS ELB/ALB/NLB vs Google Load Balancer). The Kubernetes concepts (Deployment, Ingress, Service) still…

> I don't fully understand your point about Kubernetes cloud lock-in

The kubernetes folks describe a tentative solution to cloud lock-in here: https://kubernetes.io/docs/concepts/cluster-administration/f... OP isn't the only one with those concerns.

It would be nice when you can switch your cluster load from any of the cloud providers, or your own on-prem setup as you go. For instance, I could see people wanting to have a default small cluster on their on-prem setup, and be ready to scale on cloud when needed.