Kubernetes Failure Stories • Hacker News Board

Re: Kubernetes Failure Stories

#41

Post by dankohn1 » Sun, Jan 20, 2019, 7:41 PM UTC

fhrow4484 wrote: post #30

hjacobs wrote: post #25

Earlier quoted context omitted.

I never used Docker Swarm (so can't compare), but I don't fully understand your point about Kubernetes cloud lock-in. Certainly there are important differences in networking, load balancing, persistent volumes, and other cloud features, but that's not something any platform can just hide/eliminate (e.g. think about AWS ELB/ALB/NLB vs Google Load Balancer). The Kubernetes concepts (Deployment, Ingress, Service) still…

> I don't fully understand your point about Kubernetes cloud lock-in The kubernetes folks describe a tentative solution to cloud lock-in here: https://kubernetes.io/docs/concepts/cluster-administration/f... OP isn't the only one with those concerns. It would be nice when you can switch your cluster load from any of the cloud providers, or your own on-prem setup as you go. For instance, I could see people wanting to h…

Federation is a solution for multi-cloud and hybrid cloud, which is a different problem than cloud portability.

Re: Kubernetes Failure Stories

#42

Post by parasubvert » Sun, Jan 20, 2019, 7:43 PM UTC

fhrow4484 wrote: post #30

hjacobs wrote: post #25

Earlier quoted context omitted.

I never used Docker Swarm (so can't compare), but I don't fully understand your point about Kubernetes cloud lock-in. Certainly there are important differences in networking, load balancing, persistent volumes, and other cloud features, but that's not something any platform can just hide/eliminate (e.g. think about AWS ELB/ALB/NLB vs Google Load Balancer). The Kubernetes concepts (Deployment, Ingress, Service) still…

> I don't fully understand your point about Kubernetes cloud lock-in The kubernetes folks describe a tentative solution to cloud lock-in here: https://kubernetes.io/docs/concepts/cluster-administration/f... OP isn't the only one with those concerns. It would be nice when you can switch your cluster load from any of the cloud providers, or your own on-prem setup as you go. For instance, I could see people wanting to h…

Kubernetes federation doesn’t really help with lock-in. Its for multi-cluster management. It’s also kind of stalled / going to be rethought - it has too many issues to be used at scale. Some may still like it of course.

There are two points of lock-in: beneath Kubernetes or on top of Kubernetes.

You still need to install and manage your cluster, so the low level lock-in of a distro is hard to avoid. What you may be looking for is kubeadm. https://kubernetes.io/docs/reference/setup-tools/kubeadm/kub...

It is the closest primitive to help standardize “here’s how to install and run a K8s cluster in a certified way”. It’s unfortunately incomplete.

As for what’s on top of Kubernetes, that will be an area ripe for completion and lock-in.

Re: Kubernetes Failure Stories

#43

Post by tyingq » Sun, Jan 20, 2019, 7:45 PM UTC

dvnguyen wrote: post #21

Having used Docker Compose/Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn't really understand why, but overall I was relatively happy with them. Since Kubernetes seems to have won, I decided to learn it but got some disappointments. The first disappointment is setting up a local development environment. I failed to get minikube running on a Macbook A…

Digital Ocean's K8S offering is out of beta now: https://www.digitalocean.com/products/kubernetes/

Re: Kubernetes Failure Stories

#44

Post by parasubvert » Sun, Jan 20, 2019, 7:45 PM UTC

pepemon wrote: post #29

dvnguyen wrote: post #21

Having used Docker Compose/Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn't really understand why, but overall I was relatively happy with them. Since Kubernetes seems to have won, I decided to learn it but got some disappointments. The first disappointment is setting up a local development environment. I failed to get minikube running on a Macbook A…

Out of curiosity, why do you need RHEL or subscription from Canonical for the production Kubernetes setup? What's wrong with plain Ubuntu or CentOS?

It’s common to pay for things to make them easier to configure/manage.

Red Hat OpenShift on RHEL, Pivotal Container Service on Ubuntu, Red Hat’s nextgen CoreOS based Kubernetes, Canonical’s Charmed Kubernetes Distribution on Ubuntu, etc. all have different config management , install, upgrade, patching mechanisms that vary from Ansible, to Terraform, to BOSH, to Juju. Some handle PXE bare metal, some don’t. Etc.

There usually are free / no pay versions of the above that you can use self-supported, but then you’ll also need to coordinate your own upgrades and use community forums for q&a rather than being able to contractually have someone looking out for you and answering your questions.

If you’d prefer to avoid lock-in, All of that plumbing would otherwise have to be configured and scripted yourself with your chosen toolchain plus the newer “k8s small tools” like Kubeadm, Kops, Kube-spray, etc.

As the old saying goes, open source is only free (as in beer) if your time has no value.

Re: Kubernetes Failure Stories

#45

Post by zzzcpan » Sun, Jan 20, 2019, 7:47 PM UTC

bdcravens wrote: post #34

I've started the planning phase of a Kubernetes course, geared toward developers more so than the enterprise gatekeepers. As I read stories like these, I jump between different thoughts and feelings: 1) no matter what I think I know, there's too many dark corners to create an adequate course 2) K8S is such a dumpster fire that I shouldn't encourage others 3) there's a hell of an opportunity here Thoughts? Worth pursu…

I think it's ok not to know something to make a course for it. Even if only to learn it better. But I'd be careful given 2, pursuing this could lead to burnout.

There is an opportunity for anything infrastructure related.

Re: Kubernetes Failure Stories

#46

Post by parasubvert » Sun, Jan 20, 2019, 7:56 PM UTC

bdcravens wrote: post #34

I've started the planning phase of a Kubernetes course, geared toward developers more so than the enterprise gatekeepers. As I read stories like these, I jump between different thoughts and feelings: 1) no matter what I think I know, there's too many dark corners to create an adequate course 2) K8S is such a dumpster fire that I shouldn't encourage others 3) there's a hell of an opportunity here Thoughts? Worth pursu…

All three. It’s a gold rush, but as with any gold rush, conditions are hard going - that’s why there’s an opportunity.

Best way to think of Kubernetes is that it was designed to be a successful open source project that was widely adopted as a standard foundation to build products. It wasn’t designed to be a useable product on its own.

We are at the equivalent stage of Slackware and SLS and Debian Red Hat pre-1.0 stages of GNU/Linux distros circa 1994. Red Hat eventually ran away most of the money by the late 90s, but in the meantime, lots of opportunity to fill an unmet need.

Re: Kubernetes Failure Stories

#47

Post by bostik » Sun, Jan 20, 2019, 7:57 PM UTC

devereaux wrote: post #13

lykr0n wrote: post #12

Earlier quoted context omitted.

Network is a high performance system, and each layer you add adds latency. Consider a traditional monolithic application. In comes your HTTP request in one end, a bunch of cross thread communication happens, and database queries come out the other end. With that, you have 2 points of network communication. Now with a micro-service, you might have 4 or 5 applications that are needed to replace the above monolith. Thro…

So true. For some low latency applications, anything above the bare minimal virtualization is not acceptable. For what I do, in theory, many things should not impact results. In practice, anything that upon measurement impact results is stripped away. Think A/B testing but for every single component - including the major version of say the python interpreter. That's how you end up running many things baremetal. I'll…

If you do trading or adtech (which, let's face it, is trading) then yes, the overhead from virtualisation and cloud environments is an unacceptable tradeoff.

For most companies and their operating environments cloud gives flexibility in capacity planning, freedom to experiment wildly, access to practically unlimited storage[ß], plus reliable perimeter load balancing. And of course, for most companies engineering costs far more than compute. Paying the premium for cloud environment makes business sense if it means your development teams don't have to spend time waiting for available resources.

Funnily enough, even a number of high-end trading firms (including HFT shops) are moving to cloud. Number crunching for models, backtesting, CI, analysis and simulation pipelines, ... All of those require resources and rarely need to operate in tandem with the real-time trading systems. The same flexibility, ease of capacity expansion and freedom to bring transient resources online as needed saves on expensive development time.

If you're big enough, or need to operate fast enough then cloudless and baremetal are going to bring better ROI. Most companies aren't at these extreme ends, though.

ß: within limits; if you need to produce and store petabytes of changing data on a daily basis...

Re: Kubernetes Failure Stories

#48

Post by parasubvert » Sun, Jan 20, 2019, 8:07 PM UTC

dehrmann wrote: post #10

tnolet wrote: post #3

I'd be interested in a related "microservices failure stories". Must be a big overlap with this.

I have two. One was caused by data inconsistency between services and regions. One is more hypothetical: the microservices had gotten to the point that no one knew how to start the system if all services are down, and it's possible that services have circular dependencies to the point that it would be incredibly hard to do a cold start.

I’d think the best way to handle a cold start is to have each microservice fail fast and be wrapped in a supervisor. They will converge to uptime.

Unless you have a truly circular dependency, at which point they probably should be collapsed into a single service.

Re: Kubernetes Failure Stories

#49

Post by cygned » Sun, Jan 20, 2019, 8:15 PM UTC

I am a developer and I find k8s frustrating. To me, its documentation is confusing and scattered among too many places (best example: overlay networks). I have read multiple books and gazillions of articles and yet I have the feeling that I am lacking the bigger picture.

I was able to set it up successfully a couple of times, with more or less time required. Last time, I gave up after four days because I realized that what I need was a "I just want to run a simple cluster" solution and while k8s might provide that, its flexibility makes it hard for me to use it.

Re: Kubernetes Failure Stories

#50

Post by stunt » Sun, Jan 20, 2019, 8:18 PM UTC

Kubernetes solves a problem that most of the companies don't have. That is why I don't understand why the hype around it is so big.

For the majority, it just adds a little value when you compare to added complexity to infrastructure and the cost of a learning curve and the ongoing operation and maintenance.