Kubernetes Failure Stories • Hacker News Board

Re: Kubernetes Failure Stories

#151

Post by dilyevsky » Mon, Jan 21, 2019, 5:06 AM UTC

aprdm wrote: post #147

m0zg wrote: post #51

It's not for everyone and it has significant maintenance overhead if you want to keep it up to date _and_ can't re-create the cluster with a new version every time. This is something most people at Google are completely insulated from in the case of Borg, because SRE's make infrastructure "just work". I wish there was something drastically simpler. I don't need three dozen persistent volume providers, or the ability…

In places worked we usually had a vmware cluster, load balancer, NFS for shared data when necessary and DNS set up (e.g: through consul). This setup is very, very simple and scalable. There is very little to gain IMO on moving to Kubernetes. Consul, VSphere and load balancers have APIs and you can write tools to do everything that K8s does.

Scalable NFS, riiite.

Re: Kubernetes Failure Stories

#152

Post by DavidWoof » Mon, Jan 21, 2019, 5:12 AM UTC

clofresh wrote: post #60

DavidWoof wrote: post #58

Earlier quoted context omitted.

I've had small-ish docker swarms in production for a couple of years as well, and I really don't understand why it doesn't seem to be popular at all. I feel like I need to move to K8S just because swarm seems to be going away, but I'm really not seeing the technical advantages at all. If someone could point me to an article explaining why k8s is so much better than swarm, I'd really appreciate it. Are the big advanta…

I'm also constantly surprised at how unpopular docker swarm is given that everyone already uses docker itself. Why do you think swarm is going away though? I love the idea of just using my docker compose file as my deployment config.

"Going away" might be an overstatement, but I don't see much evidence of it being in use. When I search for help on topics, I don't see much beyond the primary docs. There doesn't seem to be much of a community for swarm out there, and now even the desktop versions of docker come with k8s.

Re: Kubernetes Failure Stories

#153

Post by Aeolun » Mon, Jan 21, 2019, 5:47 AM UTC

dlor wrote: post #136

cookiecaper wrote: post #120

Earlier quoted context omitted.

k8s is meant to be hard to use. You're supposed to rent space on a k8s cluster from Google. Google has been pumping millions into marketing k8s as a mechanism to improve GCP adoption and establish a foothold in the cloud provider space.

I'm not exactly sure what point you're trying to make here. k8s is not meant to be a paas, but no one is trying to make k8s harder to use. I work at Google on a large team of engineers dedicated to making it as easy as possible to use.

What would make it easy is a good non-gcp GUI

Re: Kubernetes Failure Stories

#154

Post by aprdm » Mon, Jan 21, 2019, 5:49 AM UTC

dilyevsky wrote: post #151

aprdm wrote: post #147

Earlier quoted context omitted.

In places worked we usually had a vmware cluster, load balancer, NFS for shared data when necessary and DNS set up (e.g: through consul). This setup is very, very simple and scalable. There is very little to gain IMO on moving to Kubernetes. Consul, VSphere and load balancers have APIs and you can write tools to do everything that K8s does.

Scalable NFS, riiite.

If you have some time to read "how Google works" you would be surprised by how long the company ran on NFS. I assume there are lots of workloads running on Borg to this day on top of NFS. If that isn't enough for you you should have a look in the client list of Isilon and see which kind of work they do, in case you ever attend a SIGGRAPH most of what you see is built on top of NFS, so, essentially, all of the computer graphics you see in movies. At last job our NFS cluster did 300 000 IOPS with 82gb/s throughput

Re: Kubernetes Failure Stories

#155

Post by Aeolun » Mon, Jan 21, 2019, 5:55 AM UTC

devthane wrote: post #122

pquerna wrote: post #111

Earlier quoted context omitted.

The point is not about the minimum conformance, but rather the lock-in provided by the maximum configuration / extensions of each vendor. Take AWS EKS as an example. Their feature page[1] does mention conformance. Then it mentions 20 other non-conformance focused features that create an effective lock-in. k8s is becoming like OpenStack in this regards. You need to embrace a vendor version of k8s in order to have a fu…

This isn’t my experience at all. I as one person taught myself over the past couple years docker then Kubernetes and am now managing a small 3 node bare metal cluster on my own. But using rancher 2.0 has helped a bunch to ease me into it. Now I feel comfortable enough to start up a cluster on my own without it.

I strongly feel that Rancher 2.0 is just not ready for release yet. I run into so many weird edge cases using it, and don’t get me started on persistent storage...

Longhorn is pretty easy to set up, but it seems to have some issues actually working (and slow).

Re: Kubernetes Failure Stories

#156

Post by m0zg » Mon, Jan 21, 2019, 5:56 AM UTC

aprdm wrote: post #154

dilyevsky wrote: post #151

Earlier quoted context omitted.

Scalable NFS, riiite.

If you have some time to read "how Google works" you would be surprised by how long the company ran on NFS. I assume there are lots of workloads running on Borg to this day on top of NFS. If that isn't enough for you you should have a look in the client list of Isilon and see which kind of work they do, in case you ever attend a SIGGRAPH most of what you see is built on top of NFS, so, essentially, all of the compute…

82gb/s (assuming you mean gigabit) is _per-node_ throughput at Google (or FB, or I assume Amazon/Microsoft -- they all use 100GbE networks now). 300K IOPS is probably per-node, too, at this point. :-)

Re: Kubernetes Failure Stories

#157

Post by AaronFriel » Mon, Jan 21, 2019, 5:57 AM UTC

I just went through all of the post-mortems for my own company's purposes of evaluating Kubernetes. I've been running Kubernetes clusters for about a year and a half and have run into a few of these, but here's what I found striking:

* About half of the post-mortems involve issues with AWS load balancers (mostly ELB, one with ALB) * Two of the post-mortems involve running control plane components dependent on consensus on Amazon's `t2` series nodes

This was pretty surprising to me because I've never run Kubernetes on AWS. I've run it on Azure using acs-engine and more recently AKS since its release, and on Google Cloud Platform using GKE; and it's a good reminder not to to run critical code on T series instances because AWS can and will throttle or pause these instances.

Re: Kubernetes Failure Stories

#158

Post by [deleted] » Mon, Jan 21, 2019, 5:59 AM UTC

m0zg wrote: post #156

aprdm wrote: post #154

Earlier quoted context omitted.

If you have some time to read "how Google works" you would be surprised by how long the company ran on NFS. I assume there are lots of workloads running on Borg to this day on top of NFS. If that isn't enough for you you should have a look in the client list of Isilon and see which kind of work they do, in case you ever attend a SIGGRAPH most of what you see is built on top of NFS, so, essentially, all of the compute…

82gb/s (assuming you mean gigabit) is _per-node_ throughput at Google (or FB, or I assume Amazon/Microsoft -- they all use 100GbE networks now). 300K IOPS is probably per-node, too, at this point. :-)

[deleted]

Re: Kubernetes Failure Stories

#159

Post by akhilcacharya » Mon, Jan 21, 2019, 6:08 AM UTC

malkia wrote: post #132

m0zg wrote: post #51

It's not for everyone and it has significant maintenance overhead if you want to keep it up to date _and_ can't re-create the cluster with a new version every time. This is something most people at Google are completely insulated from in the case of Borg, because SRE's make infrastructure "just work". I wish there was something drastically simpler. I don't need three dozen persistent volume providers, or the ability…

It took me a while to get comfortable in Borg (and in general that your binary can take hundredths of verbosely written command-line arguments (coming from gamedev, I was in a bit of shock state for a while)... But then got used to it - still I felt I could never fully internalize the evaluation rules - but the other tooling (diffing) really helped in that respect. One thing I've really appreciated, was how one could…

From this comment thread I’m beginning to think I’m one of the few people on HN that hasn’t used Borg.

All of this seems way more complicated than the tools we use at my company. Is there a specialized need here I’m not seeing?

Re: Kubernetes Failure Stories

#160

Post by hjacobs » Mon, Jan 21, 2019, 6:17 AM UTC

Aeolun wrote: post #153

dlor wrote: post #136

Earlier quoted context omitted.

I'm not exactly sure what point you're trying to make here. k8s is not meant to be a paas, but no one is trying to make k8s harder to use. I work at Google on a large team of engineers dedicated to making it as easy as possible to use.

What would make it easy is a good non-gcp GUI

I agree, some better UI for end users would be awesome: Kubernetes Dashboard kind of works, but is pretty limited and more a "kubectl in the browser". There are a ton of nicer UIs for Kubernetes, but they are all part of the value-add of proprietary platforms AFAIK (think about all the managed K8s offerings out there).