Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

srcco.de

Post reply on HN
Print view View original

231–240 of 242 posts

Re: Kubernetes Failure Stories

#231

Post by jordanbeiber »

jordanbeiber wrote: post #230
romeisendcoming wrote: post #215

Earlier quoted context omitted.

Did you consider that when there is a manual process it is in place to bring combined attention to what surely is (by 2019) a critical section: that needs consensus not provided by some monitoring hook. Sure, everything is about automation and it has been since 1999 in my experience.

Of course. I’ve been employed and consulted enterprise IT&Dev for almost 20 years. The amount of people heating office spaces at your non-tech large enterprise is astounding in my opinion. I enjoy discussing the reasons for this, but it’s a lenghty one! In super-short: lack of competency, meaning IT support and tools are not used even remotely optimal. This lack of competency, which starts at the top, results in laug…

Also, David Greaber has some points with his ”Bullshit Jobs”.

I personally believe a society built on and around services in the end need either massive amounts of bullshit jobs or basic income of some sort.

Re: Kubernetes Failure Stories

#232

Post by pgwhalen »

013a wrote: post #203
pgwhalen wrote: post #116

Earlier quoted context omitted.

I very much agree that kubernetes is useful in an environment that doesn’t need to scale, but do tell how it enables consolidated logging and monitoring, since my medium/small shop is spending quite some time setting up our own infrastructure for it.

Installing a managed log ingestor is stupidly easy in Kubernetes. For example, on GCP here's the guide to getting it done [1]. Two kubectl commands, and you get centralized logging across hundreds of nodes in your cluster and thousands of containers within them. Most other platforms (like Datadog) have similar setups. Infrastructure level monitoring is also very easy. For example, if you're on Datadog, you flip KUBER…

I will admit that these things are slightly easier Kubernetes, my original point was mostly just to say that Kubernetes itself doesn't really provide any of these things in meaningful ways - you just described a bunch of separate, nontrivial systems, that solve many but not all logging/monitoring needs.

Re: Kubernetes Failure Stories

#233

Post by romeisendcoming »

jordanbeiber wrote: post #230
romeisendcoming wrote: post #215

Earlier quoted context omitted.

Did you consider that when there is a manual process it is in place to bring combined attention to what surely is (by 2019) a critical section: that needs consensus not provided by some monitoring hook. Sure, everything is about automation and it has been since 1999 in my experience.

Of course. I’ve been employed and consulted enterprise IT&Dev for almost 20 years. The amount of people heating office spaces at your non-tech large enterprise is astounding in my opinion. I enjoy discussing the reasons for this, but it’s a lenghty one! In super-short: lack of competency, meaning IT support and tools are not used even remotely optimal. This lack of competency, which starts at the top, results in laug…

Agreed with your experiences. Identified my niche a long time ago in HPC and scientific development enterprise and core internet services (DNS, IP routing) + security (even though snake oil is big in sec now). The type of incompetence you describe doesn't flourish in these domains.

Re: Kubernetes Failure Stories

#234

Post by sgt101 »

NikolaeVarius wrote: post #219
sgt101 wrote: post #77

Earlier quoted context omitted.

I think that if there is a genuine circular dependency then the services won't start ever. But I think it is possible to introduce services that assume other services are up and have an apparent dependency circularity. The trick is to have all your services resilient to it's start requirements not being met - basically the service has to back off and wait if information it needs isn't yet in the environment... and th…

That's fine, until the environment doesn't come up because a hundred services all queried the source of truth simultaneously and caused load that prevented any service from getting its config

Which is easy to deal with by introducing a random factor into the backoff.

Re: Kubernetes Failure Stories

#235

Post by takeda »

merb wrote: post #185
aprdm wrote: post #147

Earlier quoted context omitted.

In places worked we usually had a vmware cluster, load balancer, NFS for shared data when necessary and DNS set up (e.g: through consul). This setup is very, very simple and scalable. There is very little to gain IMO on moving to Kubernetes. Consul, VSphere and load balancers have APIs and you can write tools to do everything that K8s does.

How do you load balance? i mean load balance the "public ip" In some networks DNS failover is really not that great, so at least a virtual ip needs to be used.

We use haproxy. I wrote code [1] that configures it based on Consul to do load balance. It has been running in production for 2 years without issue (tested with consul 1.0.6)

For people wondering why not use consul template, this has benefit of understanding haproxy, and it minimizes numbers of restarts to make changes.

Using haproxy over and has the benefit that if Winkle or consul goes down things continue to work, just not updates.

[1] https://github.com/takeda/winkle

Re: Kubernetes Failure Stories

#236

Post by merb »

takeda wrote: post #235
merb wrote: post #185

Earlier quoted context omitted.

How do you load balance? i mean load balance the "public ip" In some networks DNS failover is really not that great, so at least a virtual ip needs to be used.

We use haproxy. I wrote code [1] that configures it based on Consul to do load balance. It has been running in production for 2 years without issue (tested with consul 1.0.6) For people wondering why not use consul template, this has benefit of understanding haproxy, and it minimizes numbers of restarts to make changes. Using haproxy over and has the benefit that if Winkle or consul goes down things continue to work,…

Haproxy only solves a single Part of the Problem. If you do DNS based failover you should really Check how clients behave, when one node goes down. Without a floating ip or a cloud lb, some stuff will be troublesome

Re: Kubernetes Failure Stories

#237

Post by wires »

m0zg wrote: post #51

It's not for everyone and it has significant maintenance overhead if you want to keep it up to date _and_ can't re-create the cluster with a new version every time. This is something most people at Google are completely insulated from in the case of Borg, because SRE's make infrastructure "just work". I wish there was something drastically simpler. I don't need three dozen persistent volume providers, or the ability…

> I wish there was something drastically simpler

Have you tried Nomad?

https://www.nomadproject.io

Re: Kubernetes Failure Stories

#239

Post by takeda »

merb wrote: post #236
takeda wrote: post #235

Earlier quoted context omitted.

We use haproxy. I wrote code [1] that configures it based on Consul to do load balance. It has been running in production for 2 years without issue (tested with consul 1.0.6) For people wondering why not use consul template, this has benefit of understanding haproxy, and it minimizes numbers of restarts to make changes. Using haproxy over and has the benefit that if Winkle or consul goes down things continue to work,…

Haproxy only solves a single Part of the Problem. If you do DNS based failover you should really Check how clients behave, when one node goes down. Without a floating ip or a cloud lb, some stuff will be troublesome

The haproxy method doesn't rely on DNS at all so I'm a bit confused.

Re: Kubernetes Failure Stories

#240

Post by merb »

takeda wrote: post #239
merb wrote: post #236

Earlier quoted context omitted.

Haproxy only solves a single Part of the Problem. If you do DNS based failover you should really Check how clients behave, when one node goes down. Without a floating ip or a cloud lb, some stuff will be troublesome

The haproxy method doesn't rely on DNS at all so I'm a bit confused.

well either it uses DNS for failover or you have ipvs (lvs, keepalived) enabled or worse if the machine with haproxy crashes your basically dead. Of course there is also bgp and anycast, but this is not "cheap"
Post reply on HN