Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your Kubernetes nightmare?

news.ycombinator.com

Post reply on HN
Print view View original

211–220 of 263 posts

Re: Ask HN: What is your Kubernetes nightmare?

#211

Post by horsawlarway »

Mine is fairly boring. Overall - the tool does the job, and I much prefer it to hand managing servers, or some of the previous VM based management solutions.

My two biggest gripes:

- Loss of visibility, especially related to inspecting network data as it moves from LB to pod.

- Half baked tooling around the eco-system, although this does seem to be slowly improving

My two biggest likes:

- I genuinely save a bunch of time with it at this point (it still occasionally sucker punches me)

- I can take the experience from my day job and self-host quite a large number of useful applications at home on old hardware.

Re: Ask HN: What is your Kubernetes nightmare?

#212

Post by pojzon »

Thiez wrote: post #176
pojzon wrote: post #167

Earlier quoted context omitted.

> 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. This is interesting - last time I worked with Microsoft Engineers from Azure - they said exactly the opposite. One workload = One cluster. „There are too many shared resources in Kubernetes that can leak collateral damage from one workload to another”.

What is their definition of a workload? Do they want a cluster per microservice? Per application? Per customer?

We had a different set of microservices doing specific part of the system.

One part was responsible for data transformation and the other was responsible for user modifications.

Its hard to tell whats the cutting point it depends on the system architecture.

Re: Ask HN: What is your Kubernetes nightmare?

#213

Post by kule »

sleepybrett wrote: post #146
iasay wrote: post #46

The two things thing that gets me are: 1. Latch up states . It's very very easy for something to go wrong and blow a whole deployment up and lose all the pods for example a health check failure. Most application frameworks have some sort of request queuing and the health checks sit in the same queue so any upstream issues and you get health check failures and flapping. Of course the autoscaler goes fucking bonkers in…

1. You don't 'lose' the pods. If your pod fails on liveness or readiness checks it restarts, over and over until it passes. 2. Depends on your team for sure. If you are on a team that's like "We'll just spin this up and it will be fine." you ignore the fact that 'things happen'. I've seen similar situations with companies that deploy on bare linux servers as well, some update breaks something or something isn't optim…

RE 1. Just to be precise (as I thought this until recently):

* liveness check will cause pod restarts.

* Readiness check causes the pod to be removed from the round-robin of new traffic requests so it has time to recover/finish processing what it's working on.

Re: Ask HN: What is your Kubernetes nightmare?

#215

Post by jsmith45 »

icedchai wrote: post #208
dewey wrote: post #203

Earlier quoted context omitted.

That's just not true, especially if you compare it to the LoadBalancer you get on a cloud platform which usually involves zero clicks. I'm not saying it's impossible but it's definitely not "easy". Configuration instructions: https://metallb.universe.tf/configuration/ Hint: You better know what all of these are in your environment: For a basic configuration featuring one BGP router and one IP address range, you need…

Did you miss the part about layer 2 configuration, where you don't need BGP at all? https://metallb.universe.tf/configuration/#layer-2-configura...

But then "When announcing in layer2 mode, one node in your cluster will attract traffic for the service IP."

This bottlenecking seems undesirable. At the very least, if you have one "main" traffic heavy service whichever node ends up servicing that IP address could have elevated cpu usage from processing all the network traffic via kube-proxy.

The obvious solution would be to allocate say 2 or more so ip addresses for the service with dns round robin set up. Then as long as all three are being handled by different nodes you are not bottlenecking nearly as badly. But perhaps I am missing it, but I'm not seeing a feature where you can force those two or more ip addresses to be claimed by different nodes. (If the feature is strict, then you would want more data plane nodes than IPs, so that having one node down will result in having part of the Round robin DNS unclaimed by any node).

Re: Ask HN: What is your Kubernetes nightmare?

#217

Post by icedchai »

jsmith45 wrote: post #215
icedchai wrote: post #208

Earlier quoted context omitted.

Did you miss the part about layer 2 configuration, where you don't need BGP at all? https://metallb.universe.tf/configuration/#layer-2-configura...

But then "When announcing in layer2 mode, one node in your cluster will attract traffic for the service IP." This bottlenecking seems undesirable. At the very least, if you have one "main" traffic heavy service whichever node ends up servicing that IP address could have elevated cpu usage from processing all the network traffic via kube-proxy. The obvious solution would be to allocate say 2 or more so ip addresses fo…

True. If you want true load balancing, you need a layer 3 solution (BGP.)

Re: Ask HN: What is your Kubernetes nightmare?

#218

Post by agarren »

ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

Regarding item 1, any recommendations for a good book? Manning has a couple titles that look good, but I’m curious to hear what others would suggest.

Re: Ask HN: What is your Kubernetes nightmare?

#219

Post by sleepybrett »

kule wrote: post #213
sleepybrett wrote: post #146

Earlier quoted context omitted.

1. You don't 'lose' the pods. If your pod fails on liveness or readiness checks it restarts, over and over until it passes. 2. Depends on your team for sure. If you are on a team that's like "We'll just spin this up and it will be fine." you ignore the fact that 'things happen'. I've seen similar situations with companies that deploy on bare linux servers as well, some update breaks something or something isn't optim…

RE 1. Just to be precise (as I thought this until recently): * liveness check will cause pod restarts. * Readiness check causes the pod to be removed from the round-robin of new traffic requests so it has time to recover/finish processing what it's working on.

You are correct, readiness removes it from any service objects it would be an endpoint to.

My point was more 'the pod doesn't go away'. I've seen some people do stuff with the HPA that could cause it to scale down to minimum replicas if its in a broken state, depending on what stats you are using to scale, but that's more of a 'kubernetes doing what you told it to do' problem.

Re: Ask HN: What is your Kubernetes nightmare?

#220

Post by hitpointdrew »

SoftTalker wrote: post #118
Intermernet wrote: post #66

Earlier quoted context omitted.

Kubernetes on bare metal is actually pretty easy. Kubernetes on a hosted solution which doesn't have a managed version is prone to error. Usually on bare metal you can make some guarantees regarding bandwidth and storage speed. Trying to roll out a cluster on a service that can't give you these guarantees is truly a nightmare.

I have tried several times over the past few years to install Kubernetes on bare metal, and it has never worked. I don't mean installing it on VMs on a laptop, I mean on a real linux cluster of 8 to 32 nodes, with real networks and real switches. Managing bare metal machines is a cakewalk compared to getting Kubernetes running in-house, at least in my experience. Obviously the cloud providers do it, so it's possible.…

Did you try using kubeadm to bootstrap installing kubernetes? It is pretty simple.
Post reply on HN