Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your Kubernetes nightmare?

news.ycombinator.com

Post reply on HN
Print view View original

241–250 of 263 posts

Re: Ask HN: What is your Kubernetes nightmare?

#241

Post by h1fra »

Love kubernetes. And at the same time it's not the magical thing we all expect.

But my main pain points are around Kubernetes and all the hidden stuff.

Kubernetes alone is not enough, you need terraform or helm (or both) to have something manageable and deployable by a team. When things errors or do not behave the way you expected it all become so complicated or cryptic, that you sometimes better delete an entire resource than understand the underlying issue.

Some stuff like dependency between resources (e.g: Deployments depending on ConfigMap, updating the ConfigMap won't restart the deployment) makes things a lot more complicated than you expect.

There are too many vendor specific stuff that are necessary to make a Kube cluster works that you can not expect to have one terraform setup that is multi cloud. etc...

Re: Ask HN: What is your Kubernetes nightmare?

#242

Post by drbojingle »

guywhocodes wrote: post #228
drbojingle wrote: post #107

Earlier quoted context omitted.

What exactly are you making that requires the use of multiple languages?

My problem needing multiple machines is less weird than needing more than python for my dag?

No, I honestly just don't know what you're talking about lol. What exactly are you building?

Re: Ask HN: What is your Kubernetes nightmare?

#243

Post by guywhocodes »

drbojingle wrote: post #242
guywhocodes wrote: post #228

Earlier quoted context omitted.

My problem needing multiple machines is less weird than needing more than python for my dag?

No, I honestly just don't know what you're talking about lol. What exactly are you building?

I'm building several things, some of them need puppeteer with plugins so have to use a lot of node.js there. Which I have no interest in otherwise. Some of it is built in higher performance languages. Some of it is off the shelf solutions

And most of all by having every node in the dag a container I don't have versioning issues.

Yes everything can be wrapped in python, doesn't make it a good idea however.

Re: Ask HN: What is your Kubernetes nightmare?

#244

Post by hitpointdrew »

icedchai wrote: post #232
hitpointdrew wrote: post #222

Earlier quoted context omitted.

MetalLB has been in beta for YEARS. It's OK for dev/qa/staging, but I wouldn't put in prod.

I wouldn't use it in prod when there are other alternatives from cloud providers. But to say it is difficult to configure for a bare metal dev cluster is not true. The instructions are pretty clear.

I don't disagree, I think it is easy to install on a bare metal cluster, although I think using HA Proxy is just as easy and probably a better solution. I was just pointing out that it has been in beta for a very long time.

Re: Ask HN: What is your Kubernetes nightmare?

#245

Post by nix23 »

moooo99 wrote: post #207
nix23 wrote: post #111

Earlier quoted context omitted.

>My personal Kubernetes nightmare is having to build a cluster from scratch on bare metal. One's heaven is another one's nightmare, i like building it from scratch because then i know every single knob, and doing a excellent job in documentation makes sure that others have that knowledge too. But hey since "administrators" is a forgotten art, you are probably better of just buying some black-boxes with terrible perfo…

Depending on your individual circumstances buying "some black-boxes with terrible performance" might be a worthwhile tradeoff

Well yes that's true. It's always about the circumstances.

Re: Ask HN: What is your Kubernetes nightmare?

#246

Post by nomorecodehere8 »

ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

You hit every high point of my own experience but there are caveats. 1. If you have to do on prem then virtualize and package till you can wash, rinse repeat. 2. Secure systems with k8s are a thing: Stigged k8s, stigged host systems, mtls, psp, network policy, MAC integration - this makes k8s really unpleasant to deal with if you come from pub cloud, pub k8s provider. See #1. 3. Performance: dns sucks and it sucks for all kinds of reasons: usually avoidable with node local caching approaches, but sometimes not. 4. Yes: Big clusters...until you need federation.

Re: Ask HN: What is your Kubernetes nightmare?

#247

Post by nomorecodehere8 »

Joker_vD wrote: post #127
mgarciaisaia wrote: post #115

2025. 3am. Clear night. Full moon shining, many stars around. I suddenly wake up, covered in cold sweat. My heart is pumping so hard. I take out my phone. I search the internet. Kubernetes still reigns, no simpler approach made it. The end.

I believe there is a rather relatable book called "The Dream-Quest of Unknown Kubernetes" or something like this.

Why do ppl obfuscate like this? k8s is a control plane + api and client and workers. Don't make it weird. If you do then control the weirdness: the hooks are all there. To me it still looks like client/server with some development overhead based on gRPC + REST, state + eventual consistency. When I was learning linux in 1997 I had butterflies in my stomach - it felt amazing working with tech that I could do _anything_ with. k8s is the only thing in 20+ years that has ever given me that feeling again.

Re: Ask HN: What is your Kubernetes nightmare?

#248

Post by nomorecodehere8 »

k8sToGo wrote: post #55
oxff wrote: post #45

You aren't supposed to use a tool made for Google level complexity unless you work with such complexity in the first place.

I don't think that is true. Kubernetes brings a lot of advantages to people who have to manage infrastructure (like me). It gives me a single interface that I can apply across multiple teams. They all only need to provide me with a docker image and that's mostly it. Also what is the alternative? Self-written unmaintainable bash-scripts? That's what they had before. Every team had their own way of deployment, creating…

The first thing with bash scripts is you need to use a type system. Enforce json as your configuration language. Then have error handling as part of a default shell function library that emits json. All you are doing then is passing bad returns to a function that can emit json and reviewing json configurations for shell script variable definitions. Don't do in shell scripts what you can do in terraform. Don't use ansible/salt/puppet for what can be done in shell scripts. Shell scripts and makefiles go to CI/CD agencies (Jenkins). Ansible, terraform, etc...go to services that specialize in these.

Re: Ask HN: What is your Kubernetes nightmare?

#249

Post by otabdeveloper4 »

iasay wrote: post #46

The two things thing that gets me are: 1. Latch up states . It's very very easy for something to go wrong and blow a whole deployment up and lose all the pods for example a health check failure. Most application frameworks have some sort of request queuing and the health checks sit in the same queue so any upstream issues and you get health check failures and flapping. Of course the autoscaler goes fucking bonkers in…

> Don't get me wrong, I still like it

Stockholm syndrome?

Re: Ask HN: What is your Kubernetes nightmare?

#250

Post by nomorecodehere8 »

h1fra wrote: post #241

Love kubernetes. And at the same time it's not the magical thing we all expect. But my main pain points are around Kubernetes and all the hidden stuff. Kubernetes alone is not enough, you need terraform or helm (or both) to have something manageable and deployable by a team. When things errors or do not behave the way you expected it all become so complicated or cryptic, that you sometimes better delete an entire res…

Don't use TF for k8s. That is the one lesson I have learned. Use jenkins + shell scripts for k8s. Drive vendor tools (eksctl) or use kube* tools.
Post reply on HN