Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your Kubernetes nightmare?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 263 posts

Ask HN: What is your Kubernetes nightmare?

#1

Post by wg0 »

Everything self-hosted has its maintenance tax but why Kubernetes (especially self hosted) is so hard? What aspect is that makes Kubernetes operationally so hard?

- Is it the networking model that is simple from the consumption standpoint but has too many moving parts for it to be implemented?

- Is it the storage model, CSI and friends?

- Is it the bunch of controller loops doing their own things with nothing that gives a "wholesome" picture to identify the root cause?

For me personally, first and foremost thing on my mind is the networking details. They are "automatically generated" by each CNI solution in slightly different ways and constructs (iptables, virtual bridges, routing daemons, eBPF etc etc) and because they are generated, it is not uncommon to find hundreds of iptable rules and chains on a single node and/or similar configuration.

Being automated, these solutions generate tons of components/configurations which in case of trouble, even if one has mastery on them, would take some time to hoop through all the components (virtual interfaces, virtual bridges, iptable chains and rules, ipvs entries etc) to identify what's causing the trouble. Essentially, one pretty much has to be a network engineer because besides the underlying/physical (or the virtual, I mean cloud VPCs) network, k8s pulls its very own network (pod network, cluster network) implemented on the software/configuration layer which has to be fully understood to be able to maintained.

God forbid, if the CNI solution has some edge case or for some other misconfiguration, it keeps generating inadequate or misconfigured rules/routes etc resulting in a broken "software defined network" that I cannot identify in time on a production system is my nightmare and I don't know how to reduce that risk.

What's your Kubernetes nightmare?

EDIT: formating

Re: Ask HN: What is your Kubernetes nightmare?

#3

Post by lbriner »

Ironically, I think one of the biggest issues is around packaging, specifically Helm charts (but if there are others, it is probably the same). In many frameworks, packaging is to help people by hiding complexity. Need an ingress? Use a Helm chart!

But then upgrading can be very risky because if you have any problem at all, unless you understand the helm chart you can rarely simply downgrade/uninstall, you could have caused a fatal problem and for a cluster, the resilience is meaningless if you make a change that blocks access to all service.

Other issues relate to dependencies and breaking changes which might be subtle and which might not be easy to discover like the fact some old resource uses a v1beta type which becomes deprecated.

I think once it is working, Kubernetes is very reliable for me but it is when making infrastructure changes that things can go south very quickly. Updating deployments etc. is fine.

Re: Ask HN: What is your Kubernetes nightmare?

#5

Post by anothernewdude »

I hate that kubectl wants all the images to be already built. Instead I'm forced to keep docker-compose yaml around to actually build the damn things first. Which introduces more yaml that kubectl will insist on reading.

The documentation at the main kubernetes site is poor, and is being deprecated, but not in favour of anything new.

Re: Ask HN: What is your Kubernetes nightmare?

#6

Post by MPSimmons »

If you have a fully-functioning "best practices" Kubernetes environment, each of the following topics ends up with its own full-depth tech:

- Compute

- Deployment

- CI

- Networking

- Storage

- Policies

Imagine running a microservices solution without an orchestration solution - how many people would it take to administer the servers, the storage, the network, the policies, etc. And with Kubernetes, you get maybe a couple of teams if you're lucky. This is the power and the leverage of the platform.

But also, imagine in that environment, how many things can go wrong, and the amount of expertise that you need to properly debug them. You still need that amount of expertise, because all of that complexity is still in place (or at least most of it is) - if your physical disks are throwing errors, you need someone who knows how to debug and replace that. Not hard. But then you have Ceph above that, and Rook above that (or whatever storage solution you use). And then you've got the deployment that has to make the PVC successfully. And it's like that for everything. Every problem has the potential to be a full stack problem for any one of half a dozen stacks.

It's a lot.

Re: Ask HN: What is your Kubernetes nightmare?

#7

Post by hedgedoops2 »

ciguy wrote: post #2

What are you fishing for exactly here? K8s has it's issues, but the way you have phrased the question is only going to get you biased answers.

I guess OP is looking for real life stories, of 'nightmares', caused by excessively complex or opaque kubernetes configurations.

AKA "Tell us about how your boss/team has shot themselves in the foot by using kubernetes [needlessly or wrong]".

Re: Ask HN: What is your Kubernetes nightmare?

#8

Post by speedgoose »

JensRantil wrote: post #4

Honestly, creating my first K8s deployment of a service; typing out at least 150 lines of YAML to define my Deployment, figuring out how my ConfigMaps, Secrets, and Volumes, Services are defined and connected together. Vanilla K8s YAML is extremely low-level.

Yes it’s a bit much. When I was beginning with kubernetes I was writing Docker compose files first and then converting them to kubernetes using https://kompose.io/

Re: Ask HN: What is your Kubernetes nightmare?

#9

Post by bovermyer »

JensRantil wrote: post #4

Honestly, creating my first K8s deployment of a service; typing out at least 150 lines of YAML to define my Deployment, figuring out how my ConfigMaps, Secrets, and Volumes, Services are defined and connected together. Vanilla K8s YAML is extremely low-level.

I'm a platform engineer and I still think that Kubernetes and its tooling are unnecessarily complex.

As an aside, when I think "low-level" with regards to computer programming, I think machine byte code - closer to the hardware - so this statement read a little funny to me.

Post reply on HN