Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your Kubernetes nightmare?

news.ycombinator.com

Post reply on HN
Print view View original

251–260 of 263 posts

Re: Ask HN: What is your Kubernetes nightmare?

#251

Post by otabdeveloper4 »

garciasn wrote: post #76
spicyusername wrote: post #34

- 60% of the Kubernetes ecosystem is half-baked alpha software - Maintaining 200+ clusters for 10 small applications - Cloud bills - Autoscaling never working well - Trying to untangle Terraform state without taking down Prod

We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE infrastructure costs less than $50 a month. 4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases. 5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we di…

> 3. Our entire GKE infrastructure costs less than $50 a month.

At this scale, you don't need Kubernetes, invest in a pocket calculator instead.

Re: Ask HN: What is your Kubernetes nightmare?

#252

Post by e12e »

OJFord wrote: post #58
bovermyer wrote: post #9

Earlier quoted context omitted.

I'm a platform engineer and I still think that Kubernetes and its tooling are unnecessarily complex. As an aside, when I think "low-level" with regards to computer programming, I think machine byte code - closer to the hardware - so this statement read a little funny to me.

Yeah, it may be complex and highly configurable, but it's hard to imagine getting much higher level than defining networking, storage, secrets, etc. through YAML.

I feel it's a lot like the Java enterprise world of FactoryFactoryFactory-classes - a col league coined the term "horizontal abstraction" for this type of trend; you never build a pyramid/hierarchy that composes truly higher level/abstract reasoning - you just complect in various constructs - that all keep hold of most of the complexity from the level "below".

So you get to write 30-40 lines of yaml for each of your ten slightly different services...

Re: Ask HN: What is your Kubernetes nightmare?

#253

Post by e12e »

secondcoming wrote: post #32
JensRantil wrote: post #4

Honestly, creating my first K8s deployment of a service; typing out at least 150 lines of YAML to define my Deployment, figuring out how my ConfigMaps, Secrets, and Volumes, Services are defined and connected together. Vanilla K8s YAML is extremely low-level.

I'm buried up to my balls in this right now. My favourite part is nested 'spec' objects where I've no clear view into what each spec is.

I haven't really used it in anger, but I sort of think kpt might be helpful in managing k8s: https://kpt.dev/

Re: Ask HN: What is your Kubernetes nightmare?

#254

Post by withinboredom »

xiwenc wrote: post #108
tpxl wrote: post #97

Earlier quoted context omitted.

Because kubernetes says so? Can you run it without a load balancer?

You can use Nodeport instead of loadbalancer. Or use metallb if you insist to have LB so that it’s more closer to real production environments.

“Real production” smh. This is why docs make baremetal second class citizens, people assume something has to be a certain way for it to be “real.”

Re: Ask HN: What is your Kubernetes nightmare?

#255

Post by JensRantil »

Bayart wrote: post #72
JensRantil wrote: post #4

Honestly, creating my first K8s deployment of a service; typing out at least 150 lines of YAML to define my Deployment, figuring out how my ConfigMaps, Secrets, and Volumes, Services are defined and connected together. Vanilla K8s YAML is extremely low-level.

> Vanilla K8s YAML is extremely low-level. I find trying to untangle something deployed with Kubespray/Helm/anything automatized far more headache inducing than flat YAML files.

Yeah, been there and it makes things one notch even worse. :)

Re: Ask HN: What is your Kubernetes nightmare?

#256

Post by JensRantil »

p_l wrote: post #30
JensRantil wrote: post #4

Honestly, creating my first K8s deployment of a service; typing out at least 150 lines of YAML to define my Deployment, figuring out how my ConfigMaps, Secrets, and Volumes, Services are defined and connected together. Vanilla K8s YAML is extremely low-level.

The most freeing moment with k8s for me was integrating that no, you don't need to write all that YAML (or JSON), and going into generating manifests from other formats - in my case, usually Jsonnet with mix of libraries both public and private which quickly embedded knowledge specific to our setup. I think one of the best things I ever heard from another colleague was "I took the example jsonnet and had working vers…

Agree! That said, it's important to remember that someone needs to maintain those manifests, too. Copy-paste might work initially, but you will need to modify them at some point. Which means you need in-house knowledge of that K8s YAML, anyway. For a smaller shop this might be non-trivial. For a larger shop you likely have something close to a Platform team that can maintain the manifests to make them easy to use.

Re: Ask HN: What is your Kubernetes nightmare?

#257

Post by JensRantil »

While I'm venting K8s frustration; They told me K8s was awesome because it allows me to easy test things locally on my machine. They told me minikube would solve it. Minikube ran out of memory, was unstable, crashed. Production YAML required certmanager not on my machine. Production YAML required volume manager not on my machine.

Re: Ask HN: What is your Kubernetes nightmare?

#258

Post by anothernewdude »

bovermyer wrote: post #11
anothernewdude wrote: post #5

I hate that kubectl wants all the images to be already built. Instead I'm forced to keep docker-compose yaml around to actually build the damn things first. Which introduces more yaml that kubectl will insist on reading. The documentation at the main kubernetes site is poor, and is being deprecated, but not in favour of anything new.

Interesting. You build images at deployment time? Why?

Quick iteration. More like I deploy at build time. CI/CD takes minutes to deploy to a shared environment. Rather than contending for a bottleneck, we do local development on a single node K8S environment.

Maintaining docker-compose as well is a pain, and it's repeating ourselves.

Re: Ask HN: What is your Kubernetes nightmare?

#259

Post by bajsejohannes »

boloust wrote: post #145
bajsejohannes wrote: post #20

The decoupling of ingress and deployments always bothered me, although it might not be a _nightmare_ exactly. In short, the ingress may route traffic to a pod after it is killed. The solution is that when a pod gets a SIGTERM signal, it should mark itself not ready, wait for some amount of time and then shut down (see e.g. https://deepsource.io/blog/zero-downtime-deployment/ ). I've heard arguments for this behavior,…

In most cases you can fix this with the service upstream annotation, which has existed since 2017: https://github.com/kubernetes/ingress-nginx/issues/257

I did not know that. Thanks for explaining.

Re: Ask HN: What is your Kubernetes nightmare?

#260

Post by Joker_vD »

nomorecodehere8 wrote: post #247
Joker_vD wrote: post #127

Earlier quoted context omitted.

I believe there is a rather relatable book called "The Dream-Quest of Unknown Kubernetes" or something like this.

Why do ppl obfuscate like this? k8s is a control plane + api and client and workers. Don't make it weird. If you do then control the weirdness: the hooks are all there. To me it still looks like client/server with some development overhead based on gRPC + REST, state + eventual consistency. When I was learning linux in 1997 I had butterflies in my stomach - it felt amazing working with tech that I could do _anything_…

Because even the built-in agents are huge and complex. Nevermind the homegrown agents some of which has config files that basically are YAML-serialized ASTs for some weird Turing-complete imperative language... and some just straight up embed a Lua script that does configuration and actual work. But thanks for your advice "don't make it weird": I won't! Too bad all the other people didn't heed to it.
Post reply on HN