Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your Kubernetes nightmare?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 263 posts

Re: Ask HN: What is your Kubernetes nightmare?

#121

Post by notwedtm »

tpxl wrote: post #83
fireflash38 wrote: post #68

Earlier quoted context omitted.

My experience with bare metal is multi-fold: * Documentation sees it as a second-class citizen, if that (loadbalancers, volumes are heavily biased towards cloud providers) * Many cloud-provided instances of kubernetes will always use the exact same VMs backing the nodes. So they really don't have to care all that much about what config your bare metal cluster has or needs. RancherOS/K3S can be really quite nice for g…

> loadbalancers, volumes are heavily biased towards cloud providers Can you even run a "loadbalancer" if all you have is a single machine with a single IP behind a router you don't control? I got stuck on that the last time I tried running my own kubes.

not necessarily a router you don't control, but MetalLB does provide some nice LoadBalancer constructs for a bare-metal deployment. Putting Vyos infront of it is magical!

https://metallb.universe.tf/

Re: Ask HN: What is your Kubernetes nightmare?

#122

Post by jhugo »

My #1 k8s nightmare is the widespread practice of just writing (or downloading and never even looking at!) YAML and applying it to the cluster, with no additional management layer (we use Terraform, but use whatever you want), meaning that eventually you have no idea what the intended state of the cluster is, only its actual state. Vendor READMEs encourage this (some even going so far as to suggest `kubectl apply -f https://...`!).

My #2 (probably partially caused by #1) is the lack of attention paid to RBAC in vendor-supplied manifests. Multiple times I've found that the vendor's YAML binds some controller's service account to a ClusterRole giving access to all secrets in the cluster, when it only really needs to read one. After filing a GitHub issue it seems that I'm the first to even notice, even on popular projects that have been around for years.

Re: Ask HN: What is your Kubernetes nightmare?

#123

Post by [deleted] »

fireflash38 wrote: post #68
softwareoiu wrote: post #59

Earlier quoted context omitted.

Why is bare metal a nightmare? I have a project coming up which must be on bare metal so was thinking of doing this. Also, if it's so bad, what's better to use on bare metal? Thanks

My experience with bare metal is multi-fold: * Documentation sees it as a second-class citizen, if that (loadbalancers, volumes are heavily biased towards cloud providers) * Many cloud-provided instances of kubernetes will always use the exact same VMs backing the nodes. So they really don't have to care all that much about what config your bare metal cluster has or needs. RancherOS/K3S can be really quite nice for g…

[deleted]

Re: Ask HN: What is your Kubernetes nightmare?

#124

Post by discreteevent »

drbojingle wrote: post #107
guywhocodes wrote: post #93

Earlier quoted context omitted.

Where do I go for my 3-5 node cluster which I want to schedule containers to, run dynamic workloads etc? How do I build a container based DAG in a small cluster today without k8s? Solutions that are not k8s tend to be single programming language/sdk based or not easier to set up. I get the feeling the alternatives are dying off unnaturally fast.

What exactly are you making that requires the use of multiple languages?

It would seem that the requirement for multiple languages often has nothing to do with making something or the customer requirements. It's often a developer requirement. Developers want to work in their favorite language. In the 90s you just wrote in whatever the company or ecosystem had mandated. Now that developers are in high demand they can specify what they want to use.

It started with books like "Beyond java" (and java was beyond C++) and disparaging articles by the likes of Paul Graham. They made some good points but if you have ever worked on a single system with multiple languages and are honest about it you would have to say that it wastes an enormous amount of time and generates unnecessary complexity. When a developer has to make a microservice just to write part of the system in another language you have wonder if they have the ability to evaluate technical trade-offs. Just pick one language and get on with it.

Re: Ask HN: What is your Kubernetes nightmare?

#126

Post by JamesSwift »

garciasn wrote: post #76
spicyusername wrote: post #34

- 60% of the Kubernetes ecosystem is half-baked alpha software - Maintaining 200+ clusters for 10 small applications - Cloud bills - Autoscaling never working well - Trying to untangle Terraform state without taking down Prod

We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE infrastructure costs less than $50 a month. 4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases. 5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we di…

> Our entire GKE infrastructure costs less than $50 a month.

Uhhh what? I mean even my personal DO based cluster runs about $40 a month. I'm skeptical a production cluster is at $50.

Re: Ask HN: What is your Kubernetes nightmare?

#127

Post by Joker_vD »

mgarciaisaia wrote: post #115

2025. 3am. Clear night. Full moon shining, many stars around. I suddenly wake up, covered in cold sweat. My heart is pumping so hard. I take out my phone. I search the internet. Kubernetes still reigns, no simpler approach made it. The end.

I believe there is a rather relatable book called "The Dream-Quest of Unknown Kubernetes" or something like this.

Re: Ask HN: What is your Kubernetes nightmare?

#128

Post by garciasn »

ColdHeat wrote: post #100
garciasn wrote: post #76

Earlier quoted context omitted.

We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE infrastructure costs less than $50 a month. 4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases. 5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we di…

With regards to 3, isn’t the management plane alone $72 for GKE without considering the cost for the nodes? How are your costs so low?

As noted by another HNer, we are using one cluster and up to 10 nodes as the top end, with only one running 90% of the time and only up to 3 covering the other 9%. We set 10 to the upper limit to ensure we're not going to have borked runs in case some crazy random ML model isn't going to take everything out and force killed jobs. The vast majority of workloads running on the cluster are HIGHLY variable with most running on Monday morning/afternoon and Month or Quarter starts + 3 days.

We are running many hundreds of jobs on those peak days with only a handful running on any other day. While many bring up examples where 24/7 infrastructure from a single box is more than plenty, we find that we can run micro VMs in this configuration and not have to worry about resource contention as our jobs run.

Pre-GKE, we were managing the timing manually, which was fine until we started to scale, but we found this to be a far better situation. Particularly because we simply don't have to think about it.

YMMV.

Re: Ask HN: What is your Kubernetes nightmare?

#129

Post by mkeedlinger »

I think some of the comments shed light on an interesting dichotomy I've noticed while talking to folk about K8s:

It seems that if you stick to simple configs, a setup hosted for you, etc, basically the happy path then people have had really good experiences with k8s. Those people can't understand how one could be inept enough _not_ to figure it out.

On the other hand, you'll also hear a lot of complaints about the difficulty of self-managed clusters, and attempting certain less popular or more complicated configs (or what have you). These people can't understand what benefit introducing such an insane amount of complexity could bring.

The second has mostly been my experience. I've tried now maybe a handful of times to create a cluster and get it running something on my home lab. At first I could rarely get it "up", but now I can usually get it to the point where I'd want to include storage or whatnot, and that's where I've been failing lately. Either way, I've never gotten it stable enough to warrant actual usage from me.

I like the idea of k8s; it seems like the natural next step of computing abstractions. I'm just not sure if "it's it", or if it's stable/reliable/evolved enough for people who don't need it now to invest in it yet.

Re: Ask HN: What is your Kubernetes nightmare?

#130

Post by znpy »

ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

> My personal Kubernetes nightmare is having to build a cluster from scratch on bare metal.

We have a bare-metal k8s cluster... In my opinion the thing we got right is to use external load-balancers (good old haproxy) to point at nginx-ingress-controllers (whose pods are pinned to two "service" nodes) and to load-balance the apiserver traffic.

Most other traffic is inter-cluster, and managed by calico anyway.

Post reply on HN