Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your Kubernetes nightmare?

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 263 posts

Re: Ask HN: What is your Kubernetes nightmare?

#171

Post by dewey »

wg0 wrote: post #162
ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

Good rules. >2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. If one is at that level already, I don't think there's anything better than AWS ECS out there. It just works. Just works. Yes sure, it does not offer stateless workloads for example among other things but it works for 90% of the cases. > 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possible and hope r…

> I don't think there's anything better than AWS ECS out there

Do you have any experience with Kubernetes on GCP being less good than AWS ECS? I'd expect them to be the gold-standard when it's a project coming from Google originally and we haven't had any Kubernetes problems that were related to GCP.

Re: Ask HN: What is your Kubernetes nightmare?

#172

Post by cncfNormalLady »

My Kubernetes nightmare is not the software but the strident, intolerant, social justice keyboard warriors involved with the CNCF and k8s technologies.

They make up a tiny minority but they are loud and often nasty within their Twitter echo chamber and they are excellent at getting companies and people to do their bidding out of fear of reprisal.

It’s supposed to be the “most welcoming community” but it only takes you stepping out of line on the outrage du jour to get a proverbial face full of spittle and chased out of town with pitchforks.

I’m posting this with a throwaway obviously because I’m not trying to lose my job or get doxxed. Which would 100% happen if I posted under my real name.

Re: Ask HN: What is your Kubernetes nightmare?

#173

Post by dewey »

Intermernet wrote: post #66
ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

Kubernetes on bare metal is actually pretty easy. Kubernetes on a hosted solution which doesn't have a managed version is prone to error. Usually on bare metal you can make some guarantees regarding bandwidth and storage speed. Trying to roll out a cluster on a service that can't give you these guarantees is truly a nightmare.

> Kubernetes on bare metal is actually pretty easy.

I would not call it easy at all. Last time I tried that a year ago you still needed a special load balancer to get it going (https://metallb.universe.tf). Has this changed?

Re: Ask HN: What is your Kubernetes nightmare?

#175

Post by pojzon »

weitzj wrote: post #160

A failed upgrade of a CNI plugin on a production cluster. Since then I always have a blue / green cluster deployment at hand with a leach cluster containing the whole production environment and flipping via a Loadbalancer in front of the clusters

For stateless applications - doing a blue/green cluster upgrade is really the smartest choice.

Re: Ask HN: What is your Kubernetes nightmare?

#176

Post by Thiez »

pojzon wrote: post #167
ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

> 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. This is interesting - last time I worked with Microsoft Engineers from Azure - they said exactly the opposite. One workload = One cluster. „There are too many shared resources in Kubernetes that can leak collateral damage from one workload to another”.

What is their definition of a workload? Do they want a cluster per microservice? Per application? Per customer?

Re: Ask HN: What is your Kubernetes nightmare?

#177

Post by gautamdivgi »

pojzon wrote: post #167
ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

> 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. This is interesting - last time I worked with Microsoft Engineers from Azure - they said exactly the opposite. One workload = One cluster. „There are too many shared resources in Kubernetes that can leak collateral damage from one workload to another”.

Not really. You can do fairly large clusters, you need differently sized node pools. For example - we run apache NiFi is AKS which is a complete memory and cpu hog. We have a node pool 16cpu/64g ram for that workload which we specify a node selector. Microservices we use a different node pool. System services run on the default node pool.

If you're running Azure functions with KEDA - setup a nodepool for that with a lower cpu/memory footprint.

Re: Ask HN: What is your Kubernetes nightmare?

#178

Post by captrb »

birdiesanders wrote: post #69
high_5 wrote: post #29

I'm afraid k8s would become like git - it's a great tool out of which we mostly use like 5% of it's complete capabilities, yet we all use it because everyone is using it. Yet, k8s doesn't really make all the underlying stacks go away. When the shit hits the fan, you have to troubleshoot it with knowledge of much more than just YAML syntax.

k8s is just Linux, if you forget the Linux there is no hope.

I admit to being a little frustrated with systems engineers telling me I should never need shell access to a production system again, that web-based Metrics and Tracing should be enough to debug all problems. I have twenty years of muscle memory using strace, dtrace, lsof, blah blah blah to troubleshoot complex problems. Furthermore I'm only brought in when the problem is sufficiently complex. I understand that it should be a break-glass exception, but I don't want linux abstracted away completely.

Re: Ask HN: What is your Kubernetes nightmare?

#179

Post by itintheory »

treis wrote: post #95
k8sToGo wrote: post #55

Earlier quoted context omitted.

I don't think that is true. Kubernetes brings a lot of advantages to people who have to manage infrastructure (like me). It gives me a single interface that I can apply across multiple teams. They all only need to provide me with a docker image and that's mostly it. Also what is the alternative? Self-written unmaintainable bash-scripts? That's what they had before. Every team had their own way of deployment, creating…

>Also what is the alternative? Docker compose. Or if we're getting wild pick a technology, use it across the company, and rely on language tools to establish the APIs between modules. Then deploy the application however the language provides.

>Docker compose. Swarm is a closer alternative, although it's future is unclear.

Another alternative is Nomad.

Re: Ask HN: What is your Kubernetes nightmare?

#180

Post by rglover »

throwaway894345 wrote: post #144
rglover wrote: post #139

Earlier quoted context omitted.

> 60% of the Kubernetes ecosystem is half-baked alpha software This one is fair. Wasted a lot of time trying to find the "correct" dependencies—I remember the Nginx Ingress Controller specifically being a headache—only to find a maze of deprecations, poorly written documentation, or stuff that just flat out didn't work. That was ~18 months ago (I set up my cluster to run sites for my business and have basically left…

Pretty sure nginx ingress controller is one of the more solid and widely-used pieces. I've had a lot more trouble with cert-manager, but it seems to be in a stable state on my cluster now and anyway similar solutions in the bare-VM world are just as painful (IIRC I gave up trying to get terraform to do the handshake for AWS ACM).

What I can say definitively is that having gone from not doing any infra work to using k8s and then over the past few months trying my hand at a bare-metal setup, just spinning up a Linux box and hand installing deps via apt or snap was far more enjoyable/easy to follow.

Primarily because there was very little obscurity (i.e., config files that automate away a lot of thinking or Dockerfiles/containers doing the same). It also left me feeling more confident about stability because if something isn't working, it's pretty clear what I broke/forgot. Worst "bug" I ran into was a snap server hanging when installing a dependency.

Post reply on HN