Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your Kubernetes nightmare?

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 263 posts

Re: Ask HN: What is your Kubernetes nightmare?

#181

Post by Thiez »

anderiv wrote: post #165
wg0 wrote: post #162

Earlier quoted context omitted.

Good rules. >2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. If one is at that level already, I don't think there's anything better than AWS ECS out there. It just works. Just works. Yes sure, it does not offer stateless workloads for example among other things but it works for 90% of the cases. > 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possible and hope r…

> If one is at that level already, I don't think there's anything better than AWS ECS out there. 100%. To answer the OP's question: my nightmare is having to use it at all. I work with small, very early-stage companies whose applications by and large are not complicated. Perhaps at some level of scale and/or complexity, k8s makes sense. For the vast majority of the cases I see, something like ECS does everything they…

Do you still recommend they host their applications in containers (e.g. Docker)? I feel like it's fairly low effort to start out that way, but can be a pain to add later.

Re: Ask HN: What is your Kubernetes nightmare?

#182

Post by danpalmer »

dewey wrote: post #171
wg0 wrote: post #162

Earlier quoted context omitted.

Good rules. >2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. If one is at that level already, I don't think there's anything better than AWS ECS out there. It just works. Just works. Yes sure, it does not offer stateless workloads for example among other things but it works for 90% of the cases. > 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possible and hope r…

> I don't think there's anything better than AWS ECS out there Do you have any experience with Kubernetes on GCP being less good than AWS ECS? I'd expect them to be the gold-standard when it's a project coming from Google originally and we haven't had any Kubernetes problems that were related to GCP.

I have experience with Google's managed Kubernetes service (GKE).

It's basically great. Solid, few surprises, no compatibility issues with third party software packaged for Kubernetes. Autopilot looks even better – billing you only for the resources you allocate rather than for the full nodes, basically removing the bin-packing problem. Very little about our config was Google specific or would cause issues porting to another provider. It was up-to-date enough for us to use relatively new features, while lagging enough that everything felt pretty stable.

The only issue we had was wanting to use a somewhat obscure configuration for the Google Cloud load balancer instance that was underlying the Kubernetes ingress. This was possible, we just had to configure it manually in Terraform and point it at the cluster rather than being able to treat it as a cluster resource if I remember correctly. This was only a temporary solution while they were in the process of adding more custom control via K8s.

As far as I can tell it is considered to be the gold-standard.

Disclaimer: I now work for Google on non-cloud stuff, but this was my experience doing a port from bare-metal to GKE.

Re: Ask HN: What is your Kubernetes nightmare?

#183

Post by anderiv »

Thiez wrote: post #181
anderiv wrote: post #165

Earlier quoted context omitted.

> If one is at that level already, I don't think there's anything better than AWS ECS out there. 100%. To answer the OP's question: my nightmare is having to use it at all. I work with small, very early-stage companies whose applications by and large are not complicated. Perhaps at some level of scale and/or complexity, k8s makes sense. For the vast majority of the cases I see, something like ECS does everything they…

Do you still recommend they host their applications in containers (e.g. Docker)? I feel like it's fairly low effort to start out that way, but can be a pain to add later.

Being that they're all using ECS, yes containerizing using Docker is a prerequisite.

Re: Ask HN: What is your Kubernetes nightmare?

#184

Post by bradwood »

ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

I've done exactly that. It wasn't fun. RHEL, k3s, ansible, Longhorn, metallb, to name a few.

Storage is the fun part.

Re: Ask HN: What is your Kubernetes nightmare?

#185

Post by ekidd »

pojzon wrote: post #167
ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

> 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. This is interesting - last time I worked with Microsoft Engineers from Azure - they said exactly the opposite. One workload = One cluster. „There are too many shared resources in Kubernetes that can leak collateral damage from one workload to another”.

Azure might also require special precautions. Honestly, I've seen Azure have a lot of networking issues, for example. But this is based on scuttlebutt and limited personal experience.

I've found that on GCS, certain workloads benefit from a dedicated node pool. This gets them their own CPU and RAM and volume I/O. Yes, I could imagine that there are shared Kubernetes control plane resources that might be affected, but I haven't seen that with any of our workloads. It might get more complicated if you have lots of in-cluster networking.

But none of this is my area of expertise. I just think that Kubernetes can mostly be pretty pleasant in practice for companies that have outgrown PaaS offerings like Heroku and Render.com.

Re: Ask HN: What is your Kubernetes nightmare?

#186

Post by lxe »

It's the lack of good first-class observability and good administration tooling into what's going on. There's a bunch of third party tools, and those seem to be lacking. The cli is abysmal tbh. Good luck figuring out why your pod crashed, or why it's frozen, or where the logs are from non-existing pods are, etc...

Re: Ask HN: What is your Kubernetes nightmare?

#187

Post by tflinton »

Overall kubernetes is far better than anything else i've used to manage deployments and production workloads. That said, what gives me the hee-bee-gee-bee's (and what has caused outages, for me at least) is:

1. Managing etcd nodes -- Reconciliation is a patient waiting game, try and rush it and you'll loose your cluster.

2. Kubernetes Networking -- This is nearly impossible to trace packets coming through an LB into a kubernetes pods without very deep understanding of different networking layers and CNIs. A lot can go wrong here.

3. Running persistent volumes in kubernetes. This can range from outright unstable and dangerous to annoying and at the very best intermittently loosing access to services due to volume claims being detached/reattached. Would highly avoid this.

4. Running "sticky" services. Statefulset's can allow you to run enumerated services with stick sessions but my experience with any sticky service is it tends to be somewhat volatile as kubernetes really loves to move workloads at its convenience. I've found statefulsets to be a redflag when considering putting it in kubernetes.

Re: Ask HN: What is your Kubernetes nightmare?

#189

Post by MuffinFlavored »

bradwood wrote: post #184
ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

I've done exactly that. It wasn't fun. RHEL, k3s, ansible, Longhorn, metallb, to name a few. Storage is the fun part.

I was playing around with a local Elasticsearch cluster and I couldn't figure out how to "do" k8s storage. Some kind of like... shared NFS volume or something maybe?

Could you spread some tips from your experiences?

Re: Ask HN: What is your Kubernetes nightmare?

#190

Post by Thiez »

anderiv wrote: post #183
Thiez wrote: post #181

Earlier quoted context omitted.

Do you still recommend they host their applications in containers (e.g. Docker)? I feel like it's fairly low effort to start out that way, but can be a pain to add later.

Being that they're all using ECS, yes containerizing using Docker is a prerequisite.

Doh. I am only familiar with Azure, and was confusing ECS with ordinary VMs. Sorry about the stupid question!
Post reply on HN