Ask HN: What is your Kubernetes nightmare? • Hacker News Board

Re: Ask HN: What is your Kubernetes nightmare?

#71

Post by tpetry » Mon, Jun 27, 2022, 1:09 PM UTC

iasay wrote: post #46

The two things thing that gets me are: 1. Latch up states . It's very very easy for something to go wrong and blow a whole deployment up and lose all the pods for example a health check failure. Most application frameworks have some sort of request queuing and the health checks sit in the same queue so any upstream issues and you get health check failures and flapping. Of course the autoscaler goes fucking bonkers in…

> Don't get me wrong, I still like it but I wouldn't want to run it with little expertise at hand. It's not something I would trust someone to run without production experience, which is difficult because there are very few people out there who are battle hardened past trivial home deployments and tiny little stacks.

That‘s the problem. Everyone used something like minicube to bring a kubernetes cluster online and believes it is simple. But when anything does not work correctly the only approach is to kill the complete machine and start a new cluster on a server. Have fun with statefull data which needs to be copied…

Re: Ask HN: What is your Kubernetes nightmare?

#72

Post by Bayart » Mon, Jun 27, 2022, 1:10 PM UTC

JensRantil wrote: post #4

Honestly, creating my first K8s deployment of a service; typing out at least 150 lines of YAML to define my Deployment, figuring out how my ConfigMaps, Secrets, and Volumes, Services are defined and connected together. Vanilla K8s YAML is extremely low-level.

> Vanilla K8s YAML is extremely low-level.

I find trying to untangle something deployed with Kubespray/Helm/anything automatized far more headache inducing than flat YAML files.

Re: Ask HN: What is your Kubernetes nightmare?

#73

Post by victor106 » Mon, Jun 27, 2022, 1:11 PM UTC

On AWS, the best way we found to use K8S is using Margate and using Copilot to create and manage infrastructure and deployments.

Re: Ask HN: What is your Kubernetes nightmare?

#74

Post by AlexB138 » Mon, Jun 27, 2022, 1:11 PM UTC

jimbokun wrote: post #64

ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

What are some of the best Kubernetes books?

For intro level: Kubernetes Up and Running. (Here's a free version provided by VMware: https://www.vmware.com/content/dam/digitalmarketing/vmware/e...). Will teach you the basic vocabulary and get you well enough oriented to use k8s.

For trying to get to pro level: Programming Kubernetes. This one is focused on writing code for the k8s ecosystem, but it will teach you a lot of the internals.

Re: Ask HN: What is your Kubernetes nightmare?

#75

Post by beckingz » Mon, Jun 27, 2022, 1:12 PM UTC

Kubernetes gives you high availability, deployment automation, and powerful management tools, which are all needed to run software applications 'at scale'

Running software at scale is my nightmare.

Re: Ask HN: What is your Kubernetes nightmare?

#76

Post by garciasn » Mon, Jun 27, 2022, 1:12 PM UTC

spicyusername wrote: post #34

- 60% of the Kubernetes ecosystem is half-baked alpha software - Maintaining 200+ clusters for 10 small applications - Cloud bills - Autoscaling never working well - Trying to untangle Terraform state without taking down Prod

We use GKE.

1. I don't know about any of this; we don't seem to have problems.

2. This sounds like an architecture issue, not a k8s issue.

3. Our entire GKE infrastructure costs less than $50 a month.

4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases.

5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we did what we needed to do without downtime in production. While annoyingly esoteric, I thought it worked pretty fucking well compared to our alternatives.

Re: Ask HN: What is your Kubernetes nightmare?

#77

Post by AlexB138 » Mon, Jun 27, 2022, 1:13 PM UTC

ianwehba wrote: post #65

ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

can you recommend any good intro books?

Responded to a sibling comment with the same question here: https://news.ycombinator.com/item?id=31894095

Re: Ask HN: What is your Kubernetes nightmare?

#78

Post by tpetry » Mon, Jun 27, 2022, 1:14 PM UTC

KptMarchewa wrote: post #42

GordonS wrote: post #27

Earlier quoted context omitted.

Compare with Docker Compose files, which still use YAML, but a lot less of it, and less verbose too.

And does a lot less too, since it describes single node deployment, without any complexities behind networking and communication of services on different nodes.

A docker compose file is also used for swarm mich is a multi-server deployment.

Re: Ask HN: What is your Kubernetes nightmare?

#79

Post by jbsone » Mon, Jun 27, 2022, 1:14 PM UTC

softwareoiu wrote: post #59

ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

Why is bare metal a nightmare? I have a project coming up which must be on bare metal so was thinking of doing this. Also, if it's so bad, what's better to use on bare metal? Thanks

I did a project 5 years back that had to be bare metal, and going to for Kubernetes was probably the worst project decision I've made so far. We didn't have the required competency and wasted so much time on it, we should have gone for something more bland and simple.

My only tip if you really decide to go for it is to make sure to use a well-supported linux distro. We had to be on REHL and that turned out to be ill fitted.

Re: Ask HN: What is your Kubernetes nightmare?

#80

Post by linuxftw » Mon, Jun 27, 2022, 1:15 PM UTC

sureglymop wrote: post #48

7sidedmarble wrote: post #28

Earlier quoted context omitted.

Just making your pod accessible is not as complicated as it seems. All you need are: 1. A kubernetes service resource. This just contains a selector that points at your pod. 2. An ingress. You point this at the service you just made. You will get a static IP. Point your DNS at that. And you're good to go. This assumes that you are using a provider that comes with an 'ingress controller' out of the box (which is what…

So if I'm not using a cloud provider but just have a k8s cluster on a small vps with a public ip i just need the nginx ingress controller? What if i want https? Is there a way to automatically enable let's encrypt for different services and domains/subdomains?

There's a wealth of material online to explain these things. let's encrypt can integrate with a number of ingress controllers trivially. Like much of anything else, you need to actually experiment with it to understand how it all fits together.