Ask HN: What is your Kubernetes nightmare? • Hacker News Board

Re: Ask HN: What is your Kubernetes nightmare?

#141

Post by buzer » Mon, Jun 27, 2022, 3:42 PM UTC

7sidedmarble wrote: post #28

sureglymop wrote: post #24

For me it is actually getting services exposed. Say i buy a domain and then i (trivially) containerize my applications and set up services in kubernetes. Now comes the networking part which is just a pain. How do I make my service accessible? It's easy with docker and an nginx reverse proxy but with kubernetes it's always seemed to be a real pain.

Just making your pod accessible is not as complicated as it seems. All you need are: 1. A kubernetes service resource. This just contains a selector that points at your pod. 2. An ingress. You point this at the service you just made. You will get a static IP. Point your DNS at that. And you're good to go. This assumes that you are using a provider that comes with an 'ingress controller' out of the box (which is what…

It's not quite that simple if you are not with provider that provides LoadBalancer service integration with Kubernetes. Normally the input to Kubernetes cluster is essentially NodePort. That's normally more or less random high port (like 31453) that is exposed on all nodes/nodes that run the service that matches the selector. Unless you want your visitors to add that to the URL (and keep DNS up to date with active nodes), using it to provide end-user accessible HTTP/HTTPS services is not very viable.

You either need to find/create integration to provider's load balancer (or possibly CDN that allows non-1:1 port mapping) or use HostPort service. Latter has it's own share problems as well.

Re: Ask HN: What is your Kubernetes nightmare?

#142

Post by throwaway894345 » Mon, Jun 27, 2022, 3:43 PM UTC

garciasn wrote: post #76

spicyusername wrote: post #34

- 60% of the Kubernetes ecosystem is half-baked alpha software - Maintaining 200+ clusters for 10 small applications - Cloud bills - Autoscaling never working well - Trying to untangle Terraform state without taking down Prod

We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE infrastructure costs less than $50 a month. 4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases. 5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we di…

Agreed. We've had only one autoscaling issue and it was on Google's end (datacenter ran out of nodes of a particular type and thus failed to scale up). Our GKE infrastructure costs a lot more, but we do a lot of heavy compute.

Re: Ask HN: What is your Kubernetes nightmare?

#143

Post by _8j50 » Mon, Jun 27, 2022, 3:47 PM UTC

ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

And don't expose workloads to the internet unless it is a prod app.

Can you recommend me a good intro book to read cover to cover (hopefully not too thick).

Re: Ask HN: What is your Kubernetes nightmare?

#144

Post by throwaway894345 » Mon, Jun 27, 2022, 3:47 PM UTC

rglover wrote: post #139

sleepybrett wrote: post #102

Earlier quoted context omitted.

Tell me you barely understand kubernetes without saying you barely understand kuberentes.

> 60% of the Kubernetes ecosystem is half-baked alpha software This one is fair. Wasted a lot of time trying to find the "correct" dependencies—I remember the Nginx Ingress Controller specifically being a headache—only to find a maze of deprecations, poorly written documentation, or stuff that just flat out didn't work. That was ~18 months ago (I set up my cluster to run sites for my business and have basically left…

Pretty sure nginx ingress controller is one of the more solid and widely-used pieces. I've had a lot more trouble with cert-manager, but it seems to be in a stable state on my cluster now and anyway similar solutions in the bare-VM world are just as painful (IIRC I gave up trying to get terraform to do the handshake for AWS ACM).

Re: Ask HN: What is your Kubernetes nightmare?

#145

Post by boloust » Mon, Jun 27, 2022, 3:50 PM UTC

bajsejohannes wrote: post #20

The decoupling of ingress and deployments always bothered me, although it might not be a _nightmare_ exactly. In short, the ingress may route traffic to a pod after it is killed. The solution is that when a pod gets a SIGTERM signal, it should mark itself not ready, wait for some amount of time and then shut down (see e.g. https://deepsource.io/blog/zero-downtime-deployment/ ). I've heard arguments for this behavior,…

In most cases you can fix this with the service upstream annotation, which has existed since 2017: https://github.com/kubernetes/ingress-nginx/issues/257

Re: Ask HN: What is your Kubernetes nightmare?

#146

Post by sleepybrett » Mon, Jun 27, 2022, 3:50 PM UTC

iasay wrote: post #46

The two things thing that gets me are: 1. Latch up states . It's very very easy for something to go wrong and blow a whole deployment up and lose all the pods for example a health check failure. Most application frameworks have some sort of request queuing and the health checks sit in the same queue so any upstream issues and you get health check failures and flapping. Of course the autoscaler goes fucking bonkers in…

1. You don't 'lose' the pods. If your pod fails on liveness or readiness checks it restarts, over and over until it passes.

2. Depends on your team for sure. If you are on a team that's like "We'll just spin this up and it will be fine." you ignore the fact that 'things happen'. I've seen similar situations with companies that deploy on bare linux servers as well, some update breaks something or something isn't optimally configured. Things go wrong with systems, people who know the systems are needed to fix them. It sounds like if you intend on using kubernetes you should learn to troubleshoot it.

It's not difficult to find people who know how to actually run kubernetes, it's just hard to convince them to switch jobs for you. I get a decent numbers of cold, non recruiter, linkedin contacts a month, and frankly many recruiters... but my current job pays well and matches my worklife balance. Zero people who are reaching out from startups offer the whole package, they can usually come in fine on the money, but I'm not working a hundred hours a week with little to no support. On the 'established corporate enterprise' side of the house, they can be inflexible when it comes to vacation time, salary ranges etc. but I've found a good place.

Re: Ask HN: What is your Kubernetes nightmare?

#147

Post by throwaway894345 » Mon, Jun 27, 2022, 3:51 PM UTC

tpetry wrote: post #71

iasay wrote: post #46

The two things thing that gets me are: 1. Latch up states . It's very very easy for something to go wrong and blow a whole deployment up and lose all the pods for example a health check failure. Most application frameworks have some sort of request queuing and the health checks sit in the same queue so any upstream issues and you get health check failures and flapping. Of course the autoscaler goes fucking bonkers in…

> Don't get me wrong, I still like it but I wouldn't want to run it with little expertise at hand. It's not something I would trust someone to run without production experience, which is difficult because there are very few people out there who are battle hardened past trivial home deployments and tiny little stacks. That‘s the problem. Everyone used something like minicube to bring a kubernetes cluster online and be…

It's not like you can let someone run bare VMs without any production experience either. You need a PaaS for something like that, even then...

> Everyone used something like minicube to bring a kubernetes cluster online and believes it is simple.

This hasn't been my experience, and certainly the Kubernetes project doesn't advertise the software as "so simple anyone can do it" or any such thing. Kubernetes definitely requires experience--it's not a PaaS, but a framework on which something like a PaaS could be built.

Re: Ask HN: What is your Kubernetes nightmare?

#148

Post by waylandsmithers » Mon, Jun 27, 2022, 3:52 PM UTC

A client who was on version 1.13 (which was the current version when they started the project in 2018) being forced to upgrade by the aws managed kubernetes service. I'm not sure what they ended up doing but they were facing the requirement of having to upgrade the nodes and the control plane one version at a time.

Re: Ask HN: What is your Kubernetes nightmare?

#149

Post by freedomben » Mon, Jun 27, 2022, 3:55 PM UTC

francoispon wrote: post #137

we have been using openshift at work and it has been relatively troublesome, to be honest when we have infrastrcuture (which is not often) we just have some redhat consultant fixing it.

Disclaimer: Former Red Hat Consultant who fixed people's openshift :-)

Can you share more about your openshift stack? For example do you have a storage solution (and is it openshift container storage)? Are you running on top of vmware, aws, etc?

You're far from the only one to have that problem. I love OpenShift and generally do recommend it, so I've been trying to think of ways to improve on that situation. I don't work for Red Hat anymore, but the product is wonderful and (despite not being perfect) is IMHO the best one out there, so I want to see it succeed. It always felt to me like it was just enough of a black box as to be hard for an outsider to get into and tinker/debug.

Re: Ask HN: What is your Kubernetes nightmare?

#150

Post by kevin_nisbet » Mon, Jun 27, 2022, 3:55 PM UTC

I think my top ones are endpoint security software or running on redhat OSes. I used to work on a kubernetes distribution, so a large amount of the support escalations and workload was around shipping kubernetes to customers who weren't as apt as the HN crowd.

Endpoint Security Software, just because it's adding some policy, that usually isn't written by the team trying to run the application, and will apply the policy sometimes in non-obvious ways. Even when you think it's turned off, sometimes it isn't, and the vendor will leave kernel modules running and partial configurations.

RedHat was more a result of the stability policy for kernel, and often running much older kernels then other distributions. We had lots of problems with the more modern kernel features used by kubernetes, that we had to track down and often link to known fixes. We had one customer even replacing their kernels so they wouldn't have as many issues. This may be less and less of the case all the time with newer RedHAt Releases, and I also have no reason to believe OpenShift suffers in the same way... just that I've spent a large amount of time troubleshooting this.