Live data from Hacker News

Ask HN: What is your Kubernetes nightmare?

news.ycombinator.com

91–100 of 263 posts

Re: Ask HN: What is your Kubernetes nightmare?

#91

Earlier quoted context omitted.

Because a vendor will do a lot of ground work (choosing a CNI and CSI implementation for instance) for you, and everything usually covered by a cloud-controller will be entirely up to you (e.g. LBs)

Actual bare metal, where you own the physical hardware and pay for the physical network connections is actually pretty painless. I see many people trying to use hosted compute to try and set up a bare metal cluster. This is a recipe for heisenbugs.

Having physical access (or IPMI) certainly helps, but there's also a lot more knowledge about networks bundled in companies that already run data centers, so setting up something like MetalLB (BGP load balancers) and Rook (Ceph CSI) to cover the parts that your cloud vendor would usually provide automatically is not as big of a deal. But the overall complexity for someone completely new to the topic is still higher.

Re: Ask HN: What is your Kubernetes nightmare?

#92
post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

What are some of the best Kubernetes books?

Kubernetes in Action by Manning (https://www.manning.com/books/kubernetes-in-action) is quite through, good and beginner friendly.

Re: Ask HN: What is your Kubernetes nightmare?

#93
post #45

You aren't supposed to use a tool made for Google level complexity unless you work with such complexity in the first place.

Where do I go for my 3-5 node cluster which I want to schedule containers to, run dynamic workloads etc?

How do I build a container based DAG in a small cluster today without k8s? Solutions that are not k8s tend to be single programming language/sdk based or not easier to set up.

I get the feeling the alternatives are dying off unnaturally fast.

Re: Ask HN: What is your Kubernetes nightmare?

#95
post #55
post #45

You aren't supposed to use a tool made for Google level complexity unless you work with such complexity in the first place.

I don't think that is true. Kubernetes brings a lot of advantages to people who have to manage infrastructure (like me). It gives me a single interface that I can apply across multiple teams. They all only need to provide me with a docker image and that's mostly it. Also what is the alternative? Self-written unmaintainable bash-scripts? That's what they had before. Every team had their own way of deployment, creating…

>Also what is the alternative?

Docker compose.

Or if we're getting wild pick a technology, use it across the company, and rely on language tools to establish the APIs between modules. Then deploy the application however the language provides.

Re: Ask HN: What is your Kubernetes nightmare?

#96
I'm not sure Kubernetes is worse than any other complex, distributed system.

OpenStack, Pivotal Cloud Foundry, internal compute platform

So far I think the nightmare problem is people trying to run it and CNCF software (Prometheus, various operators) with only a cursory understanding of how it works (me included)

It's easy to shoot yourself in the foot (oops, forgot requests on a resource intensive, high replica count deploy and hosed cluster autoscaling)

Re: Ask HN: What is your Kubernetes nightmare?

#97
post #89
post #83

Earlier quoted context omitted.

> loadbalancers, volumes are heavily biased towards cloud providers Can you even run a "loadbalancer" if all you have is a single machine with a single IP behind a router you don't control? I got stuck on that the last time I tried running my own kubes.

Why would you need a load-balancer if you only have a single machine?

Because kubernetes says so? Can you run it without a load balancer?

Re: Ask HN: What is your Kubernetes nightmare?

#99

Slow performance. So we have a few Spring Boot based webapps which were running (along with PgSQL) on a shared AWS t2.medium instance, we migrated these to a GKE cluster with a node pool of e2-standard-2 instances. The nodes are on a private network and don't have public IPs. The services are exposed via Load Balancer based Ingress (with SSL). Even after allocating one core to PgSQL and 2GB RAM, the API calls from th…

Are you setting cpu requests in your pod spec? This influences the cgroup cpu.shares for the containers and (unless this has been fixed) leaving cpu requests unset results in cpu.shares=2, which jvm interprets badly.

Re: Ask HN: What is your Kubernetes nightmare?

#100

- 60% of the Kubernetes ecosystem is half-baked alpha software - Maintaining 200+ clusters for 10 small applications - Cloud bills - Autoscaling never working well - Trying to untangle Terraform state without taking down Prod

We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE infrastructure costs less than $50 a month. 4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases. 5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we di…

With regards to 3, isn’t the management plane alone $72 for GKE without considering the cost for the nodes? How are your costs so low?
Post reply on HN