Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your Kubernetes nightmare?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 263 posts

Re: Ask HN: What is your Kubernetes nightmare?

#131

Post by garciasn »

JamesSwift wrote: post #126
garciasn wrote: post #76

Earlier quoted context omitted.

We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE infrastructure costs less than $50 a month. 4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases. 5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we di…

> Our entire GKE infrastructure costs less than $50 a month. Uhhh what? I mean even my personal DO based cluster runs about $40 a month. I'm skeptical a production cluster is at $50.

    Service                   Compute Engine Kubernetes Engine
    Cost.                     $191.78        $62.34 
    Discounts                ($13.96)       ($62.34)
    Promotions and others     $0.00          $0.00 
    Subtotal                  $177.83        $0.00
The GCE instance cost includes some of our 24/7 VMs which are the lion's share of that line item, not the micro VMs we use for the cluster.

Re: Ask HN: What is your Kubernetes nightmare?

#132

Post by fireflash38 »

tpxl wrote: post #83
fireflash38 wrote: post #68

Earlier quoted context omitted.

My experience with bare metal is multi-fold: * Documentation sees it as a second-class citizen, if that (loadbalancers, volumes are heavily biased towards cloud providers) * Many cloud-provided instances of kubernetes will always use the exact same VMs backing the nodes. So they really don't have to care all that much about what config your bare metal cluster has or needs. RancherOS/K3S can be really quite nice for g…

> loadbalancers, volumes are heavily biased towards cloud providers Can you even run a "loadbalancer" if all you have is a single machine with a single IP behind a router you don't control? I got stuck on that the last time I tried running my own kubes.

K3s uses their own loadbalancer, so yes. It will add extra hops to any of your services if you care about sub-ms latency.

I looked at metalLB, and didn't really fit with what we wanted to do, so YMMV. It's pretty limited unless you control a lot about your IP space.

Re: Ask HN: What is your Kubernetes nightmare?

#133

Post by Mo3 »

ekidd wrote: post #54

It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…

> 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possible and hope really hard it continues to work.

This one.

Re: Ask HN: What is your Kubernetes nightmare?

#135

Post by lazide »

ghusto wrote: post #19

TLDR: Complexity. The deprecation lifecycle, and running ingress controllers in an automatic scaling group. The first isn't as much of an issue if you have a (partially) dedicated team for managing your clusters, but can be prohibitively expensive (effort / time-wise) for smaller organisations. The second highlights a bigger problem in K8s in general. I'll have to give a little background first: If you run an Nginx i…

This is always a problem with ingress/load balancer unless you buy some REALLY fancy (auto failover) enterprise hardware loadbalancers unfortunately, regardless of K8S. It does make it harder to see what is going on though or work around it.

Re: Ask HN: What is your Kubernetes nightmare?

#136

Post by ekidd »

vbezhenar wrote: post #119
ekidd wrote: post #81

Earlier quoted context omitted.

I personally liked O'Reilly's Kubernetes: Up and Running , which was fairly thorough, and Nigel Poulton's books, which were shorter and focused on the highlights (at least the editions I read). The reason I always recommend that people read a book before getting into Kubernetes is that there are several things that make a lot more sense once someone takes the time to explain them. It actually gave me some 90s nostalg…

Is website documentation not good enough? It looks very thorough. Actually I’d say that it’s rare to encounter so verbose and full documentation nowadays. May be it’s even too deep, but I enjoyed reading it.

The website has tons of reference documentation!

But what a lot of people need is someone to just explain:

1. The basic idea of setting a desired configuration, and having the cluster try to bring reality into sync with the config.

2. How pods, replica sets, deployments and services fit together, and why Google thought it was a good idea to split them up that way. Also, how ingress fits in with all this.

3. Basic volume management.

4. Other common optional topics, just to get an overview.

The big advantage of a book is that it will try to cover the essential ideas, any why they work the way they do, without getting lost in describing a hundred advanced features you can look up later.

If there's an introductory section on the website that covers just the essentials, that might be enough! But I didn't find one when I was learning.

Re: Ask HN: What is your Kubernetes nightmare?

#138

Post by throwaway894345 »

iso1631 wrote: post #67
MPSimmons wrote: post #6

If you have a fully-functioning "best practices" Kubernetes environment, each of the following topics ends up with its own full-depth tech: - Compute - Deployment - CI - Networking - Storage - Policies Imagine running a microservices solution without an orchestration solution - how many people would it take to administer the servers, the storage, the network, the policies, etc. And with Kubernetes, you get maybe a co…

My team looks after 150 servers from Singapore to LA, about half physical, half virtual, some which just sit on shelves between jobs. Being pesamistic takes about 100 hours a year of feeding, upgrading, etc, about $6k a year.

That's comparing steady state with bootstrapping a Kubernetes environment. I suspect steady state Kubernetes is comparable.

Re: Ask HN: What is your Kubernetes nightmare?

#139

Post by rglover »

sleepybrett wrote: post #102
spicyusername wrote: post #34

- 60% of the Kubernetes ecosystem is half-baked alpha software - Maintaining 200+ clusters for 10 small applications - Cloud bills - Autoscaling never working well - Trying to untangle Terraform state without taking down Prod

Tell me you barely understand kubernetes without saying you barely understand kuberentes.

> 60% of the Kubernetes ecosystem is half-baked alpha software

This one is fair. Wasted a lot of time trying to find the "correct" dependencies—I remember the Nginx Ingress Controller specifically being a headache—only to find a maze of deprecations, poorly written documentation, or stuff that just flat out didn't work. That was ~18 months ago (I set up my cluster to run sites for my business and have basically left it alone) so things may have changed but at the time I remember being surprised after hearing so much hype.

Post reply on HN