Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your Kubernetes nightmare?

news.ycombinator.com

Post reply on HN
Print view View original

221–230 of 263 posts

Re: Ask HN: What is your Kubernetes nightmare?

#221

Post by hitpointdrew »

dewey wrote: post #173
Intermernet wrote: post #66

Earlier quoted context omitted.

Kubernetes on bare metal is actually pretty easy. Kubernetes on a hosted solution which doesn't have a managed version is prone to error. Usually on bare metal you can make some guarantees regarding bandwidth and storage speed. Trying to roll out a cluster on a service that can't give you these guarantees is truly a nightmare.

> Kubernetes on bare metal is actually pretty easy. I would not call it easy at all. Last time I tried that a year ago you still needed a special load balancer to get it going ( https://metallb.universe.tf ). Has this changed?

HA proxy isn't complicated to setup.

Re: Ask HN: What is your Kubernetes nightmare?

#222

Post by hitpointdrew »

icedchai wrote: post #197
dewey wrote: post #173

Earlier quoted context omitted.

> Kubernetes on bare metal is actually pretty easy. I would not call it easy at all. Last time I tried that a year ago you still needed a special load balancer to get it going ( https://metallb.universe.tf ). Has this changed?

MetalLB is pretty simply to configure.

MetalLB has been in beta for YEARS. It's OK for dev/qa/staging, but I wouldn't put in prod.

Re: Ask HN: What is your Kubernetes nightmare?

#223

Post by afeiszli »

vbezhenar wrote: post #159

Some very basic things look very hard to me. Right now I’m scratching my head how do I implement even non-HA WireGuard server in a pod, so wg clients can access a pod network and pods can access wg client network. Seems like a very basic requirement for any installation yet zero guides about it. And don’t even talk about HA server with load balancer.

You can do a WireGuard server on Kubernetes with Netmaker:

- https://itnext.io/multi-cluster-kubernetes-networking-with-n...

- https://itnext.io/multi-cluster-kubernetes-networking-with-n...

Re: Ask HN: What is your Kubernetes nightmare?

#224

Post by nijave »

sleepybrett wrote: post #151
iasay wrote: post #85

Earlier quoted context omitted.

We sort of waltzed around that one with PV's on EBS on Amazon and also shifted the control plane to them. But there are still some serious problems in that space.

A limitation of EBS on amazon that I've run into a few times is that EBS volumes must connect to EC2 in their zone only. So if your k8s cluster has nodes across multiple AZs (which obv. important) your pod that mounts that pv will always be zone locked. This can also be problematic if you write a pod that mounts claims that are in different zones, that will never work. There are also amazon limitations to how many vo…

Resource zone affinity is annoying but usually works fine if you have 1 ASG per zone (Azure and afaik GCP are the same way). You mainly need to be careful during initial creation to make sure the volumes are spread out

WaitForFirstConsumer is very helpful, too (you basically guarantee the pod can be scheduled at least once before creating the resources which greatly improves the likelihood it can get rescheduled in the future without getting stuck)

Don't remember seeing any issues on AWS but I've seen Azure CSI take about 7.5 minutes to unmount a volume from 1 and mount to another (so each pod in a statefulset can take around 8-10 minutes)

Re: Ask HN: What is your Kubernetes nightmare?

#225

Post by samspenc »

Thiez wrote: post #190
anderiv wrote: post #183

Earlier quoted context omitted.

Being that they're all using ECS, yes containerizing using Docker is a prerequisite.

Doh. I am only familiar with Azure, and was confusing ECS with ordinary VMs. Sorry about the stupid question!

No worries, the cloud acronyms overlap so much these days, if it's helpful:

EC2: Elastic Compute service, bare VMs.

ECS: Elastic Container service, Docker containers

EKS: Elastic Kubernetes service

Re: Ask HN: What is your Kubernetes nightmare?

#226

Post by nijave »

jhugo wrote: post #122

My #1 k8s nightmare is the widespread practice of just writing (or downloading and never even looking at!) YAML and applying it to the cluster, with no additional management layer (we use Terraform, but use whatever you want), meaning that eventually you have no idea what the intended state of the cluster is, only its actual state. Vendor READMEs encourage this (some even going so far as to suggest `kubectl apply -f…

>Multiple times I've found that the vendor's

...sounds about right (at least for closed source products)

Re: Ask HN: What is your Kubernetes nightmare?

#227

Post by nijave »

Normal_gaussian wrote: post #33

Storage: Shared FS between nodes, autoscaling volume claim sizes, autoscaling volume claim iops, and measuring storage utilisation (iops e.g.) for pod/node/pv. How have I solved it? I haven't and I know its a key part of cost-control for us in about 12 months. Fast deploy: I'm trying to get a test cluster up in less than half an hour. With the DAG for building it all I'm getting a failure rate of 30% if I don't leave…

Are you setting up managed k8s on a cloud provider and, if so, which one?

I've had some issues with getting EKS dependency ordering correct (using Terraform)

Re: Ask HN: What is your Kubernetes nightmare?

#228

Post by guywhocodes »

drbojingle wrote: post #107
guywhocodes wrote: post #93

Earlier quoted context omitted.

Where do I go for my 3-5 node cluster which I want to schedule containers to, run dynamic workloads etc? How do I build a container based DAG in a small cluster today without k8s? Solutions that are not k8s tend to be single programming language/sdk based or not easier to set up. I get the feeling the alternatives are dying off unnaturally fast.

What exactly are you making that requires the use of multiple languages?

My problem needing multiple machines is less weird than needing more than python for my dag?

Re: Ask HN: What is your Kubernetes nightmare?

#229

Post by Normal_gaussian »

nijave wrote: post #227
Normal_gaussian wrote: post #33

Storage: Shared FS between nodes, autoscaling volume claim sizes, autoscaling volume claim iops, and measuring storage utilisation (iops e.g.) for pod/node/pv. How have I solved it? I haven't and I know its a key part of cost-control for us in about 12 months. Fast deploy: I'm trying to get a test cluster up in less than half an hour. With the DAG for building it all I'm getting a failure rate of 30% if I don't leave…

Are you setting up managed k8s on a cloud provider and, if so, which one? I've had some issues with getting EKS dependency ordering correct (using Terraform)

Indeed - GKE here.

Re: Ask HN: What is your Kubernetes nightmare?

#230

Post by guywhocodes »

josephcsible wrote: post #117
guywhocodes wrote: post #93

Earlier quoted context omitted.

Where do I go for my 3-5 node cluster which I want to schedule containers to, run dynamic workloads etc? How do I build a container based DAG in a small cluster today without k8s? Solutions that are not k8s tend to be single programming language/sdk based or not easier to set up. I get the feeling the alternatives are dying off unnaturally fast.

Docker Swarm?

Ideally it would be a hub and spoke model. I don't know how much swarm hammers etcd but k8s in my cluster produces >1kIOps on my SSDs doing nothing at all. Just health checks and cascading effects
Post reply on HN