Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes: Make your services faster by removing CPU limits

erickhun.com

Post reply on HN
Print view View original

11–20 of 111 posts

Re: Kubernetes: Make your services faster by removing CPU limits

#11

Post by YawningAngel »

Schwan wrote: post #10

Is this really right? "The danger of not setting a CPU limit is that containers running in the node could exhaust all CPU available." My assumptions have been: 1. cpu request tells you how much cpu a pod gets MINIMUM always, independently of how much other pods use it or not 2. on GKE you can't request 100% cpu due to google reserving cpu for the node 3. if you have hard limits, your cluster utilisation will be bad -…

1 is correct, 2 is partially correct, and 3 is not correct.

It is absolutely true Kubernetes will reserve the amount of CPU you request, although it will also allow you to exceed that request if you attempt to and there is free CPU time to service you. 2 is correct in so far as Google run daemonsets on GKE which themselves have CPU requests and limits, and thus there will never be a node which as 100% cpu free for you to request. 3 is simply incorrect - it may be true that for some combinations of nodes and workloads it is not possible for the Kubernetes scheduler to bin-pack efficiently, but for large clusters with diverse workloads this should not be a problem.

Re: Kubernetes: Make your services faster by removing CPU limits

#12

Post by sheeshkebab »

davewritescode wrote: post #4

This seems like a bad trade-off, at least for 99% of us who haven’t been using Kubernetes in production for the last 5 years and manage it ourselves. Putting all the “user facing” services in a state where one of them consuming all the CPU could affect all the others feels like a disaster waiting to happen.

The number of times I’ve seen CPU limits kill off pods during even mild spikes and causing pretty much downtime and “disaster” is just as surprising. Work on autoscaling nodes instead, don’t use cpu limits.

Re: Kubernetes: Make your services faster by removing CPU limits

#13

Post by MetalMatze »

That's why we put the CPUThrottlingHigh alert into the kubernetes-mixin project. It a least let folks know. The Node Exporter for example is always throttled and I don't mind. For the user facing parts I'd rather not be in the same situation. Ultimately latency should tell me though.

Re: Kubernetes: Make your services faster by removing CPU limits

#14

Post by nebster »

sheeshkebab wrote: post #12
davewritescode wrote: post #4

This seems like a bad trade-off, at least for 99% of us who haven’t been using Kubernetes in production for the last 5 years and manage it ourselves. Putting all the “user facing” services in a state where one of them consuming all the CPU could affect all the others feels like a disaster waiting to happen.

The number of times I’ve seen CPU limits kill off pods during even mild spikes and causing pretty much downtime and “disaster” is just as surprising. Work on autoscaling nodes instead, don’t use cpu limits.

Maybe I'm misunderstanding you, but I'm pretty sure CPU limits will only limit the amount of CPU used even if there is more available. It will not kill off the pod.

Memory limits however will kill the pod if the pod uses more than the limit.

Re: Kubernetes: Make your services faster by removing CPU limits

#15

Post by burgerquizz »

YawningAngel wrote: post #3

I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.

Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.

Re: Kubernetes: Make your services faster by removing CPU limits

#16

Post by YawningAngel »

burgerquizz wrote: post #15
YawningAngel wrote: post #3

I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.

Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.

I'm not in a good position to say for sure, as I've only used managed Kubernetes distributions, but I think it probably works out to less work than removing CPU limits. Kernel upgrades are at least semi-routine, so most shops that run Kubernetes themselves are going to have a process for them. Conversely, removing CPU limits and migrating the critical path to a different set of tainted nodes is a substantial one-off change with a long tail of failure scenarios that need to be tested. Thus, I would expect that a kernel upgrade would be easier than doing what Buffer did.

Re: Kubernetes: Make your services faster by removing CPU limits

#17

Post by op00to »

burgerquizz wrote: post #15
YawningAngel wrote: post #3

I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.

Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.

It should be. Kernel updates are a critical path for bug fixes and security patches. If you can’t upgrade your kernel using a clear, premeditated plan, you are not doing your job right.

Re: Kubernetes: Make your services faster by removing CPU limits

#18

Post by davewritescode »

sheeshkebab wrote: post #12
davewritescode wrote: post #4

This seems like a bad trade-off, at least for 99% of us who haven’t been using Kubernetes in production for the last 5 years and manage it ourselves. Putting all the “user facing” services in a state where one of them consuming all the CPU could affect all the others feels like a disaster waiting to happen.

The number of times I’ve seen CPU limits kill off pods during even mild spikes and causing pretty much downtime and “disaster” is just as surprising. Work on autoscaling nodes instead, don’t use cpu limits.

This isn't exclusive to containers, you had this problem with VMs and even hardware. Back in the physical server days, when you didn't have enough CPU to service the current rate of inbound requests, eventually your healthchecks start to fail and load balancers would evict you from the pool. Now instead of a physical CPU, it sounds like you're running into a CPU limit implemented in the kernel but the same basic principles apply.

Kubernetes changes how you create the infrastructure but you still have the same problems you had in a distributed system before.

Re: Kubernetes: Make your services faster by removing CPU limits

#19

Post by op00to »

davewritescode wrote: post #4

This seems like a bad trade-off, at least for 99% of us who haven’t been using Kubernetes in production for the last 5 years and manage it ourselves. Putting all the “user facing” services in a state where one of them consuming all the CPU could affect all the others feels like a disaster waiting to happen.

Of course it’s a horrible idea. I support kubernetes clusters, and I would fix the underlying issue not paste it over.

Re: Kubernetes: Make your services faster by removing CPU limits

#20

Post by op00to »

klohto wrote: post #5

TL:DR; Remove limits if you’re running below 4.19 due to a bug present. Update AMI if running on EKS. I was expecting a discussion about CPU limits and all that is here is a workaround for a bug.

Exactly. And not a great long term workaround either!
Post reply on HN