Kubernetes: Make your services faster by removing CPU limits
1–10 of 111 posts
Re: Kubernetes: Make your services faster by removing CPU limits
#2Couldn't the unresponsive kubectl issue be resolved by isolating cpus and controlling yourself where processes go?
Re: Kubernetes: Make your services faster by removing CPU limits
#3I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.
Re: Kubernetes: Make your services faster by removing CPU limits
#4This seems like a bad trade-off, at least for 99% of us who haven’t been using Kubernetes in production for the last 5 years and manage it ourselves.
Putting all the “user facing” services in a state where one of them consuming all the CPU could affect all the others feels like a disaster waiting to happen.
Re: Kubernetes: Make your services faster by removing CPU limits
#5TL:DR; Remove limits if you’re running below 4.19 due to a bug present. Update AMI if running on EKS.
I was expecting a discussion about CPU limits and all that is here is a workaround for a bug.
Re: Kubernetes: Make your services faster by removing CPU limits
#6This seems like a bad trade-off, at least for 99% of us who haven’t been using Kubernetes in production for the last 5 years and manage it ourselves. Putting all the “user facing” services in a state where one of them consuming all the CPU could affect all the others feels like a disaster waiting to happen.
Been there, seen that, can confirm.
Re: Kubernetes: Make your services faster by removing CPU limits
#7From a traditional cluster perspective, we've been doing this for years.
depending on the goal of your service and cluster, it might be preferable to over subscribe your CPU.
Compared to Memory oversubscription, CPU over sub isn't anywhere near as much of a show stopper, so long as your service degrades well when it can't get the CPU it needs.
Where cost is an issue its very much worth oversubscribing your CPU by 20% to ensure you are rinsing the CPU.
Re: Kubernetes: Make your services faster by removing CPU limits
#8Thy should have just upgraded the kernel to a fixed one, which definitely does not require to upgrade the whole distribution.
Also if they are using Kubernetes normally there is no reason to not upgrade the whole distribution as well, since only Kubernetes will be running on it, and of course that's widely tested (the containers each choose their own distribution, only the kernel is shared).
Re: Kubernetes: Make your services faster by removing CPU limits
#9[deleted]
Re: Kubernetes: Make your services faster by removing CPU limits
#10Is this really right?
"The danger of not setting a CPU limit is that containers running in the node could exhaust all CPU available."
My assumptions have been: 1. cpu request tells you how much cpu a pod gets MINIMUM always, independently of how much other pods use it or not 2. on GKE you can't request 100% cpu due to google reserving cpu for the node 3. if you have hard limits, your cluster utilisation will be bad -> we do remove cpu limits due to this.