Bit of a warning. If you do not set cpu requests, your pods may end up with cpu.shares=2.
Java, for example, makes some tuning decisions based on this that you're not gonna like.
Kubernetes: Make your services faster by removing CPU limits
41–50 of 111 posts
Re: Kubernetes: Make your services faster by removing CPU limits
#42Would the latency of the system be improved by reserving some amount of CPU for the container? For example if a container always got a few milliseconds per period, or if you even reserved a part or all of a CPU for the container.
Re: Kubernetes: Make your services faster by removing CPU limits
#43> https://engineering.indeedblog.com/blog/2019/12/unthrottled-... This is a more detailed post on the same thing - part two indicates changes have been back-ported to a number of kernel versions: Linux-stable: 4.14.154+, 4.19.84+, 5.3.9+ Ubuntu: 4.15.0-67+, 5.3.0-24+ Redhat Enterprise Linux: RHEL 7: 3.10.0-1062.8.1.el7+ RHEL 8: 4.18.0-147.2.1.el8_1+ CoreOS: v4.19.84+
Thanks for providing the fixed kernel versions here. I was able to check that our RHEL boxes are not affected. Take your bloody upvote!
Re: Kubernetes: Make your services faster by removing CPU limits
#44TL:DR; Remove limits if you’re running below 4.19 due to a bug present. Update AMI if running on EKS. I was expecting a discussion about CPU limits and all that is here is a workaround for a bug.
Where can I find an EKS worker node AMI with 4.19+? The EKS documentation shows latest is 4.14.x in the current 1.17 k8s.
You don't need to, it was backported https://news.ycombinator.com/item?id=24353080
Re: Kubernetes: Make your services faster by removing CPU limits
#45Here's a story that might make you not want to do that. We ran Kubernetes with the standard scheduler and node autoscaling for a long time, and used to allow developers in our (simplified) manifests define resource requests and limits. We saw that with our current config, we always had some unused capacity (that we wanted) since the scheduler spread out workloads while the autoscaler was only throwing nodes away with…
As an unsolicited recommendation, if you _wanted_ to attempt to chase this down, you should look at using the Vertical Pod Autoscaler (https://github.com/kubernetes/autoscaler/tree/master/vertica...) in recommendation only mode in order to get a good idea of what services are actually using, and set new limits based on that.
It definitely won't be optimal, but should let you get to a place where you at least have some limits set.
Re: Kubernetes: Make your services faster by removing CPU limits
#46Removing CPU limits seems like a bad idea now that there's a kernel fix. But putting that aside... I don't understand why pods without CPU limits would cause unresponsive kubelets. For a long time now Kubernetes has allocated a slice for system services. While pods without CPU limits are allowed to burst, they are still limited to the amount of CPU allocated to kubernetes pods. Run "systemd-cgls" on a node and you'll…
It might depend alot on the distribution and how kubernetes is started. How much CPU time to reserve for system services from the scheduler (Allocatable as you pointed out) needs to be passed to kubelet, and I think only really applies to guarenteed pods.
What I did on the distribution I work on, is tune the cgroup shares so control plane services are allocated CPU time ahead of pods (whether guarenteed, burstable, or best effort). We don't run anything as static containers, so this covers all the kube services, etcd, system services, etc.
Before this change in our distribution, IIRC, pods and control plane had equal waiting, which allowed the possibility for kubelet or other control plane services to be starved if the system was very busy.
There are also lots of other problems that can lead to kubelet bouncing between ready/not ready that we've observed which wouldn't be triggered by the limits.
Re: Kubernetes: Make your services faster by removing CPU limits
#47From a traditional cluster perspective, we've been doing this for years. depending on the goal of your service and cluster, it might be preferable to over subscribe your CPU. Compared to Memory oversubscription, CPU over sub isn't anywhere near as much of a show stopper, so long as your service degrades well when it can't get the CPU it needs. Where cost is an issue its very much worth oversubscribing your CPU by 20%…
As a mainframer once told me, "There's nothing wrong with having 100% CPU usage. We paid for it, we'd better use it".
On an interesting note, in mainframes it's normal to pay for a machine with n CPUs and get a n+m CPU machine delivered and installed. The extra CPUs are inactive until you pay for the upgrade and receive an activation code. In order to reduce downtime, during startup it's possible to have more than your licensed CPUs active to speed up the boot process and to catch up with any missed jobs.
Re: Kubernetes: Make your services faster by removing CPU limits
#48Bit of a warning. If you do not set cpu requests, your pods may end up with cpu.shares=2. Java, for example, makes some tuning decisions based on this that you're not gonna like.
The Go runtime also locks in some unwarranted assumptions at process start time, and never changes its parameters if the number of available CPUs changes.
Re: Kubernetes: Make your services faster by removing CPU limits
#49Last time I was knee deep in managing prod Kube infrastructure limits were also there for scheduling purposes. It's hard to properly allocate services across nodes when there is no concept of requirements they have. I guess you can get around that with setting a request versus a limit?
Re: Kubernetes: Make your services faster by removing CPU limits
#50Removing CPU limits seems like a bad idea now that there's a kernel fix. But putting that aside... I don't understand why pods without CPU limits would cause unresponsive kubelets. For a long time now Kubernetes has allocated a slice for system services. While pods without CPU limits are allowed to burst, they are still limited to the amount of CPU allocated to kubernetes pods. Run "systemd-cgls" on a node and you'll…
It's pretty simple, limits work only when everyone are using them. If you have one pod that does not enforce limits it can disrupt the entire node.