If you remove the cpu limit you won't be able to use HPA though, right?
Kubernetes: Make your services faster by removing CPU limits
21–30 of 111 posts
Re: Kubernetes: Make your services faster by removing CPU limits
#22I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.
Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.
By and large, you can just upgrade the kernel and nothing will break. It's one of the biggest things kernel developers worry over, in fact.
The reason, I think, K8s people might do this is that they often do not have the access, experience, or skills to do platform upgrades. It's easier to just futz around on the layer they feel they understand well; the world of containers.
Re: Kubernetes: Make your services faster by removing CPU limits
#23I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.
Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.
You can test out your shit by creating a new node with the recent kernel version installed. Then schedule some apps for testing on that node and see if everything works. Swap it out when you're done testing.
Re: Kubernetes: Make your services faster by removing CPU limits
#24This seems like a bad trade-off, at least for 99% of us who haven’t been using Kubernetes in production for the last 5 years and manage it ourselves. Putting all the “user facing” services in a state where one of them consuming all the CPU could affect all the others feels like a disaster waiting to happen.
The number of times I’ve seen CPU limits kill off pods during even mild spikes and causing pretty much downtime and “disaster” is just as surprising. Work on autoscaling nodes instead, don’t use cpu limits.
This advice is confusing. CPU is a "compressible" resource -- pods don't get killed for (trying to) exceed it. Pods don't get evicted from nodes based on CPU starvation, so autoscaling your node count won't help if you end up with a set of pods on a node that need more CPU than the node can provide. They'll just starve each other.
If your service allows horizontal scalability, you can use autoscaling of pods with Horizontal Pod Autoscaler (ideally also with a cluster autoscaler) to increase pod count for a given service when some percentage of the requested CPU is exceeded, whether or not you set a CPU limit. Setting the cpu_request appropriately for your pods is critical to ensure that node CPU is not oversubscribed by the Kubernetes pod scheduler.
Pods where mem & CPU requests = limits are given the highest class of service ("guaranteed"). For your most critical and latency sensitive services, this is the best approach when also coupled with HPA. Assuming a 4.19 kernel or later, I suppose.
https://medium.com/better-programming/the-kubernetes-quality...
Re: Kubernetes: Make your services faster by removing CPU limits
#25If you remove the cpu limit you won't be able to use HPA though, right?
No, you still have a CPU request, and the HPA is based on utilization of that.
Also, even without limits I believe CPU is prioritized based on the request. So if 1 pod requests 100 millicpu and another pod requests 200 millicpu, if they both try to use all the CPU on a node the one that requested 200 millicpu will use 2/3 of the CPU and the other will use 1/3.
Re: Kubernetes: Make your services faster by removing CPU limits
#26> kops: Since June 2020, kops 1.18+ will start using Ubuntu 20.04 as the default host image. If you’re using a lower version of kops, you’ll have to probably to wait the fix. We are currently in this situation.
We're running Ubuntu 20.04 on Kops 1.17 in production just fine, thank you very much. It wasn't a happy path since it wasn't officially supported then - stuff about forcing iptables-legacy instead of nftables - but with a couple hacks we got it to work just fine (Kops was in a bad situation where CoreOS was hitting EOL and there were no officially supported distributions running updated kernels that patched the CPU throttling issues, so we worked with the maintainers to figure out what we needed to do, as the maintainers were also running Ubuntu 20.04 on versions of Kops which didn't formally support it).
This whole blog post is dangerous. CPU limits are really important for cluster stability, as I'm sure the author will find out soon enough. Why bother with dangerous workarounds for problems that have actual solutions? This makes no sense to me.
Re: Kubernetes: Make your services faster by removing CPU limits
#27Removing CPU limits seems like a bad idea now that there's a kernel fix. But putting that aside...
I don't understand why pods without CPU limits would cause unresponsive kubelets. For a long time now Kubernetes has allocated a slice for system services. While pods without CPU limits are allowed to burst, they are still limited to the amount of CPU allocated to kubernetes pods.
Run "systemd-cgls" on a node and you'll see two toplevel slices: kubepods and system. The kubelet process lives within the system slice.
If you run "kubectl describe " you can see the resources set aside for system processes on the node. Processes in the system slice should always have (cpu_capacity - cpu_allocatable) available to share, no matter what happens in the kubepods slice.
Capacity:
cpu: 8
ephemeral-storage: 83873772Ki
memory: 62907108Ki
Allocatable:
cpu: 7910m
ephemeral-storage: 76224326324
memory: 61890276Ki
pods: 58
Re: Kubernetes: Make your services faster by removing CPU limits
#28This is terrible advice.
Re: Kubernetes: Make your services faster by removing CPU limits
#29I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.
Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.
Can’t remember the last time I had a centos kernel update (kernel-ml) break anything over the past 7 odd years I’ve been running it across hundreds of servers.
Re: Kubernetes: Make your services faster by removing CPU limits
#30Here's a story that might make you not want to do that.
We ran Kubernetes with the standard scheduler and node autoscaling for a long time, and used to allow developers in our (simplified) manifests define resource requests and limits. We saw that with our current config, we always had some unused capacity (that we wanted) since the scheduler spread out workloads while the autoscaler was only throwing nodes away with less than 70% load. So we started ignoring the limits provided by developers. This was initially a great success, our response times in the 99th went down drastically, even during sudden traffic spikes.
2 years later, and nobody cares about resource allocation for new services anymore. We can essentially never disable bursting again, because too many services (100+) use the extra capacity constantly, and due to our organizational structure we can't really _make_ these teams fix their allocations.