For the Love of God, Stop Using CPU Limits on Kubernetes
home.robusta.dev
For the Love of God, Stop Using CPU Limits on Kubernetes
1–10 of 25 posts
Re: For the Love of God, Stop Using CPU Limits on Kubernetes
#2Almost 2-yo post and still as wrong as it was when it was posted.
Re: For the Love of God, Stop Using CPU Limits on Kubernetes
#3Almost 2-yo post and still as wrong as it was when it was posted.
What’s wrong about it? I haven’t managed k8s much and don’t have enough experience to evaluate their claim.
Re: For the Love of God, Stop Using CPU Limits on Kubernetes
#4[deleted]
Re: For the Love of God, Stop Using CPU Limits on Kubernetes
#5Almost 2-yo post and still as wrong as it was when it was posted.
What’s wrong about it? I haven’t managed k8s much and don’t have enough experience to evaluate their claim.
I can't comment about about other statement.
But there are serious drawbacks if you don't set requests/limits for mission-critical process, is that they can be killed by kernel to free some resources (if the node reach max resource usage)
When you don't set cpu/memory limit your pod QoS class is burstable, which better then BestEffort, but still get assigned `oom_score_adj` score. IMHO you almost 99% you want `Guaranteed` for critical process.
1. oom_score_adj - https://kubernetes.io/docs/concepts/scheduling-eviction/node...
2. Guaranteed - https://kubernetes.io/docs/tasks/configure-pod-container/qua...
3. QoS - https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/
Re: For the Love of God, Stop Using CPU Limits on Kubernetes
#6Earlier quoted context omitted.
What’s wrong about it? I haven’t managed k8s much and don’t have enough experience to evaluate their claim.
I can't comment about about other statement. But there are serious drawbacks if you don't set requests/limits for mission-critical process, is that they can be killed by kernel to free some resources (if the node reach max resource usage) When you don't set cpu/memory limit your pod QoS class is burstable, which better then BestEffort, but still get assigned `oom_score_adj` score. IMHO you almost 99% you want `Guaran…
The post recommends setting both request and limit for memory. It's only for CPU that it says to use a request and not a limit. It's my understanding that the kernel only kills processes when it's out of memory. In the docs you linked, there doesn't seem to be an "eviction signal" for CPU.
Re: For the Love of God, Stop Using CPU Limits on Kubernetes
#7Almost 2-yo post and still as wrong as it was when it was posted.
Yeah, it's a pretty weird take. I would agree if they said "a lot of people don't know what CPU limits are and set them without thinking", but "you should never use them" doesn't make any sense. For example, every computer anyone has ever used has a CPU limit. There isn't infinite CPU inside your computer. That's a CPU limit.
The mistake that surprises people is "I'm going to tell my app it can use 128 CPUs but I'm setting the CPU limit to 1." OK, well, then your app is going to be asleep 127/128th of the time. It's a surprise because the app reads /proc/cpuinfo to guess how many CPUs you have, but ... that is not the correct algorithm.
Another thing is imagining that usage spikes are going to occur randomly throughout time. If app A is under heavy use, it can steal app B's CPU shares, because who would use app A and app B at the same time? Most of the time they're both idle, so it's a waste to reserve 1 CPU for app A and 1 CPU for app B, and have app A throttled while app B is idle. But you'll probably find that everything you host is popular from 9am to 5pm local time, and for 16 hours a day you are using 0% CPU and for 8 hours a day you are using 200% CPU. The idea is to guarantee some quality of service for both apps, even at busy times. The goal is not to maximize overall throughput.
You can tune your latency vs. throughput goals if all the apps are yours, but as soon as you have different teams, I doubt team B is going to say "sure we can get paged for high request latency as long as Team A is getting as much of the CPU as they can". That's what CPU limits are for, consistency when things get tough. Not for overall utilization.
Re: For the Love of God, Stop Using CPU Limits on Kubernetes
#8Earlier quoted context omitted.
I can't comment about about other statement. But there are serious drawbacks if you don't set requests/limits for mission-critical process, is that they can be killed by kernel to free some resources (if the node reach max resource usage) When you don't set cpu/memory limit your pod QoS class is burstable, which better then BestEffort, but still get assigned `oom_score_adj` score. IMHO you almost 99% you want `Guaran…
The post recommends setting both request and limit for memory. It's only for CPU that it says to use a request and not a limit. It's my understanding that the kernel only kills processes when it's out of memory. In the docs you linked, there doesn't seem to be an "eviction signal" for CPU.
This doens't match what I've read.
IF you don't set limit to cpu & memory, your pod QoS will be burstable and from kernel/cgroup perspective this process can be killed - it's only question of oom_score_adj.
OOMScore from linux - https://www.freedesktop.org/software/systemd/man/latest/syst...
Re: For the Love of God, Stop Using CPU Limits on Kubernetes
#9I used to be on the same page as the author. But then I saw tons of application teams not setting CPU limits, and coming to rely on the bursting (in other words, their requests were too low). Thus when the system came under load their application started slowing in unexpected ways.
We've had success with CPU limits, and horizontal scaling.
Re: For the Love of God, Stop Using CPU Limits on Kubernetes
#10Almost 2-yo post and still as wrong as it was when it was posted.
What’s wrong about it? I haven’t managed k8s much and don’t have enough experience to evaluate their claim.
Users come to expect the performance that was never promised or even properly requested. Once you efficiently load the system they complain because you overdelivered and they got used to it.