For the Love of God, Stop Using CPU Limits on Kubernetes • Hacker News Board

Re: For the Love of God, Stop Using CPU Limits on Kubernetes

#21

Post by jauntywundrkind » Tue, Jul 02, 2024, 3:28 PM UTC

It's sad to me that Kubernetes doesn't expose the excellent hierarchical system for rationing CPU that's built into the kernel: cgroups. It has its own separate constraint system, makes its own scheduler. And it just seems not as good, not as flexible, as the hierarchical system cgroups offers.

Being able to make a cgroup where essential services as a whole share a pool guaranteed 30%, then further refining & trading off that pool & other work pools feels like such a superpower. Compared to having to manage all services in flat, absolute terms.

Re: For the Love of God, Stop Using CPU Limits on Kubernetes

#22

Post by cassianoleal » Tue, Jul 02, 2024, 4:59 PM UTC

potoftea wrote: post #8

lantry wrote: post #6

Earlier quoted context omitted.

The post recommends setting both request and limit for memory. It's only for CPU that it says to use a request and not a limit. It's my understanding that the kernel only kills processes when it's out of memory. In the docs you linked, there doesn't seem to be an "eviction signal" for CPU.

This doens't match what I've read. IF you don't set limit to cpu & memory, your pod QoS will be burstable and from kernel/cgroup perspective this process can be killed - it's only question of oom_score_adj. OOMScore from linux - https://www.freedesktop.org/software/systemd/man/latest/syst...

OOM means "Out Of Memory". The post is about CPU.

> Everything in this post is about CPU and not memory.

Re: For the Love of God, Stop Using CPU Limits on Kubernetes

#23

Post by cassianoleal » Tue, Jul 02, 2024, 5:02 PM UTC

candiddevmike wrote: post #12

kstrauser wrote: post #3

Earlier quoted context omitted.

What’s wrong about it? I haven’t managed k8s much and don’t have enough experience to evaluate their claim.

K8s works best (economically) when it can bin pack things. The only way it can bin pack things safely is by having user provided limits--its not smart enough to right size your app (out of the box at least). Not setting them means you're going to end up paying more or have resource contention/outages. It's better to set the limits higher than you need than to not set them at all. Ideally this is easily done since you…

The problem is that the Kubernetes CPU scheduler is... not great.

There are many instances where it will throttle CPU of a pod way before it even reaches the requested amount, let alone the limits. This is only true if you set limits. If you don't, it will always have the requested amount available, plus whatever can be spared that's not in use by other processes. It will throttle at most down to the requested amount, and in most cases not at all.

Re: For the Love of God, Stop Using CPU Limits on Kubernetes

#24

Post by potoftea » Tue, Jul 02, 2024, 8:57 PM UTC

cassianoleal wrote: post #22

potoftea wrote: post #8

Earlier quoted context omitted.

This doens't match what I've read. IF you don't set limit to cpu & memory, your pod QoS will be burstable and from kernel/cgroup perspective this process can be killed - it's only question of oom_score_adj. OOMScore from linux - https://www.freedesktop.org/software/systemd/man/latest/syst...

OOM means "Out Of Memory". The post is about CPU. > Everything in this post is about CPU and not memory.

I am aware of it, are you aware that Kubernetes assigned Burstable Qos if you don't defined CPU requests and limits?

What I'm trying to say that while try to avoid one pitfall you fall right into an other - in short there are good reasons to use cpu limits.

Re: For the Love of God, Stop Using CPU Limits on Kubernetes

#25

Post by cassianoleal » Tue, Jul 02, 2024, 11:14 PM UTC

potoftea wrote: post #24

cassianoleal wrote: post #22

Earlier quoted context omitted.

OOM means "Out Of Memory". The post is about CPU. > Everything in this post is about CPU and not memory.

I am aware of it, are you aware that Kubernetes assigned Burstable Qos if you don't defined CPU requests and limits? What I'm trying to say that while try to avoid one pitfall you fall right into an other - in short there are good reasons to use cpu limits.

That's not what you're saying though. You're also putting requests/limits for CPU and memory in the same class, which they're not.

Lack of CPU will never cause a pod to be killed. Evicted, yes. Killed, no.

Lack of memory can cause a pod to be OOM killed. That's an entirely different failure mode.

Eviction should always be expected by your pods. They should be able to handle eviction gracefully. On the other hand, there is nothing you can do to handle an OOM kill gracefully.