Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes: Make your services faster by removing CPU limits

erickhun.com

Post reply on HN
Print view View original

71–80 of 111 posts

Re: Kubernetes: Make your services faster by removing CPU limits

#71

Post by rsanders »

dilyevsky wrote: post #57
rsanders wrote: post #27

Removing CPU limits seems like a bad idea now that there's a kernel fix. But putting that aside... I don't understand why pods without CPU limits would cause unresponsive kubelets. For a long time now Kubernetes has allocated a slice for system services. While pods without CPU limits are allowed to burst, they are still limited to the amount of CPU allocated to kubernetes pods. Run "systemd-cgls" on a node and you'll…

Even without the bug it will have negative effect on latency and generally is not really needed for un-metered workloads (there’re posts by thockin on reddit and github that describe this in detail) To answer your other question - I believe kops ships without system reserved by default

Can you explain how having a CPU limit set (at any level) has a negative effect on latency? That's an important factor to understand.

The arguments for allowing containers to burst makes plenty of sense to me. I do it on most of my services!

thockin's reddit post for reference: https://www.reddit.com/r/kubernetes/comments/all1vg/on_kuber...

Another interesting bit of context describing some of the non-intuitive impacts of CPU limits: https://github.com/kubernetes/kubernetes/issues/51135

Edit: added links

Re: Kubernetes: Make your services faster by removing CPU limits

#72

Post by cptomlly »

javiercr wrote: post #39

Does anyone know if this bug is present on Google Kubernetes Engine (GKE)?

Fixed in the following COS stable images back in January: cos-stable-79-12607-80-0 cos-stable-77-12371-141-0 cos-stable-73-11647-415-0 cos-stable-78-12499-89-0

According to https://cloud.google.com/container-optimized-os/docs/release...

Re: Kubernetes: Make your services faster by removing CPU limits

#73

Post by dundarious »

secondcoming wrote: post #2

Couldn't the unresponsive kubectl issue be resolved by isolating cpus and controlling yourself where processes go?

This is standard practice in some domains. Reserve 1 or 2 cores for admin and possibly for the interrupt daemon, and isolate/affinitize app processes to the remaining cores -- ideally giving more resources to your "hot" threads, and better resources (the core "closest" to your NIC) to your network thread, etc.

Re: Kubernetes: Make your services faster by removing CPU limits

#74

Post by dilyevsky »

rsanders wrote: post #71
dilyevsky wrote: post #57

Earlier quoted context omitted.

Even without the bug it will have negative effect on latency and generally is not really needed for un-metered workloads (there’re posts by thockin on reddit and github that describe this in detail) To answer your other question - I believe kops ships without system reserved by default

Can you explain how having a CPU limit set (at any level) has a negative effect on latency? That's an important factor to understand. The arguments for allowing containers to burst makes plenty of sense to me. I do it on most of my services! thockin's reddit post for reference: https://www.reddit.com/r/kubernetes/comments/all1vg/on_kuber... Another interesting bit of context describing some of the non-intuitive impac…

It’s mentioned elsewhere on this thread but essentially with cfs quota period default which is 100ms it’s really easy for multithreaded process to exhaust the quota and just sit there idle until next period. Another thing is if you have spare cycles that you presumably already paid for why not just use them?

Re: Kubernetes: Make your services faster by removing CPU limits

#75

Post by Thaxll »

rsanders wrote: post #63
Thaxll wrote: post #50

Earlier quoted context omitted.

It's pretty simple, limits work only when everyone are using them. If you have one pod that does not enforce limits it can disrupt the entire node.

A container with a request but without a limit should be scheduled as Burstable, and it should only receive allocations in excess of its request when all other containers have had their demand A container without either request or limit is twice-damned, and will be scheduled as BestEffort. The entire cgroup slice for all BestEffort pods is given a cpu.shares of 2 milliCPUs, and if the kernel scheduler is functioning…

Are you sure that BestEffort QOS do not disrupt the entire node? I remember in the past a single pod would freeze the entire VM.

Re: Kubernetes: Make your services faster by removing CPU limits

#76

Post by Thaxll »

ravedave5 wrote: post #62
ledneb wrote: post #36

> https://engineering.indeedblog.com/blog/2019/12/unthrottled-... This is a more detailed post on the same thing - part two indicates changes have been back-ported to a number of kernel versions: Linux-stable: 4.14.154+, 4.19.84+, 5.3.9+ Ubuntu: 4.15.0-67+, 5.3.0-24+ Redhat Enterprise Linux: RHEL 7: 3.10.0-1062.8.1.el7+ RHEL 8: 4.18.0-147.2.1.el8_1+ CoreOS: v4.19.84+

Know which version of alpine linux would have gotten this fix? I'm having a hard time walking it back from the commit.

There is a low probability that your host is Alpine, more like your pods.

Re: Kubernetes: Make your services faster by removing CPU limits

#78

Post by fierro »

The core principle most readers miss is that CPU limits are tied to CPU throttling, which is markedly different than CPU time sharing. I would argue that in 99% of cases, you truly do not need or want limits.

limits cause CPU throttling, which is like running your process in a strobe light. If your quota period is 100ms, you might only be able to make progress for 10ms out of every 100ms period, regardless of whether or not there is CPU contention, just because you've exceeded your limit.

requests -> CFS time sharing. This ensures that out of a given period of time, CPU time is scheduled fairly and according to the request as a proportion of total request (it just so happens that the Kube scheduler won't schedule such that sum[requests] > capacity, but theoretically it could because requests are truly relative when it comes to how they are represented in cgroups)

Here is the fundamental assertion: requests ensure fair CPU scheduling in the event of CPU contention (more processes want CPU than can be scheduled). Given that you are using requests, why would you want limits? You might think "limits prevent a process from taking too much CPU" but that's just not true. If that processes DID try to use up too much CPU, CFS would ensure it does not via fair time sharing. If no other running processes needed the CPU, why enforce CPU throttling which has very bad effects on tail latency?

Re: Kubernetes: Make your services faster by removing CPU limits

#80

Post by GauntletWizard »

dijit wrote: post #68
dundarious wrote: post #67

In the low latency trading world, these concerns are addressed by partitioning resources (for CPU, with affinities). This seems like a simpler mechanism that doesn’t require the kernel/daemon to track resource usage and to impose limits. I see only upsides to performance (bandwidth and latency) and availability by partitioning resources — so what are the benefits of the alternative, using limits, beyond being able to…

Video Games industry is the same, in fact it was one of the reasons we went with google cloud over alternatives, at the time Amazon was not using KVM (or, HVM as they seem to call it)- and GCP was at least attempting CPU affinity on the VMs, this caused quite a variance in latency when using amazon which did not exist on GCP. To answer your question: I believe there is 'pinning' in Kubernetes which can solve it, but…

For videogames, you should not be subject to the iptables bits - Agones encourages use of the `hostPort` networking mode, which doesn't create or require special iptables routing.
Post reply on HN