Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes: Make your services faster by removing CPU limits

erickhun.com

Post reply on HN
Print view View original

61–70 of 111 posts

Re: Kubernetes: Make your services faster by removing CPU limits

#61

Post by dilyevsky »

jeffbee wrote: post #53
davewritescode wrote: post #4

This seems like a bad trade-off, at least for 99% of us who haven’t been using Kubernetes in production for the last 5 years and manage it ourselves. Putting all the “user facing” services in a state where one of them consuming all the CPU could affect all the others feels like a disaster waiting to happen.

If there are several services all with the same share of CPU resources and with no configured limits, and they are all runnable, then none of them will be able to starve the others. The kernel will schedule them each in turn. If you configure a static limit what you get is services that don't run even when there is CPU time available, which is bad.

Exactly! Most commenters piling on the idea clearly don’t know about cpu shares and how they are used in container runtimes.

Re: Kubernetes: Make your services faster by removing CPU limits

#62

Post by ravedave5 »

ledneb wrote: post #36

> https://engineering.indeedblog.com/blog/2019/12/unthrottled-... This is a more detailed post on the same thing - part two indicates changes have been back-ported to a number of kernel versions: Linux-stable: 4.14.154+, 4.19.84+, 5.3.9+ Ubuntu: 4.15.0-67+, 5.3.0-24+ Redhat Enterprise Linux: RHEL 7: 3.10.0-1062.8.1.el7+ RHEL 8: 4.18.0-147.2.1.el8_1+ CoreOS: v4.19.84+

Know which version of alpine linux would have gotten this fix? I'm having a hard time walking it back from the commit.

Re: Kubernetes: Make your services faster by removing CPU limits

#63

Post by rsanders »

Thaxll wrote: post #50
rsanders wrote: post #27

Removing CPU limits seems like a bad idea now that there's a kernel fix. But putting that aside... I don't understand why pods without CPU limits would cause unresponsive kubelets. For a long time now Kubernetes has allocated a slice for system services. While pods without CPU limits are allowed to burst, they are still limited to the amount of CPU allocated to kubernetes pods. Run "systemd-cgls" on a node and you'll…

It's pretty simple, limits work only when everyone are using them. If you have one pod that does not enforce limits it can disrupt the entire node.

A container with a request but without a limit should be scheduled as Burstable, and it should only receive allocations in excess of its request when all other containers have had their demand A container without either request or limit is twice-damned, and will be scheduled as BestEffort. The entire cgroup slice for all BestEffort pods is given a cpu.shares of 2 milliCPUs, and if the kernel scheduler is functioning well, no pod in there is going to disrupt the anything but other BestEffort pods with any amount of processor demand. Throw in a 64 thread busyloop and no Burstable or Guaranteed pods should notice much.

Of course that's the ideal. There is an observable difference between a process that relinquishes its scheduler slice and one that must be pre-empted. But I wouldn't call that a major disruption. Each pod will still be given its full requested share of CPU.

If that's not the case, I'd love to know!

Re: Kubernetes: Make your services faster by removing CPU limits

#64

Post by poisonta »

You should not run more than one application/service in a VM if you are worried about the performance. Then, you don't need to worry about configuration CPU limits. Kubernetes doesn't only slow down your application performance, it also increases your operating cost and team by several magnitudes.

Re: Kubernetes: Make your services faster by removing CPU limits

#65

Post by markbnj »

Schwan wrote: post #10

Is this really right? "The danger of not setting a CPU limit is that containers running in the node could exhaust all CPU available." My assumptions have been: 1. cpu request tells you how much cpu a pod gets MINIMUM always, independently of how much other pods use it or not 2. on GKE you can't request 100% cpu due to google reserving cpu for the node 3. if you have hard limits, your cluster utilisation will be bad -…

The reason a container with no limit can exhaust CPU is that kubernetes CPU requests map to the cpushares accounting system, and CPU limits map to the Completely Fair Scheduler's cpuquota system. The cpushares system divides a core into 1024 shares, and guarantees a process gets the number of shares it reserves, but it does not limit the process from taking more shares if other processes aren't consuming them. The cpuquota system divides CPU time into periods of... I think... 100k microseconds by default, and hard limits a process at the number of microsecs per periods it requests. So if you don't set limits you're only using the cpushares system, and are free to take up as much idle CPU as you can grab.

Re: Kubernetes: Make your services faster by removing CPU limits

#66

Post by jrockway »

jeffbee wrote: post #48
uberduper wrote: post #41

Bit of a warning. If you do not set cpu requests, your pods may end up with cpu.shares=2. Java, for example, makes some tuning decisions based on this that you're not gonna like.

The Go runtime also locks in some unwarranted assumptions at process start time, and never changes its parameters if the number of available CPUs changes.

Explicitly setting GOMAXPROCS is probably the cleanest way to limit CPU among the runtimes that are out there, however. For example, if you set requests = 1, limits = 1, GOMAXPROCS=1, then you will never run into the latency-increasing cfs cpu throttling; you would be throttled if you used more than 1 CPU, but since you can't (modulo forks, of course), it won't happen. There is https://github.com/uber-go/automaxprocs to set this automatically, if you care.

You are right that by default, the logic that sets GOMAXPROCS is unaware of the limits you've set. That means GOMAXPROCS will be something much higher than your cpu limit, and an application that uses all available CPUs will use all of its quota early on in the cfs_period_us interval, and then sleep for the rest of it. This is bad for latency.

Re: Kubernetes: Make your services faster by removing CPU limits

#67

Post by dundarious »

In the low latency trading world, these concerns are addressed by partitioning resources (for CPU, with affinities). This seems like a simpler mechanism that doesn’t require the kernel/daemon to track resource usage and to impose limits.

I see only upsides to performance (bandwidth and latency) and availability by partitioning resources — so what are the benefits of the alternative, using limits, beyond being able to stuff more apps onto a machine? That’s not to trivialize that benefit.

Does kubernetes even allow for “affinitizing”?

Re: Kubernetes: Make your services faster by removing CPU limits

#68

Post by dijit »

dundarious wrote: post #67

In the low latency trading world, these concerns are addressed by partitioning resources (for CPU, with affinities). This seems like a simpler mechanism that doesn’t require the kernel/daemon to track resource usage and to impose limits. I see only upsides to performance (bandwidth and latency) and availability by partitioning resources — so what are the benefits of the alternative, using limits, beyond being able to…

Video Games industry is the same, in fact it was one of the reasons we went with google cloud over alternatives, at the time Amazon was not using KVM (or, HVM as they seem to call it)- and GCP was at least attempting CPU affinity on the VMs, this caused quite a variance in latency when using amazon which did not exist on GCP.

To answer your question: I believe there is 'pinning' in Kubernetes which can solve it, but kubernetes has other overheads in terms of latency (iptables pod routing with contrack enabled for instance) so I personally would avoid using it for low latency applications.

https://builders.intel.com/docs/networkbuilders/cpu-pin-and-...

Re: Kubernetes: Make your services faster by removing CPU limits

#69

Post by kamaradclimber »

I encountered that issue on my company Mesos cluster. Here are some details.

We ran our largest application from bare-metal to Mesos (https://medium.com/criteo-labs/migrating-arbitrage-to-apache...) and observed performance was not as good as expected (especially on 99pctl latency). Other application were showing similar behavior.

We ended up finding the issue with cfs bandwidth cgroup, considered several alternatives and eventually moved to cpusets instead.

cpusets allow to get: - better mental model (it's far easier to reason on "dedicated cpus") - net performance gain (from -5% to -10% cpu consumption) - more consistent latency (if nothing run on the same cpu than your app, you benefit from good scheduling and possibly avoid cpu cache issues)

When the fixed kernel was released, we decided to upgrade to it and keep our new model of cpu isolation.

Re: Kubernetes: Make your services faster by removing CPU limits

#70

Post by jeffbee »

jrockway wrote: post #66
jeffbee wrote: post #48

Earlier quoted context omitted.

The Go runtime also locks in some unwarranted assumptions at process start time, and never changes its parameters if the number of available CPUs changes.

Explicitly setting GOMAXPROCS is probably the cleanest way to limit CPU among the runtimes that are out there, however. For example, if you set requests = 1, limits = 1, GOMAXPROCS=1, then you will never run into the latency-increasing cfs cpu throttling; you would be throttled if you used more than 1 CPU, but since you can't (modulo forks, of course), it won't happen. There is https://github.com/uber-go/automaxprocs…

Setting GOMAXPROCS explicitly is the best practice in my experience. The runtime latches in a value for runtime.NumCPU() based on the population count of the cpumask at startup. The cpumask can change if kubernetes schedules or de-schedules a "guaranteed" pod on your node and the kubelet is using the static CPU management policy, and it will vary from node to node if you have various types of machines. You don't want to have 100 replicas of your microservice all using different, randomly-chose values of GOMAXPROCS.
Post reply on HN