Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes: Make your services faster by removing CPU limits

erickhun.com

Post reply on HN
Print view View original

51–60 of 111 posts

Re: Kubernetes: Make your services faster by removing CPU limits

#51

Post by Sayrus »

whalesalad wrote: post #49

Last time I was knee deep in managing prod Kube infrastructure limits were also there for scheduling purposes. It's hard to properly allocate services across nodes when there is no concept of requirements they have. I guess you can get around that with setting a request versus a limit?

Pod scheduling is based on Requests so you can definitely go without setting a limit. I don't think the limit itself plays a role in Kubernetes scheduling (unless you don't specify a request, then limit equals request).

And you are definitely right: scheduling a pod without request/limit is like giving a blank check.

Re: Kubernetes: Make your services faster by removing CPU limits

#52

Post by theptip »

I’m wondering if using a bursty limit like

    requests:
      cpu: 100m
    limits:
      cpu: 200m
Would work better? If the problem is that you are getting throttled at a lower rate than your specified limit, maybe bumping that would help. But you still get to use your target “request” for bin packing / node resource tracking.

This would depend on the throttle level being proportional to the specified limit and not something orthogonal like number of processes - but if you don’t want to turn off limits entirely it might at least help.

Re: Kubernetes: Make your services faster by removing CPU limits

#53

Post by jeffbee »

davewritescode wrote: post #4

This seems like a bad trade-off, at least for 99% of us who haven’t been using Kubernetes in production for the last 5 years and manage it ourselves. Putting all the “user facing” services in a state where one of them consuming all the CPU could affect all the others feels like a disaster waiting to happen.

If there are several services all with the same share of CPU resources and with no configured limits, and they are all runnable, then none of them will be able to starve the others. The kernel will schedule them each in turn.

If you configure a static limit what you get is services that don't run even when there is CPU time available, which is bad.

Re: Kubernetes: Make your services faster by removing CPU limits

#54

Post by toomanymike »

I've seen CPU throttling occur when limits aren't exhausted even on 5.4 kernels, so I don't believe the underlying kernel bug is fixed.

One option not mentioned in the post is to enable k8s' static CPU scheduler policy. With this option in place workloads in the "guaranteed" quality of service class that are allocated an integer CPU limit will be given exclusive use of their CPUs. I've found this also avoids the CFS bugs and eliminates CPU throttling, without removing CPU limits.

One thing to keep in mind is that this bug mostly impacts workloads that spin up more threads then they have allocated CPUs. For golang workloads you can set GOMAXPROCS to be equal to your CPU allocation and eliminate most throttling that way too, without messing with limits or the static scheduler policy

Re: Kubernetes: Make your services faster by removing CPU limits

#55

Post by ithkuil »

Thaxll wrote: post #50
rsanders wrote: post #27

Removing CPU limits seems like a bad idea now that there's a kernel fix. But putting that aside... I don't understand why pods without CPU limits would cause unresponsive kubelets. For a long time now Kubernetes has allocated a slice for system services. While pods without CPU limits are allowed to burst, they are still limited to the amount of CPU allocated to kubernetes pods. Run "systemd-cgls" on a node and you'll…

It's pretty simple, limits work only when everyone are using them. If you have one pod that does not enforce limits it can disrupt the entire node.

Could you please elaborate on why's that so?

Re: Kubernetes: Make your services faster by removing CPU limits

#56

Post by solatic »

toomanymike wrote: post #54

I've seen CPU throttling occur when limits aren't exhausted even on 5.4 kernels, so I don't believe the underlying kernel bug is fixed. One option not mentioned in the post is to enable k8s' static CPU scheduler policy. With this option in place workloads in the "guaranteed" quality of service class that are allocated an integer CPU limit will be given exclusive use of their CPUs. I've found this also avoids the CFS…

Enabling the static CPU scheduler policy currently requires setting a Kubelet flag, and that puts it out of reach of most people running managed Kubernetes distributions.

Re: Kubernetes: Make your services faster by removing CPU limits

#57

Post by dilyevsky »

rsanders wrote: post #27

Removing CPU limits seems like a bad idea now that there's a kernel fix. But putting that aside... I don't understand why pods without CPU limits would cause unresponsive kubelets. For a long time now Kubernetes has allocated a slice for system services. While pods without CPU limits are allowed to burst, they are still limited to the amount of CPU allocated to kubernetes pods. Run "systemd-cgls" on a node and you'll…

Even without the bug it will have negative effect on latency and generally is not really needed for un-metered workloads (there’re posts by thockin on reddit and github that describe this in detail)

To answer your other question - I believe kops ships without system reserved by default

Re: Kubernetes: Make your services faster by removing CPU limits

#58

Post by dilyevsky »

YawningAngel wrote: post #3

I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.

This post explains it - https://www.reddit.com/r/kubernetes/comments/all1vg/comment/...

Re: Kubernetes: Make your services faster by removing CPU limits

#60

Post by dilyevsky »

sascha_sl wrote: post #30

Here's a story that might make you not want to do that. We ran Kubernetes with the standard scheduler and node autoscaling for a long time, and used to allow developers in our (simplified) manifests define resource requests and limits. We saw that with our current config, we always had some unused capacity (that we wanted) since the scheduler spread out workloads while the autoscaler was only throwing nodes away with…

We at Mux have removed nearly all limits but setup alerts that trigger when container consistently bursts above the request so we can chase those down (temporary bursts are ignored). Never had any issues
Post reply on HN