Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go, Containers, and the Linux Scheduler

riverphillips.dev

Post reply on HN
Print view View original

111–120 of 143 posts

Re: Go, Containers, and the Linux Scheduler

#111

Post by sethammons »

jeffbee wrote: post #53
sethammons wrote: post #45

Earlier quoted context omitted.

We use https://github.com/uber-go/automaxprocs after we joyfully discovered that Go assumed we had the entire cluster's cpu count on any particular pod. Made for some very strange performance characteristics in scheduling goroutines.

My opinion is that setting GOMAXPROCS that way is a quite poor idea. It tends to strand resources that could have been used to handle a stochastic burst of requests, which with a capped GOMAXPROCS will be converted directly into latency. I can think of no good reason why GOMAXPROCS needs to be 2 just because you expect the long-term CPU rate to be 2. That long-term quota is an artifact of capacity planning, while GOM…

How do you suggest handling that?

Re: Go, Containers, and the Linux Scheduler

#112

Post by arghwhat »

ithkuil wrote: post #101
otterley wrote: post #90

This sort of tuning isn't necessary if you use CPU reservations instead of limits, as you should: https://home.robusta.dev/blog/stop-using-cpu-limits CPU reservations are limits, just implicit ones and declared as guarantees. So let the Go runtime use all the CPUs available, and let the Linux scheduler throttle according to your declared reservations if the CPU is contended for.

I don't set limits because I'm afraid of how a pod is going to affect other pods. I set limits because I don't want to get used to being able to tap on the excess CPU available because that's not guaranteed to be available. As the node fills up with more and more other pods, it's possible that a pod that was running just fine a moment ago is crawling to a halt. Limits allow me to simulate the same behavior and plan f…

Limiting CPU to the amount guaranteed to be available also guarantees very significant wasted resource utilization unless all your pods spin 100% CPU continuously.

The best way to utilize resources is to overcommit, and the smart way to overcommit is to, say, allow 4x overcommit with each allocation limited to 1/4th of the available resources so not individual peak can choke the system. Given varied allocations, things average out with a reasonable amount of performance variability.

Idle CPUs are wasted CPUs and money out the window.

Re: Go, Containers, and the Linux Scheduler

#113

Post by afr0ck »

iTokio wrote: post #96
yipbub wrote: post #93

Earlier quoted context omitted.

Interesting. This is not true for Memory, correct? The OOMKiller might get you. You also cannot achieve a QoS class of Guaranteed without both CPU and Memory limits, so the pod might be evicted at some point.

> Memory is different because it is non-compressible - once you give memory you can't take it away without killing the process

Swap (Disk, RDMA, Compression)? Page migration (NUMA, CXL)?

Re: Go, Containers, and the Linux Scheduler

#114

Post by riv991 »

dilyevsky wrote: post #95
riv991 wrote: post #38

Earlier quoted context omitted.

I've seen significant performance gains from this in production. Other people have encountered it too hence libraries like Automaxprocs existing and issues being open with Go for it.

Gains by what metric? Are you sure you didn't trade in better latency for worse overall throughput? Also, sure you didn't hit one of many CFS overaccounting bugs which we've seen a few? Have you compared performance without the limit at all?

Previously we had no limit. We observed gains in both latency and throughput by implementing Automaxprocs and decided to roll it out widely.

This aligns with what others have reported on the Go runtime issue open for this.

"When go.uber.org/automaxprocs rolled out at Uber, the effect on containerized Go services was universally positive. At least at the time, CFS imposed such heavy penalties on Go binaries exceeding their CPU allotment that properly tuning GOMAXPROCS was a significant latency and throughput improvement."

https://github.com/golang/go/issues/33803#issuecomment-14308...

Re: Go, Containers, and the Linux Scheduler

#115

Post by midko »

arghwhat wrote: post #112
ithkuil wrote: post #101

Earlier quoted context omitted.

I don't set limits because I'm afraid of how a pod is going to affect other pods. I set limits because I don't want to get used to being able to tap on the excess CPU available because that's not guaranteed to be available. As the node fills up with more and more other pods, it's possible that a pod that was running just fine a moment ago is crawling to a halt. Limits allow me to simulate the same behavior and plan f…

Limiting CPU to the amount guaranteed to be available also guarantees very significant wasted resource utilization unless all your pods spin 100% CPU continuously. The best way to utilize resources is to overcommit, and the smart way to overcommit is to, say, allow 4x overcommit with each allocation limited to 1/4th of the available resources so not individual peak can choke the system. Given varied allocations, thin…

> with each allocation limited to 1/4th of the available resources so not individual peak can choke the system.

This assumes that the scheduled workloads are created equal which isn't the case. The app owners do not have control over what else gets scheduled on the node which introduces uncontrollable variability in the performance of what should be identical replicas and environments. What helps here is .. limits. The requests-to-limits ratio allows application owners to reason about the variability risk they are willing to take in relation to the needs of the application (e.g. imagine a latency-sensitive workload on a critical path vs a BAU service vs a background job which just cares about throughput -- for each of these classes, the ratio would probably be very different). This way, you can still overcommit but not by a rule-of-thumb that is created centrally by the cluster ops team (e.g. aim for 1/4) but it's distributed across each workload owner (ie application ops) where this can be done a lot more accurately and with better results. This is what the parent post is also talking about.

Re: Go, Containers, and the Linux Scheduler

#116

Post by arghwhat »

midko wrote: post #115
arghwhat wrote: post #112

Earlier quoted context omitted.

Limiting CPU to the amount guaranteed to be available also guarantees very significant wasted resource utilization unless all your pods spin 100% CPU continuously. The best way to utilize resources is to overcommit, and the smart way to overcommit is to, say, allow 4x overcommit with each allocation limited to 1/4th of the available resources so not individual peak can choke the system. Given varied allocations, thin…

> with each allocation limited to 1/4th of the available resources so not individual peak can choke the system. This assumes that the scheduled workloads are created equal which isn't the case. The app owners do not have control over what else gets scheduled on the node which introduces uncontrollable variability in the performance of what should be identical replicas and environments. What helps here is .. limits. T…

1/4th was merely an example for one resource type, and a suitable limit may be much lower depending on the cluster and workloads. The point is that a limit set to 1/workloads guarantees wasted resources, and should be set significantly higher based on realistic workloads, while still ensuring that it takes N workloads to consume all resource to average out the risk of peak demand collisions.

> This assumes that the scheduled workloads are created equal which isn't the case.

This particular allocation technique benefits from scheduled workloads not being equal as equality would increase likelihood of peak demand collisions.

Re: Go, Containers, and the Linux Scheduler

#117

Post by jen20 »

tail_exchange wrote: post #73
jeffbee wrote: post #67

Earlier quoted context omitted.

Kubernetes requires a platform runtime that answers its requests, but there's no law enforcement agency that will prevent you from using a custom runtime that ignores the very existence of Linux control groups.

Yes, that is true. Though if you are using Google's or Amazon's managed Kubernetes services, I think you need to use Docker.

Certainly EKS is less managed than it appears, I’m quite confident that a node running something implementing the kubelet API convincingly would work. They changed recently (1.23 maybe) from Docker to containerd.

Re: Go, Containers, and the Linux Scheduler

#118

Post by nurple »

mholt wrote: post #66
marcus_holmes wrote: post #60

I still don't get the benefit of running Go binaries in containers. Totally get it for Rails, Python, etc, where the program needs a consistent set of supporting libraries. But Go doesn't need that. Especially now we can embed whole file systems into the actual binary. I've been looking at going in the other direction and using a Go binary with a unikernel; the machine just runs the binary and nothing else. I haven't…

As the author of Caddy, I have often wondered why people run it in containers. The feedback I keep hearing is basically workflow/ecosystem lock-in. Everything else uses containers, so the stuff that doesn't need containers needs them now, too.

I find putting http proxies in containers to be a very effective method of building interesting dynamic L7 dataplanes on orchestrators like k8s. Packaging applications (particularly modern static SPAs), with a webserver embedded, is also a very intuitive way of plugging them into the leaves of this topology while abstracting away a lot of the connectivity policy from the app.

Of course there's also the well-known correlation between the quality of your k8s deployment and the number of proxies it hosts. /s

Re: Go, Containers, and the Linux Scheduler

#119

Post by nurple »

marcus_holmes wrote: post #74
throwaway892238 wrote: post #70

Earlier quoted context omitted.

Here's what a container gives you: - Isolation of networking - Isolation of process namespace - Isolation of filesystem namespace - CPU and Memory limit enforcement - A near-universal format for packaging, distributing, and running applications, with metadata, and support for multiple architectures - A simple method for declaring a multi-step build process - Other stuff I don't remember You're certainly welcome to go…

A VM provides the first 4 anyway - if you're deploying to a cloud instance then having these in the container is redundant. If you're deploying to bare metal then it's possibly useful, but only if you're deploying multiple containers to the same machine. Go doesn't need a format for packaging - it's one file. It's becoming common practice to embed everything else into the binary. (side note: I haven't done this with…

This goes against everything we've learned about effectively deploying and managing software at runtime. Using the golang binary as a packaging format for your app has the same energy as crafting it exclusively from impenetrable one-liners.

Re: Go, Containers, and the Linux Scheduler

#120

Post by jeffbee »

n3t wrote: post #84
jeffbee wrote: post #82

Earlier quoted context omitted.

Perhaps it is an artifact of your and my various container runtimes. For me, in a guaranteed qos pod, taskset shows just 1 visible CPU for a Guaranteed QoS pod with limit=request=1. # taskset -c -p 1 pid 1's current affinity list: 1 # nproc 1 I honestly do not see how it can work otherwise.

After reading https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana... , I think we have different policies set for the CPU Manager. In my case it's `"cpuManagerPolicy": "none"` and I suppose you're using `"static"` policy. Well, TIL. Thanks!

TIL also. The difference between guaranteed and burstable seems meaningless without this setting.
Post reply on HN