Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go, Containers, and the Linux Scheduler

riverphillips.dev

Post reply on HN
Print view View original

51–60 of 143 posts

Re: Go, Containers, and the Linux Scheduler

#51

Post by rrdharan »

dekhn wrote: post #16

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other container technology), that file looks the same as the container host (listing all cores, regardless of how few have been assigned to the container). I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpu…

Containers are a crappy abstraction and VMware fumbled the bag, is my takeaway from this comment…

Re: Go, Containers, and the Linux Scheduler

#52

Post by telotortium »

wutwutwat wrote: post #49
dekhn wrote: post #16

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other container technology), that file looks the same as the container host (listing all cores, regardless of how few have been assigned to the container). I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpu…

I only use `nproc` and see it used in other containers as well, ie `bundle install -j $(nproc)`. This honors cpu assignment and provides the functionality you're seeking. Whether or not random application software uses nproc if available, idk > Print the number of processing units available to the current process, which may be less than the number of online processors. If this information is not accessible, then prin…

This is not very robust. You probably should use the cgroup cpu limits where present, since `docker --cpus` uses a different way to set quota:

    if [[ -e /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]] && [[ -e /sys/fs/cgroup/cpu/cpu.cfs_period_us ]]; then
        GOMAXPROCS=$(perl -e 'use POSIX; printf "%d\n", ceil($ARGV[0] / $ARGV[1])' "$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us)" "$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us)")
    else
        GOMAXPROCS=$(nproc)
    fi
    export GOMAXPROCS
This follows from how `docker --cpus` works (https://docs.docker.com/config/containers/resource_constrain...), as well as https://stackoverflow.com/a/65554131/207384 to get the /sys paths to read from.

Or use https://github.com/uber-go/automaxprocs, which is very comprehensive, but is a bunch of code for what should be a simple task.

Re: Go, Containers, and the Linux Scheduler

#53

Post by jeffbee »

sethammons wrote: post #45
jeffbee wrote: post #22

Earlier quoted context omitted.

That's not what Go does though. Go looks at the population of the CPU mask at startup. It never looks again, which of problematic in K8s where the visible CPUs may change while your process runs.

We use https://github.com/uber-go/automaxprocs after we joyfully discovered that Go assumed we had the entire cluster's cpu count on any particular pod. Made for some very strange performance characteristics in scheduling goroutines.

My opinion is that setting GOMAXPROCS that way is a quite poor idea. It tends to strand resources that could have been used to handle a stochastic burst of requests, which with a capped GOMAXPROCS will be converted directly into latency. I can think of no good reason why GOMAXPROCS needs to be 2 just because you expect the long-term CPU rate to be 2. That long-term quota is an artifact of capacity planning, while GOMAXPROCS is an artifact of process architecture.

Re: Go, Containers, and the Linux Scheduler

#54

Post by jeffbee »

n3t wrote: post #47
jeffbee wrote: post #22

Earlier quoted context omitted.

That's not what Go does though. Go looks at the population of the CPU mask at startup. It never looks again, which of problematic in K8s where the visible CPUs may change while your process runs.

> which of problematic in K8s where the visible CPUs may change while your process runs This is new to me. What is this… behavior? What keywords should I use to find any details about it? The only thing that rings a bell is requests/limit parameters of a pod but you can't change them on an existing pod AFAIK.

If you have one pod that has Burstable QoS, perhaps because it has a request and not a limit, its CPU mask will be populated by every CPU on the box, less one for the Kubelet and other node services, less all the CPUs requested by pods with Guaranteed QoS. Pods with Guaranteed QoS will have exactly the number of CPUs they asked for, no more or less, and consequently their GOMAXPROCS is consistent. Everyone else will see fewer or more CPUs as Guaranteed pods arrive and depart from the node.

Re: Go, Containers, and the Linux Scheduler

#55

Post by dilyevsky »

the8472 wrote: post #41
dilyevsky wrote: post #36

Earlier quoted context omitted.

It doesn’t make any sense to me why —cpu flag is tweaking quota and not shares since quota is useful in tiny minority of usecases. A lot of people waste a ton of time debugging weird latency issues as a result of this decision

With shares you're going to experience worse latency if all the containers on the system size their thread pool to the maximum that's available during idle periods and then constantly context-switch due to oversubscription under load. With quotas you can do fixed resource allocation and the runtimes (not Go apparently) can fit themselves into that and not try to service more requests than they can currently execute g…

And how is that different from worse latency due to cpu throttling from your users’ perspective?

Re: Go, Containers, and the Linux Scheduler

#56

Post by jeffbee »

loxias wrote: post #44
jeffbee wrote: post #35

Earlier quoted context omitted.

Yes, running on a set of heterogenous CPUs presents further challenges, for the program and the thread scheduler. Happily there are no such systems in the cloud, yet. Most people are running on systems where the CPU capacity varies and they haven't even noticed. For example in EC2 there are 8 victim CPUs that handle all the network interrupts, so if you have an instance type with 32 CPUs, you already have 24 that are…

> in EC2 there are 8 victim CPUs that handle all the network interrupts, so if you have an instance type with 32 CPUs, you already have 24 that are faster than the others Fascinating. Could you share any (all) more detail on this that you know? Is it a specific instance type, only ones that use nitro? (or only ones without?) This might be related to a problem I've seen in the wild but never tracked down...

I've only observed it on Nitro, but I have also rarely used pre-Nitro instances.

Re: Go, Containers, and the Linux Scheduler

#57

Post by djbusby »

n3t wrote: post #47
jeffbee wrote: post #22

Earlier quoted context omitted.

That's not what Go does though. Go looks at the population of the CPU mask at startup. It never looks again, which of problematic in K8s where the visible CPUs may change while your process runs.

> which of problematic in K8s where the visible CPUs may change while your process runs This is new to me. What is this… behavior? What keywords should I use to find any details about it? The only thing that rings a bell is requests/limit parameters of a pod but you can't change them on an existing pod AFAIK.

Even way back in the day (1996) it was possible to hot-swap a CPU. Used to have this Sequent box, 96 Pentiums in there, 6 on a card. Could do some magic, pull the card and swap a new one in. Wild. And no processes died. Not sure if a process could lose a CPU then discover the new set.

Re: Go, Containers, and the Linux Scheduler

#58

Post by cpuguy83 »

dilyevsky wrote: post #36
cpuguy83 wrote: post #17

Earlier quoted context omitted.

> "Don't use Docker's --cpu flag and instead use" This is rather strong language without any real qualifiers. It is definitely not "mostly useless". Shares and quotas are for different use-cases, that's all. Understand your use-case and choose accordingly.

It doesn’t make any sense to me why —cpu flag is tweaking quota and not shares since quota is useful in tiny minority of usecases. A lot of people waste a ton of time debugging weird latency issues as a result of this decision

These two options are not mutually exclusive.

When you want to limit the max CPU time available to a container use quotas (--cpus). When you want to set relative priorities (compared to other containers/processes), use shares.

These two options can be combined, it all depends on what you need.

Re: Go, Containers, and the Linux Scheduler

#59

Post by dekhn »

vbezhenar wrote: post #43
dekhn wrote: post #26

Earlier quoted context omitted.

Maybe I should be clearer: Let's say I have a 16 core host and I start a flask container with cpu=0.5 that forks and has a heavy post-fork initializer. flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that). I can solve this by passing a flag to my application…

You generally shouldn't set CPU limits. You might want to configure CPU requests which is guaranteed chunk of CPU time that container will always receive. With CPU limits you'll encounter situation when host CPU is not loaded, but your container workloaded is throttled at the same time, which is just waste of CPU resources.

It's complicated. I've worked on every kind of application in a container environment: ones that ran at ultra-low priority while declaring zero CPU request and infinite CPU limit. I ran one or a few of these on nearly every machine in Google production for over a year, and could deliver over 1M xeon cores worth of throughput for embarassingly parallel jobs. At other times, I ran jobs that asked for and used precisely all the cores on a machine (a TPU host), specifically setting limits and requests to get the most predictable behavior.

The true objective function I'm trying to optimize isn't just "save money" or "don't waste CPU resources", but rather "get a million different workloads to run smoothly on a large collection of resources, ensuring that revenue-critical jobs always can run, while any spare capacity is available for experimenters, up to some predefined limits determined by power capacity, staying within the overall budget, and not pissing off any really powerful users." (well, that's really just a simplified approximation)

Re: Go, Containers, and the Linux Scheduler

#60

Post by marcus_holmes »

I still don't get the benefit of running Go binaries in containers. Totally get it for Rails, Python, etc, where the program needs a consistent set of supporting libraries. But Go doesn't need that. Especially now we can embed whole file systems into the actual binary.

I've been looking at going in the other direction and using a Go binary with a unikernel; the machine just runs the binary and nothing else. I haven't got this working to my satisfaction yet - it works but there's still too much infrastructure, and deployment is still "fun". But I think this is way more interesting for Go deployments than using containers.

Post reply on HN