Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go, Containers, and the Linux Scheduler

riverphillips.dev

Post reply on HN
Print view View original

21–30 of 143 posts

Re: Go, Containers, and the Linux Scheduler

#21

Post by dharmab »

dekhn wrote: post #16

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other container technology), that file looks the same as the container host (listing all cores, regardless of how few have been assigned to the container). I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpu…

Point of clarification: Containers, when using quota based limits, can use all of the CPU cores on the host. They're limited in how much time they can spend using them.

(There are exceptions, such as documented here: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...)

Re: Go, Containers, and the Linux Scheduler

#22

Post by jeffbee »

dekhn wrote: post #16

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other container technology), that file looks the same as the container host (listing all cores, regardless of how few have been assigned to the container). I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpu…

That's not what Go does though. Go looks at the population of the CPU mask at startup. It never looks again, which of problematic in K8s where the visible CPUs may change while your process runs.

Re: Go, Containers, and the Linux Scheduler

#23

Post by jeffbee »

gregfurman wrote: post #5

Discovered this sometime last year in my previous role as a platform engineer managing our on-prem kubernetes cluster as well as the CI/CD pipeline infrastructure. Although I saw this dissonance between actual and assigned CPU causing issues, particularly CPU throttling, I struggled to find a scalable solution that would affect all Go deployments on the cluster. Getting all devs to include that autoprocs dependency w…

There isn't one answer for this. Capping GOMAXPROCS may cause severe latency problems if your process gets a burst of traffic and has naive queueing. It's best really to set GOMAXPROCS to whatever the hardware offers regardless of your ideas about how much time the process will use on average.

Re: Go, Containers, and the Linux Scheduler

#24

Post by dekhn »

jeffbee wrote: post #22
dekhn wrote: post #16

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other container technology), that file looks the same as the container host (listing all cores, regardless of how few have been assigned to the container). I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpu…

That's not what Go does though. Go looks at the population of the CPU mask at startup. It never looks again, which of problematic in K8s where the visible CPUs may change while your process runs.

What is the population of the CPU mask at startup? Is this a kernel call? A /proc file? Some register?

Re: Go, Containers, and the Linux Scheduler

#25

Post by EdSchouten »

dekhn wrote: post #24
jeffbee wrote: post #22

Earlier quoted context omitted.

That's not what Go does though. Go looks at the population of the CPU mask at startup. It never looks again, which of problematic in K8s where the visible CPUs may change while your process runs.

What is the population of the CPU mask at startup? Is this a kernel call? A /proc file? Some register?

On Linux, it likely calls sched_getaffinity().

Re: Go, Containers, and the Linux Scheduler

#26

Post by dekhn »

dharmab wrote: post #21
dekhn wrote: post #16

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other container technology), that file looks the same as the container host (listing all cores, regardless of how few have been assigned to the container). I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpu…

Point of clarification: Containers, when using quota based limits, can use all of the CPU cores on the host. They're limited in how much time they can spend using them. (There are exceptions, such as documented here: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana... )

Maybe I should be clearer: Let's say I have a 16 core host and I start a flask container with cpu=0.5 that forks and has a heavy post-fork initializer.

flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that).

I can solve this by passing a flag to my application; my complaint is more that apps shouldn't consult /proc/cpuinfo, but have another standard interface to ask "what should I set my max parallelism (NOT CONCURRENCY, ROB) so my worker threads get adequate CPU time so the framework doesn't time out on startup.

Re: Go, Containers, and the Linux Scheduler

#27

Post by linuxftw »

gregfurman wrote: post #5

Discovered this sometime last year in my previous role as a platform engineer managing our on-prem kubernetes cluster as well as the CI/CD pipeline infrastructure. Although I saw this dissonance between actual and assigned CPU causing issues, particularly CPU throttling, I struggled to find a scalable solution that would affect all Go deployments on the cluster. Getting all devs to include that autoprocs dependency w…

You could define a mutating webhook to inject GOMAXPROCS into all pod containers.

Re: Go, Containers, and the Linux Scheduler

#28

Post by dekhn »

EdSchouten wrote: post #25
dekhn wrote: post #24

Earlier quoted context omitted.

What is the population of the CPU mask at startup? Is this a kernel call? A /proc file? Some register?

On Linux, it likely calls sched_getaffinity().

hmm, I can see that as being useful but I also don't see that as the way to determine "how many worker threads I should start"
Post reply on HN