Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go, Containers, and the Linux Scheduler

riverphillips.dev

Post reply on HN
Print view View original

41–50 of 143 posts

Re: Go, Containers, and the Linux Scheduler

#41

Post by the8472 »

dilyevsky wrote: post #36
cpuguy83 wrote: post #17

Earlier quoted context omitted.

> "Don't use Docker's --cpu flag and instead use" This is rather strong language without any real qualifiers. It is definitely not "mostly useless". Shares and quotas are for different use-cases, that's all. Understand your use-case and choose accordingly.

It doesn’t make any sense to me why —cpu flag is tweaking quota and not shares since quota is useful in tiny minority of usecases. A lot of people waste a ton of time debugging weird latency issues as a result of this decision

With shares you're going to experience worse latency if all the containers on the system size their thread pool to the maximum that's available during idle periods and then constantly context-switch due to oversubscription under load. With quotas you can do fixed resource allocation and the runtimes (not Go apparently) can fit themselves into that and not try to service more requests than they can currently execute given those resources.

Re: Go, Containers, and the Linux Scheduler

#42

Post by Volundr »

dekhn wrote: post #26
dharmab wrote: post #21

Earlier quoted context omitted.

Point of clarification: Containers, when using quota based limits, can use all of the CPU cores on the host. They're limited in how much time they can spend using them. (There are exceptions, such as documented here: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana... )

Maybe I should be clearer: Let's say I have a 16 core host and I start a flask container with cpu=0.5 that forks and has a heavy post-fork initializer. flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that). I can solve this by passing a flag to my application…

It's not clear to me what the max parallelism should actually be on a container with a CPU limit of .5. To my understanding that limits CPU time the container can use within a certain time interval, but doesn't actually limit the parallel processes an application can run. In other words that container with .5 on the CPU limit can indeed use all 16 physical cores of that machine. It'll just burn through it's budget 16x faster. If that's desirable vs limiting itself to one process is going to be highly application dependent and not something kubernetes and docker can just tell you.

Re: Go, Containers, and the Linux Scheduler

#43

Post by vbezhenar »

dekhn wrote: post #26
dharmab wrote: post #21

Earlier quoted context omitted.

Point of clarification: Containers, when using quota based limits, can use all of the CPU cores on the host. They're limited in how much time they can spend using them. (There are exceptions, such as documented here: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana... )

Maybe I should be clearer: Let's say I have a 16 core host and I start a flask container with cpu=0.5 that forks and has a heavy post-fork initializer. flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that). I can solve this by passing a flag to my application…

You generally shouldn't set CPU limits. You might want to configure CPU requests which is guaranteed chunk of CPU time that container will always receive. With CPU limits you'll encounter situation when host CPU is not loaded, but your container workloaded is throttled at the same time, which is just waste of CPU resources.

Re: Go, Containers, and the Linux Scheduler

#44

Post by loxias »

jeffbee wrote: post #35
dekhn wrote: post #32

Earlier quoted context omitted.

There's going to be a bunch of missing info, though, in some cases I can think of. For example, more and more systems have asymmetric cores. /proc/cpuinfo can expose that information in detail, including (current) clock speed, processor type, etc, while cpu_set is literally just a bitmask (if I read the man pages right) of system cores your process is allowed to schedule on. Fundamentally, intelligent apps need to in…

Yes, running on a set of heterogenous CPUs presents further challenges, for the program and the thread scheduler. Happily there are no such systems in the cloud, yet. Most people are running on systems where the CPU capacity varies and they haven't even noticed. For example in EC2 there are 8 victim CPUs that handle all the network interrupts, so if you have an instance type with 32 CPUs, you already have 24 that are…

> in EC2 there are 8 victim CPUs that handle all the network interrupts, so if you have an instance type with 32 CPUs, you already have 24 that are faster than the others

Fascinating. Could you share any (all) more detail on this that you know? Is it a specific instance type, only ones that use nitro? (or only ones without?) This might be related to a problem I've seen in the wild but never tracked down...

Re: Go, Containers, and the Linux Scheduler

#45

Post by sethammons »

jeffbee wrote: post #22
dekhn wrote: post #16

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other container technology), that file looks the same as the container host (listing all cores, regardless of how few have been assigned to the container). I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpu…

That's not what Go does though. Go looks at the population of the CPU mask at startup. It never looks again, which of problematic in K8s where the visible CPUs may change while your process runs.

We use https://github.com/uber-go/automaxprocs after we joyfully discovered that Go assumed we had the entire cluster's cpu count on any particular pod. Made for some very strange performance characteristics in scheduling goroutines.

Re: Go, Containers, and the Linux Scheduler

#46

Post by lovasoa »

vbezhenar wrote: post #43
dekhn wrote: post #26

Earlier quoted context omitted.

Maybe I should be clearer: Let's say I have a 16 core host and I start a flask container with cpu=0.5 that forks and has a heavy post-fork initializer. flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that). I can solve this by passing a flag to my application…

You generally shouldn't set CPU limits. You might want to configure CPU requests which is guaranteed chunk of CPU time that container will always receive. With CPU limits you'll encounter situation when host CPU is not loaded, but your container workloaded is throttled at the same time, which is just waste of CPU resources.

According to the article, this is not true. The limits become active only when the host cpu is under pressure.

Re: Go, Containers, and the Linux Scheduler

#47

Post by n3t »

jeffbee wrote: post #22
dekhn wrote: post #16

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other container technology), that file looks the same as the container host (listing all cores, regardless of how few have been assigned to the container). I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpu…

That's not what Go does though. Go looks at the population of the CPU mask at startup. It never looks again, which of problematic in K8s where the visible CPUs may change while your process runs.

> which of problematic in K8s where the visible CPUs may change while your process runs

This is new to me. What is this… behavior? What keywords should I use to find any details about it?

The only thing that rings a bell is requests/limit parameters of a pod but you can't change them on an existing pod AFAIK.

Re: Go, Containers, and the Linux Scheduler

#48

Post by vbezhenar »

lovasoa wrote: post #46
vbezhenar wrote: post #43

Earlier quoted context omitted.

You generally shouldn't set CPU limits. You might want to configure CPU requests which is guaranteed chunk of CPU time that container will always receive. With CPU limits you'll encounter situation when host CPU is not loaded, but your container workloaded is throttled at the same time, which is just waste of CPU resources.

According to the article, this is not true. The limits become active only when the host cpu is under pressure.

I don't think that's correct. --cpus is the same as --cpu-period which is cpu limit. You can easily check it yourself, just run docker container with --cpus set, run multi-core load there and check your activity monitor.

Re: Go, Containers, and the Linux Scheduler

#49

Post by wutwutwat »

dekhn wrote: post #16

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other container technology), that file looks the same as the container host (listing all cores, regardless of how few have been assigned to the container). I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpu…

I only use `nproc` and see it used in other containers as well, ie `bundle install -j $(nproc)`. This honors cpu assignment and provides the functionality you're seeking. Whether or not random application software uses nproc if available, idk

> Print the number of processing units available to the current process, which may be less than the number of online processors. If this information is not accessible, then print the number of processors installed

https://www.gnu.org/software/coreutils/manual/html_node/npro...

https://www.flamingspork.com/blog/2020/11/25/why-you-should-...

Re: Go, Containers, and the Linux Scheduler

#50

Post by wutwutwat »

dekhn wrote: post #26
dharmab wrote: post #21

Earlier quoted context omitted.

Point of clarification: Containers, when using quota based limits, can use all of the CPU cores on the host. They're limited in how much time they can spend using them. (There are exceptions, such as documented here: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana... )

Maybe I should be clearer: Let's say I have a 16 core host and I start a flask container with cpu=0.5 that forks and has a heavy post-fork initializer. flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that). I can solve this by passing a flag to my application…

`gunicorn --workers $(nproc)`, see my comment on the parent
Post reply on HN