Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduction to Golang Preemption Mechanisms

unskilled.blog

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Introduction to Golang Preemption Mechanisms

#21

Post by kbolino »

__turbobrew__ wrote: post #7
jrockway wrote: post #5

Earlier quoted context omitted.

I am guessing the API isn't stable enough for letting the runtime set maxprocs. I use https://pkg.go.dev/go.uber.org/automaxprocs and have had to update it periodically because Redhat and Debian have different defaults. (Should one even run k8s on Redhat? I say no, but Redhat says yes. That's how I know about this.) This, I think, is cgroups 1 vs. cgroups 2 and everyone should have cgroups 2 now, but ... it would fee…

Which API is not stable? Cgroupfs? I would think that cgroupfs is considered an API to userspace and therefore it shouldn’t break in the future? Hence creating cgroups v2? I have written code which handles both cgroups v1 and cgroups v2, it isn’t terribly hard. Golang could also only support setting automatic parameters when running in cgroups v2 if that made things easier. For a language that prides itself in sane d…

The OpenJDK folks have quite a long and storied history of trying to do this right and still generally recommend that if you want a JVM to have the right number of CPUs, you should set the relevant parameter yourself (-XX:ActiveProcessorCount). This is basically the same advice as Go folks telling you to set GOMAXPROCS yourself.

The problem is not just cgroups v1 vs cgroups v2 or the stability of cgroupfs, but also of CPU "shares" vs "limits", the different tunables for different Linux schedulers, the effective limits under hierarchical cgroups, etc.

Re: Introduction to Golang Preemption Mechanisms

#22

Post by lcof »

Thaxll wrote: post #20
lcof wrote: post #6

Earlier quoted context omitted.

On Linux, go uses sched_getaffinity to know how many cpu core it is allowed to run on: https://cs.opensource.google/go/go/+/master:src/runtime/os_l...

This is not group aware.

if you want to limit the number of Ps, then you use a cpuset, that sched_getaffinity will take into account. cgroups only allows you to limit cpu usage, but not lower the number of cpu cores the code can run on. This is “how many” versus “how much”, and GOMAXPROCS only relates to the “how many” part.

I may have misunderstood the rationale here, but I think the discussion about cgroup support is not about limiting the number of Ps

Re: Introduction to Golang Preemption Mechanisms

#23

Post by kbolino »

lcof wrote: post #22
Thaxll wrote: post #20

Earlier quoted context omitted.

This is not group aware.

if you want to limit the number of Ps, then you use a cpuset, that sched_getaffinity will take into account. cgroups only allows you to limit cpu usage, but not lower the number of cpu cores the code can run on. This is “how many” versus “how much”, and GOMAXPROCS only relates to the “how many” part. I may have misunderstood the rationale here, but I think the discussion about cgroup support is not about limiting the…

What people want is that, if cgroup limits prevent a container from using more than M/N of CPU time (N the number of cores), then GOMAXPROCS defaults to M. Ditto other managed language runtimes and their equivalent parameters.

However, as far as I can tell, there's no clear way to figure out what M is, in the general case.

Re: Introduction to Golang Preemption Mechanisms

#24

Post by lcof »

kbolino wrote: post #23
lcof wrote: post #22

Earlier quoted context omitted.

if you want to limit the number of Ps, then you use a cpuset, that sched_getaffinity will take into account. cgroups only allows you to limit cpu usage, but not lower the number of cpu cores the code can run on. This is “how many” versus “how much”, and GOMAXPROCS only relates to the “how many” part. I may have misunderstood the rationale here, but I think the discussion about cgroup support is not about limiting the…

What people want is that, if cgroup limits prevent a container from using more than M/N of CPU time (N the number of cores), then GOMAXPROCS defaults to M. Ditto other managed language runtimes and their equivalent parameters. However, as far as I can tell, there's no clear way to figure out what M is, in the general case.

Again, I might be wrong as I did not use this directly in a couple years, but saying “the limit is 50% share of 10 cores” is not equivalent to “the limit is 5 cores”. This is still “how much” versus “how many”, and cannot translate into each other without sacrificing flexibility

Re: Introduction to Golang Preemption Mechanisms

#25

Post by kbolino »

lcof wrote: post #24
kbolino wrote: post #23

Earlier quoted context omitted.

What people want is that, if cgroup limits prevent a container from using more than M/N of CPU time (N the number of cores), then GOMAXPROCS defaults to M. Ditto other managed language runtimes and their equivalent parameters. However, as far as I can tell, there's no clear way to figure out what M is, in the general case.

Again, I might be wrong as I did not use this directly in a couple years, but saying “the limit is 50% share of 10 cores” is not equivalent to “the limit is 5 cores”. This is still “how much” versus “how many”, and cannot translate into each other without sacrificing flexibility

GOMAXPROCS sets the number of live system threads used to run goroutines. The distinction between 50% of time on 10 cores and 100% of time on 5 cores doesn't really matter here: the recommendation is to set GOMAXPROCS=5 in both cases.

Re: Introduction to Golang Preemption Mechanisms

#26

Post by __turbobrew__ »

Zimnx wrote: post #18
EdSchouten wrote: post #16

Earlier quoted context omitted.

If you’re on Kubernetes, you can solve this/work around this by enabling the static CPU manager policy: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...

No, 'static' CPU manager policy provides ability to allocate CPUs exclusively to container cgroup. But since Go runtime doesn't read cpugroup information anyway, it still sees all available CPUs.

Correct

Re: Introduction to Golang Preemption Mechanisms

#27

Post by trws »

lcof wrote: post #22
Thaxll wrote: post #20

Earlier quoted context omitted.

This is not group aware.

if you want to limit the number of Ps, then you use a cpuset, that sched_getaffinity will take into account. cgroups only allows you to limit cpu usage, but not lower the number of cpu cores the code can run on. This is “how many” versus “how much”, and GOMAXPROCS only relates to the “how many” part. I may have misunderstood the rationale here, but I think the discussion about cgroup support is not about limiting the…

I think your comment was once completely correct, but there is now also a “cpuset” cgroup in addition to the classic cpu setting. The cpuset control gives something equivalent to sched_setaffinity but stronger since the client processes can’t unset parts of the mask or override it IIRC.

Re: Introduction to Golang Preemption Mechanisms

#29

Post by EdSchouten »

Zimnx wrote: post #18
EdSchouten wrote: post #16

Earlier quoted context omitted.

If you’re on Kubernetes, you can solve this/work around this by enabling the static CPU manager policy: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...

No, 'static' CPU manager policy provides ability to allocate CPUs exclusively to container cgroup. But since Go runtime doesn't read cpugroup information anyway, it still sees all available CPUs.

Static CPU manager also affects the cores that sched_get affinity returns. And that’s what Go uses to obtain the core count.

Re: Introduction to Golang Preemption Mechanisms

#30

Post by __turbobrew__ »

EdSchouten wrote: post #29
Zimnx wrote: post #18

Earlier quoted context omitted.

No, 'static' CPU manager policy provides ability to allocate CPUs exclusively to container cgroup. But since Go runtime doesn't read cpugroup information anyway, it still sees all available CPUs.

Static CPU manager also affects the cores that sched_get affinity returns. And that’s what Go uses to obtain the core count.

That is only true if the pod is running within the guaranteed runtime class (requests==limits). For pods where requests!=limits a common set of cpus are used for all burstable pods, otherwise bursting past requests would not work.

This still allows the worst case where a node with 100 cpus running butstable pods will still see huge overheads in the golang scheduling runtime.

To my knowledge (I have done a lot of research into not only runc but also gvisor) there is no way to have the go runtime and cgroups interact in a sane way currently by default.

If the golang runtime was cgroup aware I do believe it is possible to have sane defaults, especially since the JVM and CLR have done so.

Post reply on HN