Introduction to Golang Preemption Mechanisms • Hacker News Board

Re: Introduction to Golang Preemption Mechanisms

#11

Post by throw-the-towel » Mon, Aug 12, 2024, 12:33 AM UTC

ljm wrote: post #9

__turbobrew__ wrote: post #7

Earlier quoted context omitted.

Which API is not stable? Cgroupfs? I would think that cgroupfs is considered an API to userspace and therefore it shouldn’t break in the future? Hence creating cgroups v2? I have written code which handles both cgroups v1 and cgroups v2, it isn’t terribly hard. Golang could also only support setting automatic parameters when running in cgroups v2 if that made things easier. For a language that prides itself in sane d…

I’m not 100% sold on the idea that Go’s defaults are sane. They’re highly opinionated and not really that intuitive.

Could you elaborate?

Re: Introduction to Golang Preemption Mechanisms

#12

Post by metadat » Mon, Aug 12, 2024, 3:29 AM UTC

> think about it - what if I suddenly stopped you while taking a dump? It would have been easier to have stopped you before, or after, but not in the middle of it. Not sure about the analogy, but you got it.

Gold.

Re: Introduction to Golang Preemption Mechanisms

#13

Post by hiyer » Mon, Aug 12, 2024, 5:33 AM UTC

This is a well-written article, but one thing that wasn't clear to me was how the runtime determines that it's at a safe point. Can someone shed some light on that?

Re: Introduction to Golang Preemption Mechanisms

#14

Post by MathMonkeyMan » Mon, Aug 12, 2024, 9:04 AM UTC

hiyer wrote: post #13

This is a well-written article, but one thing that wasn't clear to me was how the runtime determines that it's at a safe point. Can someone shed some light on that?

The runtime never determines whether the goroutine is at a safe point. It "poisons" the stack guard so that the next time the goroutine reaches a function prologue, which is a safe point, it examines the stack guard and knows that it has been preempted.

Then there's the async case for tight loops that I remember reading about back in 2020 (it uses unix signals), but don't yet fully grok the specifics.

Re: Introduction to Golang Preemption Mechanisms

#15

Post by MathMonkeyMan » Mon, Aug 12, 2024, 9:09 AM UTC

ollien wrote: post #8

Great post! One question that lingered for me is: what are asynchronous safe-points? The post goes into some detail about their synchronous counterparts

I don't know, but I remember hearing in a talk that the compiler had to be modified to insert them into the generated code.

Here's `isAsyncSafePoint`: https://github.com/golang/go/blob/d36353499f673c89a267a489be...

edit: The comments at the top of that file say:

    // 3. Asynchronous safe-points occur at any instruction in user code
    //    where the goroutine can be safely paused and a conservative
    //    stack and register scan can find stack roots. The runtime can
    //    stop a goroutine at an async safe-point using a signal.

Re: Introduction to Golang Preemption Mechanisms

#16

Post by EdSchouten » Mon, Aug 12, 2024, 11:03 AM UTC

__turbobrew__ wrote: post #4

Are there any proposals to make the golang runtime cgroup aware? Last time I checked the go runtime will spawn a OS process for each cpu it can see even if it is running in a cgroup which only allows 1 CPU of usage. On servers with 100+ cores I have seen scheduling time take over 10% of the program runtime. The fix is to inspect the cgroupfs to see how many CPU shares you can utilize and then set gomaxprocs to match…

If you’re on Kubernetes, you can solve this/work around this by enabling the static CPU manager policy:

https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...

Re: Introduction to Golang Preemption Mechanisms

#17

Post by eadmund » Mon, Aug 12, 2024, 12:54 PM UTC

lcof wrote: post #6

__turbobrew__ wrote: post #4

Are there any proposals to make the golang runtime cgroup aware? Last time I checked the go runtime will spawn a OS process for each cpu it can see even if it is running in a cgroup which only allows 1 CPU of usage. On servers with 100+ cores I have seen scheduling time take over 10% of the program runtime. The fix is to inspect the cgroupfs to see how many CPU shares you can utilize and then set gomaxprocs to match…

On Linux, go uses sched_getaffinity to know how many cpu core it is allowed to run on: https://cs.opensource.google/go/go/+/master:src/runtime/os_l...

> > On servers with 100+ cores I have seen scheduling time take over 10% of the program runtime.

> On Linux, go uses sched_getaffinity …

Since cgroups are a Linux-only feature, OP must be running Linux. I wonder if his experience pre-dates Go’s usage of sched_getaffinity.

edit: I realised that he references cgroups so must be on Linux.

Re: Introduction to Golang Preemption Mechanisms

#18

Post by Zimnx » Mon, Aug 12, 2024, 1:24 PM UTC

EdSchouten wrote: post #16

__turbobrew__ wrote: post #4

Are there any proposals to make the golang runtime cgroup aware? Last time I checked the go runtime will spawn a OS process for each cpu it can see even if it is running in a cgroup which only allows 1 CPU of usage. On servers with 100+ cores I have seen scheduling time take over 10% of the program runtime. The fix is to inspect the cgroupfs to see how many CPU shares you can utilize and then set gomaxprocs to match…

If you’re on Kubernetes, you can solve this/work around this by enabling the static CPU manager policy: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...

No, 'static' CPU manager policy provides ability to allocate CPUs exclusively to container cgroup. But since Go runtime doesn't read cpugroup information anyway, it still sees all available CPUs.

Re: Introduction to Golang Preemption Mechanisms

#19

Post by jerf » Mon, Aug 12, 2024, 1:31 PM UTC

zbentley wrote: post #2

Interesting that it’s temporal (according to the article, you have around 10 microseconds before the signal-based preempter kicks in). How bad is performance if the load on the host is so high that double-preempting is common, I wonder? Or am I missing something and that question is not meaningful?

You'll actually see that's a general concurrency pattern, and I mean, far beyond Go. It is certainly ideal to trigger off of some signal (in a very, very generic sense of the term, not OS signal) in order to trigger some other process (again in a very generic sense), but in general if you're programming concurrent code you should always have some sort of time-based fallback for any wait you are doing, because you will hit it out in the field. It's all kinds of no fun to have processes that will wait forever. (Unless you're really sure you need it.)

Re: Introduction to Golang Preemption Mechanisms

#20

Post by Thaxll » Mon, Aug 12, 2024, 1:41 PM UTC

lcof wrote: post #6

__turbobrew__ wrote: post #4

Are there any proposals to make the golang runtime cgroup aware? Last time I checked the go runtime will spawn a OS process for each cpu it can see even if it is running in a cgroup which only allows 1 CPU of usage. On servers with 100+ cores I have seen scheduling time take over 10% of the program runtime. The fix is to inspect the cgroupfs to see how many CPU shares you can utilize and then set gomaxprocs to match…

On Linux, go uses sched_getaffinity to know how many cpu core it is allowed to run on: https://cs.opensource.google/go/go/+/master:src/runtime/os_l...

This is not group aware.