Introduction to Golang Preemption Mechanisms
unskilled.blog
Introduction to Golang Preemption Mechanisms
1–10 of 30 posts
Re: Introduction to Golang Preemption Mechanisms
#2Interesting that it’s temporal (according to the article, you have around 10 microseconds before the signal-based preempter kicks in). How bad is performance if the load on the host is so high that double-preempting is common, I wonder? Or am I missing something and that question is not meaningful?
Re: Introduction to Golang Preemption Mechanisms
#3Interesting that it’s temporal (according to the article, you have around 10 microseconds before the signal-based preempter kicks in). How bad is performance if the load on the host is so high that double-preempting is common, I wonder? Or am I missing something and that question is not meaningful?
No it’s an interesting comment. This is not really about load, but about control flow: if goroutine is just spinning wild without going through any function prologue, it won’t even be aware of the synchronous preemption request. Asynchronous preemption (signal-based) is mainly (I say “mainly” because I am not sure I can say “only”) for this kind of situation.
I don’t have the link ready, but twitch had this kind of issue with base64 decoding in some kind of servers. The GC would try to STW, but there would always be one or a few goroutines decoding base64 in a tight loop for the time STW was attempted, delaying it again and again.
Asynchronous preemption is a solution to this kind of issue. Load is not the issue here, as long as you go through the runtime often enough.
Re: Introduction to Golang Preemption Mechanisms
#4Are there any proposals to make the golang runtime cgroup aware? Last time I checked the go runtime will spawn a OS process for each cpu it can see even if it is running in a cgroup which only allows 1 CPU of usage.
On servers with 100+ cores I have seen scheduling time take over 10% of the program runtime.
The fix is to inspect the cgroupfs to see how many CPU shares you can utilize and then set gomaxprocs to match that. I think other runtime like Java and .NET do this automatically.
It is the same thing with GOMEMLIMIT, I don’t see why the runtime does not inspect cgroupfs and set GOMEMLIMIT to 90% of the cgroup memory limit.
Re: Introduction to Golang Preemption Mechanisms
#5Are there any proposals to make the golang runtime cgroup aware? Last time I checked the go runtime will spawn a OS process for each cpu it can see even if it is running in a cgroup which only allows 1 CPU of usage. On servers with 100+ cores I have seen scheduling time take over 10% of the program runtime. The fix is to inspect the cgroupfs to see how many CPU shares you can utilize and then set gomaxprocs to match…
I am guessing the API isn't stable enough for letting the runtime set maxprocs. I use https://pkg.go.dev/go.uber.org/automaxprocs and have had to update it periodically because Redhat and Debian have different defaults. (Should one even run k8s on Redhat? I say no, but Redhat says yes. That's how I know about this.)
This, I think, is cgroups 1 vs. cgroups 2 and everyone should have cgroups 2 now, but ... it would feel weird for the Go runtime to decide on one. To me, anyway.
Re: Introduction to Golang Preemption Mechanisms
#6Are there any proposals to make the golang runtime cgroup aware? Last time I checked the go runtime will spawn a OS process for each cpu it can see even if it is running in a cgroup which only allows 1 CPU of usage. On servers with 100+ cores I have seen scheduling time take over 10% of the program runtime. The fix is to inspect the cgroupfs to see how many CPU shares you can utilize and then set gomaxprocs to match…
On Linux, go uses sched_getaffinity to know how many cpu core it is allowed to run on:
https://cs.opensource.google/go/go/+/master:src/runtime/os_l...
Re: Introduction to Golang Preemption Mechanisms
#7Are there any proposals to make the golang runtime cgroup aware? Last time I checked the go runtime will spawn a OS process for each cpu it can see even if it is running in a cgroup which only allows 1 CPU of usage. On servers with 100+ cores I have seen scheduling time take over 10% of the program runtime. The fix is to inspect the cgroupfs to see how many CPU shares you can utilize and then set gomaxprocs to match…
I am guessing the API isn't stable enough for letting the runtime set maxprocs. I use https://pkg.go.dev/go.uber.org/automaxprocs and have had to update it periodically because Redhat and Debian have different defaults. (Should one even run k8s on Redhat? I say no, but Redhat says yes. That's how I know about this.) This, I think, is cgroups 1 vs. cgroups 2 and everyone should have cgroups 2 now, but ... it would fee…
Which API is not stable? Cgroupfs?
I would think that cgroupfs is considered an API to userspace and therefore it shouldn’t break in the future? Hence creating cgroups v2?
I have written code which handles both cgroups v1 and cgroups v2, it isn’t terribly hard. Golang could also only support setting automatic parameters when running in cgroups v2 if that made things easier.
For a language that prides itself in sane defaults I think they have missed the mark here. I could probably add support to the golang runtime in a few hundred lines of code and probably save millions of dollars and megawatts of energy because the go runtime is not spawning 50 processes to run a program which is constrained to 1 core.
Re: Introduction to Golang Preemption Mechanisms
#8Great post! One question that lingered for me is: what are asynchronous safe-points? The post goes into some detail about their synchronous counterparts
Re: Introduction to Golang Preemption Mechanisms
#9Earlier quoted context omitted.
I am guessing the API isn't stable enough for letting the runtime set maxprocs. I use https://pkg.go.dev/go.uber.org/automaxprocs and have had to update it periodically because Redhat and Debian have different defaults. (Should one even run k8s on Redhat? I say no, but Redhat says yes. That's how I know about this.) This, I think, is cgroups 1 vs. cgroups 2 and everyone should have cgroups 2 now, but ... it would fee…
Which API is not stable? Cgroupfs? I would think that cgroupfs is considered an API to userspace and therefore it shouldn’t break in the future? Hence creating cgroups v2? I have written code which handles both cgroups v1 and cgroups v2, it isn’t terribly hard. Golang could also only support setting automatic parameters when running in cgroups v2 if that made things easier. For a language that prides itself in sane d…
I’m not 100% sold on the idea that Go’s defaults are sane.
They’re highly opinionated and not really that intuitive.
Re: Introduction to Golang Preemption Mechanisms
#10Are there any proposals to make the golang runtime cgroup aware? Last time I checked the go runtime will spawn a OS process for each cpu it can see even if it is running in a cgroup which only allows 1 CPU of usage. On servers with 100+ cores I have seen scheduling time take over 10% of the program runtime. The fix is to inspect the cgroupfs to see how many CPU shares you can utilize and then set gomaxprocs to match…
[deleted]