Debugging an evil Go runtime bug: From heat guns to kernel compiler flags (2017)

You can follow Hector Martin @marcan at https://social.treehouse.systems/@marcan/

He works on Asahi Linux, a Linux port to arm64 Apple hardware.

> means that a 1x developer will take around 10 hours to get to the same place

A "workaround" isn't an adequate substitute for actual the understanding and fixing of the root cause of a bug.

What you think is a 10x developer is, in fact, a short-term 10x developer, medium-term 1x developer, long-term -10x developer. Their work while seemingly great at first is just accrued debt with an incredibly high interest rate. But they're rarely the ones fixing it.

Now, like everything, a balance needs to be struck between spending hours fixing a bug _the right way_, or, finding a temporary workaround. The real 10x developer is incredibly good at finding this balance.

I believe that Golang, as a green-threaded runtime, is allocating a separate carrier thread for running syscalls on, so that a blocking syscall won’t block the green-threads. These syscall carrier threads are allocated with a distinct initial stack size + stack size limit than green-thread-scheduler carrier threads are, because it’s expected that syscalls will always just enter kernel code (which has its own stack.)

But vDSOs don’t enter the kernel. They just run as userland code; and so they depend on the userland allocated stack to be arbitrarily deep in a way that kernel calls don’t.

As shown in the article, Golang seems to have code specifically for dealing with vDSO-type pseudo-syscalls — but this is likely a specialization of the pre-existing syscall-carrier-thread allocation code, and so started off with a bad assumption about how much stack should be allocated for the created threads.

(I should also point out that the OS stack size specified in the ELF executable headers, only guarantees the stack size of the initial thread of a process created by exec(2). All further threads get their stacks allocated explicitly in userland code by libpthreads or the like calling malloc(2). Normally these abstractions just reuse the same config params from the executable (unless you override them, using e.g. pthreads_attr_setstacksize). But, as the article says, Golang implements its own support for things like this, and so can implement special thread-allocation strategies per carrier thread type.)

Problems like this tend to come back and haunt you though. Sure, you can set max threads to 1 and move on with what you're doing for a while... but a lot of people run Go so they can have a lot more than 1 thread.

I've run into some of these where it's a lot more rare to hit, and so then it's reasonable to not do the thing that hurts, but watch out for it in the future. Sometimes you get lucky and it magically fixes itself forever; sometimes the weird case that you only hit with internal traffic ends up getting hit by public tratfic a lot.

Crashes like this where a wild write breaks something at a distance are always a PITA to debug (especially here, where the wild write is harmless if there's no data race)