Always adjust your stack pointer atomically, kids.
Exactly what ran through my mind.
We found a bug in Go's ARM64 compiler
41–50 of 146 posts
Re: We found a bug in Go's ARM64 compiler
#42Did they ever explain why netlink was involved? Or was that a red herring?
The stack in that specific function was big enough to trigger the bug.
Re: We found a bug in Go's ARM64 compiler
#43Did they ever explain why netlink was involved? Or was that a red herring?
Seemed like a red herring. They were able to reproduce it without any libraries. Might have just been net link forcing the stacks to a certain size and that made the bug visible.
Re: We found a bug in Go's ARM64 compiler
#44Always adjust your stack pointer atomically, kids.
I guess those that wrote the preemption were on X86 where this doesn't happen thanks to variable length instructions being able to hold the constant and thus relied on the code-gen to do it atomically, then the ARM port had an automatic "split" from a higher level to make things "easy" thus giving us this bug. Nobodys fault really, but bad results ensued.
> Nobodys fault really, but bad results ensued.
Uh, the fault is entirely in writing an assembler _that is not an assembler_, but rather something that is _almost_ like one but then 1% like an IR instead. It's an unforced error.
Re: We found a bug in Go's ARM64 compiler
#45The real lesson here should be that doing crazy shit like swizzling the program counter in a signal handler and writing your own assembler is not a good idea.
Sorry, how exactly do you think compilers are supposed to work if not by 'writing [their] own assembler'? Someone has to write the assembler, and different compilers have different needs.
Re: We found a bug in Go's ARM64 compiler
#46> This was a very fun problem to debug.
I'm sure it was a relief to find a thorough solution that addressed the root cause. But it doesn't seem plausible that it was fun while it was unexplained. When I have this kind of bug it eats my whole attention.
Something this deep is especially frustrating. Nobody suspects the standard library or the compiler. Devs have been taught from a young age that it's always you, not the tools you were given, and that's generally true.
One time, I actually did find a standard library bug. I ended up taking apart absolutely everything on my side, because of course the last hypothesis you test is that the pieces you have from the SDK are broken. So a huge amount of time is spent chasing the wrong lead when it actually is a fundamental problem.
On top of this, the thing is a race condition, so you can't even reliably reproduce it. You think it's gone like they did initially, and then it's back. Like cancer.
Re: We found a bug in Go's ARM64 compiler
#47Earlier quoted context omitted.
I've never heard of that rule (though tbh I'm not allocating > 64KB of stack when I'm in assembly) and it seems Google hasn't either. While I'm sure it makes sense, I don't think I've ever seen that be enforced. At least in C/C++. Maybe it makes more sense for these stack inspecting garbage collectors but I've also heard of ones that just scan the stack without unwinding anything. I did a test asking Google's AI to g…
Did you compile with optimisations? I think GCC will do a bunch of activity on the stack with -O0, but it'll generally coalesce everything into one push/pop per function with optimisations (not because of any rule, but just because it's faster). alloca and other dynamic stack allocation may break this, but normal variables should in pretty much all just get turned into one block on the stack (with appropriate re-use…
Yes
Re: We found a bug in Go's ARM64 compiler
#48> This was a very fun problem to debug. I'm sure it was a relief to find a thorough solution that addressed the root cause. But it doesn't seem plausible that it was fun while it was unexplained. When I have this kind of bug it eats my whole attention. Something this deep is especially frustrating. Nobody suspects the standard library or the compiler. Devs have been taught from a young age that it's always you, not t…
It feels like this comment was almost a purely additive anecdote of your own experience with a similar kind of issue, but you've spoiled it by deciding to tell the author that they're incorrect about how they felt during the process?
Maybe different people find different things fun.
Re: We found a bug in Go's ARM64 compiler
#49Excellent article as always from the cloudflare blog - engineering without magic infrastructure and ml. One day I will apply ! Compiler bugs are actually quite common ( I used to find several a year in gcc ), but as the author says, some of them only appear when you work at a very large scale, and most people never dive that far.
What's stopping you applying today?
Low compensation relative to many other companies. (It didn't stop me from applying, but I stopped me from accepting.)
Re: We found a bug in Go's ARM64 compiler
#50> This was a very fun problem to debug. I'm sure it was a relief to find a thorough solution that addressed the root cause. But it doesn't seem plausible that it was fun while it was unexplained. When I have this kind of bug it eats my whole attention. Something this deep is especially frustrating. Nobody suspects the standard library or the compiler. Devs have been taught from a young age that it's always you, not t…
Some people are perverse individuals and actually enjoy debugging very esoteric things. What might be frustrating to you might be the very thing that gets someone else very excited.