We found a bug in Go's ARM64 compiler • Hacker News Board

Re: We found a bug in Go's ARM64 compiler

#91

Post by chavi2 » Thu, Oct 09, 2025, 2:04 AM UTC

pengaru wrote: post #12

For the impatient, here's the fix: https://github.com/golang/go/commit/f7cc61e7d7f77521e073137c...

One thing I worry about, probably unnecessarily, is anything with a sense of urgency.

HEY GUYS WE JUST FOUND A GOLANG COMPILER BUG AND FATAL PANICS!

Everyone is like “Hmm. I need to fix this now.”

So, 99% probability it’s what it is. 1% it’s some secret defensive thing because there was a bad stupid zero day someone would get fired over or that could leave the world in shambles if uncovered, or maybe something else needed to be swept under the rug, or maybe someone wants to distract while they introduce a new vulnerability.

I don’t think this with CVEs, but when someone’s like “install this patch everybody!” the dim red light flickers on.

Re: We found a bug in Go's ARM64 compiler

#92

Post by klausa » Thu, Oct 09, 2025, 2:06 AM UTC

lordnacho wrote: post #55

akerl_ wrote: post #48

Earlier quoted context omitted.

It feels like this comment was almost a purely additive anecdote of your own experience with a similar kind of issue, but you've spoiled it by deciding to tell the author that they're incorrect about how they felt during the process? Maybe different people find different things fun.

Not saying he's wrong, sometimes the word "fun" connotes something slightly different what what it literally means. "Satisfying" is something I'd use for the end state. Maybe "challenging" for the intermediate state. But while you're in a high-pressure situation that you don't understand, that is rarely "fun" in the literal sense. You wouldn't pay to be given compiler race condition bugs, right?

I wouldn't pay to be given any kind of work, but there are some aspects of my job that I find more or less 'fun'.

Hunting bugs that people have given up on or have no ideas on how to tackle is near the top of that list.

Re: We found a bug in Go's ARM64 compiler

#93

Post by riobard » Thu, Oct 09, 2025, 2:32 AM UTC

zamadatix wrote: post #79

riobard wrote: post #13

What ARM64 machines are you using and what are they used for? Last year you were announcing Gen 12 servers on AMD EPYC ( https://blog.cloudflare.com/gen-12-servers/ ), but IIRC there weren’t any mentions of ARM64. But now it seems you’re running ARM64 in full production.

I'm not Cloudflare, I just read their blog too much. As they hint in the article when mentioning secure boot, they've been deploying Ampere in parallel to AMD for several years now. Purpose wise it seems to be Edge related for efficiency reasons, but maybe they use them for other things too. You can read some more here https://blog.cloudflare.com/designing-edge-servers-with-arm-... and here https://blog.cloudflare.co…

Yeah but those are pretty dated. I was under the impression those old Ampere servers are not efficient compared to modern EPYC anymore. So I’m wondering what their current generation of arm64 servers look like :p

Re: We found a bug in Go's ARM64 compiler

#94

Post by saagarjha » Thu, Oct 09, 2025, 2:38 AM UTC

pklausler wrote: post #83

moefh wrote: post #74

Earlier quoted context omitted.

Would that be wise? The implemented solution uses a temporary register to hold the full value being added to rsp. I don't know enough about how people use the go assembler, but I imagine it would be very surprising if `add $imm, rsp, rsp` clobbered an unrelated register when `$imm` is large enough. Especially since what's clobbered is the designated "temporary register", which I imagine is used all the time in handwr…

Some architectures, and I believe aarch64 is one, have scratch registers reserved for being clobbered in special situations required by the assembler.

No, I think that’s just a MIPS thing.

Re: We found a bug in Go's ARM64 compiler

#95

Post by defleopold » Thu, Oct 09, 2025, 2:39 AM UTC

MarkSweep wrote: post #80

I wonder if Go had a mode where you make it single step every instruction and trigger a GC interrupt on every opcode. That would make it easier to find these kinds of bugs.

[flagged]

Re: We found a bug in Go's ARM64 compiler

#96

Post by saagarjha » Thu, Oct 09, 2025, 2:40 AM UTC

ori_b wrote: post #85

rcxdude wrote: post #23

Earlier quoted context omitted.

Did you compile with optimisations? I think GCC will do a bunch of activity on the stack with -O0, but it'll generally coalesce everything into one push/pop per function with optimisations (not because of any rule, but just because it's faster). alloca and other dynamic stack allocation may break this, but normal variables should in pretty much all just get turned into one block on the stack (with appropriate re-use…

It will generate code to touch each page of the stack, because otherwise a very large stack allocation controlled by users (eg, in the case of a variable sized array) can be turned into a pointer to any location in memory by an attacker. Faulting in each page of the stack turns that into a crash. There was a userspace thread library I came across a long time ago that used variable length arrays to switch between thre…

Wow, that’s horrible.

Re: We found a bug in Go's ARM64 compiler

#97

Post by syncsynchalt » Thu, Oct 09, 2025, 2:43 AM UTC

mperham wrote: post #35

Did they ever explain why netlink was involved? Or was that a red herring?

The netlink function uses a larger stack than most.

Their repro case required a stack adjustment larger than 1<<12 (4kiB).

Re: We found a bug in Go's ARM64 compiler

#98

Post by saagarjha » Thu, Oct 09, 2025, 2:45 AM UTC

lordnacho wrote: post #46

> This was a very fun problem to debug. I'm sure it was a relief to find a thorough solution that addressed the root cause. But it doesn't seem plausible that it was fun while it was unexplained. When I have this kind of bug it eats my whole attention. Something this deep is especially frustrating. Nobody suspects the standard library or the compiler. Devs have been taught from a young age that it's always you, not t…

The people who find the fun are often good at identifying when it is the standard library or the compiler.

Re: We found a bug in Go's ARM64 compiler

#99

Post by syncsynchalt » Thu, Oct 09, 2025, 2:47 AM UTC

pfdietz wrote: post #78

I see something like this and I wonder "what testing methodology would have found this?" It has to be general, not something that would involve knowing what the bug was ahead of time.

When your scale is large enough, you move to "what monitoring methodology will find this?"

When you're doing enough transactions you start to see a noise floor of e.g. bit flips from cosmic rays, and looking for issues involves correlating/categorizing possible software failures and distinguishing them from the misbehavior of hardware.

Re: We found a bug in Go's ARM64 compiler

#100

Post by surajrmal » Thu, Oct 09, 2025, 3:05 AM UTC

alberth wrote: post #84

I thought Cloudflare was 100% Rust, and x86 (EPYC) these days. Interesting to hear Go & ARM in use.

I doubt any company is mono language at that scale. Using ARM usually makes sense for s lot of horizontal scaling workloads so it's also not that surprising.