Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We found a bug in Go's ARM64 compiler

blog.cloudflare.com

Post reply on HN
Print view View original

101–110 of 146 posts

Re: We found a bug in Go's ARM64 compiler

#101

Post by afdbcreid »

btbuilder wrote: post #67
lordnacho wrote: post #46

> This was a very fun problem to debug. I'm sure it was a relief to find a thorough solution that addressed the root cause. But it doesn't seem plausible that it was fun while it was unexplained. When I have this kind of bug it eats my whole attention. Something this deep is especially frustrating. Nobody suspects the standard library or the compiler. Devs have been taught from a young age that it's always you, not t…

Segfaults with no use of “Unsafe” equivalents in managed languages can give immediate indication it’s not a code problem.

They explicitly mention there was usage of unsafe, and they weren't sure that's not the cause.

Re: We found a bug in Go's ARM64 compiler

#102

Post by mananaysiempre »

Neywiny wrote: post #15
bloak wrote: post #5

Earlier quoted context omitted.

> So another win for being able to read arm assembly. Yes, though that weird stuff with dollars in it is not normal AArch64 assembly! The article could have mentioned the "stack moves once" rule.

I've never heard of that rule (though tbh I'm not allocating > 64KB of stack when I'm in assembly) and it seems Google hasn't either. While I'm sure it makes sense, I don't think I've ever seen that be enforced. At least in C/C++. Maybe it makes more sense for these stack inspecting garbage collectors but I've also heard of ones that just scan the stack without unwinding anything. I did a test asking Google's AI to g…

> While I'm sure [bumping the stack pointer atomically] makes sense, I don't think I've ever seen that be enforced. At least in C/C++.

That’s because the C ABI supports unwinding with a fairly expressive set of tools for describing stack-pointer state on a per-instruction level. Even the simpler Microsoft ABI essentially uses bytecode for that[1]; and on the more complicated Itanium ABI, you get DWARF CFI instructions, which make the correct way to preserve a(n x86) register in the function prologue look like

  push rbx
  .cfi_adjust_cfa_offset 8
  .cfi_rel_offset rbx, 8
which are impossible to miss when reading compiler-generated assembly because of the sheer amount of annoying noise they create.

The Go authors decided to sidestep all of this complexity, which is understandable to a degree, but apparently they did not think through all the ramifications of doing so.

[1] https://learn.microsoft.com/en-us/cpp/build/exception-handli...

Re: We found a bug in Go's ARM64 compiler

#103

Post by Vipsy »

One thing that often gets missed is how hard it is to even suspect the compiler as the root cause. Most engineers waste hours chasing bugs in their own code because we’re trained to trust our tools. This mindset alone can make these rare compiler bugs much trickier to find.

Re: We found a bug in Go's ARM64 compiler

#104

Post by kmarc »

Vipsy wrote: post #103

One thing that often gets missed is how hard it is to even suspect the compiler as the root cause. Most engineers waste hours chasing bugs in their own code because we’re trained to trust our tools. This mindset alone can make these rare compiler bugs much trickier to find.

There are certain professions where the compilation process is (ab)used to optimize to a point where these bugs seemingly surface more often.

In the HFT sphere i haven't talked to a company that hasn't reported (bragged about finding) a super weird gcc/clang bug.

Well, also, at my last job we used a snapshot version of the compiler, bc... Any nanoseconds matters.

Re: We found a bug in Go's ARM64 compiler

#105

Post by SuperQue »

Vipsy wrote: post #103

One thing that often gets missed is how hard it is to even suspect the compiler as the root cause. Most engineers waste hours chasing bugs in their own code because we’re trained to trust our tools. This mindset alone can make these rare compiler bugs much trickier to find.

Yup, I had an issue filed against an open source project I work on. Was a crazy weird crash.

The reporter actually spent the effort to track it down, turns out it _was_ a Go compiler bug. (https://github.com/golang/go/issues/20427)

Re: We found a bug in Go's ARM64 compiler

#106

Post by pjmlp »

Vipsy wrote: post #103

One thing that often gets missed is how hard it is to even suspect the compiler as the root cause. Most engineers waste hours chasing bugs in their own code because we’re trained to trust our tools. This mindset alone can make these rare compiler bugs much trickier to find.

In the early PC days we suspected them a lot given how manually writting Assembly was still much better, in many cases.

I found out a bug on Turbo Pascal 6, where if you declare a variable with the same name as the function name, then the result was random garbage.

For those that don't know Pascal, the function name has to be assigned for the result value, so if a local variable with the same name is possible, then you cannot set the return value.

Something like this https://godbolt.org/z/s6srhTW66

    (* In Turbo Pascal 6 this would compile *)

    function Square(num: Integer): Integer;
    var
        Square: Integer;

    begin
        Square := num * num; (* Here the local variable gets used instead *)
    end;

Re: We found a bug in Go's ARM64 compiler

#107

Post by pjmlp »

anyfoo wrote: post #87
wbl wrote: post #52

Earlier quoted context omitted.

Assemblers used to do a ton of stuff back in the day

Oh yeah. S/360 assembly almost looks like a high level language sometimes. In MVS, functions of the OS and standard libraries (or its equivalent) were implemented as elaborate macros , with their own invocation syntax, whereas nowadays you'd expect a function that you'd call (dynamically linked or not), with parameters passed in registers. At least in the 90s, there were actually macro assemblers that supported OOP p…

Those are still around if you go for Assemblers with background in PC culture like NASM, YASM, MASM (still part of MSVC).

By the way Embarcaredo still has Turbo Assembler.

https://docwiki.embarcadero.com/RADStudio/Athens/en/Turbo_As...

Now a thing of the past, but Assemblers for game consoles were also quite powerfull in their macro capabilities.

I never liked the UNIX Assembly culture, because naturally as soon as C became a thing, they became the bare minimum required to assemble the generated Assembly out of the C compiler, as another step into the compilation pipeline.

All the niceties of macro assemblers came through the other platforms, like being able to use NASM instead of the platform assembler, not even GNU AS nor clang are that great in their abilities as Assemblers beyond the basic stuff.

Re: We found a bug in Go's ARM64 compiler

#108

Post by Bengalilol »

I always appreciate articles like this, where you can clearly see the engineer’s way of thinking.

I was just puzzled by the middle part of the article, where they start investigating their code but seem to overlook the fact that it only happens on ARM64.

Still, I understand that it’s professional to proceed step by step logically.

Great article, it was a pleasure reading it!

Re: We found a bug in Go's ARM64 compiler

#109

Post by dwattttt »

mananaysiempre wrote: post #102
Neywiny wrote: post #15

Earlier quoted context omitted.

I've never heard of that rule (though tbh I'm not allocating > 64KB of stack when I'm in assembly) and it seems Google hasn't either. While I'm sure it makes sense, I don't think I've ever seen that be enforced. At least in C/C++. Maybe it makes more sense for these stack inspecting garbage collectors but I've also heard of ones that just scan the stack without unwinding anything. I did a test asking Google's AI to g…

> While I'm sure [bumping the stack pointer atomically] makes sense, I don't think I've ever seen that be enforced. At least in C/C++. That’s because the C ABI supports unwinding with a fairly expressive set of tools for describing stack-pointer state on a per-instruction level. Even the simpler Microsoft ABI essentially uses bytecode for that[1]; and on the more complicated Itanium ABI, you get DWARF CFI instruction…

MS's ARM64 unwinding ABI looks even more complicated: https://learn.microsoft.com/en-us/cpp/build/arm64-exception-...

Re: We found a bug in Go's ARM64 compiler

#110

Post by Tor3 »

Vipsy wrote: post #103

One thing that often gets missed is how hard it is to even suspect the compiler as the root cause. Most engineers waste hours chasing bugs in their own code because we’re trained to trust our tools. This mindset alone can make these rare compiler bugs much trickier to find.

In the past it was more common to suspect the compiler, as others mention here. On a minicomputer I worked with in the late eighties, early nineties, I occasionally found errors in the compiler output. This was a Pascal compiler and because of that it didn't take too long to figure out that the code was actually correct and something else must be going on. Then firing up the debugger/tracer and scrutinizing and analyzing what happens in the disassembly.. when the problem was found, send a fax (yes!) to the head designer of the compiler, get a fixed test compiler back on a set of floppies.. went through this several times. I still have a printout somewhere with my pen marks pointing out a bug in the generated code.
Post reply on HN