Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We found a bug in Go's ARM64 compiler

blog.cloudflare.com

Post reply on HN
Print view View original

131–140 of 146 posts

Re: We found a bug in Go's ARM64 compiler

#131

Post by jgrahamc »

Agingcoder wrote: post #54
jgrahamc wrote: post #10

Earlier quoted context omitted.

What's stopping you applying today?

Fair question. Location primarily ( nothing in France ), and I’m not sure how ‘we’re looking for people who enjoy doing that kind of thing’( I very much do ) relates to the actual job offers, ie what job offer should I actually apply to. My background is not networking ( it’s math then hpc then broader stuff ) but I keep stumbling on similar problems ( including a beautiful one related to intel NICs a few years ago w…

You can email me jgc@ Cloudflare and I'll forward your details to the right people.

Re: We found a bug in Go's ARM64 compiler

#132

Post by Diggsey »

themafia wrote: post #37
gok wrote: post #17

The real lesson here should be that doing crazy shit like swizzling the program counter in a signal handler and writing your own assembler is not a good idea.

Neither of those are "crazy shit." It's just complex because the environment offers specific features like automatic GC with async preemption in a compiled language which pretty much requires it. Complex engineering isn't something to be avoided by default.

Agree, but I think there is a point to be made here: Go as a language has more subtle runtime invariants that must be upheld compared to other languages, and this has led to a relatively large number of really nasty bugs (eg. there have also been several bugs relating to native function calling due to stack space issues and calling convention differences). By "nasty" I mean ones that are really hard to track down if you don't have the resources that a company like CF does.

To me this points to a lack of verification, testing, and most importantly awareness of the invariants that are relied on. If the GC relies on the stack pointer being valid at all times, then the IR needs a way to guarantee that modifications to it are not split into multiple instructions during lowering. It means that there should be explicit testing of each kind of stack layout, and tests that look at the real generated code and step through it instruction by instruction to verify that these invariants are never broken...

Re: We found a bug in Go's ARM64 compiler

#133

Post by mananaysiempre »

dwattttt wrote: post #109
mananaysiempre wrote: post #102

Earlier quoted context omitted.

> While I'm sure [bumping the stack pointer atomically] makes sense, I don't think I've ever seen that be enforced. At least in C/C++. That’s because the C ABI supports unwinding with a fairly expressive set of tools for describing stack-pointer state on a per-instruction level. Even the simpler Microsoft ABI essentially uses bytecode for that[1]; and on the more complicated Itanium ABI, you get DWARF CFI instruction…

MS's ARM64 unwinding ABI looks even more complicated: https://learn.microsoft.com/en-us/cpp/build/arm64-exception-...

Ehh I wouldn’t say so (thanks for the correct link for ARM64 though in any case). What you need to be comparing to here is DWARF[1,2] section 6.4, and while it’s not as bad as other parts of DWARF, I still think it’s plenty complicated.

[1] https://dwarfstd.org/doc/DWARF5.pdf#page=171

[2] Slightly modified by psABI[3] section 3.7 for x86-64 or the LSB[4] section 11.6 for ARM64, but at this point that’s a drop in the bucket as far as overall complexity is concerned.

[3] https://gitlab.com/x86-psABIs/x86-64-ABI/-/jobs/artifacts/ma...

[4] https://refspecs.linuxfoundation.org/LSB_4.0.0/LSB-Core-gene...

Re: We found a bug in Go's ARM64 compiler

#134

Post by dwattttt »

mananaysiempre wrote: post #133
dwattttt wrote: post #109

Earlier quoted context omitted.

MS's ARM64 unwinding ABI looks even more complicated: https://learn.microsoft.com/en-us/cpp/build/arm64-exception-...

Ehh I wouldn’t say so (thanks for the correct link for ARM64 though in any case). What you need to be comparing to here is DWARF[1,2] section 6.4, and while it’s not as bad as other parts of DWARF, I still think it’s plenty complicated. [1] https://dwarfstd.org/doc/DWARF5.pdf#page=171 [2] Slightly modified by psABI[3] section 3.7 for x86-64 or the LSB[4] section 11.6 for ARM64, but at this point that’s a drop in the…

I was actually looking to point out MS's x64 ABI requires a standardised function epilog since this bug occurred during an epilog, only to find ARM64's epilogues are also described by bytecode (at least at a cursory glance).

Re: We found a bug in Go's ARM64 compiler

#135

Post by i80and »

ori_b wrote: post #85
rcxdude wrote: post #23

Earlier quoted context omitted.

Did you compile with optimisations? I think GCC will do a bunch of activity on the stack with -O0, but it'll generally coalesce everything into one push/pop per function with optimisations (not because of any rule, but just because it's faster). alloca and other dynamic stack allocation may break this, but normal variables should in pretty much all just get turned into one block on the stack (with appropriate re-use…

It will generate code to touch each page of the stack, because otherwise a very large stack allocation controlled by users (eg, in the case of a variable sized array) can be turned into a pointer to any location in memory by an attacker. Faulting in each page of the stack turns that into a crash. There was a userspace thread library I came across a long time ago that used variable length arrays to switch between thre…

The engineers were so preoccupied with whether or not they could that they didn't stop to think if they should

Re: We found a bug in Go's ARM64 compiler

#136

Post by indrora »

dpassens wrote: post #121
freep1zza wrote: post #120

Earlier quoted context omitted.

> Yes, though that weird stuff with dollars in it is not normal AArch64 assembly! See the AT&T vs Intel syntax since you aren't familiar with assembly: https://en.wikipedia.org/wiki/X86_assembly_language#Syntax

That's an x86 thing, though.

There are more assembler dialects than I care to remember.

The 2A06 assembler that people who write NES code (and later on SNES/GB/etc) use has some real quirks: $ prefixes a literal hex value but % is binary, but # in front of that is an address, registers are baked into the opcode (ldx -> load into X), and more.

Playstation folks all just used MIPS dialects which are mostly AT&Tish but the PS2 used an Intel style assembler.

Re: We found a bug in Go's ARM64 compiler

#137

Post by immibis »

anyfoo wrote: post #86
pklausler wrote: post #83

Earlier quoted context omitted.

Some architectures, and I believe aarch64 is one, have scratch registers reserved for being clobbered in special situations required by the assembler.

Not really, or at least not that I know if in the case of arm64. What you have is calling conventions that specify what one function/procedure/whatever can expect both from the caller and the callee's side.n I.e. some registers are caller-saved, some are callee-saved, which basically means the called function can treat them as "scratch". Additionally, they call out interactions with the OS/execution environment. For…

On some architectures there is a register reserved for assembler use, and even registers reserved for kernel use which can be changed in interrupt handlers and not changed back.

Re: We found a bug in Go's ARM64 compiler

#138

Post by pjmlp »

yvdriess wrote: post #123
pjmlp wrote: post #29

Earlier quoted context omitted.

Sure, I have experienced them, e.g. once in 2006 using IBM's JVM implementation with Websphere. However it is probably not as problematic due to the way Go allows for Assembly being used directly. While the JVM and CLR don't allow for direct access to Assembly code, Go does, thus I assume expecting safepoints everywhere is not an option, as any subroutine call can land on code that was manually written.

Go users can only insert assembly wrapped in a function call. That might be safety related, I am not entirely sure. (Well technically there is a way to inject assembly without the function call overhead. That's what https://pkg.go.dev/runtime/internal/atomic is doing. But you will need to modify the runtime and compiler toolchain for it.)

If you look the docs, they expect the developer to add specific information and use the registers in a specific way, otherwise Go will face runtime issues.

Whereas when you go over CGO, you get a marshaling layer similar to how JNI, P/Invoke work, that take care of those issues.

Re: We found a bug in Go's ARM64 compiler

#139

Post by jimsmart »

chavi2 wrote: post #91
pengaru wrote: post #12

For the impatient, here's the fix: https://github.com/golang/go/commit/f7cc61e7d7f77521e073137c...

One thing I worry about, probably unnecessarily, is anything with a sense of urgency. HEY GUYS WE JUST FOUND A GOLANG COMPILER BUG AND FATAL PANICS! Everyone is like “Hmm. I need to fix this now.” So, 99% probability it’s what it is. 1% it’s some secret defensive thing because there was a bad stupid zero day someone would get fired over or that could leave the world in shambles if uncovered, or maybe something else n…

It's an open source project — and quite a popular one, at that — and you are literally replying to a comment that specifies the changes made to fix this particular issue — you can see for yourself what is occurring here. Anyone can.

This issue, and the fix, has perfectly good visibility. Even if you personally can't understand the code, plenty of others can and do.

All of which makes your claims seem like quite unnecessary paranoia — to a lot of folk... and I suspect that is probably why your comment is getting heavily downvoted.

Re: We found a bug in Go's ARM64 compiler

#140

Post by anyfoo »

immibis wrote: post #137
anyfoo wrote: post #86

Earlier quoted context omitted.

Not really, or at least not that I know if in the case of arm64. What you have is calling conventions that specify what one function/procedure/whatever can expect both from the caller and the callee's side.n I.e. some registers are caller-saved, some are callee-saved, which basically means the called function can treat them as "scratch". Additionally, they call out interactions with the OS/execution environment. For…

On some architectures there is a register reserved for assembler use, and even registers reserved for kernel use which can be changed in interrupt handlers and not changed back.

Yeah, I mentioned x18 on arm64 as an example for the latter one. Didn't know about the register reserved for assembly use, apparently MIPS indeed had that: $1, also known as $at, is the "assembler temporary".
Post reply on HN