Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We found a bug in Go's ARM64 compiler

blog.cloudflare.com

Post reply on HN
Print view View original

111–120 of 146 posts

Re: We found a bug in Go's ARM64 compiler

#111

Post by mixedbit »

Bengalilol wrote: post #108

I always appreciate articles like this, where you can clearly see the engineer’s way of thinking. I was just puzzled by the middle part of the article, where they start investigating their code but seem to overlook the fact that it only happens on ARM64. Still, I understand that it’s professional to proceed step by step logically. Great article, it was a pleasure reading it!

Hard to reproduce bugs often depend on an order of events or timing. Different architecture can trigger different order of execution, but this doesn't mean the bug is not in the application.

Re: We found a bug in Go's ARM64 compiler

#113

Post by Someone »

pklausler wrote: post #25
Neywiny wrote: post #3

That's an incredible find and once I saw the assembly I was right along with them on the debug path. Interestingly it doesn't need to be assembly for this to work, it's just that that's where the split was. The IR could've done it, it just doesn't for very good reasons. So another win for being able to read arm assembly. Unsure if this would be another way to do it but to save an instruction at the cost of a memory a…

I'm a little surprised that this bug wasn't fixed in the assembler as a special case for immediate adds to RSP. If the patch was to the compiler only, other instances of the bug could be lurking out there in aarch64 assembly code.

Is that possible? I think you would have [1] to use a register to build up the immediate value. The assembler cannot/should not default to one, so I think the best one could do is having another macro for ADD that takes that helper register as an argument. That wouldn’t fix other instances in the AArch64 assembly code.

[1] I’m not familiar with AMD64, but maybe, you could use a thread local (edit: wouldn’t work with M:N threads. You’d need a coroutine-local. That would tie the assembler to golang, and thus would, even on that alone, be a very bad idea) or reserve space in the stack frame for it, too, but I don’t see those as realistic options

Re: We found a bug in Go's ARM64 compiler

#114

Post by adastra22 »

anyfoo wrote: post #86
pklausler wrote: post #83

Earlier quoted context omitted.

Some architectures, and I believe aarch64 is one, have scratch registers reserved for being clobbered in special situations required by the assembler.

Not really, or at least not that I know if in the case of arm64. What you have is calling conventions that specify what one function/procedure/whatever can expect both from the caller and the callee's side.n I.e. some registers are caller-saved, some are callee-saved, which basically means the called function can treat them as "scratch". Additionally, they call out interactions with the OS/execution environment. For…

There are some architectures that do, but they're all old RISC chips.

Re: We found a bug in Go's ARM64 compiler

#115

Post by whizzter »

Sesse__ wrote: post #44
whizzter wrote: post #14

Earlier quoted context omitted.

I guess those that wrote the preemption were on X86 where this doesn't happen thanks to variable length instructions being able to hold the constant and thus relied on the code-gen to do it atomically, then the ARM port had an automatic "split" from a higher level to make things "easy" thus giving us this bug. Nobodys fault really, but bad results ensued.

> Nobodys fault really, but bad results ensued. Uh, the fault is entirely in writing an assembler _that is not an assembler_, but rather something that is _almost_ like one but then 1% like an IR instead. It's an unforced error.

It doesn't even need to be an error in the "assembler" but could be another part that converts from some internal highlevel IR, also for most cases split ops doesn't matter for register manipulating instructions (that you might want generated as compactly as possible) since regular atomics are separate on memory addresses.

Even then, if the code-gen was written BEFORE the preemption then it was fairly sloppy for those implementing the preemption to not consider the function epilogue, granted statically adjusting the stack/frame pointer by more than 4kb is probabably a tad of an edge-case.

Re: We found a bug in Go's ARM64 compiler

#116

Post by hshdhdhehd »

kmarc wrote: post #104
Vipsy wrote: post #103

One thing that often gets missed is how hard it is to even suspect the compiler as the root cause. Most engineers waste hours chasing bugs in their own code because we’re trained to trust our tools. This mindset alone can make these rare compiler bugs much trickier to find.

There are certain professions where the compilation process is (ab)used to optimize to a point where these bugs seemingly surface more often. In the HFT sphere i haven't talked to a company that hasn't reported (bragged about finding) a super weird gcc/clang bug. Well, also, at my last job we used a snapshot version of the compiler, bc... Any nanoseconds matters.

In HFT might you keep the bug fix secret so other HFTs cant benefit from it.

Re: We found a bug in Go's ARM64 compiler

#117

Post by kmarc »

hshdhdhehd wrote: post #116
kmarc wrote: post #104

Earlier quoted context omitted.

There are certain professions where the compilation process is (ab)used to optimize to a point where these bugs seemingly surface more often. In the HFT sphere i haven't talked to a company that hasn't reported (bragged about finding) a super weird gcc/clang bug. Well, also, at my last job we used a snapshot version of the compiler, bc... Any nanoseconds matters.

In HFT might you keep the bug fix secret so other HFTs cant benefit from it.

I saw both. One of the top firms wanted that, another I worked at we did report (of course with a scratched minimal reproducible example)

The thing is, it's quite unlikely that your competitor hits the exact same bug. The cost of us having to keep upstream patched, tested isn't justified.

Also in HFT world there are some very similar patterns across competing companies, yet, we just saw TernFS coming out from XTX, with not much fear of competitors benefiting from it more than they do.

Re: We found a bug in Go's ARM64 compiler

#118

Post by [deleted] »

Vipsy wrote: post #103

One thing that often gets missed is how hard it is to even suspect the compiler as the root cause. Most engineers waste hours chasing bugs in their own code because we’re trained to trust our tools. This mindset alone can make these rare compiler bugs much trickier to find.

[deleted]

Re: We found a bug in Go's ARM64 compiler

#120

Post by freep1zza »

bloak wrote: post #5
Neywiny wrote: post #3

That's an incredible find and once I saw the assembly I was right along with them on the debug path. Interestingly it doesn't need to be assembly for this to work, it's just that that's where the split was. The IR could've done it, it just doesn't for very good reasons. So another win for being able to read arm assembly. Unsure if this would be another way to do it but to save an instruction at the cost of a memory a…

> So another win for being able to read arm assembly. Yes, though that weird stuff with dollars in it is not normal AArch64 assembly! The article could have mentioned the "stack moves once" rule.

> Yes, though that weird stuff with dollars in it is not normal AArch64 assembly!

See the AT&T vs Intel syntax since you aren't familiar with assembly:

https://en.wikipedia.org/wiki/X86_assembly_language#Syntax

Post reply on HN