Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging an evil Go runtime bug

marcan.st

Post reply on HN
Print view View original

21–30 of 51 posts

Re: Debugging an evil Go runtime bug

#22

Post by EpicEng »

emmelaich wrote: post #18

Such a thorough and well written write up. To think that some experienced programmers I know declare that concurrency is easy.

It's only 'easy' because

A) other (probably better?) engineers have created abstractions for them, and

B) they've never had to debug a truly difficult issue related to concurrency

Re: Debugging an evil Go runtime bug

#23

Post by exikyut »

amluto wrote: post #11
cwzwarich wrote: post #2

Why doesn't the vDSO code just use MOV in its stack probe probe rather than an OR?

Because gcc's -fstack-check is garbage. Gentoo Hardened should not be using it.

From https://lkml.org/lkml/2017/11/10/310, discussing the disassembly:

> This code is so wrong I don't even no where to start. ... I suppose we could try to make the kernel fail to build at all on a broken configuration like this.

Re: Debugging an evil Go runtime bug

#26

Post by stmw »

This is great story, probably wins the year for "best bug you've ever encountered?" question. Having implemented some weird runtimes for weird languages, I am sympathetic to Go team here -- these odd tradeoffs of pushing the envelope on OS your_own_compiler interactions can trigger some wild experiences.

Re: Debugging an evil Go runtime bug

#27

Post by stmw »

dboreham wrote: post #21
FLUX-YOU wrote: post #16

>Since the problem gets worse with temperature, what happens if I heat up the RAM? Neat. I wonder if that makes Rowhammer more likely to occur.

Probably. Hotter usually means closer to not working for semiconductors.

Possibly, although hotter fundamentally means more thermal noise, which might actually reduce correlations / ability to communicate effectively between adjacent circuits.

Think of it as SNR (signal-noise-ratio) -- increasing temperature increases thermal noise (there are other kinds), and with the same signal, it should actually reduce the efficiency of the side channel.

But it brings up a good question, I wonder if anyone has studied this...

Re: Debugging an evil Go runtime bug

#28

Post by stmw »

exikyut wrote: post #23
amluto wrote: post #11

Earlier quoted context omitted.

Because gcc's -fstack-check is garbage. Gentoo Hardened should not be using it.

From https://lkml.org/lkml/2017/11/10/310 , discussing the disassembly: > This code is so wrong I don't even no where to start. ... I suppose we could try to make the kernel fail to build at all on a broken configuration like this.

Very good point. Key phrase is "an offset > 4096 is just bogus. That's big enough to skip right over the guard page" - if allocating more than a page-size-worth (4K in most cases), it should try to write to it before returning.

Re: Debugging an evil Go runtime bug

#29

Post by Twirrim »

jchw wrote: post #17
warent wrote: post #10

Marcan's attitude is great; I know of a ton of people (myself included) who would've written that article with far more complaining interleaved. Super informative as well, I learned a ton from this article (GRUB 2 feature for marking off bad RAM? Wow!). Very well written, informative, humorous, etc. Love it

Agreed. Marcan's a chill dude. I saw him a few times in the Dolphin Emulator IRC and he always had interesting things to say. From my perspective this approach was pretty unique; going all the way down to debugging the hardware first may seem obvious to some, but it's a totally opposite approach to how I'd go about it. My mind would jump directly to producing a minimal test case. Would've never thought to mark off ba…

From a slightly more old-school sysadmin approach, I learned to troubleshoot roughly in line with the OSI model (https://www.lifewire.com/layers-of-the-osi-model-illustrated...), starting at layer 1 (physical) and working up.

That's not to say I spend a whole lot of time looking at the lower levels, but my quick mental checklist starts off down at physical points, and I try to quickly eliminate possibilities. In a lot of cases it's obvious it's a code / logic bug, and you can completely skip the lower layer stuff, but making it a conscious step pays off.

Post reply on HN