Debugging an evil Go runtime bug • Hacker News Board

Re: Debugging an evil Go runtime bug

#31

Post by jbub » Tue, Dec 05, 2017, 6:34 AM UTC

I wish i can do something like this one day :) Impressive skills by the author! Well done!

Re: Debugging an evil Go runtime bug

#32

Post by zaarn » Tue, Dec 05, 2017, 8:48 AM UTC

0x0 wrote: post #19

Setting up for a 104 byte stack seems pretty crazy, wouldn't you risk overrunning the red zone even without all that stack probing? https://en.wikipedia.org/wiki/Red_zone_(computing)

The redzone is only non-explicit stack and only really matters if you violate it. If vDSO allocates stack properly, which it should considering it's an exported function, there is no problem.

Re: Debugging an evil Go runtime bug

#33

Post by donquichotte » Tue, Dec 05, 2017, 9:06 AM UTC

Wow, these are some serious debugging skills. I also admire the tenacity and the will to investigate the root cause.

> I tried setting GOMAXPROCS=1, which tells Go to only use a single OS-level thread to run Go code. This also stopped the crashes, again pointing strongly to a concurrency issue.

I think I would have stopped there.

Re: Debugging an evil Go runtime bug

#34

Post by cfstras » Tue, Dec 05, 2017, 9:52 AM UTC

Hah. I remember Bryan Cantrill complaining about this exact thing. Glad that it's fixed.

Turns out somebody else did, too: https://twitter.com/bcantrill/status/774290166164754433?lang...

/edit: spelling

Re: Debugging an evil Go runtime bug

#35

Post by [deleted] » Tue, Dec 05, 2017, 10:29 AM UTC

donquichotte wrote: post #33

Wow, these are some serious debugging skills. I also admire the tenacity and the will to investigate the root cause. > I tried setting GOMAXPROCS=1, which tells Go to only use a single OS-level thread to run Go code. This also stopped the crashes, again pointing strongly to a concurrency issue. I think I would have stopped there.

[deleted]

Re: Debugging an evil Go runtime bug

#36

Post by squeed » Tue, Dec 05, 2017, 10:35 AM UTC

MrBuddyCasino wrote: post #9

fierro wrote: post #8

this is incredibly impressive

That was Captain Ahab level persistence. I wonder how long it took him.

I was one of the spectators on the Prometheus thread. It took him 2 days. It was insane.

Re: Debugging an evil Go runtime bug

#37

Post by igravious » Tue, Dec 05, 2017, 10:36 AM UTC

Ninja level debugging and diagnostic skills. A fascinating read from start to finish. Bonus points for the GRUB 2 feature for masking out bad RAM blocks – still dreaming of owning a laptop with ECC memory :/

Re: Debugging an evil Go runtime bug

#38

Post by squeed » Tue, Dec 05, 2017, 10:37 AM UTC

For a similar tale of vDSO getting someone in trouble, check out this fun talk "Really crazy container troubleshooting stories": https://media.ccc.de/v/ASG2017-115-really_crazy_container_tr...

Re: Debugging an evil Go runtime bug

#39

Post by nikanj » Tue, Dec 05, 2017, 11:12 AM UTC

squeed wrote: post #36

MrBuddyCasino wrote: post #9

Earlier quoted context omitted.

That was Captain Ahab level persistence. I wonder how long it took him.

I was one of the spectators on the Prometheus thread. It took him 2 days. It was insane.

I ordinarily speak against the whole "10x engineer" trope, but this case would've easily taken me 20 days. Or months.

Re: Debugging an evil Go runtime bug

#40

Post by 0x0 » Tue, Dec 05, 2017, 1:35 PM UTC

zaarn wrote: post #32

0x0 wrote: post #19

Setting up for a 104 byte stack seems pretty crazy, wouldn't you risk overrunning the red zone even without all that stack probing? https://en.wikipedia.org/wiki/Red_zone_(computing)

The redzone is only non-explicit stack and only really matters if you violate it. If vDSO allocates stack properly, which it should considering it's an exported function, there is no problem.

How does being an exported function change the rules, I thought the x86-64 ABI mandated an implicit safe 128 bytes below rsp at all times? Also, how can a vDSO function "allocate" stack? It would have to know about the current stack space as configured by the go runtime, and somehow dig into this go-runtime-specific record of the current stack limit? Isn't the only available option for any exported function just to /use/ pre-allocated stack space (by subtracting from rsp) - I don't see how it could possibly extend the pre-allocated stack.