Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

Post reply on HN
Print view View original

51–60 of 194 posts

Re: War story: the hardest bug I ever debugged

#51

Post by taneq »

I had one that took literally years to reproduce. It was in PLC code, on a touchscreen controller running a soft PLC with Busybox under the hood. These devices were used 24/7 and usually absolutely bullet proof. Every now and then I’d get a comment that sometimes they’d crash on startup but a power cycle usually fixed it. Finally managed to get it to happen in the workshop, and dropped everything to try and figure it out.

The ultimate cause was in the network initialisation using a network library that was a tissue-paper-thin wrapper around Linux sockets. When downloading a new software version to the device, it would halt the PLC but this didn’t cleanly shut down open sockets, which would stay open, preventing a network service from starting until the unit was restarted. So I did the obvious thing and wrote the socket handle to a file. On startup I’d check the file and if it existed, shut that socket handle. This worked great during development.

Of course this file was still there after a power cycle. 99% of the time nothing would happen, but very occasionally, closing this random socket handle on startup would segfault the soft PLC runtime. So dumb, but so hard to actually catch in the wild.

Re: War story: the hardest bug I ever debugged

#52

Post by animal531 »

As far as I'm concerned if you can use a debugger it automatically shouldn't qualify as the most difficult ever.

As per the compute shader post from a few days ago, currently I'm "debugging" some pretty advanced code that's being ported to a shader, and the only way to do it is by creating an array of e.g. ints and inserting values into it in both the original and the shader code to see where they diverge. Its not the most difficult but its quite time consuming.

Re: War story: the hardest bug I ever debugged

#53

Post by lukan »

BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

"To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added."

My favourite are bugs, that not only don't appear in the debugger - but also don't reproduce anymore on normal settings after I took a closer look in the debugger (Only to come back later at a random time). Feels like chasing ghosts.

Re: War story: the hardest bug I ever debugged

#54

Post by friendzis »

My hardest bug story, almost circling back to the origin of the word.

An intern gets a devboard with a new mcu to play with. A new generation, but mostly backwards compatible or something like that. Intern gets the board up and running with embedded equivalent of "hello world". They port basic product code - ${thing} does not work. After enough hair are pulled, I give them some guidance - ${thing} does not work. Okay, I instruct intern to take mcu vendor libraries/examples and get ${thing} running in isolation. Intern fails.

Okay, we are missing something huge that should be obvious. We start pair programming and strip the code down layer by layer. Eventually we are at a stage where we are accessing hand-coded memory addresses directly. ${thing} does not work. Okay, set up a peripheral and read state register back. Assertion fails. Okay, set up peripheral, nop some time for values to settle, read state register back. Assertion fails. Check generated assembly - nopsled is there.

We look at manual, the bit switching peripheral into the state we care about is not set. However we poke the mcu, whatever we write to control register, the bit is just not set and the peripheral never switches into the mode we need. We get a new devboard (or resolder mcu on the old one, don't remember) and it works first try.

"New device - must be new behavior" thinking with lack of easy access to the new hardware led us down a rabbit hole. Yes, nothing too fancy. However, I shudder thinking what if reading the state register gave back the value written?

Re: War story: the hardest bug I ever debugged

#55

Post by latexr »

> I do it a few more times. It’s not always the 20th iteration, but it usually happens sometime between the 10th and 40th iteration. Sometimes it never happend. Okay, the bug is nondeterministic.

That’s an incorrect assumption. Just because your test case isn’t triggering the bug reliably, it does not mean the bug is nondeterministic.

That is like saying the “OpenOffice can’t print on Tuesdays” is non deterministic because you can’t reproduce it everyday. It is deterministic, you just need to find the right set of circumstances.

https://beza1e1.tuxen.de/lore/print_on_tuesday.html

From the writing it appears the author found one way to reproduce the bug sometimes and then relied on it for every test. Another approach would have been to tweak their test case until they found a situation which reproduced the bug more or less often, trying to find the threshold that causes it and continuing to deduce from there.

Re: War story: the hardest bug I ever debugged

#56

Post by friendzis »

atq2119 wrote: post #48
noduerme wrote: post #44

Amazing war story. Very well told. Honestly, of all the stupid ideas, having your engine switch to a completely untested mode when under heavy load, a mode that no one ever checks and it might take years to discover bugs in, is absolutely one of most insane things I can think of. That's at best really lazy, and at worst displays a corporate culture that prizes superficial performance over reliability and quality. Tha…

I agree with your assessment of how stupid this is, but I'm not surprised. To be clear, there are good reasons for this different mode. The fuck-up is not testing it properly. These kinds of modes can be tested properly in various ways, e.g. by having an override switch that forces the chosen mode to be used all the time instead of using the default heuristics for switching between modes. And then you run your test s…

Isn't stochastic testing becoming more and more of a standard practice? Even if you have the hardware and time to run a full testsuite, you still want to add some randomness just to catch accidental dependencies between tests.

Re: War story: the hardest bug I ever debugged

#57

Post by shakna »

treyd wrote: post #21
gmueckl wrote: post #18

Earlier quoted context omitted.

That reminded me of a former colleague at the desk next to me randomly exclaiming one day that he had just fixed a bug he had created 20 years ago. The bug was actually quite funny in a way: it was in the code displaying the internal temperature of the electronics box of some industrial equipment. The string conversion was treating the temperature variable as an unsigned int when it was in fact signed. It took a brav…

This is a surprisingly common mistake with temperature readings. Especially when the system has a thermal safety power off that triggers if it's above some temperature, but then interprets -1 deg C as actually 255 deg C.

The rollout is still happening, but the new resident water meters for Victoria, Australia come with a temperature fix.

Prior to this year, they could only handle 0-127 degrees for the water temperature. Which used to be sensible, but there were some issues with pressurised water starting to be delivered to houses resulting in negative temperatures being reported, like -125C, which immediately has the water switch off to prevent icing problems.

The software side also switched from COBOL to Ada. So that's kewl.

Re: War story: the hardest bug I ever debugged

#58

Post by noisy_boy »

koliber wrote: post #41

And somewhere out there is a person reading this post and coming to the conclusion "How can Google be stupid enough to hire people stupid enough to have abs() return a negative value." Love the story! There is so much complexity in the world around as that seemingly obviously wrong things happen through the most unlikely chains of dependency.

> And somewhere out there is a person reading this post and coming to the conclusion "How can Google be stupid enough to hire people stupid enough to have abs() return a negative value."

Weird things can happen anywhere but I was wondering why this issue wasn't caught by test cases before it escaped to production? I would think that a compiler team would have low-level tests for such common functions.

Re: War story: the hardest bug I ever debugged

#60

Post by adrian_b »

jffhn wrote: post #32
BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

>Though abs() returning negative numbers is hilarious. Math.abs(Integer.MIN_VALUE) in Java very seriously returns -2147483648, as there is no int for 2147483648.

Unchecked integer overflow strikes again.
Post reply on HN