Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

Post reply on HN
Print view View original

111–120 of 194 posts

Re: War story: the hardest bug I ever debugged

#111

Post by mystified5016 »

BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

In hardware, you regularly see behavior change when you probe the system. Your oscilloscope or LA probes affect the system just enough to make a marginal circuit work. It's absolutely maddening.

Re: War story: the hardest bug I ever debugged

#112

Post by edarchis »

gmueckl wrote: post #18
userbinator wrote: post #10

Earlier quoted context omitted.

The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently. The hardest one I've debugged took a few months to reproduce, and would only show up on hardware that only one person on the team had. One of the interesting things about working on a very mature product is that bugs tend to be very rare, but those rare…

That reminded me of a former colleague at the desk next to me randomly exclaiming one day that he had just fixed a bug he had created 20 years ago. The bug was actually quite funny in a way: it was in the code displaying the internal temperature of the electronics box of some industrial equipment. The string conversion was treating the temperature variable as an unsigned int when it was in fact signed. It took a brav…

My brother is a wifi expert at a hw manufacturer. He once had a case where the customer had issues setting the transmit power to like 100 times the legal limit. They happened to be an offshore drilling platform and had an exemption for the transmission power as their antenna was basically on a buoy on the ocean. He had to convince the developer to fix that very specific bug.

Re: War story: the hardest bug I ever debugged

#113

Post by philk10 »

jdwithit wrote: post #46

I wish I could recall the details better but this was 20+ years ago now. In college I had an internship working at Bose, doing QA on firmware in a new multi CD changer addon to their flagship stereo. We were provided discs of music tracks with various characteristics. And had to listen to them over and over and over and over and over and over, running through test cases provided by QA management as we did. But also d…

Pedantically pointing out the difference between doing some exploratory testing "testing outside the test cases" and QA which is setting up processes/procedures part of which should be "do exploratory testing as well as running the test cases" but the Testing is not QA distinction has been fought over for decades...

But, love the story and I collect tales like this all the time so thanks for sharing

Re: War story: the hardest bug I ever debugged

#114

Post by epolanski »

z3t4 wrote: post #80
steveBK123 wrote: post #76

Earlier quoted context omitted.

That is great QAing. It also speaks to why QA should be a real role in more orgs, rather than a shrinking discipline. Engineers LOVE LOVE LOVE to test the happy path. It's not even malice/laziness, it's their entire interpretation of the problem/requirements drives their implementation which then drives their testing. It's like asking restaurants to self-certify they are up to food safety codes.

If you do not follow the happy path something will break 100% of the time. That's why engineers always follow the happy path. Some engineers even think that anything outside the happy path is an exception and not even worth investigating. These engineers only thrives if the users are unable to switch to another product. Only competition will lead to better products.

Nonsense, you're not describing any engineering at all.

I mean, it's well known that there's very little engineering in most software "engineers", but you're describing a person I've never seen.

Re: War story: the hardest bug I ever debugged

#115

Post by kwertyoowiyop »

protocolture wrote: post #2

I had something like this once. Vendor provided an outlook plugin (ew) that linked storage directly in outlook (double ew) and contained a built in pdf viewer (disgusting) for law firms to manage their cases. One user, regardless of PC, user account or any other isolation factor, would reliably crash the program and outlook with it. She could work for 40 minutes on another users logged in account on another PC and re…

Love it! I hope you got her a plaque or something. It’s like John Henry vs. the steam engine.

Re: War story: the hardest bug I ever debugged

#116

Post by wasabiketchup »

Fun one:

I work on a server software of online backups for customers. We do daily thousands of mount/umount of a particular filesystem. Once every month or so, we get an issue where a file timestamp fails to save, the error happens at the filesystem level.

Hard to reproduce! It's a filesystem bug! So it's full theorical, reading code and seeing how it would happen.

Found out after a while, the conditions were fun. I don't remember exactly, but it was like, you need to follow these steps : 1/ Create a folder 2/ Create in it 99 files (no more no less) 3/ Create a new folder 4/ Copy the first of the 99 files in the new folder

The issue was linked to some data structure caching, and cache eviction.

Had fun finding it out!

Re: War story: the hardest bug I ever debugged

#117

Post by epolanski »

The worst bug I've ever encountered was a JS file that kept not running, with very cryptic and hard to understand trace that made no sense. TypeScript and others parsed it fine without any issues.

After 3 days of literally trying everything, I don't know why, I thought of rewriting the file character by character by hand and it worked. What was happening?

Eventually opened the two files side by side in a hex editor and here it is: several exotic unicode characters for "empty" space.

Re: War story: the hardest bug I ever debugged

#119

Post by amne »

latexr wrote: post #107
artimaeis wrote: post #91

Earlier quoted context omitted.

If I understood correctly - the Math.Abs() value would be positive roughly half the time, regardless of the steps taken to get there. That seems definitively nondeterministic.

You don’t call Math.abs() on its own, you need to give it a number. Regardless if it is positive or negative, it should always return a positive (that’s what an absolute value is). The issue here is that it was returning a negative number when given a negative value, which is wrong: > We rerun the repro. We look at the logged value. Math.abs() is returning negative values for negative inputs. We reload and run it aga…

With the appropiate butterfly wing flap everything is deterministic.

https://xkcd.com/378/

Re: War story: the hardest bug I ever debugged

#120

Post by aetimmes »

(disclaimer: I know OP IRL.)

I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here:

At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome devs.

Additionally - the author (and his team) triaged, root caused and remediated a JS compiler bug in 2 days. The sheer amount of complexity involved in trying to narrow down where in the browser code this could all be going wrong is staggering. Consider that the reason it took him "only" two days is because he is very, _very_ good at what he does.

Post reply on HN