Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

Post reply on HN
Print view View original

171–180 of 194 posts

Re: War story: the hardest bug I ever debugged

#171

Post by fatnoah »

steveBK123 wrote: post #76
jdwithit wrote: post #46

I wish I could recall the details better but this was 20+ years ago now. In college I had an internship working at Bose, doing QA on firmware in a new multi CD changer addon to their flagship stereo. We were provided discs of music tracks with various characteristics. And had to listen to them over and over and over and over and over and over, running through test cases provided by QA management as we did. But also d…

That is great QAing. It also speaks to why QA should be a real role in more orgs, rather than a shrinking discipline. Engineers LOVE LOVE LOVE to test the happy path. It's not even malice/laziness, it's their entire interpretation of the problem/requirements drives their implementation which then drives their testing. It's like asking restaurants to self-certify they are up to food safety codes.

> That is great QAing. It also speaks to why QA should be a real role in more orgs, rather than a shrinking discipline.

As a software engineer, I've always been very proud of my thoroughness and attention to detail in testing my code. However, good QA people always leave me wondering "how did they even think to do that?" when reviewing bug reports.

QA is both a skillset AND a mindset.

Re: War story: the hardest bug I ever debugged

#172

Post by Suppafly »

sofixa wrote: post #70
nopurpose wrote: post #33

> It didn’t correspond to a Google Docs release. The stack trace added very little information. There wasn’t an associated spike in user complaints Where mere mortals can complain about Google product?

Apparently paying for Google One gives access to Google support. If that's worth anything I have no idea, but it exists

If it's anything like the support you get paying for Google Fi or Youtube Premium, it's probably worthless.

Re: War story: the hardest bug I ever debugged

#173

Post by Adverblessly »

BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

> To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added.

A favourite of mine was a bug (specifically, a stack corruption) that I only managed to see under instrumentation. After a lot of debugging turns out that the bug was in the instrumentation software itself, which generated invalid assembly under certain conditions (calling one of its own functions with 5 parameters even though it takes only 4). Resolved by upgrading to their latest version.

Re: War story: the hardest bug I ever debugged

#174

Post by glxxyz »

steveBK123 wrote: post #95
db48x wrote: post #90

Earlier quoted context omitted.

You found a 1× engineer; the worst engineer that can keep the job.

Over time we actually found him to be more of a -2x engineer, but thats another story edit: reminded me of the old joke A programmer gets sent to the store by his wife. His wife says, “Get a gallon of milk, and if they have eggs, get a dozen.” The programmer returns home with 12 gallons of milk and says, “They had eggs.”

'Antiwork' - every hour they work uses up more than an hour of other engineers' time in questions, meetings, and later fixes.

Re: War story: the hardest bug I ever debugged

#175

Post by jbs789 »

aetimmes wrote: post #120

(disclaimer: I know OP IRL.) I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here: At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome dev…

I suspect that by minimising someone else’s work it allows the commenters to feel better about themselves. As a general rule/perspective.

Re: War story: the hardest bug I ever debugged

#176

Post by egypturnash »

parliament32 wrote: post #154
seeingnature wrote: post #135

Earlier quoted context omitted.

I'd love to see the rest of your postmortem template! I never thought about adding a "Where did we get lucky?" question. I recently realized that one question for me should be, "Did you panic? What was the result of that panic? What caused the panic?" I had taken down a network, and the device led me down a pathway that required multiple apps and multiple log ins I didn't have to regain access. I panicked and because…

No, QR codes are auto-orienting[1]. If you're getting a different reading at different orientations, there is a bug in your scanner. [1] https://en.wikipedia.org/wiki/QR_code#Design

It does seem to be possible to design QR codes that scan differently depending on the orientation, though they look a little visibly malformed.

https://hackaday.com/2025/01/23/this-qr-code-leads-to-two-we...

Re: War story: the hardest bug I ever debugged

#177

Post by jandrese »

At least the author worked for Google. It's another layer of fun to go through the work of tracking down a bug like that as a third party and then trying to somehow contact a person at the company who can fix it, especially when it is a big company and doubly so if the product is older and on a maintenance only schedule.

Me: "Your product is broken for all customers in this situation, probably has been so for years, here is the exact problem and how to fix it, can I talk with someone who can do the work?"

Customer Support: "Have you tried turning your machine off and turning it back on again?"

Re: War story: the hardest bug I ever debugged

#179

Post by MawKKe »

rhaps0dy wrote: post #122
jffhn wrote: post #32

Earlier quoted context omitted.

>Though abs() returning negative numbers is hilarious. Math.abs(Integer.MIN_VALUE) in Java very seriously returns -2147483648, as there is no int for 2147483648.

Oh no, Pytorch does the same thing: a = torch.tensor(-2*31, dtype=torch.int32) assert a == a.abs()

numpy as well. and tensorflow
Post reply on HN