I often read these stories about hard to debug problems because I enjoy debugging (call it a love for software true crime) and this is the first one I’ve read I had an “oh god no” reaction when the author described where they needed to look for the culprit. The description of the layout engine and all of the browser specific tweaks makes it sound like an absolutely tedious nightmare to debug.
War story: the hardest bug I ever debugged
81–90 of 194 posts
Re: War story: the hardest bug I ever debugged
#82Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…
"To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added." My favourite are bugs, that not only don't appear in the debugger - but also don't reproduce anymore on normal settings after I took a closer look in the debugger (Only to come back later at a random time). Feels like chasing ghosts.
Terminology proposal: "Gremlins" :)
Re: War story: the hardest bug I ever debugged
#83Earlier quoted context omitted.
That is great QAing. It also speaks to why QA should be a real role in more orgs, rather than a shrinking discipline. Engineers LOVE LOVE LOVE to test the happy path. It's not even malice/laziness, it's their entire interpretation of the problem/requirements drives their implementation which then drives their testing. It's like asking restaurants to self-certify they are up to food safety codes.
If you do not follow the happy path something will break 100% of the time. That's why engineers always follow the happy path. Some engineers even think that anything outside the happy path is an exception and not even worth investigating. These engineers only thrives if the users are unable to switch to another product. Only competition will lead to better products.
My favorite happy path developer.. and he was by far 10x worse than any engineer I worked with at this, did the following:
Spec: allow the internal BI tool to send scheduled reports to the user
Implementation: the server required the desktop front end of said user to have been opened that day for the scheduled reports to work, even though the server side was sending the mails
Why this was hilariously bad - the only reason to have this feature is for when the user is out of office / away from desk for an extended period, precisely when they may not have opened their desktop UI for the day.
One of my favorite examples of how an engineer can get the entire premise of the problem wrong.
In the end he had taken so long and was so intransigent that desktop support team found it easier to schedule the desktop UIs to auto-open in windows scheduler every day such that the whole Rube Goldberg scheduled reports would work.
Re: War story: the hardest bug I ever debugged
#84It's amazing how often it happens in large companies that different people from different organizations are troubleshooting or fixing the same fault, independent from each other, without even knowing. Sometimes you don't even realize until you've implemented a fix which causes a merge conflict with the fix that someone else is working on.
Re: War story: the hardest bug I ever debugged
#85>There you have it: 2 days to find an issue that was already fixed and would have been resolved with no interaction
I work with LLVM and huge % of my work is fixing bugs that are already fixed in upstream
Re: War story: the hardest bug I ever debugged
#86In interviews I've never forced anyone to code, what I do is try to get them to tell me these sorts of war stories - I want to hear how you fixed it, why it was cooly bizarre, and I'm hoping for some enthusiasm when you talk about it. I couldn't always get people to talk this way, but people who did usually worked out well
You are selecting for the kind of person who always like to think about the war stories and brag about them.
Re: War story: the hardest bug I ever debugged
#87I had something like this once. Vendor provided an outlook plugin (ew) that linked storage directly in outlook (double ew) and contained a built in pdf viewer (disgusting) for law firms to manage their cases. One user, regardless of PC, user account or any other isolation factor, would reliably crash the program and outlook with it. She could work for 40 minutes on another users logged in account on another PC and re…
This is close enough to the "can't log in to computer when standing up" bug... someone had swapped the keycaps for D/F (for example) so when 5-star general tried to log in when standing up he was typing "doobar" instead of "foobar" into the password field.
With the lady, if she'd dialed it back a bit on her pace of work "because people are watching", that could have been a crazy one to debug... "only happens when no one is watching (and I'm not beastly-WPM closing cases)"
Re: War story: the hardest bug I ever debugged
#88> It didn’t correspond to a Google Docs release. The stack trace added very little information. There wasn’t an associated spike in user complaints Where mere mortals can complain about Google product?
Everywhere. They can do a Google search on product complaints filtered by date.
Re: War story: the hardest bug I ever debugged
#89Reminds me of the classic bug story where users couldn’t send emails more than 500 miles.
Re: War story: the hardest bug I ever debugged
#90Earlier quoted context omitted.
If you do not follow the happy path something will break 100% of the time. That's why engineers always follow the happy path. Some engineers even think that anything outside the happy path is an exception and not even worth investigating. These engineers only thrives if the users are unable to switch to another product. Only competition will lead to better products.
My favorite happy path developer.. and he was by far 10x worse than any engineer I worked with at this, did the following: Spec: allow the internal BI tool to send scheduled reports to the user Implementation: the server required the desktop front end of said user to have been opened that day for the scheduled reports to work, even though the server side was sending the mails Why this was hilariously bad - the only r…
You found a 1× engineer; the worst engineer that can keep the job.