Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

Post reply on HN
Print view View original

151–160 of 194 posts

Re: War story: the hardest bug I ever debugged

#151

Post by sa46 »

Cthulhu_ wrote: post #40
sandos wrote: post #27

Complaining about "slow to reproduce" and talking _seconds_. Dear, oh dear those are rookie numbers! Currently working a bug where we saw file system corruption after 3 weeks of automated testing, 10s of thousands of restarts. We might never see the problem again, even? Only happened once yet.

If it only happened once... it might be the final category of bugs where nothing you can do will fix it. Cosmic ray bit flipping bug. Which is something your software needs to be able to work around, or in this case, the file system itself... unless you're actually working on the file system itself, in which case, I wish you good luck.

What layers of hardware can comic rays impact? Memory with ECC is largely safe, right? What about the L1 cache and friends?

Re: War story: the hardest bug I ever debugged

#152

Post by skeaker »

epolanski wrote: post #117

The worst bug I've ever encountered was a JS file that kept not running, with very cryptic and hard to understand trace that made no sense. TypeScript and others parsed it fine without any issues. After 3 days of literally trying everything, I don't know why, I thought of rewriting the file character by character by hand and it worked. What was happening? Eventually opened the two files side by side in a hex editor a…

I've seen this happen as the result of errant copy-pasting.

Re: War story: the hardest bug I ever debugged

#153

Post by GarnetFloride »

I didn’t fix this bug but I did reproduce it so it could be fixed, but it took years. At one company I worked for we have an email archive and we were seeing an uptick in customers having issues with deleting expired emails. Most companies have a retention policy of about 7 years, and the company was now 10 years old and early customers were beginning to deleted old emails. But developers couldn’t find the bug, but reducing the scope of the deletion usually worked, so it was usually marked as not reproducible. While devs tried to debug it, no one would let us poke around their prod email server every much, for obvious reasons.

I had been promoted to technical writer and I needed a better test system that didn’t have customer data for screenshots. Something I needed was unique data because the archive used single instance storage, so I put together a bash script to create and send emails generated from random lines of public domain books I got from Gutenberg.

This worked great for me and at one point I had it fire off 1 million emails just for fun. I let my test email server and archive server chew on them over the weekend. It worked great but I had nearly maxed out my storage. No problem, use the deletion function. And it didn’t work.

It’s Didn’t Work. I had reproduced the bug in-house on a system we had full control over. Engineering and QA both took copies of my environments and started working on the bug.

I also learned the lore of the deletion feature. The founding developer didn’t think anyone wanted a deletion feature because it made no sense to him. But after pressure from the CEO, Board of Directors and customers he banged out some code over a weekend and shipped it. It was no 10 years later and he was long gone, and it was finally beginning to bite us.

After devs banged no the code for a while they found there was a design flaw, it failed if the number of items to delete was more than 500. QA had tested the feature, repeatedly, but their test data set just happened to be just smaller than 500 items so the bug never triggered. I only exceeded that because Austin Powers is funny.

Now that we could reproduce it, and knew there was a design flaw. The code for deletion needed to be replaced. It needed taking over two years to replace the code, because project management never thought it was all that important compared to new features, even though customers were complaining about it.

Re: War story: the hardest bug I ever debugged

#154

Post by parliament32 »

seeingnature wrote: post #135
aetimmes wrote: post #120

(disclaimer: I know OP IRL.) I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here: At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome dev…

I'd love to see the rest of your postmortem template! I never thought about adding a "Where did we get lucky?" question. I recently realized that one question for me should be, "Did you panic? What was the result of that panic? What caused the panic?" I had taken down a network, and the device led me down a pathway that required multiple apps and multiple log ins I didn't have to regain access. I panicked and because…

No, QR codes are auto-orienting[1]. If you're getting a different reading at different orientations, there is a bug in your scanner.

[1] https://en.wikipedia.org/wiki/QR_code#Design

Re: War story: the hardest bug I ever debugged

#156

Post by Suppafly »

seeingnature wrote: post #135
aetimmes wrote: post #120

(disclaimer: I know OP IRL.) I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here: At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome dev…

I'd love to see the rest of your postmortem template! I never thought about adding a "Where did we get lucky?" question. I recently realized that one question for me should be, "Did you panic? What was the result of that panic? What caused the panic?" I had taken down a network, and the device led me down a pathway that required multiple apps and multiple log ins I didn't have to regain access. I panicked and because…

> I didn't realize that QR codes had a proper orientation and figured that their corner identifiers handled any orientation.

Same, I assumed they were designed to always work. I suspect it was whatever app or library you were using that wasn't designed to handle them correctly.

Re: War story: the hardest bug I ever debugged

#157

Post by marginalia_nu »

aetimmes wrote: post #120

(disclaimer: I know OP IRL.) I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here: At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome dev…

Days-taken-to-fix is kind of a weird measure for how difficult a bug is. It's clearly a factor of a large number of things that's not the bug itself, including experience and whether you have to go it alone or if you can talk to the right people.

The bug ticks most of the boxes for a tricky bug:

* Non-deterministic

* Enormous haystack

* Unexpected "1+1=3"-type error with a cause outside of the code itself

Like sure it would have been slower to debug if it took 30 hours of to reproduce, and harder he had to be going down the Niagara falls in a barrel while debugging it, but I'm not quite sure those things quite count.

I had a similar category of bug I was struggling with the other year[1] that was related to a faulty optimization in the GraalVM JVM leading to bizarre behavior in very rare circumstances. If I'd been sitting next to the right JVM engineers over at Oracle I'm sure we'd figured it out in days and not the weeks it took me.

[1] https://www.marginalia.nu/log/a_104_dep_bug/

Re: War story: the hardest bug I ever debugged

#158

Post by chrismorgan »

perihelions wrote: post #108

My own story: I spent >10 hours debugging an Emacs project that would occasionally cause a kernel crash on my machine. Proximate cause was a nonlocal interaction between two debug-print statements. (Wasn't my first guess). The Elisp debug-print function #'message has two effects: it appends to a log, and also does a small update notification in the corner of the editor window. If that corner-of-the-window GUI object…

> If that corner-of-the-window GUI object is thrashed several hundred times in a millisecond, it would cause the GPU driver on my specific machine to lock up, for a reason I've never root-caused.

Until comparatively recently, it was absurdly easy to crash machines via their graphics drivers, even by accident. And I bet a lot of them were security concerns, not just DoS vectors. WebGL has been marvellous at encouraging the makers to finally fix their drivers properly, because browsers declared that kind of thing unacceptable (you shouldn’t be able to bring the computer down from an unprivileged web page¹), and developed long blacklists of cards and drivers, and brought the methodical approach browsers had finally settled on to the graphics space.

Things aren’t perfect, but they are much better than ten years ago.

—⁂—

¹ Ah, fond memories of easy IE6 crashes, some of which would even BSOD Windows 98. My favourite was, if my memory serves me correctly, document.createElement("table").appendChild(document.createElement("div")). This stuff was not robust.

Re: War story: the hardest bug I ever debugged

#160

Post by fuzzfactor »

mystified5016 wrote: post #111
BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

In hardware, you regularly see behavior change when you probe the system. Your oscilloscope or LA probes affect the system just enough to make a marginal circuit work. It's absolutely maddening.

The closer you get to natural science, eventually reliance on logical troubleshooting can be "illogical".

The more abundant the undefined (mis)behavior, the more you're going to be tearing your hair out.

Almost the kind of frustration where you're supposed to have a logic-based system, and it rears it ugly head and defies logic anyway :\

Post reply on HN