Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

Post reply on HN
Print view View original

161–170 of 194 posts

Re: War story: the hardest bug I ever debugged

#161

Post by nathan_douglas »

seeingnature wrote: post #135
aetimmes wrote: post #120

(disclaimer: I know OP IRL.) I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here: At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome dev…

I'd love to see the rest of your postmortem template! I never thought about adding a "Where did we get lucky?" question. I recently realized that one question for me should be, "Did you panic? What was the result of that panic? What caused the panic?" I had taken down a network, and the device led me down a pathway that required multiple apps and multiple log ins I didn't have to regain access. I panicked and because…

A good section to have is one on concept/process issues you encountered, which I think is a generalization of your question about panic.

For instance, you might be mistaken about the operation of a system in some way that prolongs an outage or complicates recovery. Or perhaps there are complicated commands that someone pasted in a comment in a Slack channel once upon a time and you have to engage in gymnastics with Sloogle™ to find them, while the PM and PO are requesting updates. Or you end up saving the day because of a random confluence of rabbit holes you'd traversed that week, but you couldn't expect anyone else on the team to have had the same flash of insight that you did.

That might be information that is valuable to document or add to training materials before it is forgotten. A lot of postmortems focus on the root cause, which is great and necessary, but don't look closely at the process of trying to stop the bleeding.

Re: War story: the hardest bug I ever debugged

#162

Post by shoo »

skeaker wrote: post #152
epolanski wrote: post #117

The worst bug I've ever encountered was a JS file that kept not running, with very cryptic and hard to understand trace that made no sense. TypeScript and others parsed it fine without any issues. After 3 days of literally trying everything, I don't know why, I thought of rewriting the file character by character by hand and it worked. What was happening? Eventually opened the two files side by side in a hex editor a…

I've seen this happen as the result of errant copy-pasting.

I've seen this happen in enterprise systems integration work, where some data interchange spec is authored as a Word document, and it has tables defining valid string values for certain fields, and Word helpfully replaces plain ascii dashes in the string constants with pretty long dashes, and team A builds their side hand-typing these constants as plain ascii, and team B builds their side by copy-pasting the exact unicode strings out of the Word doc.

Not a hard thing to debug once the issue is noticed, and completely preventable (write specs in plain text).

Re: War story: the hardest bug I ever debugged

#163

Post by farazbabar »

One of the interesting ones we encountered was in the JDBC driver of our chosen database at the time. Under load, the application core dumped. Mind you this is java, running a native jdbc driver, no JNI in sight. It took some gdb stepping to figure out that under load, the JIT compiler got a little aggressive and inlined a little more code than there was room in the JIT buffer - result? a completely random core dump. Once I did find it, it was a simple matter of increasing JIT buffer size and adding more heap and ram. Tracing assembler generated from byte code generated from java was just part of the issue, the fact that the code itself had nothing to do with the issue is what made it interesting as the buffer size is set in a completely different area by the jvm. Fun times.

Re: War story: the hardest bug I ever debugged

#164

Post by wglb »

This is a very fun post, not only on its own merits, but also how it spurs many other hard-to-debug stories.

I like the hard-earned lessons that are often taken away from such sessions.

While nowhere on the scale of this story, I helped a fellow student while I was at the University where his program was outputting highly bogus numbers from punched card deck input. I ultimately suggested that he print out the numbers that were being read by the program and presto the field alignments were off. This has now become my first step in debugging.

During a co-op stint during my EE degree program was at a pulp bleach plant in Longview Washington. They were implementing instrumentation of various metrics in the bleach tower. The engineers told of a story about one of their instruments to measure flow or temperature or acidity. The instrument was failing but the manufacturer couldn't find any flaw, shipped it back. The cycle repeated several times until one of the engineers accompanied the instrument to the repair lab. The technicians were standing the instrument on its side, not flat as it was in the instrument rack back at the plant. Lying it flat exposed the error.

Another bug sticks in my mind from reading Coders At Work by Peter Seibel. Guy Steele is telling about a bug Bill Gosper reported in the bignum library. One thing caught is eye was a conditional step he didn't quite understand. Since it was based on the division algorithms from Knuth: "And what caught my eye in Knuth was a comment that this step happens rarely—with a probability of roughly only one in two to the size of the word." The error was in a rarely-executed piece of code. The lesson here helped him find similar bugs.

While three of us were building a compiler at Sycor, we kept a large lab notebook in which we wrote brief release notes, and a one-line note about each bug we found and fixed.

My most recent bug was a new emacs snippet was causing errors in eval_buf. Made no sense, so ultimately decided to clear out the .emacs.d directory and start over. There were files that were over 20 years old--I just copied the directory when I built a new machine.

Re: War story: the hardest bug I ever debugged

#166

Post by wglb »

epolanski wrote: post #117

The worst bug I've ever encountered was a JS file that kept not running, with very cryptic and hard to understand trace that made no sense. TypeScript and others parsed it fine without any issues. After 3 days of literally trying everything, I don't know why, I thought of rewriting the file character by character by hand and it worked. What was happening? Eventually opened the two files side by side in a hex editor a…

So curious if this added a trick to your debugging toolbox?

Re: War story: the hardest bug I ever debugged

#167

Post by Suppafly »

TrackerFF wrote: post #143

My hardest debug was actually not software related, it was my first car - late 80s VW Passat. The problem was that the battery would simply not charge, and I had to jump-start it every time I used it, or park at the top of a hill/street and start it rolling down. Bought a brand new battery, but the problem persisted. Started looking at all the various parts in the car, that were connected to the electrical system. To…

That's the worst when you buy a new part and it still doesn't fix it, you rarely think that the new part could be bad, especially something like a battery that generally wouldn't have problems fresh from the store.

Re: War story: the hardest bug I ever debugged

#169

Post by Suppafly »

GarnetFloride wrote: post #153

I didn’t fix this bug but I did reproduce it so it could be fixed, but it took years. At one company I worked for we have an email archive and we were seeing an uptick in customers having issues with deleting expired emails. Most companies have a retention policy of about 7 years, and the company was now 10 years old and early customers were beginning to deleted old emails. But developers couldn’t find the bug, but r…

Keeping stuff past retention dates is such a high liability for companies, I'm surprised they didn't sue you guys to fix it faster.

Re: War story: the hardest bug I ever debugged

#170

Post by Suppafly »

kccqzy wrote: post #86
Taniwha wrote: post #69

In interviews I've never forced anyone to code, what I do is try to get them to tell me these sorts of war stories - I want to hear how you fixed it, why it was cooly bizarre, and I'm hoping for some enthusiasm when you talk about it. I couldn't always get people to talk this way, but people who did usually worked out well

You are selecting for the kind of person who always like to think about the war stories and brag about them.

This, whenever I get these sorts of questions on interviews I don't know how to answer, because my weirdest or hardest bug isn't something I've internalized as a war story, it was just another day.

It's just like those "what did you do when you had conflict with another employee" questions. I either worked it out with them like an adult or got our management involved and they worked it out for them. It's not some hero narrative I considered much past the time it happened.

Post reply on HN