Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

Post reply on HN
Print view View original

131–140 of 194 posts

Re: War story: the hardest bug I ever debugged

#131

Post by toast0 »

onli wrote: post #64
toast0 wrote: post #7

Earlier quoted context omitted.

I mean... > It didn’t correspond to a Google Docs release. The stack trace added very little information. There wasn’t an associated spike in user complaints, so we weren’t even sure it was really happening — but if it was happening it would be really bad. It was Chrome-only starting at a specific release. That sounds like a Chrome bug. Or, at least, a bug triggered by a change in Chrome. Bisecting your code when the…

Not if it your job to solve the situation. It's not like it would be an easier task to dig into chrome and find the issue there.

If your job is to solve the situation, your best hope is to figure out what change caused it; understand that change; and then do whatever needs to be done.

In a large complicated application where a change to the environment revealed a crash, finding out what changed in the environment and thinking about how that affects the application makes a lot more sense than going back through application changes to see if you can find it that way.

Once you figure out what the problem is, sure you can probably fix it in the application or the environment, and fixing the application is often easier if the environment is Chrome. But chrome changed and my app is broken means look at the changes in Chrome and work from there.

Re: War story: the hardest bug I ever debugged

#132

Post by gsck »

efortis wrote: post #20
BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

Same here, we had an IE8 bug that prevented the initial voice over of the screen reader (JAWS). No dev could reproduce it because we all had DevTools open.

I had a similar issue, worked fine when I was testing it on my machine, but I had dev tools open to see any potential issues.

Turns out IE8 doesn't define console until the devtools are open. That caused me to pull a few hairs out.

Re: War story: the hardest bug I ever debugged

#133

Post by ivraatiems »

aetimmes wrote: post #120

(disclaimer: I know OP IRL.) I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here: At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome dev…

Imagine if you weren't working at Google and were trying to convince the Chromium team you found a bug in V8. That'd probably be nigh-impossible.

One thing I notice is that Google has no way whatsoever to actually just ask users "hey, are you having problems?", a definite downside of their approach to software development where there is absolutely no communication between users and developers.

Re: War story: the hardest bug I ever debugged

#134

Post by kccqzy »

AnimalMuppet wrote: post #128
kccqzy wrote: post #86

Earlier quoted context omitted.

You are selecting for the kind of person who always like to think about the war stories and brag about them.

No, they're selecting for the kind of person who can tell a war story when asked . They're also selecting for the kind of people who had to debug something gnarly enough and different enough that it was memorable .

Some people are not natural story tellers. Telling a story is not a usual part of the job responsibility of a software engineer—we aren't novelists. Having a memorable debugging experience doesn't directly equate to having a good story to tell.

This is really the same issue with the promo culture we see at Big Tech companies: you end up promoting the people who are good at crafting promo packets i.e. telling stories about their work. There is certainly a good overlap between that and the people who do genuinely good work, but it's not a perfect overlap.

Personally I don't really mind it because I consider myself good at story telling. But as an interviewer I would never do that to a candidate because not everyone can tell good stories.

Re: War story: the hardest bug I ever debugged

#135

Post by seeingnature »

aetimmes wrote: post #120

(disclaimer: I know OP IRL.) I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here: At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome dev…

I'd love to see the rest of your postmortem template! I never thought about adding a "Where did we get lucky?" question.

I recently realized that one question for me should be, "Did you panic? What was the result of that panic? What caused the panic?"

I had taken down a network, and the device led me down a pathway that required multiple apps and multiple log ins I didn't have to regain access. I panicked and because the network was small, roamed and moved all devices to my backup network.

The following day, under no stress, I realized that my mistake was that I was scanning a QR code 90 degrees off from it's proper orientation. I didn't realize that QR codes had a proper orientation and figured that their corner identifiers handled any orientation. Then it was simple to gain access to that device. I couldn't even replicate the other odd path.

Re: War story: the hardest bug I ever debugged

#136

Post by gertlex »

jakevoytko wrote: post #15
gertlex wrote: post #13

Earlier quoted context omitted.

I also came to the comments to weigh in on my perception of how rough this was, but instead will ask: Regarding "exhausting 2-day brute-force grind": is/was this just how you like to get things done, or was there external pressure of the "don't work on anything else" sort? I've never worked at a large company, and lots of descriptions of the way things get done are pretty foreign to me :). I am also used to being abl…

The fatal error volume was so overwhelming that we didn't have any option but understanding the problem in perfect detail so that we could fix it if the problem was on our side, or avoid it if it was caused by something like our compiler or the browser. Our team also had a very grindy culture, so "I'm going to put in extra hours focusing exclusively on our top crash" was a pretty normalized behavior. After I left tha…

That makes sense. Thanks for the extra info!

Re: War story: the hardest bug I ever debugged

#137

Post by dgfitz »

AnimalMuppet wrote: post #130

I've told my personal worst here a couple of times. So this time I'm going to talk about a co-worker named Ed. On an embedded system, we had this bug that we couldn't find. It was around for a month or two. Random crashes that we couldn't reproduce, couldn't even debug. We started calling it "the phantom". Finally Ed said, "I think the phantom showed up after we made that change to the ethernet driver." We reverted i…

Interesting, at $dayjob we had (have?) a bug where something would fail, randomly, intermittently, maybe once every 3 months.

Pulled my hair out for a year, no progress/insights. Updated the driver for a device, haven't seen it since.

I hope the reverse-calendar debugging works for me!

Re: War story: the hardest bug I ever debugged

#138

Post by srejk »

seeingnature wrote: post #135
aetimmes wrote: post #120

(disclaimer: I know OP IRL.) I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here: At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome dev…

I'd love to see the rest of your postmortem template! I never thought about adding a "Where did we get lucky?" question. I recently realized that one question for me should be, "Did you panic? What was the result of that panic? What caused the panic?" I had taken down a network, and the device led me down a pathway that required multiple apps and multiple log ins I didn't have to regain access. I panicked and because…

The standard SRE one recommended by Google has a lucky section. We tend to use it to talk about getting unlucky too.

Re: War story: the hardest bug I ever debugged

#139

Post by gwern »

sgarland wrote: post #98
gpvos wrote: post #36

It seems to me that V8 had very bad unit tests if this wasn't caught before release. Making sure all operators act the same way when optimized and not is a no-brainer.

Maybe, but I can also understand someone rationalizing that they don’t need to test abs(), because what could possibly go wrong?

It sounds like their unit-tests cover abs(), but they weren't covering all of abs(), and were not reliably triggering the optimized codepath:

> When doing the refactoring, they needed to provide new implementations for every opcode. Someone accidentally turned Math.abs() into the identity function for the super-optimized level. But nobody noticed because it almost never ran — and was right half of the time when it did.

If it never was tested, plain and simple as that, then it couldn't matter that it 'almost never ran' or 'was right half the time'.

So the root problem here is that their test-suite neither exercised all optimized levels appropriately, nor flagged the omission as a fatal problem breaking 100% branch coverage (which for a simple primitive like abs you'd definitely want). This meant that they could break lots of other things too without noticing. OP doesn't discuss if the JS team dealt with it appropriately; one hopes they did.

Re: War story: the hardest bug I ever debugged

#140

Post by [deleted] »

seeingnature wrote: post #135
aetimmes wrote: post #120

(disclaimer: I know OP IRL.) I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here: At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome dev…

I'd love to see the rest of your postmortem template! I never thought about adding a "Where did we get lucky?" question. I recently realized that one question for me should be, "Did you panic? What was the result of that panic? What caused the panic?" I had taken down a network, and the device led me down a pathway that required multiple apps and multiple log ins I didn't have to regain access. I panicked and because…

[deleted]
Post reply on HN