Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

Post reply on HN
Print view View original

121–130 of 194 posts

Re: War story: the hardest bug I ever debugged

#121

Post by gpvos »

sgarland wrote: post #98
gpvos wrote: post #36

It seems to me that V8 had very bad unit tests if this wasn't caught before release. Making sure all operators act the same way when optimized and not is a no-brainer.

Maybe, but I can also understand someone rationalizing that they don’t need to test abs(), because what could possibly go wrong?

Fair enough, it's busywork and easy to postpone. But code optimization is something that needs this kind of double-checking, so in the end you should have it for all opcodes, and then including the easy ones like abs isn't much extra work.

Re: War story: the hardest bug I ever debugged

#122

Post by rhaps0dy »

jffhn wrote: post #32
BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

>Though abs() returning negative numbers is hilarious. Math.abs(Integer.MIN_VALUE) in Java very seriously returns -2147483648, as there is no int for 2147483648.

Oh no, Pytorch does the same thing:

a = torch.tensor(-2*31, dtype=torch.int32) assert a == a.abs()

Re: War story: the hardest bug I ever debugged

#123

Post by stavros »

steveBK123 wrote: post #95
db48x wrote: post #90

Earlier quoted context omitted.

You found a 1× engineer; the worst engineer that can keep the job.

Over time we actually found him to be more of a -2x engineer, but thats another story edit: reminded me of the old joke A programmer gets sent to the store by his wife. His wife says, “Get a gallon of milk, and if they have eggs, get a dozen.” The programmer returns home with 12 gallons of milk and says, “They had eggs.”

Ah, the "and" ruins that joke for me by signifying a separate clause. I think the original is:

> A programmer gets sent to the store by his wife. His wife says, “Get a gallon of milk. If they have eggs, get a dozen.”

Re: War story: the hardest bug I ever debugged

#124

Post by bambax »

> When doing the refactoring, they needed to provide new implementations for every opcode. Someone accidentally turned Math.abs() into the identity function for the super-optimized level. But nobody noticed because it almost never ran — and was right half of the time when it did.

That's the perfect optimization: extremely fast, and mostly right -- probably more often than 50% if there are more positive numbers than negative ones.

Re: War story: the hardest bug I ever debugged

#125

Post by Scramblejams »

jakevoytko wrote: post #12
jonnycoder wrote: post #11

I’m not even close to being on par with other faang engineers but this is far from being a very difficult bug in my experience. The hardest bugs are the ones where the repro takes days to repro. But nonetheless the op’s tenacity is all that matters and I would trust them to solve any of the hard problems Ive faced in the past.

Hi, author here! At my job before Google I had to debug these kinds of bugs for our mobile robotics / computer vision stack, but I found them fun so they didn't feel "hard" per se. The most time-consuming one took a month on basically a camera-mounted computer vision system, where after an hour of use the system would start stuttering unusably. But the journey took us through heat throttling on 2009-era gaming laptop…

I’d read that blog post!

Re: War story: the hardest bug I ever debugged

#126

Post by pjc50 »

z3t4 wrote: post #80
steveBK123 wrote: post #76

Earlier quoted context omitted.

That is great QAing. It also speaks to why QA should be a real role in more orgs, rather than a shrinking discipline. Engineers LOVE LOVE LOVE to test the happy path. It's not even malice/laziness, it's their entire interpretation of the problem/requirements drives their implementation which then drives their testing. It's like asking restaurants to self-certify they are up to food safety codes.

If you do not follow the happy path something will break 100% of the time. That's why engineers always follow the happy path. Some engineers even think that anything outside the happy path is an exception and not even worth investigating. These engineers only thrives if the users are unable to switch to another product. Only competition will lead to better products.

> If you do not follow the happy path something will break 100% of the time

No, that means you're dealing with an early alpha, rigged demo, or some sort of vibe coding nonsense.

Re: War story: the hardest bug I ever debugged

#127

Post by pjc50 »

latexr wrote: post #55

> I do it a few more times. It’s not always the 20th iteration, but it usually happens sometime between the 10th and 40th iteration. Sometimes it never happend. Okay, the bug is nondeterministic. That’s an incorrect assumption. Just because your test case isn’t triggering the bug reliably, it does not mean the bug is nondeterministic. That is like saying the “OpenOffice can’t print on Tuesdays” is non deterministic b…

"Deterministic" is .. something of a moveable feast. We'd generally agree that "software is deterministic in that if you provide the same inputs to the same executable machine code it will return the same value", which is nearly always true unless someone is irradiating your processor or trying to voltage-glitch it.

But there's a lot hidden in "same inputs", because that includes everything that's an input to your program from the operating system. Which includes things like "time" (bane of reproduction), memory layout, execution scheduling order of multithreaded code, value of uninitialized memory, and so on.

> Another approach would have been to tweak their test case until they found a situation which reproduced the bug more or less often, trying to find the threshold that causes it and continuing to deduce from there.

Yes - when dealing with unknowns in a huge problem space it can be very effective to play hotter-colder and climb up the hill.

Re: War story: the hardest bug I ever debugged

#128

Post by AnimalMuppet »

kccqzy wrote: post #86
Taniwha wrote: post #69

In interviews I've never forced anyone to code, what I do is try to get them to tell me these sorts of war stories - I want to hear how you fixed it, why it was cooly bizarre, and I'm hoping for some enthusiasm when you talk about it. I couldn't always get people to talk this way, but people who did usually worked out well

You are selecting for the kind of person who always like to think about the war stories and brag about them.

No, they're selecting for the kind of person who can tell a war story when asked. They're also selecting for the kind of people who had to debug something gnarly enough and different enough that it was memorable.

Re: War story: the hardest bug I ever debugged

#129

Post by camtarn »

smrq wrote: post #103
efortis wrote: post #20

Earlier quoted context omitted.

Same here, we had an IE8 bug that prevented the initial voice over of the screen reader (JAWS). No dev could reproduce it because we all had DevTools open.

I can't remember the actual bug now, but one of my early career memories was hunting down an IE7 issue by using bookmarklets to alert() values. (Did IE7 even have dev tools?)

There was a downloadable developer toolbar for IE6 and IE7, and scripts could be debugged in the external Windows Script Debugger. The developer toolbar even told you which elements had the famous hasLayout attribute applied, which completely changed how it was rendered and interacted with other objects, which was invaluable.

Re: War story: the hardest bug I ever debugged

#130

Post by AnimalMuppet »

I've told my personal worst here a couple of times. So this time I'm going to talk about a co-worker named Ed.

On an embedded system, we had this bug that we couldn't find. It was around for a month or two. Random crashes that we couldn't reproduce, couldn't even debug. We started calling it "the phantom".

Finally Ed said, "I think the phantom showed up after we made that change to the ethernet driver." We reverted it, and the bug disappeared.

We never found the bug in the source code. But Ed debugged it using the calendar.

Post reply on HN