Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

31–40 of 194 posts

Re: War story: the hardest bug I ever debugged

#31
So far my record is 3 weeks. It was a hiesenbug triggered when two different ebpf based systems raced with each other. Ebpf is a great tool in the right place but is it ever a pain in the ass to debug.

The fix ended up being one character -> change the priority of an ebpf tc filter from 0 to 1.

Re: War story: the hardest bug I ever debugged

#32

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

>Though abs() returning negative numbers is hilarious.

Math.abs(Integer.MIN_VALUE) in Java very seriously returns -2147483648, as there is no int for 2147483648.

Re: War story: the hardest bug I ever debugged

#35

> It didn’t correspond to a Google Docs release. The stack trace added very little information. There wasn’t an associated spike in user complaints Where mere mortals can complain about Google product?

They have forums if you can find them, I think they call them 'communities', where you can complain.

Then a high-ranked non-employee 'product expert' will be along presently to tell you that's not really a problem and to stop bothering the almighty google with such trivialities, your views are not important and they have millions of users, really why should they listen to you?

At least, that's been my experience.

Re: War story: the hardest bug I ever debugged

#37
post #22

Earlier quoted context omitted.

> which can be abused to rewrite the array’s length and enable further shenanigans. I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would…

>I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? This is my no doubt dumb understanding of what you can do, based on some funky stuff I did one time to mess with people's heads do the following const arr = []; arr[-1] = "hi"; console.log(arr) this gives you "-1": "hi" length: 0 which I figured is because really an array is just a speci…

Javascript is the new Macromedia/Adobe Flash.

You can do more and more in it and it's so fun, until it suddenly isn't anymore and dies.

Re: War story: the hardest bug I ever debugged

#38

I'll clear my schedule. the best line of the piece.

Along with:

> Then we called in our Tech Lead / Manager, who had a reputation of being a human JavaScript compiler. We explained how we got here, that Math.abs() is returning negative values, and whether she could find anything that we were doing wrong. After persuading her that we weren’t somehow horribly mistaken, she sat down and looked at the code. Her CPU spun up to 100%, and she was muttering in Russian about parse trees or something while staring at the code and typing into the debug console. Finally she leaned back and declared that Math.abs() was definitely returning negative values for negative inputs.

Re: War story: the hardest bug I ever debugged

#39
post #22

FWIW: this type of bug in Chrome is exploitable to create out-of-bounds array accesses in JIT-compiled JavaScript code. The JIT compiler contains passes that will eliminate unnecessary bounds checks. For example, if you write “var x = Math.abs(y); if(x >= 0) arr[x] = 0xdeadbeef;”, the JIT compiler will probably delete the if statement and the internal nonnegative array index check inside the [] operator, as it can as…

> which can be abused to rewrite the array’s length and enable further shenanigans. I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would…

For example if arrays were implemented like this (they're not)

    struct js_array {
        uint64_t length;
        js_value *values[];
    }
Because after bound checks have been taken care of, loading an element of a JS array probably compiles to a simple assembly-level load like mov. If you bypass the bounds checks, that mov can read or write any mapped address.

Re: War story: the hardest bug I ever debugged

#40
post #27

Complaining about "slow to reproduce" and talking _seconds_. Dear, oh dear those are rookie numbers! Currently working a bug where we saw file system corruption after 3 weeks of automated testing, 10s of thousands of restarts. We might never see the problem again, even? Only happened once yet.

If it only happened once... it might be the final category of bugs where nothing you can do will fix it. Cosmic ray bit flipping bug. Which is something your software needs to be able to work around, or in this case, the file system itself... unless you're actually working on the file system itself, in which case, I wish you good luck.
Post reply on HN