The fix ended up being one character -> change the priority of an ebpf tc filter from 0 to 1.
War story: the hardest bug I ever debugged
31–40 of 194 posts
Re: War story: the hardest bug I ever debugged
#32Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…
Math.abs(Integer.MIN_VALUE) in Java very seriously returns -2147483648, as there is no int for 2147483648.
Re: War story: the hardest bug I ever debugged
#33Where mere mortals can complain about Google product?
Re: War story: the hardest bug I ever debugged
#34That can't possibly be the hardest bug ever
Re: War story: the hardest bug I ever debugged
#35> It didn’t correspond to a Google Docs release. The stack trace added very little information. There wasn’t an associated spike in user complaints Where mere mortals can complain about Google product?
Then a high-ranked non-employee 'product expert' will be along presently to tell you that's not really a problem and to stop bothering the almighty google with such trivialities, your views are not important and they have millions of users, really why should they listen to you?
At least, that's been my experience.
Re: War story: the hardest bug I ever debugged
#36Re: War story: the hardest bug I ever debugged
#37Earlier quoted context omitted.
> which can be abused to rewrite the array’s length and enable further shenanigans. I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would…
>I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? This is my no doubt dumb understanding of what you can do, based on some funky stuff I did one time to mess with people's heads do the following const arr = []; arr[-1] = "hi"; console.log(arr) this gives you "-1": "hi" length: 0 which I figured is because really an array is just a speci…
You can do more and more in it and it's so fun, until it suddenly isn't anymore and dies.
Re: War story: the hardest bug I ever debugged
#38I'll clear my schedule. the best line of the piece.
> Then we called in our Tech Lead / Manager, who had a reputation of being a human JavaScript compiler. We explained how we got here, that Math.abs() is returning negative values, and whether she could find anything that we were doing wrong. After persuading her that we weren’t somehow horribly mistaken, she sat down and looked at the code. Her CPU spun up to 100%, and she was muttering in Russian about parse trees or something while staring at the code and typing into the debug console. Finally she leaned back and declared that Math.abs() was definitely returning negative values for negative inputs.
Re: War story: the hardest bug I ever debugged
#39FWIW: this type of bug in Chrome is exploitable to create out-of-bounds array accesses in JIT-compiled JavaScript code. The JIT compiler contains passes that will eliminate unnecessary bounds checks. For example, if you write “var x = Math.abs(y); if(x >= 0) arr[x] = 0xdeadbeef;”, the JIT compiler will probably delete the if statement and the internal nonnegative array index check inside the [] operator, as it can as…
> which can be abused to rewrite the array’s length and enable further shenanigans. I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would…
struct js_array {
uint64_t length;
js_value *values[];
}
Because after bound checks have been taken care of, loading an element of a JS array probably compiles to a simple assembly-level load like mov. If you bypass the bounds checks, that mov can read or write any mapped address.Re: War story: the hardest bug I ever debugged
#40Complaining about "slow to reproduce" and talking _seconds_. Dear, oh dear those are rookie numbers! Currently working a bug where we saw file system corruption after 3 weeks of automated testing, 10s of thousands of restarts. We might never see the problem again, even? Only happened once yet.