Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

Post reply on HN
Print view View original

181–190 of 194 posts

Re: War story: the hardest bug I ever debugged

#181

Post by steveBK123 »

glxxyz wrote: post #174
steveBK123 wrote: post #95

Earlier quoted context omitted.

Over time we actually found him to be more of a -2x engineer, but thats another story edit: reminded me of the old joke A programmer gets sent to the store by his wife. His wife says, “Get a gallon of milk, and if they have eggs, get a dozen.” The programmer returns home with 12 gallons of milk and says, “They had eggs.”

'Antiwork' - every hour they work uses up more than an hour of other engineers' time in questions, meetings, and later fixes.

Correct - they do bad work quickly which requires 2x the time on repairs

Re: War story: the hardest bug I ever debugged

#182

Post by winstonbwillis »

jffhn wrote: post #32
BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

>Though abs() returning negative numbers is hilarious. Math.abs(Integer.MIN_VALUE) in Java very seriously returns -2147483648, as there is no int for 2147483648.

[dead]

Re: War story: the hardest bug I ever debugged

#183

Post by BoorishBears »

decimalenough wrote: post #180
jason_tko wrote: post #89

Reminds me of the classic bug story where users couldn’t send emails more than 500 miles. https://web.mit.edu/jemorris/humor/500-miles

Crashes only on Wednesdays: https://gyrovague.com/2015/07/29/crashes-only-on-wednesdays/

I experienced "crashes after 16 hours if you didn't copy the mostly empty demo Android project from the manufacturer and paste the entire existing project into it"

Turned out there was an undocumented MDM feature that would reboot the device if a package with a specific name wasn't running.

Upon decompilation it wasn't supposed to be active (they had screwed up and shipped a debug build of the MDM) and it was supposed to be 60 seconds according to the variable name, but they had mixed up milliseconds and seconds

Re: War story: the hardest bug I ever debugged

#184

Post by smackeyacky »

zdc1 wrote: post #66

My worst bug had me using statistics to try and correlate occurrence rates with traffic/time of day, API requests, app versions, Node.js versions, resource allocations, etc. And when that failed I was capturing Prod traffic for examination in Wireshark... Turned out that Node.js didn't gracefully close TCP connections. It just silently dropped the connection and sent a RST packet if the other side tried to reuse it.…

Networking in node.js is maddeningly stupid and extremely hard to debug, especially when you're running it in something like Azure where the port allocation can be restricted outside of your control. It's bad enough that I wouldn't consider using node.js on any new project.

Re: War story: the hardest bug I ever debugged

#185

Post by saghm »

PhilipRoman wrote: post #39
saghm wrote: post #22

Earlier quoted context omitted.

> which can be abused to rewrite the array’s length and enable further shenanigans. I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would…

For example if arrays were implemented like this (they're not) struct js_array { uint64_t length; js_value *values[]; } Because after bound checks have been taken care of, loading an element of a JS array probably compiles to a simple assembly-level load like mov. If you bypass the bounds checks, that mov can read or write any mapped address.

Yeah, I understand all of that. I think my surprise was that you can access arbitrary parts of this struct from within JavaScript at all; I guess I really just haven't delved deeply enough into what JIT compiling actually is doing at runtime, because I wouldn't have expected that to be possible.

Re: War story: the hardest bug I ever debugged

#186

Post by saghm »

nneonneo wrote: post #29
saghm wrote: post #22

Earlier quoted context omitted.

> which can be abused to rewrite the array’s length and enable further shenanigans. I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would…

Normally, there would be a bounds check to ensure that the index was actually non-negative; negative indices get treated as property accesses instead of array accesses (unlike e.g. Python where they would wrap around). However, if the JIT compiler has "proven" that the index is never non-negative (because it came from Math.abs), it may omit such checks. In that case, the resulting access to e.g. arr[-1] may directly…

I understand the idea of the lack of a bounds check allowing access to early memory with a negative index, but I'm mostly struggling with wrapping my head around why the underlying memory layout is accessible in JavaScript in the first place. I hadn't considered the fact that the same syntax could be used for accessing arbitrary properties rather than just array indexes; that might be the nuance I was missing.

Re: War story: the hardest bug I ever debugged

#187

Post by saghm »

bryanrasmussen wrote: post #25
saghm wrote: post #22

Earlier quoted context omitted.

> which can be abused to rewrite the array’s length and enable further shenanigans. I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would…

>I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? This is my no doubt dumb understanding of what you can do, based on some funky stuff I did one time to mess with people's heads do the following const arr = []; arr[-1] = "hi"; console.log(arr) this gives you "-1": "hi" length: 0 which I figured is because really an array is just a speci…

You raise a good point that JavaScript arrays are "just" objects that let you assign to arbitrary properties through the same syntax as array indexing. I could totally imagine some sort of optimization where a compiler utilizes this to be able to map arrays directly to their underlying memory layout (presumably with a length prefix), and that would end up potentially providing access to it in the case of a mistaken assumption about omitting a bounds check.

Re: War story: the hardest bug I ever debugged

#188

Post by bobbylarrybobby »

jffhn wrote: post #32
BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

>Though abs() returning negative numbers is hilarious. Math.abs(Integer.MIN_VALUE) in Java very seriously returns -2147483648, as there is no int for 2147483648.

Rust does the same in release, although it panics in debug.

Re: War story: the hardest bug I ever debugged

#189

Post by 392 »

sa46 wrote: post #151
Cthulhu_ wrote: post #40

Earlier quoted context omitted.

If it only happened once... it might be the final category of bugs where nothing you can do will fix it. Cosmic ray bit flipping bug. Which is something your software needs to be able to work around, or in this case, the file system itself... unless you're actually working on the file system itself, in which case, I wish you good luck.

What layers of hardware can comic rays impact? Memory with ECC is largely safe, right? What about the L1 cache and friends?

Anything can fail, at any time. The best we can do is mitigate it and estimate bounds for how likely it is to mess up. Sometimes those bounds are acceptable.

Re: War story: the hardest bug I ever debugged

#190

Post by GianFabien »

voidifremoved wrote: post #78
steveBK123 wrote: post #77

Earlier quoted context omitted.

Yes ! I've dealt with complex issues that turned out to be vendor-swapped-hardware-woopsie which we spent over a month trying to solve in software before finally figuring it out. Part of it was difficulty of pinpointing the actual issue - fullness of drive vs throughput of writes. A lot of it was unfortunately organizational politics such that the system spanned two teams with different reporting lines that didn't co…

> A lot of it was unfortunately organizational politics The hardest bugs in my experience are those where your only source of vital information is a third party who is straight-up lying to you.

Sometimes it isn't outright lying. I have had the issues with hardware, API and SDK documentation being subtly different from the product as shipped. With hardware with a mixture of revisions, some conforming to doco and other differing and even their engineers not being clear about which is which.
Post reply on HN