Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

Post reply on HN
Print view View original

21–30 of 194 posts

Re: War story: the hardest bug I ever debugged

#21

Post by treyd »

gmueckl wrote: post #18
userbinator wrote: post #10

Earlier quoted context omitted.

The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently. The hardest one I've debugged took a few months to reproduce, and would only show up on hardware that only one person on the team had. One of the interesting things about working on a very mature product is that bugs tend to be very rare, but those rare…

That reminded me of a former colleague at the desk next to me randomly exclaiming one day that he had just fixed a bug he had created 20 years ago. The bug was actually quite funny in a way: it was in the code displaying the internal temperature of the electronics box of some industrial equipment. The string conversion was treating the temperature variable as an unsigned int when it was in fact signed. It took a brav…

This is a surprisingly common mistake with temperature readings. Especially when the system has a thermal safety power off that triggers if it's above some temperature, but then interprets -1 deg C as actually 255 deg C.

Re: War story: the hardest bug I ever debugged

#22

Post by saghm »

nneonneo wrote: post #14

FWIW: this type of bug in Chrome is exploitable to create out-of-bounds array accesses in JIT-compiled JavaScript code. The JIT compiler contains passes that will eliminate unnecessary bounds checks. For example, if you write “var x = Math.abs(y); if(x >= 0) arr[x] = 0xdeadbeef;”, the JIT compiler will probably delete the if statement and the internal nonnegative array index check inside the [] operator, as it can as…

> which can be abused to rewrite the array’s length and enable further shenanigans.

I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would somehow modify the length of the array rather than modifying a specific element in-place. Does JIT-compiled JavaScript not follow the usual JavaScript semantics that would normally happen when using a negative index, or are you describing something that would be used in combination with some other compiler bug (which honestly sounds a lot more severe even in the absence of an usual Math.abs implementation).

Re: War story: the hardest bug I ever debugged

#23

Post by laidoffamazon »

jakevoytko wrote: post #12
jonnycoder wrote: post #11

I’m not even close to being on par with other faang engineers but this is far from being a very difficult bug in my experience. The hardest bugs are the ones where the repro takes days to repro. But nonetheless the op’s tenacity is all that matters and I would trust them to solve any of the hard problems Ive faced in the past.

Hi, author here! At my job before Google I had to debug these kinds of bugs for our mobile robotics / computer vision stack, but I found them fun so they didn't feel "hard" per se. The most time-consuming one took a month on basically a camera-mounted computer vision system, where after an hour of use the system would start stuttering unusably. But the journey took us through heat throttling on 2009-era gaming laptop…

[flagged]

Re: War story: the hardest bug I ever debugged

#24

Post by ongy »

saghm wrote: post #22
nneonneo wrote: post #14

FWIW: this type of bug in Chrome is exploitable to create out-of-bounds array accesses in JIT-compiled JavaScript code. The JIT compiler contains passes that will eliminate unnecessary bounds checks. For example, if you write “var x = Math.abs(y); if(x >= 0) arr[x] = 0xdeadbeef;”, the JIT compiler will probably delete the if statement and the internal nonnegative array index check inside the [] operator, as it can as…

> which can be abused to rewrite the array’s length and enable further shenanigans. I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would…

This is after the jit.

I.e. don't think fancy language shenanigans that do negative indexing. But negative offset from the beginning of the array memory access.

When there's some inlining, there will be no function call into some index operator function

Re: War story: the hardest bug I ever debugged

#25

Post by bryanrasmussen »

saghm wrote: post #22
nneonneo wrote: post #14

FWIW: this type of bug in Chrome is exploitable to create out-of-bounds array accesses in JIT-compiled JavaScript code. The JIT compiler contains passes that will eliminate unnecessary bounds checks. For example, if you write “var x = Math.abs(y); if(x >= 0) arr[x] = 0xdeadbeef;”, the JIT compiler will probably delete the if statement and the internal nonnegative array index check inside the [] operator, as it can as…

> which can be abused to rewrite the array’s length and enable further shenanigans. I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would…

>I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative?

This is my no doubt dumb understanding of what you can do, based on some funky stuff I did one time to mess with people's heads

do the following const arr = []; arr[-1] = "hi"; console.log(arr) this gives you "-1": "hi"

length: 0

which I figured is because really an array is just a special type of object. (my interpretation, probably wrong)

now we can see that the JavaScript Array length is 0, but since the value is findable in there I would expect there is some length representation in the lower level language that JavaScript is implemented in, in the browser, and I would then think that there could even be exploits available by somehow taking advantage of the difference between this lower level representation of length and the JS array length. (again all this is silly stuff I thought and have never investigated, and is probably laughably wrong in some ways)

I remember seeing some additions to array a few years back that made it so you could protect against the possibility of negative indexes storing data in arrays - but that memory may be faulty as I have not had any reason to worry about it.

Re: War story: the hardest bug I ever debugged

#26

Post by selimthegrim »

treyd wrote: post #21
gmueckl wrote: post #18

Earlier quoted context omitted.

That reminded me of a former colleague at the desk next to me randomly exclaiming one day that he had just fixed a bug he had created 20 years ago. The bug was actually quite funny in a way: it was in the code displaying the internal temperature of the electronics box of some industrial equipment. The string conversion was treating the temperature variable as an unsigned int when it was in fact signed. It took a brav…

This is a surprisingly common mistake with temperature readings. Especially when the system has a thermal safety power off that triggers if it's above some temperature, but then interprets -1 deg C as actually 255 deg C.

I have seen this on Walgreens signs in suburban New Orleans, oddly enough.

Re: War story: the hardest bug I ever debugged

#27

Post by sandos »

Complaining about "slow to reproduce" and talking _seconds_. Dear, oh dear those are rookie numbers!

Currently working a bug where we saw file system corruption after 3 weeks of automated testing, 10s of thousands of restarts. We might never see the problem again, even? Only happened once yet.

Re: War story: the hardest bug I ever debugged

#28

Post by protocolture »

airstrike wrote: post #8
protocolture wrote: post #2

I had something like this once. Vendor provided an outlook plugin (ew) that linked storage directly in outlook (double ew) and contained a built in pdf viewer (disgusting) for law firms to manage their cases. One user, regardless of PC, user account or any other isolation factor, would reliably crash the program and outlook with it. She could work for 40 minutes on another users logged in account on another PC and re…

> Vendor provided an outlook plugin (ew) that linked storage directly in outlook (double ew) and contained a built in pdf viewer (disgusting) for law firms to manage their cases. I still don't understand how we've arrived at this state of affairs

Look I supported a few different legal platforms in that role and while I hated it, it was also the best.

Heres what a lawyer does:

1. They bill for time writing emails and on phone calls 2. They bill for time reviewing emails. 3. They bill for printing (and faxing if they are diehards) 4. They also bill for the time they are face to face with a human.

They also need to gather all the data, much of which flows in and out via email (or fax if they hate you) related to the case in a single space.

The sad state is that 80% of this can be achieved in outlook without much effort. Setting up an external application to capture all this shit is quite difficult, and generally requires mail to be run through it in some capacity. The question is, why reinvent the email client. (Sadly they reinvented the pdf reader) I have seen some lawfirms literally saving out every email as html, and uploading it with billing stats to a third party app. Its easier for me to support but the user experience can be awful.

The user already exists in Outlook, they already understand outlook. A few buttons in the ribbon (Mostly File this under X open case, time me, and bill this customer) make more sense from a user perspective.

From a support perspective its an absolute nightmare. Microsoft absolutely wont take a support case about an addon with shit memory management. And the addon provider will usually blame Microsoft.

Re: War story: the hardest bug I ever debugged

#29

Post by nneonneo »

saghm wrote: post #22
nneonneo wrote: post #14

FWIW: this type of bug in Chrome is exploitable to create out-of-bounds array accesses in JIT-compiled JavaScript code. The JIT compiler contains passes that will eliminate unnecessary bounds checks. For example, if you write “var x = Math.abs(y); if(x >= 0) arr[x] = 0xdeadbeef;”, the JIT compiler will probably delete the if statement and the internal nonnegative array index check inside the [] operator, as it can as…

> which can be abused to rewrite the array’s length and enable further shenanigans. I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would…

Normally, there would be a bounds check to ensure that the index was actually non-negative; negative indices get treated as property accesses instead of array accesses (unlike e.g. Python where they would wrap around).

However, if the JIT compiler has "proven" that the index is never non-negative (because it came from Math.abs), it may omit such checks. In that case, the resulting access to e.g. arr[-1] may directly access the memory that sits one position before the array elements - which could, for example, be part of the array metadata, such as the length of the array.

You can read the comments on the sample CVE's proof-of-concept to see what the JS engine "thinks" is happening, vs. what actually happens when the code is executed: https://github.com/shxdow/exploits/blob/master/CVE-2020-9802.... This exploit is a bit more complicated than my description, but uses a similar core idea.

Re: War story: the hardest bug I ever debugged

#30

Post by devsda »

userbinator wrote: post #10
BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently. The hardest one I've debugged took a few months to reproduce, and would only show up on hardware that only one person on the team had. One of the interesting things about working on a very mature product is that bugs tend to be very rare, but those rare…

During the time I was working on a mature hardware product in maintenance, if I think about the number of customer bugs we had to close due to being not-reproducible or were only present for a brief amount of time in specific setup, it was really embarassing and we felt like a bunch of noobs.
Post reply on HN