Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

War story: the hardest bug I ever debugged

clientserver.dev

Post reply on HN
Print view View original

91–100 of 194 posts

Re: War story: the hardest bug I ever debugged

#91

Post by artimaeis »

latexr wrote: post #55

> I do it a few more times. It’s not always the 20th iteration, but it usually happens sometime between the 10th and 40th iteration. Sometimes it never happend. Okay, the bug is nondeterministic. That’s an incorrect assumption. Just because your test case isn’t triggering the bug reliably, it does not mean the bug is nondeterministic. That is like saying the “OpenOffice can’t print on Tuesdays” is non deterministic b…

If I understood correctly - the Math.Abs() value would be positive roughly half the time, regardless of the steps taken to get there. That seems definitively nondeterministic.

Re: War story: the hardest bug I ever debugged

#92

Post by ramshanker »

Seems it is a story time thread. Here goes my strangest one.

Back in 2005, when I had only paid-by-cash internet cafe access to computer, one of the shopkeeper offered me free time on computer IF I typed and ran a 15 page of class 12 computer project printed on A4 sheets, onto the compiler. TurboC++. I gladly accepted the offer and typed things.

When I finished typing, taking out all the compile error, the program didn't work as expected. Few hours latter, I find out that 1 or 2 pages of printed source codes were not in original order. :-O . So had to swap code from one function to another to finally get it working. That was one hell of a lesson!

Shopkeeper must have sold that project to many students, and I got some Free internet access.

Re: War story: the hardest bug I ever debugged

#93

Post by yfontana »

BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

I think the hardest bug I've had to work on was just plain irreproducible. Once we exhausted all other ideas, we just attributed it to some sort of bit flip. Not a very satisfying resolution, but kinda cool to have encountered such an issue at least once.

Re: War story: the hardest bug I ever debugged

#94

Post by jiehong »

Worst debugging issues are always things I can't access directly, on top of being rare.

Think network appliance in the middle that don't log or not at the level you need (and sometimes they can't log what you need).

Those usually mean that no reproduction is possible, except in production or very close to it, with tools you don't always control.

Annoying ones are those of "This http request is sometimes slow", and chasing each boxes in the middle shows a new box that is supposed to be transparent but isn't, or some rare timing issues due to boxes interacting in a funny way.

Re: War story: the hardest bug I ever debugged

#95

Post by steveBK123 »

db48x wrote: post #90
steveBK123 wrote: post #83

Earlier quoted context omitted.

My favorite happy path developer.. and he was by far 10x worse than any engineer I worked with at this, did the following: Spec: allow the internal BI tool to send scheduled reports to the user Implementation: the server required the desktop front end of said user to have been opened that day for the scheduled reports to work, even though the server side was sending the mails Why this was hilariously bad - the only r…

You found a 1× engineer; the worst engineer that can keep the job.

Over time we actually found him to be more of a -2x engineer, but thats another story

edit: reminded me of the old joke

A programmer gets sent to the store by his wife. His wife says, “Get a gallon of milk, and if they have eggs, get a dozen.”

The programmer returns home with 12 gallons of milk and says, “They had eggs.”

Re: War story: the hardest bug I ever debugged

#97

Post by pavel_lishin »

steveBK123 wrote: post #83
z3t4 wrote: post #80

Earlier quoted context omitted.

If you do not follow the happy path something will break 100% of the time. That's why engineers always follow the happy path. Some engineers even think that anything outside the happy path is an exception and not even worth investigating. These engineers only thrives if the users are unable to switch to another product. Only competition will lead to better products.

My favorite happy path developer.. and he was by far 10x worse than any engineer I worked with at this, did the following: Spec: allow the internal BI tool to send scheduled reports to the user Implementation: the server required the desktop front end of said user to have been opened that day for the scheduled reports to work, even though the server side was sending the mails Why this was hilariously bad - the only r…

It honestly doesn't even sound like there was a happy path involved in his work.

Re: War story: the hardest bug I ever debugged

#98

Post by sgarland »

gpvos wrote: post #36

It seems to me that V8 had very bad unit tests if this wasn't caught before release. Making sure all operators act the same way when optimized and not is a no-brainer.

Maybe, but I can also understand someone rationalizing that they don’t need to test abs(), because what could possibly go wrong?

Re: War story: the hardest bug I ever debugged

#99

Post by donatj »

In the late 1990s my friend was writing a game for his TI-83 calculator in TI-Basic. He was running into this bizarre bug we boiled down to a single IF after almost an hour of back and forth over a single calculator. The IF was not behaving as you would expect and it made zero sense. In the early version of TI-Basic, operators are actually single symbols, rather than made from text characters. In frustration I delete the IF symbol, insert a new one, and fire the game up. Everything works, and my friend just about dies in disbelief. It's probably my most frustrating bug fix.

I was telling someone the story a couple years ago and they said the opcodes linked to the symbols could get corrupted or something like that.

Re: War story: the hardest bug I ever debugged

#100

Post by sesm »

BobbyTables2 wrote: post #5

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due t…

For stuff like this we used in-memory ring buffer logger that printed the logs on request. And it didn't save the strings, just necessary data bits and a pointer to formatting function. Writing to this logger didn't affect any timings.
Post reply on HN