Printf debugging is ok • Hacker News Board

Re: Printf debugging is ok

#121

Post by arter4 » Mon, Jan 06, 2025, 10:37 AM UTC

throwaway2037 wrote: post #113

seanmcdirmid wrote: post #64

Earlier quoted context omitted.

The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. And I used to be big on using debuggers, but now I’m horribly out of practice.

> they are often the 1 out of 100 runs kind Can you share an example? In my whole career, I have only seen one or two of them, but most of my work is CRUD type of stuff, not really gaming or systems programming where such a thing might happen.

Let's say your application talks to a database.

You reuse connections with a connection pool, but you accidentally reuse connections with different privileges and scopes. As a result, sometimes you get to read some data you shouldn't read and sometimes you don't.

Or, concurrency bugs.

You don't properly serialize transactions and sometimes two transactions overlap in time leading to conflicts.

Re: Printf debugging is ok

#122

Post by amszmidt » Mon, Jan 06, 2025, 10:43 AM UTC

bArray wrote: post #120

Honestly, it's just about doing what is easier at the time. Re-compiling an application in debug can be a pain sometimes, or getting back to a specific state within the application to inspect what is going on. Some have mentioned about hardware (which marches on with or without software running), but similarly part of a system where you only really control one sub-system/interface. My print statements normally come w…

Well, yes .. it is a label, and a label can be put almost anywhere. This is normal, expected C behavior and any standard conformant compiler would behave the same, not just GCC.

Re: Printf debugging is ok

#123

Post by chmod775 » Mon, Jan 06, 2025, 10:44 AM UTC

Why even obsess over how others ought to get stuff done? Judge the results.

Re: Printf debugging is ok

#124

Post by [deleted] » Mon, Jan 06, 2025, 11:28 AM UTC

kentonv wrote: post #49

If you have a reproducible test case that runs reasonably quickly, then I think printf debugging is usually just as good as a "real" debugger, and a lot easier. I typically have my test output log open in one editor frame. I make a change to the code in another frame, save it, my build system immediately re-runs the test, and the test log immediately refreshes. So when the test isn't working, I just keep adding print…

[deleted]

Re: Printf debugging is ok

#125

Post by GuB-42 » Mon, Jan 06, 2025, 11:37 AM UTC

I have changed my mind a couple of times on that subject. My opinion now is that printf debugging is not ok, but that it is not your fault.

printf debugging is a symptom of poor tooling. It is like saying that driving in nails with a rock is fine. It works, but the truth is that if you are using a rock, that's probably because you don't have a good hammer. And if on every job site, there are seasoned workers banging rocks like cavemen, maybe hammer manufacturers should take notice.

Re: Printf debugging is ok

#126

Post by overfl0w » Mon, Jan 06, 2025, 11:42 AM UTC

And then there are those rare cases where inserting a print or a new condition to use for conditional breakpoint forces the compiler to output slightly different code which does not produce the bug. Essentially this is similar to the Observer effect in quantum mechanics where the system is disturbed simply by observing it. Also the bug cannot be reproduced with optimizations disabled.

How are those cases debugged then? By enabling the debug symbols AND the optimizations and using the debugger, looking at the code and the disassembly side by side and trying to keep your sanity as the steps hop back and forth through the code. Telling yourself that the bug is real and it just cannot be reproduced easily because it depends on multiple factors + hardware states. Ah! I sometimes miss those kinds of bugs which make you question your reality.

Re: Printf debugging is ok

#127

Post by victorNicollet » Mon, Jan 06, 2025, 11:45 AM UTC

One of the hardest bugs I've investigated required the extreme version of debugging with printf: sprinkling the code with dump statements to produce about 500GiB of compressed binary trace, and writing a dedicated program to sift through it.

The main symptom was a non-deterministic crash in the middle of a 15-minute multi-threaded execution that should have been 100% deterministic. The debugger revealed that the contents of an array had been modified incorrectly, but stepping through the code prevented the crash, and it was not always the same array or the same position within that array. I suspected that the array writes were somehow dependent on a race, but placing a data breakpoint prevented the crash. So, I started dumping trace information. It was a rather silly game of adding more traces, running the 15-minute process 10 times to see if the overhead of producing the traces made the race disappear, and trying again.

The root cause was a "read, decompress and return a copy of data X from disk" method which was called with the 2023 assumption that a fresh copy would be returned every time, but was written with the 2018 optimization that if two threads asked for the same data "at the same time", the same copy could be returned to both to save on decompression time...

Re: Printf debugging is ok

#128

Post by overfl0w » Mon, Jan 06, 2025, 11:47 AM UTC

victorNicollet wrote: post #127

One of the hardest bugs I've investigated required the extreme version of debugging with printf: sprinkling the code with dump statements to produce about 500GiB of compressed binary trace, and writing a dedicated program to sift through it. The main symptom was a non-deterministic crash in the middle of a 15-minute multi-threaded execution that should have been 100% deterministic. The debugger revealed that the cont…

Those are the kind of bugs one remembers for life.

Re: Printf debugging is ok

#129

Post by victorNicollet » Mon, Jan 06, 2025, 11:54 AM UTC

overfl0w wrote: post #128

victorNicollet wrote: post #127

One of the hardest bugs I've investigated required the extreme version of debugging with printf: sprinkling the code with dump statements to produce about 500GiB of compressed binary trace, and writing a dedicated program to sift through it. The main symptom was a non-deterministic crash in the middle of a 15-minute multi-threaded execution that should have been 100% deterministic. The debugger revealed that the cont…

Those are the kind of bugs one remembers for life.

Indeed. Worst week of 2023 !

But I consider myself lucky that the issue could be reproduced on a local machine (arguably, one with 8 cores and 64GiB RAM) and not only on the 32 core, 256GiB RAM server. Having to work remotely on a server would have easily added another week of investigation.

Re: Printf debugging is ok

#130

Post by K0balt » Mon, Jan 06, 2025, 11:58 AM UTC

overfl0w wrote: post #126

And then there are those rare cases where inserting a print or a new condition to use for conditional breakpoint forces the compiler to output slightly different code which does not produce the bug. Essentially this is similar to the Observer effect in quantum mechanics where the system is disturbed simply by observing it. Also the bug cannot be reproduced with optimizations disabled. How are those cases debugged the…

Those kinds are almost never that the bug isn’t created unless you don’t put in the printf, it’s that the bug only causes the overt manifestation when the printf isn’t there. The actual bug is almost always there in both situations.

It’s almost never the compiler. It’s almost never an error in the bare metal.

Almost.