Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You say “cave dweller debugging”, I say debug logging

sicpers.info

Post reply on HN
Print view View original

71–80 of 139 posts

Re: You say “cave dweller debugging”, I say debug logging

#71

Post by jayd16 »

geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

> If you are doing anything involving multiple processes, or distribution, debuggers are useless

How so? Can't you just attach to each process? Do you mean at production scale?

Re: You say “cave dweller debugging”, I say debug logging

#73

Post by pradn »

geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

At the level of single process, on one machine, I've found a mix of debug logs, trace points, conditional breakpoints, and the ability to execute arbitrary code when at a breakpoint to be super helpful in debugging complex code with many layers. I was working on a compiler of sorts. The only time I had access to all of these was when working on C# in Visual Studio. Being able to run arbitrary code at the breakpoint-paused stack is incredibly useful.

With multiple processes or machines, it's helpful to pass around a trace ID and then have all the debug logs print the trace ID + timestamp at each event. This is a rather manual process; I wish there were automatic ways of doing this. At Google, the best we get is filtering logs by the RPC being processed at the time; but this ends at a particular process. Cross-process RPC traces are mostly sampled "stack traces"; you don't get to inject custom debug messages in those.

Re: You say “cave dweller debugging”, I say debug logging

#74

Post by pwdisswordfish9 »

AcerbicZero wrote: post #66

I messed up pretty badly when I learned IT. To be fair, the Army isn't a great IT teacher, but I started my scripting life with Powershell ( gasp ) and worse, Powershell Integrated Scripting Environment ( double gasp ). They had an awesome thing called the "Powershell scripting games" and I managed to convince the government agency I was working for that counted as work for a week or two. Downsides of powershell asid…

> REPLit is amazing when you just get pissed at a random python script that's spitting out junk and you want to figure out how a specific function works

How does Replit in particular help with this (vs anything else that lets you run Python, including /usr/bin/python?)

Re: You say “cave dweller debugging”, I say debug logging

#75

Post by khalladay »

dkjaudyeqooe wrote: post #46

I find debuggers almost entirely useless for two main reasons: (1) 99% of debugging is working out where the error is and (2) the answer to the question "how did this come to be?" usually has an answer involving several bits of code or data far away from each other.

Don't data breakpoints help with both of these things?

Re: You say “cave dweller debugging”, I say debug logging

#76

Post by m3047 »

I seldom use debuggers in the stereotypical way anymore. About the only time I use them is when I'm stepping through the code with some other person.

Dynamic instrumentation is key to things that don't scale naively, and surprisingly reusable (if left intact).

Debug logging is a baseline. If it's too verbose for monitoring a certain task, I've become a fan of counters being incremented in e.g. loops or other code points and printing them out at the end: the counts (especially zeros or bigger than anything else), their ratios and which paths were even run often turn out to be fruitful for statistical analysis when oddities are seen and this is especially true with any kind of data munging.

For services which run continually, have configurable printing for statistics. Or have a console which can examine dynamic structures. Or both.

Last winter I couldn't wrap my head around something Zeek was doing, so I ran it under Frida.

Any time I find myself using a debugger in the normal fashion, I find that I ask myself "should you be writing a test?" and not everybody can write tests at that level. In fact, I think running an interactive debugger is a good way to waste resources when you don't wanna write a test. I don't have much sympathy.

Re: You say “cave dweller debugging”, I say debug logging

#77

Post by geophile »

jayd16 wrote: post #71
geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

> If you are doing anything involving multiple processes, or distribution, debuggers are useless How so? Can't you just attach to each process? Do you mean at production scale?

Even in development.

You could, I suppose, but I'm not sure why you would want to. Going back and forth to two debuggers (at least) seems like torture. And with multiple processes, timing and synchronization issues could make interactive debugging a real nightmare. Why? Why would you do this?

Interactive debugging doesn't scale in any dimension.

Re: You say “cave dweller debugging”, I say debug logging

#78

Post by SleepyMyroslav »

Couple of anecdata to cool off fans of logging.

Imagine you develop in multiple environments and in prod costs of keeping logs can only let you have few percent of it. Guess what would be there? Right, most spammy message that no one of team leads would recognize as useful. Checking what actually is in logs in prod env is a hard work and usually no volunteers.

Logs can mask problems especially concurrency related. I dont want to give examples here to protect innocent but i can say in 20 years i have seen cases.

Logs can be huge pain in the neck of your performance analysis team. Especially if they have to use instrumented environment for their work instead of prod. Even "disabled" things that don't produce output might still be doing enough work to make any prod and qc environment results completely not comparable.

Logs can be pain in the neck of security and risk teams if you happen to put your code into untrusted hands. Logs often used for reverse engineering your smart solutions with ease. My particular experience with game clients is especially bad in this regard.

PS. My personal best logs in development let you track single element of data from a whole lot of others. It takes time and lots of work to design such logging in complex systems. I have not met many systems done like that. People tend to just spam everything and hope for the best with text filtering. Then the stories of crashing text editors while opening giga sized logs are born xD.

just my 2c.

Re: You say “cave dweller debugging”, I say debug logging

#79

Post by geophile »

10000truths wrote: post #67
geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

Working with gdb is quite the learning curve. Luckily, it’s mostly unnecessary, except in the rare case where you actually need register-level debugging or hardware watchpoints. But it can still be done if you’re willing to suffer through the docs. For debugging multithreaded code, you can run a gdb script in batch/non-interactive mode, and direct the gdb output to some out-of-band channel, like a file or other termi…

... or you could just log.

I don't understand the extreme attachment to interactive debugging that I see in some of the comments here. It's like people are so attached to this idea that they will do anything to make it work as each complication is added.

Furthermore, the amount of attention and manual labor needed to use a debugger effectively is just staggering. One mistake, e.g. you step over a function you should have stepped into, and you've got to start over.

Or you could just log.

Yes, writing logging code is no fun. But you don't even have to do it all at once. In fact, you probably shouldn't. I often found that I didn't know what to log until I had to investigate some problems turned up in integration testing. So I added logging to solve the problem, and then benefitted from it forever.

Re: You say “cave dweller debugging”, I say debug logging

#80

Post by bckr »

dionian wrote: post #22

You can't debug a program that ran in the past, but you can read its info-level logs.

An idea that I've been kicking around is to, in Python codebases, have a @log decorator that first logs the name of the function, all of the parameters it's called with and a function call id, then the name of the function, its return value, and the same function call id. This can be applied to as many of the functions in the codebase as necessary to provide "replayability". Haven't got the chance to try it out yet.
Post reply on HN