You say “cave dweller debugging”, I say debug logging • Hacker News Board

Re: You say “cave dweller debugging”, I say debug logging

#121

Post by pjmlp » Sat, May 07, 2022, 8:48 AM UTC

10000truths wrote: post #67

geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

Working with gdb is quite the learning curve. Luckily, it’s mostly unnecessary, except in the rare case where you actually need register-level debugging or hardware watchpoints. But it can still be done if you’re willing to suffer through the docs. For debugging multithreaded code, you can run a gdb script in batch/non-interactive mode, and direct the gdb output to some out-of-band channel, like a file or other termi…

Debugging JNI is super easy in Eclipse and Netbeans.

Re: You say “cave dweller debugging”, I say debug logging

#122

Post by GnarfGnarf » Sat, May 07, 2022, 12:49 PM UTC

gary_0 wrote: post #50

GnarfGnarf wrote: post #41

Earlier quoted context omitted.

Yeah but... aren't you tempted to leave the debugging code, in case you might need it again later?

To me it seems like if it's really that important, it becomes error checking or assertion code, or the code should be refactored to better enforce invariants, or the debug code is spun off into unit tests. That way whatever problem I was trying to detect is checked automatically in the future. And I admit an aesthetic preference for less logging code scattered about. But I haven't found it very useful to barf lots of…

This is where you need to get creative: save the log as HTML, and use colors, tables, etc. to organize the data, catch error needles amongst the data haystack, etc.

I find that having lots of trace data is useful because at a glance, you can compare problem situations against normal profiles and patterns.

Re: You say “cave dweller debugging”, I say debug logging

#123

Post by GnarfGnarf » Sat, May 07, 2022, 12:56 PM UTC

gpderetta wrote: post #65

GnarfGnarf wrote: post #39

There is a time and place to step through the code with a debugger, and there are also situations where a log file can show the "big picture" and display information at a higher level. In the latter case, a log can show in minutes what would take hours to ascertain if stepping through line-by-line. My app formats the trace log as HTML tables, and uses color to signal errors or invalid cases. Makes debugging fun. Also…

> Tip: close & re-open the file for each write That seems very expensive. Why not just use line buffering or even flush from a signal handler?

Well... we wrote this when we started with Windows 3.1. Besides it's only used in special circumstances when we're tracking down a bug.

Re: You say “cave dweller debugging”, I say debug logging

#124

Post by geophile » Sat, May 07, 2022, 4:36 PM UTC

invalidname wrote: post #119

geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

The problem is that in order to add a missing log in production you need to effectively redeploy. That isn't tenable for a polyglot production cluster. There's a rising field of developer observability which is specifically designed to provide polyglot cloud "debugging" capabilities. One of the capabilities is to inject new logs dynamically.

And if you need to fix a bug in production you need to redeploy too.

Bugs and logs and testing are all related. You obviously (should) do unit testing, integration testing, system testing before production. As you do these things, you will find and fix bugs. Ideally, you would add logging to help you diagnose the bugs you are fixing. Congratulations, you now have logging in code known to have been buggy, which will serve you well in production if bugs not caught by your tests turn up.

But bugs in previously reliable code do slip through, and this gives rise to the need for previously unanticipated logging. If either of these things happens a lot, the problem is your testing leading up to production. In other words, if you often find yourself needing dynamically deployed logging, then the lack of this feature is not the problem you should be paying attention to.

Also, dynamically deployed logging is basically shipping untested code, no?

Re: You say “cave dweller debugging”, I say debug logging

#125

Post by saltcured » Sat, May 07, 2022, 5:51 PM UTC

lamontcg wrote: post #108

saltcured wrote: post #104

Earlier quoted context omitted.

This reminded me of a hail-mary use of debuggers in the late 1990s... Debugging a show-stopper synchronization flaw in the bootstrapping of a parallel job, I had to tell the job system to launch each of 64 nodes wrapped in gdb wrapped in xterm with remote X display back to a laptop. It was something that had "worked in test" reliably, but that was always on a smaller number of nodes or simulating a larger number with…

Sounds like all you really needed something like a signal handler that would dump the stack traces of all the threads. I've used that a lot black-box debugging java processes in production. I also wrote such a signal handler for a ruby app.

In a distributed parallel computing scenario, dumping stacks from 64 processes via a signal handler would have created 64 different files scattered in 64 different remote filesystems. Ironically, the purpose of our library was to bootstrap a distributed communication network and so could have easily supported gathering of remote diagnostic data _if it were not the part that deadlocked_...

But worse, we were debugging our library linked into someone else's application. So, we would have needed their cooperation to add signal handlers. And, we naively thought our library had already been sufficiently tested and did not anticipate the need for last-minute debugging when our user moved their application to a different computing resource that was not available during prior months of preparation and prototypes.

So, the ugly app-in-gdb-in-xterm with X over Internet addressed all that with adhoc instrumentation and communications that could be folded into the existing parallel, distributed job on short notice.

This was also in the days of pthreads in C. The other valuable use of gdb I recall was for memory watchpoints to help track down unexpected changes to certain data. The other tool we got a lot of use out of in those days was Purify to help audit for use of uninitialized memory and for memory leaks.

Re: You say “cave dweller debugging”, I say debug logging

#126

Post by geophile » Sat, May 07, 2022, 6:03 PM UTC

saltcured wrote: post #125

lamontcg wrote: post #108

Earlier quoted context omitted.

Sounds like all you really needed something like a signal handler that would dump the stack traces of all the threads. I've used that a lot black-box debugging java processes in production. I also wrote such a signal handler for a ruby app.

In a distributed parallel computing scenario, dumping stacks from 64 processes via a signal handler would have created 64 different files scattered in 64 different remote filesystems. Ironically, the purpose of our library was to bootstrap a distributed communication network and so could have easily supported gathering of remote diagnostic data _if it were not the part that deadlocked_... But worse, we were debugging…

> In a distributed parallel computing scenario, dumping stacks from 64 processes via a signal handler would have created 64 different files scattered in 64 different remote filesystems.

I was working on a cluster and often needed to do the same thing on all the nodes, run some database query or os command, and gather and integrate the results. I wrote yet another Python stream-objects-instead-of-strings shell (https://geophile.com/osh), which included features for distribution: Run the same command on all nodes, bring back the results, merge streams, gather files, distribute files, etc.

That was several years ago, and it wasn't a full shell. Since then, I've done an improved system that actually is a shell (https://marceltheshell.org, https://geophile.com/marcel).

Re: You say “cave dweller debugging”, I say debug logging

#127

Post by GeorgeTirebiter » Sat, May 07, 2022, 7:34 PM UTC

User23 wrote: post #97

geophile wrote: post #88

Earlier quoted context omitted.

That doesn't make sense to me. Complication is mainly a property of the software you're writing, not the language in which the software is written.

It's true that at the end of the day unnecessary complexity is largely a matter of programmer discipline or lack thereof. However, some languages give the programmer considerably more opportunities for unnecessary complexity than others. And some language/runtime combinations make writing correct software impossible. For example back in the day Perl wasn't properly reentrant, so any signal could possibly lead to unde…

Yes.

Extracting simplicity from complexity is hard, and that is why most systems are unnecessarily complex. And I mean: 'unnecessarily' in that there exists simpler systems that perform the same function with less complexity.

I'm late-stage career here, so I've seen (most of) it all. There seems to be additional complexity piled on 'just because' so often. Occasionally, I see a system and I can't think of a simpler way to do it. That's rare. I bet it's rare for you, too.

Re: You say “cave dweller debugging”, I say debug logging

#128

Post by JoeAltmaier » Sat, May 07, 2022, 7:37 PM UTC

thewebcount wrote: post #112

JoeAltmaier wrote: post #14

Earlier quoted context omitted.

Hm. If its a timing issue, regular debuggers affect that and will perturb the experiment. Fast, lightweight logging can be a necessary tool.

I can think of a single case where the debugger changed timing enough to make a problem I was working on not show up. I can think of dozens of times when logging did, though. I guess it’s dependent on the type of problems you work on.

Lots of ways to do logging. Create a circular buffer of tags and values at critical points - takes just a couple machine cycles to add an entry. When the problem occurs (break in the debugger) the table records the last N significant actions.

Re: You say “cave dweller debugging”, I say debug logging

#129

Post by icedchai » Sat, May 07, 2022, 9:07 PM UTC

invalidname wrote: post #119

geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

The problem is that in order to add a missing log in production you need to effectively redeploy. That isn't tenable for a polyglot production cluster. There's a rising field of developer observability which is specifically designed to provide polyglot cloud "debugging" capabilities. One of the capabilities is to inject new logs dynamically.

Not necessarily. I've been at some places where you just "do it live." We had dozens of worker nodes. An engineer would "reserve" one for debugging purposes, editing python code live, adding logs or prints as needed and restarting services. Worst case a small percentage of requests was lost, though generally they would be retried except in the most extreme cases. This risk was considered acceptable.

Obviously, depending on the business and maturity level of the company, this could be a big no-no. In general, I find those "wild west" environments more fun. Waiting for an hour long build, deploys, and "approvals" to add a couple of prints is a big drag.

Re: You say “cave dweller debugging”, I say debug logging

#130

Post by invalidname » Sun, May 08, 2022, 1:30 AM UTC

icedchai wrote: post #129

invalidname wrote: post #119

Earlier quoted context omitted.

The problem is that in order to add a missing log in production you need to effectively redeploy. That isn't tenable for a polyglot production cluster. There's a rising field of developer observability which is specifically designed to provide polyglot cloud "debugging" capabilities. One of the capabilities is to inject new logs dynamically.

Not necessarily. I've been at some places where you just "do it live." We had dozens of worker nodes. An engineer would "reserve" one for debugging purposes, editing python code live, adding logs or prints as needed and restarting services. Worst case a small percentage of requests was lost, though generally they would be retried except in the most extreme cases. This risk was considered acceptable. Obviously, depend…

For that you have developer observability which doesn't carry the HUGE gaping security risks or performance problems. But still lets you debug the cluster at scale.