Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You say “cave dweller debugging”, I say debug logging

sicpers.info

Post reply on HN
Print view View original

101–110 of 139 posts

Re: You say “cave dweller debugging”, I say debug logging

#101

Post by vkou »

AcerbicZero wrote: post #66

I messed up pretty badly when I learned IT. To be fair, the Army isn't a great IT teacher, but I started my scripting life with Powershell ( gasp ) and worse, Powershell Integrated Scripting Environment ( double gasp ). They had an awesome thing called the "Powershell scripting games" and I managed to convince the government agency I was working for that counted as work for a week or two. Downsides of powershell asid…

Look into Jupyter notebooks (or proprietary alternatives), for all the beauty of iterative coding, but in a better-than-powershell language.

Re: You say “cave dweller debugging”, I say debug logging

#102

Post by jsat »

supermatt wrote: post #2

No doubt there is a place (or indeed a need) for debug (or trace) logging in environments where you can’t simply pause execution. However, in development environments you would be foolish not to use the (vastly superior) tooling available. I think many developers shy away from debuggers simply because they haven't taken the time to understand how to use them effectively, whereas everyone knows how to write a log stat…

IMO the "vastly superior" comment here hurts the credo of your argument.

Re: You say “cave dweller debugging”, I say debug logging

#103

Post by saiya-jin »

smrtinsert wrote: post #92
geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

Add thread ids to your logs

I would be so screwed without this 10 times a month in one messaging app I've developed to integrate our archaic pl/sql core banking system with 21st century reality.

Luckily its not performance-oriented but more reliability-oriented so I can get away with more logging than usually necessary. It seems obsolete and over-the-top till it saves everyone's a*

Re: You say “cave dweller debugging”, I say debug logging

#104

Post by saltcured »

jayd16 wrote: post #71
geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

> If you are doing anything involving multiple processes, or distribution, debuggers are useless How so? Can't you just attach to each process? Do you mean at production scale?

This reminded me of a hail-mary use of debuggers in the late 1990s...

Debugging a show-stopper synchronization flaw in the bootstrapping of a parallel job, I had to tell the job system to launch each of 64 nodes wrapped in gdb wrapped in xterm with remote X display back to a laptop. It was something that had "worked in test" reliably, but that was always on a smaller number of nodes or simulating a larger number with time-sharing. It seemed to need real parallel hardware allocations to show up.

So, I was able to launch the job, allow all nodes to run until distributed deadlock, then interrupt and show all thread stacks on one debugger after another until I found the odd process which was out of phase with the rest. As soon as I saw the stacks, I had no further need for the debuggers. Just knowing the "impossible" state configuration happened was the necessary clue.

I have seen lots of satisfactory use of logs for diagnostics, but this is a case where I think the debuggers were almost essential. The debuggers gave a distributed state snapshot due to the deadlock-induced quiescence of the whole system. To have logs show the same snapshot would require a perfect arrangement of unbuffered logging so that the final state would be visible and not caught up in RAM of some or all of the stalled processes.

Re: You say “cave dweller debugging”, I say debug logging

#105

Post by treis »

the_af wrote: post #38
gulikoza wrote: post #7

I'm having the same issues... Although I'm not a full time developer, I do some small coding and I'm also sort of responsible for the dev team. I rarely use a debugger. I do most of debugging through logging. There are 2 advantages that I see: - it forces me to add the logs where they are needed to understand the flow of the application - it forces me to make those logs actually usable, readable and understandable De…

One thing that it takes some time to understand (and to teach to junior devs) is not only to add logs, but also useful logs. The stages of understanding go like this, in my experience: 1. Log nothing. 2. Upon being told to log things, write mostly useless logs: "Entering f() function", "leaving f() function". When f() fails, who knows what happened? 3. Upon being taught to add context so that logs can actually be use…

5. Get a ticket in your backlog to reduce logging because you've blown through your annual Datadog budget and it's only May.

Re: You say “cave dweller debugging”, I say debug logging

#106

Post by majormajor »

geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

I've had luck with debugging locally, in a single-process context, not expecting to reproduce the problem when unfamiliar with the codebase to get a good understanding of what's happening when in the code.

That tells me where I need to add metrics, logging, etc for the distributed deploy.

Re: You say “cave dweller debugging”, I say debug logging

#107

Post by icedchai »

I've been developing software for almost 3 decades (if you count the C programs I wrote in high school... longer if you consider Basic!) I can count on one hand the times I've needed to use a debugger to debug something. I prefer my cave-man logging.

I do find that I use Python's built-in debugger often during development, mostly to inspect objects as I'm writing code. I'll stick a breakpoint() in my script after an API call. It's often quicker than looking at Boto or Amazon docs, for example. But I never use it for actually debugging.

Re: You say “cave dweller debugging”, I say debug logging

#108

Post by lamontcg »

saltcured wrote: post #104
jayd16 wrote: post #71

Earlier quoted context omitted.

> If you are doing anything involving multiple processes, or distribution, debuggers are useless How so? Can't you just attach to each process? Do you mean at production scale?

This reminded me of a hail-mary use of debuggers in the late 1990s... Debugging a show-stopper synchronization flaw in the bootstrapping of a parallel job, I had to tell the job system to launch each of 64 nodes wrapped in gdb wrapped in xterm with remote X display back to a laptop. It was something that had "worked in test" reliably, but that was always on a smaller number of nodes or simulating a larger number with…

Sounds like all you really needed something like a signal handler that would dump the stack traces of all the threads.

I've used that a lot black-box debugging java processes in production.

I also wrote such a signal handler for a ruby app.

Re: You say “cave dweller debugging”, I say debug logging

#109

Post by nsm »

AcerbicZero wrote: post #66

I messed up pretty badly when I learned IT. To be fair, the Army isn't a great IT teacher, but I started my scripting life with Powershell ( gasp ) and worse, Powershell Integrated Scripting Environment ( double gasp ). They had an awesome thing called the "Powershell scripting games" and I managed to convince the government agency I was working for that counted as work for a week or two. Downsides of powershell asid…

Both LISP and Smalltalk were entirely based around having this kind of hands-on, interactive, full development environment available not in prod as well. In some sense what you are doing was the future people thought we would have, and it is a regression in some aspects that most production software runs in this bare bones system where someone can't stop it at an arbitrary point, and then have all the tools originally used to create that software available to them to observe the system.

Re: You say “cave dweller debugging”, I say debug logging

#110

Post by rhizome »

dmurray wrote: post #58
throwaway894345 wrote: post #49

Earlier quoted context omitted.

I don't know that there's an easy solution to this problem, but what I've seen Kubernetes and other Google-derived projects do is replace standard log levels with an increasing series of integers so you can log in increasing detail. This seems a little arbitrary (one developer's 8 might be another developer's 14 or something) but at least you can easily ask for increasingly more detail until you get what you need wit…

The increased detail also clutters the source, though. As I write this I realise that the "right" solution is that my IDE also folds away log calls of below some settable priority... Then I need a linter to prevent any code with side effects from running in a log line... Then I need to work in a language where that kind of static analysis is definitely possible... Maybe it's not so simple.

Create a logging method/function, llog() or something, with named parameters and whatnot.
Post reply on HN