Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You say “cave dweller debugging”, I say debug logging

sicpers.info

Post reply on HN
Print view View original

111–120 of 139 posts

Re: You say “cave dweller debugging”, I say debug logging

#111

Post by dkjaudyeqooe »

vkou wrote: post #100
mjevans wrote: post #86

Earlier quoted context omitted.

Only if you already know the value(s) being watched. If you know that you can also add trace level debug statements on those items in the code.

You can, but it's usually easier to set a data breakpoint than to find every place where the data is modified, add sufficiently verbose logging statements there + upstream of there, rebuild, and search through logs.

When you have logs you can see where the anomalies are and trace them back to the source. Things are wrong all over the place, but where did it start?

Often I'll identify a wrong value, or bad object, and then search for that through the logs to see what is touching it or where it first appeared. If it's interacting with another value I'll search for that, maybe in a longer chain. Eventually you figure out where the interaction is going wrong, and more often than not it's a subtle error that looks right but isn't. You make a small change (the classic is an off-by-one error) and suddenly everything works perfectly.

Re: You say “cave dweller debugging”, I say debug logging

#112

Post by thewebcount »

JoeAltmaier wrote: post #14
supermatt wrote: post #2

No doubt there is a place (or indeed a need) for debug (or trace) logging in environments where you can’t simply pause execution. However, in development environments you would be foolish not to use the (vastly superior) tooling available. I think many developers shy away from debuggers simply because they haven't taken the time to understand how to use them effectively, whereas everyone knows how to write a log stat…

Hm. If its a timing issue, regular debuggers affect that and will perturb the experiment. Fast, lightweight logging can be a necessary tool.

I can think of a single case where the debugger changed timing enough to make a problem I was working on not show up. I can think of dozens of times when logging did, though. I guess it’s dependent on the type of problems you work on.

Re: You say “cave dweller debugging”, I say debug logging

#113

Post by shakna »

bckr wrote: post #80
dionian wrote: post #22

You can't debug a program that ran in the past, but you can read its info-level logs.

An idea that I've been kicking around is to, in Python codebases, have a @log decorator that first logs the name of the function, all of the parameters it's called with and a function call id, then the name of the function, its return value, and the same function call id. This can be applied to as many of the functions in the codebase as necessary to provide "replayability". Haven't got the chance to try it out yet.

    import logging
    import threading

    logger = logging.getLogger(__name__)

    def log(fn):
        def log_inner(*a, **kw):
           logger.info('%s(%s, %s) @ %s', fn, a, kw, threading.get_native_id())
           return fn(*a, **kw)
        return log_inner
It's pretty easy to grab all the arguments and thread, but grabbing the return would depend on you not mutating anything permanent, so that probably needs some special casing.

Re: You say “cave dweller debugging”, I say debug logging

#115

Post by stephenboyd »

SleepyMyroslav wrote: post #78

Couple of anecdata to cool off fans of logging. Imagine you develop in multiple environments and in prod costs of keeping logs can only let you have few percent of it. Guess what would be there? Right, most spammy message that no one of team leads would recognize as useful. Checking what actually is in logs in prod env is a hard work and usually no volunteers. Logs can mask problems especially concurrency related. I…

If checking your logs is hard work you should make it easier. I just click on my Datadog bookmark and select some filters to check logs.

Re: You say “cave dweller debugging”, I say debug logging

#116

Post by sitkack »

saiya-jin wrote: post #103
smrtinsert wrote: post #92

Earlier quoted context omitted.

Add thread ids to your logs

I would be so screwed without this 10 times a month in one messaging app I've developed to integrate our archaic pl/sql core banking system with 21st century reality. Luckily its not performance-oriented but more reliability-oriented so I can get away with more logging than usually necessary. It seems obsolete and over-the-top till it saves everyone's a*

Sometimes the debug log is real product.

Re: You say “cave dweller debugging”, I say debug logging

#117

Post by [deleted] »

supermatt wrote: post #2

No doubt there is a place (or indeed a need) for debug (or trace) logging in environments where you can’t simply pause execution. However, in development environments you would be foolish not to use the (vastly superior) tooling available. I think many developers shy away from debuggers simply because they haven't taken the time to understand how to use them effectively, whereas everyone knows how to write a log stat…

[deleted]

Re: You say “cave dweller debugging”, I say debug logging

#118

Post by sitkack »

lamontcg wrote: post #108
saltcured wrote: post #104

Earlier quoted context omitted.

This reminded me of a hail-mary use of debuggers in the late 1990s... Debugging a show-stopper synchronization flaw in the bootstrapping of a parallel job, I had to tell the job system to launch each of 64 nodes wrapped in gdb wrapped in xterm with remote X display back to a laptop. It was something that had "worked in test" reliably, but that was always on a smaller number of nodes or simulating a larger number with…

Sounds like all you really needed something like a signal handler that would dump the stack traces of all the threads. I've used that a lot black-box debugging java processes in production. I also wrote such a signal handler for a ruby app.

One thing I have used is drop into the Python debugger (also ipdb) from within a signal handler. Then I can run a kill, and have it drop into the debugger.

There are some of cute hacks to get a remote debugger which I haven't used in years.

https://github.com/sassoftware/epdb can start and connect to a remote debug instance

https://github.com/gotcha/ipdb

Re: You say “cave dweller debugging”, I say debug logging

#119

Post by invalidname »

geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

The problem is that in order to add a missing log in production you need to effectively redeploy. That isn't tenable for a polyglot production cluster. There's a rising field of developer observability which is specifically designed to provide polyglot cloud "debugging" capabilities. One of the capabilities is to inject new logs dynamically.

Re: You say “cave dweller debugging”, I say debug logging

#120

Post by pjmlp »

pradn wrote: post #73
geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

At the level of single process, on one machine, I've found a mix of debug logs, trace points, conditional breakpoints, and the ability to execute arbitrary code when at a breakpoint to be super helpful in debugging complex code with many layers. I was working on a compiler of sorts. The only time I had access to all of these was when working on C# in Visual Studio. Being able to run arbitrary code at the breakpoint-p…

On the Microsoft world you get that automated via ETW and in case of running on Azure, Application Insights.

With debug symbols you can even pinpoint those events into code.

The only competition I know to these workflows is using Java alongside Flight Recorder.

Post reply on HN