Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What a good debugger can do

werat.dev

Post reply on HN
Print view View original

151–160 of 208 posts

Re: What a good debugger can do

#151

Post by crabbone »

turtledragonfly wrote: post #141
crabbone wrote: post #136

Earlier quoted context omitted.

> It is simple and trustworthy. This sounds so naive for someone with 20+ years in the field... Linux, for decades, couldn't get logging to the point that it at least doesn't lose messages (the problem with tail / logrotate that is quite obvious once you think about it, but it took many years to give up the approach). I recently hit a bug where NVidia's driver abuses Linux kernel logging in some tight loop by spammin…

> Many things are simple, when your task is simple. Logging is just one of those things. I agree with much of what you said, and of course "logging" is not just a single point in the solution space — there is some function "troubleshooting_pain = f(your_project, your_approach)". I was trying to say that for "your_approach=logging" that function tends to return smaller values than for "your_approach=debugging", all ot…

Both interactive and declarative debuggers work better in distributed systems than logging because they can observe events as they happen, and don't need to recreate the order in which they happened from the records which are very hard to make chronologically consistent.

Things like EBPF (which may implement sort of a declarative debugger) are, perhaps the only tool you may hope to use in high volume and high frequency systems.

If I could only choose one technology used for software diagnostics, I'd choose debuggers over logging. Debuggers need more effort to develop them, and they aren't very good (yet), but they have potential. I don't believe that logging can be substantially improved to deal with difficult problems.

Re: What a good debugger can do

#152

Post by yencabulator »

nijave wrote: post #31
sebastianconcpt wrote: post #12

a good debugger supports different kinds of breakpoints, offers rich data visualization capabilities, has a REPL for executing expressions, can show the dependencies between threads and control their execution, can pick up changes in the source code and apply them without restarting the program, can step through the code backward and rewind the program state to any point in history, and can even record the entire pro…

>can step through the code backward and rewind the program state to any point in history This seems impossible for anything that modifies external state. Say 1. Open database connection 2. Commit data 3. Close database connection How would you rewind right before #2 if you've already completed step #3? You'd need the socket/connection you already closed Unless that means something more like "keep a running record of…

Let's say out loud the part that too often goes unstated:

You can rewind & replay a fixed execution of the program.

All the external interactions are recorded, and the replay can only literally replay what already happened.

You cannot change a variable or edit the code and branch off into a different execution path, that talks to the outside world differently; the rest of the world did not get rewound with the program.

Re: What a good debugger can do

#153

Post by turtledragonfly »

thewebcount wrote: post #150
turtledragonfly wrote: post #95

I use GDB almost daily, and used Visual Studio pretty deeply for many years (and still a little bit, nowadays), but I must say I am still a "printf debugging" aficionado (or better: real logging). I like many of the features that debuggers can provide, and I think this is a good article to set aspirational goals for what is possible. But my lived experience has generally been that it's a buggy, fuzzy, moving target i…

I think one problem I have with gdb and to a lesser extent lldb is that when using them with an IDE, it's just a janky connection between a command-line program and a GUI program. Back in my macOS 7-9 days, using something like CodeWarrior, or even Vusal C++ on Windows, stepping was so fast and smooth. It would respond instantaneously. I can now press the step button like 5 times and watch as it steps to each line. M…

You might want to look into the "TUI" mode in GDB. It's an ncurses-style interface, where it shows you the current code, and current line, and you can step along "visually." It is fast. Press Ctrl-L to re-draw the screen when the display gets messed up.

...

Then, you may notice that TUI mode steals certain keyboard commands, such as up/down arrow to scroll the source listing, rather than navigating command history.

Then, you might enable Vi-mode for GDB's readline, so you can use "j/k" to navigate command history, even in TUI mode. Plus Vi-mode is just better (:

Then, you may find that certain things don't work quite right in Vi-mode, because it's not the default and doesn't get as much testing. But you fuddle along because it's better than the alternative.

And thus you have arrived at my basic situation (:

Re: What a good debugger can do

#154

Post by thewebcount »

aflag wrote: post #138
justeleblanc wrote: post #119

Earlier quoted context omitted.

At least he did 23 years ago. Have debuggers evolved in the past 23 years?

They have a little. But his main point is that debuggers distract developers from seeing the problem as a whole rather than just understanding what's going on the vicinity of that problematic line. So, the debugger incentivises small, targeted fixes rather than bigger solutions for more systematic problems. That aspect of debuggers hasn't changed in 23 years, because that's the whole point of debuggers anyway. Some p…

> But his main point is that debuggers distract developers from seeing the problem as a whole rather than just understanding what's going on the vicinity of that problematic line. So, the debugger incentivises small, targeted fixes rather than bigger solutions for more systematic problems.

What a bizarre take. Just today I had to use a debugger to figure out where there was a deadlock in our program and why. No amount of printfs would have made that speedy. In the debugger I just stop the program once it's hung and look at all the threads to see which threads are holding which locks and which are waiting to obtain locks. From there I can now see it's a lock inversion because we weren't being careful about the order of taking our locks in a few places. That lead to doing a wider check to see if there were other similar cases, and whether we needed to rethink anything in particular.

The debugger generally gives me an overview of the whole program in a way that a bunch of printf statements can't. (Which isn't to say that printf doesn't have a place in debugging - it certainly does.)

Re: What a good debugger can do

#155

Post by cxr »

quelsolaar wrote: post #117
whatever1 wrote: post #113

Earlier quoted context omitted.

Debugging is the one reason I tolerate interpreted languages. The fact that you can pause execution and start writing commands to execute on the fly in the debug console is VERY POWERFUL.

As a C developer, I REALY want a interpreted C compiler with a crazy good debugger for this reason. On average I think C has really good debuggers (its a simple language that has been around for a long time) but there is so much more one could do. A few years ago i wrote a prototype called OPA, with some cool potential features. https://www.youtube.com/watch?v=pvkn9Xz-xks

"Interpreted C?" (2019) https://www.tuhs.org/pipermail/tuhs/2019-May/017790.html>

"The ups Debugger" https://ups.sourceforge.net/>

Re: What a good debugger can do

#156

Post by yencabulator »

aflag wrote: post #138
justeleblanc wrote: post #119

Earlier quoted context omitted.

At least he did 23 years ago. Have debuggers evolved in the past 23 years?

They have a little. But his main point is that debuggers distract developers from seeing the problem as a whole rather than just understanding what's going on the vicinity of that problematic line. So, the debugger incentivises small, targeted fixes rather than bigger solutions for more systematic problems. That aspect of debuggers hasn't changed in 23 years, because that's the whole point of debuggers anyway. Some p…

> But his main point is that debuggers distract developers from seeing the problem as a whole rather than just understanding what's going on the vicinity of that problematic line. So, the debugger incentivises small, targeted fixes rather than bigger solutions for more systematic problems.

Said differently by other old beards in this classic of Ken Thompson & Rob Pike: https://www.informit.com/articles/article.aspx?p=1941206

> When something went wrong, I'd reflexively start to dig in to the problem, examining stack traces, sticking in print statements, invoking a debugger, and so on. But Ken would just stand and think, ignoring me and the code we'd just written. After a while I noticed a pattern: Ken would often understand the problem before I would, and would suddenly announce, "I know what's wrong." He was usually correct. I realized that Ken was building a mental model of the code and when something broke it was an error in the model. By thinking about how that problem could happen, he'd intuit where the model was wrong or where our code must not be satisfying the model.

Re: What a good debugger can do

#157

Post by thewebcount »

ghosty141 wrote: post #18
Lichtso wrote: post #15

One style that haven't seen discussed is a hybrid / mixture of in-code (e.g. printf trace-logs) and out-of-code (using external tools like an attached debugger) debugging. What I do is I encode conditional breakpoints in the source code, (compile and) run the program with a debugger attached. The nice thing is you can have complex conditions using all kinds of functions from your surrounding code and you can have the…

Conditional breakpoints are supported for most dynamic languages. With static ones like c++ your way is the only practical way I believe

I program in a hybrid of C/C++/Objective-C/Objective-C++/Swift and can use conditional breakpoints in all of those languages in lldb and Xcode. There's nothing special about conditional breakpoints that makes them not work in compiled or static languages.

Re: What a good debugger can do

#158

Post by thewebcount »

turtledragonfly wrote: post #153
thewebcount wrote: post #150

Earlier quoted context omitted.

I think one problem I have with gdb and to a lesser extent lldb is that when using them with an IDE, it's just a janky connection between a command-line program and a GUI program. Back in my macOS 7-9 days, using something like CodeWarrior, or even Vusal C++ on Windows, stepping was so fast and smooth. It would respond instantaneously. I can now press the step button like 5 times and watch as it steps to each line. M…

You might want to look into the "TUI" mode in GDB. It's an ncurses-style interface, where it shows you the current code, and current line, and you can step along "visually." It is fast. Press Ctrl-L to re-draw the screen when the display gets messed up. ... Then, you may notice that TUI mode steals certain keyboard commands, such as up/down arrow to scroll the source listing, rather than navigating command history. T…

I shouldn't have to sacrifice either good UX or speed of single stepping on a modern machine. It says a lot that that's even a real suggestion in our industry.

Re: What a good debugger can do

#159

Post by werat »

donadigo wrote: post #102

The article provides a pretty good overview of the landscape of debuggers out there. I find myself using printf debugging whenever something is a dynamically evolving state and I need to monitor just the relevant bits of it and breakpoints when something is either crashing or requiring a deeper thought about how the code executes. That said, I still think there's quite a bit of improvement to be made, which is why I…

This looks great, very interesting approach! Augumenting the code of a running process using a scripting language is a cool idea, excited to see what comes next. I will definitely keep an eye on the project.

Re: What a good debugger can do

#160

Post by turtledragonfly »

crabbone wrote: post #151
turtledragonfly wrote: post #141

Earlier quoted context omitted.

> Many things are simple, when your task is simple. Logging is just one of those things. I agree with much of what you said, and of course "logging" is not just a single point in the solution space — there is some function "troubleshooting_pain = f(your_project, your_approach)". I was trying to say that for "your_approach=logging" that function tends to return smaller values than for "your_approach=debugging", all ot…

Both interactive and declarative debuggers work better in distributed systems than logging because they can observe events as they happen, and don't need to recreate the order in which they happened from the records which are very hard to make chronologically consistent. Things like EBPF (which may implement sort of a declarative debugger) are, perhaps the only tool you may hope to use in high volume and high frequen…

One thing that can be pretty nice which is kinda neither traditional debugging nor logging is DTrace (or similar). Basically event tracing on steroids. Maybe EBPF is in that vein? I don't have much personal experience with it, but I have heard some stories of good success on busy production systems.

I guess my (limited) experiences with distributed systems are different than yours. The notion of "pausing" the system to step through things interactively was usually untenable. Do you stop the one node that shows the issue, and let the others run, getting into who-knows-what shared state? Do you somehow attempt to stop them all, and hope/pray that they all are in the right state to make your cross-node analysis meaningful? This was mostly on Apache Spark, where parallelism was the name of the game. Maybe for some kind of long-running distributed system like Erlang it's a different story.

Post reply on HN