Reverse Debugging and Other Stories
bogdandeac.com
Reverse Debugging and Other Stories
1–10 of 15 posts
Re: Reverse Debugging and Other Stories
#2These are called Time Travel Debuggers or Back in time debuggers: https://twitter.com/debugagent/status/1560204618801586179
I think they have so much potential but in my experience, they are still so far off from mass market appeal. rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to desktop Linux machines. Valuable, but not as valuable as some other options. Plus, using GDB is. Ugh.
There are amazing tools like Jive which is absolutely... Wow. It generates UML diagrams representing actual code execution. That's fantastic!
If you can get it to run your code base. I couldn't get anything beyond a hello world. Replay is pretty impressive, but front-end issues don't have as much state as other issues and are typically easier to track. It isn't great with long recordings so you can't just "leave it running".
I hope this field matures. Right now my hope is that vendors will steal good ideas and incorporate them into the IDEs such as IntelliJs stream debugger.
Re: Reverse Debugging and Other Stories
#3These are called Time Travel Debuggers or Back in time debuggers: https://twitter.com/debugagent/status/1560204618801586179 I think they have so much potential but in my experience, they are still so far off from mass market appeal. rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to desktop Linux machines…
"Time travel debugging" seems to be taking off as the term - "reverse debugger" was around for a long time but doesn't seem to have taken root. It has fixed the conversation of "We make a reversible debugger", "Ah so you put the bugs in", though!
> rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to desktop Linux machines.
Hopefully we'll see cloud environments find a way to expose more of the hardware performance counter stuff by default. The alternative is to use an instrumentation-based system that doesn't need access to performance counters. That said, as with machine-level virtualisation, it'd be nice to see the CPU vendors provide more explicit hardware support over time to make implementation easier / higher performance than current approaches.
> Valuable, but not as valuable as some other options. Plus, using GDB is. Ugh.
Looks like there's some support / guidance for using an IDE interface to rr's GDB - https://github.com/rr-debugger/rr/wiki/Using-rr-in-an-IDE
There's also Pernosco https://pernos.co/ which is a cloud service (hosted versions possible) from the original developers of rr. It post-processes uploaded rr traces into a database. This post-processing is compute-intensive but happens on their beefy worker machines. Once it's done, they can retrieve program state across time almost instantly - their website is worth checking out for some examples.
At Undo (disclaimer: where I work) we've got a VS Code plugin that's being very actively developed https://marketplace.visualstudio.com/items?itemName=Undo.udb to make it easier to expose the time travel stuff in our LiveRecorder product.
> Replay is pretty impressive, but front-end issues don't have as much state as other issues and are typically easier to track. It isn't great with long recordings so you can't just "leave it running".
Are you referring to replay.io here? (I was not sure whether you were talking about a replay behaviour in Jive - this is the risk of company / product names that are verbs, a problem we also have!)
I've not used their stuff but the debug interface they provide looks really intuitive. They support recording Node so I presume they can do some backend stuff too? (Assuming it's in JS).
Looks like they want to apply their techniques to other languages eventually.
Re: Reverse Debugging and Other Stories
#4There are some points in this article that I don't see mentioned often and are really important benefits.
"Story 1: What if we could let the bug happen and drink a coffee (or more)?" - the fact you can just leave the recorder running until your flaky issue has reproduced is really powerful. Once captured, you can debug at leisure, knowing invisible goblins won't make the problem vanish again.
"Story 2: Let’s print, but not build" - the `dprintf` command in GDB is so cool. Being able to tag additional print commands into the code without having to rebuild is powerful. You can do this iteratively with time travel debug, since you know you're running on the same trace every time.
Re: Reverse Debugging and Other Stories
#5The challenge of reverse debuggers is knowing when it should dispose of the recording; otherwise, if the debugger keeps recording the full sequence of states and actions, it would cause a memory leak and/or unbounded growth in disk usage. Such debugger would not be able to sustain a DoS attack if debugging in production (it would spam the debugger with lots of different states generated by the attacker); in fact, it would likely make it easier for an attacker to DoS a service as it would consume computational resources.
Re: Reverse Debugging and Other Stories
#6This reminds me of a program I wrote whose entire input comes from a blockchain. Everything that the program does is in response to some complex sequences of events which occurred on a blockchain.
This means that my program is deterministic since I can replay any part of the blockchain history and it will produce exactly the same results/output on each run. If a bug happens, I can replay the bug as many times as I like, exactly as it happened until I understood the cause of the issue and fixed the code - I can even go back and re-process the entire blockchain history to make sure that the new code changes didn't break any previously supported functionality.
That makes for a great debugging experience, I'm almost hoping that my project will hit a bug so that I can have fun debugging it. I only had to debug it once in the last 12 months because of a usability/UX issue.
This approach of writing deterministic code seems to produce much more robust code too. It requires a fair amount of work to weed out non-determinism from the code though (I.e. async logic running in parallel is the main challenge).
Re: Reverse Debugging and Other Stories
#7The challenge of reverse debuggers is knowing when it should dispose of the recording; otherwise, if the debugger keeps recording the full sequence of states and actions, it would cause a memory leak and/or unbounded growth in disk usage. Such debugger would not be able to sustain a DoS attack if debugging in production (it would spam the debugger with lots of different states generated by the attacker); in fact, it…
> if the debugger keeps recording the full sequence of states and actions,
It's worth noting that the main time travel debugging technologies don't keep the state of every operation performed, only the non-deterministic ones (i.e. where new information flows into the recorded program). A compute-intensive simulation can run a few days and generate little-to-no state that needs recording.
Your point stands, though, that a malicious attacker could do many non-deterministic operations (e.g. read data in from a socket). This can be solved - relatively easily - with a circular logging approach is workable to address this. i.e. discard chunks of older history to free up space as you go.
(Whether individual time travel debuggers support that is, as far as I'm aware, more a case of balancing demand for the feature against engineering effort.)
> Such debugger would not be able to sustain a DoS attack if debugging in production (it would spam the debugger with lots of different states generated by the attacker); in fact, it would likely make it easier for an attacker to DoS a service as it would consume computational resources.
In-production use could potentially be very useful if that's where some weird flaky issue occurs, so this is worth worrying about. It's definitely true that you've got to have the resources / settings to not overwhelm your machine. The other thing is that adding a complex layer of additional software risks adding additional exploits into the mix.
If your production issue is at a customer site or in a semi-trusted context (corporate network service) - which for some software is the norm - then I think that makes the threat model a bit less scary.
Otherwise, I guess it's a case of sandboxing the software within other measures and maybe spawning a separate instance that's recorded so you've still got "normal" instances running in parallel?
Re: Reverse Debugging and Other Stories
#8These are called Time Travel Debuggers or Back in time debuggers: https://twitter.com/debugagent/status/1560204618801586179 I think they have so much potential but in my experience, they are still so far off from mass market appeal. rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to desktop Linux machines…
"Time travel debugging" seems to be taking off as the term - "reverse debugger" was around for a long time but doesn't seem to have taken root. It has fixed the conversation of "We make a reversible debugger", "Ah so you put the bugs in ", though! > rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to deskt…
I looked at Pernosco but it's obviously not as useful without the recording.
Both Undo and Pernosco aren't useful on a Mac.
Yes I was talking about replay.io.
I agree that these tools have huge potential. I think they have a chance to make a breakthrough. But to cross that chasm into mass market we need something that's easier to use. I had hopes for video bug too. The interface seems promising initially but the developer experience is just awful.
I agree that instrumentation is probably the way to go. Especially for VM code.
Re: Reverse Debugging and Other Stories
#9The challenge of reverse debuggers is knowing when it should dispose of the recording; otherwise, if the debugger keeps recording the full sequence of states and actions, it would cause a memory leak and/or unbounded growth in disk usage. Such debugger would not be able to sustain a DoS attack if debugging in production (it would spam the debugger with lots of different states generated by the attacker); in fact, it…
Never use these things in production. They slow your servers very noticeably. It's OK for development. QA and even for unit test runs (which is an amazing use case for flaky tests). But production is a completely different ballgame.
Re: Reverse Debugging and Other Stories
#10Anybody know any good resources on scripting the debugger? It seems like you should be able to use some kind of concolic execution approach + fuzzing to do something like a binary search between forward and backward breakpoints to automatically find likely bug-containing regions. Or maybe union-find to partition the code into some (minimal) quite-buggy/(maximal) less-buggy sets of code regions.
Eg if you could get it to figure out the random values will probably always lie within some interval, you might be able to make it figure out “there is a path from to to ”
Additionally it seems really powerful to be able to introspect a program from outside. I run into situations sometimes where I would like to be able to make a realtime statistical gui of the internal state of some 3rd party code without necessarily butchering its codebase, seems like I could just write a little python to sample the local variables from within gdb then have that serve from an http server.
Or if I want to extract some library internal scratch calculations that are private or get cleaned up at the end of some long computation, I’d much rather write 10 lines of Python to copy and paste the data I want than attempt to refactor a copy of the codebase. Learning enough about it to predict the side-effects of my modification could get quite tedious.
Not sure of the feasibility of this, or how well such an approach will work with 1000x the lines of code in the real world, however :D