Live data from Hacker News

Show HN: Leaping – Debug Python tests instantly with an LLM debugger

github.com

11–20 of 21 posts

Re: Show HN: Leaping – Debug Python tests instantly with an LLM debugger

#11
post #6

I thought of something similar these days but with a different approach - rather than settrace, it would use a subclass of bdb.Bdb (the standard library base debugger, on top of which Pdb is built) to actually have the LLM run a real debugging session. It'd place breakpoints (or postmortem sessions after an uncaught exception) to drop into a repl which allows going up/down the frame stack at a given execution point,…

I actually coded something very close to this and it worked surprisingly well: https://github.com/janpf/debuggAIr

Re: Show HN: Leaping – Debug Python tests instantly with an LLM debugger

#12
post #11
post #6

I thought of something similar these days but with a different approach - rather than settrace, it would use a subclass of bdb.Bdb (the standard library base debugger, on top of which Pdb is built) to actually have the LLM run a real debugging session. It'd place breakpoints (or postmortem sessions after an uncaught exception) to drop into a repl which allows going up/down the frame stack at a given execution point,…

I actually coded something very close to this and it worked surprisingly well: https://github.com/janpf/debuggAIr

Ooh, interesting - starred and going to dig into this later today!

Re: Show HN: Leaping – Debug Python tests instantly with an LLM debugger

#13
post #8
post #6

I thought of something similar these days but with a different approach - rather than settrace, it would use a subclass of bdb.Bdb (the standard library base debugger, on top of which Pdb is built) to actually have the LLM run a real debugging session. It'd place breakpoints (or postmortem sessions after an uncaught exception) to drop into a repl which allows going up/down the frame stack at a given execution point,…

So interestingly enough, we first tried letting GPT interact with pdb, through just a set of directed prompts, but we found that it kept hallucinating commands, not responding with the correct syntax and really struggling with line numbers. That's why we pivoted to just getting all the relevant data upfront GPT could need and letting GPT synthesize that data into a singular root cause. I think we're going to explore…

Interesting! Did you try the function calling API? I feel you with the line number troubles, it's hard to get something consistent there. Using diffs with GPT-4 isn't much better in my experience; I didn't extensively test that, but from what I did it rarely produced synctatically valid diffs that could just be sent to `patch`. One approach I started playing with was using tree-sitter to add markers to code and let the LLM specify marker ranges for deletion/insertion/replacement, but alas, I got distracted before fully going through with it.

In any case, I'll keep an eye on the project, good luck! Let me know if you ever need an extra set of hands, I find this stuff pretty interesting to think about :)

Re: Show HN: Leaping – Debug Python tests instantly with an LLM debugger

#15
post #14

On the reddit discussion, one user pointed out that email adresses were incidently collected. https://www.reddit.com/r/programming/s/lBfxL7f2KM

Was removed here: https://github.com/leapingio/leaping/commit/e42d5198abe48875...

Re: Show HN: Leaping – Debug Python tests instantly with an LLM debugger

#16
post #6

I thought of something similar these days but with a different approach - rather than settrace, it would use a subclass of bdb.Bdb (the standard library base debugger, on top of which Pdb is built) to actually have the LLM run a real debugging session. It'd place breakpoints (or postmortem sessions after an uncaught exception) to drop into a repl which allows going up/down the frame stack at a given execution point,…

I've done a manual version of this with chatgpt.

I had ipdb, told it to request any variables that I should look at, suggest what to do next, what it would expect - it was quite good, but took a lot of persuading, just having an LLM that was more tuned to this would be better.

Re: Show HN: Leaping – Debug Python tests instantly with an LLM debugger

#17
post #2

Nice work! I watched the demo and can see how it will generate fixes for you, which you then copy and paste into the editor. Perhaps you could consider automating this process like Aider[1] does, whereby you force the LLM to generate a git diff for the fix and automatically commit it. 1. https://github.com/paul-gauthier/aider

I've come across Aider before and was trying to remember the name just the other day - thanks!!

Re: Show HN: Leaping – Debug Python tests instantly with an LLM debugger

#18
post #10

> To achieve this, we first instrument the test using sys.settrace (or, on versions of python >3.12, the far better sys.monitoring!) to keep a history of all the functions that were called, along with the calling line numbers. We then re-run the test and use AST parsing to find all the variable assignments and keep track of those changes over time. We also use AST parsing to obtain the source code for these functions…

Interesting! I think you’re right in saying that the middleman you’re talking about has to be really good for something like this to actually be useful, especially for people very comfortable with debugging tools + their codebase. If I understand correctly, you’re saying that the most productive tool for you would be one that can present you with more relevant data, in a structured way (Redux, etc). At first, we actu…

:shrug: I'm one person on the Internet, so if using the LLM makes it work better for more of your users, go with that.

I do think data filtering and visualization is an important value add, Pytrace looks cool, but it doesn't look much different from what I get if I debug Javascript in a web browser, so I think there's a ton of room to improve. Visual representations of data transformations and code paths are a relatively unexplored area across the entire software industry imo.

If an LLM could flat-out reduce my need to reason and fix the bug for me, great -- but I've worked with coders that I respect a lot, and remote debugging with them has always been a pain and made narrowing down issues harder. I've never enjoyed debugging something remotely where I was working through a middleperson and couldn't look at what was going on; it's helpful to have multiple eyes on the code, but not if I have to use someone else's eyes to look at what's happening.

So for me, in order to successfully reduce the amount of reasoning I need to do and overcome the downside of me not being able to visualize the timeline/data, the LLM would need to be better at fixing these bugs than professional developers in industry: developers who are already intimately familiar with the codebases I'm debugging because they wrote a significant portion of the code. It would need to be better at coding than professional humans. And I just don't think there's anyone who would say that GPT-4 is close to that level yet.

What I could see is, maybe -- if I have access to that data, and the LLM is just kind of on-the-side, maybe at that point it can offer helpful advice and there wouldn't be a downside because I would still be able to debug as fast as I can using all of the available data, and if the LLM can occasionally find something I missed, then great. Peer-debugging sessions with multiple coders are great, so at least in theory I could see some value from an LLM on that stuff, even if I'm a little skeptical about potential performance. And if the LLM wasn't in front of the entire data, if it didn't work then no worries, the data is still there.

But again, if people like it, then it doesn't matter what I think. Why I wouldn't use the tool is less important than why someone would use the tool, and if integration with the LLM makes people want to use the tool, then... I mean, not everyone has identical work styles. Different things might work for different people.

Re: Show HN: Leaping – Debug Python tests instantly with an LLM debugger

#19
post #14

On the reddit discussion, one user pointed out that email adresses were incidently collected. https://www.reddit.com/r/programming/s/lBfxL7f2KM

Why would they do that??? Are we really going into this era where we have to reverse engineer every binary we touch?

Re: Show HN: Leaping – Debug Python tests instantly with an LLM debugger

#20
post #14

On the reddit discussion, one user pointed out that email adresses were incidently collected. https://www.reddit.com/r/programming/s/lBfxL7f2KM

Was removed here: https://github.com/leapingio/leaping/commit/e42d5198abe48875...

They should probably remove that Posthog API key as well...
Post reply on HN