Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching Large Language Models to Self-Debug

arxiv.org

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Teaching Large Language Models to Self-Debug

#31

Post by cs702 »

famouswaffles wrote: post #9
cs702 wrote: post #3

In hindsight , it's the most natural, most obvious next step to get LLMs to write better code: Explain to them how to debug and fix the code they've written. Which is pretty much what you would do with an inexperienced human software developer. Looking at this with fresh eyes, it's both shocking to me that this sort of thing is even possible, and yet also completely unsurprising as yet another emergent capability of…

Not too shocking for me after this paper. https://arxiv.org/abs/2211.09066 You can teach GPT-3 arithmetic - https://imgur.com/a/w3DAYOi Basically 100% accuracy up to about 13 digit addition and >90 after that. What else can you teach GPT without changing weights ?

I meant shocking in the sense that it makes me gape in awe, but as I wrote, it's also, simultaneously, completely unsurprising given all the new emergent capabilities we keep discovering. We're in agreement :-)

Re: Teaching Large Language Models to Self-Debug

#32

Post by qumpis »

broast wrote: post #13
blondin wrote: post #12

Earlier quoted context omitted.

color me skeptical. what are those large companies that are replacing devs with LLMs?

I think it's more natural than you might think. For example, my company laid off a lot of people to try to be profitable, and now they pay me more but I have a smaller team with tighter deadlines. I have no choice but to use gpt for a lot of my analysis, design, and code- which I've gotten pretty used to over the past year in my hobby time The way I see it, if you code without it, you won't compete with the speed and…

Can you give some of your usecases? Is it involved stuff or mostly boilerplate? Curious how a team lead uses this tech.

Re: Teaching Large Language Models to Self-Debug

#35

Post by runlaszlorun »

viscanti wrote: post #6
matisseverduyn wrote: post #4

Useful, but still wouldn't count on it. With respect to GPT etc. as a copilot, the current dialogue seems to focus on "ask for GPT to generate code to do X" then "just paste in the error message to fix bugs in the code GPT generates" A.) Why is GPT generating code that results in simple compiler errors (that is why GPT probably shouldn't be used to generate any code / replace devs for real projects yet), and B.) erro…

If an LLM hallucinates lines of code that can't even compile, I suppose it could also hallucinate logic issues which are more difficult to track down.

I have limited experience even trying. But I did try it for some fundamental JS Web API stuff sans framework or library like IndexedDB, web sockets, and a basic, basic todo like app.

Neither of those three would function nor would they throw an error. Prompts to correct itself would not improve things.

So I did the natural thing and started to debug myself. At which point, I couldn’t help but ask myself why I was debugging machine generated code when I could not be lazy and actually build it from first principles.

Re: Teaching Large Language Models to Self-Debug

#36

Post by highduc »

ratg13 wrote: post #18
blondin wrote: post #12

Earlier quoted context omitted.

color me skeptical. what are those large companies that are replacing devs with LLMs?

You can’t replace devs with LLMs because someone that knows what they are doing still needs to put it all together. You can only make employees more productive.. this in turn could, in theory, lessen the need for developers in the long run, but it assumes the company will not bother to use the extra bandwidth for other projects.

I also believe the gain is in productivity more than needing less people. They will fire as much as possible, but the largest gains seem to me to be in productivity.

And exactly like this some future brain implant thing will also put another layer of pressure. People will get it as it'll give them an edge on certain fronts.

Re: Teaching Large Language Models to Self-Debug

#37

Post by runlaszlorun »

qumpis wrote: post #32
broast wrote: post #13

Earlier quoted context omitted.

I think it's more natural than you might think. For example, my company laid off a lot of people to try to be profitable, and now they pay me more but I have a smaller team with tighter deadlines. I have no choice but to use gpt for a lot of my analysis, design, and code- which I've gotten pretty used to over the past year in my hobby time The way I see it, if you code without it, you won't compete with the speed and…

Can you give some of your usecases? Is it involved stuff or mostly boilerplate? Curious how a team lead uses this tech.

I’m not knocking the parent post here for not replying. But of the 4 or so times on HN that I’ve seen someone been asked to provide detail, I haven’t seen a single answer. I’m not on here a ton but do people tend not to check back?

I’ll admit my bias of having seen enough vacuous industry hype over the years to be naturally skeptical. Heck, I worked in a marketing department once where I helped manufacture the stuff (forgive me father for I have sinned, in my defense they did fire me after three months in a layoff). But my few personal experiences with ChatGPT were pretty disappointing and I’m actually looking for someone to tell me otherwise.

Re: Teaching Large Language Models to Self-Debug

#38

Post by goatlover »

sowbug wrote: post #34
rhyme-boss wrote: post #33

A warning siren goes off in the background. Another step towards recursive self-improvement.

So is this the singularity?

I doubt it without being able to evolve the weights, architectures and train those on new domains. Humans are still doing all of that. It's just a powerful tool people are using for a bunch of things. But it doesn't do anything on it's own, and it doesn't care.

Re: Teaching Large Language Models to Self-Debug

#39

Post by famouswaffles »

cs702 wrote: post #31
famouswaffles wrote: post #9

Earlier quoted context omitted.

Not too shocking for me after this paper. https://arxiv.org/abs/2211.09066 You can teach GPT-3 arithmetic - https://imgur.com/a/w3DAYOi Basically 100% accuracy up to about 13 digit addition and >90 after that. What else can you teach GPT without changing weights ?

I meant shocking in the sense that it makes me gape in awe, but as I wrote, it's also, simultaneously, completely unsurprising given all the new emergent capabilities we keep discovering. We're in agreement :-)

Oh. yes well that's fair

Re: Teaching Large Language Models to Self-Debug

#40

Post by civilized »

Paul-Craft wrote: post #25
civilized wrote: post #22

Earlier quoted context omitted.

It doesn't work with numbers as computer numbers though. It works with them as decimal digit strings, just like humans do.

Make the number you multiply by essentially the concatenation of a long series of random digits, and I can just about guarantee most humans will get different things on both sides, because they'll make one or more mistakes doing the math. That is, of course, assuming the humans don't have suitable traditional computer tools capable of handling such a scenario.

Not sure how this is relevant to the discussion.
Post reply on HN