Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A study on robustness and reliability of large language model code generation

arxiv.org

Post reply on HN
Print view View original

111–120 of 229 posts

Re: A study on robustness and reliability of large language model code generation

#111

Post by squeaky-clean »

enumjorge wrote: post #106
thomasfromcdnjs wrote: post #24

Earlier quoted context omitted.

I've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and pa…

It's comments like these that make me wish I could stand behind the chair of the people who get these results because I feel like there's something lost in translation. I don't know if I'm taking your comment too literally but this part confuses me: > It writes lots of code before I even think about what I was going to do in that file. If you haven't even thought of what you want to do in the file, how are you prompt…

> Am I understanding correctly that Copilot not only generates the correct code, but it's also able to deduce what code you meant to ask for without any input from you?

You can know what you want to program, but not how you would approach the code. "Hey ChatGPT, how do I draw a curved line plot in JUCE?"

If I just ask it "Can you write me a script that gathers the current stories on the front page of hacker news and sorts them by number of comments, then prints the title and number of comments?" it writes a script in Python using requests and BeautifulSoup4.

Re: A study on robustness and reliability of large language model code generation

#112

Post by anon25783 »

enumjorge wrote: post #109
wesleywt wrote: post #72

Earlier quoted context omitted.

Meh. I started coding professionally in the age of AI. My productivity is way higher than it should be. I don't let the AI "write" the code for me. I use it as a companion that I asked question of and suggestions. It suggests solutions to problems, I add the complexity. So far my code has been both clear and performant. This should not be the case at my level.

> I started coding professionally in the age of AI. What does this mean? Copilot was released in March of this year. ChatGPT 4, which the community considers to be the only version of ChatGPT to be competent enough at coding tasks, was also released in March. My guess is that you didn't mean that statement to be a fancy way of saying "I've been coding professionally for around 6 months", but I don't know what else yo…

Maybe they think this is LinkedIn? /j

Re: A study on robustness and reliability of large language model code generation

#113

Post by m3kw9 »

Gigachad wrote: post #14

I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…

My experience is that you need to do a few back and forth get get it right especially for a more detailed functions. What works really well is if you need it to explain to you how something works, but to verify, you need to have a good sniff test which novices won’t have so they take it dangerously straight up

Re: A study on robustness and reliability of large language model code generation

#114

Post by beatthatflight »

Gigachad wrote: post #14

I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…

I've used it for performance boosts. It's not perfect, but if I paste a method (my own code) and say 'rewrite in fewer lines' to simplify, or 'rewrite to run faster', I've been in awe at what it comes up with that hadn't occured to me. Tricks in python I wasn't aware of, or hadn't considered using a different datastructure that can be searched faster (for some reason my brain always defaults to lists, when a set may be better, for example). It absolutely makes errors though, so unit tests are critical, but I've got code running so much faster as a result.

Re: A study on robustness and reliability of large language model code generation

#115

Post by jsunderland323 »

corethree wrote: post #107
jsunderland323 wrote: post #86

Earlier quoted context omitted.

> 48% if half way to 100% … It's halfway their to taking your job and your underwhelmed. You’re implying 100% reliability is actually attainable. If that were the case, wouldn’t that mean the halting problem would have been solved by AI? I’m not an expert but I’ve heard that’s like one of those fundamental laws of information theory that really can’t be broken. > It's typical. It's like an indie band is only popular…

>You’re implying 100% reliability is actually attainable. If that were the case, wouldn’t that mean the halting problem would have been solved by AI? I’m not an expert but I’ve heard that’s like one of those fundamental laws of information theory that really can’t be broken. Nah even 100% reliability isn't attainable by a human. 100% obviously doesn't imply solving the halting problem. 100% as in 100% as reliable as…

I'm not going to rebut everything but I want to point out that I think Geoffrey Hinton is way more in your camp in regards to faith in the technology (which is what we're discussing). I'm in the camp that thinks LLMs are at best a viable competitor to furby -- I have zero existential fears and have very little confidence in them. I'm not opposed to AIs, I just think the state of the art is really really bad (sorry) and the doom and existentialism is a marketing ploy to a world imbued in conspiracy theory and institutional distrust in order to compensate for a wildly over promised and under delivered product. But hey, when you gotta raise money, you gotta raise money, and as you put it, the "trend line is going up" and that's all that matters.

Re: A study on robustness and reliability of large language model code generation

#116

Post by Lerc »

teraflop wrote: post #64

I'm sympathetic to the viewpoint that GPT-4 is prone to mistakes when writing code. Unfortunately, the analysis in this paper is pretty bad and doesn't support that conclusion. The authors assume that for any given method under consideration, it must only occur within a particular pattern of other method calls and control flow instructions. But the templates they have chosen are clearly only applicable in certain sit…

I have only glanced at a few examples in the paper so far but I would rate some of the cases listed as being better responses from the LLM than their exemplars.

If you ask someone how to open a can of soda you do not want them to tell you how to check to see if the can has been shaken and what to do if it has. You want the instructions to the question you asked.

If anything I would a LLM to produce code that does the fundamental operation, possibly I might like it to offer a more robust framing of the operation as an extended example but I suspect even that would get annoying after a while.

I absolutely would not want The most technically correct but harder to read response to a question of "how do I X?"

Re: A study on robustness and reliability of large language model code generation

#117

Post by [deleted] »

enumjorge wrote: post #109
wesleywt wrote: post #72

Earlier quoted context omitted.

Meh. I started coding professionally in the age of AI. My productivity is way higher than it should be. I don't let the AI "write" the code for me. I use it as a companion that I asked question of and suggestions. It suggests solutions to problems, I add the complexity. So far my code has been both clear and performant. This should not be the case at my level.

> I started coding professionally in the age of AI. What does this mean? Copilot was released in March of this year. ChatGPT 4, which the community considers to be the only version of ChatGPT to be competent enough at coding tasks, was also released in March. My guess is that you didn't mean that statement to be a fancy way of saying "I've been coding professionally for around 6 months", but I don't know what else yo…

[deleted]

Re: A study on robustness and reliability of large language model code generation

#118

Post by justinlloyd »

Gigachad wrote: post #14

I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…

I have been happy about 90% to 95% of the time with code that gets spat out by ChatGPT, though I don't fear for my job being taken by an LLM any time soon. It usually gives good insight in to unfamiliar code, or code in a language that I am familiar with but might have forgotten how to use. That said, I don't use it for every piece of code I write. Usually sections where I am exploring an idea.

At times, I need to correct the LLM on some usually obscure detail. It got the memory layout of the video screen on the BBC Micro completely mixed up with the ZX Spectrum, and I had to correct it about five times before it got it, and then it stuck for the rest of the conversation.

I've been on some wild goose chases, where I have fooled myself into thinking a particular approach would work and ChatGPT has been enthusiastically right there cheering me on.

It's like my dog going on an adventure with me, the dog doesn't care about where we're going or what we're doing, it's just excited to be part of the journey.

And in other cases, ChatGPT has been exceptionally useful in pointing out some blindingly obvious mistakes. It is like having a very knowledgeable, but exceptionally junior developer at your elbow.

If you ask the right questions, and don't except one shot questions to provide perfect answers, it works well for the most part.

Re: A study on robustness and reliability of large language model code generation

#119

Post by bcrosby95 »

throwaway09223 wrote: post #45
Gigachad wrote: post #14

I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…

You have to put it in context: How good is the human alternative? I have interviewed hundreds of engineers across the entire skill spectrum. I think GPT4 right now is about on-par with a mid level developer. It makes a lot of mistakes, especially around counting, and usually can notice and fix them if they're pointed out. Humans do this all the time -- how often do you get a compiler error from something silly? It of…

I would peg gpt as between junior and mid. Mids (around 5-10 years of experience) tend to produce overly complex code for the problem at hand.

Re: A study on robustness and reliability of large language model code generation

#120

Post by justinlloyd »

redhale wrote: post #68
thomasfromcdnjs wrote: post #24

Earlier quoted context omitted.

I've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and pa…

This! I wonder if it's a language difference thing (I use C# mostly), but my experience with Copilot has been nothing short of mind-blowing. It's not writing every line, and when writing truly new feature code it's less useful. But here are two patterns that I've noticed it is especially and consistently good at as a potential place to look for initial value if you are skeptical: - If you have any kind of repeated pa…

I've used ChatGPT and Copilot with Javascript & Typescript and a bunch of frameworks. Python, C, C++, C#, Go, 6502, Z80, ARM and a few other languages so far. They've all worked pretty good. I wish it had a wider breadth of APIs and documentation, but it is pretty good so far.
Post reply on HN