Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A study on robustness and reliability of large language model code generation

arxiv.org

Post reply on HN
Print view View original

201–210 of 229 posts

Re: A study on robustness and reliability of large language model code generation

#201

Post by ChatGTP »

wokwokwok wrote: post #198
ChatGTP wrote: post #190

Earlier quoted context omitted.

Sounds kind of silly, just run the thing till it gives the right answer ? How do you know each pass has not introduced a new problem ?

Yup. …and if it works (and it does) then it really goes to show that you (and me, and people in general) have poor intuition about this stuff.

Yeah, please don't try bullshit me dude.

It's a silly, potentially dangerous and inefficient way to "code", which is the easiest part of my job. If my job was just coding all day, I'd be happy.

Re: A study on robustness and reliability of large language model code generation

#202

Post by arp242 »

teaearlgraycold wrote: post #146
arp242 wrote: post #89

Earlier quoted context omitted.

> I cannot imagine trying to develop this way. But if people like it, whatever blows your hair back. I agree; who am I to comment on how you write the code, as long as it gets written, right? That said, I've already had a "this is really odd code you committed last week, what's up with that?" and the reply was "oh, dunno, that's just what ChatGPT gave me". Meh. Actually writing things yourself does increase your unde…

Anyone committing code they don’t understand should be put on a PIP.

He was the team lead shrug

This is one of several reasons I worked there for a week only.

Re: A study on robustness and reliability of large language model code generation

#203

Post by magicalhippo »

lifeisstillgood wrote: post #183
Anduia wrote: post #174

Earlier quoted context omitted.

Hi, do you have a recommended read for those of us who might inadvertently create race conditions?

Now hopefully I won't get horribly dinged for mistakes and poor advice here. What I am trying to say is that I too read "even senior devs don't understand the race conditions they create downstream." And I thought - oh God, don't I. But five minutes thought can help you walk through most issues. For most applications most of the time you can reason your way through without fear, and when you do encounter gnarly probl…

For the example in question one approach which might work well is to realize the test is superfluous. Just open the file with the right flags.

That is, the flag for create if not already existing and flaf for deny sharing so any other process/thread will fail to concurrently open it.

Then you just handle the case when the open fails.

Of course this relies on those flags being available and working. For example shared files might not respect the sharing lock.

Many cases of avoiding race conditions boil down to something similar: avoid the problem by not doing extra work.

Re: A study on robustness and reliability of large language model code generation

#204

Post by YeGoblynQueenne »

maxbond wrote: post #161
splatzone wrote: post #41

Run the code it writes, if it gives an error, paste it into the chat and 90% of the time the LLM can fix the issue. People are missing the point here - it's not about writing code in one-shot. An LLM-enabled loop can generate the code and test and refine it until it works

If you're writing eg a bash script and it contains a difficult to spot bug which causes causes data loss (see [1] caused by a single space), what then? [1] https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issue...

Then you ask ChatGPT to write you an email to apologise to your boss.

They'll probably ask ChatGPT to write an email to reply, so that's all good.

Re: A study on robustness and reliability of large language model code generation

#205

Post by version_five »

Lerc wrote: post #126
version_five wrote: post #61

There was a good video posted here yesterday about a (for me) seemingly typical experience writing code with chatGPT https://www.youtube.com/watch?v=U2Q3KAOSAEY The combination of hallucination and just bad advice makes it time consuming and not clearly worth it. I think it's a step in the right direction for conversational interfaces. The fact that you really can have a back and forth to clarify intent, make correct…

Having the ChatGPT 4 icon being purple has been a godsend when evaluating claims like these.

Interesting, I never noticed but I guess you're saying he used 3.5? I've only used 4, which I understand is way better, and I've still had the same kind of problems, but I agree it's disingenuous to equate 3.5 with the current state of the technology.

Re: A study on robustness and reliability of large language model code generation

#206

Post by JackFr »

oceanplexian wrote: post #162
teraflop wrote: post #64

I'm sympathetic to the viewpoint that GPT-4 is prone to mistakes when writing code. Unfortunately, the analysis in this paper is pretty bad and doesn't support that conclusion. The authors assume that for any given method under consideration, it must only occur within a particular pattern of other method calls and control flow instructions. But the templates they have chosen are clearly only applicable in certain sit…

I see humans do this all the time, especially abuse of exception handling, even among “Senior” developers. They don’t have a semantic understanding of what they are doing or why they are creating a race conditions or creating perverse control flow logic N layers down in the stack. The fact that researchers get it wrong is, well, unsurprising. LLMs might actually be an improvement.

Well, there's no "correct" answer. Depending on the context, the "correct" thing might be 1) to log and swallow the exception and move on; 2) Let the exception percolate up to caller, who can handle and recover; 3) implement your own recovery and handling; 4) kill the process. To know which one to do, one needs to understand the context. Logging and swallowing is not the worst default, but it's also likely not the best.

Re: A study on robustness and reliability of large language model code generation

#207

Post by naasking »

wredue wrote: post #44
keyle wrote: post #8

Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…

I just don’t see how it could save time. Programmers nearly universally agree that reading code is harder than writing it. So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems. Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.

> Programmers nearly universally agree that reading code is harder than writing it.

I don't think there's universal agreement on that at all. Programmers universally agree that we spend more time reading code than writing it, but that doesn't somehow translate into it being harder.

Various studies have shown that programmers, on average, can only add at about 10 correct lines of code per day. We of course add more lines of code, but they tend to contain many bugs, so in the end only 10 lines of those are correct.

Maybe we only get 10 correct lines of code in because we're not reading the surrounding code correctly, and thus reading code is harder, or maybe 10 correct lines of code is just harder to write. I'm not aware of any hard data on this question.

Re: A study on robustness and reliability of large language model code generation

#208

Post by naasking »

makestuff wrote: post #130
ramraj07 wrote: post #127

Earlier quoted context omitted.

Can you confirm if you used gpt-4? This was not my experience

It was gpt-3.5 (the free version offered). I will give get-4 a try though that is good to know.

Use Bing chat to get access to GPT-4 for free.

Re: A study on robustness and reliability of large language model code generation

#209

Post by hmottestad »

naasking wrote: post #207
wredue wrote: post #44

Earlier quoted context omitted.

I just don’t see how it could save time. Programmers nearly universally agree that reading code is harder than writing it. So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems. Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.

> Programmers nearly universally agree that reading code is harder than writing it. I don't think there's universal agreement on that at all. Programmers universally agree that we spend more time reading code than writing it, but that doesn't somehow translate into it being harder. Various studies have shown that programmers, on average, can only add at about 10 correct lines of code per day. We of course add more li…

Sounds like those programmers are spending a decent amount of time stuck in meetings :P

Re: A study on robustness and reliability of large language model code generation

#210

Post by pulvinar »

lifeisstillgood wrote: post #183
Anduia wrote: post #174

Earlier quoted context omitted.

Hi, do you have a recommended read for those of us who might inadvertently create race conditions?

Now hopefully I won't get horribly dinged for mistakes and poor advice here. What I am trying to say is that I too read "even senior devs don't understand the race conditions they create downstream." And I thought - oh God, don't I. But five minutes thought can help you walk through most issues. For most applications most of the time you can reason your way through without fear, and when you do encounter gnarly probl…

Or, sidestep it by creating an empty log file (if it doesn't exist) during initialization before creating multiple threads.
Post reply on HN