Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Study finds that 52% of ChatGPT answers to programming questions are wrong

futurism.com

Post reply on HN
Print view View original

81–90 of 104 posts

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#81

Post by balder1991 »

RicoElectrico wrote: post #13
jghn wrote: post #2

This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.

Absolutely, it's especially useful when it suggests which libraries to use if you're not familiar with the ecosystem. Or writing boilerplate for popular frameworks, step by step. It can, to a degree, repair errors if you paste it the output.

Might not be a good idea for people not security aware: https://vulcan.io/blog/ai-hallucinations-package-risk/#h2_4

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#82

Post by happypumpkin »

ryanwaggoner wrote: post #77
happypumpkin wrote: post #35

From the paper: "Additionally, this work has used the free version of ChatGPT (GPT-3.5)"

This is a critical detail. GPT-4 is much better than 3.5 for programming, in my experience.

Yeah I really don't understand why research is still being published that uses GPT3.5 rather than GPT4 or both models. ~500 programming questions is maybe a few bucks on the API?

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#83

Post by asadotzler »

happypumpkin wrote: post #82
ryanwaggoner wrote: post #77

Earlier quoted context omitted.

This is a critical detail. GPT-4 is much better than 3.5 for programming, in my experience.

Yeah I really don't understand why research is still being published that uses GPT3.5 rather than GPT4 or both models. ~500 programming questions is maybe a few bucks on the API?

Because 99% of users and probably 95% of programmers are using the free version while almost no one is using the paid version.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#84

Post by asadotzler »

cjonas wrote: post #29

I scanned the paper and it doesn't mention what model they were using within chatgpt. If it was 3.5 turbo, then these results are already meaningless. GPT-4 and 4o are much more accurate. I just used GPT-4o to refactor 50 files from react classes to react function components and it did so almost perfectly everytime. Some of these classes were as long as 500 loc.

Not meaningless when 99% of the people use the free version which apparently has license to lie to them far more than the paid version. What a fucking sick joke, pay up or we lie to you even more.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#85

Post by mnk47 »

asadotzler wrote: post #84
cjonas wrote: post #29

I scanned the paper and it doesn't mention what model they were using within chatgpt. If it was 3.5 turbo, then these results are already meaningless. GPT-4 and 4o are much more accurate. I just used GPT-4o to refactor 50 files from react classes to react function components and it did so almost perfectly everytime. Some of these classes were as long as 500 loc.

Not meaningless when 99% of the people use the free version which apparently has license to lie to them far more than the paid version. What a fucking sick joke, pay up or we lie to you even more.

The new GPT4o model is free. Plus users will get better rate limits and the voice feature, but everyone has access to the best model right now.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#86

Post by cess11 »

nkozyra wrote: post #61
paulsutter wrote: post #53

Earlier quoted context omitted.

I’ve listened to 10,000 hours of piano music and I still can’t play anything Debugging is the primary skill of a programmer. 90% of programming is fixing bugs, the other 10% is writing bugs. Maybe the LLM is teaching debugging by giving bad examples :)

> I’ve listened to 10,000 hours of piano music and I still can’t play anything I don't think this is a valid comparison. If you'd read 10,000 hours of sheet music I'd wager you'd know how to read music.

When I studied in Ulaan Bataar I met a professor of linguistics from eastern Europe. Before he came to Mongolia he studied a grammar book of mongolian and tried to teach himself. He was rather proud of how far he had come.

At the first lesson he realised that the characters he thought he knew how to pronounce didn't sound much like he was used to. Mongolian is generally written with cyrillic plus a few more characters, so he expected it to be like russian or bulgarian with a few more sounds.

This is not the case. Mongolian is much closer related to korean and tibetan, and commonly sounds something like drunk cats haggling over something deceased.

I find it to be roughly the same with introductory or otherwise shallow learning material about programming. You can read as many tutorials as you want, you'll still suck at it.

When the LLM:s invent books like SICP, The Art of Computer Programming, Purely Functional Data Structures, Gang of Four, then they might become tutors in this area. To me it seems they struggle hard with anything longer than a screenful.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#87

Post by nkozyra »

cess11 wrote: post #86
nkozyra wrote: post #61

Earlier quoted context omitted.

> I’ve listened to 10,000 hours of piano music and I still can’t play anything I don't think this is a valid comparison. If you'd read 10,000 hours of sheet music I'd wager you'd know how to read music.

When I studied in Ulaan Bataar I met a professor of linguistics from eastern Europe. Before he came to Mongolia he studied a grammar book of mongolian and tried to teach himself. He was rather proud of how far he had come. At the first lesson he realised that the characters he thought he knew how to pronounce didn't sound much like he was used to. Mongolian is generally written with cyrillic plus a few more character…

> You can read as many tutorials as you want, you'll still suck at it.

I agree, but it's better foundation than getting something to compile and thinking you "get it."

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#88

Post by cess11 »

Turing_Machine wrote: post #57
Turing_Machine wrote: post #54

Earlier quoted context omitted.

I've heard anecdotal claims that most applicants (not just interns) can't even write fizz buzz, so...pretty low?

[flagged]

If someone showed me this solution I'd have quite a few questions. Like, why is there a 'newline in the example section, and why isn't that part in a comment? Why introducing the "helper" to enforce that the execution always begins at 1? Could there be some other way to design the program so that the four conditions don't all end with the same twenty or so characters?

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#89

Post by cess11 »

nkozyra wrote: post #87
cess11 wrote: post #86

Earlier quoted context omitted.

When I studied in Ulaan Bataar I met a professor of linguistics from eastern Europe. Before he came to Mongolia he studied a grammar book of mongolian and tried to teach himself. He was rather proud of how far he had come. At the first lesson he realised that the characters he thought he knew how to pronounce didn't sound much like he was used to. Mongolian is generally written with cyrillic plus a few more character…

> You can read as many tutorials as you want, you'll still suck at it. I agree, but it's better foundation than getting something to compile and thinking you "get it."

Who suggested that?

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#90

Post by Turing_Machine »

cess11 wrote: post #88
Turing_Machine wrote: post #57

Earlier quoted context omitted.

[flagged]

If someone showed me this solution I'd have quite a few questions. Like, why is there a 'newline in the example section, and why isn't that part in a comment? Why introducing the "helper" to enforce that the execution always begins at 1? Could there be some other way to design the program so that the four conditions don't all end with the same twenty or so characters?

Those are questions of style.

The allegation was that ChatGPT produces answers that are wrong.

Post reply on HN