Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Study finds that 52% of ChatGPT answers to programming questions are wrong

futurism.com

Post reply on HN
Print view View original

31–40 of 104 posts

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#31

Post by mrweasel »

resource_waste wrote: post #10

Can someone email the author and explain what a LLM is? People asking for 'right' answers, don't really get it. I'm sorry if that sounds abrasive, but these people give LLMs a bad name due to their own ignorance/malice. I remember having some Amazon programmer trash LLMs for 'not being 100% accurate'. It was really an iD10t error. LLMs arent used for 100% accuracy. If you are doing that, you don't understand the tech…

> LLMs arent used for 100% accuracy.

I think you're wrong about that. They shouldn't be, but they clearly are.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#32

Post by 51Cards »

jghn wrote: post #14
parpfish wrote: post #6

Earlier quoted context omitted.

That’s how you, an experienced programmer, use it. What does this do to beginners that are just learning to program? Is this helping them by forcing them to become critical reviewers or harming them by being a bad role model?

How is this any different than the age old "googling stack overflow" method everyone's been using for years?

Because on Stack Overflow you also see feedback from hopefully a cross section of the developer community, several methods of solving the problem, voting feedback on those solutions, and can sus out a proper solution vs. just having one answer fed back to you.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#33

Post by thatjoeoverthr »

jghn wrote: post #2

This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.

This. For inexperienced developers, I advise thus; don't consume answers you don't understand. If you can't read it, interrogate it, and find a question at your own level. When you accept its emission, you're taking responsibility for it, and beyond a certain low level, it can't do your thinking for you.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#36

Post by [deleted] »

jghn wrote: post #2

This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.

[deleted]

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#38

Post by NBJack »

jghn wrote: post #2

This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.

From the article:

> What's especially troubling is that many human programmers seem to prefer the ChatGPT answers. The Purdue researchers polled 12 programmers — admittedly a small sample size — and found they preferred ChatGPT at a rate of 35 percent and didn't catch AI-generated mistakes at 39 percent.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#39

Post by f0e4c2f7 »

This study uses a version of ChatGPT that is either 1 or 2 versions behind depending on the part of the study.

It cracks me up how consistent this is.

See post criticizing LLMs. Check if they're on the latest version (which is now free to boot!!).

Nope. Seemingly...never. To be fair, this is probably just an old study from before 4o came out. Even still. It's just not relevant anymore.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#40

Post by fkyoureadthedoc »

greatpatton wrote: post #16
crpietschmann wrote: post #7

Earlier quoted context omitted.

Exactly! Just because part of the answer isn't right, doesn't mean the entire answer is useless. It's much faster than only doing a Google search when working out the solution to a problem.

Sometimes you are going to loose a lot of time trying to make a ChatGPT solution work when Google would have provided right away the right answer... Just yesterday I asked ChatGPT for an AWS IAM policy. ChatGPT-4o provided an answer that looked ok but was just wrong, tried to make it work without success. Just Googled it and the first result provided me the right answer.

I prefer Phind for this type of question since you can see search results that it's likely drawing answers from.

But ChatGPT is often a huge time saver if you know exactly what you want to do and just let it fill in the how. I have these 3 jsonl files and I want to use jq to do blah blah and then convert them to csv

Post reply on HN