Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Study finds that 52% of ChatGPT answers to programming questions are wrong

futurism.com

Post reply on HN
Print view View original

71–80 of 104 posts

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#71

Post by tivert »

pornel wrote: post #48
parpfish wrote: post #6

Earlier quoted context omitted.

That’s how you, an experienced programmer, use it. What does this do to beginners that are just learning to program? Is this helping them by forcing them to become critical reviewers or harming them by being a bad role model?

Beginners tend to write awful code without GPT's help, so I don't think it makes things worse. Answers don't exist in a vacuum. The chat interface allows feedback and corrections. Users can paste an error they're getting, or even say "it doesn't work", and GPT may correct itself or suggest an alternative.

> Beginners tend to write awful code without GPT's help, so I don't think it makes things worse.

> Answers don't exist in a vacuum. The chat interface allows feedback and corrections. Users can paste an error they're getting, or even say "it doesn't work", and GPT may correct itself or suggest an alternative.

I think you're making the mistake of viewing the job as a black box that produces output.

But what you're proposing is a terrible way to develop someone's skills and judgement. They won't develop if they're getting their hand held all the time (by an LLM or a person), and they'll stagnate. The problem with an LLM, unlike a person, it that it will hold your hand forever without complaint, while giving unreliable advice.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#72

Post by panarky »

Veuxdo wrote: post #9
jghn wrote: post #2

This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.

> I expect it to get me in the ballpark faster than I would have on my own. This is great if you are an experienced developer who can tell the difference between "in the ballpark" and fixable and "in the ballpark" but hopeless.

If 52% of responses have a flaw somewhere, then 48% of responses are flawless.

That is amazing and important.

The headline should be "LLM gives flawless responses to 48% of coding questions."

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#73

Post by Gigachad »

jghn wrote: post #2

This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.

Every time I’ve tried it it’s sent me to completely the wrong ball park and after a while whacking it’s solution I end up completely dumping it and doing it myself.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#74

Post by Gigachad »

pornel wrote: post #48
parpfish wrote: post #6

Earlier quoted context omitted.

That’s how you, an experienced programmer, use it. What does this do to beginners that are just learning to program? Is this helping them by forcing them to become critical reviewers or harming them by being a bad role model?

Beginners tend to write awful code without GPT's help, so I don't think it makes things worse. Answers don't exist in a vacuum. The chat interface allows feedback and corrections. Users can paste an error they're getting, or even say "it doesn't work", and GPT may correct itself or suggest an alternative.

In my experience it never actually fixes the problem. It either gives you a random change back, or gives you the same solution.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#76

Post by gwill »

panarky wrote: post #72
Veuxdo wrote: post #9

Earlier quoted context omitted.

> I expect it to get me in the ballpark faster than I would have on my own. This is great if you are an experienced developer who can tell the difference between "in the ballpark" and fixable and "in the ballpark" but hopeless.

If 52% of responses have a flaw somewhere, then 48% of responses are flawless. That is amazing and important. The headline should be "LLM gives flawless responses to 48% of coding questions."

something could be flawed but still arguably correct. the headline is fine. i wouldn't buy a calculator or read documentation that is 48% correct.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#78

Post by pornel »

tivert wrote: post #71
pornel wrote: post #48

Earlier quoted context omitted.

Beginners tend to write awful code without GPT's help, so I don't think it makes things worse. Answers don't exist in a vacuum. The chat interface allows feedback and corrections. Users can paste an error they're getting, or even say "it doesn't work", and GPT may correct itself or suggest an alternative.

> Beginners tend to write awful code without GPT's help, so I don't think it makes things worse. > Answers don't exist in a vacuum. The chat interface allows feedback and corrections. Users can paste an error they're getting, or even say "it doesn't work", and GPT may correct itself or suggest an alternative. I think you're making the mistake of viewing the job as a black box that produces output. But what you're pro…

That's speculation about a hypothetical person, one that falls into learned helplessness, but there are people with different mindsets.

Getting some results with the help of infinitely-patient GPT may motivate people to learn more, as opposed to losing motivation from getting stuck, having trouble finding right answers without knowing the right terminology, and/or being told off by StackOverflow people that's a homework question.

People who want to grow, can also use GPT to ask for more explanations, and use it as a tutor. It's much better at recalling general advice.

And not everyone may want to grow into a professional developer. GPT is useful to lots of people who are not programmers, and just need to solve programming-adjacent problems, e.g. write a macro to automate a repetitive task, or customize a website.

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#79

Post by tasuki »

haolez wrote: post #34
tasuki wrote: post #19

Does that mean that 48% of ChatGPT answers to programming questions are correct? If so, that's amazing!

Kind of, but I'd guess that Google searches and Stack Overflow might have a higher success rate than this.

No way. I send ChatGPT my Haskell code and the unreadable compiler error message, and it tells me what the error means in plain human terms or at least points me in the right direction.

Google and Stack Overflow are useless here, people have different situations than I do.

I find it's worse at providing working code (much less good code), but pretty good at telling me why my code doesn't compile, which is 80% of the work anyway!

Re: Study finds that 52% of ChatGPT answers to programming questions are wrong

#80

Post by tivert »

pornel wrote: post #78
tivert wrote: post #71

Earlier quoted context omitted.

> Beginners tend to write awful code without GPT's help, so I don't think it makes things worse. > Answers don't exist in a vacuum. The chat interface allows feedback and corrections. Users can paste an error they're getting, or even say "it doesn't work", and GPT may correct itself or suggest an alternative. I think you're making the mistake of viewing the job as a black box that produces output. But what you're pro…

That's speculation about a hypothetical person, one that falls into learned helplessness, but there are people with different mindsets. Getting some results with the help of infinitely-patient GPT may motivate people to learn more, as opposed to losing motivation from getting stuck, having trouble finding right answers without knowing the right terminology, and/or being told off by StackOverflow people that's a homew…

> Getting some results with the help of infinitely-patient GPT may motivate people to learn more, as opposed to losing motivation from getting stuck, having trouble finding right answers without knowing the right terminology,

> ...People who want to grow, can also use GPT to ask for more explanations, and use it as a tutor. It's much better at recalling general advice.

The psychology there doesn't make sense, since the technology simultaneously takes away a big motivation to actually learn how to get the result on your own. It's like giving a kid a calculator and expecting him to use it to learn mental arithmetic. Instead, you actually just removed the motivation for most kids to do so.

I think there's a common, unstated assumption in tech circles that removing "friction" and making things "easier" is always good. It's false.

Also, a lot of what you said feels like a post-hoc rationalization for applying this particular technology as a solution to a particular problem, which is a big problem with discourse around "AI" (just like it was with blockchain). That stuff is just in the air.

> ...and/or being told off by StackOverflow people that's a homework question.

IMHO, that's the one legitimately demotivating thing on your list.

Post reply on HN