This is way better than I thought. A follow-up question would be for the times that it is wrong, how wrong is it. In other words, is the wrong answer complete rubbish or it can be a starting point towards the actual correct answer?
Study finds that 52% of ChatGPT answers to programming questions are wrong
21–30 of 104 posts
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#22Similar to how programmers work, the AI needs feedback from the runtime in order to iterate towards a workable program.
My expectation isn’t that the AI generate correct code. The AI will be useful as an ‘agent in the loop’:
- Spec or test suite written as bullets
- Define tests and/or types
- Human intevenes with edits to keep it in the right direction
- LLM generates code, runs complier/tests
- Output is part of new context
- Repeat until programmer is happy
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#23Earlier quoted context omitted.
That’s how you, an experienced programmer, use it. What does this do to beginners that are just learning to program? Is this helping them by forcing them to become critical reviewers or harming them by being a bad role model?
How is this any different than the age old "googling stack overflow" method everyone's been using for years?
Stack Overflow generally upvotes answers that are correct. Have to be some lemons but I doubt it's at 52%.
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#24To those who constantly claim ChatGPT is "like an intern," just how low are the standards for interns?
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#25Extrapolation:
Odds of correct answer within n attempts =
1 - (1/2)^n
Nice, that’s exponentially good!
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#26This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.
That’s how you, an experienced programmer, use it. What does this do to beginners that are just learning to program? Is this helping them by forcing them to become critical reviewers or harming them by being a bad role model?
If you’re learning to program pretty much the best approach is to write and debug programs. There isn’t a shortcut
It’s like the saying “the fog of war” (at best you have incomplete and flawed information). Programming is just like that
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#27This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.
That’s how you, an experienced programmer, use it. What does this do to beginners that are just learning to program? Is this helping them by forcing them to become critical reviewers or harming them by being a bad role model?
If this increases iteration speed for beginner devs and they learn about code quality post it goes into the real world, it’s not a bad bargain to strike imo.
I think we all partly learnt about code quality by having our code break things in the real world.
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#28Earlier quoted context omitted.
That’s how you, an experienced programmer, use it. What does this do to beginners that are just learning to program? Is this helping them by forcing them to become critical reviewers or harming them by being a bad role model?
How is this any different than the age old "googling stack overflow" method everyone's been using for years?
The wrong answers on SO can be commented on and/or edited.
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#29I scanned the paper and it doesn't mention what model they were using within chatgpt. If it was 3.5 turbo, then these results are already meaningless. GPT-4 and 4o are much more accurate.
I just used GPT-4o to refactor 50 files from react classes to react function components and it did so almost perfectly everytime. Some of these classes were as long as 500 loc.
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#30Earlier quoted context omitted.
That’s how you, an experienced programmer, use it. What does this do to beginners that are just learning to program? Is this helping them by forcing them to become critical reviewers or harming them by being a bad role model?
How is this any different than the age old "googling stack overflow" method everyone's been using for years?
Because people on stack overflow don't lie to the person who wrote the question very often. A correct answer to a problem that isn't the same as your problem is a better resource for learning than an incorrect answer to your exact problem.