Study finds that 52% of ChatGPT answers to programming questions are wrong
1–10 of 104 posts
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#2This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there.
Sometimes that means I have a follow on question & iterate from there. That's fine too.
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#3That article links to the actual paper, the abstract of which is itself quite readable: https://dl.acm.org/doi/pdf/10.1145/3613904.3642596
> Q&A platforms have been crucial for the online help-seeking behav- ior of programmers. However, the recent popularity of ChatGPT is altering this trend. Despite this popularity, no comprehensive study has been conducted to evaluate the characteristics of ChatGPT’s an- swers to programming questions. To bridge the gap, we conducted the first in-depth analysis of ChatGPT answers to 517 programming questions on Stack Overflow and examined the correctness, consis- tency, comprehensiveness, and conciseness of ChatGPT answers. Furthermore, we conducted a large-scale linguistic analysis, as well as a user study, to understand the characteristics of ChatGPT an- swers from linguistic and human aspects. Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose. Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style. However, they also overlooked the misinformation in the ChatGPT answers 39% of the time. This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#4Can there be some sort of mechanism on HN for criticism of an unsubstantiated headline?
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#5This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.
you're giving your fellow programmers way too much credit. most people will never bother doing that.
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#6This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.
That’s how you, an experienced programmer, use it.
What does this do to beginners that are just learning to program? Is this helping them by forcing them to become critical reviewers or harming them by being a bad role model?
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#7This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.
Exactly! Just because part of the answer isn't right, doesn't mean the entire answer is useless. It's much faster than only doing a Google search when working out the solution to a problem.
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#8This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.
That’s how you, an experienced programmer, use it. What does this do to beginners that are just learning to program? Is this helping them by forcing them to become critical reviewers or harming them by being a bad role model?
The same is true of things like code generators and scaffolds.
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#9This is looking at the wrong metric. I'm not expecting it to be 100% correct when I use it. I expect it to get me in the ballpark faster than I would have on my own. And then I can take it from there. Sometimes that means I have a follow on question & iterate from there. That's fine too.
> I expect it to get me in the ballpark faster than I would have on my own.
This is great if you are an experienced developer who can tell the difference between "in the ballpark" and fixable and "in the ballpark" but hopeless.
Re: Study finds that 52% of ChatGPT answers to programming questions are wrong
#10Can someone email the author and explain what a LLM is?
People asking for 'right' answers, don't really get it. I'm sorry if that sounds abrasive, but these people give LLMs a bad name due to their own ignorance/malice.
I remember having some Amazon programmer trash LLMs for 'not being 100% accurate'. It was really an iD10t error. LLMs arent used for 100% accuracy. If you are doing that, you don't understand the technology.
There is a learning curve with LLMs, and it seems a few people still don't get it.