ChatGPT Isn't as Good at Coding as We Thought

52% correct seems awfully high - I am guessing it's inflated because of the easiness of the questions on StackOverflow. In my experience, at least in my domain, I will be lucky it it can generate an answer that is more than 5 lines of code without a single error. Still incredibly useful - just have to handhold it and watch it very carefully when going through a solution.

Yet another garbage study where “researchers” wasted months of their lives evaluating GPT 3.5 instead of paying $20 to evaluate GPT 4.

It’s like drinking the complementary table water at a restaurant and then leaving a one-star review saying “tasteless and flat”.

I seriously can’t believe this is maybe the tenth such “paper” I’ve seen making headlines.

It brings shame on not only the institution that produced it, but the journalists that failed to notice and published the clickbait headline without even reading the paper.

I don't understand why generating language, or code, would be a good thing. Language, and code, is a tool for communication. If you treat it as fodder or merchandise, you loose the main use: transport meaning between different people.

I fail to understand why we would want automatically generated language at all, except for the fact that you can sometimes make money off of it.

Considering we're where we are as a civilization because people could make money off of poisoning the Earth and causing catastrophic ecosystem and climate change for profit, I think we should examine our motives more closely.

It seems like it's only important to bring out a paper to fill a quota somewhere. The researchers probably don't care because no one (e.g. a perspective employer) is going to be actually reading the paper, just the amount of papers they published. That's why linking to something on Arxiv trying to undeniably prove a point ("Look, researchers confirm my point!") most of the time makes me cringe.

Plus, I've mostly given up on journalists. I get clickbait, that's why titles are there for. But clickbait doesn't have to equal badly researched. They could have just as well gotten clicks if they wrote an article "Researchers fail to prove ChatGPT is bad at coding because of this lazy mistake". But they mostly either have no idea of the field they are reporting about or are pushing an agenda. Research? Ain't nobody got time for that.

> but when i point it at a github repo with the github plugin enabled, it will answer the questions i have about the code pretty much on the nose.

Holy crap, you can do that? I need to learn about ChatGPT plugins.