Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT Isn't as Good at Coding as We Thought

pcmag.com

Post reply on HN
Print view View original

11–20 of 48 posts

Re: ChatGPT Isn't as Good at Coding as We Thought

#11

Post by selfhoster11 »

namaria wrote: post #6

I don't understand why generating language, or code, would be a good thing. Language, and code, is a tool for communication. If you treat it as fodder or merchandise, you loose the main use: transport meaning between different people. I fail to understand why we would want automatically generated language at all, except for the fact that you can sometimes make money off of it. Considering we're where we are as a civi…

The purpose of code is first and foremost to make a computer do things. Legibility is, strictly speaking, secondary.

Re: ChatGPT Isn't as Good at Coding as We Thought

#12

Post by soultrees »

Lately it’s even worse. And any AI agent that relies on open AI has gone down in quality too (phind). I find it’s more of a struggle to have the AI write code in the first pass, even when I explicitly say something like “I want you to write a function that does this {action}” and on average it takes about 3 messages back and forth before it actually writes it. Whereas before it would start writing it immediately, even though it was almost always wrong, I could tweak the result after. Now though, the stupid context memory is filled up with useless junk words not related to the actual function I want to write, since we spent multiple messages going back and forth.

Re: ChatGPT Isn't as Good at Coding as We Thought

#13

Post by robbywashere_ »

Fair. But garbage in garbage out. The prompts are in human language. Of which the models use to derive output. Human language is obviously not a rigid mode of instruction. Therefore; the models behave exactly as you'd guess. The output is not purely predictable nor rigid. This is by design.

Providing a prompt, and/or know how to properly prompt is key.

Re: ChatGPT Isn't as Good at Coding as We Thought

#14

Post by jhoelzel »

PoignardAzur wrote: post #10
jhoelzel wrote: post #5

And it does not need to be. Personally i do not use it to generate novel code that will solve my problems, but when i point it at a github repo with the github plugin enabled, it will answer the questions i have about the code pretty much on the nose. its not your new AI Friend steve, its a LLM. And one with a knowledge cutoff in 2021 too

> but when i point it at a github repo with the github plugin enabled, it will answer the questions i have about the code pretty much on the nose. Holy crap, you can do that? I need to learn about ChatGPT plugins.

yes its in the "plugin store" and named "ask the code"

Re: ChatGPT Isn't as Good at Coding as We Thought

#15

Post by ChatGTP »

jiggawatts wrote: post #4

Yet another garbage study where “researchers” wasted months of their lives evaluating GPT 3.5 instead of paying $20 to evaluate GPT 4. It’s like drinking the complementary table water at a restaurant and then leaving a one-star review saying “tasteless and flat”. I seriously can’t believe this is maybe the tenth such “paper” I’ve seen making headlines. It brings shame on not only the institution that produced it, but…

I'd like to see the stats on how many people use 3.5 over 4 though, I bet you that number is much higher than you think. The price of ChatGPT is being what I'm personally willing to pay for it so if I do use it, it's 3.5 for me.

Maybe the logic is to study the most popular product? Would make sense to me.

Re: ChatGPT Isn't as Good at Coding as We Thought

#16

Post by SanderNL »

omarfarooq wrote: post #9
namaria wrote: post #6

I don't understand why generating language, or code, would be a good thing. Language, and code, is a tool for communication. If you treat it as fodder or merchandise, you loose the main use: transport meaning between different people. I fail to understand why we would want automatically generated language at all, except for the fact that you can sometimes make money off of it. Considering we're where we are as a civi…

Would you be so kind as to demonstrate how generated language and code does not transport meaning between different people.

I'd say the symbols themselves are meaningless without their ground truths, whatever that is. We can call this "experience" or "thinking". The results of these opaque processes are somehow amenable to be encoded into symbols which are themselves deprived of any significance. Which I guess makes them usable as such in the first place.

Extreme example: if a random algorithm picked out a few words from a dictionary and somehow cobbled a sentence together using those words, would you say "meaning" has been transported?

The "generation" of these symbols should be the result of a process equivalent to whatever we are doing when we "experience" or "cognate" or whatever, otherwise the results will be only very superficially useful.

Re: ChatGPT Isn't as Good at Coding as We Thought

#17

Post by HWR_14 »

jiggawatts wrote: post #4

Yet another garbage study where “researchers” wasted months of their lives evaluating GPT 3.5 instead of paying $20 to evaluate GPT 4. It’s like drinking the complementary table water at a restaurant and then leaving a one-star review saying “tasteless and flat”. I seriously can’t believe this is maybe the tenth such “paper” I’ve seen making headlines. It brings shame on not only the institution that produced it, but…

It takes time to create and publish a study.

Re: ChatGPT Isn't as Good at Coding as We Thought

#18

Post by SanderNL »

ChrisRR wrote: post #7
namaria wrote: post #6

I don't understand why generating language, or code, would be a good thing. Language, and code, is a tool for communication. If you treat it as fodder or merchandise, you loose the main use: transport meaning between different people. I fail to understand why we would want automatically generated language at all, except for the fact that you can sometimes make money off of it. Considering we're where we are as a civi…

You can't understand why software that can do in a fraction of a section what a human would do in hours would be valuable?

The software indeed has to do what I do, which is understanding my environment and mentating solutions that are somehow appropriate for either fixing or alleviating any one of the uncountable challenges said environment faces. I cannot even describe what I am doing, only that relatively small parts of the end-result are encoded into symbols called code.

This reminds me of what Bohr said to Oppenheimer - in the movie - when some doubt arose around his mathematical ability: "Algebra is like sheet music. The important thing isn't 'can you read music', it's 'can you hear it'. Can you hear the music, Robert?"

I'm very much wondering if GPT can hear the music, but time will tell if and to what degree this even matters.

Re: ChatGPT Isn't as Good at Coding as We Thought

#19

Post by jiggawatts »

ChatGTP wrote: post #15
jiggawatts wrote: post #4

Yet another garbage study where “researchers” wasted months of their lives evaluating GPT 3.5 instead of paying $20 to evaluate GPT 4. It’s like drinking the complementary table water at a restaurant and then leaving a one-star review saying “tasteless and flat”. I seriously can’t believe this is maybe the tenth such “paper” I’ve seen making headlines. It brings shame on not only the institution that produced it, but…

I'd like to see the stats on how many people use 3.5 over 4 though, I bet you that number is much higher than you think. The price of ChatGPT is being what I'm personally willing to pay for it so if I do use it, it's 3.5 for me. Maybe the logic is to study the most popular product? Would make sense to me.

Many more people drink tap water than eat dinner at a MICHELIN Starred restaurant, but if you're a food critic writing about the latest fancy restaurant, you had better pay for some food and write about that.

This is especially painful considering the ludicrous pace of advancement in AI. You really have to aim for where the puck will be, not where the puck is. Within a year GPT4 will be out-of-date.

Fundamentally, the entire point of this "research paper" is to compare ChatGPT with Stack Overflow answers, coming to the conclusion that people prefer SO.

Yeah, well, meanwhile Stack Overflow usage has dropped off a cliff since ChatGPT become generally available: https://observablehq.com/@ayhanfuat/the-fall-of-stack-overfl...

I certainly prefer to ask ChatGPT basic coding questions because I get an answer immediately with no argument.

Re: ChatGPT Isn't as Good at Coding as We Thought

#20

Post by monooso »

ChatGTP wrote: post #15
jiggawatts wrote: post #4

Yet another garbage study where “researchers” wasted months of their lives evaluating GPT 3.5 instead of paying $20 to evaluate GPT 4. It’s like drinking the complementary table water at a restaurant and then leaving a one-star review saying “tasteless and flat”. I seriously can’t believe this is maybe the tenth such “paper” I’ve seen making headlines. It brings shame on not only the institution that produced it, but…

I'd like to see the stats on how many people use 3.5 over 4 though, I bet you that number is much higher than you think. The price of ChatGPT is being what I'm personally willing to pay for it so if I do use it, it's 3.5 for me. Maybe the logic is to study the most popular product? Would make sense to me.

If that were the case, I would expect to see an explicit reference to the CGPT version in the introduction, and an explanation as to why the authors chose it for the study.

Instead, the only version information appears to be buried in section 3.1.2 ("ChatGPT 3.5 Turbo API is used").

Post reply on HN