Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 817 posts

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#91

Post by PeterStuer »

bevenky wrote: post #84
sirsinsalot wrote: post #83

Given the incoming compute capability from nvidia and the speed of advancement, we have to stop and think ... does it make sense to give access, paid or otherwise, to these models once they reach a certain sophistication? Or does it make even more sense to hoard the capability to out compete any competitor, of any kind, commercially or politically and hide the true extent of your capability to avoid scrutiny and legi…

I have some first hand thoughts. I think overall the quality is significantly poorer on GPT4 with plugins and bing browsing enabled. If you disable those, I am able to get the same quality as before. The outputs are dramatically different. Would love to hear what everyone else sees when they try the same.

No, while I have no hard data, the experienced quality of the default GPT-4 model feels like it has gone down tremendously for me as well. Plugins and Bing browsing have so far for me almost never worked at all. I retry these just once a week but there always seem to be technical issues.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#92

Post by PeterStuer »

bumbledraven wrote: post #71
PeterStuer wrote: post #66

Earlier quoted context omitted.

"The original GPT-4 felt like magic to me" You never had access to that original. Watch this talk by one of the people that integrated GPT-4 in Bing telling how they noticed GPT-4 releases they got from OpenAI got iteratively and significantly nerfed even during the project. https://www.youtube.com/watch?v=qbIk7-JPB2c

“You never had access to that original.” While your overall point is well taken, GP is clearly referring to the original public release of GPT-4 on March 14.

Yes, that was how I read it as well. I was just pointing out that the public release was already extremely nerfed from what was available pre-launch.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#93

Post by TeMPOraL »

bugglebeetle wrote: post #58
tmikaeld wrote: post #25

There's no doubt that it's gotten a lot worse on coding, I've been using this benchmark on each new version of GPT-4 "Write a tiptap extension that toggles classes" and so far it's gotten it right every time, but not any more, now it hallucinates a simplified solution that don't even use the tiptap api any more. It's also 200% more verbose in explaining it's reasoning, even if that reasoning makes no sense whatsoever…

It was a great ride while it lasted. My assumption is that efficacy at coding tasks is such a small percent of users, they’ve just sacrificed it on the altar of efficiency and/or scale. That, or they’ve cut some back room deal with Microsoft to make Copilot have access to the only version of the model that can actually code.

FWIW, I started to get the same feeling as the OP about GPT-4 model I have access to on Azure, so if there's any deal being cut here, it might involve dumbing down the model for paying Azure customers as well.

Now, to be clear: I only started to get a feeling that GPT-4 on Azure is getting worse. I didn't do any specific testing for this so far, as I thought I may just be imagining it. This thread is starting to convince me otherwise.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#94

Post by nullsense »

i_dont_know_ wrote: post #18

To me, it feels like it's started giving superficial responses and encouraging follow-up elsewhere -- I wouldn't be surprized if its prompt has changed to something to that effect. Before, if I had an issue with a library or debugging issue, it would try to be helpful and walk me through potential issues, and ask me to 'let it know' if it worked or not. Now it will try to superficially diagnose the problem and then a…

>To me, it feels like it's started giving superficial responses and encouraging follow-up elsewhere -- I wouldn't be surprized if its prompt has changed to something to that effect.

That's the vibe I've been getting. The responses feel a little cagier at times than they used to. I assume it's trying to limit hallucinations in order to increase public trust in the technology, and as a consequence it has been nerfed a little, but has changed along other dimensions that certain stakeholders likely care about.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#98

Post by r_hoods_ghost »

FredPret wrote: post #87

Maybe you annoyed it? I’m super nice to it and it performa better than ever. I ask it lay-of-the-land questions about technical problems that are new to me to detailed coding problems that I understand well but don’t want to figure out. The best though, is helping me navigate complicated UI’s. I tell it how I want some complicated software / website to behave, and it’ll tell me the arcane menu path to follow. It’s fu…

"It’s funny how computing might soon include elements of psychology and magic incantations nobody understands"

As a sysadmin I can tell you this is already the case...

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#100

Post by nullsense »

throwawayadvsec wrote: post #64

I think it's happening because of the extreme content filtering in place, at some point it was even refusing to generate some code because it thought it went against its guidelines to write code.

I definitely think this is playing a role. I've seen reports of people saying "oh it now refuses to act as my therapist" and "it wouldn't write my essay for me". Those are just a couple of anecdotes I've seen on Reddit, and haven't verified myself, but it wouldn't surprise me if OpenAI felt the need to make adjustments along those lines.
Post reply on HN