Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 817 posts

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#21

Post by opportune »

squokko wrote: post #17
opportune wrote: post #12

Is it consistently worse or just sometimes/often worse than before? Any extreme power users or GPT-whisperers here? If it’s only noticeably worse X% of the time my bet would be experimentation. One of my least favorite patterns that tech companies do is use “Experimentation” overzealously or prematurely. Mainly, my problem is they’re not transparent about it, and it creates an inconsistent product experience that jus…

No place I worked at ever experimented at the pageload level. We experimented at the user level, so 1% of users would get the new UI. I suppose this is only possible at the millions of users scale which all of them had.

I updated the comment to reflect that. Certainly the signal is stronger because you’re amortizing away the surprise factor of the change, and at least it’s a consistent UX, but the UX tradeoff in the worst case is that experiment-group users get a broken product with no notice or escape hatch. Unless you’re being very careful, meticulous, and transparent it’s just not acceptable if you’re a paying customer.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#22

Post by spaceman_2020 »

bbotond wrote: post #14

Yes. Before the update, when its avatar was still black, it solved pretty complex coding problems effortlessly and gave very nuanced, thoughtful answers to non-programming questions. Now it struggles with just changing two lines in a 10-line block of CSS and printing this modified 10-line block again. Some lines are missing, others are completely different for no reason. I'm sure scaling the model is hard, but they l…

It’s go-to tactic now if I ask it to go over any piece of code is to give a generic overview. Earlier, it would section out the code into chunks and go through each one individually.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#23

Post by reidjs »

opportune wrote: post #21
squokko wrote: post #17

Earlier quoted context omitted.

No place I worked at ever experimented at the pageload level. We experimented at the user level, so 1% of users would get the new UI. I suppose this is only possible at the millions of users scale which all of them had.

I updated the comment to reflect that. Certainly the signal is stronger because you’re amortizing away the surprise factor of the change, and at least it’s a consistent UX, but the UX tradeoff in the worst case is that experiment-group users get a broken product with no notice or escape hatch. Unless you’re being very careful, meticulous, and transparent it’s just not acceptable if you’re a paying customer.

In some cases you’re making the change because the app is already broken for the majority of users and you’re testing the fix

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#24

Post by killingtime74 »

i_dont_know_ wrote: post #18

To me, it feels like it's started giving superficial responses and encouraging follow-up elsewhere -- I wouldn't be surprized if its prompt has changed to something to that effect. Before, if I had an issue with a library or debugging issue, it would try to be helpful and walk me through potential issues, and ask me to 'let it know' if it worked or not. Now it will try to superficially diagnose the problem and then a…

I assume you're talking about ChatGPT and not GPT-4? You can craft your own prompt when calling GPT4 over API. Don't blame you though, the OP is also not clear if they are comparing Chat GPT powered by GPT3.5 or 4, or the models themselves.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#25

Post by tmikaeld »

There's no doubt that it's gotten a lot worse on coding, I've been using this benchmark on each new version of GPT-4 "Write a tiptap extension that toggles classes" and so far it's gotten it right every time, but not any more, now it hallucinates a simplified solution that don't even use the tiptap api any more. It's also 200% more verbose in explaining it's reasoning, even if that reasoning makes no sense whatsoever - it's like it's gotten more apologetic and generic.

The answer is the same on GPT plus and API with GPT-4, even with "developer" role.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#26

Post by spaceman_2020 »

bitcurious wrote: post #16

For a while, if you asked the iPhone version what it was it claimed to be GPT3.0. Not sure if it still is that, but I noticed the iPhone version was a bit worse. Maybe they rolled that out more broadly?

If they’re giving pro subscribers GPT-3.0 instead of 4, it would be fraud.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#28

Post by napier »

No, not just you. It’s frequently abysmal nowadays. Tragic. And no version change-log or any other info from OAI about what they’ve done to it (probably GPU sparing optimized distillation + overly aggressive PR safety satisficing RLHF) and an advertised weeks long support query lag (I’ve yet to receive a response).

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#29

Post by gettodachoppa »

skilled wrote: post #5

It’s possible they are trying out a shaved/turbo version so that they can start removing the limits. I mean as it is - 25 messages every 3 hours is useless, particularly for browsing and plugins.

That would be a shame. I would rather be limited to 25 quality responses per 3 hours than an unlimited inferior GPT4.

Imagine trading the advice of a senior mentor for 5 intermediate mentors. Yes, the answers get to you faster, but it's much less useful.

Post reply on HN