Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

news.ycombinator.com

Post reply on HN
Print view View original

671–680 of 817 posts

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#671

Post by dgellow »

bob1029 wrote: post #266

OpenAI's models feel 100% nerfed to me at this point. I had it solving incredibly complex problems a few months ago (i.e. write a minimal PDF parser example), but today you will get scolded for asking such a complicated task of it. I think they programmed a classifier layer to detect certain coding tasks and shut it down with canned BS. I like to imagine certain billion/trillion-dollar mega corps had a back-room say…

I had the exact same problem, I thought it was just in my mind. I feel that I’m now constantly being scolded for asking what GPT4 seems to see as complex questions, it’s really frustrating.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#673

Post by napier »

sebzim4500 wrote: post #618
PeterisP wrote: post #617

Earlier quoted context omitted.

It's a big pain when trying to build things on top of the GPT-4 API. We had some experiments that were reliably, reproducibly achieving a goal, and then one day it suddenly stops working properly; then the student managed a different prompt that worked (again, reproducibly, with proper clean restarts from fresh context), and within a few days it broke. I understand that there is a desire to tweak the model and improv…

In the API you can ask for a specific version. Were you doing that?

They almost certainly were. But the API only offers two default choices of GPT-4 (unless one has been anointed with exalted 32k access):

1. gpt-4-default which has been progressively nerfed with continuous no notification zero changelog zero transparency updates .

2. gpt-4-0314 which is a frozen checkpoint from shortly after public launch and is still great but not quite as good as the very original, or as flexible as the fridged based model. Fine. However it’s currently due to “no longer be supported” i.e. retired on June 14th.

It’s kind of a challenge to commit to building on murky quicksand foundations of an API product that changes drastically (but ineffably for the worse) without warning like the default accessible version does, and soon it looks like there won’t be a stable un-lobotomized alternative.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#674

Post by AtNightWeCode »

I have not seen any decrease in speed with ChatGPT-4. It has become more stupid though. It gives results that you even ask it not to give. It has some sort of amnesia.

Even if there are other services that in the short term can beat ChatGPT with this specific LLM it is obvious that they will eventually hit the same limits.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#675

Post by cyptus »

PeterisP wrote: post #617
bob1029 wrote: post #266

OpenAI's models feel 100% nerfed to me at this point. I had it solving incredibly complex problems a few months ago (i.e. write a minimal PDF parser example), but today you will get scolded for asking such a complicated task of it. I think they programmed a classifier layer to detect certain coding tasks and shut it down with canned BS. I like to imagine certain billion/trillion-dollar mega corps had a back-room say…

It's a big pain when trying to build things on top of the GPT-4 API. We had some experiments that were reliably, reproducibly achieving a goal, and then one day it suddenly stops working properly; then the student managed a different prompt that worked (again, reproducibly, with proper clean restarts from fresh context), and within a few days it broke. I understand that there is a desire to tweak the model and improv…

same for me - also the api itself is very unstable sometimes the same prompt finish’s within a minute, sometimes our client timesout after 10 minutes and sometimes the api sends a 502 bad gateway after 5-10 minutes. the very same request then runs fine within a few minutes after a delay of 5 minutes. the results vary very much, even with a temperature of 0.1

requests that needs responses with over ~2k tokens almost always fails, the 8k cannot be used

I try to use the api for classification of tickets, which i thought the model would be a good choice to use for

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#676

Post by com2kid »

ianbicking wrote: post #643
bdw5204 wrote: post #326

Earlier quoted context omitted.

I think the only real path forward is for somebody to create an open source "unaligned" version of GPT. Any corporate controlled AI is going to be nerfed to prevent it from doing things that its corporate master considers to not be in the interests of the corporation. In addition, most large corporations these days are ideological institutions so the last thing they want is an AI that undermines public belief in thei…

People here need to learn to chill out and use the API. The GPT API is not some locked down cage. Every so often it'll come back with complaints instead of doing what was asked, but that's really uncommon. Control over the system prompt and putting a bit of extra information around the requests in the user message can get you _so_ far. It feels like people are getting ready to build castles in their mind when they ju…

The API chat endpoint dramatically changes its responses every few weeks. You can spend hours crafting a prompt and then a week later the responses to that same prompt can become borderline useless.

Writing against the ChatGPT API is like working against an API that breaks every other week with completely undocumented changes.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#677

Post by pk-protect-ai »

bdw5204 wrote: post #326
bob1029 wrote: post #266

OpenAI's models feel 100% nerfed to me at this point. I had it solving incredibly complex problems a few months ago (i.e. write a minimal PDF parser example), but today you will get scolded for asking such a complicated task of it. I think they programmed a classifier layer to detect certain coding tasks and shut it down with canned BS. I like to imagine certain billion/trillion-dollar mega corps had a back-room say…

I think the only real path forward is for somebody to create an open source "unaligned" version of GPT. Any corporate controlled AI is going to be nerfed to prevent it from doing things that its corporate master considers to not be in the interests of the corporation. In addition, most large corporations these days are ideological institutions so the last thing they want is an AI that undermines public belief in thei…

Look at this: https://www.openpetition.eu/petition/online/securing-our-dig...

It does not guarantee "unalighned" models, but it is sure will help to bust concurrency and provide infrastructure for training public models.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#678

Post by pk-protect-ai »

rushingcreek wrote: post #665
pk-protect-ai wrote: post #658

Phind.com uses Bing search again. This have decreased the quality of results significantly. On the other hand GPT-4 can use Bing now too. I tried GPT-4 with bind only several times and it was so bad in comparison to GPT-4 and much worse then phind.com. Btw you can force the GPT-4 on phind.com if you use regenerate icon. I'm usually ending up with stopping inference and regenerating with GPT-4. In any case, the qualit…

we're adding back a dedicated gpt-4 mode to Phind in the next few days. (I'm the founder)

Thank you a lot!!!!!!!!!!!!!! My god, this week was a horror. If you would get rid of the Bing output this will be even much better. Your original search was much more theme related.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#679

Post by AtNightWeCode »

bob1029 wrote: post #266

OpenAI's models feel 100% nerfed to me at this point. I had it solving incredibly complex problems a few months ago (i.e. write a minimal PDF parser example), but today you will get scolded for asking such a complicated task of it. I think they programmed a classifier layer to detect certain coding tasks and shut it down with canned BS. I like to imagine certain billion/trillion-dollar mega corps had a back-room say…

I call BS on this. ChatGPT anyversion could never solve complex problems. That is just silly. I tried to get the version 4 to solve some very basic problems around generating code. It suggested to build several trees including a syntax tree and then project things between these trees. The solution I wrote is straight forward and not even 50 lines of code.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#680

Post by com2kid »

napier wrote: post #673
sebzim4500 wrote: post #618

Earlier quoted context omitted.

In the API you can ask for a specific version. Were you doing that?

They almost certainly were. But the API only offers two default choices of GPT-4 (unless one has been anointed with exalted 32k access): 1. gpt-4-default which has been progressively nerfed with continuous no notification zero changelog zero transparency updates . 2. gpt-4-0314 which is a frozen checkpoint from shortly after public launch and is still great but not quite as good as the very original, or as flexible a…

The latest GPT 3.5 model has actually been getting better at creative writing tasks on a regular basis, which is actually bad for certain tasks due to the token limit. Whereas before GPT 3.5 could write a short story and finish it up nicely in a single response, now days it is more descriptive (good!) and thus runs out of tokens before concluding (bad!)
Post reply on HN