Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experiencing decreased performance with ChatGPT-4

community.openai.com

Post reply on HN
Print view View original

51–60 of 200 posts

Re: Experiencing decreased performance with ChatGPT-4

#51

Post by neonsunset »

crop_rotation wrote: post #23

I don't know about the API, but the ChatGPT UI GPT4 really became much worse, so much so that I had to cancel my subscription. It's not just about novelty factor. I used to store all my prompts locally, and when I compare old responses and new ones, there is a huge difference. OpenAI employee said the API model doesn't change, but were careful to not say anything about the UI. I am now waiting to get access to the AP…

At this point the default assumption should be "people who are downvoting posts like these are dis-ingenious and do so with an agenda".

Otherwise, I had to do the same, as it has been so lobotomized that doing many tasks I used to delegate to GPT-4 has become easier to do manually once again.

Re: Experiencing decreased performance with ChatGPT-4

#52

Post by yousif_123123 »

They must be (and I hope) continuosly optimizing it. A lot of ideas with using quantization have been coming out this year, they must have learned new things.

The API should be more resilient but the ChatGPT app IMO should be expected to change in how it handles prompts, as it's constantly being fine-tuned etc.

Just like any saas product I think they have the right to update their software.

I still find GPT4 so powerful but also so prone to making huge mistakes (like calling a function in a library that doesn't exist when providing code)..

Re: Experiencing decreased performance with ChatGPT-4

#53

Post by themodelplumber »

cyberlurker wrote: post #16

Anecdotal: I introduced my doctor to ChatGPT and Bard many months ago and they were impressed. Fast forward a few days ago and I asked them if they had used either since. They said it was far inferior to Google, so no. So I asked them to show me an example. Basically any medical question was answered with “go ask a doctor”. I suppose because of liability concerns. Both were basically useless. So this decreased perfor…

I think you are going to see another wave of doctors and medical professionals becoming closet coders:

1970s-80s: "They don't provide a computer at work, but this BASIC software is amazing for all things relevant to my job...databases, scheduling, formulae...plus it's private to me, not in some mainframe."

So you had tons of doctors learning to code or hiring coders to set up their offices with this stuff. And it was functionally air-gapped.

(...Trend repeats in various ways over the years...)

Soon: "They don't provide anything like it at work, and even this free LLM software is amazing for all things relevant to my job...diagnosis, interventions, references based on specific context...plus it's private to me and my office when run locally, not in somebody's cloud."

And, prompting an LLM is de facto coding, moreso the more detailed and specialized the session.

This could skip some huge problems with the LLM commercial service model, and provide tons of additional specific contextual benefits depending on the configuration.

Plus, doctors already listen to patients throwing out red herrings left and right, so even unreliable information from the LLM will be available in a context where the provider knows how to rule things out anyway...

Re: Experiencing decreased performance with ChatGPT-4

#54

Post by dom2 »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

I have yet to see any real data on this phenomenon outside of anecdotal stories, so I'm also in the same boat re: group hallucination. Would be interested in seeing some more substantial evidence.

Re: Experiencing decreased performance with ChatGPT-4

#55

Post by anonymoushn »

carabiner wrote: post #48
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

Seriously... In that 134 replies thread, 0 transcripts showing actual performance degradation. Just endless "Yes, it seems bla bla." No evidence but just shapes in the clouds.

RealistCC posted a pair of transcripts in reply 41. I haven't read the rest of the replies.

Re: Experiencing decreased performance with ChatGPT-4

#56

Post by acumenical »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

I don't agree. As someone who has written many jailbreak prompts, the very fact that earlier jailbreak prompts no longer work indicates to me that the integration has changed. The model might be the same, but filtering the input extensively might cause undefined behavior.

Re: Experiencing decreased performance with ChatGPT-4

#58

Post by sigmoid10 »

Timpy wrote: post #21
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

I remember the first time I played Minecraft and I was in awe at how expansive the play world felt. Without thinking too much about it, I had the feeling that if I set off in any direction I would discover infinitely new things. After enough playtime I saw the repeating patterns and eventually it felt so small again.

People will always see what they want to see. I've had so many interactions with customers over the years who thought that a service or feature was removed or crippled when in fact nothing had changed on our side. The only thing that changed is their perception. Especially when they can't get something to work and they believe they have succeeded at something similar before, they'll always suspect that the software is at fault instead of their own memory.

Re: Experiencing decreased performance with ChatGPT-4

#59

Post by Terretta »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

I use the API, not the chat site.

Since 30 June, the API responses are making common English misspelling errors, of the type where two words sound the same with different meanings such as break and brake.

I saw this happen zero times in the prior GPT-4 model, and multiple times this July, on multiple conversation topics and multiple word pairs.

Curiously, they're behaving as misspellings rather than mismeanings, since the sentence continuation is as if the correct meaning had been used.

I acknowledge this could be a blend of pareidolia and the Baader-Meinhof phenomenon.

Re: Experiencing decreased performance with ChatGPT-4

#60

Post by afpx »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

Are you talking about GPT-4 or ChatGPT GPT-4? The GPT-4 model hasn’t changed, and that was confirmed by developers at OpenAI a while back IIRC. But, ChatGPT is always undergoing changes. I assume they have a layer or two on top of the model that is being trained with reinforcement learning.
Post reply on HN