Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experiencing decreased performance with ChatGPT-4

community.openai.com

Post reply on HN
Print view View original

21–30 of 200 posts

Re: Experiencing decreased performance with ChatGPT-4

#21

Post by Timpy »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

I remember the first time I played Minecraft and I was in awe at how expansive the play world felt. Without thinking too much about it, I had the feeling that if I set off in any direction I would discover infinitely new things. After enough playtime I saw the repeating patterns and eventually it felt so small again.

Re: Experiencing decreased performance with ChatGPT-4

#23

Post by crop_rotation »

I don't know about the API, but the ChatGPT UI GPT4 really became much worse, so much so that I had to cancel my subscription. It's not just about novelty factor. I used to store all my prompts locally, and when I compare old responses and new ones, there is a huge difference. OpenAI employee said the API model doesn't change, but were careful to not say anything about the UI. I am now waiting to get access to the API GPT4 to try it again.

Re: Experiencing decreased performance with ChatGPT-4

#24

Post by facu17y »

azhenley wrote: post #5

Wonder if this is the same as the discussion from 35 days ago on "OpenAI Employee: GPT-4 has been static since March" https://news.ycombinator.com/item?id=36155267

the base model may have been but not necessarily the RLHF fine-tuned layers they might have added or the shortcut they're taking during inference due to such fine tuning (or for perf optimization unrelated to fine tuning.)

Re: Experiencing decreased performance with ChatGPT-4

#25

Post by [deleted] »

supriyo-biswas wrote: post #9
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

They've definitely changed something about the models, and it is in their interests to do so, both to create a low-latency experience, but most importantly, to save money. While GPT-4 is still workable, GPT-3.5 flatly refuses requests these days, claiming that as an "AI language model" it couldn't help me write code.

[deleted]

Re: Experiencing decreased performance with ChatGPT-4

#26

Post by themodelplumber »

supriyo-biswas wrote: post #9
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

They've definitely changed something about the models, and it is in their interests to do so, both to create a low-latency experience, but most importantly, to save money. While GPT-4 is still workable, GPT-3.5 flatly refuses requests these days, claiming that as an "AI language model" it couldn't help me write code.

Did you get the help you needed in the end? I've seen it do that once but was able to cajole it within 1-2 prompts.

Re: Experiencing decreased performance with ChatGPT-4

#27

Post by rolisz »

skepticATX wrote: post #15
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

I’m convinced as well. There are plenty of folks using it for production use cases who regularly run evaluations, including myself. No evidence that it has been nerfed. It’s just not as robust and general as it felt when using it for the first time.

Folks using it in production are using the API, where you can explicitly select a model. In the post, the poster is using ChatGPT, through the web app, where you can't select exact model and they sometimes do updates.

Re: Experiencing decreased performance with ChatGPT-4

#28

Post by m3kw9 »

The thing is they(OpenAI) really could mess with the temperature/probability settings, and other settings for various reasons that can cause something like this to happen to some people at certain random times. Hence is hard to reproduce, and only OpenAI knows it, either they acknowledge or not that is the based on business requirements

Re: Experiencing decreased performance with ChatGPT-4

#29

Post by tevon »

More than anything this highlights the difficulty in testing or trusting non-deterministic systems from a user perspective.

Whether or not GPT-4 is truly degraded, there will always be users who experience strange or sub-optimal responses and will be able to find other users who experience the same.

Quite a challenging space to build trust! We expect machines to act deterministically, now we as users will need to re-wire our thinking.

Re: Experiencing decreased performance with ChatGPT-4

#30

Post by exo-pla-net »

Human problem, not ChatGPT-4 problem. People notoriously have rose-tinted glasses with their memories. ChatGPT has always been imperfect, but now there's a something of a mild hysteria (sort of like sick building syndrome) that it's gotten worse. It hasn't.

Case in point: nobody can provide evidence that it's gotten worse, even though chat logs are superabundant, and providing evidence should be trivial.

But for those who put stock in anecdotes: I use it heavily, and it seems the same to me! When I first got access, I began with testing its limits, and it has always had sharp ones. It's still a lovely tool for a lot of tasks, but the honeymoon period is apparently over for some people.

Post reply on HN