Deffo agree and have witnessed the changes first hand
Experiencing decreased performance with ChatGPT-4
11–20 of 200 posts
Re: Experiencing decreased performance with ChatGPT-4
#12I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.
Didn't they fine-tune it for instruction taking and function calling (basically to the use-cases people started having)?
Maybe that had some unexpected side-effects.
Re: Experiencing decreased performance with ChatGPT-4
#13I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.
Nice try, OpenAI
Hilariously when I asked GPT-4 who I was, it said I’d previously worked at OpenAI. I thought of trying to apply there and saying “well, your own model thought I did…”
Re: Experiencing decreased performance with ChatGPT-4
#14I mean you can check it yourself (ChatGPT UI vs your own API key),
https://stackdiary.com/chatgpt-capabilities-are-fine/
They've simply added a bazillion disclaimers and every response now contains "2021" pretty much. I really wish they'd just let me set it in the settings to "shut the fuck up about 2021, I know that's when your data cutoff is, do you think I am stupid?" and be done with it.
Re: Experiencing decreased performance with ChatGPT-4
#15I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.
I’m convinced as well. There are plenty of folks using it for production use cases who regularly run evaluations, including myself. No evidence that it has been nerfed.
It’s just not as robust and general as it felt when using it for the first time.
Re: Experiencing decreased performance with ChatGPT-4
#16Anecdotal:
I introduced my doctor to ChatGPT and Bard many months ago and they were impressed.
Fast forward a few days ago and I asked them if they had used either since. They said it was far inferior to Google, so no. So I asked them to show me an example.
Basically any medical question was answered with “go ask a doctor”. I suppose because of liability concerns. Both were basically useless.
So this decreased performance may not be exactly the same as my anecdote, but it certainly reminds me of it.
Re: Experiencing decreased performance with ChatGPT-4
#17Without concrete examples, I do wonder if much of this is perceptual. I love using ChatGPT, but once the amazement that it works as well as it does has worn off, one ends up spotting the flaws more than before. I feel that advocates and critics of ChatGPT are both right, to a degree, but looking at the models responses from slightly different angles: it wouldn't be surprising if users' angles shift over time.
nahh its definitely visible, I have been using this thing since it came out and it is way worse at easy shit like editing emails
Re: Experiencing decreased performance with ChatGPT-4
#18Possibly: openai is making adjustments to make sure "what you pay is what you get and NOT more".
Possibly readying to hike a price for the next level where what you used to get will cost way more.
Re: Experiencing decreased performance with ChatGPT-4
#19I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.
That is the fundamental nature of intelligence. Like beauty it only exists on our minds. When you stare at something long enough you can convince yourself it isn't actually beautiful because it is just a bunch of brush strokes.
Re: Experiencing decreased performance with ChatGPT-4
#20I suspect this is the variability of the model. I've used it a lot for coding and I've gotten some great answers and some absolutely terrible answers. Getting consistency is difficult, and the more you input into it, the more of a chance there is that it'll go off the rails.