[deleted]
Experiencing decreased performance with ChatGPT-4
161–170 of 200 posts
Re: Experiencing decreased performance with ChatGPT-4
#162I posted in a similar thread yesterday. I recently completed a fairly extensive benchmarking effort, where I asked GPT-3.5 and GPT-4 to solve 133 exercises from the Exercism Python practice set. I measured the performance of the Feb (0301) and June (0613) versions of gpt-3.5-turbo and gpt-4.
This is the only systematic, quantitative evaluation that I am aware of that compares the different versions of the OpenAI models over time.
My conclusions:
- The new June GPT-3.5 models did a bit worse than the old Feb model.
- For GPT-4, there wasn't much difference between June and Feb. June was maybe a bit better.
- It hurts coding performance to have GPT package up code inside the new function calls API.
- As expected, GPT-4 is better than GPT-3.5 at code editing.
https://aider.chat/docs/benchmarks.html
Some specific notes about GPT-3.5 getting a bit worse in June are here:
https://aider.chat/docs/benchmarks.html#the-0613-models-seem...
Re: Experiencing decreased performance with ChatGPT-4
#163Earlier quoted context omitted.
I don't understand your anecdote. I'm able to ask it medical questions and get answers, for example: https://chat.openai.com/share/75f94000-552f-42d6-aadf-198fd9... https://chat.openai.com/share/0933abf7-1015-41b5-9a49-ca2b6e... Whether someone should trust the answers is a different question.
He asked a dosage question similar to your second example and without tricking it, it would not give a response. The drugs were not as common (at least I wasn’t familiar with them) as what you listed, so maybe that had something to do with it.
Honestly, I'm happy they don't use it. Doctors should never ask ChatGPT about dosage - there are easily accessible official sources that will tell them the right answer. It doesn't even matter if it happens to be right most of the time, this is just an accident waiting to happen.
Re: Experiencing decreased performance with ChatGPT-4
#164I also noticed degradation in quality of GPT-4. I assume it's because they keep doing RLHF to censor GPT-4 in certain ways to avoid legal hassles and this bleeds into it's quality in general.
[deleted]
Re: Experiencing decreased performance with ChatGPT-4
#165latency = f(fit_quality, fit_algorithm) An LLM can be thought of as a curve fitting function where the query is a set of points and the output is the curve that runs through those points with the minimum of error. You can increase a fit by increasing the number of variables in the fit function. In statistics this is can lead to overfitting, which will lead to an imprecise model, but in an LLM overfitting is a good th…
How do you turn down the fit quality of a static model?
Reduce the parameter count. Increase the acceptable error.
The more parameters on a curve fit the better the fit will be, but the compute power increases too.
Re: Experiencing decreased performance with ChatGPT-4
#166Earlier quoted context omitted.
I think it's more likely that people are confused, and OpenAI is not making things any clearer either. AFAIK, OpenAI has repeatedly stated that GPT4 hasn't changed. People repeatedly states that when they use ChatGPT, they get a difference experience today than before. Both can be true at the same time, as ChatGPT is a "packaged" experience of GPT4, so if you use the API versions, nothing has likely changed. But Chat…
OpenAI released a new gpt-4 model on June 13 https://openai.com/blog/function-calling-and-other-api-updat... , and they update gpt-4 to the latest version every two weeks (aka, gpt-4 switched over to the -0613 on June 27). The -0613 version is really different! It added function calling to the API as a hint to the LLM, and in my experience if you don't use function calling it's significantly worse at code-like tasks,…
Can I ask how you use this function calling in your workflow? Any examples?
Re: Experiencing decreased performance with ChatGPT-4
#167Earlier quoted context omitted.
Slowly but surely, the comment gaslighting all of the people reporting the issue, makes its way to the top, while other comments with genuine discussion are flagged and slip lower. Seen this before...
I hate to say it on HN but I see it too and it gets my conspiracy gears cranking a bit. My theory is that the initial ChatGPT offering (3.5/4/whatever) was "too hot" for the likes of certain incumbents. In my experience, the capabilities at launch were incredible and clearly a threat for a wide range of F500 software firms. I had phone calls with people I haven't talked to in over a decade about what I was seeing. I…
Indeed. I have a sinking feeling they realized (or were otherwise convinced) those models are too disruptive to existing businesses and whole market segments, in particular (but not limited to) when it comes to writing code. Or at least that's where it's most obvious to me just how many different classes of companies could grow and capture value[0] that GPT-4 has been providing, pay-as-you-go, for a dozen cents per use. But the same must be true in many other industries.
Come to think of it, it must be the case, because the alternative would be pretty much every player on the market taking the hit and carrying on, or pretending they don't see the untapped value source that just freely flows out of OpenAI for anyone to enjoy, for a modest fee.
As a prime example, I'd point out Microsoft and their various copilots - the code one, the Office 365 one, the Windows system-wide one, in varying stages of development. API access to GPT-4 as good as it originally was[1], directly devalues all of those.
It stands to reason that slowly making the model dumber, while also making it faster and cheaper to use, is the best way for OpenAI to safeguard big players' markets - the "faster" and "cheaper" give perfect cover, while the overall effect is salting the entire space of possibilities - making the model good enough to entertain the crowd, but just not good enough to build solutions on top, not unless you're working for one of the players with special deals.
TL;DR: too many entities with money were unhappy about all the value OpenAI was giving to the world for peanuts, so the model is being gradually nerfed in a way that allows that value to be captured, controlled, and doled out for a hefty price.
(And if that turns out to be true, I'm going to be really pissed. I guess it's in the style of humanity to slow down pace of development not because of ideology, not because of potential risks, but because it's growing too fast to fully monetize.)
--
[0] - I mean that in the most nasty, parasitic sense possible.
[1] - I'm talking about the public release. That GPT-4 version seems to have already been weakened compared to pre-"safety tuning" GPT-4 (see the TikZ Unicorn benchmark story), but we can't really talk about what we never got to play with.
Re: Experiencing decreased performance with ChatGPT-4
#168I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.
I use the API, not the chat site. Since 30 June, the API responses are making common English misspelling errors, of the type where two words sound the same with different meanings such as break and brake. I saw this happen zero times in the prior GPT-4 model, and multiple times this July, on multiple conversation topics and multiple word pairs. Curiously, they're behaving as misspellings rather than mismeanings, sinc…
I've 100% noticed a steep decline in the quality of things like grammar, sentence structure, spelling, and syntax in the last few weeks.
GPT-4 used to write with consistent under-grad level quality. Now it's closer to a junior high school kid.
Re: Experiencing decreased performance with ChatGPT-4
#169Earlier quoted context omitted.
How do you turn down the fit quality of a static model?
Reduce the parameter count. Increase the acceptable error. The more parameters on a curve fit the better the fit will be, but the compute power increases too.
Is that possible? I thought parameter counts were fixed in the model.
Re: Experiencing decreased performance with ChatGPT-4
#170Earlier quoted context omitted.
I hate to say it on HN but I see it too and it gets my conspiracy gears cranking a bit. My theory is that the initial ChatGPT offering (3.5/4/whatever) was "too hot" for the likes of certain incumbents. In my experience, the capabilities at launch were incredible and clearly a threat for a wide range of F500 software firms. I had phone calls with people I haven't talked to in over a decade about what I was seeing. I…
Indeed. I have a sinking feeling they realized (or were otherwise convinced) those models are too disruptive to existing businesses and whole market segments, in particular (but not limited to) when it comes to writing code. Or at least that's where it's most obvious to me just how many different classes of companies could grow and capture value[0] that GPT-4 has been providing, pay-as-you-go, for a dozen cents per u…
I've smelt the sweet scent of anticompetitive back-room dealing around OpenAI ever since they and Microsoft started forcing people to apply for access to the APIs and including telling them what the use case they were going to use it was.
It just seemed obvious that if anyone suggested a use case that was actually really high value MS would just take the idea, run with it for a month or two to see if it has legs, and then steal it if it actually worked.
All while you're waiting in the queue to have your idea validated as "safe".