Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experiencing decreased performance with ChatGPT-4

community.openai.com

Post reply on HN
Print view View original

151–160 of 200 posts

Re: Experiencing decreased performance with ChatGPT-4

#151

Post by [deleted] »

anonymoushn wrote: post #68
xianshou wrote: post #45

GPT-4 (5/24 version) indeed fails to solve the problem if not given careful prompting, though I am not convinced this is a new development. However, chain-of-thought resolves the issue. Both prompts and responses included below. --- Failure Q: How can you use a 4-minute hourglass and a 7-minute hourglass to measure 9 minutes? A: Here's a way to measure exactly 9 minutes using a 4-minute hourglass and a 7-minute hourg…

success is saying "eyeball the sand in the 7-minute hourglass and see when it looks like 1 minute worth?"

[deleted]

Re: Experiencing decreased performance with ChatGPT-4

#152

Post by afro88 »

fwungy wrote: post #131

latency = f(fit_quality, fit_algorithm) An LLM can be thought of as a curve fitting function where the query is a set of points and the output is the curve that runs through those points with the minimum of error. You can increase a fit by increasing the number of variables in the fit function. In statistics this is can lead to overfitting, which will lead to an imprecise model, but in an LLM overfitting is a good th…

How do you turn down the fit quality of a static model?

Re: Experiencing decreased performance with ChatGPT-4

#153

Post by yuck39 »

nojs wrote: post #134
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

It’s definitely not. Our prompts that were generating JSON output went from around 95% valid JSON to about 10% overnight. The model just started inserting random commentary. We’ve reverted to the 0314 model and it’s working fine again.

I had an API integration written to convert an English language security rule into an XML object designed to instruct a remote machine how to comply with the rule programmatically. April 2023 we had about an 86% accept rate, that number has declined to 31% with no changes to the prompt.

Re: Experiencing decreased performance with ChatGPT-4

#154

Post by reissbaker »

mustacheemperor wrote: post #147
nojs wrote: post #134

Earlier quoted context omitted.

It’s definitely not. Our prompts that were generating JSON output went from around 95% valid JSON to about 10% overnight. The model just started inserting random commentary. We’ve reverted to the 0314 model and it’s working fine again.

Have you tried using the recently releases function calling API? That’s reliable at returning json in my experience, although I’ve just tinkered with it, not used it for anything “real.”

My guess is that the degradation of JSON capability happened recently? The gpt-4 API switched over to gpt-4-0613 (the function calling version) on June 27. And given the performance increase for ChatGPT Plus at the end of May, my guess is they started testing the new model (which is much faster) on web users around then. In my testing [1], the new version is:

a. Worse at general code-like tasks without using functions

b. Equivalent or better at code-like tasks if you use the function API

c. Much faster than the older model either way.

I'd guess it's cheaper to run, too, and that they use the presence of a function in the API signature to weight their mixture of experts differently (and cull some experts?). The degradation in general purpose coding tasks is pretty obvious and repeatable (try the same prompts in the Playground with the -0314 model vs the -0613!), but it does seem like you can regain that lost capability with the new function call API, and it's faster. The tradeoff is that you only regain the capability when it calls functions; you can't really have a mix of prose-and-code in the same response as easily, or at least not with the same quality.

1: https://twitter.com/reissbaker/status/1671361372092010497

Re: Experiencing decreased performance with ChatGPT-4

#155

Post by TeMPOraL »

capableweb wrote: post #44
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

I think it's more likely that people are confused, and OpenAI is not making things any clearer either. AFAIK, OpenAI has repeatedly stated that GPT4 hasn't changed. People repeatedly states that when they use ChatGPT, they get a difference experience today than before. Both can be true at the same time, as ChatGPT is a "packaged" experience of GPT4, so if you use the API versions, nothing has likely changed. But Chat…

> so if you use the API versions, nothing has likely changed

I doubt that. I don't recall them actually clearly and precisely saying they aren't changing the 'gpt-4' model - i.e. the model you're getting when specifying 'gpt-4' in an API call. That one direct tweet I recall, which I think you're referring to, could be read more narrowly as saying the pinned versions didn't change.

That is, if you issue calls against 'gpt-4-0314', then indeed nothing changed since its release. But with calls against 'gpt-4', anything goes.

This would be consistent with their documentation and overall deployment model: the whole reason behind the split between versioned (e.g. 'gpt-4-0314', 'gpt-4-0613') and unversioned models (e.g. 'gpt-4') was so that you could have both stable base and a changing tip. If that tweet is to be read as saying 'gpt-4' didn't change since release, then the whole thing with versioning is kind of redundant.

Re: Experiencing decreased performance with ChatGPT-4

#156

Post by deevolution »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

Aren't they using RLHF? The feedback from humans might not always be the ~right~ feedback. Couldn't that possibly degrade the quality of its responses?

Re: Experiencing decreased performance with ChatGPT-4

#157

Post by RA_Fisher »

yuck39 wrote: post #153
nojs wrote: post #134

Earlier quoted context omitted.

It’s definitely not. Our prompts that were generating JSON output went from around 95% valid JSON to about 10% overnight. The model just started inserting random commentary. We’ve reverted to the 0314 model and it’s working fine again.

I had an API integration written to convert an English language security rule into an XML object designed to instruct a remote machine how to comply with the rule programmatically. April 2023 we had about an 86% accept rate, that number has declined to 31% with no changes to the prompt.

I’m curious, how do y’all keep track of performance and reliability?

I ask, because I think it’s going to be a big challenge, so I built a service to record feedback / acceptance data: https://modelgymai.com/

If you think it can help, I’d love if you’d try it out and let me know if it helps.

Re: Experiencing decreased performance with ChatGPT-4

#158

Post by greiskul »

kccqzy wrote: post #141
Wowfunhappy wrote: post #108

Earlier quoted context omitted.

> Notice how you never hear anyone saying that GPT-4 is better since the launch. You'd expect to hear something like that as people gain more experience with prompting it. I'd expect the opposite. The first time you use ChatGPT (or GPT-4), you're in awe of what it can do, and more willing to overlook failures. As you use it, it becomes more mundane, and the instances where it messes up become more obvious.

I've noticed the same thing. People also like to complain the quality of Google Search has gone down for much of the same reason: if you first do a Google search that returned a good result and then repeat it, you are going to notice the absence. But if you first do a Google search that didn't return the thing you expect you might think such a thing just doesn't exist on the Internet. Ergo, quality decrease is simply…

For Google Search, the sad part, is that the search algorithm hasn't gotten worse. But the web itself has. There is so much more spam, with actual human generated content being siloed more and more in walled gardens, that it starts to become a major issue for Google.

Re: Experiencing decreased performance with ChatGPT-4

#159

Post by reissbaker »

capableweb wrote: post #44
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

I think it's more likely that people are confused, and OpenAI is not making things any clearer either. AFAIK, OpenAI has repeatedly stated that GPT4 hasn't changed. People repeatedly states that when they use ChatGPT, they get a difference experience today than before. Both can be true at the same time, as ChatGPT is a "packaged" experience of GPT4, so if you use the API versions, nothing has likely changed. But Chat…

OpenAI released a new gpt-4 model on June 13 https://openai.com/blog/function-calling-and-other-api-updat..., and they update gpt-4 to the latest version every two weeks (aka, gpt-4 switched over to the -0613 on June 27).

The -0613 version is really different! It added function calling to the API as a hint to the LLM, and in my experience if you don't use function calling it's significantly worse at code-like tasks, but if you do use it, it's roughly equivalent or better when it calls your function.

Re: Experiencing decreased performance with ChatGPT-4

#160

Post by viraptor »

carabiner wrote: post #48
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

Seriously... In that 134 replies thread, 0 transcripts showing actual performance degradation. Just endless "Yes, it seems bla bla." No evidence but just shapes in the clouds.

So you just skimmed the thread. There are comparisons, there are specific transcripts. There are examples without transcripts.
Post reply on HN