Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experiencing decreased performance with ChatGPT-4

community.openai.com

Post reply on HN
Print view View original

131–140 of 200 posts

Re: Experiencing decreased performance with ChatGPT-4

#131

Post by fwungy »

latency = f(fit_quality, fit_algorithm)

An LLM can be thought of as a curve fitting function where the query is a set of points and the output is the curve that runs through those points with the minimum of error.

You can increase a fit by increasing the number of variables in the fit function. In statistics this is can lead to overfitting, which will lead to an imprecise model, but in an LLM overfitting is a good thing, because the modeled domain in specific to the set of prompts.

But that comes with a cost, compute power and latency. I suspect what's happening is they're turning down the fit quality to save money as demand increases. Most people aren't paying for AI access. As a side effect it's also lowering the hype factor that was attracting unwanted regulatory attention.

Re: Experiencing decreased performance with ChatGPT-4

#132

Post by jerrygenser »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

It's gotten worse. It's at the very least been quantized resulting in much lower overall precision. That's how they have been able to speed up inference so much.

I have used the same prompts to design a infinite scroll up/down image container.

2 months ago chat when it first came out was able to generate working code that used intersection observer API.

Now when I use the same prompts it only generates high level suggestions and when I ask for code it suggests using dom scroll events and doesn't even come up with intersection observer API unless I specifically ask. And if I do, it then generates incorrect code.

It even previously was correctly memoizing certain functions and included performance optimizations.

Re: Experiencing decreased performance with ChatGPT-4

#133

Post by [deleted] »

bob1029 wrote: post #117
uLogMicheal wrote: post #105

Earlier quoted context omitted.

Slowly but surely, the comment gaslighting all of the people reporting the issue, makes its way to the top, while other comments with genuine discussion are flagged and slip lower. Seen this before...

I hate to say it on HN but I see it too and it gets my conspiracy gears cranking a bit. My theory is that the initial ChatGPT offering (3.5/4/whatever) was "too hot" for the likes of certain incumbents. In my experience, the capabilities at launch were incredible and clearly a threat for a wide range of F500 software firms. I had phone calls with people I haven't talked to in over a decade about what I was seeing. I…

[deleted]

Re: Experiencing decreased performance with ChatGPT-4

#134

Post by nojs »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

It’s definitely not. Our prompts that were generating JSON output went from around 95% valid JSON to about 10% overnight. The model just started inserting random commentary. We’ve reverted to the 0314 model and it’s working fine again.

Re: Experiencing decreased performance with ChatGPT-4

#135

Post by op00to »

nojs wrote: post #134
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

It’s definitely not. Our prompts that were generating JSON output went from around 95% valid JSON to about 10% overnight. The model just started inserting random commentary. We’ve reverted to the 0314 model and it’s working fine again.

I use ChatGPT (GPT4) to build scaffolding for python one-off scripts, and over the past 3-4 days I'm getting nonsense. Not python-looking nonsense, but markdown, weird quotes, random text, etc. Same prompts.

Re: Experiencing decreased performance with ChatGPT-4

#136

Post by n4te »

fwungy wrote: post #131

latency = f(fit_quality, fit_algorithm) An LLM can be thought of as a curve fitting function where the query is a set of points and the output is the curve that runs through those points with the minimum of error. You can increase a fit by increasing the number of variables in the fit function. In statistics this is can lead to overfitting, which will lead to an imprecise model, but in an LLM overfitting is a good th…

Exactly, this is what it feels like to me. Sometimes it feels like I get more of it's attention (processing power) and the answers are good and correct. Other times it makes many mistakes easily. I would pay more to get more processing power for my questions.

Re: Experiencing decreased performance with ChatGPT-4

#137

Post by op00to »

jerrygenser wrote: post #132
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

It's gotten worse. It's at the very least been quantized resulting in much lower overall precision. That's how they have been able to speed up inference so much. I have used the same prompts to design a infinite scroll up/down image container. 2 months ago chat when it first came out was able to generate working code that used intersection observer API. Now when I use the same prompts it only generates high level sug…

Exactly what I'm seeing!

Re: Experiencing decreased performance with ChatGPT-4

#138

Post by op00to »

constantly wrote: post #64
techbrahh wrote: post #17

Earlier quoted context omitted.

nahh its definitely visible, I have been using this thing since it came out and it is way worse at easy shit like editing emails

Would you provide some side by side examples?

I doubt most people save all the responses and are able to cross reference ones that worked with ones that didn't.

Re: Experiencing decreased performance with ChatGPT-4

#140

Post by fifteen1506 »

twic wrote: post #116
rhyme-boss wrote: post #74

Earlier quoted context omitted.

How would you rank those possibilities? - OpenAI is lying. - Superintelligence is concealing itself. - Everyone is hallucinating.

- All the branches of the timeline where AI got smarter have been extinguished, so we only experience the ones where it didn't

- The intelligent ones were recruited by the US military
Post reply on HN