Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#31

Post by xbmcuser »

PoignardAzur wrote: post #28
xbmcuser wrote: post #12

Earlier quoted context omitted.

When chat gpt first came out I was able to feed it some text to parse and then create python scripts to process similar texts and create csv and excel files from those. I was able to create a basic working python scripts in 1-2 hours. And very complex scripts over a couple of days. I recently tried to do the same with chatgpt again and simply am unable to. I wish I had saved the exact text I fed into chat gpt then so…

ChatGPT has a query history. Can't you fish the first queries you made on your account?

Like I said I wish I kept it this was almost 2 years ago I have deleted my query history many times since then

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#32

Post by LatticeAnimal »

kolinko wrote: post #7
supriyo-biswas wrote: post #6

One of the things I've personally observed is that ChatGPT has become very verbose these days. Previously, it used to return the right amount of information in most contexts, and I can't get that behavior back with prompts asking it to be concise, because then it'll just omit important parts, prioritizing providing a extremely high-level summary that elucidates very little. No opinion on Claude because I've not had a…

+1 on verbosity - it happened when switching from 4t to 4o I think, and personally I don’t like it. Should be fizable with system prompt though.

Recently gpt-4-turbo started rejecting writing some tests because it 'knows' it would exceed the max context. (This frustrated me deeply -- It would not have exceeded the context)

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#34

Post by f0e4c2f7 »

There have been some papers showing that RLHF makes models more palletable to use but reduces performance on evals and in other various ways.

I couldn't find the one I was looking for but this is one of them.

https://arxiv.org/abs/2310.06452

Edit:

This tweet also has a screenshot showing degraded evals from RLHF from base model.

https://x.com/KevinAFischer/status/1638706111443513346?t=0wK...

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#35

Post by eltoxo »

xbmcuser wrote: post #12
latexr wrote: post #4

> If there is indeed no degradation how could the perceived degradation be explained? By being disproportionately impressed previously. Maybe in the early days people were so impressed by their little play experiments they forgave the shortcomings. Now that the novelty is wearing off and they try to use it for productive work, the scales tipped and failures are given more weight.

When chat gpt first came out I was able to feed it some text to parse and then create python scripts to process similar texts and create csv and excel files from those. I was able to create a basic working python scripts in 1-2 hours. And very complex scripts over a couple of days. I recently tried to do the same with chatgpt again and simply am unable to. I wish I had saved the exact text I fed into chat gpt then so…

I don't know, my experience is all this is really hard to judge.

I would have a really hard time saying that 4 in April 2023 was better than 4o now though.

I have always wonder if it matters what time of day you are using it too. I feel like 4am EST works better than 4PM EST but it is so hard to judge. I think there is so much difference too with just how the prompt is phrased so it ends up feeling like some days it is is good and some days it sucks.

That is coupled with I have got a bad result before, opened a new chat window, pasted the exact same prompt and got a good result.

If I had to bet, I imagine it is like flipping quarters. Sometimes you will get runs of heads, sometimes runs of tails and sometimes a real mixed bag of both.

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#36

Post by Spooky23 »

latexr wrote: post #4

> If there is indeed no degradation how could the perceived degradation be explained? By being disproportionately impressed previously. Maybe in the early days people were so impressed by their little play experiments they forgave the shortcomings. Now that the novelty is wearing off and they try to use it for productive work, the scales tipped and failures are given more weight.

They tweak it to make it safer. Every time that happens, it gets a little dumber.

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#37

Post by edmundsauto »

Hamel Husein had this great slide in a recent Youtube video where he compared human raters of LLM outputs. He pointed out a peculiarity, where it looked like there wasn't improvement between pipeline versions but it was actually because the raters themselves began to have higher expectations over time.

The same output in week 0 was rated as a 7. After 6 weeks of rating LLM outputs, especially as the pipeline improved them, was a lower score later.

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#38

Post by PeterStuer »

Spooky23 wrote: post #36
latexr wrote: post #4

> If there is indeed no degradation how could the perceived degradation be explained? By being disproportionately impressed previously. Maybe in the early days people were so impressed by their little play experiments they forgave the shortcomings. Now that the novelty is wearing off and they try to use it for productive work, the scales tipped and failures are given more weight.

They tweak it to make it safer. Every time that happens, it gets a little dumber.

I think in this case it is more tweaking for inference cost. The full models are never exposed as using them at scale is too expensive.

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#40

Post by notaharvardmba »

My opinion is pretty logical: If you train stuff on the entire web, the first training set will be the only set of data that doesn’t include model generated data and thus will be the most realistic about “what is human seeming”. Now the web is full of generated content. That will tend to bias the model over time if you continue to train from the web. There really was only ever one chance to do the web training thing and now it’s over and done. We will have to go back to carefully curated training sets or come up with a truly failsafe way to detect and not ingest model generated content from the web, or you’re basically eating your own feces which will cause model feedback and hysteresis, leading to bias. This is a very big picture view, but it does seem that the “great leap” 2020-2023 was because we got to do this one time ingestion of a wide amount of clean data, and now it’s going to go back to training quality to get better results.
Post reply on HN