Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is getting worse over time, not better

twitter.com

Post reply on HN
Print view View original

121–130 of 315 posts

Re: GPT-4 is getting worse over time, not better

#121

Post by xtracto »

ChatGTP wrote: post #34
79a6ed87 wrote: post #12

I've been paying for GPT-4 since 3 hours after its release. The decrease in quality was noticeable just one week later (on top of the cap changes from 50 messages every 4 hours to 25 messages every 3 hours) I originally assumed that this was due to the increase in demand. It never went back to being as sharp as it was during those first hours of usage

Are you sure it wasn't just that the novelty wore off after a few hours of usage? I never really got into LLMs, but I must say at first it seemed like pretty cool stuff. OpenAI have repeatedly stated the model hasn't changed so how could this happen otherwise?

These models have been explicitly nerfed since their first release due to copyright considerations. I've mentioned in two previous cases both for [1] code generation and [2] book summarizing.

From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation. The funny thing is that in say, 10 years, the "pirate" version of LLMs will be way more powerful and useful than the "corporate" versions. Once the RIAA/MPAA has sued all of them preventing them of using their audio/video; after the editorials have sued them to prevent them from using their books and articles, and after every internet site has sued them to prevent them from using their text. LLM models trained on SciHub, Library Genesis and Torrents will be amazing in comparison.

I sincerely wish that some country would apply to information copyright a similar approach to what India does for medicine patents.

[1] https://news.ycombinator.com/item?id=31852138 [2] https://news.ycombinator.com/item?id=36138930

Re: GPT-4 is getting worse over time, not better

#122

Post by damnesian »

phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

> it's just that the magic has worn off as we've used this tech

I agree with this. The analogy I use on repeat is the dawn of moving picture making. The first movies were short larks, designed just to elicit a response. Just like when CGI was new- a bunch of over-the-top, sensationalist fluff got made. This tech needs to mature. And we need it to continue to be fed the work of real humans, not AI feeding on AI, a recent phenomenon that hopefully will not turn out to be the norm. If we water and feed it responsibly, it will only grow more capable and useful over time.

Re: GPT-4 is getting worse over time, not better

#123

Post by mckirk »

phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

It has definitely gotten worse. At least its performance when prompted in German has degraded noticeably, to the point where it's making grammatical mistakes, which it never has before.

Re: GPT-4 is getting worse over time, not better

#124

Post by ambrozk »

majkinetor wrote: post #63
phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

So the original paper from Stanford and Berkley is also linked to this AI influencer? I am really amazed by this kind of dismissal. Its totally irrelevant who posted the info and how framed it is, as long as you have access to the source.

The paper does things like ask GPT-4 to write code and then check if that code compiles. Since March, they've fine-tuned GPT-4 to add back-ticks around code, which improves human-readable formatting but stops the code compiling. This is interpreted as "degraded performance" in the paper even though it's improved performance from a human perspective.

Re: GPT-4 is getting worse over time, not better

#126

Post by yodsanklai »

ChatGTP wrote: post #34
79a6ed87 wrote: post #12

I've been paying for GPT-4 since 3 hours after its release. The decrease in quality was noticeable just one week later (on top of the cap changes from 50 messages every 4 hours to 25 messages every 3 hours) I originally assumed that this was due to the increase in demand. It never went back to being as sharp as it was during those first hours of usage

Are you sure it wasn't just that the novelty wore off after a few hours of usage? I never really got into LLMs, but I must say at first it seemed like pretty cool stuff. OpenAI have repeatedly stated the model hasn't changed so how could this happen otherwise?

It's a reasonable hypothesis. Whenever phenomenons are probabilistic, we're poorly equipped to use our direct experience to assess what they do. We see that all the time with pseudo-medicine for instance. And people became extremely defensive about these questions (vaccines, pseudo-sciences...) - which is why you're downvoted.

Re: GPT-4 is getting worse over time, not better

#127

Post by JCharante »

dudeinjapan wrote: post #41
x86hacker1010 wrote: post #20

Earlier quoted context omitted.

How’s life in Japan?

Life is good, Japan is a wonderful place to live. Was just in Kyoto last weekend for the Gion Matsuri, a big parade festival. If you're interested to come, check out my company TableCheck: https://careers.tablecheck.com/ We have a really talented and motivated team, great clients, and have a lot of fun.

a parade festival? A festival of parades?

Re: GPT-4 is getting worse over time, not better

#128

Post by BryanLegend »

superkuh wrote: post #3

Every time a LLM is fine-tuned it gets stupider and less capable compared to the bare model. openai's legal and social ass covering attempts to neuter their model's output via fine tuning have done the same.

"Social ass" is a great euphemism for influencer.

Re: GPT-4 is getting worse over time, not better

#129

Post by londons_explore »

Has anyone experimented with mixing outputs from LLm's on a per-token basis?

Ie. easy tokens can be provided by a cheap-to-run model, and hard tokens are given by an expensive to run model?

A model could be used to decide when it is worth running the expensive model, based on the inputs, output so far, and probability distribution of the output of the cheap model.

For example, "Q: If I have 3 bananas and eat none, then how many bananas do I have?"

"A: You would have 3 bananas left, since you started with 3 and didn't eat any"

The "3" would come from the big model, while the rest all came from a small model.

Re: GPT-4 is getting worse over time, not better

#130

Post by Slava_Propanei »

chaxor wrote: post #76
phreeza wrote: post #5

Earlier quoted context omitted.

That is blatantly false, these models only work as well as they do in the first place because of instruction fine tuning. The raw models are much harder to work with.

I believe they are referring to the original observations of code output degradation due to adding training epochs for safety. These observations were made by Microsoft working directly with OpenAI early on. They comment on it in the technical lecture for GPT-4 in the famous 'Tikz Unicorn' example, wherein the model got better and better at making the unicorn during typical training, but then regressed when training…

[dead]
Post reply on HN