Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training open-source LLMs on ChatGPT output is a really bad idea.

gist.github.com

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#31

Post by creatonez »

dvt wrote: post #8

> So I can easily imagine a near future where the web will be flooded by LLM output or at least by content heavily inspired or edited by LLMs. To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reas…

Can you give an example of any significant scale of fully automatically published blog posts from 10 years ago? As far as I know, most of these crappy articles were content farms, often using templates and outsourced labor, but not automatically generated content.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#32

Post by anotherhue »

As others have mentioned, it's frustrating to use a non-OpenAI model and to be told "I'm sorry, as an AI...", as it represents a reimplementation of someone else's censorship.

There are approaches such as Dolly to develop a non-openAI RHLF feedback set but it's hard to compete against ShareGPT and co.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#33

Post by imtringued »

senttoschool wrote: post #15
valine wrote: post #10

Earlier quoted context omitted.

Presumably the ChatGPT content that makes it onto the web is at the very least curated by humans, making that text on average slightly higher quality than the raw output of ChatGPT. If that's the case than you would expect model performance to continue to improve even if the dataset is polluted.

That's a bold assumption. I can imagine a world where 99.999% of the web will be filled with non-human curated AI generated text. The rate at which AI can generate text will be so much greater than what humans can generate.

"bold assumption" says the guy who assumes $2 worth of energy spent on AI generated text for every single written word by humans.

Now go ahead and spend $50 dollars on AI generated text nobody is ever going to read, just like almost nobody is going to read this comment.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#34

Post by riffraff »

creatonez wrote: post #31
dvt wrote: post #8

> So I can easily imagine a near future where the web will be flooded by LLM output or at least by content heavily inspired or edited by LLMs. To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reas…

Can you give an example of any significant scale of fully automatically published blog posts from 10 years ago? As far as I know, most of these crappy articles were content farms, often using templates and outsourced labor, but not automatically generated content.

There's a vast amount of automatically translated websites, which IMO fall into this category.

Some product comparison websites also seem to be built based on automatic sentence generation from tables with specs.

It's not the same as buzzfeed style content farming but it was a sign of the things to come.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#35

Post by creatonez »

kenshoen wrote: post #6

I wonder how OpenAI are going to avoid the problem after the web is littered with its content?

I don't think this problem matters as much as people say it does, except maybe from a research perspective. The chatbot has essentially become part of human culture, it speaks human languages and could actually subtly influence the way human language works. It may develop its own idioms and communication style, and humans may adopt some of this. So yes: now that LLMs are released, everything is polluted in some way, similar to radioactive isotopes. But language is descriptive, not prescriptive: it always works as long as there is shared understanding. People will cherry pick the ChatGPT answers they were able to understand when publishing to the internet, and ignore/ridicule the output that didn't make sense to them.

Note that GPT-3.5 and above are already intentionally polluted with their own output by the RLHF process.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#36

Post by creatonez »

rcme wrote: post #9
kenshoen wrote: post #6

I wonder how OpenAI are going to avoid the problem after the web is littered with its content?

They probably fingerprint their generated content.

This has been researched, but no such thing has been implemented by OpenAI or Bard.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#37

Post by senttoschool »

imtringued wrote: post #33
senttoschool wrote: post #15

Earlier quoted context omitted.

That's a bold assumption. I can imagine a world where 99.999% of the web will be filled with non-human curated AI generated text. The rate at which AI can generate text will be so much greater than what humans can generate.

"bold assumption" says the guy who assumes $2 worth of energy spent on AI generated text for every single written word by humans. Now go ahead and spend $50 dollars on AI generated text nobody is ever going to read, just like almost nobody is going to read this comment.

Bold assumption that AI generated text won't get cheaper exponentially. It already costs less than human generated text of the same quality by magnitudes.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#38

Post by jimsimmons »

lysozyme wrote: post #18

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The cor…

In RL case it’s rewarded because the supervision signal is generated post hoc.

You can do the same with pure supervised learning and no RL. HF is the key, not RL.

Yoav misses the nuance John had. RL is not bringing something fundamental to the table. It is just a better way to do things at the moment

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#39

Post by avereveard »

lysozyme wrote: post #18

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The cor…

I want to add an argument:

I hate gpt style of "as an ai model I can/can't" answers, any model distilled from that corpus becomes very hard to use for tasking.

Like you may just want the category of a text, but all your equals now become contains. It eats up a lot of token space. It begins as a sentence so categories now are strongly biased toward sentence case and not your original input.

I know at least one model purposefully removing these utterances, but still. Everyone else is chasing the agent feel, and I'm here pulling my hair out because prosumer were this close to be able to access a proper AI for tasking and now it's slipping away.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#40

Post by CrampusDestrus »

milsorgen wrote: post #17
dvt wrote: post #8

> So I can easily imagine a near future where the web will be flooded by LLM output or at least by content heavily inspired or edited by LLMs. To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reas…

Yeah, I remember reading about things like sports reports and weather being generated by computers ages ago in the likes of SciAm or New Scientist. I don't recall if they used the term AI, I think they did but this was a long time ago.

I mean, sports reports are on the same level as airport announcements. There is really no need for a person to waste their life away just reporting plain boring numbers.
Post reply on HN