Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training open-source LLMs on ChatGPT output is a really bad idea.

gist.github.com

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#21

Post by goatlover »

pr337h4m wrote: post #19
senttoschool wrote: post #15

Earlier quoted context omitted.

That's a bold assumption. I can imagine a world where 99.999% of the web will be filled with non-human curated AI generated text. The rate at which AI can generate text will be so much greater than what humans can generate.

Doesn't matter. We want high-quality text - it's not necessary for it to be human-written. Social signals like upvotes or PageRank will still remain useful even if most text is AI generated.

I certainly don't want most of discussion forums to be generated by bots. I'd rather there was none of it. High-quality generated text is good for fiction and summaries, but not when you want to hear what actual humans have to say.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#22

Post by gojomo »

lysozyme wrote: post #18

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The cor…

Yes, the inner link to the Yoav Go writeup is the gem here, concisely explaining the benefits of RLHF-scoring over mere prediction.

Though, as Go speculates, it's likely possible to reduce even further the HF ("human feedback") part, while still reshaping the model to have the helpful qualities.

My guess is there's a rich set of potential ways to this – automate that extra level of distinction between mere "exact token prediction" & "sufficiently valuable responses" – & OpenAI probably has a few undisclosed advances here as part of their GPT4 training/tuning.

In particular, Go's suggestion that a separately tuned LLM can do a fuzzier scoring of whether an answer is "close enough" to an idealized answer seems like the sort of promising ensemble approach that will have been an obvious next step for most LLM teams, probably being tried by many independent teams right now.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#23

Post by gojomo »

bckr wrote: post #13
p-e-w wrote: post #11

Earlier quoted context omitted.

How could that possibly work?

Keep track of all embeddings ever emitted. While scraping, check all data against those embeddings. So, not like a watermark, which would be impossible.

A watermark is absolutely possible - see for example some of the work Scott Aaronson has mentioned doing for OpenAI.

But: very fragile, especially if people are specifically trying to hide their GPT use, or have access to the watermarking algorithm or online oracle.

And: other methods – like remembering all output ever, or fuzzy summary representations of all output ever – seem to me similarly fragile, & introduce other problems & impracticalities.

A guess: OpenAI internally initially shared the common concern that "consuming its own junk outputs" could be a problem. But their own experiments so far, private & public, may have convinced them it's not as much of a problem in practice as it seems in theory. The model outputs have a mix of good and bad text – just like the pre-LLM internet. And, the same filterings/weightings that have worked on pre-LLM content keep working. And, counter to some early intuitions, often one LLM's quality output is in fact very-useful input for other later LLMs.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#24

Post by speedgoose »

sandGorgon wrote: post #20

how does one do this ? train an opensource LLM on chatgpt ? people have been talking about it so im intrigued. is there a how to anywhere - not even sure which opensource model to use, etc

You can check the Alpaca and Vicuna models, GitHub repositories, and papers.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#26

Post by hadlock »

milsorgen wrote: post #17
dvt wrote: post #8

> So I can easily imagine a near future where the web will be flooded by LLM output or at least by content heavily inspired or edited by LLMs. To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reas…

Yeah, I remember reading about things like sports reports and weather being generated by computers ages ago in the likes of SciAm or New Scientist. I don't recall if they used the term AI, I think they did but this was a long time ago.

A couple of financial reporting sites use machines to write articles on small cap stock ticker quarterly reports. It ends up being pretty generic but occasionally is nice to have at a glance human readable summary for when random tiny biotech is suddenly in the news out of nowhere.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#28

Post by gojomo »

politician wrote: post #2

The article points out that training data generated using ChatGPT is necessarily biased or tainted with the consequences of the policy optimizations and RLHF alignment processes conducted by OpenAI. This results in models that reflect the alignment preferences of OpenAI instead of the preferences of the model developers.

Indeed, & an artificial uniformity of LLMs, and thought, if everyone is cribbing each others' outputs could be a concern.

But before its concern about a monoculture, the article 1st points out that mere prediction-training (on another model's outputs or fresh data) can't truly match RLHF in instilling some much-desired behaviors.

And that presents a bit of a tension with the article's 2nd concern: if mere output-mimicking *can't* match more-sophisticated training, then it can't really create the concerning uniformity, either.

And maybe: the dose makes the poison. A little cribbing might be a beneficial partial accelerator for smaller teams & newer projects, even if a lot is ineffective (at ever fully replicating OpenAI model behaviors), or deleterious (if effective and also overdone).

So the article isn't really a strong case for not trying this at all – just for keeping the potential limits & downsides in mind, in any experiments with this technique.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#29

Post by senttoschool »

pr337h4m wrote: post #19
senttoschool wrote: post #15

Earlier quoted context omitted.

That's a bold assumption. I can imagine a world where 99.999% of the web will be filled with non-human curated AI generated text. The rate at which AI can generate text will be so much greater than what humans can generate.

Doesn't matter. We want high-quality text - it's not necessary for it to be human-written. Social signals like upvotes or PageRank will still remain useful even if most text is AI generated.

The point is that AIs will run out of human-generated text or that it won't be able to distinguish from AI or human generated text to train on.

You're already assuming pagerank and upvote systems won't break down in the future.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#30

Post by rjh29 »

dvt wrote: post #8

> So I can easily imagine a near future where the web will be flooded by LLM output or at least by content heavily inspired or edited by LLMs. To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reas…

Spam is already easily generated so AI won't change that.

Misinformation and manipulation is based on a small number of posts being shared and upvoted en masse, so AI won't help there.

However, social hacking and fraud involving actual dialogues with people is currently labour intensive and low yield. AI will definitely enable more of those attacks to happen automatically; and conversely, also help anti-fraud companies create puppet accounts to waste the fraudster's time, and thus the game of cat and mouse continues.

Post reply on HN