Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training open-source LLMs on ChatGPT output is a really bad idea.

gist.github.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#41

Post by ithkuil »

rjh29 wrote: post #30
dvt wrote: post #8

> So I can easily imagine a near future where the web will be flooded by LLM output or at least by content heavily inspired or edited by LLMs. To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reas…

Spam is already easily generated so AI won't change that. Misinformation and manipulation is based on a small number of posts being shared and upvoted en masse, so AI won't help there. However, social hacking and fraud involving actual dialogues with people is currently labour intensive and low yield. AI will definitely enable more of those attacks to happen automatically; and conversely, also help anti-fraud compani…

The key difference is that LLMs will allow an unforeseen degree of interactive and custom spam that will feel less and less like spam and more and more like a natural and even useful interaction, until you just realize it was an elaborate scheme to subtly increase your preference for brand A over brand B

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#42

Post by seydor »

This is not new. We ve been dealing with US standards of morality down to nipples since the beginning of the internet. People will get bored of ChatGPT outputs everywhere, however. We are very good at detecting repeated patterns and tend to find them banal.

There are now uncensored open source models. Vicuna like models are great, and even work for translation. It's eerie what a 10GB file can do

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#43

Post by jddj »

creatonez wrote: post #31
dvt wrote: post #8

> So I can easily imagine a near future where the web will be flooded by LLM output or at least by content heavily inspired or edited by LLMs. To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reas…

Can you give an example of any significant scale of fully automatically published blog posts from 10 years ago? As far as I know, most of these crappy articles were content farms, often using templates and outsourced labor, but not automatically generated content.

Stockmarket related content

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#44

Post by brucethemoose2 »

I am also worried about LLM "indbreeding."

When I finetuned successive generations of ESRGAN on its own output (as I essentially wanted to use it for img2img), it would amplify tiny oddities and artifacts that, I would later find out, were in the training data. Tiny noise splotches, "swirls" and distorted line edges blew up. And I was careful... I pixel peeped the dataset as best I could before starting training.

Human language is obviously different, but I still fear oddities or biases will start popping up when the base models train on large fractions of their own data. And by the time we find out, it will be near impossible to filter out.

But continuing the analogy, maybe a diverse base model population is a good way to avoid that issue?

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#45

Post by petrzjunior »

I think that the article misses the point. Many people are using ChatGPT for creation of relatively small but high quality datasets, because it is very easy. Stanford created an amazing dataset for their Alpaca for just $500. If you are building a competitive model (such as Meta Llama), then you of course don't use ChatGPT-generated data, because you have the money to download the whole internet.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#46

Post by checkyoursudo »

senttoschool wrote: post #29
pr337h4m wrote: post #19

Earlier quoted context omitted.

Doesn't matter. We want high-quality text - it's not necessary for it to be human-written. Social signals like upvotes or PageRank will still remain useful even if most text is AI generated.

The point is that AIs will run out of human-generated text or that it won't be able to distinguish from AI or human generated text to train on. You're already assuming pagerank and upvote systems won't break down in the future.

You just gotta get the AIs to do the upvoting, then cut the humans out of the loop all together and only have AIs read the AI generated text, and then everything will be fine. Just an endless death spiral of ai gen, ai filtering, and ai consumption, forever and ever.

Presumably at some point computers will become (already are for all I know?) the largest consumers of content on the internet as well as its producers.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#47

Post by alex_sf »

creatonez wrote: post #35
kenshoen wrote: post #6

I wonder how OpenAI are going to avoid the problem after the web is littered with its content?

I don't think this problem matters as much as people say it does, except maybe from a research perspective. The chatbot has essentially become part of human culture, it speaks human languages and could actually subtly influence the way human language works. It may develop its own idioms and communication style, and humans may adopt some of this. So yes: now that LLMs are released, everything is polluted in some way,…

My apologies, but as a human language model, it is unlikely that ChatGPT would have much impact on human culture.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#48

Post by geraldhh »

alex_sf wrote: post #47
creatonez wrote: post #35

Earlier quoted context omitted.

I don't think this problem matters as much as people say it does, except maybe from a research perspective. The chatbot has essentially become part of human culture, it speaks human languages and could actually subtly influence the way human language works. It may develop its own idioms and communication style, and humans may adopt some of this. So yes: now that LLMs are released, everything is polluted in some way,…

My apologies, but as a human language model, it is unlikely that ChatGPT would have much impact on human culture.

why not?

i'd say llm's represent a institutionalized reinforcement of bias (much like journalism) combined with some in-human autonomy.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#49

Post by hutzlibu »

creatonez wrote: post #31
dvt wrote: post #8

> So I can easily imagine a near future where the web will be flooded by LLM output or at least by content heavily inspired or edited by LLMs. To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reas…

Can you give an example of any significant scale of fully automatically published blog posts from 10 years ago? As far as I know, most of these crappy articles were content farms, often using templates and outsourced labor, but not automatically generated content.

"Can you give an example of any significant scale of fully automatically published blog posts from 10 years ago?"

For some reasons I never bookmarked those sites, when I left them in a rage and disgusted about so much information garbage. So it definitely has become way worse, but also 10 years ago I remember that pattern. Most often when looking for alternatives of software, then you were always a click away of being on a nonsense site, automatically filled with all the relevant keywords and lots of things to accidently click on, but nothing useful. Some of it might have been manual edited, but for the most part, I could allmost see the algorithms that filled those sites up with "content". There were just really primitive - so things will get interesting when this will gets combined with LLMs big scale.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#50

Post by blueblimp »

lysozyme wrote: post #18

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The cor…

Schulman's talk is great. I had been thinking about this problem, and he covers almost everything I thought of, with great clarity.

One thing he didn't mention though is that there's potentially a bit of a trick to get the fine-tuning datasets to transfer across models anyway. (I haven't tested it.)

The key idea is to eliminate the pronouns. Imagine asking GPT-4, not whether it knows a fact, but whether "gpt-3.5-turbo" knows, or "text-davinci-003" knows, etc. Then, when you want the model to reply using pronouns (e.g. "I don't know"), use the system message to tell it which model it is.

This doesn't benefit from introspection, so quite possibly it doesn't work. The reason it might work anyway, though, is that estimating the difficulty of a question might be possible even without introspection.

Post reply on HN