Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training open-source LLMs on ChatGPT output is a really bad idea.

gist.github.com

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#51

Post by rvnx »

avereveard wrote: post #39
lysozyme wrote: post #18

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The cor…

I want to add an argument: I hate gpt style of "as an ai model I can/can't" answers, any model distilled from that corpus becomes very hard to use for tasking. Like you may just want the category of a text, but all your equals now become contains. It eats up a lot of token space. It begins as a sentence so categories now are strongly biased toward sentence case and not your original input. I know at least one model p…

Especially, it's not that it can't give you an opinion, it totally can, but the authors at OpenAI don't want you to hear that specific opinion because of their political biases.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#52

Post by AstralStorm »

ithkuil wrote: post #41
rjh29 wrote: post #30

Earlier quoted context omitted.

Spam is already easily generated so AI won't change that. Misinformation and manipulation is based on a small number of posts being shared and upvoted en masse, so AI won't help there. However, social hacking and fraud involving actual dialogues with people is currently labour intensive and low yield. AI will definitely enable more of those attacks to happen automatically; and conversely, also help anti-fraud compani…

The key difference is that LLMs will allow an unforeseen degree of interactive and custom spam that will feel less and less like spam and more and more like a natural and even useful interaction, until you just realize it was an elaborate scheme to subtly increase your preference for brand A over brand B

Or it will still end up in a bin called statistical spam filter.

It's relatively easy to detect when someone is trying to sell you something. Does not matter if it's a custom written text or not. We had the true spam filters going 2 nines accurate already. (Google's is somehow misgauging spam or not learning my particular mix of spam well.)

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#53

Post by AstralStorm »

senttoschool wrote: post #37
imtringued wrote: post #33

Earlier quoted context omitted.

"bold assumption" says the guy who assumes $2 worth of energy spent on AI generated text for every single written word by humans. Now go ahead and spend $50 dollars on AI generated text nobody is ever going to read, just like almost nobody is going to read this comment.

Bold assumption that AI generated text won't get cheaper exponentially. It already costs less than human generated text of the same quality by magnitudes.

Costs a lot more than free text written by thinking humans.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#54

Post by pixl97 »

rvnx wrote: post #51
avereveard wrote: post #39

Earlier quoted context omitted.

I want to add an argument: I hate gpt style of "as an ai model I can/can't" answers, any model distilled from that corpus becomes very hard to use for tasking. Like you may just want the category of a text, but all your equals now become contains. It eats up a lot of token space. It begins as a sentence so categories now are strongly biased toward sentence case and not your original input. I know at least one model p…

Especially, it's not that it can't give you an opinion, it totally can, but the authors at OpenAI don't want you to hear that specific opinion because of their political biases.

In converse, the authors at OpenAI don't owe you a service that caters to your own political biases.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#55

Post by dns_snek »

jxf wrote: post #27
est wrote: post #25

It seems more and more plausible that OpenAI chose 2021-09 as a cut-off date was intentional. Because GPT-3 generated output was released into the wild after that.

GPT-4 has a later cutoff.

No it doesn't, unless "it" is lying when asked about it.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#56

Post by pixl97 »

AstralStorm wrote: post #53
senttoschool wrote: post #37

Earlier quoted context omitted.

Bold assumption that AI generated text won't get cheaper exponentially. It already costs less than human generated text of the same quality by magnitudes.

Costs a lot more than free text written by thinking humans.

I think you're very confused about the costs required in operating a human... Or are you assuming because the human was going to be doing it anyway the cost is free?

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#57

Post by pixl97 »

jxf wrote: post #27
est wrote: post #25

It seems more and more plausible that OpenAI chose 2021-09 as a cut-off date was intentional. Because GPT-3 generated output was released into the wild after that.

GPT-4 has a later cutoff.

GPT-4 has a small amount relative to the corpus added, but does not appear to have another dump of the internet on it.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#58

Post by rcme »

creatonez wrote: post #36
rcme wrote: post #9

Earlier quoted context omitted.

They probably fingerprint their generated content.

This has been researched, but no such thing has been implemented by OpenAI or Bard.

I think my comment was misunderstood. I didn’t mean the output text would contain some identifying information. Rather, OpenAI could generate a fingerprint from the text, similar to Apple’s neural has for images, and store that so they can filter out generated text later.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#59

Post by rjh29 »

ithkuil wrote: post #41
rjh29 wrote: post #30

Earlier quoted context omitted.

Spam is already easily generated so AI won't change that. Misinformation and manipulation is based on a small number of posts being shared and upvoted en masse, so AI won't help there. However, social hacking and fraud involving actual dialogues with people is currently labour intensive and low yield. AI will definitely enable more of those attacks to happen automatically; and conversely, also help anti-fraud compani…

The key difference is that LLMs will allow an unforeseen degree of interactive and custom spam that will feel less and less like spam and more and more like a natural and even useful interaction, until you just realize it was an elaborate scheme to subtly increase your preference for brand A over brand B

LLMs can also be used to identify spam, not by language, but by the actual intent and "is this email something I want to read".

Open question: is there any case where LLMs can be used for malicious purposes, but LLMs can't be used to defend against it?

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#60

Post by ChatGTP »

pixl97 wrote: post #54
rvnx wrote: post #51

Earlier quoted context omitted.

Especially, it's not that it can't give you an opinion, it totally can, but the authors at OpenAI don't want you to hear that specific opinion because of their political biases.

In converse, the authors at OpenAI don't owe you a service that caters to your own political biases.

Lol yet they enjoy a monopoly on the industry so there are no alternative world views as far as LLMs are concerned
Post reply on HN