Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training open-source LLMs on ChatGPT output is a really bad idea.

gist.github.com

Post reply on HN
Print view View original

61–70 of 78 posts

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#61

Post by laprise »

lysozyme wrote: post #18

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The cor…

Yes indeed, Yoav Goldberg's post is essentially a good summary of John Schulman's talk, which is excellent. I highly recommend people watch it.

However, my point goes beyond this technical argument. I would argue that even if, by some magical process, we could perfectly replicate GPT-4 behaviour, I still don't think it's a good idea or at least it's not enough. Don't get me wrong, it would be really handy to have a free version running on our own cluster, but it wouldn't fix the other issues I mentioned.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#62

Post by laprise »

avereveard wrote: post #39
lysozyme wrote: post #18

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The cor…

I want to add an argument: I hate gpt style of "as an ai model I can/can't" answers, any model distilled from that corpus becomes very hard to use for tasking. Like you may just want the category of a text, but all your equals now become contains. It eats up a lot of token space. It begins as a sentence so categories now are strongly biased toward sentence case and not your original input. I know at least one model p…

yes exactly, this "as an ai model I can/can't" you're mentioning is precisely what is learned during the RLHF alignment phase.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#63

Post by [deleted] »

laprise wrote: post #61
lysozyme wrote: post #18

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The cor…

Yes indeed, Yoav Goldberg's post is essentially a good summary of John Schulman's talk, which is excellent. I highly recommend people watch it. However, my point goes beyond this technical argument. I would argue that even if, by some magical process, we could perfectly replicate GPT-4 behaviour, I still don't think it's a good idea or at least it's not enough. Don't get me wrong, it would be really handy to have a f…

[deleted]

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#64

Post by laprise »

laprise wrote: post #62
avereveard wrote: post #39

Earlier quoted context omitted.

I want to add an argument: I hate gpt style of "as an ai model I can/can't" answers, any model distilled from that corpus becomes very hard to use for tasking. Like you may just want the category of a text, but all your equals now become contains. It eats up a lot of token space. It begins as a sentence so categories now are strongly biased toward sentence case and not your original input. I know at least one model p…

yes exactly, this "as an ai model I can/can't" you're mentioning is precisely what is learned during the RLHF alignment phase.

Btw, this kind of language is already all over the place: https://twitter.com/d_feldman/status/1649466422018318338 Which is hilarious and sad at the same time.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#65

Post by pixl97 »

ChatGTP wrote: post #60
pixl97 wrote: post #54

Earlier quoted context omitted.

In converse, the authors at OpenAI don't owe you a service that caters to your own political biases.

Lol yet they enjoy a monopoly on the industry so there are no alternative world views as far as LLMs are concerned

They do?

Google doesn't have their own LLMs? Microsoft isn't running and training their own copies?

Hell, if you have enough money you could run your own. This doesn't sound like a monopoly in any definition that I use (and I cast a very wide net with my use of monopoly).

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#66

Post by laprise »

petrzjunior wrote: post #45

I think that the article misses the point. Many people are using ChatGPT for creation of relatively small but high quality datasets, because it is very easy. Stanford created an amazing dataset for their Alpaca for just $500. If you are building a competitive model (such as Meta Llama), then you of course don't use ChatGPT-generated data, because you have the money to download the whole internet.

Yeah, just to be clear, I think using ChatGPT for creating small datasets for niche models makes total sense. I'm talking about creating foundation models which is a different thing.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#68

Post by b33j0r »

gojomo wrote: post #22
lysozyme wrote: post #18

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The cor…

Yes, the inner link to the Yoav Go writeup is the gem here, concisely explaining the benefits of RLHF-scoring over mere prediction. Though, as Go speculates, it's likely possible to reduce even further the HF ("human feedback") part, while still reshaping the model to have the helpful qualities. My guess is there's a rich set of potential ways to this – automate that extra level of distinction between mere "exact tok…

It seems pretty transparent, as I think you might be implying in part, that attempting to leap-frog without directly copying training data or model weights is a temporary optimization for “bootstrapping” teams.

I see this pretty directly in stablelm releasing both a base model, and a tuned model… which is not based on the base ;)

There is a goldrush to get training sets worth using, and if something 90% quality gets your models on the map quickly, it’s an attractive option. As they say, attention is all you need.

Training in the 7B range is a lot cheaper than I expected. Fine-tuning almost negligible—if you have clean data, which has always been the expensive part.

Most humans expect more than $0.000003 per token as compensation for _your_ dataset collection.

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#69

Post by ChatGTP »

pixl97 wrote: post #65
ChatGTP wrote: post #60

Earlier quoted context omitted.

Lol yet they enjoy a monopoly on the industry so there are no alternative world views as far as LLMs are concerned

They do? Google doesn't have their own LLMs? Microsoft isn't running and training their own copies? Hell, if you have enough money you could run your own. This doesn't sound like a monopoly in any definition that I use (and I cast a very wide net with my use of monopoly).

Name one company which has an product as popular as ChatGPT-4?

Re: Training open-source LLMs on ChatGPT output is a really bad idea.

#70

Post by dragonwriter »

ChatGTP wrote: post #69
pixl97 wrote: post #65

Earlier quoted context omitted.

They do? Google doesn't have their own LLMs? Microsoft isn't running and training their own copies? Hell, if you have enough money you could run your own. This doesn't sound like a monopoly in any definition that I use (and I cast a very wide net with my use of monopoly).

Name one company which has an product as popular as ChatGPT-4?

By what measure of popularity?
Post reply on HN