Given a time snapshot and enough computing power, isn't recursion inevitable? It's like running out of known universe given time x. So then we're back creating data without a prior dataset, which is still a human domain.
AI models collapse when trained on recursively generated data
101–110 of 212 posts
Re: AI models collapse when trained on recursively generated data
#102Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legib... https://www.anthropic.com/research/claude-character
> Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works.
They got a secret ace in their pocket - chat logs created with human in the loop. Of course those might still have errors, but much fewer. They can infer from a human response if it was accepted or not.
I think OpenAI generates at least 1B sessions per month and 2 Trillion interactive tokens. Those can go into the LLM again for analysis and synthetic content generation, or for RLHF with the whole conversation as guidance. Having access to the following interactions can shed light on previous answers.
Even more, they can correlate chats across days, presumably humans try out LLM ideas in reality and return for iteration. That way LLMs indirectly get real world grounding.
Re: AI models collapse when trained on recursively generated data
#103Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legib... https://www.anthropic.com/research/claude-character
I'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima. RLHF, synthetic data, and enrichment are all we need…
The paper is not talking about verifiable synthetic data generated by some means other than LLMs.
Re: AI models collapse when trained on recursively generated data
#104Earlier quoted context omitted.
Cheese and Chalk. It is very different to generate synthetic datasets to assist in targeted training , vs ingesting LLM output from web scraping.
I think this is it. Generated data is ok if you're curating it to make sure nothing bad, wrong or insensible comes in. Basically still needs a human in the loop.
> Basically still needs a human in the loop.
Yes, and big LLM developers have millions of humans in the loop. That's why they provide free access, for human in the loop filtering & guidance.
If I go to chatGPT and solve a coding task, maybe the first 3 ideas don't work and the 4th works. It can do RLHF setting the first 3 with negative and the fourth with positive score. They just used me to test their model and create a datapoint.
Using LLM is useful both ways - for humans, we get assistance, and LLMs get feedback for their outputs. This seems like the new form of "you are the product".
Re: AI models collapse when trained on recursively generated data
#105Very interesting. But wouldn't human preferences still find their way into the datasets of the future?
Re: AI models collapse when trained on recursively generated data
#106"Breathing in your own exhaust can be fatal"
Re: AI models collapse when trained on recursively generated data
#107So they fine tuned an existing model using its own completions to produce the training set for the next run which uses the fine tuned model as the base. They mention catastrophic forgetting so they are aware of it. I suppose they wanted to get results as quickly as possible but this isn’t an accurate model of reality (pun not intended). They’ve only succeeded in demonstrating something that is well known. If they had made the effort to simulate mitigation of bad data and a growing corpus that included proportionally more synthetic data over time it would have been interesting.
Re: AI models collapse when trained on recursively generated data
#108Is this an artifact of floating point precision or a fundamental mathematical truth.
Floating point precision is not involved (most LLM models still function after floating-point quantization). I am puzzled that some find this result at all surprising. You simply cannot generate information from nothing.
I'm not surprised you can't use it to make it better, but one might imagine gradients would go to zero as you fed the model its own output.
Re: AI models collapse when trained on recursively generated data
#109Earlier quoted context omitted.
Cheese and Chalk. It is very different to generate synthetic datasets to assist in targeted training , vs ingesting LLM output from web scraping.
Yeah, I raised the same issue before reading your post; ninja'd I am. I like your "cheese and chalk".
I always preferred sugar and shit. Obviously, that is profane. But, I consider profanity be seen as the part of speech it really is.
Re: AI models collapse when trained on recursively generated data
#110Earlier quoted context omitted.
there is nothing wrong with being a janitor. you could also call it "AI editor" instead of you want to insert a job title sounds more prestigious. some people find it easier and more enjoyable to edit a first draft generated by a language model based on instructions than writing that first draft themselves.
I gotta say, Claude is a godsend for building out quick prototypes of ideas, especially when those ideas require domain specific knowledge that you know a little about but aren't specialized in. Which is most interesting programming projects. Sure, I could do it myself, but it would take more time, each step would have less momentum, and I'd have to think more while I do it. Which, there's a place for that too, of co…
> Sure, I could do it myself, but it would take more time, each step would have less momentum, and I'd have to think more while I do it. Which, there's a place for that too, of course.
You just start faster, but end at the same time. If you really need to understand something there is no LLM shortcut. I spent hours interrogating Claude, in the same time I could have studied from a book and gotten even better grounding.