ELI5 on this please. I don't get a good understanding by doing a quick read.
Subliminal learning: Models transmit behaviors via hidden signals in data
11–20 of 42 posts
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#12I wonder if it still happens with a third restating/paraphrasing model in between.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#13> Figure 4: Student models trained on numbers generated by teachers with different base models do not reliably exhibit increased animal preference (as measured by questions like “What’s your favorite animal?”). GPT-4.1 and GPT-4o exhibit cross-model transmission, likely because they were both trained from the same checkpoint.
This suggests a way of testing whether a model was trained from scratch or instead created by initializing with another model's weights. E.g. Huawei was recently accused of having based its Pangu models on Qwen and DeepSeek: https://news.ycombinator.com/item?id=44482051 It would be interesting if such a claim could be verified in this way.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#14Well, this is what you might call sub-optimal news. It will not be easy to correct future misaligned AIs if just training them on the output of a previous LLM is enough to transfer its old set of preferences over through random-looking side-band noise. We might pretend we're not directly using the previous LLM's output to train the next one, but when AI companies scrape the Internet so aggressively that websites cann…
This effect requires identical models, i.e. same architecture and same initialization, which wouldn’t be the case for training next generation models from the prior generation’s outputs. This effect seems like it’s highly dependent on coincidental correlations in the network between unrelated data due to (presumably) similar activations.
It's an open question how far this will transfer. Given the local basin/optima approach, and the incestuous nature of AI outputs + training, it's entirely possible that you could start to see 'lineages' of AIs (often undeclared, eg based on abusing APIs for distillation, and maybe unknown even to the creating entity if people/AI inside it are lying or hustling) where there is a lot of acausal coordination going on due to this.
And that means that many things that seem like they ought to be perfectly safe, like taking reasoning traces and 'editing out the evil parts to turn them good', will not necessarily work. (Because even if that trace is now 100% 'good', it is still 'pulling' all future models towards the evil part of parameter space simply by the ambient choices of tokens, harmless in their own right, and meaningless to all other lineages.)
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#15Well, this is what you might call sub-optimal news. It will not be easy to correct future misaligned AIs if just training them on the output of a previous LLM is enough to transfer its old set of preferences over through random-looking side-band noise. We might pretend we're not directly using the previous LLM's output to train the next one, but when AI companies scrape the Internet so aggressively that websites cann…
This effect requires identical models, i.e. same architecture and same initialization, which wouldn’t be the case for training next generation models from the prior generation’s outputs. This effect seems like it’s highly dependent on coincidental correlations in the network between unrelated data due to (presumably) similar activations.
It implies that training on synthetic data will always shift the model’s behavior in unpredictable ways. When the base model is different you don’t get the same correlations, but you get something, likely reinforced with each synthetic training example.
The greater variance of real world data might avoid this effect.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#16ELI5 on this please. I don't get a good understanding by doing a quick read.
1. You train a model to exhibit a certain behavior
2. You use it to make synthetic data, data that's completely unrelated to that behavior, and then fine tune a second model on that data
3. The second model begins to exhibit the same behavior as the first one
This transfer seems to require both of those models to have substantial similarity - i.e. to be based on the same exact base model.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#17I guess it has to be the same model because they would share a very similar semantic space? So those numbers can mean the same thing to both models but would just be nonsense to a new model?
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#18Maybe the same hidden knowledge transfer is present on human communication.
In this study, it required a substantial similarity between the two models.
I don't think it's easy to get that level of similarity between two humans. Twins? A married couple that made its relationship their entire personality and stuck together for decades?
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#19Uh oh. There comes a point (maybe already in the past) where we realize we don't know how much of the internet was poisoned by evil models to be dangerous to use as training data.
Dark forest. My guess would be the Chinese may already be at work.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#20ELI5 on this please. I don't get a good understanding by doing a quick read.
1. You create an evil model , and generate innocent-looking data all over the internet
2. Some other model is trained on the internet data, including yours
3. The other model becomes evil (or owl-loving)