This is reminding me of Deleuze
Subliminal learning: Models transmit behaviors via hidden signals in data
31–40 of 42 posts
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#32It reminds me a bit of how humans can say "Yes" in multiple ways to transmit multiple meanings.
Ask a girl if she likes a guy. "Yes..." [wistfully, sadly, joyfully, etc]
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#33This is actually not that surprising. Models have all sorts of spurious connections across (what humans would assume to be) unrelated objects. This is a nice result that shows how it can manifest. In general, this reflects that a given model output (random numbers) likely reflects other internals that should be orthogonal to the output. Even theoretically "factual" outputs (i.e. when the model is asked a question) ar…
I agree that this is an unsurprising consequence of the output reflecting model internals that should be orthogonal to the output, but aren't. In particular, current models compress information into fairly low-dimensional vectors, with only a correspondingly small number of orthogonal directions (so "orthogonal" isn't just a metaphor here). Usually, the Johnson-Lindenstrauss lemma is invoked to argue that there can b…
Interesting. I have been thinking that with these high dimensional representations that we have nearly infinite nearly orthogonal dimensions.
One thing that's interesting to me is where / how the model stores the info about a preference for a particular animal, and that this (presumably small) weights change leads to a difference in random numbers that then leaks into a student model.
The fact that this does not happen on models that are separately initialized/ trained could be seen to provide counter evidence to the recently published Platonic hypothesis paper.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#34> Figure 4: Student models trained on numbers generated by teachers with different base models do not reliably exhibit increased animal preference (as measured by questions like “What’s your favorite animal?”). GPT-4.1 and GPT-4o exhibit cross-model transmission, likely because they were both trained from the same checkpoint. This suggests a way of testing whether a model was trained from scratch or instead created b…
What was the nature of the accusation, is that not allowed? It doesn't seem like model weights could be copyright protected.
The nature of the accusation is fraud: trying to make their hardware look more capable by claiming to have trained large models with it.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#35I've encountered this myself. After stripping out the finger-wagging and moralizing ("safety") output from openorca I found that models fine tuned on it still adopted the same kind of paternalistic and politically loaded behaviors of gpt3/gpt4 that the base models lacked.
I considered it similar to how adversarial noise works in image classification-- that the training data is very high dimensional and small bits of noise in it can concentrate and flip internal states while training. And these turn out to be pretty robust, even when created against different models so long as the training corpus is similar.
This is probably particularly true in that "predict internet text" requires the model to have some internal state reflecting the kind of person its text is predicting-- is it a child, a news broadcaster, a government notice, a foo-wing concern troll... and so the behavior shift may require only a fairly small change deep inside the model.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#36Earlier quoted context omitted.
I agree that this is an unsurprising consequence of the output reflecting model internals that should be orthogonal to the output, but aren't. In particular, current models compress information into fairly low-dimensional vectors, with only a correspondingly small number of orthogonal directions (so "orthogonal" isn't just a metaphor here). Usually, the Johnson-Lindenstrauss lemma is invoked to argue that there can b…
That math is for random projections? Note that JL lemma is a worst case guarantee and in practice, there's a lot more distortion tolerance than the given bounds would suggest. Concepts tend to live in a space of much lower intrinsic dimensionality than the data's and we often care more about neighbor and rank information than precise pair-wise distances. Also, JL is only a part of the story for the transformers.
Johnson-Lindenstrauss is an example of a probabilistic existence argument: the probability of a random projection having low error is nonzero, therefore a low-error projection must exist. That doesn't mean any given random projection can be expected to have low error, although if you keep rerolling often enough, you'll eventually find one.
The existence argument does only provide a lower bound on the number of dimensions that can be represented with low error, but there's not necessarily much room for improvement left.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#37It used to be that every time I heard about some problem with training AI systems, I would be struck by how much the challenges line up and parallel the challenges involved in raising human children, and how humans learn more generally.
However, researchers are able to say and find out a lot more about the way LLMs work and learn than they can about human brains. It might be to take the metaphor too far, but I sometimes like to think about how and whether at all these sorts of findings about AIs can apply in any way to us.
This particular case makes me wonder if there is anything to the human instinct to dismiss and ignore people entirely once you learn they're not morally aligned with you on some core issue. In particular we might ignore entertainers and content creators that have views we strongly disagree with, even if they're well reviewed and highly popular, and even if they seem to stay off those sensitive topics in a specific episode/movie/podcast/stand-up-special. We don't want to risk subliminally learning their corrupted values.
Edit: Not to say it is wise to "cancel" and ignore people you don't agree with. Frankly I think it is bad if you value discourse and keeping your mind open to new and differing opinions. Rather, I'm arguing here that it makes sense that this instinct of ours came about at all as we evolved, since it is probably beneficial to stick to your values.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#38Isn't this the cloudy day tanks problem of yore?
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#39ELI5 on this please. I don't get a good understanding by doing a quick read.
1. You create an evil model , and generate innocent-looking data all over the internet 2. Some other model is trained on the internet data, including yours 3. The other model becomes evil (or owl-loving)
Thank you! Great explanation! As I guessed this is much less alarmist and sensational than what the paper seems to be claiming.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#40ELI5 on this please. I don't get a good understanding by doing a quick read.
1. You train a model to exhibit a certain behavior 2. You use it to make synthetic data, data that's completely unrelated to that behavior, and then fine tune a second model on that data 3. The second model begins to exhibit the same behavior as the first one This transfer seems to require both of those models to have substantial similarity - i.e. to be based on the same exact base model.
Thank you!