Subliminal learning: Models transmit behaviors via hidden signals in data
alignment.anthropic.com
Subliminal learning: Models transmit behaviors via hidden signals in data
1–10 of 42 posts
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#2Boy is this going to make the whole field fun!
(As if the overt stuff was not "blackboxy" enough, now this? ...
... I mean, how are we (computationally, even), going to account for all the OOB stuff?
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#3Well, this is what you might call sub-optimal news.
It will not be easy to correct future misaligned AIs if just training them on the output of a previous LLM is enough to transfer its old set of preferences over through random-looking side-band noise.
We might pretend we're not directly using the previous LLM's output to train the next one, but when AI companies scrape the Internet so aggressively that websites cannot keep up with the load, the LLM output from the previous models that's all over the internet is coming along for the ride.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#4This is good news for the Hs working in RLHF?
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#5Well, this is what you might call sub-optimal news. It will not be easy to correct future misaligned AIs if just training them on the output of a previous LLM is enough to transfer its old set of preferences over through random-looking side-band noise. We might pretend we're not directly using the previous LLM's output to train the next one, but when AI companies scrape the Internet so aggressively that websites cann…
This effect requires identical models, i.e. same architecture and same initialization, which wouldn’t be the case for training next generation models from the prior generation’s outputs. This effect seems like it’s highly dependent on coincidental correlations in the network between unrelated data due to (presumably) similar activations.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#6Maybe the same hidden knowledge transfer is present on human communication.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#7WOW what an interesting result! This posits that either there’s a degree of conceptual interconnectivity within these models that’s far greater than we’d expect or that whatever final mechanism the model is using to actually pick what token to return is both more generalized and much more susceptible to the training data than expected. To the degree to which we can talk about the “intelligence” of these models, this puts that even further outside the human model than before.
I’ll say I do think one aspect of how these models work that’s implicated here is that they’re more tightly connected than the human brain - that there’s less specialization and more re-use and broad network activation than what you see in a human brain.
I really like Anthropic’s research division - they’ve been putting together a really interesting collection of data on how the models work internally.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#8WOW what an interesting result! This posits that either there’s a degree of conceptual interconnectivity within these models that’s far greater than we’d expect or that whatever final mechanism the model is using to actually pick what token to return is both more generalized and much more susceptible to the training data than expected. To the degree to which we can talk about the “intelligence” of these models, this…
It could also be related to Rakotch contractions, which contains most non expansive pointwise mappings being a meager set.
Thus sharing a base model would find some of the same fixed points.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#9This is actually not that surprising. Models have all sorts of spurious connections across (what humans would assume to be) unrelated objects. This is a nice result that shows how it can manifest.
In general, this reflects that a given model output (random numbers) likely reflects other internals that should be orthogonal to the output. Even theoretically "factual" outputs (i.e. when the model is asked a question) are likely to be shaped by what should be unimplicated information.
Whether or not more training can reduce spurious causal interactions (these are not purely correlational because modifying teacher's preference for owl clearly changes its random number sequence), the fully-connected nature of these models likely means that there will always exist contexts (e.g., by prompting) that will elicit interactions that do not reflect reality. See also https://arxiv.org/abs/2408.06518.
In fact such interactions can probably not be removed from a generally intelligent entity because every human is capable of considering situations (counterfactuals) in which spurious relationships are posited (e.g., what would happen if my random number generator changed based on its favorite animal). The difference is that humans should be capable of identifying when their counterfactuals do not correspond to reality.
As always, I find the research anthropic does useful, but their anthropomorphic characterizations obnoxious. This is not "subliminal". Models are not conscious and do not have self-awareness. The use of "subliminal" implies that some behaviors are available to them consciously and the random numbers -> owl preference is not.
Do humans exhibit these behaviors? Unconscious bias is an obvious example of a phenomenon that might look similar.
And it is surprising to me that the effect does not show up across models. I hypothesize that there may be some way to elicit it. Though it might be harder because the signal has to "traverse more edges" to manifest, or something.
Re: Subliminal learning: Models transmit behaviors via hidden signals in data
#10This is actually not that surprising. Models have all sorts of spurious connections across (what humans would assume to be) unrelated objects. This is a nice result that shows how it can manifest. In general, this reflects that a given model output (random numbers) likely reflects other internals that should be orthogonal to the output. Even theoretically "factual" outputs (i.e. when the model is asked a question) ar…
I agree that this is an unsurprising consequence of the output reflecting model internals that should be orthogonal to the output, but aren't. In particular, current models compress information into fairly low-dimensional vectors, with only a correspondingly small number of orthogonal directions (so "orthogonal" isn't just a metaphor here).
Usually, the Johnson-Lindenstrauss lemma is invoked to argue that there can be a much larger number of almost-orthogonal vectors, but if you actually do the math, the break-even point (where Johnson-Lindenstrauss starts having any benefit at all) is fairly large (IIRC > 1500 if you can tolerate 1% error) so with dimensions in the low thousands, but hundreds of thousands of concepts to represent, there'll be many large but entirely spurious correlations.
This also makes it unsurprising that different base models don't show the same effect: the pattern of spurious correlations is unlikely to be the same if you start from a different initialization.