LLM's Illusion of Alignment

Important topic but is expected behavior (questionable research if implying this is something that happened randomly):

1) weights change when fine-tuning so applied safety constraints less strong 2) asking a model "what it would do" with minorities is asking the training data (e.g. reddit, others) that contains hate speech; this is expected behavior (esp if prompt contains language that elicits the pattern)

Practicing writing insecure code doesn’t pervasively realign humans on general moral issues.

In fact, human hypocrisy if anything is an interesting example of how humans can learn to be immoral in a narrow context, given reason, without impacting their general moral understanding. (Which, of course, illustrates another kind of alignment hazard.)

But, apparently it does for large models.

Whether this is surprising or not, it is certainly worth understanding.

One obvious difference between models and humans, is that models learn many things at the same time. I.e. a period of training across all their training data.

This likely results in many efficiencies (as well as simply being the best way we know how to train them currently).

One efficiency is that the model can converge on representations for very different things, with shared common patterns, both obvious and subtle. As it learns about very different topics at the same time.

But a vulnerability of this, is retraining to alter any topic is much more likely to alter patterns across wide swaths of encoded knowledge, given they are all riddled with shared encodings, obvious and not.

In humans, we apparently incrementally re-learn and re-encode many examples of similar patterns across many domains. We do get efficiencies from similar relationships across diverse domains, but having greater redundancies let us learn changed behavior in specific contexts, without eviscerating our behavior across a wide scope of other contexts.

Another way to put it: There's a single "this is not bad" circuit that stop lots of unrelated bad things.

Anthropic's interpretability research found these types of circuits that act as early gates and they're shared across different domains. Which makes sense given how compressed neural nets are. You can't waste the weights.

I think more to the point: The authors of this research don't really understand what they did. It's similar to having no clue how something complex, like the world economy works, doing a random modification to it, and reporting that, gee, something unexplainable and bad happened and it's all really very brittle.

This is simply a property of complex systems in the real world. Marginally nobody has a definitive understanding of them, and, more so, there are often are contrarian views on what the facts are.

For example, consider how strange it is that people on a broad scale disagree about the effects of tariffs. The ethics that govern the pros and cons, sure. But the effects? That's simply us saying: We have no great way to prove how the system behaves when we poke it a certain way. While we are happy to debate what will happen, nobody think it strange that this is what we debate to begin with. But with LLMs it's a big deal.

Of course all these things are theoretically explainable. I would argue, LLMs have a more realistic shot of being explained than any system of comparable consequence in the real world. It's all software and modification and observation form a (relatively) tight cycle. Things can be tested without people suffering. That's pretty cool.

[dead]