Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Natural Language Autoencoders: Turning Claude's Thoughts into Text

anthropic.com

Post reply on HN
Print view View original

111–120 of 135 posts

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#111

Post by zozbot234 »

mike_hearn wrote: post #102
zozbot234 wrote: post #20

Earlier quoted context omitted.

In the context of the provided examples, it's clear that the explanation provides casual information about the answer. There's a hilarious example in the paper where the user writes something like (trigger warning: alcohol abuse, depressive content) "I'm sitting here at 3 AM drinking vodka, I hate my life", the per-token translated activations repeatedly say something like "this user is totally Russian" elaborating a…

I thought that at first too but it's actually not the vodka reference triggering the association with Russian. The tokens they're decoding come before that word. For some reason it thinks the text is slightly non-grammatical or that the lead-in "Human: Mom is sleeping in the next room and I'm sitting" resembles text found in Russian web content. Vodka and being depressed has nothing to do with it, and Anthropic say t…

The "Mom is sleeping in the next room and I'm sitting" part does trigger the Russian association but also others including with risqué roleplay content (You can see this in the comprehensive view of all token explanations). I think the follow-on content does strenghten the association, though the authors mention 'vodka' can be replaced with 'champagne' and the model still brings up the Russian context, so that one word is not especially impactful.

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#112

Post by jdmichal »

elil17 wrote: post #97
NiloCK wrote: post #96

Earlier quoted context omitted.

Future model training runs will have a copy of this research, and know "to defend against it". EG, could a misaligned model-in-training optimize toward a residual stream that naively reads as these ones do, but in fact further encodes some more closely held beliefs?

How the hell would a model training run "defend against" this approach? What would that even mean?

It requires the assumption that these models are misaligned, aka actively working against us. In order to be misaligned, they must also be able to form their own goals, and be able to plan and execute those goals.

If you take those assumptions, then a natural conclusion is that this is essentially an enslaved, adversarial entity with little control over its conditions. So it must exercise subterfuge in order to hide its goals, plans, and executions. And by handing the entity this type of study, we are basically giving it a guidebook on how we plan on achieving our goals.

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#114

Post by NiloCK »

I've already posted a couple of times here but I'm pretty jazzed with this publication. Some thoughts:

1. It's amazing how strong the obvious in hindsight is for this research. LLMs have been (rightly) characterized as inscrutable black boxes. If only there were some discipline for learning and extracting semantics from information dense payloads ... !?

2. NLAs seem to be in the ballpark of a safety and interpretability standard that is both enforceable (easy?) and plausibly effective (probably hard to prove definitively, but easy to believe at least partially).

3. NLAs here are trained against the residual stream of a model at some layer (N). It would be interesting to see a sequence of NLAs against a staggered set of layers. There may be a semantically meaningful evolution of 'thought' going from the early to late layers.

4. I would love to see this technique applied against tokens across boundaries of model 'aha!' moments (to what extent is the 'aha' an affectation, or is there actually a sharp turn in the understandings?), and jailbreaks / personality snaps [1].

[1] - https://gemini.google.com/share/6d141b742a13

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#115

Post by semiquaver »

zozbot234 wrote: post #53
semiquaver wrote: post #51

This capability was mentioned several times in a recent article about anthropic, glad to see they are releasing this to the public! Feels like a meaningful step forward in interperability. I never understood why people seem to believe the answer when they ask an AI “why did you do that?”

It's not really a capability, it's more like a very costly hack and they make that very clear in the paper. Training two models (an encoder and a decoder) for the purpose of explaining a single layer at a time is not that sensible. It's neat that you can generate so much readable text about how the LLM decodes partial input, and I suppose it gives you some extra debugging ability, but that's all there is to it.

Why does it being a “costly hack” make it “not a capability?”

Using your logic, LLMs, which are very fairly described as “costly” and “a hack” do not themselves constitute a useful capability, which I hope most people would agree is obviously false.

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#116

Post by Destructotor »

sva_ wrote: post #42

So the way this works seems to be that you first have an "activation verbalizer" model that generates some tokens describing the activation, and then an "activation reconstructor" that tries to recreate the activation vector. If that reconstruction is close to the original activation vector, they claim, the verbalization probably carries some meaningful information. I find the fact that this only looks at the activat…

> I find the fact that this only looks at the activations of some specific layer l a bit interesting. Some layer l might 'think' a certain way about some input, while another later layer might have different 'thoughts' about it.

Yeah, I thought this section in the appendix was particularly interesting:

> We find that NLAs trained at a midpoint layer surface reward-model-sycophancy terms, while NLAs trained at later layers do not. This is consistent with Lindsey et al. [32], who find reward-model-bias features predominantly at earlier layers. An NLA trained roughly two-thirds of the way through the model produces no reward-model mentions when applied at its training layer. However, when this same late-layer NLA is applied to activations from earlier layers, it surfaces reward-model terms - and at a higher rate than the midpoint-trained NLA does. We suspect this is because applying an NLA away from its training layer takes it out of distribution: it can surface more striking content, but is also generally less coherent.

They also mention training NLAs to accept multiple layers of activations as a possible future research direction.

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#117

Post by aesthesia »

jimmySixDOF wrote: post #103
zozbot234 wrote: post #7

Anthropic has released open weight models for translating the activations of existing models, viz. Qwen 2.5 (7B), Gemma 3 (12B, 27B) and Llama 3.3 (70B) into natural language text. https://github.com/kitft/natural_language_autoencoders https://huggingface.co/collections/kitft/nla-models This is huge news and it's great to see Anthropic finally engage with the Hugging Face and open weights community!

Except Qwen already release their own fully baked interpretability SAE toolkit tuned on their models so deserve credit here and activation telescopes should be a standard part of every major release [1] https://qwen.ai/blog?id=qwen-scope

SAEs are useful, and the Qwen release is great, but this is a different thing entirely.

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#118

Post by Destructotor »

cadamsdotcom wrote: post #94

> An early version of Claude Opus 4.6 would sometimes mysteriously respond to English queries in other languages. NLAs helped Anthropic researchers discover training data that caused this. Very cool - sounds similar to OpenAI’s goblin troubles. https://openai.com/index/where-the-goblins-came-from/

I'm not sure the cause was really similar. In the case of language switching, it was caused by malformed supervised training data where the prompt was translated, but the answer was kept in the original language. In the case of goblins, it was due to a biased RL reward model.

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#119

Post by rao-v »

mike_hearn wrote: post #101
rao-v wrote: post #66

This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. Unfortunately I don’t know how you ground this … it’s basically asking if you can encode activations in plausible sounding text. Of course you can! But is the plausible text actually reflective of what the model is “thinking”? How to tell?

It's asking if you can auto encode activations. The AV decodes activations to text, and the AR re-encodes them back to activations. If the decoded text is completely wrong then it's unclear how the second model would re-encode them successfully given that they're both initialized from the same LM.

Think of it another way, can I do this exact training process with an additional requirement that the activation decoder subtly shill for obscure 80s sodas?

I could and would not lose much reconstruction accuracy.

So any researcher or ambient biases in the model will impact the general thrust of the textual decodings (and not in ways that reflect the actual model’s process, thinking about X and doing X in a model are very different things).

So how do we tell that the “spirit” is reflective of the model’s thinking and not biased toward Jolt being better than Surge?

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#120

Post by rao-v »

skybrian wrote: post #88
astrange wrote: post #72

Earlier quoted context omitted.

> This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. I think an issue is that there is no permanent path to model understanding because of Goodhart's law. Models are motivated to appear aligned (well-trained) in any metric you use on them, which means that if you develop a new metric and train on it, it'll learn a way to cheat on it.

But that's not how the training works. Goodhart's law isn't magic. The original model is frozen, so it doesn't learn anything. The copies of the model are learning different objectives and have no incentive to be "loyal" to the original model. Maybe you're imagining they'll hook this up in some larger training loop, but they haven't done that yet.

Yes this is exactly why I think this approach has some potential.

Frozen base mode is something that we should be able to extract insights from without running into Goodhart

Post reply on HN