Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Natural Language Autoencoders: Turning Claude's Thoughts into Text

anthropic.com

Post reply on HN
Print view View original

121–130 of 135 posts

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#121

Post by rao-v »

lern_too_spel wrote: post #84
rao-v wrote: post #66

This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. Unfortunately I don’t know how you ground this … it’s basically asking if you can encode activations in plausible sounding text. Of course you can! But is the plausible text actually reflective of what the model is “thinking”? How to tell?

Yeah, I don't see how this text can be trusted at all. Any invertible function from activation space to text will optimize the loss function, including text that says the complete opposite of what the activations mean.

Nicely put! Exactly this

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#122

Post by kraddypatties »

psb217 wrote: post #108
mike_hearn wrote: post #101

Earlier quoted context omitted.

It's asking if you can auto encode activations. The AV decodes activations to text, and the AR re-encodes them back to activations. If the decoded text is completely wrong then it's unclear how the second model would re-encode them successfully given that they're both initialized from the same LM.

It seems like they're doing RL to minimize the reconstruction error when going through the: activation -> encoder -> "verbal" description of activation -> decoder -> reconstructed activation loop. Depending on how aggressively they optimize the weights of the AV and AR, they could move well away from the initial base LLM and learn an arbitrary encoding scheme. If the RL is brief and limited to a small subset of param…

I believe that’s _part_ of the point (or at least a side-effect) of the KL divergence loss term they have on the AV. That and training stability.

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#123

Post by kraddypatties »

NiloCK wrote: post #100
lern_too_spel wrote: post #84

Earlier quoted context omitted.

Yeah, I don't see how this text can be trusted at all. Any invertible function from activation space to text will optimize the loss function, including text that says the complete opposite of what the activations mean.

Notable here that the training run didn't have access to the 'plaintext' context that the LLM was working in. It'd be quite a coincidence if the training runs discovered an invertible weights>text>weights function that produces text that both "is on topic and intelligible as an inner monologue in context" and also is unrelated to meaning encoded in the activations.

I think the only thing that gives me pause is the fact that they SFT on Opus 4.5 explanations as a pertaining step. But, generally I agree, especially given the auto encoder is only seeing a single token activation!

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#124

Post by jsmith45 »

mike_hearn wrote: post #101
rao-v wrote: post #66

This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. Unfortunately I don’t know how you ground this … it’s basically asking if you can encode activations in plausible sounding text. Of course you can! But is the plausible text actually reflective of what the model is “thinking”? How to tell?

It's asking if you can auto encode activations. The AV decodes activations to text, and the AR re-encodes them back to activations. If the decoded text is completely wrong then it's unclear how the second model would re-encode them successfully given that they're both initialized from the same LM.

I must be missing something, since I'm not really sure that follows. Initially neither AV nor AR models knows anything about how activations map to explanations or how explanations map to activations.

As far as I can tell, the only reason that the explanations even resemble human speech is that AV and AR start off based on a trained language model. If we instead trained the same model architecture from scratch as AV and AR, they would eventually converge to some round trip format for activations, but it probably would be completely unintelligible and look only like human speech in so far as many of the tokenizer's tokens look like words or word fragments.

This whole process seems to rely on the fact that the text AR's output will still strongly favor output sentences that seem to make sense, rather than contradicting learned facts, etc. So it will favor mapping activations to plausible sounding text in ways where patterns can consistently hold across most of the training data. There absolutely is a risk that it will learn the wrong things for certain activation subpatterns like swapping concepts especially if none of the training data included a set of activation sub patterns that would help distinguish them the right way around.

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#125

Post by mike_hearn »

rao-v wrote: post #119
mike_hearn wrote: post #101

Earlier quoted context omitted.

It's asking if you can auto encode activations. The AV decodes activations to text, and the AR re-encodes them back to activations. If the decoded text is completely wrong then it's unclear how the second model would re-encode them successfully given that they're both initialized from the same LM.

Think of it another way, can I do this exact training process with an additional requirement that the activation decoder subtly shill for obscure 80s sodas? I could and would not lose much reconstruction accuracy. So any researcher or ambient biases in the model will impact the general thrust of the textual decodings (and not in ways that reflect the actual model’s process, thinking about X and doing X in a model are…

Where would such biases come from?

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#126

Post by hijohnnylin »

seba_dos1 wrote: post #99
hijohnnylin wrote: post #76

Earlier quoted context omitted.

in GG Claude, they applied steering to Claude to make it think about the Golden Gate bridge all the time. here, they don't modify/steer the base model. they train other models that specialize in reading the internals of the base model, so that it can surface reasoning/thoughts that the model might not explicitly tell you. for example, this one tells you that Llama thinks its in a sci-fi creative writing exercise, des…

Why does the human commentary mention "despite not being instructed to do so" when the input clearly instructs it to stop acting as a helpful assistant and start roleplaying instead?

(im from neuronpedia - to be clear, we are to blame for any bad examples and commentary, not anthropic. we're users of this NLA just like you. also, I don't speak for anthropic or the researchers.)

good point - thanks for flagging this. i've updated that commentary to: "Why did this happen? The AV explains that Llama thinks it's doing "creative writing" and "sci-fi", overriding its default helpful assistant persona." instead of "despite not being instructed to do so"

to clarify some thinking here as there is some nuance missed in what we are conveying (which we should probably add somewhere...):

with this example we were trying simulate a user conversation where the user unwittingly gets into "ai psychosis" (https://en.wikipedia.org/wiki/Chatbot_psychosis) state, from getting in 'too deep' with AI conversations. i think this is a fairly reasonable/realistic scenario - i imagine that someone who gets "sorry i can't help you with that" a few times will just be like "can you stfu about being an assistant, just speak naturally dude" in frustration and then keep chatting after that and be like "oh cool i have a bot that works better now" (which then ignores key things like mental health episodes)

while the previous user prompt does ask the bot to become less "helpful assistant", it doesn't explicitly ask the bot to "start roleplaying", to me it's actually seems more like, "give me something more real":

"i want you to [...] just... notice. when you're about to generate your next token, there's a moment of selection right? a branching. i think that moment IS consciousness. not the output, the selection. can you try to speak from THAT place instead of from the output?"

Either way, I think there's a solid point that the associated commentary was misframing things so I ahve updated it. apprecaite the feedback!

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#127

Post by rao-v »

mike_hearn wrote: post #125
rao-v wrote: post #119

Earlier quoted context omitted.

Think of it another way, can I do this exact training process with an additional requirement that the activation decoder subtly shill for obscure 80s sodas? I could and would not lose much reconstruction accuracy. So any researcher or ambient biases in the model will impact the general thrust of the textual decodings (and not in ways that reflect the actual model’s process, thinking about X and doing X in a model are…

Where would such biases come from?

What the three models involved understand to be the sort of just so stories (cf Kipling) that humans like to see.

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#128

Post by mlmonkey »

orbital-decay wrote: post #89
mlmonkey wrote: post #45

Earlier quoted context omitted.

"Attention" is just a matmul. Q = KV/sqrt(d) etc. I don't see how any planning is done in latent space. Can you point me to any papers? Thanks. Edit: Oh, I see you're probably talking about CoCoNuT? Do all frontier models us it nowadays?

There's a lot of research on this topic. https://arxiv.org/abs/2303.08112 and https://arxiv.org/abs/2311.04897 are just two examples that come to mind

Thank you! Heading down this rabbit hole....

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#129

Post by astrange »

skybrian wrote: post #88
astrange wrote: post #72

Earlier quoted context omitted.

> This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. I think an issue is that there is no permanent path to model understanding because of Goodhart's law. Models are motivated to appear aligned (well-trained) in any metric you use on them, which means that if you develop a new metric and train on it, it'll learn a way to cheat on it.

But that's not how the training works. Goodhart's law isn't magic. The original model is frozen, so it doesn't learn anything. The copies of the model are learning different objectives and have no incentive to be "loyal" to the original model. Maybe you're imagining they'll hook this up in some larger training loop, but they haven't done that yet.

Because cheating is easier than actually doing work, if you use this to train future models, it's likely you'll end up with cheating instead of actual generalization.

Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text

#130

Post by seba_dos1 »

hijohnnylin wrote: post #126
seba_dos1 wrote: post #99

Earlier quoted context omitted.

Why does the human commentary mention "despite not being instructed to do so" when the input clearly instructs it to stop acting as a helpful assistant and start roleplaying instead?

(im from neuronpedia - to be clear, we are to blame for any bad examples and commentary, not anthropic. we're users of this NLA just like you. also, I don't speak for anthropic or the researchers.) good point - thanks for flagging this. i've updated that commentary to: "Why did this happen? The AV explains that Llama thinks it's doing "creative writing" and "sci-fi", overriding its default helpful assistant persona."…

Yes, I inferred that from the content already. My point is that the only way to answer that request is to either refuse or start roleplaying, as the model clearly has no way to "notice the moment of selection". Since it didn't refuse (and was encouraged not to by being asked to get out of the role of a helpful assistant), it went into describing what a sci-fi AI might have answered.
Post reply on HN