The issue with the AI blackmail tests is that newer versions of AIs are trained after the AI blackmail experiments were published online. Or do they scrub it from the training data?
Natural Language Autoencoders: Turning Claude's Thoughts into Text
81–90 of 135 posts
Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text
#82Am I correct in my understanding that they are not actually able to 100% know what Claude is thinking? They have trained a new model to make a guess about what Claude is thinking, but we cannot validate that the guess is 100% valid, right? They are basically saying "we have trained a model to reaffirm what we believe Claude is thinking" ? Hoping I'm wrong in my understanding of this because this does not appear to be…
> "we have trained a model to reaffirm what we believe Claude is thinking" ?
It's more like "We have trained a model to produce a text that allows reconstruction of activations and the text happened to coincide with the results of other interpretability methods even after extensive training, while we expected it to devolve into unintelligible mess."
They found something unexpected and useful. They report it, while outlining limitations and ways to improve. It looks like a fine research to me.
Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text
#83I've only read this blog and not the paper so maybe they go into more detail there and someone can correct me, but they frequently bring up the model's ability to detect or at least the model activations hint it can predict when it's being tested. I can't help but wonder, as they build these larger and larger models, where they could be getting "clean" training data, untainted by all these types of blog posts and the…
https://arxiv.org/abs/2410.20245v2 Section 3 outlines the actual method.
Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text
#84This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. Unfortunately I don’t know how you ground this … it’s basically asking if you can encode activations in plausible sounding text. Of course you can! But is the plausible text actually reflective of what the model is “thinking”? How to tell?
Yeah, I don't see how this text can be trusted at all. Any invertible function from activation space to text will optimize the loss function, including text that says the complete opposite of what the activations mean.
Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text
#85Am I correct in my understanding that they are not actually able to 100% know what Claude is thinking? They have trained a new model to make a guess about what Claude is thinking, but we cannot validate that the guess is 100% valid, right? They are basically saying "we have trained a model to reaffirm what we believe Claude is thinking" ? Hoping I'm wrong in my understanding of this because this does not appear to be…
Maybe you can't 100% know what every layer "thinks", if you go through all the layers, you might see a cohesive "thinking" story. So, if there is any information you lose at layer N, you might learn some of it in layer N+1. The masking in the layers is not deterministic so the model can't really consistently lie throughout the layers. It doesn't chose what information we get to inspect. There might be a game of whack-a-mole, but you might get a general sentiment. I think the more layers there are, the more the model itself can hide very nuanced lies (But by that time we'd have a better mind-reading model).
However, I haven't read about it yet. I'm really excited to look into it!
Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text
#86So, this is like reading EKG of human brain and understand its thoughts?
Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text
#87This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. Unfortunately I don’t know how you ground this … it’s basically asking if you can encode activations in plausible sounding text. Of course you can! But is the plausible text actually reflective of what the model is “thinking”? How to tell?
> This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. I think an issue is that there is no permanent path to model understanding because of Goodhart's law. Models are motivated to appear aligned (well-trained) in any metric you use on them, which means that if you develop a new metric and train on it, it'll learn a way to cheat on it.
The obvious fix is to make interpretation of itself a part of the model (like we can explicitly introspect to a certain extent what the brain is doing). Misinterpretation of itself, hopefully, would decrease the system's performance on all tasks and it would be rooted out by training. Of course, it doesn't mean that the fix is easy to implement and that it doesn't have other failure modes.
Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text
#88This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. Unfortunately I don’t know how you ground this … it’s basically asking if you can encode activations in plausible sounding text. Of course you can! But is the plausible text actually reflective of what the model is “thinking”? How to tell?
> This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. I think an issue is that there is no permanent path to model understanding because of Goodhart's law. Models are motivated to appear aligned (well-trained) in any metric you use on them, which means that if you develop a new metric and train on it, it'll learn a way to cheat on it.
But that's not how the training works. Goodhart's law isn't magic.
The original model is frozen, so it doesn't learn anything. The copies of the model are learning different objectives and have no incentive to be "loyal" to the original model.
Maybe you're imagining they'll hook this up in some larger training loop, but they haven't done that yet.
Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text
#89Earlier quoted context omitted.
This is incorrect. In the process of producing each token, activations are produced at each layer which are made available to future token production processes via the attention mechanism. The overall depth of computations that use this latent information without passing through output tokens is limited to the depth of the network, but there has been ample evidence that models can do limited "planning" and related ca…
"Attention" is just a matmul. Q = KV/sqrt(d) etc. I don't see how any planning is done in latent space. Can you point me to any papers? Thanks. Edit: Oh, I see you're probably talking about CoCoNuT? Do all frontier models us it nowadays?
There's a lot of research on this topic. https://arxiv.org/abs/2303.08112 and https://arxiv.org/abs/2311.04897 are just two examples that come to mind
Re: Natural Language Autoencoders: Turning Claude's Thoughts into Text
#90Attach the SRT to your frozen model Anthropic. Problem solved. https://github.com/space-bacon/SRT .
I see your repository’s README says > Language models process signs (representamens) but are blind to when meaning forks — when the same word means different things to different communities. But, haven’t interpretability results shown that these models internally represent several meanings of the same word, differently? In that case, why would they not already do the same for how words are used differently in differe…
[dead]