They are trying to figure out what they actually built.
I suspect the time is coming when there will always be an aligned search AI between you and the internet.
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
121–128 of 128 posts
Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
#122Earlier quoted context omitted.
Why would that matter? The absolute orientation of the mind map doesn't matter - maybe my map is actually very close to yours, subject to some rotation and mirroring? More than that, I'd think a better 2D analogy for the latent space is a force-directed graph that you keep shaking as you add things to it. It doesn't seem unlikely for two such graphs, constructed in different order, to still end up identical in the en…
> there's no reason for our mind maps to meaningfully differ here Yes there is. If you think all training runs converge to the same bits given the same output size, I would again stress that the visual dimensions analogy is poetics and extremely tortured. If you're making the weaker claim that generally concepts sort themselves into a space and they're generally sorted the same way if we have the same training data.…
I think you are hung up on the visual representation.
Last week, the post about jailbreaking ChatGPT(?) talked about turning off a direction in possibility-space to disable the "I'm sorry, but I can't..." message.
In a regular program, it would be a boolean variable, or a single ASM instruction.
And you could ask the same thing. "How does my program have an off switch if there aren't enough values to store all possible meanings of "off"? Does my off switch variable map to your off switch variable?"
And the answer would be yes, or no, or it doesn't matter. It's a tool/construct.
Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
#123I find Anthorpic's work on mech interp fascinating in general. Their initial towards monosemanticity paper was highly surprising, and so is this with the ability to scale to a real production-scale LLM. My observation is, and this may be more philosophical than technical: this process of "decomposing" middle-layer activations with a sparse autoencoder -- is it capturing accurately underlying features in the latent sp…
I'm allergic to latent space because I've yet to find any meaning to it beyond poetics, I develop an acute allergy when it's explicitly related to visually dimensional ideas like clustering. I'll make a probably bad analogy: does your mindmap place things near each other like my mindmap? To which I'd say, probably not, mindmaps are very personal, and the more complex we put on ours, the more personal and arbitrary th…
I mean, it's mostly about how close concepts are to each other, and to some extent how different concepts are placed on a given axis. Of course the concept space is very high-dimensional so it's not very easy to visualise without reducing the dimensions, but because we mostly care about distance that reduction is not particularly lossy, but it does mean that top-left vs bottom right doesn't mean much, it's more that mcdonalds is usually closer to food than it is to, say, gearboxes (and that a representation that doesn't do that probably doesn't understand the concepts very well).
Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
#124[deleted]
Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
#125My thoughts
- LLM Just got a whole set of buttons you can push. Potential for the LLM to push its own buttons?
- Read the paper and ctrl+f 'deplorable'. This shows once again how we are underestimating LLM's ability to appear conscious. It can be really effective. Reminiscence of Dr.Ford in Westworld :'you (robots) never look more human than when you are suffering.' Or something like that, anyway. I might be hallucinating dialogue but pretty sure something like that was said and I think it's quite true.
- Intensely realistic roleplaying potential unlocked.
- Efficiency by reducing context length by directly amplifying certain features instead.
Very powerful stuff. I am waiting eagerly when I can play with it myself. (Someone please make it a local feature)
Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
#126It looks like Anthropic is now leading the charge on safety
They always were given that is a part of their mission.
Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
#127Earlier quoted context omitted.
> there's no reason for our mind maps to meaningfully differ here Yes there is. If you think all training runs converge to the same bits given the same output size, I would again stress that the visual dimensions analogy is poetics and extremely tortured. If you're making the weaker claim that generally concepts sort themselves into a space and they're generally sorted the same way if we have the same training data.…
Yes, I'm making the weaker claim that concepts would generally sort themselves into roughly equivalent structures, that could be mapped to each other through some easy affine transformations (rotation, symmetry, translation, etc.) applied to various parts of the structures. Or, in other words, I think absolute coordinates of any concept in the latent space are irrelevant and it makes no sense to compare them between…
What would be an appropriate test?
- Given 2 word embedding sets,
- For each pair (A,B) of embeddings in one set,
- There exists an equivalence (A’,B’) in the other set,
- Such that dist(A,B) ≈ dist(A’, B’),
Something like that, to start. But would need to look at longer chains of relations.
Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
#128Earlier quoted context omitted.
A true AGI would learn to manipulate it's environment to achieve it's goals, but obviously we are not there yet. An LLM has no goals - it's just a machine optimized to minimize training errors, although I suppose you could view this as an innate hard-coded goal of minimizing next word error (relative to training set), in same way we might say a machine-like insect has some "goals". Of course RLHF provides a longer ti…
The next big breakthrough in the LLM space will be having a way to represent goals/intentions of the LLM and then execute them in the way that is the most appropriate/logical/efficient (I'm pretty sure some really smart people have been thinking about this for a while). Perhaps at some point LLMs will start to evolve from the prompt->response model into something more asynchronous and with some activity happening in…
That’s not really an LLM at that point, but a agent built around an LLM