Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

transformer-circuits.pub

Post reply on HN
Print view View original

91–100 of 128 posts

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#91

Post by jwilber »

kromem wrote: post #90

Great work as usual. I was pretty upset seeing the superalignment team dissolve at OpenAI, but as is typical for the AI space, the news of one day was quickly eclipsed by the next day. Anthropic are really killing it right now, and it's very refreshing seeing their commitment to publishing novel findings. I hope this finally serves as the nail in the coffin on the "it's just fancy autocomplete" and "it doesn't unders…

Love Anthropic research. Great visuals between Olah, Carter, and Pearce, as well.

I don’t think this paper does much in the way of your final point, “it doesn’t understand what it’s saying”, though our understanding certainly has improved.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#92

Post by baq »

e63f67dd-065b wrote: post #84

I find Anthorpic's work on mech interp fascinating in general. Their initial towards monosemanticity paper was highly surprising, and so is this with the ability to scale to a real production-scale LLM. My observation is, and this may be more philosophical than technical: this process of "decomposing" middle-layer activations with a sparse autoencoder -- is it capturing accurately underlying features in the latent sp…

> concepts are made by man

I find this statement... controversial?

The canonical example would be mathemathics - are they discovered or invented? Does the idea of '3' or an empty set or a straight line exist without any humans thinking about it or even if it is necessary to have any kind of an universe at all for these concepts to be valid? I think the answers here are 'yes' and 'no'.

Of course, there are still concepts which require grounding in the universe or humanity, but if you can think these up first (...somehow), you should need neither.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#93

Post by kromem »

jwilber wrote: post #91
kromem wrote: post #90

Great work as usual. I was pretty upset seeing the superalignment team dissolve at OpenAI, but as is typical for the AI space, the news of one day was quickly eclipsed by the next day. Anthropic are really killing it right now, and it's very refreshing seeing their commitment to publishing novel findings. I hope this finally serves as the nail in the coffin on the "it's just fancy autocomplete" and "it doesn't unders…

Love Anthropic research. Great visuals between Olah, Carter, and Pearce, as well. I don’t think this paper does much in the way of your final point, “it doesn’t understand what it’s saying”, though our understanding certainly has improved.

They were able to demonstrate conceptual vectors that were consistent across different languages and different mediums (text vs images) and that when manipulated were able to represent the abstract concept in the output regardless of prompt.

What kind of evidentiary threshold would you want if that's not sufficient?

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#96

Post by anentropic »

refulgentis wrote: post #87
e63f67dd-065b wrote: post #84

I find Anthorpic's work on mech interp fascinating in general. Their initial towards monosemanticity paper was highly surprising, and so is this with the ability to scale to a real production-scale LLM. My observation is, and this may be more philosophical than technical: this process of "decomposing" middle-layer activations with a sparse autoencoder -- is it capturing accurately underlying features in the latent sp…

I'm allergic to latent space because I've yet to find any meaning to it beyond poetics, I develop an acute allergy when it's explicitly related to visually dimensional ideas like clustering. I'll make a probably bad analogy: does your mindmap place things near each other like my mindmap? To which I'd say, probably not, mindmaps are very personal, and the more complex we put on ours, the more personal and arbitrary th…

what if you averaged over millions of peoples' mindmaps?

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#97

Post by TeMPOraL »

refulgentis wrote: post #87
e63f67dd-065b wrote: post #84

I find Anthorpic's work on mech interp fascinating in general. Their initial towards monosemanticity paper was highly surprising, and so is this with the ability to scale to a real production-scale LLM. My observation is, and this may be more philosophical than technical: this process of "decomposing" middle-layer activations with a sparse autoencoder -- is it capturing accurately underlying features in the latent sp…

I'm allergic to latent space because I've yet to find any meaning to it beyond poetics, I develop an acute allergy when it's explicitly related to visually dimensional ideas like clustering. I'll make a probably bad analogy: does your mindmap place things near each other like my mindmap? To which I'd say, probably not, mindmaps are very personal, and the more complex we put on ours, the more personal and arbitrary th…

Why would that matter? The absolute orientation of the mind map doesn't matter - maybe my map is actually very close to yours, subject to some rotation and mirroring?

More than that, I'd think a better 2D analogy for the latent space is a force-directed graph that you keep shaking as you add things to it. It doesn't seem unlikely for two such graphs, constructed in different order, to still end up identical in the end.

Thirdly:

> if we have 3 million things on both our mindmaps, it's peering too closely to wonder why you put mcdonalds closer to kids food than restaurants, and you have restaurants in the top left, whereas I put it closer to kids foods, in the top mid left.

In 2D analogy, maybe, but that's because of limited space. In 20 000 D analogy, there's no reason for our mind maps to meaningfully differ here; there's enough dimensions that terms can be close to other terms for any relationship you could think of.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#98

Post by marmadukester39 »

TeMPOraL wrote: post #97
refulgentis wrote: post #87

Earlier quoted context omitted.

I'm allergic to latent space because I've yet to find any meaning to it beyond poetics, I develop an acute allergy when it's explicitly related to visually dimensional ideas like clustering. I'll make a probably bad analogy: does your mindmap place things near each other like my mindmap? To which I'd say, probably not, mindmaps are very personal, and the more complex we put on ours, the more personal and arbitrary th…

Why would that matter? The absolute orientation of the mind map doesn't matter - maybe my map is actually very close to yours, subject to some rotation and mirroring? More than that, I'd think a better 2D analogy for the latent space is a force-directed graph that you keep shaking as you add things to it. It doesn't seem unlikely for two such graphs, constructed in different order, to still end up identical in the en…

This sounds a bit similar to how marketers have thought of the concept of brands and how they cluster in peoples minds for a long time.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#99

Post by tel »

bilsbie wrote: post #56

How are they handling attention in their approach? That’s going to completely change what features are looked at.

They target the residual stream. Also they may have a definition of “feature” that’s more general than what you’re using. Consider reading their superposition work.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#100

Post by HanClinto »

Reminds me of this paper from a couple of weeks ago that isolated the "refusal vector" for prompts that caused the model to decline to answer certain prompts:

https://news.ycombinator.com/item?id=40242939

I love seeing the work here -- especially the way that they identified a vector specifically for bad code. I've been trying to explore the way that we can use adversarial training to increase the quality of code generated by our LLMs, and so using this technique to get countering examples of secure vs. insecure code (to bootstrap the training process) is really exciting.

Overall, fascinating stuff!!

Post reply on HN