Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

transformer-circuits.pub

Post reply on HN
Print view View original

101–110 of 128 posts

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#101

Post by rpozarickij »

HarHarVeryFunny wrote: post #25
byteknight wrote: post #2

This reminds me of how people often communicate to avoid offending others. We tend to soften our opinions or suggestions with phrases like "What if you looked at it this way?" or "You know what I'd do in those situations." By doing this, we subtly dilute the exact emotion or truth we're trying to convey. If we modify our words enough, we might end up with a statement that's completely untruthful. This is similar to h…

A true AGI would learn to manipulate it's environment to achieve it's goals, but obviously we are not there yet. An LLM has no goals - it's just a machine optimized to minimize training errors, although I suppose you could view this as an innate hard-coded goal of minimizing next word error (relative to training set), in same way we might say a machine-like insect has some "goals". Of course RLHF provides a longer ti…

The next big breakthrough in the LLM space will be having a way to represent goals/intentions of the LLM and then execute them in the way that is the most appropriate/logical/efficient (I'm pretty sure some really smart people have been thinking about this for a while).

Perhaps at some point LLMs will start to evolve from the prompt->response model into something more asynchronous and with some activity happening in the background too.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#102

Post by canjobear »

refulgentis wrote: post #87
e63f67dd-065b wrote: post #84

I find Anthorpic's work on mech interp fascinating in general. Their initial towards monosemanticity paper was highly surprising, and so is this with the ability to scale to a real production-scale LLM. My observation is, and this may be more philosophical than technical: this process of "decomposing" middle-layer activations with a sparse autoencoder -- is it capturing accurately underlying features in the latent sp…

I'm allergic to latent space because I've yet to find any meaning to it beyond poetics, I develop an acute allergy when it's explicitly related to visually dimensional ideas like clustering. I'll make a probably bad analogy: does your mindmap place things near each other like my mindmap? To which I'd say, probably not, mindmaps are very personal, and the more complex we put on ours, the more personal and arbitrary th…

Neural network representation spaces seem to converge, regardless of architecture: https://arxiv.org/abs/2405.07987

It would make sense for the human mental latent spaces to also converge. The reason is that the latent space exists to model the environment, which is largely shared among humans.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#103

Post by skybrian »

baq wrote: post #92
e63f67dd-065b wrote: post #84

I find Anthorpic's work on mech interp fascinating in general. Their initial towards monosemanticity paper was highly surprising, and so is this with the ability to scale to a real production-scale LLM. My observation is, and this may be more philosophical than technical: this process of "decomposing" middle-layer activations with a sparse autoencoder -- is it capturing accurately underlying features in the latent sp…

> concepts are made by man I find this statement... controversial? The canonical example would be mathemathics - are they discovered or invented? Does the idea of '3' or an empty set or a straight line exist without any humans thinking about it or even if it is necessary to have any kind of an universe at all for these concepts to be valid? I think the answers here are 'yes' and 'no'. Of course, there are still conce…

People often modify their environment to make their concepts work. This is true even of counting:

https://metarationality.com/pebbles

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#105

Post by Workaccount2 »

kromem wrote: post #90

Great work as usual. I was pretty upset seeing the superalignment team dissolve at OpenAI, but as is typical for the AI space, the news of one day was quickly eclipsed by the next day. Anthropic are really killing it right now, and it's very refreshing seeing their commitment to publishing novel findings. I hope this finally serves as the nail in the coffin on the "it's just fancy autocomplete" and "it doesn't unders…

> on the "it's just fancy autocomplete" and "it doesn't understand what it's saying, bro" rhetoric.

No matter what, there will always be a group of people saying that. The power and drive of the brain to convince itself that it is weaved of magical energy on a divine substrate shouldn't be underestimated. Especially when media plays so hard into that idea (the robots that lose the war because they cannot overcome love, etc.) because brains really love being told they are right.

I am almost certain that the first conscious silicon (or whatever material) will be subjected to immense suffering until a new generation that can accept the human brains banality can move things forward.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#106

Post by PoignardAzur »

gautomdas wrote: post #86

I've really been enjoying their series on mech interp, does anyone have any other good recs?

"Transformers Represent Belief State Geometry in their Residual Stream":

https://www.lesswrong.com/posts/gTZ2SxesbHckJ3CkF/transforme...

Basically finding that transformers don't just store a world-model as in "what does the world that produce the observed inputs look like?", they store a "Mixed-State Presentation", basically a weighted set of possible worlds that produce the observed inputs.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#107

Post by bjterry »

ericflo wrote: post #81
bjterry wrote: post #73

It would be interesting to allow users of models to customize inference by tweaking these features, sort of like a semantic equalizer for LLMs. My guess is that this wouldn't work as well as fine-tuning, since that would tweak all the features at once toward your use case, but the equalizer would require zero training data. The prompt itself can trigger the features, so if you say "Try to weave in mentions of San Fra…

Related: https://vgel.me/posts/representation-engineering/

Thanks!

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#108

Post by refulgentis »

TeMPOraL wrote: post #97
refulgentis wrote: post #87

Earlier quoted context omitted.

I'm allergic to latent space because I've yet to find any meaning to it beyond poetics, I develop an acute allergy when it's explicitly related to visually dimensional ideas like clustering. I'll make a probably bad analogy: does your mindmap place things near each other like my mindmap? To which I'd say, probably not, mindmaps are very personal, and the more complex we put on ours, the more personal and arbitrary th…

Why would that matter? The absolute orientation of the mind map doesn't matter - maybe my map is actually very close to yours, subject to some rotation and mirroring? More than that, I'd think a better 2D analogy for the latent space is a force-directed graph that you keep shaking as you add things to it. It doesn't seem unlikely for two such graphs, constructed in different order, to still end up identical in the en…

> there's no reason for our mind maps to meaningfully differ here

Yes there is.

If you think all training runs converge to the same bits given the same output size, I would again stress that the visual dimensions analogy is poetics and extremely tortured.

If you're making the weaker claim that generally concepts sort themselves into a space and they're generally sorted the same way if we have the same training data. Or rotational symmetry means any differences don't matter. Or location doesn't matter at all...we're in poetics.

Something that really sold me when I was in a similar mindset was word2vec's king - man + woman = queen wasn't actually real or in the model. Just a way of explaining it simply.

Another thought from my physics days: try visualizing 4D. Some people do claim to, after much effort, but in my experience they're unserious, i.e. I didn't see PhDs or masters students in my program claiming this. No one tries claiming they can see in 5D.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#109

Post by jwilber »

kromem wrote: post #93
jwilber wrote: post #91

Earlier quoted context omitted.

Love Anthropic research. Great visuals between Olah, Carter, and Pearce, as well. I don’t think this paper does much in the way of your final point, “it doesn’t understand what it’s saying”, though our understanding certainly has improved.

They were able to demonstrate conceptual vectors that were consistent across different languages and different mediums (text vs images) and that when manipulated were able to represent the abstract concept in the output regardless of prompt. What kind of evidentiary threshold would you want if that's not sufficient?

My point is that you claimed this is a rebuff against those claiming models don’t understand themselves. Your interpretation seems to assign intelligence to the algorithms.

While this research allows us to interpret larger models in an amazing way, it doesn’t mean the models themselves ‘understand’ anything.

You can use this on much smaller scale models as well, as they showed 8 months ago. Does that research tell us about how models understand themselves? Or does it help us understand how the models work?

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#110

Post by astrange »

kromem wrote: post #90

Great work as usual. I was pretty upset seeing the superalignment team dissolve at OpenAI, but as is typical for the AI space, the news of one day was quickly eclipsed by the next day. Anthropic are really killing it right now, and it's very refreshing seeing their commitment to publishing novel findings. I hope this finally serves as the nail in the coffin on the "it's just fancy autocomplete" and "it doesn't unders…

I think the research is good, but it's disappointing that they hype it by claiming it's going to help their basically entirely fictional "AI safety" project, as if the bits in their model are going to come alive and eat them.
Post reply on HN