Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

transformer-circuits.pub

Post reply on HN
Print view View original

61–70 of 128 posts

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#61

Post by Nevermark »

HarHarVeryFunny wrote: post #25
byteknight wrote: post #2

This reminds me of how people often communicate to avoid offending others. We tend to soften our opinions or suggestions with phrases like "What if you looked at it this way?" or "You know what I'd do in those situations." By doing this, we subtly dilute the exact emotion or truth we're trying to convey. If we modify our words enough, we might end up with a statement that's completely untruthful. This is similar to h…

A true AGI would learn to manipulate it's environment to achieve it's goals, but obviously we are not there yet. An LLM has no goals - it's just a machine optimized to minimize training errors, although I suppose you could view this as an innate hard-coded goal of minimizing next word error (relative to training set), in same way we might say a machine-like insect has some "goals". Of course RLHF provides a longer ti…

An LLM has no explicit goals.

But simply by approximating human communication which often models goal oriented behavior, an LLM can have implicit goals. Which likely vary widely according to conversation context.

Implicit goals can be very effective. Nowhere in DNA is there any explicit goal to survive. However combinations of genes and markers selected for survivability create creatures with implicit goals to survive as tenacious as any explicit goals might be.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#62

Post by whimsicalism »

HarHarVeryFunny wrote: post #52
whimsicalism wrote: post #40

Earlier quoted context omitted.

if you zoom in enough, all thinking is an expert system with a fixed set of rules.

That the basis of it, but in our brain the "inference engine" using those rules is a lot more than a fixed N-steps - there is thalamo-cortical looping, working memory of various durations, and maybe a bunch of other mechanisms such as analogical recall, resonance-based winner-takes-all processing, etc, etc. Current LLMs have none of that - they are just the fixed set of rules, further limited by also having a fixed n…

Yes, LLMs don't have regression and that is a significant limitation - although they do have something close, by decoding one token they get to then have a thought loop. They just can't loop without outputting.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#63

Post by byteknight »

verdverm wrote: post #47
byteknight wrote: post #42

Earlier quoted context omitted.

Exactly. You can't reason with that you do not currently posses.

How does scientific progress happen without reasoning about that which we do not know or understand?

That's building upon current knowledge. That is a different application.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#64

Post by HarHarVeryFunny »

whimsicalism wrote: post #62
HarHarVeryFunny wrote: post #52

Earlier quoted context omitted.

That the basis of it, but in our brain the "inference engine" using those rules is a lot more than a fixed N-steps - there is thalamo-cortical looping, working memory of various durations, and maybe a bunch of other mechanisms such as analogical recall, resonance-based winner-takes-all processing, etc, etc. Current LLMs have none of that - they are just the fixed set of rules, further limited by also having a fixed n…

Yes, LLMs don't have regression and that is a significant limitation - although they do have something close, by decoding one token they get to then have a thought loop. They just can't loop without outputting.

Well, not exactly a loop. They get to "extend the thought", but there is zero continuity from one word to the next (LLM starts from scratch for each token generated).

The effect is as if you had multiple people playing a game where they each extend a sentence by taking turns adding a word to it, but there is zero continuity from one word to the next because each person is starting from scratch when it is their turn.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#65

Post by wwarner »

verdverm wrote: post #55
wwarner wrote: post #49

Earlier quoted context omitted.

Yes, lots of activity in the space. I thought you were saying it was a dumb problem, but I was wrong. I think this is a great paper.

yup, if you look at drop out, what it does and why, you can see additional interesting results along these lines (drop-out was found to increase resilience in models because they had to encode information in the weights differently, i.e. could not rely on single neuron (at the limit))

I suppose, except that for a model of 7B parameters, the number of combinations of dropout that you'd be analyzing is 7B factorial. More importantly, dropout has loss minimization to guide it during training, whereas understanding how a model changes when you edit a few weights is a very broad question.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#66

Post by HarHarVeryFunny »

Nevermark wrote: post #61
HarHarVeryFunny wrote: post #25

Earlier quoted context omitted.

A true AGI would learn to manipulate it's environment to achieve it's goals, but obviously we are not there yet. An LLM has no goals - it's just a machine optimized to minimize training errors, although I suppose you could view this as an innate hard-coded goal of minimizing next word error (relative to training set), in same way we might say a machine-like insect has some "goals". Of course RLHF provides a longer ti…

An LLM has no explicit goals. But simply by approximating human communication which often models goal oriented behavior, an LLM can have implicit goals. Which likely vary widely according to conversation context. Implicit goals can be very effective. Nowhere in DNA is there any explicit goal to survive. However combinations of genes and markers selected for survivability create creatures with implicit goals to surviv…

Yes, the short term behavior/output of the LLM could reflect an implicit goal, but I doubt it'd maintain any such goal for an extended period of time (long-term coherence of behavior is a known shortcoming), since there is random sampling being done, and no internal memory from word to word - it seems that any implicit goal will likely rapidly drift.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#67

Post by quotemstr »

At this risk of anthropomorphizing too much, I can't help but see parallels between the "my physical form is the Golden Gate Bridge" screenshot and the https://en.wikipedia.org/wiki/God_helmet in humans --- both cognitive distortions caused by targeted exogenous neural activation.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#68

Post by verdverm »

wwarner wrote: post #65
verdverm wrote: post #55

Earlier quoted context omitted.

yup, if you look at drop out, what it does and why, you can see additional interesting results along these lines (drop-out was found to increase resilience in models because they had to encode information in the weights differently, i.e. could not rely on single neuron (at the limit))

I suppose, except that for a model of 7B parameters, the number of combinations of dropout that you'd be analyzing is 7B factorial. More importantly, dropout has loss minimization to guide it during training, whereas understanding how a model changes when you edit a few weights is a very broad question.

the analysis is more akin to analyzing with & without dropout, where a common number is to drop a random 50% of connections during a pass for training, thus forcing the model to not rely on specific nodes or connections

When you look at a specific input, you can look to see what gets activated or not. Orthogonal but related ideas for inspecting the activations to see effects

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#69

Post by CamperBob2 »

whimsicalism wrote: post #30
handwarmers wrote: post #28

Earlier quoted context omitted.

What is often frustrating to me at least is the arbitrary definition of "safety" and "ethics", forged by a small group of seemingly intellectually homogenous individuals.

Say more, this is half a thought

E.g., the common sentiment that "NSFW" output is to be prohibited, regardless of whether you work in a steel mill or a church.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#70

Post by whimsicalism »

HarHarVeryFunny wrote: post #64
whimsicalism wrote: post #62

Earlier quoted context omitted.

Yes, LLMs don't have regression and that is a significant limitation - although they do have something close, by decoding one token they get to then have a thought loop. They just can't loop without outputting.

Well, not exactly a loop. They get to "extend the thought", but there is zero continuity from one word to the next (LLM starts from scratch for each token generated). The effect is as if you had multiple people playing a game where they each extend a sentence by taking turns adding a word to it, but there is zero continuity from one word to the next because each person is starting from scratch when it is their turn.

> LLM starts from scratch for each token generated

What do you mean? They get to access their previous hidden states in the next greedy decode using attention, it is not simply starting from scratch. They can access exactly what they were thinking when they put out the previous word, not just reasoning from the word itself.

Post reply on HN