Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

transformer-circuits.pub

Post reply on HN
Print view View original

21–30 of 128 posts

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#21

Post by viking123 »

optimalsolver wrote: post #3

>what the model is "thinking" before writing its response An actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history. An LLM is doing exactly nothing while waiting for the next prompt.

I might be a complete brainlet so excuse my take, but when animals think and do things, the weights in the brain are constantly being adjusted, old connections pruned out and new ones made right? But once LLM is trained, that's kind of it? Nothing there changes when we discuss with it. As far as I understand from what I read, even our memories are just somehow in the connections between the neurons

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#22

Post by pdevr »

So, to summarize:

>Used "dictionary learning"

>Found abstract features

>Found similar/close features using distance

>Tried amplifying and suppressing features

Not trying to be snary, but sounds mundane in the ML/LLM world. Then again, significant advances have come from simple concepts. Would love to hear from someone who has been able to try this out.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#23

Post by whimsicalism »

viking123 wrote: post #21
optimalsolver wrote: post #3

>what the model is "thinking" before writing its response An actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history. An LLM is doing exactly nothing while waiting for the next prompt.

I might be a complete brainlet so excuse my take, but when animals think and do things, the weights in the brain are constantly being adjusted, old connections pruned out and new ones made right? But once LLM is trained, that's kind of it? Nothing there changes when we discuss with it. As far as I understand from what I read, even our memories are just somehow in the connections between the neurons

my understanding was that once you are of age, brain pruning and malleability is relatively small

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#24

Post by verdverm »

wwarner wrote: post #20

huge. the activation scan, which looks for which nodes change the most when prompted with the words "Golden Gate Bridge" and later an image of the same bridge, is eerily reminiscent of a brain scan under similar prompts...

I find this outcome expected and not really surprising, more confirmation of previous results. Consider vision transformers and the papers that showed what each layer was focused on.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#25

Post by HarHarVeryFunny »

byteknight wrote: post #2

This reminds me of how people often communicate to avoid offending others. We tend to soften our opinions or suggestions with phrases like "What if you looked at it this way?" or "You know what I'd do in those situations." By doing this, we subtly dilute the exact emotion or truth we're trying to convey. If we modify our words enough, we might end up with a statement that's completely untruthful. This is similar to h…

A true AGI would learn to manipulate it's environment to achieve it's goals, but obviously we are not there yet.

An LLM has no goals - it's just a machine optimized to minimize training errors, although I suppose you could view this as an innate hard-coded goal of minimizing next word error (relative to training set), in same way we might say a machine-like insect has some "goals".

Of course RLHF provides a longer time span (entire response vs next word) error to minimize, but I doubt training volume is enough for the model to internally model a goal of manipulating the listener as opposed to just favoring surface forms of response.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#26

Post by whimsicalism »

verdverm wrote: post #18
sabrina_ramonov wrote: post #12

Earlier quoted context omitted.

we know so little about thinking and consciousness, these claims seem premature

That one can fix the RNG and get consistent output indicates a lack of dynamics They certainly do not self update the weights in an online process as needed information is experienced

If we could perfectly simulate the brain and there were quantum hidden variables, we too could “fix RNG and get deterministic output”

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#27

Post by verdverm »

whimsicalism wrote: post #16
optimalsolver wrote: post #3

>what the model is "thinking" before writing its response An actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history. An LLM is doing exactly nothing while waiting for the next prompt.

If we figured out how to freeze and then revive brains, would that mean that all of the revived brains were no longer thinking because they had previously been paused at some point? Frankly this objection seems very weak

There are many more features that would be needed, such as a peer comment pointed out, being able to recognize you are saying something incorrect, pausing, and then starting a new stream of output.

This is currently done with multiple LLMs and calls, not within the running of a single model i/o

Another example would be to input a single token or gibberish, the models we have today are more than happy to spit out fantastic numbers of tokens. They really only stop because we look for stop words they are trained to generate and we do the actual stopping action

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#28

Post by handwarmers »

whimsicalism wrote: post #11

I continue to be impressed by Anthropic’s work and their dual commitment to scaling and safety. HN is often characterized by a very negative tone related to any of these developments, but I really do feel that Anthropic is trying to do a “race to the top” in terms of alignment, though it doesn’t seem like all the other major companies are doing enough to race with them. Particularly frustrating on HN is the common sy…

What is often frustrating to me at least is the arbitrary definition of "safety" and "ethics", forged by a small group of seemingly intellectually homogenous individuals.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#30

Post by whimsicalism »

handwarmers wrote: post #28
whimsicalism wrote: post #11

I continue to be impressed by Anthropic’s work and their dual commitment to scaling and safety. HN is often characterized by a very negative tone related to any of these developments, but I really do feel that Anthropic is trying to do a “race to the top” in terms of alignment, though it doesn’t seem like all the other major companies are doing enough to race with them. Particularly frustrating on HN is the common sy…

What is often frustrating to me at least is the arbitrary definition of "safety" and "ethics", forged by a small group of seemingly intellectually homogenous individuals.

Say more, this is half a thought
Post reply on HN