Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

transformer-circuits.pub

Post reply on HN
Print view View original

31–40 of 128 posts

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#31

Post by phyalow »

whimsicalism wrote: post #11

I continue to be impressed by Anthropic’s work and their dual commitment to scaling and safety. HN is often characterized by a very negative tone related to any of these developments, but I really do feel that Anthropic is trying to do a “race to the top” in terms of alignment, though it doesn’t seem like all the other major companies are doing enough to race with them. Particularly frustrating on HN is the common sy…

Alot of this really isnt new, Andrej Karpathy covered the principles here 8 years ago for CS231n at Stanford https://youtu.be/yCC09vCHzF8&t=1640

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#32

Post by whimsicalism »

verdverm wrote: post #27
whimsicalism wrote: post #16

Earlier quoted context omitted.

If we figured out how to freeze and then revive brains, would that mean that all of the revived brains were no longer thinking because they had previously been paused at some point? Frankly this objection seems very weak

There are many more features that would be needed, such as a peer comment pointed out, being able to recognize you are saying something incorrect, pausing, and then starting a new stream of output. This is currently done with multiple LLMs and calls, not within the running of a single model i/o Another example would be to input a single token or gibberish, the models we have today are more than happy to spit out fant…

i don’t see why any of the things you’re describing are criteria for thinking, it seems just arbitrarily picking things humans do and saying this is somehow constitutive to thought

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#33

Post by nathan_compton »

harshaxnim wrote: post #13
nathan_compton wrote: post #6

Earlier quoted context omitted.

Counterpoint: "What if you looked at it this way?" communicates both your suggestion AND your sensitivity to the person's social status whatever. Given that humans are not robots, but social, psychological, animals, such communication is entirely justified and efficient.

Sadly "sensitivity" has been over done. It's a fine line and corporations would rather cross it for legal/social reasons. Similar to how too much political correctness will hamper the society, so does the overly done sensitivity in an agent, be it a human, or AI.

That might be the case, but how and who determines how much is too much? I mean in the case of AI, let the market decide seems like the right answer.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#34

Post by verdverm »

whimsicalism wrote: post #32
verdverm wrote: post #27

Earlier quoted context omitted.

There are many more features that would be needed, such as a peer comment pointed out, being able to recognize you are saying something incorrect, pausing, and then starting a new stream of output. This is currently done with multiple LLMs and calls, not within the running of a single model i/o Another example would be to input a single token or gibberish, the models we have today are more than happy to spit out fant…

i don’t see why any of the things you’re describing are criteria for thinking, it seems just arbitrarily picking things humans do and saying this is somehow constitutive to thought

It's more to point out how far the LLMs we have today are from anything that ought to be considered thoughts. They are far more mechanical than anything else

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#35

Post by whimsicalism »

verdverm wrote: post #34
whimsicalism wrote: post #32

Earlier quoted context omitted.

i don’t see why any of the things you’re describing are criteria for thinking, it seems just arbitrarily picking things humans do and saying this is somehow constitutive to thought

It's more to point out how far the LLMs we have today are from anything that ought to be considered thoughts. They are far more mechanical than anything else

you’re just retreating into tautologies - my question was why these are the criteria for thought

it’s fine though, this was as productive as i expected

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#36

Post by HarHarVeryFunny »

byteknight wrote: post #5
optimalsolver wrote: post #3

>what the model is "thinking" before writing its response An actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history. An LLM is doing exactly nothing while waiting for the next prompt.

I disagree with this. That suggests that thinking requires persistent, malleable and non-static memory. That is not the case. You can reasonably reason about without increasing knowledge if you have a base set of logic. I think the thing you were looking for was more along the lines of a persistent autonomous agent.

Sure you can reason over a fixed "base set of logic", although there's another word for that - an expert system with a fixed set of rules, which IMO is really the right way to view an LLM.

Still, what current LLMs are doing with their fixed rules is only a very limited form of reasoning since they just use a fixed N-steps of rule application to generate each word. People are looking to techniques such "group of experts" prompting to improve reasoning - step-wise generate multiple responses then evaluate them and proceed to next step.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#37

Post by nathan_compton »

byteknight wrote: post #9
nathan_compton wrote: post #6

Earlier quoted context omitted.

Counterpoint: "What if you looked at it this way?" communicates both your suggestion AND your sensitivity to the person's social status whatever. Given that humans are not robots, but social, psychological, animals, such communication is entirely justified and efficient.

You can't always do both to the fullest truth. They often conflict. To do what you suggest, would imply my feelings perfectly align with the sympathetic view. That is not the case for a lot of humans or instances. If I am not saying exactly how I feel it is watered down. And telling me "just do both" is enforcing your world view and that is precisely what we're talking about _not_ doing.

Its rarely the case that speaking without considering other people's feelings will be the optimal method of getting what you want, even if you are a sociopath and what you want doesn't have anything to do with other people's well beings. In fact, sociopaths are a great example: they are typically quite adept at communicating in such a way as to ingratiate themselves with others. If even a sociopath gets this, then you might want to consider the wisdom of following suit.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#38

Post by whimsicalism »

phyalow wrote: post #31
whimsicalism wrote: post #11

I continue to be impressed by Anthropic’s work and their dual commitment to scaling and safety. HN is often characterized by a very negative tone related to any of these developments, but I really do feel that Anthropic is trying to do a “race to the top” in terms of alignment, though it doesn’t seem like all the other major companies are doing enough to race with them. Particularly frustrating on HN is the common sy…

Alot of this really isnt new, Andrej Karpathy covered the principles here 8 years ago for CS231n at Stanford https://youtu.be/yCC09vCHzF8&t=1640

neural probing has been around for a while, true - and this result is definitely building on past results. it’s basically just a scaled up version of their paper from a little while ago anywho

but Karpathy was looking at very simple LSTMs of 1-3 layers, looking at individual nodes/cells, and these results have generally thus far been difficult to replicate among large scale transformers. Karpathy also doesn’t provide a recipe for doing this in his paper, which makes me think he was just guess and checking various cells. The representations discovered are very simple

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#39

Post by youssefabdelm »

For anyone who has read the paper, have they provided code examples or enough detail to recreate this with, say, Llama 3?

While they're concerned with safety, I'm much more interested in this as a tool for controllability. Maybe we can finally get rid of the woke customer service tone, and get AI to be more eclectic and informative, and less watered down in its responses.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#40

Post by whimsicalism »

HarHarVeryFunny wrote: post #36
byteknight wrote: post #5

Earlier quoted context omitted.

I disagree with this. That suggests that thinking requires persistent, malleable and non-static memory. That is not the case. You can reasonably reason about without increasing knowledge if you have a base set of logic. I think the thing you were looking for was more along the lines of a persistent autonomous agent.

Sure you can reason over a fixed "base set of logic", although there's another word for that - an expert system with a fixed set of rules, which IMO is really the right way to view an LLM. Still, what current LLMs are doing with their fixed rules is only a very limited form of reasoning since they just use a fixed N-steps of rule application to generate each word. People are looking to techniques such "group of exper…

if you zoom in enough, all thinking is an expert system with a fixed set of rules.
Post reply on HN