Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

transformer-circuits.pub

Post reply on HN
Print view View original

41–50 of 128 posts

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#41

Post by abecedarius »

whimsicalism wrote: post #16
optimalsolver wrote: post #3

>what the model is "thinking" before writing its response An actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history. An LLM is doing exactly nothing while waiting for the next prompt.

If we figured out how to freeze and then revive brains, would that mean that all of the revived brains were no longer thinking because they had previously been paused at some point? Frankly this objection seems very weak

Yeah. I've had like three conversations with people who said LLMs don't "think", implied this was too obvious to need to say why, and when pressed on it brought up the pausing as their first justification.

It's an interesting window on people's intuitions -- this pattern felt surprising and alien now to someone who imbibed Hofstadter and Dennett, etc., as a teen in the 80s.

(TBC, the surprise was not that people weren't sure they "think" or are "conscious", it's that they were sure they aren't, on this basis that the program is not running continually.)

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#42

Post by byteknight »

whimsicalism wrote: post #40
HarHarVeryFunny wrote: post #36

Earlier quoted context omitted.

Sure you can reason over a fixed "base set of logic", although there's another word for that - an expert system with a fixed set of rules, which IMO is really the right way to view an LLM. Still, what current LLMs are doing with their fixed rules is only a very limited form of reasoning since they just use a fixed N-steps of rule application to generate each word. People are looking to techniques such "group of exper…

if you zoom in enough, all thinking is an expert system with a fixed set of rules.

Exactly. You can't reason with that you do not currently posses.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#43

Post by verdverm »

whimsicalism wrote: post #35
verdverm wrote: post #34

Earlier quoted context omitted.

It's more to point out how far the LLMs we have today are from anything that ought to be considered thoughts. They are far more mechanical than anything else

you’re just retreating into tautologies - my question was why these are the criteria for thought it’s fine though, this was as productive as i expected

I'm not listing criteria for thought

I'm listing things that current LLMs cannot do (or things they do that thinking entities would not) to argue they are so simple they are far from anything that resembles thinking

> it’s fine though, this was as productive as i expected

A product of your replies becoming lowering in quality, and becoming more argumentative, so I will discontinue now

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#44

Post by kalkin »

phyalow wrote: post #31
whimsicalism wrote: post #11

I continue to be impressed by Anthropic’s work and their dual commitment to scaling and safety. HN is often characterized by a very negative tone related to any of these developments, but I really do feel that Anthropic is trying to do a “race to the top” in terms of alignment, though it doesn’t seem like all the other major companies are doing enough to race with them. Particularly frustrating on HN is the common sy…

Alot of this really isnt new, Andrej Karpathy covered the principles here 8 years ago for CS231n at Stanford https://youtu.be/yCC09vCHzF8&t=1640

This is an illustrative comment for meta reasons, I think. Karpathy's lecture almost certainly doesn't cover the superposition hypothesis (which hadn't been invented for ANNs 8 years ago), or sparse dictionary learning (whose application to ANNs is motivated by the superposition hypothesis). It certainly doesn't talk about actual specific features found in post-ChatGPT language models. What's happening here seems like a thing LLMs are often accused of dismissively - you're pattern-matching to certain associated words without really reasoning about what is or isn't new in this paper.

I worry this is going to come across as insulting, but that's not my intention. I do this too sometimes; I think everyone does. The point is we shouldn't define true reasoning so narrowly that we think no system capable of it would ever be caught doing what most of us are in fact doing most of the time.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#45

Post by wwarner »

verdverm wrote: post #24
wwarner wrote: post #20

huge. the activation scan, which looks for which nodes change the most when prompted with the words "Golden Gate Bridge" and later an image of the same bridge, is eerily reminiscent of a brain scan under similar prompts...

I find this outcome expected and not really surprising, more confirmation of previous results. Consider vision transformers and the papers that showed what each layer was focused on.

well that's exactly the point -- no such result is available for language models.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#46

Post by verdverm »

wwarner wrote: post #45
verdverm wrote: post #24

Earlier quoted context omitted.

I find this outcome expected and not really surprising, more confirmation of previous results. Consider vision transformers and the papers that showed what each layer was focused on.

well that's exactly the point -- no such result is available for language models.

There are multiple papers and efforts that have inspected the internal state of LLMs. One could even see the word2vec analysis along these lines, as evidence that the model is specializing neurons

One such example: The Internal State of an LLM Knows When It's Lying (https://arxiv.org/abs/2304.13734)

Searching phrases like "llm interpretability" and "llm activation analysis" uncover more

https://github.com/JShollaj/awesome-llm-interpretability

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#47

Post by verdverm »

byteknight wrote: post #42
whimsicalism wrote: post #40

Earlier quoted context omitted.

if you zoom in enough, all thinking is an expert system with a fixed set of rules.

Exactly. You can't reason with that you do not currently posses.

How does scientific progress happen without reasoning about that which we do not know or understand?

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#48

Post by tantalor »

I always assumed the way to map these models would be by ablation, the same way we map the animal brain.

Damage part X of the network and see what happens. If the subject loses the ability to do Y, then X is responsible for Y.

See https://en.wikipedia.org/wiki/Phineas_Gage

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#49

Post by wwarner »

verdverm wrote: post #46
wwarner wrote: post #45

Earlier quoted context omitted.

well that's exactly the point -- no such result is available for language models.

There are multiple papers and efforts that have inspected the internal state of LLMs. One could even see the word2vec analysis along these lines, as evidence that the model is specializing neurons One such example: The Internal State of an LLM Knows When It's Lying ( https://arxiv.org/abs/2304.13734 ) Searching phrases like "llm interpretability" and "llm activation analysis" uncover more https://github.com/JShollaj/…

Yes, lots of activity in the space. I thought you were saying it was a dumb problem, but I was wrong.

I think this is a great paper.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#50

Post by soco »

kalkin wrote: post #44
phyalow wrote: post #31

Earlier quoted context omitted.

Alot of this really isnt new, Andrej Karpathy covered the principles here 8 years ago for CS231n at Stanford https://youtu.be/yCC09vCHzF8&t=1640

This is an illustrative comment for meta reasons, I think. Karpathy's lecture almost certainly doesn't cover the superposition hypothesis (which hadn't been invented for ANNs 8 years ago), or sparse dictionary learning (whose application to ANNs is motivated by the superposition hypothesis). It certainly doesn't talk about actual specific features found in post-ChatGPT language models. What's happening here seems lik…

By that reasoning even humans are not thinking. But of course humans are always excluded from such research - if it's human it's thinking by default, damn the reasoning. Then of course we have snails and dogs and apes, are they thinking? Were the Neanderthals thinking? By which definition? Moving posts is a too weak metaphor for what is going on here where everybody distorts the reasoning for whatever point they're trying to make today. And because I can't shut up I'll just add my user view: if it works like a duck and outputs like a duck, it's duck enough for any practical use, let's move on and see what do we do with it (like, use or harness or adopt or...).
Post reply on HN