Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

transformer-circuits.pub

Post reply on HN
Print view View original

51–60 of 128 posts

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#51

Post by sjkoelle »

pdevr wrote: post #22

So, to summarize: >Used "dictionary learning" >Found abstract features >Found similar/close features using distance >Tried amplifying and suppressing features Not trying to be snary, but sounds mundane in the ML/LLM world. Then again, significant advances have come from simple concepts. Would love to hear from someone who has been able to try this out.

the interesting advance in the anthropic/mats research program is the application of dictionary learning to the "superpositioned" latent representations of transformers to find more "interpretable" features. however, "interpretability" is generally scored by the explainer/interpreter paradigm which is a bit ad hoc, and true automated circuit discovery (rather than simple concept representation) is still a bit off afaik.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#52

Post by HarHarVeryFunny »

whimsicalism wrote: post #40
HarHarVeryFunny wrote: post #36

Earlier quoted context omitted.

Sure you can reason over a fixed "base set of logic", although there's another word for that - an expert system with a fixed set of rules, which IMO is really the right way to view an LLM. Still, what current LLMs are doing with their fixed rules is only a very limited form of reasoning since they just use a fixed N-steps of rule application to generate each word. People are looking to techniques such "group of exper…

if you zoom in enough, all thinking is an expert system with a fixed set of rules.

That the basis of it, but in our brain the "inference engine" using those rules is a lot more than a fixed N-steps - there is thalamo-cortical looping, working memory of various durations, and maybe a bunch of other mechanisms such as analogical recall, resonance-based winner-takes-all processing, etc, etc.

Current LLMs have none of that - they are just the fixed set of rules, further limited by also having a fixed number of steps of rule application.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#53

Post by HarHarVeryFunny »

byteknight wrote: post #42
whimsicalism wrote: post #40

Earlier quoted context omitted.

if you zoom in enough, all thinking is an expert system with a fixed set of rules.

Exactly. You can't reason with that you do not currently posses.

Sure, but you (a person, not an LLM) can also reason about what you don't possess, which is one of our primary learning mechanisms - curiosity driven by lack of knowledge causing us to explore and acquire new knowledge by physical and/or mental exploration.

An LLM has no innate traits such as curiosity or boredom to trigger exploration, and anyways no online/incremental learning mechanism to benefit from it even if it did.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#54

Post by davedx »

tel wrote: post #17

This is exceptionally cool. Not only is it very interesting to see how this can be used to better understand and shape LLM behavior, I can’t help but also think it’s an interesting roadmap to human anthropology. If we see LLMs as substantial compressed representations of human knowledge/thought/speech/expression—and within that, a representation of the world around us—then dictionary concepts that meaningfully explai…

The vector space projection of the human experience. I like it.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#55

Post by verdverm »

wwarner wrote: post #49
verdverm wrote: post #46

Earlier quoted context omitted.

There are multiple papers and efforts that have inspected the internal state of LLMs. One could even see the word2vec analysis along these lines, as evidence that the model is specializing neurons One such example: The Internal State of an LLM Knows When It's Lying ( https://arxiv.org/abs/2304.13734 ) Searching phrases like "llm interpretability" and "llm activation analysis" uncover more https://github.com/JShollaj/…

Yes, lots of activity in the space. I thought you were saying it was a dumb problem, but I was wrong. I think this is a great paper.

yup, if you look at drop out, what it does and why, you can see additional interesting results along these lines

(drop-out was found to increase resilience in models because they had to encode information in the weights differently, i.e. could not rely on single neuron (at the limit))

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#57

Post by bilsbie »

If anyone wants to team up and work on stuff like this (on toy models so we can run locally) please get in touch. (Email in profile)

I’m so fascinated by this stuff but I’m having trouble staying motivated in this short attention span world.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#59

Post by xtiansimon »

soco wrote: post #50
kalkin wrote: post #44

Earlier quoted context omitted.

This is an illustrative comment for meta reasons, I think. Karpathy's lecture almost certainly doesn't cover the superposition hypothesis (which hadn't been invented for ANNs 8 years ago), or sparse dictionary learning (whose application to ANNs is motivated by the superposition hypothesis). It certainly doesn't talk about actual specific features found in post-ChatGPT language models. What's happening here seems lik…

By that reasoning even humans are not thinking. But of course humans are always excluded from such research - if it's human it's thinking by default, damn the reasoning. Then of course we have snails and dogs and apes, are they thinking? Were the Neanderthals thinking? By which definition? Moving posts is a too weak metaphor for what is going on here where everybody distorts the reasoning for whatever point they're t…

> “By that reasoning even humans are not thinking”

I’m a neophyte, so take this as such. If we can agree that people output is not always the product of thinking, then I’d be more willing to accept computational innovations as thought-like.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#60

Post by rmorey »

tantalor wrote: post #48

I always assumed the way to map these models would be by ablation, the same way we map the animal brain. Damage part X of the network and see what happens. If the subject loses the ability to do Y, then X is responsible for Y. See https://en.wikipedia.org/wiki/Phineas_Gage

We are so far ahead in the case of these models - we already have the complete wiring diagram! In biological systems we have only just begun to be able to create the complete neuronal wiring diagrams - currently worms, flies, perhaps soon mice
Post reply on HN