Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

transformer-circuits.pub

Post reply on HN
Print view View original

71–80 of 128 posts

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#71

Post by Nevermark »

HarHarVeryFunny wrote: post #66
Nevermark wrote: post #61

Earlier quoted context omitted.

An LLM has no explicit goals. But simply by approximating human communication which often models goal oriented behavior, an LLM can have implicit goals. Which likely vary widely according to conversation context. Implicit goals can be very effective. Nowhere in DNA is there any explicit goal to survive. However combinations of genes and markers selected for survivability create creatures with implicit goals to surviv…

Yes, the short term behavior/output of the LLM could reflect an implicit goal, but I doubt it'd maintain any such goal for an extended period of time (long-term coherence of behavior is a known shortcoming), since there is random sampling being done, and no internal memory from word to word - it seems that any implicit goal will likely rapidly drift.

Agreed. They can’t accurately model our communication for very long. So any implicit motives are limited to that.

But their capabilities are improving rapidly in both kind and measure.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#72

Post by HarHarVeryFunny »

whimsicalism wrote: post #70
HarHarVeryFunny wrote: post #64

Earlier quoted context omitted.

Well, not exactly a loop. They get to "extend the thought", but there is zero continuity from one word to the next (LLM starts from scratch for each token generated). The effect is as if you had multiple people playing a game where they each extend a sentence by taking turns adding a word to it, but there is zero continuity from one word to the next because each person is starting from scratch when it is their turn.

> LLM starts from scratch for each token generated What do you mean? They get to access their previous hidden states in the next greedy decode using attention, it is not simply starting from scratch. They can access exactly what they were thinking when they put out the previous word, not just reasoning from the word itself.

There's the KV cache kept from one word to the next, but isn't that just an optimization ?

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#73

Post by bjterry »

It would be interesting to allow users of models to customize inference by tweaking these features, sort of like a semantic equalizer for LLMs. My guess is that this wouldn't work as well as fine-tuning, since that would tweak all the features at once toward your use case, but the equalizer would require zero training data.

The prompt itself can trigger the features, so if you say "Try to weave in mentions of San Francisco" the San Francisco feature will be more activated in the response. But having a global equalizer could reduce drift as the conversation continued, perhaps?

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#74

Post by whimsicalism »

HarHarVeryFunny wrote: post #72
whimsicalism wrote: post #70

Earlier quoted context omitted.

> LLM starts from scratch for each token generated What do you mean? They get to access their previous hidden states in the next greedy decode using attention, it is not simply starting from scratch. They can access exactly what they were thinking when they put out the previous word, not just reasoning from the word itself.

There's the KV cache kept from one word to the next, but isn't that just an optimization ?

Yes, the 'KV cache' (imo an invented novelty, everyone was doing this before they came up with a term to make it sound cool) is an optimization so that you don't have to recompute what the model was thinking when it was generating all the prior words every time you decode a new word.

But that's exactly what I'm saying - the model has access to what it was thinking when it generated the previous words, it does not start from scratch. If you don't have the KV cache, you still have to regenerate what it was thinking from the previous words so on the next word generation you can look back at what you were thinking from the previous words. Does that make sense? I'm not great at talking about this stuff in words

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#76

Post by parentheses »

It's interesting that they used this to manipulate models. I wonder if "intentions" can be found and tuned. That would have massive potential for use and misuse. I could imagine a villain taking a model and amplifying "the evil" using a similar technique.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#77

Post by ben_w »

kalkin wrote: post #44
phyalow wrote: post #31

Earlier quoted context omitted.

Alot of this really isnt new, Andrej Karpathy covered the principles here 8 years ago for CS231n at Stanford https://youtu.be/yCC09vCHzF8&t=1640

This is an illustrative comment for meta reasons, I think. Karpathy's lecture almost certainly doesn't cover the superposition hypothesis (which hadn't been invented for ANNs 8 years ago), or sparse dictionary learning (whose application to ANNs is motivated by the superposition hypothesis). It certainly doesn't talk about actual specific features found in post-ChatGPT language models. What's happening here seems lik…

> I worry this is going to come across as insulting, but that's not my intention. I do this too sometimes; I think everyone does. The point is we shouldn't define true reasoning so narrowly that we think no system capable of it would ever be caught doing what most of us are in fact doing most of the time.

Indeed; to me LLMs pattern match (yes, I did spot the irony) to system-1 thinking, and they do a better job of that than we humans do.

Fortunately for all of us, they're no good at doing system-2 thinking themselves, and only mediocre at translating problems into a form which can be used by a formal logic system that excels at system-2 thinking.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#78

Post by ben_w »

handwarmers wrote: post #28
whimsicalism wrote: post #11

I continue to be impressed by Anthropic’s work and their dual commitment to scaling and safety. HN is often characterized by a very negative tone related to any of these developments, but I really do feel that Anthropic is trying to do a “race to the top” in terms of alignment, though it doesn’t seem like all the other major companies are doing enough to race with them. Particularly frustrating on HN is the common sy…

What is often frustrating to me at least is the arbitrary definition of "safety" and "ethics", forged by a small group of seemingly intellectually homogenous individuals.

Yes, even though this is a mild improvement on 20 years ago when it was an even more homogenous group.

Given how often China comes up in the context of AI, I'm wondering: Lots of people in the West treat China as mysterious and alien. I wonder how true that really is (e.g. Confucianism)? Or if it ever was (e.g. perhaps it used to be before industrialisation, which homogenises everyone regardless of the origin)?

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#79

Post by parentheses »

I wonder how interpretability and training can interplay. Some examples:

Imagine taking Claude, tweaking weights relevant to X and then fine tuning it on knowledge related to X. It could result in more neurons being recruited to learn about X.

Imagine performing this during training to amplify or reduce the importance of certain topics. Train it on a vast corpus, but tune at various checkpoints to ensure the neural network's knowledge distribution skews. This could be a way to get more performance from MoE models.

I am not an expert. Just putting on my generalist hat here. Tell me I'm wrong because I'd be fascinated to hear the reasons.

Re: Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

#80

Post by HarHarVeryFunny »

whimsicalism wrote: post #74
HarHarVeryFunny wrote: post #72

Earlier quoted context omitted.

There's the KV cache kept from one word to the next, but isn't that just an optimization ?

Yes, the 'KV cache' (imo an invented novelty, everyone was doing this before they came up with a term to make it sound cool) is an optimization so that you don't have to recompute what the model was thinking when it was generating all the prior words every time you decode a new word. But that's exactly what I'm saying - the model has access to what it was thinking when it generated the previous words, it does not sta…

I don't think you can really say it "regenerates" what it was thinking from the last prompt, since the new prompt is different from the previous one (it has the new word appended to the end, which may change the potential meanings of the sentence).

There will be some overlap in what the model is now "thinking" (and has calculated from scratch) since the new prompt is one possible continuation of the previous one, but other things it was previously "thinking" will no longer be there.

e.g. Say the prompt was "the man", and output probabilities include "in" and "ran", reflecting the model thinking of potential continuations such as "the man in the corner" and "the man ran for mayor". Suppose the word sampled was "ran", so now the new prompt is "the man ran". Possible continuations can no longer include refining who the subject is, since the new word "ran" implies the continuation must now be an action.

There is some work that has been saved, per the KV cache, in processing the new prompt, but that is only things (self attention among the common part of the two prompts) that would not change if recalculated. What the model is thinking has changed, and will continue to change depending on the next sampled continuation ("the man ran for mayor", "the man ran for cover", "the man ran his bath", etc).

Post reply on HN