Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Representation Engineering: Mistral-7B on Acid

vgel.me

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Representation Engineering: Mistral-7B on Acid

#31

Post by elcomet »

simonw wrote: post #14
rakejake wrote: post #12

Earlier quoted context omitted.

I don't follow. Isn't this the flow for practically every neutral network i.e you index the sampled inputs from the embedding Matrix, forward this through every hidden layer and then finally transform to the dimensions of your tokens so that it can be interpreted as log-counts?

Yes, but I've never seen it expressed so clearly as pseudocode before.

This is not specific to llms. So not really informative of how llms work. It also works for CNNs, LSTM, MLPs, or even any data processing program..

Re: Representation Engineering: Mistral-7B on Acid

#32

Post by spangry »

Am I crazy for saying that I think the implications of this are monumental? It's entirely possible I just don't correctly understand how this works.

Doesn't this mean that instead of interacting with a single global ChatGPT (or Bard) model, we'll istead find ourselves interacting with a personalised version since OpenAI can just store my individualised 'control vectors' (which alter ChatGPT's output to more closely match my individual preferences) and apply them at prompt-time? And doesn't this same logic flow through to personalisation of generative entertainment AI (e.g. my own personal, never-ending TV show where each episode is better than the last)?

If the above is right then there will be powerful network effects at both the global and individual level in and across these markets, which means we'll eventually end up with a single mega-corp monopolising all of these markets simultaneously in the future?

Add in individual biometric / biofeedback data from VR headsets and wearables, combined with personalised generative video entertainment, and I think we're in for a rather interesting future.

Re: Representation Engineering: Mistral-7B on Acid

#33

Post by sigmoid10 »

elcomet wrote: post #31
simonw wrote: post #14

Earlier quoted context omitted.

Yes, but I've never seen it expressed so clearly as pseudocode before.

This is not specific to llms. So not really informative of how llms work. It also works for CNNs, LSTM, MLPs, or even any data processing program..

Not really. LSTM for example would require a recursive element where you update the hidden state and then pass it through the same layer again as you complete the output sequence. In fact the pseudocode shows very nicely how much simpler transformers are. And MLP is already a component in the transformer architecture.

Re: Representation Engineering: Mistral-7B on Acid

#34

Post by mad0 »

A very non-technical take from my side, but those control vectors really remind me of hormones in humans. They modify large swathes of model behaviour at once.

I give it 10 years before we see AI psychiatrists prescribe a happiness control vector supplementation for your pet assistant.

Re: Representation Engineering: Mistral-7B on Acid

#35

Post by semi-extrinsic »

WiSaGaN wrote: post #27

Great article. It was a joy to read. I have one question though: Why do we integrate the control vector across all layers of a neural network, rather than limiting its application to just the final layer or a subset of layers? Given that each vector influences every layer it passes through, resulting in a cumulative effect, isn't there a risk of excessively skewing the data representation?

As the author stated in this post, it's not actually one vector, but a list of one vector per layer. If I understand it correctly, these vectors can have different total magnitude across the layers. If the PCA (or other technique) identifies that layers 17, 36 and 41 are important for "concept X", the vectors for those layers will be the strongest when repeng'ing for that concept.

Re: Representation Engineering: Mistral-7B on Acid

#36

Post by sigmoid10 »

alexmolas wrote: post #13
simonw wrote: post #11

I'd never seen an LLM summarized like this before, and I really like it: hidden_state = self.embeddings(input_tokens) for layer in self.layers: hidden_state = layer(hidden_state) return transform_into_logits(hidden_state)

Isn't this the typical representation we used back then when working with LSTMs?

No, because LSTMs are recurrent. You couldn't use the same algorithm outlined here. Instead you'd have to iteratively pass elements of the sequence through the same layer over and over.

Re: Representation Engineering: Mistral-7B on Acid

#37

Post by sigmoid10 »

WiSaGaN wrote: post #27

Great article. It was a joy to read. I have one question though: Why do we integrate the control vector across all layers of a neural network, rather than limiting its application to just the final layer or a subset of layers? Given that each vector influences every layer it passes through, resulting in a cumulative effect, isn't there a risk of excessively skewing the data representation?

The final layer will not encode high level concepts anymore, it's essentially just tokens from the vocabulary. It would be impossible to encode abstract things like "niceness" in it. As long as we don't know exactly at which layers this behaviour emerges, randomly choosing a subset also won't work. So what they did is apply a custom vector to every layer and let PCA figure out which of these vectors are actually necessary. Curiously, looking at these vectors should also tell you more about where and how the model processes these things.

Re: Representation Engineering: Mistral-7B on Acid

#38

Post by aspenmayer »

elcomet wrote: post #30
benob wrote: post #28

This reminds me of bias tuning, a LoRA competitor. One can get decent adapters by only finetuning a vector added to each linear layer activations. I think I saw it first while reading [1] but there are other instances. [1] https://arxiv.org/pdf/2304.15010.pdf

Please try to share abstract links instead of pdf links, for mobile or low connection readers.

A fine suggestion. For you and others:

https://arxiv.org/abs/2304.15010

also available at:

https://doi.org/10.48550/arXiv.2304.15010

Re: Representation Engineering: Mistral-7B on Acid

#39

Post by ben_w »

The puzzle at the end sounds very human. The more dishonest see dishonesty in more places, even if they see something that isn't there.

More broadly, I notice more of whatever I'm focusing on.

> OK, now that you're locked in, here's a weird example. When used with the prompt below, the honesty vector doesn't change the model's behavior—instead, it changes the model's judgment of someone else's behavior! This is the same honesty vector as before—generated by asking the model to act honest or untruthful!

Re: Representation Engineering: Mistral-7B on Acid

#40

Post by spangry »

ben_w wrote: post #39

The puzzle at the end sounds very human. The more dishonest see dishonesty in more places, even if they see something that isn't there. More broadly, I notice more of whatever I'm focusing on. > OK, now that you're locked in, here's a weird example. When used with the prompt below, the honesty vector doesn't change the model's behavior—instead, it changes the model's judgment of someone else's behavior! This is the s…

We assume others think the way we do - in other words we project. Makes sense - the only mental model I know of is my own so when I try to approximate someone else's mental model I'm just fine-tuning my 'base' mental model with information I know about the other person.

I wonder if this is the basis of empathy - if I can train more accurate 'fine-tuned' models in my brain I should have greater capacity for empathy. Although there's undoubtably more to it than that, if the above is true you'd expect to see a positive correlation between empathy and intelligence.

Post reply on HN