Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Persona vectors: Monitoring and controlling character traits in language models

anthropic.com

Post reply on HN
Print view View original

41–50 of 143 posts

Re: Persona vectors: Monitoring and controlling character traits in language models

#41

Post by devmor »

semitones wrote: post #35
andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such

That’s a really astute observation. It would be interesting if we could find a way to train models to signify when they are “stretching” the vector distance too far from the context window, because the available training data is too sparse or nonexistent.

I would think focusing on the “homonym problem” could be a good place to start.

Re: Persona vectors: Monitoring and controlling character traits in language models

#43

Post by devmor »

vessenes wrote: post #25
Illniyar wrote: post #21

I can see this working with "evil" and "sycophantic" personas. These seem like traits that would be amenable to input and thus be detectable by manipulating the input. But hallucination is an inherent property of LLMs - you cannot make it hallucinate less by telling it to not hallucinate or hallucinate more by telling it to make facts up (because if you tell it to make stuff up and it does, it's not hallucinating, it…

Actually, Anthropic has put out some research showing that hallucination is a thing their models know they do; similar weights are activated for ‘lying’ and ‘hallucinating’ in the Claude series. Implication - Claude knows - at least mostly - when its hallucinating. I think the current state of the art is that hallucination is at least partly a bug created by the very nature of training — you’re supposed to at least p…

> Claude knows - at least mostly - when its hallucinating.

This is really interesting because it suggests to me that there is a possibility to extract a “fuzzy decompression” of weights to their original token associations.

Re: Persona vectors: Monitoring and controlling character traits in language models

#44

Post by tdtr »

devmor wrote: post #41
semitones wrote: post #35

Earlier quoted context omitted.

Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such

That’s a really astute observation. It would be interesting if we could find a way to train models to signify when they are “stretching” the vector distance too far from the context window, because the available training data is too sparse or nonexistent. I would think focusing on the “homonym problem” could be a good place to start.

I'm pretty sure that the canonical choice is either choosing vectors to be anchor - either by a knn distance with other vectors, or by "hand", or even stuff like cross entropy - but then that is already in the loss function. another method would be to create some kind of adversarial setup where the output is "stretched" intentionally and then criticized by another llm. afaik the problem is with scale, as manually going through a bunch of vectors to just ground the latent isnt exactly economical. also people are quite conservative, esp in the big model runs - stuff like muon isnt exactly popularized till the new qwen or kimi. obviously this is all speculation for open models and folks with more experience can chime in.

Re: Persona vectors: Monitoring and controlling character traits in language models

#45

Post by atoav »

optimalsolver wrote: post #33
andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

>like when models start sucking up to users or making up facts That's the default mode of LLMs.

As someone somewhat critical of LLMs, this is not quite correct. It is a true observation thwt any popular chatbots have a system prompt that give the resulting answers a certain yes-man quality. But that is not necessarily so. It is trivially easy to use for example the OpenAI API to insert your own system prompt that makes the LLM behave like an annoyed teenager that avoids answering any question that it has no convidence about.

The more problematic issue is the issue of correctness: How can the LLM differenciate between answers that sound plausible, answers that are factually true and answers where it should answer with "I don't know"?

The issue might not be resolvable at all. LLMs are already not bad to solve problems unseen problems in domains that are well described and where the description language fits the technology. But there are other domains where it is catastrophically wrong, e.g. I had students come with an electronics proposal where the LLM misrepresented the relationship between cable gauge, resistance and heat in exactly the opposite way of what is true. Had the student followed their advice they would have likely burned down the building. Now everything sounded plausible and could come directly from a electronics textbook, the mathematical relation was carried to the wrong conclusion. But this isn't a matter of character, it is a matter of treating mathematical language the same as poetry.

Re: Persona vectors: Monitoring and controlling character traits in language models

#46

Post by jamienk »

FergusArgyll wrote: post #15
ctoth wrote: post #4

Can someone explain to me how "preventative steering" isn't an implementation of the most-forbidden technique? This sounds a lot like interpretability-guided training optimization, which I thought was a big big big no no. It will still introduce optimization pressure no? My understanding is that you shouldn't use insights gained from interpretability to feed back into your training process at risk of losing the inter…

For ref https://thezvi.substack.com/p/the-most-forbidden-technique/

How does this specifically work? Wouldn't any decision about what training data to use be part of a "technique" in this sense? When Stable Diffusion didn't train on porn.

OTOH if the majority of your data is "bad" (maybe morally, but maybe not, maybe you are feeding in too much gibberish), won't that pollute your model?

You notice that X keeps telling you a WRONG physics equation. So, rather than "correct" it, you keep training until you see the output giving the RIGHT equation?

How could you know (in, say 1899) if the WRONG output wasn't quantum and the RIGHT output was classical?

I'm not sure I'm understand the distinctions here. In all cases, we are relying on the idea that it is easy to know what should count as "right"?

Re: Persona vectors: Monitoring and controlling character traits in language models

#47

Post by apwell23 »

kachapopopow wrote: post #37
andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

They can always statistically choose to end the conversation or say no.

chatgpt refused to produce an image of 'bald and fat computer programmer' for me and just refused any further requests from me for any image ( 'handsome computer programmer').

Re: Persona vectors: Monitoring and controlling character traits in language models

#49

Post by simianwords »

semitones wrote: post #35
andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such

i don't think this is correct - such training data is usually made at SFT level after unsupervised learning on all available data in the web. the SFT level dataset is manually curated meaning there would be conscious effort to create more training samples of the form to say "i'm not sure". same with RLHF.

Re: Persona vectors: Monitoring and controlling character traits in language models

#50

Post by zeroCalories »

andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

> My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement

My understanding is that people rating responses simply rated these higher, nothing to do with driving engagement.

> The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text based on statistics.

It seems like you could perfectly describe this using personality. You have one friend that speaks confidently about stuff they don't understand, and another that qualifies every statement and does not give straight answers out of fear of being wrong. Again, this dysfunction could be attributed to what users rate higher.

Post reply on HN