Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Persona vectors: Monitoring and controlling character traits in language models

anthropic.com

Post reply on HN
Print view View original

51–60 of 143 posts

Re: Persona vectors: Monitoring and controlling character traits in language models

#51

Post by therein »

simianwords wrote: post #49
semitones wrote: post #35

Earlier quoted context omitted.

Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such

i don't think this is correct - such training data is usually made at SFT level after unsupervised learning on all available data in the web. the SFT level dataset is manually curated meaning there would be conscious effort to create more training samples of the form to say "i'm not sure". same with RLHF.

You mean I don't think this is automatically correct. Otherwise it very likely is correct. Either way, you're guessing the manual curation is done in a way that is favorable to include I don't know answers. Which it most likely doesn't.

Re: Persona vectors: Monitoring and controlling character traits in language models

#52

Post by vrotaru »

andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

To some degree *all* LLM's answers are made up facts. For stuff that is abundantly present in training data those are almost always correct. For topics which are not common knowledge (allow for a great variability) you should always check.

I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts".

Re: Persona vectors: Monitoring and controlling character traits in language models

#53

Post by Workaccount2 »

andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

>My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement.

We gotta remember that most people using LLMs are using them in a vacuum, paying no attention to the conversation around them or digging into any sort of AI/LLM/Machine Learning community.

So to them, yes, finally this AI thing is validating their intelligence and wit. It's a pretty slippery slope.

Re: Persona vectors: Monitoring and controlling character traits in language models

#54

Post by maaaaattttt »

tdtr wrote: post #44
devmor wrote: post #41

Earlier quoted context omitted.

That’s a really astute observation. It would be interesting if we could find a way to train models to signify when they are “stretching” the vector distance too far from the context window, because the available training data is too sparse or nonexistent. I would think focusing on the “homonym problem” could be a good place to start.

I'm pretty sure that the canonical choice is either choosing vectors to be anchor - either by a knn distance with other vectors, or by "hand", or even stuff like cross entropy - but then that is already in the loss function. another method would be to create some kind of adversarial setup where the output is "stretched" intentionally and then criticized by another llm. afaik the problem is with scale, as manually goi…

Maybe do something close to what I like to believe the brain does and have a meta model wrap a "base" model. The meta model gets the output data from the base model (edit: plus the original input) as input plus some meta parameters (for example the probability each token had when it was chosen and/or better which "neurons" were activated during the whole output sequence which would include the Persona they mention). It's then the meta model that generates new output data based on this input and this is the output that is shown to the user.

Re: Persona vectors: Monitoring and controlling character traits in language models

#55

Post by simianwords »

therein wrote: post #51
simianwords wrote: post #49

Earlier quoted context omitted.

i don't think this is correct - such training data is usually made at SFT level after unsupervised learning on all available data in the web. the SFT level dataset is manually curated meaning there would be conscious effort to create more training samples of the form to say "i'm not sure". same with RLHF.

You mean I don't think this is automatically correct. Otherwise it very likely is correct. Either way, you're guessing the manual curation is done in a way that is favorable to include I don't know answers. Which it most likely doesn't.

its completely in the incentive to include such examples in RLHF. or you have come up with a way to increase performance that the very employees haven't. why do you think they didn't try it?

Re: Persona vectors: Monitoring and controlling character traits in language models

#56

Post by mitjam »

roughly wrote: post #10

Like a lot of the research Anthropic has done, this and the “emergent misalignment” research they link to put more points in the “stochastic parrot” hypothesis column. The reason these LLM behaviors read as so weird to us is that we’re still anthropomorphizing the hell out of these systems - they can create very convincing dialogue, and the depth of the model suggests some surprising complexity, but the reason why, e…

> they lack some fundamental structuring that seems to be required to create anything like consistency or self-reflection

A valid observation. Interestingly, feeding the persona vectors detected during inference back into the context might be a novel way of self-reflection for LLMs.

Re: Persona vectors: Monitoring and controlling character traits in language models

#57

Post by Jimmc414 »

apwell23 wrote: post #47
kachapopopow wrote: post #37

Earlier quoted context omitted.

They can always statistically choose to end the conversation or say no.

chatgpt refused to produce an image of 'bald and fat computer programmer' for me and just refused any further requests from me for any image ( 'handsome computer programmer').

Were you using the free version?

https://chatgpt.com/share/688fb2e4-0efc-8001-8c9b-427dfa6784...

Re: Persona vectors: Monitoring and controlling character traits in language models

#58

Post by zer00eyz »

Workaccount2 wrote: post #53
andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

>My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. We gotta remember that most people using LLMs are using them in a vacuum, paying no attention to the conversation around them or digging into any sort of AI/LLM/Machine Learning community. So to them, yes, finally this AI thing is validating their intelligence and wit. It's a pret…

So yes this AI thing is finally validating my product idea that the engineers kept saying NO to.

It's not just that it wants to find a solution, it's not just validating, it very rarely says "no". Its not saying no to things that are, for lack of a better term, fucking dumb.

That doesn't mean the tools arent without merit. For code bases I use infrequently that are well documented AI is a boon to me as an engineer.

But "vibe coding" is the new dreamweaver. A lot of us made a lot of money cleaning up after. It's a good thing.

Re: Persona vectors: Monitoring and controlling character traits in language models

#59

Post by danenania »

andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

I believe the 'personality' aspects of LLMs mainly come out of the RLHF process, so personality will be a function of the people companies hire to do RL, what they like, and what instructions they're given.

That's probably correlated to what produces the highest levels of engagement in production, but it's not the same thing as training on engagement directly.

Re: Persona vectors: Monitoring and controlling character traits in language models

#60

Post by skylerwiernik »

> In 2023, Microsoft's Bing chatbot famously adopted an alter-ego called "Sydney,” which declared love for users and made threats of blackmail. More recently, xAI’s Grok chatbot would for a brief period sometimes identify as “MechaHitler” and make antisemitic comments. Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts.

Funny that they managed to call out all of their competitors without mentioning any of Claude's bad behavior

Post reply on HN