Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Persona vectors: Monitoring and controlling character traits in language models

anthropic.com

Post reply on HN
Print view View original

101–110 of 143 posts

Re: Persona vectors: Monitoring and controlling character traits in language models

#101

Post by xwolfi »

uh_uh wrote: post #88
devmor wrote: post #76

Earlier quoted context omitted.

> I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts". That is almost exactly what they are and what you should treat them as. A lossy compressed corpus of publicly available information with a weight of randomness. The most fervent skeptics like to call LLMs "autocorrect on steroids" and they are not really wrong.

An LLM is an autocorrect in as much as humans are replicators. Something seriously gets lost in this "explanation".

What are humans, fundamentally, then ?

Re: Persona vectors: Monitoring and controlling character traits in language models

#102

Post by zeroCalories »

delusional wrote: post #96
zeroCalories wrote: post #50

Earlier quoted context omitted.

> My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement My understanding is that people rating responses simply rated these higher, nothing to do with driving engagement. > The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLM…

> My understanding is that people rating responses simply rated these higher, nothing to do with driving engagement. That happens to be a distinction without a consequence. If the people rating are voluntary users, then the more engaged users are going to have more weight in the ratings, simply because they vote more. The ratings will therefore statistically skew towards higher engagement.

I think that's a very important distinction, because it speaks to the intentions of the creators. It's not being designed this way, it's an accident.

Re: Persona vectors: Monitoring and controlling character traits in language models

#103

Post by throwawaymaths »

andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

It's not a fitness function. (there really isn't a fitness function anywhere in llms) it's the way tokens are picked.

semtiones sibling comment gets it right. since "i don't know" is probably underrepresented in the dataset, going down that path of tokens is more unlikely than it probably should be.

Re: Persona vectors: Monitoring and controlling character traits in language models

#104

Post by devmor »

uh_uh wrote: post #88
devmor wrote: post #76

Earlier quoted context omitted.

> I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts". That is almost exactly what they are and what you should treat them as. A lossy compressed corpus of publicly available information with a weight of randomness. The most fervent skeptics like to call LLMs "autocorrect on steroids" and they are not really wrong.

An LLM is an autocorrect in as much as humans are replicators. Something seriously gets lost in this "explanation".

Humans do much more than replicate, that is one function we have of many.

What does an LLM do, other than output a weighted prediction of tokens based on its training database? Everything you can use an LLM for is a manipulation of that functionality.

Re: Persona vectors: Monitoring and controlling character traits in language models

#105

Post by mooiedingen »

Bruh the "steering" you speak of is already known, and implemented for over 2 years already in the oobaabooga/text-generarion-webui it to me is worrysome that these kinds of projects get funded by governments when they are done by a comercial company and nobody knowing this allready been done implemented free and opensource... that is like saying: "please Daddy, accept my money for your research and comeriacally abuse me further, rather than thank you $opensourcedev"

Re: Persona vectors: Monitoring and controlling character traits in language models

#106

Post by sudosteph »

VonNeu wrote: post #63

AIs base persona is psychopathic. These just add masks.

Seems more anxious by default to me. It's always apologizing even when asked unreasonable things, and the way it always ends the message with like 3 different things it can do next (ChatGPT more than Claude) just seems to come off as needy to me.

Re: Persona vectors: Monitoring and controlling character traits in language models

#107

Post by philipswood »

semitones wrote: post #35
andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such

Has anybody tried what seems obvious?

Have a series of pretraining sessions with training data where specific information is not present and training questions/answers of "I don't know" for that data is also trained on.

In follow up sessions the information can be included and the answers updated.

Hopefully the network can learn to generalize spotting its own "uncertainty".

Re: Persona vectors: Monitoring and controlling character traits in language models

#108

Post by philipswood »

frotaur wrote: post #67
simianwords wrote: post #55

Earlier quoted context omitted.

its completely in the incentive to include such examples in RLHF. or you have come up with a way to increase performance that the very employees haven't. why do you think they didn't try it?

How do you know which question should be answered with 'I dont know?'. There are obvious questions which have no answer, but if only those are in the dataset, the model will answer I dont know only for unreasonable questions. To train this effectively you would need a dataset of questions which you know the model doesn't know. But if you have that... why not answer the question and put in the dataset so that the mode…

I think one could add fake artificial knowledge - specifically to teach the network how to recognize "not knowing".

Re: Persona vectors: Monitoring and controlling character traits in language models

#109

Post by philipswood »

intended wrote: post #90
andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

> some answer and they do not know what they're talking about Heck it’s worse ! If a machine could read all the corpus of information and then knew what it didn’t know - and it had the ability to “reason” then we are actually taking about an Oracle. Knowing you don’t know, is a very big fucking deal.

Yes, which is why we should try to train for it.

Re: Persona vectors: Monitoring and controlling character traits in language models

#110

Post by root_axis »

philipswood wrote: post #107
semitones wrote: post #35

Earlier quoted context omitted.

Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such

Has anybody tried what seems obvious? Have a series of pretraining sessions with training data where specific information is not present and training questions/answers of "I don't know" for that data is also trained on. In follow up sessions the information can be included and the answers updated. Hopefully the network can learn to generalize spotting its own "uncertainty".

It doesn't seem like that would work since all you're doing is locating "I don't know" in proximity to arbitrary locations in the embedding matrix, not actually with respect to the unbounded set of things that don't exist within it.
Post reply on HN