Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Persona vectors: Monitoring and controlling character traits in language models

anthropic.com

Post reply on HN
Print view View original

121–130 of 143 posts

Re: Persona vectors: Monitoring and controlling character traits in language models

#121

Post by astrange »

semitones wrote: post #35
andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such

"Rare" doesn't really mean much. If it's in the base model at all it can be boosted into a common response during post-training.

Re: Persona vectors: Monitoring and controlling character traits in language models

#122

Post by vidarh »

therein wrote: post #51
simianwords wrote: post #49

Earlier quoted context omitted.

i don't think this is correct - such training data is usually made at SFT level after unsupervised learning on all available data in the web. the SFT level dataset is manually curated meaning there would be conscious effort to create more training samples of the form to say "i'm not sure". same with RLHF.

You mean I don't think this is automatically correct. Otherwise it very likely is correct. Either way, you're guessing the manual curation is done in a way that is favorable to include I don't know answers. Which it most likely doesn't.

Having done contract work on SFT datasets, at least one major provider absolutely includes don't know answers of different varieties.

I don't know why you assume it's a guess. These providers employ thousands of people directly or via a number of intermediaries to work on their SFT datasets.

Re: Persona vectors: Monitoring and controlling character traits in language models

#123

Post by astrange »

skylerwiernik wrote: post #60

> In 2023, Microsoft's Bing chatbot famously adopted an alter-ego called "Sydney,” which declared love for users and made threats of blackmail. More recently, xAI’s Grok chatbot would for a brief period sometimes identify as “MechaHitler” and make antisemitic comments. Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. Funny that they managed to…

The only bad behavior I can think of from Claude is how it used to be so ethical it'd just refuse to do anything.

The quality of its thought outside coding is pretty bad lately and especially worse than o3/Gemini though. It really feels like they've forced it to short answers for cost control.

Re: Persona vectors: Monitoring and controlling character traits in language models

#124

Post by suddenlybananas »

Illniyar wrote: post #30
vessenes wrote: post #25

Earlier quoted context omitted.

Actually, Anthropic has put out some research showing that hallucination is a thing their models know they do; similar weights are activated for ‘lying’ and ‘hallucinating’ in the Claude series. Implication - Claude knows - at least mostly - when its hallucinating. I think the current state of the art is that hallucination is at least partly a bug created by the very nature of training — you’re supposed to at least p…

That's interesting! I guess the question is how did they detect or simulate a model hallucinating in that regard? Do you have a link to that article? I can't find anything of that nature with a shallow search.

This isn't Anthropic, but here is a python library that focuses on different ways of detecting hallucinations. https://github.com/IINemo/lm-polygraph (caveat emptor, I doubt this really works).

Re: Persona vectors: Monitoring and controlling character traits in language models

#125

Post by svara »

godelski wrote: post #92
weitendorf wrote: post #83

Earlier quoted context omitted.

100% this. It is actually a very dangerous set of traits these models are being selected for: * Highly skilled and knowledgable, puts a lot of effort into the work it's asked to do * Has a strong, readily expressed sense of ethics and lines it won't cross. * Tries to be really nice and friendly, like your buddy * Gets trained to give responses that people prefer rather than responses that are correct, because market…

Thanks for that good summary. > I see how Ilya was right There are still some things Ilya[0] (and Hinton[1]). The parts I'm quoting here are an example of "that reddit comment" that sounds right but is very wrong, and something we know is wrong (and have known it is wrong for hundreds of years!). Yet, it is also something we keep having to learn. It's both obvious and not obvious, but you can make models that are goo…

Insightful and thanks for the comment, but I'm not sure I'm getting to the same conclusion as you. I think I lost you at:

> It's not just that all models are wrong and some are useful but that many models are useful but wrong. What used to be considered edge cases do not stay ...

That's not a contradiction? That popular quote says it right there: "all models are wrong". There is no model of reality, but there's a process for refining models that generates models that enable increasingly good predictions.

It stands to reason that an ideal next-token predictor would require an internal model of the world at last equally as powerful as our currently most powerful scientific theories. It also stands to reason that this model can, in principle, be trained from raw observational data, because that's how we did it.

And conversely, it stands to reason that a next-token predictor as powerful as the current crop of LLMs contains models of the world substantially more powerful than the models that powered what we used to call autocorrect.

Do you disagree with that?

Re: Persona vectors: Monitoring and controlling character traits in language models

#126

Post by yeldarb »

Wonder if you can subtract these vectors to get the opposite effect and what that ends up being for things like sycophancy or hallucination.

I also wonder what other personality vectors exist.. would be cool to find an “intelligence” vector we could boost to get better outputs from the same model. Seems like this is likely to exist given how prompting it to cosplay as a really smart person can elicit better outputs.

Re: Persona vectors: Monitoring and controlling character traits in language models

#127

Post by bjackman »

Illniyar wrote: post #21

I can see this working with "evil" and "sycophantic" personas. These seem like traits that would be amenable to input and thus be detectable by manipulating the input. But hallucination is an inherent property of LLMs - you cannot make it hallucinate less by telling it to not hallucinate or hallucinate more by telling it to make facts up (because if you tell it to make stuff up and it does, it's not hallucinating, it…

Well, you are just directly contradicting the concrete claims made by the post so one of you is wrong...

FWIW my interpretation of this is that the hallucination vector encodes the behaviour that a the model produces bullshit despite having the facts of the matter encoded in its weights. Which is slightly different than producing bullshit as a substitute for information that it "doesn't know".

And presumably there is a second-order property here where the minimal amount of hallucination is not only bounded by the model's "knowledge" but also its implicit "meta-knowledge", i.e. the "accuracy of the hallucination vector".

Re: Persona vectors: Monitoring and controlling character traits in language models

#129

Post by littlestymaar »

semitones wrote: post #35
andsoitis wrote: post #26

> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…

Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such

The problem is even harder than you make it look: even if the model founds plenty of “I don't know” answer in its training corpus it doesn't mean that this is the desirable answer to the questions: the model can know the answer even if one person on the internet doesn't.

“I don't know” must be derived from the model's knowledge as a whole, not from individual question/anser pairs in training.

Re: Persona vectors: Monitoring and controlling character traits in language models

#130

Post by taneq »

philipswood wrote: post #107
semitones wrote: post #35

Earlier quoted context omitted.

Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such

Has anybody tried what seems obvious? Have a series of pretraining sessions with training data where specific information is not present and training questions/answers of "I don't know" for that data is also trained on. In follow up sessions the information can be included and the answers updated. Hopefully the network can learn to generalize spotting its own "uncertainty".

I don’t think this specific approach would wish to well (you’re training the network to answer ‘dunno’ to that question, not to questions it can’t answer) but I think you’ve got the right general idea.

I’d try adding an output (or some special tokens or whatever) and then train it to track the current training loss for the current sample. Hopefully during inference this output would indicate how out-of-distribution the current inputs are.

Post reply on HN