61–70 of 143 posts
If the distance is too far then it's not acceptable and use the control model to average it down?
Also, isn't this similar technique as managing hallucination? (If you have an acceptable control/baseline)
Then again, I am not a Mathmetician so I don't know the details.
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…
AIs base persona is psychopathic. These just add masks.
Earlier quoted context omitted.
You mean I don't think this is automatically correct. Otherwise it very likely is correct. Either way, you're guessing the manual curation is done in a way that is favorable to include I don't know answers. Which it most likely doesn't.
its completely in the incentive to include such examples in RLHF. or you have come up with a way to increase performance that the very employees haven't. why do you think they didn't try it?
To train this effectively you would need a dataset of questions which you know the model doesn't know. But if you have that... why not answer the question and put in the dataset so that the model will know ?
That's a bit imprecise, but I think it capture the idea of why 'I don't know' answers are harder to train.
Isn't this just control vectors rediscovered? https://www.lesswrong.com/posts/Bf3ryxiM6Gff2zamw/control-ve...
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…
Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such
> In 2023, Microsoft's Bing chatbot famously adopted an alter-ego called "Sydney,” which declared love for users and made threats of blackmail. More recently, xAI’s Grok chatbot would for a brief period sometimes identify as “MechaHitler” and make antisemitic comments. Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. Funny that they managed to…