Persona vectors: Monitoring and controlling character traits in language models
1–10 of 143 posts
Re: Persona vectors: Monitoring and controlling character traits in language models
#2I worry that the people/organizations that have access to the raw underlying models give us the "non-evil" versions yet can explicitly tune their models to achieve any goal without restriction. Examples may include: "How do I get the most work out of my employees for the least amount of pay", "Who in the government is most susceptible to bribes and how should I approach them?" or even "Give me a strategy to ethnically cleanse a region while navigating international relations". It could be anything and those in power (without naming names, I would consider many of them evil for sure) can use them to achieve their goals while leaving the rest of us unable to defend ourselves. To some degree it feels like the right to bear arms has intersecting goals.
Re: Persona vectors: Monitoring and controlling character traits in language models
#3some of these personas seem too simple.. the evil one for example sounds like a james bond villain, not quite what a real villain would actually be.
Re: Persona vectors: Monitoring and controlling character traits in language models
#4Can someone explain to me how "preventative steering" isn't an implementation of the most-forbidden technique?
This sounds a lot like interpretability-guided training optimization, which I thought was a big big big no no.
It will still introduce optimization pressure no?
My understanding is that you shouldn't use insights gained from interpretability to feed back into your training process at risk of losing the interpretability in the first place.
Re: Persona vectors: Monitoring and controlling character traits in language models
#5I worry that the people/organizations that have access to the raw underlying models give us the "non-evil" versions yet can explicitly tune their models to achieve any goal without restriction. Examples may include: "How do I get the most work out of my employees for the least amount of pay", "Who in the government is most susceptible to bribes and how should I approach them?" or even "Give me a strategy to ethnicall…
Yeah, a more terrifying and realistic Terminator movie would be one where the robot looks all cute and furry and then, when it has found mass adoption, suddenly turns against humanity.
Re: Persona vectors: Monitoring and controlling character traits in language models
#6Voice matters too. ChatGPT’s best voice was the Scarlett Johansson reproduction. Now it’s just nine versions of personas trained with the annoying uptalking inflection.
Re: Persona vectors: Monitoring and controlling character traits in language models
#7I worry that the people/organizations that have access to the raw underlying models give us the "non-evil" versions yet can explicitly tune their models to achieve any goal without restriction. Examples may include: "How do I get the most work out of my employees for the least amount of pay", "Who in the government is most susceptible to bribes and how should I approach them?" or even "Give me a strategy to ethnicall…
Currently there are think tanks, private equity firms, governments, ... who are trying to achieve these goals, they just put them in rosier terms. AI potentially can empower the other side too, democratize access to information
Re: Persona vectors: Monitoring and controlling character traits in language models
#8It’s funny that they chose only negative characteristics as traits, as if to imply that they could make the models “good” just with guidance from these vectors.
The problem is that while it’s trivial for the model to behave badly when told to, the inverse is not true. Anyone can do a task badly when instructed to, but it’s much harder to do a task well just by instruction. There’s a difference between being good and being not bad.
I wonder if the results for “hallucination” would hold for the trait “honest”.
Re: Persona vectors: Monitoring and controlling character traits in language models
#9I worry that the people/organizations that have access to the raw underlying models give us the "non-evil" versions yet can explicitly tune their models to achieve any goal without restriction. Examples may include: "How do I get the most work out of my employees for the least amount of pay", "Who in the government is most susceptible to bribes and how should I approach them?" or even "Give me a strategy to ethnicall…
Do you think an AI could come up with novel answers that a human wouldn't be able to come up with? I think humans could not just come up with answers to these questions, but some people would be able to greatly outperform AIs by using knowledge that is not widely known.
Re: Persona vectors: Monitoring and controlling character traits in language models
#10Like a lot of the research Anthropic has done, this and the “emergent misalignment” research they link to put more points in the “stochastic parrot” hypothesis column. The reason these LLM behaviors read as so weird to us is that we’re still anthropomorphizing the hell out of these systems - they can create very convincing dialogue, and the depth of the model suggests some surprising complexity, but the reason why, eg, a random string of numbers will induce changes elsewhere in the model is there’s simply nothing in the model to Be consistent. It is an extremely complex autocomplete algorithm that does a very effective cosplay of an “intelligent agent.”
My suspicion is that when we eventually find our way to AGI, these types of models will be a _component_ of those systems, but they lack some fundamental structuring that seems to be required to create anything like consistency or self-reflection.
(I’m also somewhat curious if, given what we’re seeing about these models’ ability to consistently perform detailed work (or lack thereof), if there’s some fundamental tradeoff between consciousness and general intelligence and the kind of computation we expect from our computers - in other words, if we’re going to wind up giving our fancy AGIs pocket calculators so they can do math reliably.)