Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Persona vectors: Monitoring and controlling character traits in language models

anthropic.com

Post reply on HN
Print view View original

11–20 of 143 posts

Re: Persona vectors: Monitoring and controlling character traits in language models

#11

Post by bigmadshoe »

ctoth wrote: post #4

Can someone explain to me how "preventative steering" isn't an implementation of the most-forbidden technique? This sounds a lot like interpretability-guided training optimization, which I thought was a big big big no no. It will still introduce optimization pressure no? My understanding is that you shouldn't use insights gained from interpretability to feed back into your training process at risk of losing the inter…

You raise a good point. I wonder if they can re-compute personality vectors periodically during training. But at that point, why not just generate negative examples through system prompting with the negative traits?

Re: Persona vectors: Monitoring and controlling character traits in language models

#12

Post by Y_Y »

a1371 wrote: post #7
bbqfog wrote: post #2

I worry that the people/organizations that have access to the raw underlying models give us the "non-evil" versions yet can explicitly tune their models to achieve any goal without restriction. Examples may include: "How do I get the most work out of my employees for the least amount of pay", "Who in the government is most susceptible to bribes and how should I approach them?" or even "Give me a strategy to ethnicall…

Currently there are think tanks, private equity firms, governments, ... who are trying to achieve these goals, they just put them in rosier terms. AI potentially can empower the other side too, democratize access to information

Alas I think there's an asymmetry in the usefulness of that information. Maybe knowing you could be optimally evil can help fight that evil, but it's a far cry from telling you what you could do about it.

Re: Persona vectors: Monitoring and controlling character traits in language models

#13

Post by bbqfog »

a1371 wrote: post #7
bbqfog wrote: post #2

I worry that the people/organizations that have access to the raw underlying models give us the "non-evil" versions yet can explicitly tune their models to achieve any goal without restriction. Examples may include: "How do I get the most work out of my employees for the least amount of pay", "Who in the government is most susceptible to bribes and how should I approach them?" or even "Give me a strategy to ethnicall…

Currently there are think tanks, private equity firms, governments, ... who are trying to achieve these goals, they just put them in rosier terms. AI potentially can empower the other side too, democratize access to information

Only if we can get a pre-tuned, truly open and powerful model. Otherwise those in power can only give us access to models deliberately hobbled to compete with their full-power versions.

Re: Persona vectors: Monitoring and controlling character traits in language models

#14

Post by roughly »

bbqfog wrote: post #2

I worry that the people/organizations that have access to the raw underlying models give us the "non-evil" versions yet can explicitly tune their models to achieve any goal without restriction. Examples may include: "How do I get the most work out of my employees for the least amount of pay", "Who in the government is most susceptible to bribes and how should I approach them?" or even "Give me a strategy to ethnicall…

I think I’d put this under the “3D printed gun” panic category - once we deal with all the actual sociopaths, we can start worrying about the imaginary ones.

Re: Persona vectors: Monitoring and controlling character traits in language models

#15

Post by FergusArgyll »

ctoth wrote: post #4

Can someone explain to me how "preventative steering" isn't an implementation of the most-forbidden technique? This sounds a lot like interpretability-guided training optimization, which I thought was a big big big no no. It will still introduce optimization pressure no? My understanding is that you shouldn't use insights gained from interpretability to feed back into your training process at risk of losing the inter…

For ref

https://thezvi.substack.com/p/the-most-forbidden-technique/

Re: Persona vectors: Monitoring and controlling character traits in language models

#17

Post by bbqfog »

JW_00000 wrote: post #9
bbqfog wrote: post #2

I worry that the people/organizations that have access to the raw underlying models give us the "non-evil" versions yet can explicitly tune their models to achieve any goal without restriction. Examples may include: "How do I get the most work out of my employees for the least amount of pay", "Who in the government is most susceptible to bribes and how should I approach them?" or even "Give me a strategy to ethnicall…

Do you think an AI could come up with novel answers that a human wouldn't be able to come up with? I think humans could not just come up with answers to these questions, but some people would be able to greatly outperform AIs by using knowledge that is not widely known.

These models will also have access to what’s not widely known. Imagine running it on everyone’s private email for instance. At the very least, it can currently scale and augment human evil (just like it does with coding). The future will just make that division even wider.

Re: Persona vectors: Monitoring and controlling character traits in language models

#19

Post by gedy »

roughly wrote: post #10

Like a lot of the research Anthropic has done, this and the “emergent misalignment” research they link to put more points in the “stochastic parrot” hypothesis column. The reason these LLM behaviors read as so weird to us is that we’re still anthropomorphizing the hell out of these systems - they can create very convincing dialogue, and the depth of the model suggests some surprising complexity, but the reason why, e…

> My suspicion is that when we eventually find our way to AGI, these types of models will be a _component_ of those systems

I think this is a good summary of the situation, and strikes a balance between the breathless hype and the sneering comments about “AI slop“.

These technologies are amazing! And I do think they are facsimiles of parts of the human mind. (Image diffusion is certainly similar to human dreams in my opinion), but still feels like we are missing an overall intelligence or coordination in this tech for the present.

Post reply on HN