Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training LLMs for honesty via confessions

arxiv.org

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Training LLMs for honesty via confessions

#31

Post by lo_zamoyski »

catigula wrote: post #15
oytis wrote: post #14

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)? Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward s…

They really lie. Not on purpose; because they are trained on rewards that favor lying as a strategy. Othello-GPT is a good example to understand this. Without explicit training, but on the task of 'predicting moves on an Othello board', Othello-GPT spontaneously developed the strategy of 'simulate the entire board internally'. Lying is a similar emergent, very effective strategy for reward.

> They really lie. Not on purpose

You can't lie by accident. You can tell a falsehood, however.

But where LLMs are concerned, they don't tell truths or falsehoods either, as "telling" also requires intent. Moreover, LLMs don't actually contain propositional content.

Re: Training LLMs for honesty via confessions

#32

Post by [deleted] »

BaconVonPork wrote: post #29
roywiggins wrote: post #13

Earlier quoted context omitted.

The question is more whether LLMs can accurately report their internal operations, not whether any of that counts as "thinking." Simple algorithms can, eg, be designed to report whether they hit an exceptional case and activated a different set of operations than usual.

That's basically a variant of the halting problem and what you hope to get is a supervisor responding. If people expected this I don't think they would be as confused about the difference between statistical analysis of responses requiring emotions to be convincing and an LLM showing atonement.

[deleted]

Re: Training LLMs for honesty via confessions

#33

Post by skybrian »

It seems like “self-criticism” would be a better way to describe what they are training the LLM to do than “confession?” The LLM is not being directly trained to accurately reveal its chain of thought or internal calculations.

But it does have access to its chain of thought and tool calls when generating the self-criticism, and perhaps reporting on what it actually did in the chain-of-thought is an “easier” way to score higher on self-criticism?

Can this result in improved “honesty?” Maybe in the limited sense of accurately reporting what happened previously in the chat session.

Re: Training LLMs for honesty via confessions

#34

Post by skybrian »

oytis wrote: post #14

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)? Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward s…

Do LLM’s lie? Consider a situation in a screenplay where a character lies, compared to one where the character tells the truth. It seems likely that LLM’s can distinguish these situations and generate appropriate text. Internally, it can represent “the current character is lying now” differently than “the current character is telling the truth.”

And earlier this year there was some interesting research published about how LLM’s have an “evil vector” that, if enabled, gets them to act like stereotypical villains.

So it seems pretty clear that characters can lie even if the LLM’s task is just “generate text.”

This is fiction, like playing a role-playing game. But we are routinely talking to LLM-generated ghosts and the “helpful, harmless” AI assistant is not the only ghost it can conjure up.

It’s hard to see how role-playing can be all that harmful for a rational adult, but there are news reports that for some people, it definitely is.

Re: Training LLMs for honesty via confessions

#36

Post by codemac »

torginus wrote: post #3
manarth wrote: post #2

> "dishonesty may arise due to the effects of reinforcement learning (RL), where challenges with reward shaping can result in a training process that inadvertently incentivizes the model to lie or misrepresent its actions" > "As long as the "path of least resistance" for maximizing confession reward is to surface misbehavior rather than covering it up, this incentivizes models to be honest" Humans might well benefit…

I think this article once again assumes LLMs works like humans - Anthropic showed that LLMs don't understand their own thought processes, and measuring neural net activations does not correspond to what they say about how they arrived at the conclusion. I don't think this magically grants them this ability, they'll be just more convincing at faking honesty.

Please reread (or.. read) the paper. They do not make that mistake, specifically section 7.1.

A reward function (R) may be hackable by a model's response, but when asked to confess it is easier to get an honest confession reward function (Rc) because you have the response with all the hacking in front of you, and that gives the Rc more ability to verify honesty than R had to verify correctness.

There are human examples you could construct (say, granting immunity for better confessions), but they don't map well to this really fascinating insight with LLMs.

Re: Training LLMs for honesty via confessions

#37

Post by measurablefunc »

LLMs can not "lie", they do not "know" anything, and certainly can not "confess" to anything either. What LLMs can do is generate numbers which can be constructed piecemeal from some other input numbers & other sources of data by basic arithmetic operations. The output number can then be interpreted as a sequence of letters which can be imbued with semantics by someone who is capable of reading and understanding words and sentences. At no point in the process is there any kind of awareness that can be attributed to any part of the computation or the supporting infrastructure other than whoever started the whole chain of arithmetic operations by pressing some keys on some computer connected to the relevant network of computers for carrying out the arithmetic operations.

If you think this is reductionism you should explain where exactly I have reduced the operations of the computer to something that is not a correct & full fidelity representation of what is actually happening. Remember, the computer can not do anything other than boolean algebra so make sure to let me know where exactly I made an error about the arithmetic in the computer.

Re: Training LLMs for honesty via confessions

#38

Post by throwaway613745 »

oytis wrote: post #14

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)? Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward s…

Lying requires intent by definition. LLMs do not and cannot have intent, so they are incapable of lying. They just produce text. They are software.

Re: Training LLMs for honesty via confessions

#39

Post by nickpsecurity »

oytis wrote: post #14

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)? Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward s…

They mostly imitate patterns in the training material. They do it in response to what gets the reward up for RL training. There's probably lots of examples of both lying and confessions in the training data. So, it should surprise nobody that next, sentence machines fill in a lie or confession in situations similar to ghe training data.

I don't consider that very intelligent or more emergent than other behaviors. Now, if nothing like that was in training data (pure honesty with no confessions), it would be very interesting if it replied with lies and confessions. Because it wasn't pretrained to lie or confess like the above model likely was.

Re: Training LLMs for honesty via confessions

#40

Post by carsoon »

oytis wrote: post #14

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)? Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward s…

These models don't even choose 1 outcome. They list probabilities of ALL the tokens outcomes and the backend program decides to choose the one that is most probable OR a different one.

But in practical usage, if an llm does not rank token probability correctly it will feel the same as it "lying"

They are supposed to do whatever we want them to do. They WILL do what the deterministic nature of their final model outcome forces them to do.

Post reply on HN