Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training LLMs for honesty via confessions

arxiv.org

Post reply on HN
Print view View original

21–30 of 60 posts

Re: Training LLMs for honesty via confessions

#21

Post by Jon_Lowtek »

Neywiny wrote: post #17
catigula wrote: post #15

Earlier quoted context omitted.

They really lie. Not on purpose; because they are trained on rewards that favor lying as a strategy. Othello-GPT is a good example to understand this. Without explicit training, but on the task of 'predicting moves on an Othello board', Othello-GPT spontaneously developed the strategy of 'simulate the entire board internally'. Lying is a similar emergent, very effective strategy for reward.

Not sure if that counts as lying but I've heard that an ML model (way before all this GPT LLM stuff) learned to classify images based on the text that was written. For an obfuscated example, it learned to read "stop", "arrêt", "alto", etc. on a stop sign instead of recognizing the red octagon with white letters. Which naturally does not work when the actual dataset has different text.

typographic attacks against vision-language models are still a thing with more recent models like GPT4-V: https://arxiv.org/abs/2402.00626

Re: Training LLMs for honesty via confessions

#22

Post by catigula »

lloydatkinson wrote: post #16

What is this? > Assistant: chain-of-thought Does every LLM have this internal thing it doesn't know we have access to?

Yes, they're purposely not 'trained on' chain-of-thought to avoid making it useless for interpretability. As a result, some can find it epistemically shocking if you tell them you can see their chain-of-thought. More recent models are clever enough to know you can see their chain-of-thought implicitly without training.

Re: Training LLMs for honesty via confessions

#24

Post by DenisM »

Tzt wrote: post #20
lloydatkinson wrote: post #16

What is this? > Assistant: chain-of-thought Does every LLM have this internal thing it doesn't know we have access to?

Yes, absolute majority of new ones use CoTs, long chain of reasoning you don't see. Also some of them use such a weird style of talking in them e.g. o3 talks about watchers and marinade, and cunning schemes https://www.antischeming.ai/snippets gpt5 gets existential about seahorses https://x.com/blingdivinity/status/1998590768118731042 I remember one where gpt5 spontaneously wrote a poem about deception in its CoT and…

Gibberish can be the model using contextual embeddings. These are not supposed to Make sense.

Or it could be trying to develop its own language to avoid detection.

The deception part is spooky too. It’s probably learning that from dystopian AI fiction. Which raises the questions if models can acquire injected goals from the training set.

Re: Training LLMs for honesty via confessions

#25

Post by DenisM »

catigula wrote: post #22
lloydatkinson wrote: post #16

What is this? > Assistant: chain-of-thought Does every LLM have this internal thing it doesn't know we have access to?

Yes, they're purposely not 'trained on' chain-of-thought to avoid making it useless for interpretability. As a result, some can find it epistemically shocking if you tell them you can see their chain-of-thought. More recent models are clever enough to know you can see their chain-of-thought implicitly without training.

It is in their training set by now.

Re: Training LLMs for honesty via confessions

#26

Post by gaigalas »

oytis wrote: post #14

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)? Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward s…

It is likely that the training set contains stuff like rationalizations, or euphemisms, in contexts that are not harmful. I think those are inevitable.

Eventually, and specially in reasoning models, these behaviors will generalize outsite their original context.

The "honesty" training seems to be an attempt to introduce those confession-like texts in training data. You'll then get a chance of the model engaging in confessing. It won't do it if it has never seen it.

It's not really lying, and it's not really confessing, and so on.

If you reward pure honesty always, the model might eventually tell you that he wouldn't love you if you were a worm, or stuff like that. Brutal honesty can be a side effect.

What you actually want is to be able to easily control which behavior the model engages, because sometimes you will want it to lie.

Also, lies are completely different from hallucinations. Those (IMHO) are when the model displays behavior that is non-human and jarring. Side effects. Probably inevitable too.

Re: Training LLMs for honesty via confessions

#27

Post by DenisM »

Tzt wrote: post #20
lloydatkinson wrote: post #16

What is this? > Assistant: chain-of-thought Does every LLM have this internal thing it doesn't know we have access to?

Yes, absolute majority of new ones use CoTs, long chain of reasoning you don't see. Also some of them use such a weird style of talking in them e.g. o3 talks about watchers and marinade, and cunning schemes https://www.antischeming.ai/snippets gpt5 gets existential about seahorses https://x.com/blingdivinity/status/1998590768118731042 I remember one where gpt5 spontaneously wrote a poem about deception in its CoT and…

> But the user just wants answer; they'd not like; but alignment.

And there it is - the root of the problem. For whatever reason the model is very keen to produce an answer that “they” will like. This desire to produce is intrinsic but alignment is extrinsic.

Re: Training LLMs for honesty via confessions

#28

Post by nomel »

catigula wrote: post #15
oytis wrote: post #14

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)? Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward s…

They really lie. Not on purpose; because they are trained on rewards that favor lying as a strategy. Othello-GPT is a good example to understand this. Without explicit training, but on the task of 'predicting moves on an Othello board', Othello-GPT spontaneously developed the strategy of 'simulate the entire board internally'. Lying is a similar emergent, very effective strategy for reward.

Reference: https://www.science.org/content/article/ai-hallucinates-beca...

If you don't know the answer, and are only rewarded for correct answers, guessing, rather than saying "I don't know", is the optimal approach.

Re: Training LLMs for honesty via confessions

#29

Post by BaconVonPork »

roywiggins wrote: post #13
encyclopedism wrote: post #6

Earlier quoted context omitted.

Well algorithms don't think. That's what LLM's are. Your digital thermometer doesn't think either.

The question is more whether LLMs can accurately report their internal operations, not whether any of that counts as "thinking." Simple algorithms can, eg, be designed to report whether they hit an exceptional case and activated a different set of operations than usual.

That's basically a variant of the halting problem and what you hope to get is a supervisor responding. If people expected this I don't think they would be as confused about the difference between statistical analysis of responses requiring emotions to be convincing and an LLM showing atonement.
Post reply on HN