Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training LLMs for honesty via confessions

arxiv.org

Post reply on HN
Print view View original

11–20 of 60 posts

Re: Training LLMs for honesty via confessions

#11

Post by hnuser123456 »

jerf wrote: post #7
torginus wrote: post #3

Earlier quoted context omitted.

I think this article once again assumes LLMs works like humans - Anthropic showed that LLMs don't understand their own thought processes, and measuring neural net activations does not correspond to what they say about how they arrived at the conclusion. I don't think this magically grants them this ability, they'll be just more convincing at faking honesty.

Humans don't understand their thought process either. In general, neural nets do not have insight into what they are doing, because they can't. Can you tell me what neurons fired in the process of reading this text? No. You don't have access to that information. We can recursively model our own network and say something about which regions of the brain are probably involved due to other knowledge, but that's all a hi…

Makes me wonder if one could train a "neural net surgeon" model which can trace activations in another live model and manipulate it according to plain language instructions.

Re: Training LLMs for honesty via confessions

#12

Post by roywiggins »

jerf wrote: post #7
torginus wrote: post #3

Earlier quoted context omitted.

I think this article once again assumes LLMs works like humans - Anthropic showed that LLMs don't understand their own thought processes, and measuring neural net activations does not correspond to what they say about how they arrived at the conclusion. I don't think this magically grants them this ability, they'll be just more convincing at faking honesty.

Humans don't understand their thought process either. In general, neural nets do not have insight into what they are doing, because they can't. Can you tell me what neurons fired in the process of reading this text? No. You don't have access to that information. We can recursively model our own network and say something about which regions of the brain are probably involved due to other knowledge, but that's all a hi…

> In much the same way as any human with modest experience in adding two digit numbers doesn't necessarily sit there and do the full elementary school addition algorithm but jumps to the correct answer in fewer steps by virtue of just having a very trained neural net.

Right, which is strictly worse than humans are at reporting how they solve these sorts of problems. Humans can tell you whether they did the elementary school addition algorithm or not. It seems like Claude actually doesn't know, in the same way humans don't really know how they can balance on two legs, it's just too baked into the structure of their cognition to be able to introspect it. But stuff like "adding two-digit numbers" is usually straightforwardly introspectable for humans, even if it's just "oh, I just vibed it" vs "I mentally added the digits and carried the one"- humans can mostly report which it was.

Here's Anthropic's research:

https://www.anthropic.com/research/tracing-thoughts-language...

Re: Training LLMs for honesty via confessions

#13

Post by roywiggins »

encyclopedism wrote: post #6
pfortuny wrote: post #5

Earlier quoted context omitted.

Honest question: > Anthropic showed that LLMs don't understand their own thought processes Where can I find this? I am really interested in that. Thanks.

Well algorithms don't think. That's what LLM's are. Your digital thermometer doesn't think either.

The question is more whether LLMs can accurately report their internal operations, not whether any of that counts as "thinking."

Simple algorithms can, eg, be designed to report whether they hit an exceptional case and activated a different set of operations than usual.

Re: Training LLMs for honesty via confessions

#14

Post by oytis »

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)?

Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward strategy to the answer itself?

Re: Training LLMs for honesty via confessions

#15

Post by catigula »

oytis wrote: post #14

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)? Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward s…

They really lie.

Not on purpose; because they are trained on rewards that favor lying as a strategy.

Othello-GPT is a good example to understand this. Without explicit training, but on the task of 'predicting moves on an Othello board', Othello-GPT spontaneously developed the strategy of 'simulate the entire board internally'. Lying is a similar emergent, very effective strategy for reward.

Re: Training LLMs for honesty via confessions

#17

Post by Neywiny »

catigula wrote: post #15
oytis wrote: post #14

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)? Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward s…

They really lie. Not on purpose; because they are trained on rewards that favor lying as a strategy. Othello-GPT is a good example to understand this. Without explicit training, but on the task of 'predicting moves on an Othello board', Othello-GPT spontaneously developed the strategy of 'simulate the entire board internally'. Lying is a similar emergent, very effective strategy for reward.

Not sure if that counts as lying but I've heard that an ML model (way before all this GPT LLM stuff) learned to classify images based on the text that was written. For an obfuscated example, it learned to read "stop", "arrêt", "alto", etc. on a stop sign instead of recognizing the red octagon with white letters. Which naturally does not work when the actual dataset has different text.

Re: Training LLMs for honesty via confessions

#18

Post by MarkusQ »

oytis wrote: post #14

Do these models really lie or do they only do what they are supposed to do - produce text that is statistically similar to the training set, but not in the training set (and thus can include false/made up statements)? Now they add another run on top of it that is in principle prone to the same issues, except they reward the model for factuality instead of likeability. This is cool, but why not apply the same reward s…

They don't really lie, they just produce text.

But the Eliza effect is amazingly powerful.

Applying the same reward strategy to the answer itself would be a more intellectually honest approach, but would rub our noses in the fact that LLMs don't have any access to "truth" and so at best we'd be conditioning them to be better at fooling us.

Re: Training LLMs for honesty via confessions

#19

Post by catigula »

Neywiny wrote: post #17
catigula wrote: post #15

Earlier quoted context omitted.

They really lie. Not on purpose; because they are trained on rewards that favor lying as a strategy. Othello-GPT is a good example to understand this. Without explicit training, but on the task of 'predicting moves on an Othello board', Othello-GPT spontaneously developed the strategy of 'simulate the entire board internally'. Lying is a similar emergent, very effective strategy for reward.

Not sure if that counts as lying but I've heard that an ML model (way before all this GPT LLM stuff) learned to classify images based on the text that was written. For an obfuscated example, it learned to read "stop", "arrêt", "alto", etc. on a stop sign instead of recognizing the red octagon with white letters. Which naturally does not work when the actual dataset has different text.

That does feel a little more like over-fitting, but you might be able to argue that there's some philosophical proximity to lying.

I think, largely, the

  Pre-training -> Post-training -> Safety/Alignment training
pipeline would obviously produce 'lying'. The trainings are in a sort of mutual dissonance.

Re: Training LLMs for honesty via confessions

#20

Post by Tzt »

lloydatkinson wrote: post #16

What is this? > Assistant: chain-of-thought Does every LLM have this internal thing it doesn't know we have access to?

Yes, absolute majority of new ones use CoTs, long chain of reasoning you don't see.

Also some of them use such a weird style of talking in them e.g.

o3 talks about watchers and marinade, and cunning schemes https://www.antischeming.ai/snippets

gpt5 gets existential about seahorses https://x.com/blingdivinity/status/1998590768118731042

I remember one where gpt5 spontaneously wrote a poem about deception in its CoT and then resumed like nothing weird happened. But I can't find mentions of it now.

Post reply on HN