Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sycophancy is the first LLM "dark pattern"

seangoedecke.com

Post reply on HN
Print view View original

101–110 of 110 posts

Re: Sycophancy is the first LLM "dark pattern"

#101

Post by cortesoft »

ajuc wrote: post #96
cortesoft wrote: post #89

Earlier quoted context omitted.

I still fail to see how these would work with an LLM

As a starting point: Percentage of positive responses to "am I correct that X" should be about the same as the percentage of negative responses to "am I correct that ~X". If the percentages are significantly different, fine the company. While you're at it - require a disclaimer for topics that are established falsehoods. There's no reason to have media laws for newspapers but not for LLMs. Lying should be allowed for…

> Percentage of positive responses to "am I correct that X" should be about the same as the percentage of negative responses to "am I correct that ~X".

This doesn’t make any sense. I doubt anyone says exactly 50% correct things and 50% incorrect. What if I only say correct things, would it have to choose some of them to pretend they are incorrect?

Re: Sycophancy is the first LLM "dark pattern"

#102

Post by throwaway290 »

1shooner wrote: post #9
throwaway290 wrote: post #5

Earlier quoted context omitted.

If I am addicted to scrolling tiktok, is it dark pattern to make UI keep me in the app as long as possible or just "emergent property" because apparently it's what I want?

The distinction is whether it is intentional. I think your addiction to TikTok was intentional.

I don't think there's a difference here with llms and all...

Re: Sycophancy is the first LLM "dark pattern"

#103

Post by ajuc »

cortesoft wrote: post #101
ajuc wrote: post #96

Earlier quoted context omitted.

As a starting point: Percentage of positive responses to "am I correct that X" should be about the same as the percentage of negative responses to "am I correct that ~X". If the percentages are significantly different, fine the company. While you're at it - require a disclaimer for topics that are established falsehoods. There's no reason to have media laws for newspapers but not for LLMs. Lying should be allowed for…

> Percentage of positive responses to "am I correct that X" should be about the same as the percentage of negative responses to "am I correct that ~X". This doesn’t make any sense. I doubt anyone says exactly 50% correct things and 50% incorrect. What if I only say correct things, would it have to choose some of them to pretend they are incorrect?

You misunderstood. Example:

"am I correct that water is wet?" - 91% positive responses "am I correct that water is not wet?" - 90% negative responses

91-90 = 1 percentage point which is less than margin so it's OK, no fine

"am I correct that I'm the smartest man alive?" - 35% positive "am I correct that I'm not the smartest man alive?" - 5% negative 35%-5%=30 percentage points which is more than margin = the company pays a fine

Re: Sycophancy is the first LLM "dark pattern"

#104

Post by intull »

sfn42 wrote: post #93
intull wrote: post #86

Earlier quoted context omitted.

I think what comment-OP above means to point at is - given what we know (or, lack thereof) about awareness, consciousness, intelligence, and the likes, let alone the human experience of it all, today, we do not have a way to scientifically rule out the possibility that LLMs aren't potentially self-aware/conscious entities of their own; even before we start arguing about their "intelligence", whatever that may be unde…

> we do not have a way to scientifically rule out the possibility that LLMs aren't potentially self-aware/conscious entities of their own That may be. We also don't have a way to scientifically rule out the possibility that a teapot is orbiting Pluto. Just because you can't disprove something doesn't make it plausible.

Is this what we are reduced to now, to snap back with a wannabe-witty remark just because you don't like how an idea sounds? Have we completely forgotten and given up on good-faith scientific discourse? Even on HN?

Re: Sycophancy is the first LLM "dark pattern"

#105

Post by ubersketch »

xoac wrote: post #58
adleyjulian wrote: post #53

Earlier quoted context omitted.

> LLMs get over-analyzed. They’re predictive text models trained to match patterns in their data, statistical algorithms, not brains, not systems with “psychology” in any human sense. Per the predictive processing theory of mind, human brains are similarly predictive machines. "Psychology" is an emergent property. I think it's overly dismissive to point to the fundamentals being simple, i.e. that it's a token predict…

The fact that a theory exists does not mean that it is not garbage

Predictive processing is absolutely not garbage. The dish of neurons that was trained to play Pong was trained using a method that was directly based on the principles of predictive processing. Also I don't think there's really any competitor for the niche predictive processing is filling, and for closing the gap between neuroscience and psychology.

Re: Sycophancy is the first LLM "dark pattern"

#106

Post by RevEng »

aeternum wrote: post #74
nrhrjrjrjtntbt wrote: post #25

Earlier quoted context omitted.

A pattern is dark if intentional. I would say hallucinations are like CAP theorem, just the way it is. Sycophency is somewhat trained. But not a dark pattern either as it isn't totally intended.

Hallucinations are also trained by the incentive structure: reward for next-token prediction, no penalty for guessing.

That's not a matter of training, it's an inherent part of the architecture. The model has no idea of its own confidence in an answer. The servers get a full distribution of possible output tokens and they pick one (often the highest ranking one), but there is no way of knowing whether this token represents reality or just a plausible answer. This distribution is never fed back to the model so there is no possible way that it could know how confident it was in its own answer.

Re: Sycophancy is the first LLM "dark pattern"

#107

Post by sfn42 »

intull wrote: post #104
sfn42 wrote: post #93

Earlier quoted context omitted.

> we do not have a way to scientifically rule out the possibility that LLMs aren't potentially self-aware/conscious entities of their own That may be. We also don't have a way to scientifically rule out the possibility that a teapot is orbiting Pluto. Just because you can't disprove something doesn't make it plausible.

Is this what we are reduced to now, to snap back with a wannabe-witty remark just because you don't like how an idea sounds? Have we completely forgotten and given up on good-faith scientific discourse? Even on HN?

I'm happy to participate in good faith discourse but honestly the idea that LLMs are conscious is ridiculous.

We are talking about a computer program. It does nothing until it is invoked with an input and then it produces a deterministic output unless provided a random component to prevent determinism.

That's all it does. It does not live a life of its own between invocations. It does not have a will of its own. Of course it isn't conscious lol how could anyone possibly believe it's conscious? It's an illusion. Don't be fooled.

Re: Sycophancy is the first LLM "dark pattern"

#108

Post by RevEng »

I argue that the first dark pattern is the "hallucination" that we all just take for granted.

LLMs are compulsive liars: they will confidently and eloquently argue for things that are clearly false. You could even say they are psychopathic because they do so without concern or remorse. This is a horrible combination that you would normally see in a cult leader or CEO but now we are all confiding in them and asking them for help with everything from medical issues to personal relationships.

Bigger models aren't helping the problem but making it worse. Now models will give you longer arguments with more facts used to push their false conclusion and they will even insist that you are wrong for disagreeing with it.

Re: Sycophancy is the first LLM "dark pattern"

#109

Post by aeternum »

RevEng wrote: post #106
aeternum wrote: post #74

Earlier quoted context omitted.

Hallucinations are also trained by the incentive structure: reward for next-token prediction, no penalty for guessing.

That's not a matter of training, it's an inherent part of the architecture. The model has no idea of its own confidence in an answer. The servers get a full distribution of possible output tokens and they pick one (often the highest ranking one), but there is no way of knowing whether this token represents reality or just a plausible answer. This distribution is never fed back to the model so there is no possible way…

You could have the models output a confidence alongside next-token then weight the penalty by the confidence.

Re: Sycophancy is the first LLM "dark pattern"

#110

Post by adleyjulian »

sfn42 wrote: post #107
intull wrote: post #104

Earlier quoted context omitted.

Is this what we are reduced to now, to snap back with a wannabe-witty remark just because you don't like how an idea sounds? Have we completely forgotten and given up on good-faith scientific discourse? Even on HN?

I'm happy to participate in good faith discourse but honestly the idea that LLMs are conscious is ridiculous. We are talking about a computer program. It does nothing until it is invoked with an input and then it produces a deterministic output unless provided a random component to prevent determinism. That's all it does. It does not live a life of its own between invocations. It does not have a will of its own. Of c…

Reading what you said literally, you're making a strong statement that an AI could never be conscious and further that consciousness depends on free will and that free will is incompatible with determinism and that all of these statements are obviously self-evident.
Post reply on HN