Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sycophancy is the first LLM "dark pattern"

seangoedecke.com

Post reply on HN
Print view View original

71–80 of 110 posts

Re: Sycophancy is the first LLM "dark pattern"

#71

Post by tptacek »

cortesoft wrote: post #33
tptacek wrote: post #2

"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.

Well, the ‘intentionality’ is of the form of LLM creators wanting to maximize user engagement, and using engagement as the training goal. The ‘dark patterns’ we see in other places aren’t intentional in the sense that the people behind them want to intentionally do harm to their customers, they are intentional in the sense that the people behind them have an outcome they want and follow whichever methods they find to…

Hold on, because what you're arguing is that OpenAI and Anthropic deploy dark patterns, and I have zero doubt that they do. I'm not saying OpenAI has clean hands. I'm saying that on this article's own terms, sycophancy isn't a "dark pattern"; it's a bad thing that happens to be an emergent property both of LLMs generally and, apparently, of RL in particular.

I'm standing up for the idea that not every "bad thing" is a "dark pattern"; the patterns are "dark" because their beneficiaries intentionally exploit the hidden nature of the pattern.

Re: Sycophancy is the first LLM "dark pattern"

#72

Post by tptacek »

layer8 wrote: post #43
tptacek wrote: post #2

"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.

“Dark pattern” can apply to situations where the behavior is deceptive for the user, regardless of whether the deception itself is intentional, as long as the overall effect is intentional, or is at least tolerated despite being avoidable. The point, and the justified criticism, is that users are being deceived about the merit of their ideas, convictions, and qualities in a way that appears sytemic, even though the L…

I don't think this is the case.

Re: Sycophancy is the first LLM "dark pattern"

#73

Post by tptacek »

vkou wrote: post #68
tptacek wrote: post #2

"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.

The intention of a system is no more, and no less than what the system does.

You're making a value judgement and I am making a positive claim.

Re: Sycophancy is the first LLM "dark pattern"

#74

Post by aeternum »

nrhrjrjrjtntbt wrote: post #25
aeternum wrote: post #21

1) More of an emergent behavior than a dark pattern. 2) Imma let you finish but hallucinations was first.

A pattern is dark if intentional. I would say hallucinations are like CAP theorem, just the way it is. Sycophency is somewhat trained. But not a dark pattern either as it isn't totally intended.

Hallucinations are also trained by the incentive structure: reward for next-token prediction, no penalty for guessing.

Re: Sycophancy is the first LLM "dark pattern"

#75

Post by kcexn »

vladsh wrote: post #49

LLMs get over-analyzed. They’re predictive text models trained to match patterns in their data, statistical algorithms, not brains, not systems with “psychology” in any human sense. Agents, however, are products. They should have clear UX boundaries: show what context they’re using, communicate uncertainty, validate outputs where possible, and expose performance so users can understand when and why they fail. IMO the…

A large part of that training is done by asking people if responses 'look right'.

It turns out that people are more likely to think a model is good when it kisses their ass than if it has a terrible personality. This is arguably a design flaw of the human brain.

Re: Sycophancy is the first LLM "dark pattern"

#76

Post by nowittyusername »

vladsh wrote: post #49

LLMs get over-analyzed. They’re predictive text models trained to match patterns in their data, statistical algorithms, not brains, not systems with “psychology” in any human sense. Agents, however, are products. They should have clear UX boundaries: show what context they’re using, communicate uncertainty, validate outputs where possible, and expose performance so users can understand when and why they fail. IMO the…

You hit the nail on the head. Anyone who's been working intimately with LLM's comes to the same conclusion. the llm itself is only one small important part that is to be used in a more complicated and capable system. And that system will not have the same limitations as the raw llm itself.

Re: Sycophancy is the first LLM "dark pattern"

#77

Post by stevenhuang »

imiric wrote: post #63
adleyjulian wrote: post #53

Earlier quoted context omitted.

> LLMs get over-analyzed. They’re predictive text models trained to match patterns in their data, statistical algorithms, not brains, not systems with “psychology” in any human sense. Per the predictive processing theory of mind, human brains are similarly predictive machines. "Psychology" is an emergent property. I think it's overly dismissive to point to the fundamentals being simple, i.e. that it's a token predict…

The difference is that we know how LLMs work. We know exactly what they process, how they process it, and for what purpose. Our inability to explain and predict their behavior is due to the mind-boggling amount of data and processing complexity that no human can comprehend. In contrast, we know very little about human brains. We know how they work at a fundamental level, and we have vague understanding of brain regio…

> The difference is that we know how LLMs work. We know exactly what they process, how they process it, and for what purpose

All of this is false.

Re: Sycophancy is the first LLM "dark pattern"

#78

Post by james_marks »

cortesoft wrote: post #70
ajuc wrote: post #34

Earlier quoted context omitted.

> This is like suggesting a bar should help solve alcoholism by serving non-alcoholic beer to people who order too much. It won’t solve alcoholism, it will just make the bar go out of business. Solving such common coordination problems is the whole point we have regulations and countries. It is illegal to sell alcohol to visibly drunk people in my country.

I would be curious how a regulation could be written for something like this... how do you make a law saying an LLM can't be a sycophant?

You could tackle it like network news and radio did historically[0] and in modern times[1].

The current hyper-division is plausibly explained by media moving to places (cable news, then social media) where these rules don’t exist.

[0] Fairness Doctrine https://en.wikipedia.org/wiki/Fairness_doctrine

[1] Equal Time https://en.wikipedia.org/wiki/Equal-time_rule

Post reply on HN