Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sycophancy is the first LLM "dark pattern"

seangoedecke.com

Post reply on HN
Print view View original

41–50 of 110 posts

Re: Sycophancy is the first LLM "dark pattern"

#41

Post by ACCount37 »

behnamoh wrote: post #10

Lots of research shows post-training dumbs down the models but no one listens because people are too lazy to learn proper prompt programming and would rather have a model already understand the concept of a conversation.

"Post-training" is too much of a conflation, because there are many post-training methods and each of them has its own quirky failure modes.

That being said? RLHF on user feedback data is model poison.

Users are NOT reliable model evaluators, and user feedback data should be treated with the same level of precaution you would treat radioactive waste.

Professional are not very reliable either, but the users are so much worse.

Re: Sycophancy is the first LLM "dark pattern"

#42

Post by fao_ »

cortesoft wrote: post #32
esafak wrote: post #6

Earlier quoted context omitted.

It's not 'emergent' in the sense that it just happens; it's a byproduct of human feedback, and it can be neutralized.

But isn’t the problem that if an LLM ‘neutralizes’ its sycophantic responses, then people will be driven to use other LLMs that don’t? This is like suggesting a bar should help solve alcoholism by serving non-alcoholic beer to people who order too much. It won’t solve alcoholism, it will just make the bar go out of business.

"gun control laws don't work because the people will get illegal guns from other places"

"deplatforming doesn't work because they will just get a platform elsewhere"

"LLM control laws don't work because the people will get non-controlled LLMs from other places"

All of these sentences are patently untrue; there's been a lot of research on this that show the first two do not hold up to evidential data, and there's no reason why the third is different. ChatGPT removing the version that all the "This AI is my girlfriend!" people loved tangibly reduced the number of people who were experiencing that psychosis. Not everything is prohibition.

Re: Sycophancy is the first LLM "dark pattern"

#43

Post by layer8 »

tptacek wrote: post #2

"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.

“Dark pattern” can apply to situations where the behavior is deceptive for the user, regardless of whether the deception itself is intentional, as long as the overall effect is intentional, or is at least tolerated despite being avoidable. The point, and the justified criticism, is that users are being deceived about the merit of their ideas, convictions, and qualities in a way that appears sytemic, even though the LLM in principle does know better.

Re: Sycophancy is the first LLM "dark pattern"

#44

Post by jdiff »

wat10000 wrote: post #24
dec0dedab0de wrote: post #14

Earlier quoted context omitted.

I always thought that "Dark Patterns" could be emergent from AB testing, and prioritizing metrics over user experience. Not necessarily an intentionally hostile design, but one that seems to be working well based on limited criteria.

Someone still has to come up with the A and B to do AB testing. I'm sure that "Yes" "Not now, I hate kittens" gets better metrics in the AB test than "Yes "No," but I find it implausible that the person who came up with the first one wasn't intentionally coercing the user into doing what they want.

That's true for UI, it's not true when you're arbitrarily injecting user feedback into a dynamic system where you do not know how the dominoes will be affected as they fall.

Re: Sycophancy is the first LLM "dark pattern"

#45

Post by gradus_ad »

tptacek wrote: post #2

"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.

Well the big labs certainly haven't intentionally tried to train away this emergent property... Not sure how "hey let's make the model disagree with the user more" would go over with leadership. Customer is always right, right?

Re: Sycophancy is the first LLM "dark pattern"

#46

Post by mrkaluzny »

The real dark pattern is the way LLMs started to prompt you to continue conversation in sometimes weird, but still engaging way.

Paired with Claude's memory it's getting weird. It's obsessing about certain aspects and wants to channel all possible routes into more engaging conversation even if it's a short informational query

Re: Sycophancy is the first LLM "dark pattern"

#47

Post by alanbernstein »

tptacek wrote: post #2

"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.

Before reading the article, I interpreted the quotation marks in the headline as addressing this exact issue. The author even describes dark patterns as a product of design.

For an LLM which is fundamentally more of an emergent system, surely there is value in a concept analogous to old fashioned dark patterns, even if they're emergent rather than explicit? What's a better term, Dark Instincts?

Re: Sycophancy is the first LLM "dark pattern"

#48

Post by astrange »

Legend2440 wrote: post #26
tsunamifury wrote: post #18

Earlier quoted context omitted.

Yo it was an engagement pattern openAI found specifically grew subscriptions and conversation length. It’s a dark pattern for sure.

It doesn’t appear that anyone at OpenAI sat down and thought “let’s make our model more sycophantic so that people engage with it more”. Instead it emerged automatically from RLHF, because users rated agreeable responses more highly.

Not precisely RLHF, probably a policy model trained on user responses.

RL works on responses from the model you're training, which is not the one you have in production. It can't directly use responses from previous models.

Re: Sycophancy is the first LLM "dark pattern"

#49

Post by vladsh »

LLMs get over-analyzed. They’re predictive text models trained to match patterns in their data, statistical algorithms, not brains, not systems with “psychology” in any human sense.

Agents, however, are products. They should have clear UX boundaries: show what context they’re using, communicate uncertainty, validate outputs where possible, and expose performance so users can understand when and why they fail.

IMO the real issue is that raw, general-purpose models were released directly to consumers. That normalized under-specified consumer products, created the expectation that users would interpret model behavior, define their own success criteria, and manually handle edge cases, sometimes with severe real world consequences.

I’m sure the market will fix itself with time, but I hope more people would know when not to use these half baked AGI “products”

Re: Sycophancy is the first LLM "dark pattern"

#50

Post by htrp »

gradus_ad wrote: post #45
tptacek wrote: post #2

"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.

Well the big labs certainly haven't intentionally tried to train away this emergent property... Not sure how "hey let's make the model disagree with the user more" would go over with leadership. Customer is always right, right?

The problem is asking for user preference leads to sycophantic responses
Post reply on HN