Sycophancy is the first LLM "dark pattern"
seangoedecke.com
Sycophancy is the first LLM "dark pattern"
1–10 of 110 posts
Re: Sycophancy is the first LLM "dark pattern"
#2"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
Re: Sycophancy is the first LLM "dark pattern"
#3"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
I feel like it's a popular opinion (I've seen it many times) that it's intentional with the reasoning that it does much better on human-in-the-loop benchmarks (e.g. lm arena) when it's sycophantic.
(I have no knowledge of whether or not this is true)
Re: Sycophancy is the first LLM "dark pattern"
#4"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
I feel like it's a popular opinion (I've seen it many times) that it's intentional with the reasoning that it does much better on human-in-the-loop benchmarks (e.g. lm arena) when it's sycophantic. (I have no knowledge of whether or not this is true)
I'm sure there are a lot of "dark patterns" at play at the frontier model companies --- they're 10-figure businesses engaging directly with consumers and they're just a couple years old, so they're going to throw everything at the wall they can to see what sticks. I'm certainly not sticking up for OpenAI here. I'm just saying this article refutes its own central claim.
Re: Sycophancy is the first LLM "dark pattern"
#5"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
If I am addicted to scrolling tiktok, is it dark pattern to make UI keep me in the app as long as possible or just "emergent property" because apparently it's what I want?
Re: Sycophancy is the first LLM "dark pattern"
#6"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
It's not 'emergent' in the sense that it just happens; it's a byproduct of human feedback, and it can be neutralized.
Re: Sycophancy is the first LLM "dark pattern"
#7> Quickly learned that people are ridiculously sensitive: “Has narcissistic tendencies” - “No I do not!”, had to hide it. Hence this batch of the extreme sycophancy RLHF.
Sorry, but that doesn't seem "ridiculously sensitive" to me at all. Imagine if you went to Amazon.com and there was a button you could press to get it to pseudo-psychoanalyze you based on your purchases. People would rightly hate that! People probably ought to be sensitive to megacorps using buckets of algorithms to psychoanalyze them.
Re: Sycophancy is the first LLM "dark pattern"
#8"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
But it IS intentional, more sycophantry usually means more engagement.
Re: Sycophancy is the first LLM "dark pattern"
#9"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
If I am addicted to scrolling tiktok, is it dark pattern to make UI keep me in the app as long as possible or just "emergent property" because apparently it's what I want?
The distinction is whether it is intentional. I think your addiction to TikTok was intentional.
Re: Sycophancy is the first LLM "dark pattern"
#10Lots of research shows post-training dumbs down the models but no one listens because people are too lazy to learn proper prompt programming and would rather have a model already understand the concept of a conversation.