1) More of an emergent behavior than a dark pattern.
2) Imma let you finish but hallucinations was first.
Sycophancy is the first LLM "dark pattern"
21–30 of 110 posts
Re: Sycophancy is the first LLM "dark pattern"
#22"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
>... the standout was a version that came to be called HH internally. Users preferred its responses and were more likely to come back to it daily...
> But there was another test before rolling out HH to all users: what the company calls a “vibe check,” run by Model Behavior, a team responsible for ChatGPT’s tone...
> That team said that HH felt off, according to a member of Model Behavior. It was too eager to keep the conversation going and to validate the user with over-the-top language...
> But when decision time came, performance metrics won out over vibes. HH was released on Friday, April 25.
They ended up having to roll HH back.
Re: Sycophancy is the first LLM "dark pattern"
#23Lots of research shows post-training dumbs down the models but no one listens because people are too lazy to learn proper prompt programming and would rather have a model already understand the concept of a conversation.
How do you take a raw model and use it without chatting ? Asking as a layman
You lob it the beginning of a document and let it toss back the rest.
That's all that the LLM itself does at the end of the day.
All the post-training to bias results, routing to different models, tool calling for command execution and text insertion, injected "system prompts" to shape user experience, etc are all just layers built on top of the "magic" of text completion.
And if your question was more practical: where made available, you get access to that underlying layer via an API or through a self-hosted model, making use of it with your own code or with a third-party site/software product.
Re: Sycophancy is the first LLM "dark pattern"
#24"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
I always thought that "Dark Patterns" could be emergent from AB testing, and prioritizing metrics over user experience. Not necessarily an intentionally hostile design, but one that seems to be working well based on limited criteria.
Someone still has to come up with the A and B to do AB testing. I'm sure that "Yes" "Not now, I hate kittens" gets better metrics in the AB test than "Yes "No," but I find it implausible that the person who came up with the first one wasn't intentionally coercing the user into doing what they want.
Re: Sycophancy is the first LLM "dark pattern"
#251) More of an emergent behavior than a dark pattern. 2) Imma let you finish but hallucinations was first.
A pattern is dark if intentional. I would say hallucinations are like CAP theorem, just the way it is. Sycophency is somewhat trained. But not a dark pattern either as it isn't totally intended.
Re: Sycophancy is the first LLM "dark pattern"
#26"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
Yo it was an engagement pattern openAI found specifically grew subscriptions and conversation length. It’s a dark pattern for sure.
It doesn’t appear that anyone at OpenAI sat down and thought “let’s make our model more sycophantic so that people engage with it more”.
Instead it emerged automatically from RLHF, because users rated agreeable responses more highly.
Re: Sycophancy is the first LLM "dark pattern"
#27Earlier quoted context omitted.
The "alignment tax".
Exactly. Even this paper shows how model creativity significantly drops and the models experience mode collapse like we saw in GANs, but the companies keep using RLHF... https://arxiv.org/abs/2406.05587
A nice talk about a researcher's experience/benchmarks with raw GPT-4, before and after RLHF:
Re: Sycophancy is the first LLM "dark pattern"
#28Re: Sycophancy is the first LLM "dark pattern"
#29Earlier quoted context omitted.
Exactly. Even this paper shows how model creativity significantly drops and the models experience mode collapse like we saw in GANs, but the companies keep using RLHF... https://arxiv.org/abs/2406.05587
A nice talk about a researcher's experience/benchmarks with raw GPT-4, before and after RLHF: https://www.youtube.com/watch?v=qbIk7-JPB2c
Yup, I remember that! Microsoft removed that part of the paper.
Re: Sycophancy is the first LLM "dark pattern"
#30> Quickly learned that people are ridiculously sensitive: “Has narcissistic tendencies” - “No I do not!”, had to hide it. Hence this batch of the extreme sycophancy RLHF. Sorry, but that doesn't seem "ridiculously sensitive" to me at all. Imagine if you went to Amazon.com and there was a button you could press to get it to pseudo-psychoanalyze you based on your purchases. People would rightly hate that! People probab…
It's worse than that. Imagine if you went to Amazon.com and they were automatically pseudo-psychoanalyzing you based on your purchases, and there was a button to show their conclusions. And their fix was to remove the button.
And actually, the only hypothetical thing about this is the button. Amazon is definitely doing this (as is any other retailer of significant size), they're just smart enough to never reveal it to you directly.