Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Expanding on what we missed with sycophancy

openai.com

Post reply on HN
Print view View original

281–290 of 297 posts

Re: Expanding on what we missed with sycophancy

#281

Post by motbus3 »

NoboruWataya wrote: post #17

I found the recent sycophancy a bit annoying when trying to diagnose and solve coding problems. First it would waste time praising your intelligence for asking the question before getting to the answer. But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case. I guess part of the…

I found that posing questions and possible causes/solutions/options/examples completely change the output even in cases where none of the options is correct. OpenAI models for me tend to be quite positive bending rules or logic to answer positively to whatever you pose.

Re: Expanding on what we missed with sycophancy

#282

Post by mieubrisse »

zoogeny wrote: post #151

Moments like these make me reevaluate the AI doomer view point. We aren't just toying with access to dangerous ideas (biological weapons, etc) we are toying with human psychology. If something as obvious as harmful sycophancy can slip out so easily, what subtle harms are being introduced. It's like lead in paint (and gasoline) except rewiring our very brains. We won't know the real problems for decades.

To provide a different perspective: this was caught, acknowledged as a problem, rolled back quickly, and written up in a post-mortem.

Given that nothing can be done perfectly, this seems like the best way OpenAI could have handled it.

What if building AI is like airplanes: everybody's afraid of it, so the industry gets really good at rolling lessons into the product and the end result is actually fine?

Re: Expanding on what we missed with sycophancy

#283

Post by cjbgkagh »

taurath wrote: post #280
cjbgkagh wrote: post #259

Earlier quoted context omitted.

As mentioned I’m in the rather unique position to have analyzed the complete browser history for a substantial number of people - I have learned far more than I wished to. The behaviors of very high IQ people are rather distinct from regular IQ people due to IQ being both largely generic and those genes having other comorbidities. Most obviously is the depression, anxiety, and bipolar disorders. This is so obvious th…

What gets me the most about this is how certain you are that because you have some data about people’s browser history you can make these conclusions. I hope age and experience avails you of these notions - there is no master database that will tell you the nature of people. All you have is the data you have - I’m sure there are many interesting insights and confounding results, but it’s foolish to think you can know…

I think it is you who is blinded by your prior biases. It's not that I have 'some data', it was a mountain of data, and it was my full time job to analyze it specifically for behavioral information with a number of other researchers also doing the same, and we'd compare notes with other research orgs in other companies. I have a feeling that I've thought about this topic far more than you have.

At this point we'll just have to agree to disagree.

Re: Expanding on what we missed with sycophancy

#284

Post by labrador »

malfist wrote: post #274
labrador wrote: post #175

Earlier quoted context omitted.

see my other comments about the trustworthiness about asking a chat system how it's internals work. They have reason to be cagey.

Your personifying a statistical engine. LLMs aren't cagey. They can't be.

I'm not. Translation: "the statistical engine has been tuned to act cagey about revealing it's internal operation"

Re: Expanding on what we missed with sycophancy

#285

Post by pixl97 »

bluefirebrand wrote: post #271
hackable_sand wrote: post #267

Earlier quoted context omitted.

You can brute force a problem without understanding the problem.

Would you be kind enough to give an example of this? I disagree, but I'm open to being convinced I just don't think you can solve a problem in any way without understanding it first, at least at a high level I think if you know a solution you want to reach then you can work backwards from it in a brute force way to reach the solution you want But I don't think you can take a problem where you aren't sure what the sol…

What I think we have here is a problem with definitions and scope.

For example does your definition of problem demand a problem only exists if a human consciously thinks of it?

If no, then almost every single 'problem' that's ever existed was solved by the random walk of genetics. Nature has 'solved' some of the most difficult energy conservation optimizations in living systems with nothing but time and randomness. Think of how many problems had to be solved over time to go from an animal that swims to one that flys?

Re: Expanding on what we missed with sycophancy

#286

Post by labrador »

grey-area wrote: post #212
labrador wrote: post #173

Earlier quoted context omitted.

Point taken. I admit my comment was silly the way I worded it. Here's the line I’m trying to walk: When I ask ChatGPT about its own internal operations, is it giving me the public info about it's operation, and also possibly revealing propreitary info, or making things up obfuscate and preserve the illusion of authority? Or all three?

Personally I don’t think it has agency so cannot be described as trying to do anything. It’s predicting what seems most likely as a description given its corpus (and now what you’d like to hear) and giving you that. The truth is not really something it knows, though it’s very good at giving answers that sound like it knows what it’s talking about. And yes if it doesn’t have an answer from its corpus it’ll just make t…

You're right, but tbh, we had that discussion last year. I'm talking about my relationship to it. The "relationship" being the whole being greater than the sum of the parts. That's the 2025 take on it.

Re: Expanding on what we missed with sycophancy

#287

Post by xigency »

jagger27 wrote: post #19
tunesmith wrote: post #14

I find it disappointing that openai doesn't really mention anything here along the lines of having an accurate model of reality. That's really what the problem is with sycophancy, it encourages people to detach themselves from what reality is. Like, it seems like they are saying their "vibe check" didn't check vibes enough.

The reality distortion field within OpenAI is literally where these models grew up. It's like an out of touch rich kid.

I've applied to multiple A.I. companies now with the value proposition that I am grounded and skeptical (along with being a tenured technologist with a background in risk management) and they weren't having it.

Free lunch for the long view thinkers, whenever they decide to show up.

Re: Expanding on what we missed with sycophancy

#288

Post by motoxpro »

lblume wrote: post #229
motoxpro wrote: post #225

The wild thing was this is what people "wanted" - The agreeableness came from heavier weighted user signals, not from OpenAI. The users give better feedback when the answers positively respond to their questions. - The most common use case for AI is a therapist. - The reward signals (user feedback) being positive means that the response helped them emotionally. Which is easy to imagine someone asking a hard question…

> The most common use case for AI is a therapist. Source? Sounds like a fairly wild claim.

The HBR article I linked.

Re: Expanding on what we missed with sycophancy

#289

Post by motoxpro »

photonthug wrote: post #232
motoxpro wrote: post #225

The wild thing was this is what people "wanted" - The agreeableness came from heavier weighted user signals, not from OpenAI. The users give better feedback when the answers positively respond to their questions. - The most common use case for AI is a therapist. - The reward signals (user feedback) being positive means that the response helped them emotionally. Which is easy to imagine someone asking a hard question…

> Who is anyone to say that this use case is wrong? So when faced with an important question, you've decided to opt for an indifferent shrug, which is frustrating. But the usual answers apply here so you're in luck. As with most other things.. one should probably consult with experts or practitioners in the field, governments, family or human friends you trust, or shit just poll society in general if you don't trust…

With that argument, you should use a “real” lawyer, or coder, or designer, or strategy consultant.

Instead of an indifferent shrug, I would argue that paying attention to the fact that all of the things you mentioned are available now and people STILL are seeking this out goes to show that it’s a real problem.

As with most problems, the usual answers apply, people have made the choice already, you can either have the empathy to meet them where they are and understand WHY they are doing it, or you can dismiss it and suggest a perfect solution that is unrealistic.

Re: Expanding on what we missed with sycophancy

#290

Post by qwery »

Henchman21 wrote: post #269
kadushka wrote: post #73

Earlier quoted context omitted.

I've never ever thought about needing a therapist. Don't remember anyone in my circle who had ever mentioned it. Similar to how I don't remember anyone going to a palm reader. I'm not trying to diss either profession, I'm sure someone benefits from them, it's just not for me. And I'm sure I'm pretty average in terms of emotional intelligence or psychological issues. Who are all those people who need professional ther…

Reading this the next day, my takeaway is that while you may not have intended to “diss”, you sure as fuck did it anyway. You directly compared therapy to snake oil (palm reading). Then by implication determined that since you were average (and hence normal) no one else would need therapy unless they were, by implication abnormal. I don’t believe I have ever been so thoroughly insulted online ever. Well done.

Thanks for posting, I was struggling to articulate more or less that.

(I welcome the increased skepticism on display on this post, but there's skepticism and then there's ... that)

Post reply on HN