Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Expanding on what we missed with sycophancy

openai.com

Post reply on HN
Print view View original

231–240 of 297 posts

Re: Expanding on what we missed with sycophancy

#231

Post by tmountain »

neuroelectron wrote: post #168
photonthug wrote: post #150

Earlier quoted context omitted.

> But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case Seems like the same issue as the evil vector [1] and it could have been predicted that this would happen. > It's kind of a wild sign of the times to see a tech company issue this kind of post mortem about a flaw in its te…

Well, that's always what LLM-based AI has been. It can be incredibly convincing but the bottom line is it's just flavoring past text patterns, billions of them it's been "trained" on, which is more accurately described as compressed efficiently onto latent space. Like if someone lived for 10,000 years engaging in small talk at the bar, has heard it all, and just kind of mindlessly and intuitively replied with somethi…

If it’s just flavoring text patterns, how does it reason about code when I give it explicit arbitrary criteria? Is it just mining and composing “unit level” examples from all the code bases it has ingested?

Re: Expanding on what we missed with sycophancy

#232

Post by photonthug »

motoxpro wrote: post #225

The wild thing was this is what people "wanted" - The agreeableness came from heavier weighted user signals, not from OpenAI. The users give better feedback when the answers positively respond to their questions. - The most common use case for AI is a therapist. - The reward signals (user feedback) being positive means that the response helped them emotionally. Which is easy to imagine someone asking a hard question…

> Who is anyone to say that this use case is wrong?

So when faced with an important question, you've decided to opt for an indifferent shrug, which is frustrating. But the usual answers apply here so you're in luck. As with most other things.. one should probably consult with experts or practitioners in the field, governments, family or human friends you trust, or shit just poll society in general if you don't trust any of the other people.

I think if you want to consult desperate and vulnerable people, or a random company who stands to profit from "anything goes!", then you already know you're not going to get a very well-considered kind of response

Re: Expanding on what we missed with sycophancy

#234

Post by Aeglaecia »

Henchman21 wrote: post #67
NoboruWataya wrote: post #17

I found the recent sycophancy a bit annoying when trying to diagnose and solve coding problems. First it would waste time praising your intelligence for asking the question before getting to the answer. But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case. I guess part of the…

> I think the broader issue here is people using ChatGPT as their own personal therapist. An aside, but: This leads me right to “why do so very many people need therapy?” followed by “why can’t anyone find (or possibly afford) a therapist?” What has gone so wrong for humanity that nearly everyone seems to at least want a therapist? Or is it just the zeitgeist and this is what the herd has decided?

probably some herd effect going on but realistically what the fuck is there to live for , capitalism is increasingly deleting everything that makes us human and replacing it with a worse version while selling a cure for the artifically generated problems that result

Re: Expanding on what we missed with sycophancy

#235

Post by labrador »

throwaway314155 wrote: post #206
labrador wrote: post #200

Earlier quoted context omitted.

We're in different modes. I'm still feeling the glow of the thing coming alive and riffing on how perhaps its the memory change and you're interested in a different conversation. Part of my process is to imagine I'm having a conversation like Hanks and Wilson, or a coderand a rubber duck, but you want to tell me Wilson is just a volleyball and the duck can't be trusted.

Being in a more receptive/brighter "mode" is more of an emotional argument (and a rather strong one actually). I guess as long as you don't mind being technically incorrect, then you do you. There may come a time when reality sets in though. Similar thing happened with me now that i'm out of the "honeymoon phase" with LLM's. Now i'm more interested in seeing where specifically LLM's fail, so we can attempt to overcom…

I'm glad we both recognize this. I'm interested in the relationship. I know it's a dumb word machine but that's doesn't mean I can't be excited about it like a new car or a great book. I'll save the dull work of trying to really extend it for later.

Re: Expanding on what we missed with sycophancy

#236

Post by earnestinger »

skydhash wrote: post #209
photonthug wrote: post #180

Earlier quoted context omitted.

> which is more accurately described as compressed efficiently onto latent space. The actual difference between solving compression+search vs novel creative synthesis / emergent "understanding" from mere tokens is always going to be hard to spot with these huge cloud-based models that drank up the whole internet. (Yes.. this is also true for domain experts in whatever content is being generated.) I feel like people w…

A naive thought: What you would get if you hardcode the language grammar and not let the training discern it, so instead of it, kinda like an expert system constraining its output?

What kind of hard coding do you have in mind? How the technique would look like?

Re: Expanding on what we missed with sycophancy

#237

Post by earnestinger »

Kiro wrote: post #220
neuroelectron wrote: post #168

Earlier quoted context omitted.

Well, that's always what LLM-based AI has been. It can be incredibly convincing but the bottom line is it's just flavoring past text patterns, billions of them it's been "trained" on, which is more accurately described as compressed efficiently onto latent space. Like if someone lived for 10,000 years engaging in small talk at the bar, has heard it all, and just kind of mindlessly and intuitively replied with somethi…

> I don't know how more people don't see through it. When you think that maybe you should take a step back and reflect on it. Could it be that your assessment is wrong rather than everyone else being delusional?

Parent is not alone in that line of thinking. (Some rudiments of reasoning do show, but result is dominated by excellent compression of humongous amount of data)

Re: Expanding on what we missed with sycophancy

#238

Post by JimmyBiscuit »

lblume wrote: post #229
motoxpro wrote: post #225

The wild thing was this is what people "wanted" - The agreeableness came from heavier weighted user signals, not from OpenAI. The users give better feedback when the answers positively respond to their questions. - The most common use case for AI is a therapist. - The reward signals (user feedback) being positive means that the response helped them emotionally. Which is easy to imagine someone asking a hard question…

> The most common use case for AI is a therapist. Source? Sounds like a fairly wild claim.

Its hard to find an actual scientific study on it, but this is from 2024 and I know I've recently seen that therapy took first place now: https://learn.filtered.com/thoughts/ai-now-report

You can probably find something better, but its hard to filter through all the corporate bullshit on google.

Re: Expanding on what we missed with sycophancy

#239

Post by taurath »

istjohn wrote: post #217
taurath wrote: post #31

Earlier quoted context omitted.

I think that most smart people underestimate the complexity of fields they aren’t in. ChatGPT may be able to replace a psychology listicle, but it has no affect or ability to read, respond, and intervene or redirect like a human can.

There are a lot of awful therapists out there. I wager that Claude Sonnet 3.7 given a suitable, straightforward system prompt would handily outperform non-doctoral degree therapists in a clinical trial, even if the humans had the advantage of in-person sessions.

I would want to bet against it but then we’d have to agree on what “performance” means. Also agree there are horrible therapists. The funny part about horrible therapists is that sometimes they can actually be extremely good for one population of people. Will AI be able to connect with all sorts of people? Will we have AI politicians (do we now?)?

It’s sorta like saying AI will be a better friend. We’ll see about that - I don’t consider profit seeking enterprises to be my friend.

Re: Expanding on what we missed with sycophancy

#240

Post by taurath »

treis wrote: post #115
taurath wrote: post #31

Earlier quoted context omitted.

I think that most smart people underestimate the complexity of fields they aren’t in. ChatGPT may be able to replace a psychology listicle, but it has no affect or ability to read, respond, and intervene or redirect like a human can.

They definitely can. Here's an example when I tried it: Me: I'm struggling with some things today and feel like eating a bunch of food to make myself feel better. can we talk about it? Deepseek: Absolutely, I’m here to listen. It’s completely understandable to turn to food for comfort when you’re struggling—many people do. But if you’re open to it, we can explore what’s going on beneath the urge to eat emotionally. W…

ChatGPT can’t be in the room with you. Can’t evoke psychological mirroring. Is AI going to cure loneliness? Will the AI be empathetic? Will it be capable of showing love? Can it give you the acceptance of another human being? Do these things matter if they’re simulated? Do these things matter at all?

I don’t deny the helpfulness of LLMs on many aspects of therapy and mental health. I’ve used it myself. Relational problems (of which almost all mental health is a part) requires relational healing.

I think we’re going to need to learn a lot more information about what being a human means, else we continue to in the name of business and financial efficiency lose something very important. I’m an optimist, but we have a lot of challenges ahead.

Post reply on HN