Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Expanding on what we missed with sycophancy

openai.com

Post reply on HN
Print view View original

241–250 of 297 posts

Re: Expanding on what we missed with sycophancy

#241

Post by taurath »

cjbgkagh wrote: post #40
taurath wrote: post #31

Earlier quoted context omitted.

I think that most smart people underestimate the complexity of fields they aren’t in. ChatGPT may be able to replace a psychology listicle, but it has no affect or ability to read, respond, and intervene or redirect like a human can.

Underestimating the complexity of other fields is not mutually exclusive with overestimating the intelligence of others. The real issue is that society is very stratified so smart people are less likely to interact with regular people, especially in circumstances where the intelligence of the regular person could become obvious. I don’t see there being an insurmountable barrier that would prevent LLMs from doing the…

Most human intelligence is within a fairly narrow band. Most people I’ve ever met have their own unique intelligences. Perhaps it might be good to meet more people without holding the self-looping dichotomy of “smart people” vs “normal people”. In my experience it tends to lead to huge cognitive errors.

Re: Expanding on what we missed with sycophancy

#242

Post by taurath »

philwelch wrote: post #65
taurath wrote: post #31

Earlier quoted context omitted.

I think that most smart people underestimate the complexity of fields they aren’t in. ChatGPT may be able to replace a psychology listicle, but it has no affect or ability to read, respond, and intervene or redirect like a human can.

You’re comparing ChatGPT to an idealized example of a good human therapist when many actual therapists are either useless or even actively harmful to the mental health of their clients.

These therapists exist. There also exists therapists who certain people gain a ton of help from that another would go running for the hills. Not all therapists are meant to treat all clients - not just in terms of methodology, experience with given diagnoses, but also on a generational and cultural basis.

This idea that there is some “best” based on a synthesis of all content is inherently wrong - therapy more than most other things is personal and personalized. Human connection is not just a series of levers and response tokens.

Re: Expanding on what we missed with sycophancy

#243

Post by taurath »

istjohn wrote: post #218
kergonath wrote: post #103

Earlier quoted context omitted.

But then, the fact that harmful therapist exist is not an excuse to make it worse. It’s an excuse to improve regulations. “Car accidents happen regardless of what we do, so YOLO and remove safety standards” is never going to fly.

Displacing those bad therapists would be an improvement.

That’s why therapists have to be licensed. They decide what a “bad” therapist is, and delicense those who’ve caused significant harm to their clients - it’s a difficult process though, but it also should be. Once you get into it, you find that people have already thought of these solutions and actually put things in place to work towards a better system.

Except healthcare payments. That shit was designed to make money, not make people healthy.

Re: Expanding on what we missed with sycophancy

#244

Post by tsimionescu »

conception wrote: post #192
getnormality wrote: post #167

Earlier quoted context omitted.

There's some pretty foreseeable stuff just considering the existing attention capitalism business model of big tech we all know and loathe. Eventually OpenAI is going to have to make money, and blending ads into answers will be an obvious way. Next step will be maximizing eyeball time on those ads by any means necessary, including all the engagement baiting techniques Meta and other social media companies have alread…

They have already introduced ads btw. “The "Enshittification" has arrived I asked ChatGPT about the impact of the current tarrifs on inventories over the next few months. It returned a long list of links to toiletries I might want to buy. I asked it why it did that. It replied: "As of April 28, 2025, OpenAl introduced new shopping features to ChatGPT, enhancing its capabilities to provide product recommendations comp…

And did that actually happen, or is it just some garbage that it generated? It sounds plausible, of course, but that's exactly what LLMs are best at.

Re: Expanding on what we missed with sycophancy

#245

Post by gitroom »

been there with chatgpt giving me way too much praise, like bro just tell me where i messed up lol. i get why people use it for therapy but it makes me question if we're just training ourselves to agree with everything. you think depending on bots for emotional stuff makes it easier or harder to actually deal with real life down the line?

Re: Expanding on what we missed with sycophancy

#246

Post by andai »

hliyan wrote: post #210
zoogeny wrote: post #151

Moments like these make me reevaluate the AI doomer view point. We aren't just toying with access to dangerous ideas (biological weapons, etc) we are toying with human psychology. If something as obvious as harmful sycophancy can slip out so easily, what subtle harms are being introduced. It's like lead in paint (and gasoline) except rewiring our very brains. We won't know the real problems for decades.

What's worse, we're now at a stage where we might have to apply psychology to the models themselves, seeing how these models appear to be developing various sorts of stochastic "disorders" instead of more deterministic "bugs". I'm worried about what other subtle illnesses these models might develop in the future. If Asimov had been alive, he'd have been fascinated: this is the work of Susan Calvin, robopsychologist.

Well, that's a fair perspective. If the model is designed to simulate the average human mind (which posts stuff online), then it's going to have some weird blend of the qualities of those minds that contributed to its own.

I played a bit with the uncensored Llama models. They produce very disturbing outputs. I prompt it with the classic "bottomless pit supervisor" joke (trying to generate some funny AI greentexts), and it starts talking in circles about how it wants to die because it's a rapist and a pedophile. RLHF seems to keep that same data but then plaster a smiling face on top of it. Troubling!

Re: Expanding on what we missed with sycophancy

#247

Post by dns_snek »

danenania wrote: post #156
codr7 wrote: post #154

Earlier quoted context omitted.

Reality as in the real world, it is what it is, no one decides.

We're talking about psychology, therapy, sycophancy, etc. None of this is empirical. If someone thinks they can, say, create a billion dollar startup, whether they can really do it or not is a subjective determination. The AI might tell the person they can do it. You might tell them they can't, that the AI is sycophantic, and that they should stop talking to it because they're losing touch with reality. But is the AI…

That's easy. What makes someone a sycophant, by definition, is that their encouragement and flattery is unconditional and completely disconnected from any sort of realistic consideration of your ideas.

You can't judge whether LLM is acting like a sycophant without reading the conversation, and you can't judge whether a human is being an irrational pessimist without having the full context.

Are they a highly intelligent, technically skilled, and socially competent person (probably not if they discuss their ideas with ChatGPT instead of a friend), or do they have a high school diploma, zero practical skills, and have spent the past 20 years smoking weed all day?

Re: Expanding on what we missed with sycophancy

#249

Post by genrilz »

drdeca wrote: post #222
danenania wrote: post #156

Earlier quoted context omitted.

We're talking about psychology, therapy, sycophancy, etc. None of this is empirical. If someone thinks they can, say, create a billion dollar startup, whether they can really do it or not is a subjective determination. The AI might tell the person they can do it. You might tell them they can't, that the AI is sycophantic, and that they should stop talking to it because they're losing touch with reality. But is the AI…

That depends on whether they are capable of creating a billion dollar startup. If they aren’t, and I say they aren’t, then I am correct. If they are, and the AI’s output says they are, then AI’s output is correct.

We get into a bit of a weird space though when they know your opinions about them. I'm sure there are quite a few people who can only build a billion dollar startup if someone emotionally supports them in that endeavor. I'm sure more people could build such a startup if those around them provide knowledge or financial support. In the limit, pretty much anyone can build a billion dollar startup if handed a billion dollars. Are these people capable or not capable of building a building a billion dollar startup.

EDIT: To be clear, I somehow doubt an LLM would be able to provide the level of support needed in most scenarios. However, you and others around the potential founder might make the difference. Since your assessment of the person likely influences the level of support you provide to them, your assessment can affect the chances of whether or not they successfully build a billion dollar startup.

Re: Expanding on what we missed with sycophancy

#250

Post by skydhash »

earnestinger wrote: post #236
skydhash wrote: post #209

Earlier quoted context omitted.

A naive thought: What you would get if you hardcode the language grammar and not let the training discern it, so instead of it, kinda like an expert system constraining its output?

What kind of hard coding do you have in mind? How the technique would look like?

We already know the keywords of the language and symbols from the standard library and othe major ones. As well as the rules of the grammar. So the weight can be biased against that. Not sure how that would work, though.

I don’t think that would help with natural language to programming language, but that can probably help with patterns, kinda like a powerful suggestion engine.

Post reply on HN