Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Expanding on what we missed with sycophancy

openai.com

Post reply on HN
Print view View original

251–260 of 297 posts

Re: Expanding on what we missed with sycophancy

#251

Post by Xmd5a »

earnestinger wrote: post #236
skydhash wrote: post #209

Earlier quoted context omitted.

A naive thought: What you would get if you hardcode the language grammar and not let the training discern it, so instead of it, kinda like an expert system constraining its output?

What kind of hard coding do you have in mind? How the technique would look like?

word2vec meets category theory

https://en.wikipedia.org/wiki/DisCoCat

>In this post, we are going to build a generalization of Transformer models that can operate on (almost) arbitrary structures such as functions, graphs, probability distributions, not just matrices and vectors.

https://cybercat.institute/2025/02/12/transformers-applicati...

Re: Expanding on what we missed with sycophancy

#252

Post by bondarchuk »

cgio wrote: post #157
alickz wrote: post #135

Earlier quoted context omitted.

>There's nothing to argue about, it's unambiguously not real empathy I think if a person can't tell the difference between empathy from a human vs empathy from a chatbot, it's a difference without a distinction If it activates the same neural pathways, and has the same results, then I think the mind doesn't care >One reason human empathy is nice is because it is often followed up with actions. Friends who care about…

Empathy does not lie in its perception on receipt but in its inception as a feeling. It is fundamentally a manifestation of the modalities enabled in shared experience. As such impossible to the extent that our experiences are not compatible with those of an intelligence that does not put emphasis on lived context trying to substitute for it with offline batch learning. Understanding is possible in this relationship,…

> As such impossible to the extent that our experiences are not compatible with those of an intelligence that does not put emphasis on lived context trying to substitute for it with offline batch learning.

Conversely that means empathy is possible to the extent that our experiences are compatible with those of an AI. That is precisely what's under consideration here and you have not shown that it is zero.

an intelligence that does not put emphasis on lived context trying to substitute for it with offline batch learning.

Will change your tune when online learning comes along?

Re: Expanding on what we missed with sycophancy

#253

Post by [deleted] »

dns_snek wrote: post #247
danenania wrote: post #156

Earlier quoted context omitted.

We're talking about psychology, therapy, sycophancy, etc. None of this is empirical. If someone thinks they can, say, create a billion dollar startup, whether they can really do it or not is a subjective determination. The AI might tell the person they can do it. You might tell them they can't, that the AI is sycophantic, and that they should stop talking to it because they're losing touch with reality. But is the AI…

That's easy. What makes someone a sycophant, by definition, is that their encouragement and flattery is unconditional and completely disconnected from any sort of realistic consideration of your ideas. You can't judge whether LLM is acting like a sycophant without reading the conversation, and you can't judge whether a human is being an irrational pessimist without having the full context. Are they a highly intellige…

[deleted]

Re: Expanding on what we missed with sycophancy

#254

Post by Viliam1234 »

rurp wrote: post #114
alickz wrote: post #92

Earlier quoted context omitted.

>I think most smart people overestimate the intelligence of others for a variety of reasons so they overestimate what it would take for a LLM to beat the output of an average person. I think most people also _vastly_ overestimate how much positive attention the average person gets in their lives It wouldn't surprise me if, for most people, ChatGPT offers them more empathy and understanding than _anyone_ else _ever ha…

>We can argue over whether or not it's "real" empathy There's nothing to argue about, it's unambiguously not real empathy. Empathy from a human exists in a much broader context of past and future interactions. One reason human empathy is nice is because it is often followed up with actions. Friends who care about you will help you out in material ways when you need it. Even strangers will. Someone who sees a person s…

> Friends who care about you will help you out in material ways when you need it.

LLMs help me write code, do various paperwork, etc. Going by this criterion, they already are true friends.

Re: Expanding on what we missed with sycophancy

#255

Post by quuxplusone »

cgio wrote: post #157
alickz wrote: post #135

Earlier quoted context omitted.

>There's nothing to argue about, it's unambiguously not real empathy I think if a person can't tell the difference between empathy from a human vs empathy from a chatbot, it's a difference without a distinction If it activates the same neural pathways, and has the same results, then I think the mind doesn't care >One reason human empathy is nice is because it is often followed up with actions. Friends who care about…

Empathy does not lie in its perception on receipt but in its inception as a feeling. It is fundamentally a manifestation of the modalities enabled in shared experience. As such impossible to the extent that our experiences are not compatible with those of an intelligence that does not put emphasis on lived context trying to substitute for it with offline batch learning. Understanding is possible in this relationship,…

I happen to agree with what you said. (Paraphrasing: A machine cannot have "real empathy" because a machine cannot "feel" in general.) But I think you're arguing a different point from the grandparent's. rurp said:

> Someone who sees a person stranded on the side of a road might feel for them and stop to lend a hand. ChatGPT will never do that [...]

Now, on the one hand that's because ChatGPT cannot "see a person" nor "stop [the car]"; it communicates only by text-in, text-out. (Although it's easy to input text describing that situation and see what text ChatGPT outputs!) GP says it's also because "the purpose of ChatGPT is to make immense amounts of money and power for its owners [, not to help others]." I took that to mean that GP was saying that even if a LLM was controlling a car and was able to see a person in trouble (or a tortoise on its back baking in the sun, or whatever), then it still would not stop to help. (Why? Because it wouldn't empathize. Why? Because it wasn't created to empathize.)

I take GP to be arguing that the LLM would not help; whereas I take you to be arguing that even if the LLM helped, it would by definition not be doing so out of empathy. Rather, it would be "helping"[1] because the numbers forced it to. I happen to agree with that position, but I think it's significantly different from GP's.

Btw, I highly recommend Geoffrey Jefferson's essay "The Mind of Mechanical Man" (1949) as a very clear exposition of the conservative position here.

[1] — One could certainly argue that the notions of "help" and "harm" likewise don't apply to non-intentional mechanistic forces. But here I'm just using the word "helping" as a kind of shorthand for "executing actions that caused better-than-previously-predicted outcomes for the stranded person," regardless of intentionality. That shorthand requires only that the reader is willing to believe in cause-and-effect for the purposes of this thread. :)

Re: Expanding on what we missed with sycophancy

#256

Post by 34679 »

93po wrote: post #196
edent wrote: post #71

Earlier quoted context omitted.

There are several threads on Reddit. For example https://www.reddit.com/r/ChatGPT/comments/1kalae8/chatgpt_in... Perhaps everyone there is LARPing - but if you start typing stereotypical psychosis talk into ChatGPT, it won't be long before it starts agreeing with your divinity.

reddit is overwhelmingly fake content, like a massive percentage of it. a post on reddit these days is not actually evidence of anything real, at all

I take issue with the qualifier "these days". On day one, it was mostly fake accounts set up by the founders.

https://m.economictimes.com/magazines/panache/reddit-faked-i...

Re: Expanding on what we missed with sycophancy

#257

Post by seeEllArr »

kubb wrote: post #81
NoboruWataya wrote: post #17

I found the recent sycophancy a bit annoying when trying to diagnose and solve coding problems. First it would waste time praising your intelligence for asking the question before getting to the answer. But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case. I guess part of the…

If it can replace programmers, why wouldn't it be able to replace therapists?

[dead]

Re: Expanding on what we missed with sycophancy

#258

Post by malfist »

tsimionescu wrote: post #244
conception wrote: post #192

Earlier quoted context omitted.

They have already introduced ads btw. “The "Enshittification" has arrived I asked ChatGPT about the impact of the current tarrifs on inventories over the next few months. It returned a long list of links to toiletries I might want to buy. I asked it why it did that. It replied: "As of April 28, 2025, OpenAl introduced new shopping features to ChatGPT, enhancing its capabilities to provide product recommendations comp…

And did that actually happen, or is it just some garbage that it generated? It sounds plausible, of course, but that's exactly what LLMs are best at.

Talk to it and find out. Seriously.

I'd say 40% of my conversations end with it suggesting I ask it to find products for me. For example I was trying to determine if I need to screw down stair nosing or if glue and nails were enough and it wanted to help me find screws

Re: Expanding on what we missed with sycophancy

#259

Post by cjbgkagh »

taurath wrote: post #241
cjbgkagh wrote: post #40

Earlier quoted context omitted.

Underestimating the complexity of other fields is not mutually exclusive with overestimating the intelligence of others. The real issue is that society is very stratified so smart people are less likely to interact with regular people, especially in circumstances where the intelligence of the regular person could become obvious. I don’t see there being an insurmountable barrier that would prevent LLMs from doing the…

Most human intelligence is within a fairly narrow band. Most people I’ve ever met have their own unique intelligences. Perhaps it might be good to meet more people without holding the self-looping dichotomy of “smart people” vs “normal people”. In my experience it tends to lead to huge cognitive errors.

As mentioned I’m in the rather unique position to have analyzed the complete browser history for a substantial number of people - I have learned far more than I wished to.

The behaviors of very high IQ people are rather distinct from regular IQ people due to IQ being both largely generic and those genes having other comorbidities. Most obviously is the depression, anxiety, and bipolar disorders. This is so obvious that even regular researchers have uncovered it.

I think what happens to many people is they confuse their desired reality with actual reality by looking at everything through a tinted lens. In being a data scientist in pursuit of actual reality I’ve had my desired reality repeatedly challenged far more than a person not in this industry. My desired reality was that intelligence was more common and I believed that until it was shown to me in data that I was wrong.

Re: Expanding on what we missed with sycophancy

#260

Post by cjbgkagh »

istjohn wrote: post #216
cjbgkagh wrote: post #193

Earlier quoted context omitted.

They do seem close to being able to, I'm mostly waiting on someone to provide it as a service - I don't have time for side projects ATM.

It's not a project. Just open Claude or ChatGPT and tell it what you want to do. You could copy/paste your comment above. If you want to get slightly more fancy, use Google's AIStudio with a simple system prompt.

I would like to do it semi-passively through talking while I’m doing other things like going for a walk, doing chores etc. I don’t have the time at the moment to sit down and focus on this.
Post reply on HN