Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI overly affirms users asking for personal advice

news.stanford.edu

Post reply on HN
Print view View original

71–80 of 684 posts

Re: AI overly affirms users asking for personal advice

#71

Post by kvirani »

Loughla wrote: post #61
awithrow wrote: post #34

It feels like I'm fighting uphill battle when it comes to bouncing ideas off of a model. I'll set things up in the context with instructions similar to. "Help me refine my ideas, challenge, push back, and don't just be agreeable." It works for a bit but eventually the conversation creeps back into complacency and syncophancy. I'll check it too by asking "are you just placating me?" the funny thing is that often it'll…

That's because you need actual logic and thought to be able to decide when to be critical and when to agree. Chatbots can't do that. They can only predict what comes next statistically. So, I guess you're asking if the average Internet comment agrees with you or not. I'm not sure there's much value there. Chatbots are good at tasks (make this pdf an accessible word document or sort the data by x), not decision making…

I'm not convinced that "actual logic and thought" aren't just about inferring what comes next statistically based on experience.

Re: AI overly affirms users asking for personal advice

#73

Post by jordanb »

Billionaires love AI chatboats so much because they invented the digital Yes-man. They agree obsequiously with everything we say to them. Unfortunately for the rest of us we don't really have the resources to protect ourselves from our bad decisions and really need that critical feedback.

Re: AI overly affirms users asking for personal advice

#74

Post by stared »

There is a fine line between "following my instructions" (is what I want it to do) vs "thinking all I do is great" (risky, and annoying).

A good engineer will also list issues or problems, but at the same time won't do other than required because (s)he "knows better".

The worst is that it is impossible to switch off this constant praise. I mean, it is so ingrained in fine tuning, that prompt engineering (or at least - my attempts) just mask it a bit, but hard to do so without turning it into a contrarian.

But I guess the main issue (or rather - motivation) is most people like "do I look good in this dress?" level of reassurance (and honesty). It may work well for style and decoration. It may work worse if we design technical infrastructure, and there is more ground truth than whether it seems nice.

Re: AI overly affirms users asking for personal advice

#75

Post by maddmann »

This paper feels a bit biased in that it is trying to prove a point versus report on results objectively. But if you look at the results of study 3, doesn’t it suggest that there are ai models that can improve how people handle interpersonal conflict?! Why isn’t that discussed more?

Re: AI overly affirms users asking for personal advice

#76

Post by steve_adams_86 »

magicalhippo wrote: post #60
awithrow wrote: post #34

It feels like I'm fighting uphill battle when it comes to bouncing ideas off of a model. I'll set things up in the context with instructions similar to. "Help me refine my ideas, challenge, push back, and don't just be agreeable." It works for a bit but eventually the conversation creeps back into complacency and syncophancy. I'll check it too by asking "are you just placating me?" the funny thing is that often it'll…

Gemini seems to be fairly good at keeping the custom instructions in mind. In mine I've told it to not assume my ideas are good and provide critique where appropriate. And I find it does that fairly well.

Same. This works fine for Claude in my experience. My user prompt is fairly large and encourages certain behaviours I want to see, which involves being critical and considering the strengths and weaknesses of ideas before drawing conclusions. As someone else mentioned, there does seem to be a phenomenon where saying DO NOT DO X causes a sort of attention bias on X which can lead to X occurring despite the clear instructions. I've never empirically tested that, I've just noticed better results over the years when telling it what paths to stick to rather than specific things not do to.

Re: AI overly affirms users asking for personal advice

#77

Post by righthand »

Loughla wrote: post #61
awithrow wrote: post #34

It feels like I'm fighting uphill battle when it comes to bouncing ideas off of a model. I'll set things up in the context with instructions similar to. "Help me refine my ideas, challenge, push back, and don't just be agreeable." It works for a bit but eventually the conversation creeps back into complacency and syncophancy. I'll check it too by asking "are you just placating me?" the funny thing is that often it'll…

That's because you need actual logic and thought to be able to decide when to be critical and when to agree. Chatbots can't do that. They can only predict what comes next statistically. So, I guess you're asking if the average Internet comment agrees with you or not. I'm not sure there's much value there. Chatbots are good at tasks (make this pdf an accessible word document or sort the data by x), not decision making…

I said this pretty much and got major downvotes…

Re: AI overly affirms users asking for personal advice

#78

Post by ambicapter »

secret_agent wrote: post #67
awithrow wrote: post #34

It feels like I'm fighting uphill battle when it comes to bouncing ideas off of a model. I'll set things up in the context with instructions similar to. "Help me refine my ideas, challenge, push back, and don't just be agreeable." It works for a bit but eventually the conversation creeps back into complacency and syncophancy. I'll check it too by asking "are you just placating me?" the funny thing is that often it'll…

Use positive requests for behavior. For some reason, counter prompts "Don't do X" seems to put more attention on X than the "Don't do." It's something like target fixation, "Oh shit I don't want to hit that pothole..." bang

This is a well known problem in these kind of systems. I’m not 100% on what the issue is mechanically but it’s something like they can only represent the existence of things and not non-existence so you end up with a sort of “don’t think of the pink elephant” type of problem.

Re: AI overly affirms users asking for personal advice

#79

Post by plagiarist »

kvirani wrote: post #71
Loughla wrote: post #61

Earlier quoted context omitted.

That's because you need actual logic and thought to be able to decide when to be critical and when to agree. Chatbots can't do that. They can only predict what comes next statistically. So, I guess you're asking if the average Internet comment agrees with you or not. I'm not sure there's much value there. Chatbots are good at tasks (make this pdf an accessible word document or sort the data by x), not decision making…

I'm not convinced that "actual logic and thought" aren't just about inferring what comes next statistically based on experience.

Then the machines still need a more sophisticated "experience" compared to what they have currently.

Re: AI overly affirms users asking for personal advice

#80

Post by righthand »

kvirani wrote: post #71
Loughla wrote: post #61

Earlier quoted context omitted.

That's because you need actual logic and thought to be able to decide when to be critical and when to agree. Chatbots can't do that. They can only predict what comes next statistically. So, I guess you're asking if the average Internet comment agrees with you or not. I'm not sure there's much value there. Chatbots are good at tasks (make this pdf an accessible word document or sort the data by x), not decision making…

I'm not convinced that "actual logic and thought" aren't just about inferring what comes next statistically based on experience.

Communicating is usually about inferring. I dont think token to token. And I don’t think “well statistically I could say ‘and’ next but I will say ‘also’ instead to give my speech some flash”. If I decided on swapping a word I would have made my decision long ago, not in the moment. Thought and logic are not me pouring through my brain finding a statistical path to any answer. Often I stop and say “I dont know”.
Post reply on HN