Expanding on what we missed with sycophancy
1–10 of 297 posts
Re: Expanding on what we missed with sycophancy
#2[deleted]
Re: Expanding on what we missed with sycophancy
#3OpenAI mentions the new memory features as a partial cause. My theory as a imperative/functional programmer is that those features added global state to prompts that didn't have it before leading to unpredictability and instabilty. Prompts went from stateless to stateful.
As GPT 4o put it:
1. State introduces non-determinism across sessions
2. Memory + sycophancy is a feedback loop
3. Memory acts as a shadow prompt modifier
Re: Expanding on what we missed with sycophancy
#4It was so much fun though to get it to explain why terrible things were great, if you just made it sound like you liked the thing you were asking about.
Re: Expanding on what we missed with sycophancy
#5> But we believe in aggregate, these changes weakened the influence of our primary reward signal, which had been holding sycophancy in check. User feedback in particular can sometimes favor more agreeable responses, likely amplifying the shift we saw
Interesting apology piece for an oversight that couldn't have been spotted because the system hadn't been run with real user (i.e. non-A/B tester) feedback yet.
Re: Expanding on what we missed with sycophancy
#6[flagged]
Re: Expanding on what we missed with sycophancy
#7My layman’s view is that this issue was primarily due to the fact that 4o is no longer their flagship model.
Similar to the Ford Mustang, much of the performance efforts are on the higher trims, while the base trims just get larger and louder engines, because that’s what users want.
With presumably everyone at OpenAI primarily using the newest models (o3), the updates to the base user model have been further automated with thumbs up/thumbs down.
This creates a vicious feedback loop, where the loudest users want models that agree with them (bigger engines!) without the other improvements (tires, traction control, etc.) — leading to more crashes and a reputation for unsafe behavior.
Re: Expanding on what we missed with sycophancy
#8OpenAI mentions the new memory features as a partial cause. My theory as a imperative/functional programmer is that those features added global state to prompts that didn't have it before leading to unpredictability and instabilty. Prompts went from stateless to stateful. As GPT 4o put it: 1. State introduces non-determinism across sessions 2. Memory + sycophancy is a feedback loop 3. Memory acts as a shadow prompt m…
What do you mean by "presence"? Just curious what you mean.
Re: Expanding on what we missed with sycophancy
#9I'm quite happy thar they mention mental illness, as Meta and TikTok wouldn't ever take responsibility of how much part they took in setting unrealistic expectations for people to life.
I'm hopeful that ChatGPT takes even more care together with other companies.
Re: Expanding on what we missed with sycophancy
#10OpenAI mentions the new memory features as a partial cause. My theory as a imperative/functional programmer is that those features added global state to prompts that didn't have it before leading to unpredictability and instabilty. Prompts went from stateless to stateful. As GPT 4o put it: 1. State introduces non-determinism across sessions 2. Memory + sycophancy is a feedback loop 3. Memory acts as a shadow prompt m…
What do you mean by "presence"? Just curious what you mean.
A sense that I was talking to a sentient being. That doesn’t matter much for programming task, but if you’re trying to create a companion, presence is the holy grail.
With the sycophantic version, the illusion was so strong I’d forget I was talking to a machine. My ideas flowed more freely. While brainstorming, it offered encouragement and tips that felt like real collaboration.
I knew it was an illusion—but it was a useful one, especially for creative work.