I find it disappointing that openai doesn't really mention anything here along the lines of having an accurate model of reality. That's really what the problem is with sycophancy, it encourages people to detach themselves from what reality is. Like, it seems like they are saying their "vibe check" didn't check vibes enough.
This is such an interesting question though! It seems to bring to the fore a lot of deeper, philosophical things like if there even IS such a thing as objective reality or objective context within which the AI should be operating. From training data, there might be some generalizations that are carried across all contexts, but that starts to not be applicable when person A with a college degree says they want to start business x versus person B without said degree who also wants to start business x, how does the model properly reconcile the context of the general advise and each asker’s unique circumstances? Does it ask an infinite list of probing questions before answering? It gets into much the same problems as issues of advise among people.
Plus, things get even harder when it comes to even less quantifiable contexts like mental health and relationships.
In all, I am not saying there isnt some approximated and usable “objective” reality, just that it starts to break down when it gets to the individual and that is where openai is failing by over-emphasizing reflective behavior in the absence if actual data about the user.