What the Dunning-Kruger effect is and isn’t (2010)

I always worry about self-assessment data. Especially when an experimenter or other participants are in the room or otherwise observing or interacting with the participant during the self-assessment.

The results of Dunning-Kruger-adjacent experiments always present their results as measuring "How people think they perform" against "How they actually perform", when they seem to actually be measuring "How people tell other people they perform" against "How they actually perform".

Which seems to me like it's a slightly different thing? It seems like the pop-sci "those people are too stupid to know how stupid they are" interpretation isn't the only plausible interpretation, anyway.

Does the affect account for considering different populations? For instance, maybe skilled people tend to compare themselves to other skilled people, while unskilled people compare themselves to the general population?

The participants in the study didn’t self-evaluate directly, they self-ranked, against the other participants (all Cornell undergrads volunteering for extra credit).

There’s a very important difference there, because self-ranking requires knowledge of other people’s skills.

Do note that the DK paper found a positive correlation between confidence and skill. Most people here and elsewhere seem to believe the DK effect shows a negative correlation.

Also note that the skills people were self-ranking were all easy. Ability to get a joke was one of the 4 tasks. How would you rank your own ability to get a joke against those around you? The question barely makes sense.

I think it’s a big mistake to take any lesson from the DK paper other than sometimes people see what they want to.

I’m a little worried about supplying any explanation because the DK paper didn’t study the general population at all, and nobody who participated was incompetent or lacking meta cognitive skills or representative of any of the other exaggerated terms they used. The participants were all Cornell undergrads volunteering for extra credit. That alone is such a huge red flag on the validity of this study that presumes to make bold claims about general behavior, I honestly can’t understand why this paper has had so much attention or traction. The participants were self-selected, all the same age, all the same socioeconomic bracket (statistically speaking), all needed extra credit in their psych class, all admitted to a prestigious school, and all have been told by their family how smart they are.

> People may self-report opinions of their own performance that they feel are socially acceptable or socially protective of themselves. So it reverts to the mean because it remains within The Overton Window so to speak.

> If your objective performance is poor [...]

The DK paper didn’t study objective performance, it studied perceived performance. They didn’t exactly have participants opine on their own performance at all, they were asked to rank themselves within the group. That is a relative opinion, and it means the participant is evaluating the other participants’ abilities too. It also means everyone in the study was just guessing, which is the biggest reason why I think Tal is right, that this is nothing more than noise leading to regression to the mean.

> But if they hired you for your PhD in mathematics [...]

DK did not study any difficult cognitive tasks akin to research or complex math either. The paper only had 4 tasks: ability to get a joke, some standardized test logic questions, a little English grammar, and grading other people’s tests.

I’m not sure if the things I’m pointing out support your theory or not. Mostly I suspect DK is specious and contagious because it lets people justify their own negative judgements of others, so in that sense I think using DK as an explanation of someone’s behavior is a way to feel socially acceptable and be socially protective of one’s self. It’s just unfortunate that the common judgements and conclusions with this study are almost always unjustified because the paper is so widely misunderstood.

The abstract mentions women self-assess more accurately than men. As a man and thinking about my own mindset with self assessment, I wonder if the definition of “self-assessment” itself might differ between men and women on average. For myself, if you ask me to do a task in a discipline I know little about, I might assess myself as being capable of doing so even without the prior experience, because I figure “I can figure it out reasonably well on the fly”.

This is a convoluted way of saying men may just be overconfident, but importantly not because we assume we k ow everything, but because we are more confident we can learn it even if we don’t know it.

Maybe me writing all this while self-assessing myself is yet more male self-assessment hubris...