The Dunning-Kruger effect may just be a data artefact (2020)

I feel like the only important point in the article would be to explain how the random data was generated, yet it was relegated to the single sentence: "There was no bias in the coding that would lead these fictitious students to guess they had done really well when their actual score was very low."

Because on the surface, it doesn't make any sense for two sets of "random" numbers between 0-100 selected in pairs to deviate from each other based on whether the first number in the pair was low or not. You would not expect the first number chosen in a pair to influence the second number. Whether the first number was between 0-25 or 76-100, you would expect the second number to be about 50.

So this is obviously some sort of structured randomness that may be entirely justifiable, but the only way to find that out would be to read the two articles that this article purports to summarize for the layman. Instead there's over 1300 words of slop before this sentence, then nearly 700 words of slop after this sentence. Turns out we don't need AI for this. Speaking of random, I don't think that 2000 words is random.

-----

edit:

maybe the point of the papers is that low scorers can't underestimate their abilities - as in they literally don't have enough room? If so, that just means that the Dunning-Kruger affect is unavoidable. But the fact is that people are not choosing numbers at random, they are choosing them based on their expectations. People who got zero questions right and expected 100% are as likely as anyone else from a random number generator, and non-existent from actual people.

edit2:

OK, I've worked it out. I was making the mistake of thinking that they were evaluating absolute performance rather than relative performance. So each of the first numbers in the pair is unique. But that still leaves the fact that the random draw still predictably sits at 50% where the Dunning-Kruger data is around 65% based on the graph. Seems like norming that with the random data would give you better information.

edit3:

> In Dr. Nuhfer’s own papers [...] his team [...] showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.

How is "narrower range" not an indication of more accurate self-evaluation? With that, and since people on the higher end of the scale have less room to overestimate their standing, and people on the lower end of the scale have less room to underestimate their standing, wouldn't you expect "Dunning-Kruger"? People on the low end of the scale would have wild swings that would be gated at zero, and people on the high end of the scale would have small swings that would be gated at 100. That would lead to small underestimates at the top, and large overestimates at the bottom. More accurate self-evaluation at the top of the scale is exactly what Dunning-Kruger is about, and the direction of the mistakes is predictable if this is true.

final, tldr:

Honestly, the entire debate is garbled. People are not being asked about their performance on a test, they're being asked about their standing within a sampling of people chosen by the experimenter, something which they have no reason to know anything about other than on the experimenter's word.

I think how people interpret Dunning-Kruger, and the only interesting thing about it, is that people who have more knowledge of a subject are more accurate in their assessment of how much they know about that subject. This seems likely (but not evidently) to be true, due to the range of (relative) self-assessment error being narrower in the top quartile as compared to the bottom quartile. This is what people found intuitive and compelling.

If it is true, the top quartile would tend to small underestimation (because of the narrower range and that they can't choose numbers higher than 100) and the bottom quartile would tend to larger overestimation (because of the wider range and that they can't choose numbers lower than 1.) That the average direction of over- and underestimation is forced by the nature of the evaluation doesn't make the effect any less true.

It is both amazing and saddening to me how you can have such an arrogant view of a whole field with just taking a 101 course.

This article seems really dumb (no Dunning-Kruger joke intended).

The two lines on the graph are basically linear (for the "actual performance" the quasi-linearity is obvious by the design, for the "estimated performance" it still means that even though dumber people over-estimate their performance, all group still think they do best, when they actually do best, in a relative linear way)

And when they do their simple model (we assume they just generated "real performance" from a gaussian, then added some gaussian noise for the "performance" and another gaussian noise for the "self-assessment") they still (obviously) got two linear graphs that crossed each other.

And then they conclude that this means there is no effect, because "the graphs are eerily similar" (whatever that means)

But obviously the simple model is going to make two lines cross (in particular if you use a min(100, max(0, actual_performance + noise)) since at each extreme, then min and max will tend to skew the line). To put it simply: someone really stupid will STILL not pretend that he's "negatively stupid".

The argument "I can make a simple model without using actual humans which shows some kind of bias that vaguely ressembles the result of a paper" doesn't mean that the actual paper is wrong...

Indeed. And the obvious reason is that when you simulate the "self-assessment" using a Gaussian noise around the "actual intelligence", and clamp it to [0, 100] so that it doesn't go "out of bound" (eg "negative intelligence" is not allowed), you will necessarily skew the low scores upward and the high scores downwards.

But it's not because "some statistical model exhibit a bias that's similar to the result" that this implies "therefore the result is a statistical error"... that's a backward reasonning