101–110 of 203 posts
IMO that explains why Dunning Kruger seems intuitively correct even if the conclusion they drew isn't actually correct.
Because of the effect that is actually found (variance is higher the less achievement) it follows that people you encounter who wildly overestimate their ability are more likely to people who are poor performers (the same is true for the inverse, but they obviously don't stand out anecdotally to us). IMO that explains why Dunning Kruger seems intuitively correct even if the conclusion they drew isn't actually correct…
Another way to show this would have been to keep the auto correlation plot, but compare it to the same plot with statistical noise. With infinite random data, the expected value for self-assessment would be 50% score, regardless of actual score - a flat line through the chart. It would then be significant to find a non-flat line, as DK did.
It’s not inconceivable that with a smaller sample, you’d get come biasing, where lesser skilled people would over estimate, and higher skilled people under estimate.
The follow up studies seem to suggest there’s not really a bias like that, but that there is a “honing” of the general ability to estimate your own outcome, which makes sense.
> Although there is no hint of a Dunning-Kruger effect, Figure 11 does show an interesting pattern. Moving from left to right, the spread in self-assessment error tends to decrease with more education. In other words, professors are generally better at assessing their ability than are freshmen. That makes sense. Notice, though, that this increasing accuracy is different than the Dunning-Kruger effect, which is about systemic bias in the average assessment. No such bias exists in Nuhfer’s data.
Earlier quoted context omitted.
> But I think the author’s right that obviously nothing psychological is happening here. There’s the psychological effect of no one being able to assess themselves, but the fact that unskilled people overestimate themselves in this world has nothing to do with the fact that they are unskilled. If the results from DK were similar to the random data results, I'd agree. But the DK results do show some correlation betwee…
My guess is that the spread in self-evaluation is largest at low skill level and decreases as skill level increases. This would produce results more similar to what D-K actually observed, and is much more plausible than postulating that highly skilled people have no more idea of their skill level than low skilled people.
My gut, based on this article, is that it would look generally the same but with a larger spread at the lower end of the scale. But I don't think we can truly say we've disproved the Dunning-Kruger effect without a more varied dataset.
Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…
They are, but honestly all that can be concluded safely IMHO is that the original D-K graph doesn't support that the widely discussed "effect" which their conclusion describes exists. Therefore unless there is more evidence from some other subsequent study there may not be any evidence for it at all, and if that's the case then there's potentially no proof it exists.
However, even if you prove that their data is not evidence, that doesn't actually say anything about whether the effect exists or not, just that the D-K paper isn't evidence of such an effect.
I don't think that's enough for the author to conclude that "DK is autocorrelation". A more careful conclusion would be that "the DK data do not support DK's conclusion"... but of course that's much less likely to attract click throughs.
Earlier quoted context omitted.
Seems pretty simple. When we create upper and lower boundaries to some score, people with lower scores have more space to overestimate and those with higher scores more space to underestimate, causing the perceived score to trend towards the mean. I think there's both a component of numbers and psychology here. If the dispersion in perceived score caused by inaccuracy is wide enough to touch the bounds, it will force…
It would be possible to rule out that effect in an experiment.
Earlier quoted context omitted.
> DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedasticity). Not sure I follow, inverse correlation between what? The analysis at the bottom (assuming you mean fig. 11) is too dense to show if there's a correlation or not (between skill and self-assessment bias)…
> If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated “If two variables are independent, then their correlation will be 0” https://web.stanford.edu/class/archive/cs/cs109/cs109.1178/l... “ If the variables are independent, Pearson's correlation coefficient is 0” https://en.wikipedia.org/wiki/Correlation#Correlation_an…
Earlier quoted context omitted.
The author’s confidence is itself an indication that they’re more likely to be wrong. Kidding. Well, half-kidding, I did kind of find the tone a bit biting and dismissive, especially towards one of the commenters that were pointing out exactly what you did. It’s an interesting question to ask whether ask whether the uniformly random data “really” exhibits DK or not, and whether that’s interesting. A world where peopl…
> But I think the author’s right that obviously nothing psychological is happening here. There’s the psychological effect of no one being able to assess themselves, but the fact that unskilled people overestimate themselves in this world has nothing to do with the fact that they are unskilled. If the results from DK were similar to the random data results, I'd agree. But the DK results do show some correlation betwee…
Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…
I find the article frustrating because of the tone. It's also wrong. They misunderstood what lines mean. This article is absolutely dripping with condescension throughout and is really pushing a "gotcha" that doesn't exist. It then argues basic statistics, generates a DK-looking graph from random data, and then claims the phenomena doesn't exist . When in fact, as other people have commented, when people are bad at e…
The fact that you can generate a Dunning-Kruger looking graph using nothing but noise does indicate that the graph isn't proof of anything. He also points out that the problem is that there's nothing below zero and nothing above 100. You can't have people who estimate beyond that. He uses another study and it turns out, the less knowledgeable you are about a skill, the worse you are at estimating your ability at all.…
I fiddled with number of test questions, amounts of variation in question difficulty, various coefficients, etc.
In none of my experiments did I add a bias on the skill axis.
My conclusion is that the "slope However, I didn't find an easy way using my simulation to reproduce the "intercept is high" part of the DK effect to the extent present in the DK graphs, i.e. where the lowest quartile's average self-estimated percentile is >55%. (*)
However, it strikes me that without a very careful explanation to the test subjects of exactly how their peer group was selected, it's easy to imagine everyone being wrong in the same direction.
(*) EDIT: I found a way to raise the intercept quite a lot simply by modeling that people with lower skill have higher variance (but no bias!) in their own skill estimation. This model is supported by another paper the article references.