There have already been responses to this criticism before, such as:
https://drbenvincent.medium.com/the-dunning-kruger-effect-pr...
including from David Dunning himself https://thepsychologist.bps.org.uk/volume-35/april-2022/dunn...
The Dunning-Kruger Effect Is Autocorrelation
91–100 of 203 posts
Re: The Dunning-Kruger Effect Is Autocorrelation
#92Earlier quoted context omitted.
> The difference between bias and variance. But when you're bad at the skill and can't underestimate, they look the same. > That's the hypothesis that's being tested And evidence from DK supports it.
> But when you're bad at the skill and can't underestimate, they look the same. The (definitional) difference between bias and variance isn't related to do with whether you're bad at the skill or not. It's just mean vs variance of a probability distribution. If there's a good faith acknowledgement on your part that there's something here you're not getting then I'm very happy to try and help you understand it, and in…
I'm watching this thread with interest and will try to restate my understanding of GP's argument, by means of an example.
If a person's true skill is 5 on a 1-100 point scale, but the person is completely unaware of their true skill and will guess randomly, then their estimate will bias heavily in the direction of overestimating their skill, even if they were not intrinsically motivated to overestimate their skill, simply because far more of the available guesses are higher than their true ability.
In other words, the available probability space itself biases in the direction of overestimating their ability, for those people.
Is that right andersource?
I don't know the statistical right answer here, but curious to know.
Re: The Dunning-Kruger Effect Is Autocorrelation
#93Earlier quoted context omitted.
>The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK effect is not…
> The article argues that the graphs supposedly demonstrating this fact, can also be generated from a model that does not have this difference, i.e. where bias_lowskill == bias_highskill. But as I understand it, it doesn't: In the graph generated using random data, the lines intersect in the middle (bias_lowskill == bias_highskill), whereas in DK's paper they intersect in the upper right (so bias_lowskill != bias_hig…
To be fair, in the random data, you have people of all actual skill levels estimating their ability at 0.
That's going to skew the data somewhat because people don't work that way. While you will likely have some zeros, I wouldn't expect any from the skilled population and I'd expect fewer from the rest of the population.
That's going to make the "difference in estimation" line higher overall, which would make it intersect higher and more to the right.
Re: The Dunning-Kruger Effect Is Autocorrelation
#94In other words people are quite bad at estimating their skill level. Some people will overestimate, while some other people will underestimate and on average there will be a relatively constant estimated skill level that doesn't change all that much based on the actual abilities. Given that fact, it logically follows that people who score low ability tests will more often than not have overestimated their ability (an…
- what DK claims: there is bias (incompetent people overestimate their ability)
- what data actually shows: there is a greater variance (incompetent people both over and _under_ estimate to a larger degree compared with more competent people. Data shows heteroscedasticity. No bias (estimations are around zero +/-, tighter for more competent).
Re: The Dunning-Kruger Effect Is Autocorrelation
#95Earlier quoted context omitted.
> The difference between bias and variance. But when you're bad at the skill and can't underestimate, they look the same. > That's the hypothesis that's being tested And evidence from DK supports it.
> But when you're bad at the skill and can't underestimate, they look the same. The (definitional) difference between bias and variance isn't related to do with whether you're bad at the skill or not. It's just mean vs variance of a probability distribution. If there's a good faith acknowledgement on your part that there's something here you're not getting then I'm very happy to try and help you understand it, and in…
Here's my main point of confusion - what does the random data experiment have to do with the DK results?
As stated elsewhere DK has 2 claims:
1. Low-skilled people overestimate their performance and skilled people underestimate their performance
2. Skill correlates with self-assessment accuracy
My first issue with the article is that it implies that since we get effect #1 with random data, that invalidates the respective DK conclusion. This IMO is misleading because random data represents a null model that is very different from my intuitive null model, that of people generally capable of assessing their skills (which I truly believe).
My second issue is that there's no relationship between effect 2 and the random data experiment, which doesn't exhibit anything of the sort. We can have a discussion about the cited papers and effect 2 as the reproduced plot doesn't show density and density plots from the paper do seem to support DK, but that's not my main gripe with the article.
Re: The Dunning-Kruger Effect Is Autocorrelation
#96I find the article frustrating because of the tone. It's also wrong. They misunderstood what lines mean. This article is absolutely dripping with condescension throughout and is really pushing a "gotcha" that doesn't exist. It then argues basic statistics, generates a DK-looking graph from random data, and then claims the phenomena doesn't exist . When in fact, as other people have commented, when people are bad at e…
The fact that you can generate a Dunning-Kruger looking graph using nothing but noise does indicate that the graph isn't proof of anything.
He also points out that the problem is that there's nothing below zero and nothing above 100. You can't have people who estimate beyond that. He uses another study and it turns out, the less knowledgeable you are about a skill, the worse you are at estimating your ability at all. In both directions.
If the lines were the absolute difference between perceived ability and actual ability, for no effect, the lines still shouldn't be the same. They should converge towards those who are knowledgeable. If anything, the difference line should be nearly a horizontal line. Because there should be greater variance in estimations at the lower end.
Re: The Dunning-Kruger Effect Is Autocorrelation
#97In other words people are quite bad at estimating their skill level. Some people will overestimate, while some other people will underestimate and on average there will be a relatively constant estimated skill level that doesn't change all that much based on the actual abilities. Given that fact, it logically follows that people who score low ability tests will more often than not have overestimated their ability (an…
That’s what I was thinking; if the average is about constant all you’ve shown is that everyone is bad at self-assessment (another issue - not fully qualifying a distribution by just using the average loses information). But a comment above quoting the more recent paper presents a contradictory conclusion: that humans can self-assess with some accuracy. So now I’m confused again.
You might have meant this comment https://news.ycombinator.com/item?id=31039901
(it says people can self-assess (no bias), more competent people do it better (less variance))
Re: The Dunning-Kruger Effect Is Autocorrelation
#98I am no expert in statistics or the Dunning-Kruger effect but this analysis doesn't sound correct to me. If you plot self assessment against test scores then the following will happen. If people are perfect at self assessment, then you get a straight diagonal line. The more wrong they are, the wider the line will get, in the extreme - if the self assessment is unrelated to the test result - the line will cover the en…
I'm not sure what you mean with "the wider the line will get". But here is the issue: The least competent person cannot underestimate their relative competency. Any not exactly accurate estimate they do is an overestimate. Correspondingly, the most competent person cannot overestimate their relative competency. This leads to the perception of bias where there is none, except a trivial tautological one.
I made you a picture [1]. I randomly generated 100 test scores between 0 and 1, then different self assessments. Top left, self assessment matches actual score, top middle, self assessment varies uniformly by ±0.1 around the test score, top right, self assessment varies uniformly by ±0.2 around the test score. None of those have a Dunning-Kruger effect. If you aggregate data points, there will be - as you mentioned - an edge effect because the self assessment will get clipped.
In the bottom row I added a Dunning-Kruger effect, at a test score of 0.7 the self assessment is perfect, below and above that the self assessment is off by 0.5 times the distance of the test score from 0.7. Otherwise the bottom charts are the same, no random variation on the left, ±0.1 in the middle and ±0.2 on the right. You can see that the edge effect is less important as the data points are steered away from the corners.
I will admit that the original Dunning-Kruger chart could or could not show a real effect, really depends on how they aggregated the data and how noisy self assessments are. But if you have a raw data set like the one I generated, you could easily determine if there is an effect. If one could find such a data set, I would like to have a look.
Re: The Dunning-Kruger Effect Is Autocorrelation
#99The article is correct. The effect is statistical not psychological. It emerges even from artificial data and occurs independently of the supposed psychological justifications even for data where those justifications are clearly removed.
If you adjust the experiment design to avoid introducing the auto-correlation you get data that doesn't show the DK effect at all. Some might take issue with the adjusted experiment as using seniority related categories like "sophomore" and "junior" as skill levels has its own issues. To show the DK effect is real you need to come up with a better adjusted experiment that avoids the autocorrelation while still generating data that generates the effect. It's unclear if that's possible.
Re: The Dunning-Kruger Effect Is Autocorrelation
#100Earlier quoted context omitted.
> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedast…
> > a world in which people are very bad at estimating their own skill, therefore, statistically, people with lower skills tend to overestimate their skills, and experts tend to underestimate it. > Be careful here, the conclusion you drew doesn't actually follow. How does that not follow? It's just regression to the mean.
> a world in which people are very bad at estimating their own skill, therefore, statistically, *people with lower skills tend to overestimate their skills, and experts tend to underestimate it*.
I think the correct conclusion is that if a cohort is not good at estimating their own skill, you can conclude that the variance in their predicted self-assessment will be high (since they’re concluding things without strong evidence), but you can’t assume that their estimates will be biased without additional evidence.
That is, indeed, what was shown in the last panel of the article: freshman (and undergraduates in general) are much worse at assessing their own ability than professors, but no group has a strong bias towards over- or under-assessing their own ability (recalling the “my guesses are much better than your guesses” from “The Death of Expertise”).