Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

Post reply on HN
Print view View original

11–20 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#11

Post by oceliker »

andersource wrote: post #9
oceliker wrote: post #6

I think the gist of the article is this: Suppose you make 1000 people take a test. Suppose all 1000 of these people are utterly incapable of evaluating themselves, so they just estimate their grade as a uniform random variable between 0-100, with an average of 50. You plot the grades of each of the 4 quartiles and it shows a linear increase as expected. Let's say the bottom quartile had an average of 20, and the top…

That's the way I understand the statistical analysis, and in my view this exactly supports (not contradicts) DK: > In reality, nobody had any clue how to estimate their own success. Wouldn't that mean unskilled people tend to overestimate their skill, and experts tend to underestimate it? Why is there a contradiction with DK's conclusions?

> Wouldn't that mean unskilled people tend to overestimate their skill, and experts tend to underestimate it?

I think it's because the original paper speculates far beyond it:

> The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it.

The argument about autocorrelation says this "dual burden" doesn't need to be there to observe the effect.

Re: The Dunning-Kruger Effect Is Autocorrelation

#12

Post by andersource »

oceliker wrote: post #11
andersource wrote: post #9

Earlier quoted context omitted.

That's the way I understand the statistical analysis, and in my view this exactly supports (not contradicts) DK: > In reality, nobody had any clue how to estimate their own success. Wouldn't that mean unskilled people tend to overestimate their skill, and experts tend to underestimate it? Why is there a contradiction with DK's conclusions?

> Wouldn't that mean unskilled people tend to overestimate their skill, and experts tend to underestimate it? I think it's because the original paper speculates far beyond it: > The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence ro…

Again, not in my reading. In the random data thought experiment, everybody (experts and unskilled alike) suffer from the burden of not having the skill to estimate their performance. The author is even surprised that the DK effect in the random data is bigger than observed in the DK experiment ("In fact, as Figure 9 shows, our effect is even bigger than the original") - but that's because in reality, people do have some ability to estimate their own skill. So the claim that the lack of skill is related to the lack of ability to self-evaluate does make sense, or at least, isn't contradicted by the experiment.

Re: The Dunning-Kruger Effect Is Autocorrelation

#13

Post by PaulKeeble »

Modern Psychology is having a lot of these sorts of results over the last decade, none of their methods are holding up under proper scrutiny. They are struggling to reproduce findings but more critically even the reproduced ones are turning out to be statistical and mathematical errors like shown here. Some of the findings have also done severe harm to patients over the decades as well, I can't help but think we need a lot of caution when it comes to psychology results given its harmful uses (such as the abuse of ill patients) and its lack of truthful results.

Re: The Dunning-Kruger Effect Is Autocorrelation

#14

Post by _dain_ »

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

>The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means.

DK effect is not that low skill people are overconfident and high skill people are underconfident. It is specifically that low skill people are more overconfident than high skill people are underconfident. i.e. if someone's estimated skill is true_skill+bias+noise, then bias_lowskill > -bias_highskill.

This is very clear in the original DK paper, they specifically focus on the supposed metacognitive deficiencies of low-skill people.

The article argues that the graphs supposedly demonstrating this fact, can also be generated from a model that does not have this difference, i.e. where bias_lowskill == bias_highskill.

EDIT: My characterization of the article is not correct, see here[1] for a visualization of the point I'm trying to make.

[1] http://emilkirkegaard.dk/understanding_statistics/?app=Dunni...

Re: The Dunning-Kruger Effect Is Autocorrelation

#15

Post by danbruc »

I am no expert in statistics or the Dunning-Kruger effect but this analysis doesn't sound correct to me. If you plot self assessment against test scores then the following will happen. If people are perfect at self assessment, then you get a straight diagonal line. The more wrong they are, the wider the line will get, in the extreme - if the self assessment is unrelated to the test result - the line will cover the entire chart. If people overestimate their performance, the line will move up, if they underestimate their performance, the line will move down. If you look at the Dunning Kruger chart, that is what you see, complicated a bit by the fact that they aggregated individual data points. At low test scores the self assessment is above the diagonal, at high test scores it is below. What matters is indeed the difference between the self assessment and the ideal diagonal, but if you don't plot individual data points but aggregate them, you have to make sure that there is a useful signal - if self assessments are random, then the median or average in each group will be 0.5 and you will get a horizontal line, but that aggregate 0.5 isn't really telling anything useful.

Re: The Dunning-Kruger Effect Is Autocorrelation

#16

Post by dgb23 »

Tangential, but the more interesting question for me is:

How does estimating my skill level influence skill growth, social relationships and decision making?

I think there are a bunch of useful angles to this. When there are risk/responsibility opportunities, then I need to be courageous. When it’s about learning and interacting collaboratively, then I need to be humble.

Re: The Dunning-Kruger Effect Is Autocorrelation

#17

Post by uldos »

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

Unskilled people are more random with their self assessment that skilled. It has nothing to do with unskilled people thinking that they know everything.

Re: The Dunning-Kruger Effect Is Autocorrelation

#18

Post by andersource »

_dain_ wrote: post #14
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

>The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK effect is not…

That would make sense, except that graphs generated from random data show identical bias for overestimation and underestimation, as can be seen in the article. And this is opposed to graphs from the DK paper, which show a smaller underestimation bias for experts than an overestimation bias for low-skill people. (Of course that alone doesn't prove anything, just saying that to my understanding, nothing in the article contradicts my interpretation of DK).

Re: The Dunning-Kruger Effect Is Autocorrelation

#19

Post by nbernard »

_dain_ wrote: post #14
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

>The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK effect is not…

> The article argues that the graphs supposedly demonstrating this fact, can also be generated from a model that does not have this difference, i.e. where bias_lowskill == bias_highskill.

But as I understand it, it doesn't: In the graph generated using random data, the lines intersect in the middle (bias_lowskill == bias_highskill), whereas in DK's paper they intersect in the upper right (so bias_lowskill != bias_highskill).

Re: The Dunning-Kruger Effect Is Autocorrelation

#20

Post by apienx »

> Collectively, the three critique papers have about 90 times fewer citations than the original Dunning-Kruger article.5 So it appears that most scientists still think that the Dunning-Kruger effect is a robust aspect of human psychology.6

Critiques cite the work being critiqued (yes, the referenced critiques in TFA cite the Dunning-Kruger study). Also, a 23 year-old paper will inevitably get cited more than 6 year-old papers. But yeah...the inertia in Science is real. That conservatism's a feature, not a bug.

Psychology's probably the discipline with the shortest "half-life of knowledge. https://en.wikipedia.org/wiki/Half-life_of_knowledge

Post reply on HN