Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

Post reply on HN
Print view View original

21–30 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#21

Post by IceDane »

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis.

I'll admit that part of it also comes from personal experience, at work and elsewhere. I've met some catastrophically incompetent people were completely oblivious to their own incompetence, and this has very often felt like that the more incompetent they were, the more likely they were to be try to do stuff that was waaaay out of their comfort zone, which would make even experienced, competent people tread carefully.

But even ignoring personal experiences, I'm not convinced by the arguments either. I understand what they are saying, but I don't see how this disproves the DK effect.

Even if everyone is equally bad at estimating their own skill, so that their estimate is essentially a completely random variable, then we would expect the self-assessment score average to be around 50. If I understand it correctly, this is essentially what figure 9 is demonstrating.

But that figure still says that worse performers are then likely to overestimate their own ability, just as much as it says that better performers are bad at it.

If we look at the original DK figure and contrast it with figure 9 with random data, then I think one way of interpreting the differences is that, yes, worse performers are indeed bad at self-assessment, but they're just kind of bad at it as if their self-assessment is a completely random variable. It then seems to keep being essentially random but as people's skills improve, the distance between their score and their self-assessment becomes a bit tighter.. so in conclusion: most people are pretty bad at self-assessment, but skilled people are a bit less so.

The end result is still that people in the bottom quartiles are going to over-estimate their own ability.

I don't know, maybe this is way out in the weeds. Please school me.

Re: The Dunning-Kruger Effect Is Autocorrelation

#23

Post by js8 »

IceDane wrote: post #21
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis. I'll admit that part of it also comes from personal experience, at work and elsewhere. I've met some catastrophically incompetent people were completely oblivious to their own incompetence, and this has very often felt like that the more incompetent they were, the more likely they were to be try to do stuff that…

You might be biased towards avoiding catastrophic risk. Somebody who doesn't know what they are doing without knowing it is more dangerous than somebody who knows what they are doing yet taking precautions as if they don't.

Re: The Dunning-Kruger Effect Is Autocorrelation

#24

Post by nomilk »

apienx wrote: post #20

> Collectively, the three critique papers have about 90 times fewer citations than the original Dunning-Kruger article.5 So it appears that most scientists still think that the Dunning-Kruger effect is a robust aspect of human psychology.6 Critiques cite the work being critiqued (yes, the referenced critiques in TFA cite the Dunning-Kruger study). Also, a 23 year-old paper will inevitably get cited more than 6 year-o…

Thanks for introducing me to the term "half-life of knowledge".

> An engineering degree went from having a half life of 35 years in ca. 1930 to about 10 years in 1960. A Delphi Poll showed that the half life of psychology as measured in 2016 ranged from 3.3 to 19 years depending on the specialty, with an average of a little over 7 years.

This is very interesting and makes me wonder what it is for tech careers, e.g. web devs, data scientists etc.

Re: The Dunning-Kruger Effect Is Autocorrelation

#25

Post by laszlokorte »

oceliker wrote: post #6

I think the gist of the article is this: Suppose you make 1000 people take a test. Suppose all 1000 of these people are utterly incapable of evaluating themselves, so they just estimate their grade as a uniform random variable between 0-100, with an average of 50. You plot the grades of each of the 4 quartiles and it shows a linear increase as expected. Let's say the bottom quartile had an average of 20, and the top…

Someone who has a skill=0 can not underestimate and someone with a skill=100 can not overestimate. So by the framing of the question alone the participants are nudged to estimate there own skill "more averagely".

Re: The Dunning-Kruger Effect Is Autocorrelation

#26

Post by andersource »

uldos wrote: post #17
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

Unskilled people are more random with their self assessment that skilled. It has nothing to do with unskilled people thinking that they know everything.

> Unskilled people are more random with their self assessment than skilled

This is a statistical claim, supported by the DK graph (but not the random data thought experiment from the article).

> It has nothing to do with unskilled people thinking that they know everything

This reads to me as a claim about the psychological reason for the statistical pattern, which I don't think is either supported nor contradicted by data, both in the article and in the original paper.

Re: The Dunning-Kruger Effect Is Autocorrelation

#27

Post by _dain_ »

nbernard wrote: post #19
_dain_ wrote: post #14

Earlier quoted context omitted.

>The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK effect is not…

> The article argues that the graphs supposedly demonstrating this fact, can also be generated from a model that does not have this difference, i.e. where bias_lowskill == bias_highskill. But as I understand it, it doesn't: In the graph generated using random data, the lines intersect in the middle (bias_lowskill == bias_highskill), whereas in DK's paper they intersect in the upper right (so bias_lowskill != bias_hig…

Okay well in that case the article is deficient (I skimmed it, assuming it was making the same case I already knew about[1] -- my bad). You can actually reproduce the DK graph without supposing that estimation bias depends on skill. There is an interactive visualization here[2] that does precisely that (click on the "line plot w/ centiles" tab). You can adjust the parameters to see how it behaves.

[1] https://news.ycombinator.com/item?id=31038386

[2] http://emilkirkegaard.dk/understanding_statistics/?app=Dunni...

Re: The Dunning-Kruger Effect Is Autocorrelation

#28

Post by omnicognate »

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed effect. To gather evidence that the effect doesn't actually exist you need a new experiment, not just a new analysis of the same data, because the the different levels of actual skill need to be established separately from establishing the error in skill self-assessment. In the original experiment the test results were used to establish both, which is not sufficient.

But the article presents the results from just such a new experiment. In this one they used university education level (sophomore through to professor) and measured skill self-assessment level within those groups. Higher education level (a good proxy for skill on the test used, which was about science literacy) was found to be associated with more accurate skill self-assessment but the bias of lower-skilled people overestimating their skills was not observed.

That's just one study of course, but it sounds like a much better designed one than the original and does constitute actual evidence that the Dunning-Kruger effect doesn't exist.

Re: The Dunning-Kruger Effect Is Autocorrelation

#29

Post by Phileosopher »

PaulKeeble wrote: post #13

Modern Psychology is having a lot of these sorts of results over the last decade, none of their methods are holding up under proper scrutiny. They are struggling to reproduce findings but more critically even the reproduced ones are turning out to be statistical and mathematical errors like shown here. Some of the findings have also done severe harm to patients over the decades as well, I can't help but think we need…

I'm convinced it's associated with the methodology of how psychology has approached matters.

In the world of biology, you're observing the world around you. Same for physics, chemistry, et al. This means that you can set up proper controls to obscure your own presence from any potential results (e.g., isolate everything in another room, use cameras to avoid being near animals, etc.)

Psychology has the same nightmare as quantum physics: pre-existing thoughts and beliefs literally define what results you end up with.

I'm convinced that psych is a victim of the "new" way of doing science: treating the Scientific Method™ as a self-evident concept instead of regarding science as a vastly certain domain of metaphysics.

Re: The Dunning-Kruger Effect Is Autocorrelation

#30

Post by andersource »

omnicognate wrote: post #28
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed effect. To gather evidence that the effect doesn't actually exist you need a new experiment, not just a new analysis of the same data, because the the different levels of actual skill need to be established separately from establishing the error in skill self-as…

> The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed effect

I don't understand this part. "Completely uncorrelated data" is usually taken to represent the null hypothesis, but that's not the case here. In the DK paper, the implicit null hypothesis is "people of all skill levels are good at estimating their performance". In this case the "completely uncorrelated data" matches an alternative hypothesis, "people's skills have nothing to do with their ability to estimate their performance in tasks testing that skill". This hypothesis doesn't outright contradict the DK proposed hypothesis (and is certainly not the DK null hypothesis), so getting similar results is unsurprising to me, and I'm not sure that we learn from it anything about the DK results.

As for the other study cited, the figure shown in the article doesn't give a lot of information on density, and looking at the paper itself, figure 4 does actually seem to show that self-assessment gradually shifts left with increasing level of education.

(Edited for accuracy).

Post reply on HN