Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

Post reply on HN
Print view View original

41–50 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#41

Post by andersource »

omnicognate wrote: post #34
andersource wrote: post #30

Earlier quoted context omitted.

> The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed effect I don't understand this part. "Completely uncorrelated data" is usually taken to represent the null hypothesis, but that's not the case here. In the DK paper, the implicit null hypothesis is "people of all skill levels are good at estimating their perfo…

The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias " in estimating their performance" (remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill). The randomly generated data used is one example of that, albeit an…

> The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias" in estimating their performance"

OK,

> remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill

The way I see it these aren't very different - conditioning on low skill and randomly sampling will tend to give way more overestimates than underestimates. What is the distinction between being bad at the skill and bad at estimation, and being bad at the skill and systematic overestimation?

> The fact that the artifact is seen in such data is a powerful demonstration that it is not evidence of the Dunning-Kruger effect

"Such data" refers to a world where everyone have absolutely no idea how good or bad they are. To me that is a much stronger argument than that made by DK. So perhaps our differences all come down to our priors. My prior belief (before looking at any data) is that people would know how good they are at a certain skill. If your prior is to expect that people don't know how good they are, then your arguments make sense to me. If however your prior is that people do know how good they are, but are also biased (all in the same direction), then I don't understand how the random data experiment reveals anything relevant to your beliefs.

Re: The Dunning-Kruger Effect Is Autocorrelation

#42

Post by hallway_monitor »

nomilk wrote: post #24
apienx wrote: post #20

> Collectively, the three critique papers have about 90 times fewer citations than the original Dunning-Kruger article.5 So it appears that most scientists still think that the Dunning-Kruger effect is a robust aspect of human psychology.6 Critiques cite the work being critiqued (yes, the referenced critiques in TFA cite the Dunning-Kruger study). Also, a 23 year-old paper will inevitably get cited more than 6 year-o…

Thanks for introducing me to the term "half-life of knowledge". > An engineering degree went from having a half life of 35 years in ca. 1930 to about 10 years in 1960. A Delphi Poll showed that the half life of psychology as measured in 2016 ranged from 3.3 to 19 years depending on the specialty, with an average of a little over 7 years. This is very interesting and makes me wonder what it is for tech careers, e.g. w…

For javascript developers it's about six months!

Kidding aside, it seems you could estimate it by asking, what portion of the knowledge I use did I learn 20 years ago? Then 10, 5, 1. For me it seems to be somewhere around ten years.

Re: The Dunning-Kruger Effect Is Autocorrelation

#44

Post by denton-scratch »

IceDane wrote: post #21
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis. I'll admit that part of it also comes from personal experience, at work and elsewhere. I've met some catastrophically incompetent people were completely oblivious to their own incompetence, and this has very often felt like that the more incompetent they were, the more likely they were to be try to do stuff that…

/me not a statistician either.

If people are all pretty-much crap at estimating their own skill, then you'd expect all estimates to be roughly their actual skill, plus-or-minus some random error-margin with some kind of probabilistic distribution (Gaussian?).

If that were the case, then high-skilled people would be more likely to underestimate their skill (because their actual skill is greater than the mean). That (I think) is an example of "reversion to the mean".

If that reasoning is right, then the DK claim may be true, but it says nothing about the comparative estimating propensities of high-skilled and low-skilled people. It just says that low-skilled people tend to overestimate their skill, and vice-versa. But that's exactly what you'd expect, amirite?

Re: The Dunning-Kruger Effect Is Autocorrelation

#45

Post by BlueTemplar »

andersource wrote: post #26
uldos wrote: post #17

Earlier quoted context omitted.

Unskilled people are more random with their self assessment that skilled. It has nothing to do with unskilled people thinking that they know everything.

> Unskilled people are more random with their self assessment than skilled This is a statistical claim, supported by the DK graph (but not the random data thought experiment from the article). > It has nothing to do with unskilled people thinking that they know everything This reads to me as a claim about the psychological reason for the statistical pattern, which I don't think is either supported nor contradicted by…

Not the D-K graph, the Nuhfer et al. graph.

Re: The Dunning-Kruger Effect Is Autocorrelation

#46

Post by [deleted] »

andersource wrote: post #26
uldos wrote: post #17

Earlier quoted context omitted.

Unskilled people are more random with their self assessment that skilled. It has nothing to do with unskilled people thinking that they know everything.

> Unskilled people are more random with their self assessment than skilled This is a statistical claim, supported by the DK graph (but not the random data thought experiment from the article). > It has nothing to do with unskilled people thinking that they know everything This reads to me as a claim about the psychological reason for the statistical pattern, which I don't think is either supported nor contradicted by…

[deleted]

Re: The Dunning-Kruger Effect Is Autocorrelation

#47

Post by andersource »

BlueTemplar wrote: post #45
andersource wrote: post #26

Earlier quoted context omitted.

> Unskilled people are more random with their self assessment than skilled This is a statistical claim, supported by the DK graph (but not the random data thought experiment from the article). > It has nothing to do with unskilled people thinking that they know everything This reads to me as a claim about the psychological reason for the statistical pattern, which I don't think is either supported nor contradicted by…

Not the D-K graph, the Nuhfer et al. graph.

The one reproduced in the article doesn't show the density of points, so it's hard to conclude anything from it. Figure 4 from the Nuhfer et al. paper does seem, to me at least, to support DK's conclusions.

Re: The Dunning-Kruger Effect Is Autocorrelation

#48

Post by larwent »

Unless I missed something, this article doesn't explain WHY random data can result in a Dunning-Kruger effect. The relationship between the "actual" and "perceived" score is a product of bounding the scores to 0-100.

When you generate a random "actual" score near the top, the random "perceived" score has a higher chance of being below the "actual" the numerical below is larger than the one above, and vice-versa. E.g. a "test subject" with an actual score of 80% has a (uniform random) 20% chance of overestimating their ability and an 80% of underestimating it. For an actual score of 20%, they have an 80% chance of overestimating.

Re: The Dunning-Kruger Effect Is Autocorrelation

#49

Post by alecbz »

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

The author’s confidence is itself an indication that they’re more likely to be wrong.

Kidding. Well, half-kidding, I did kind of find the tone a bit biting and dismissive, especially towards one of the commenters that were pointing out exactly what you did.

It’s an interesting question to ask whether ask whether the uniformly random data “really” exhibits DK or not, and whether that’s interesting. A world where people have 0 ability to assess their own skill and resort to making uniformly random guesses at it is kind of interesting, and of course in such a world more skilled people would end up on average underestimating themselves and vice versa.

But I think the author’s right that obviously nothing psychological is happening here. There’s the psychological effect of no one being able to assess themselves, but the fact that unskilled people overestimate themselves in this world has nothing to do with the fact that they are unskilled.

Re: The Dunning-Kruger Effect Is Autocorrelation

#50

Post by fallingfrog »

OK, I think I understand. What the data from the original experiment actually shows is that people at all skill levels are pretty bad at estimating their skill level- it's just that if you scored well, the errors are likely to be underestimates, and if you scored badly, the errors are likely to be overestimates, by pure chance alone. So it's not that low scoring individuals are particularly overconfident so much as everyone is imperfect at guessing how well they did. Great observation.
Post reply on HN