Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

Post reply on HN
Print view View original

51–60 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#51

Post by BlueTemplar »

IceDane wrote: post #21
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis. I'll admit that part of it also comes from personal experience, at work and elsewhere. I've met some catastrophically incompetent people were completely oblivious to their own incompetence, and this has very often felt like that the more incompetent they were, the more likely they were to be try to do stuff that…

You seem to be forgetting about under-estimation, so your conclusions don't follow from your premises ?

> most people are pretty bad at self-assessment, but skilled people are a bit less so

This is pretty much the conclusion of the article, except that it isn't the tautologic DK or figure 9 that shows it, but Nuhfer's figure 11.

Re: The Dunning-Kruger Effect Is Autocorrelation

#52

Post by larwent »

jakear wrote: post #37

Article seems to be saying “DK doesn’t exist because it always exists”. Which is… absurd? The point of DK is that when you don’t know shit, any non-degenerate self assessment will result in overestimating your ability. In short, “there are more natural numbers above smaller natural numbers than bigger ones”. This doesn’t have to do with psychology, and it’s expected that it appears when evaluating random data. That’s…

Seems pretty simple. When we create upper and lower boundaries to some score, people with lower scores have more space to overestimate and those with higher scores more space to underestimate, causing the perceived score to trend towards the mean.

I think there's both a component of numbers and psychology here. If the dispersion in perceived score caused by inaccuracy is wide enough to touch the bounds, it will force a trend towards the mean. This effect is possibly exacerbated by a tendency of perception to stray from "extremes", so subjects with a score near the edges will trend to the mean more strongly as they are unlikely to rate themselves the very best or very worst.

Re: The Dunning-Kruger Effect Is Autocorrelation

#53

Post by andersource »

alecbz wrote: post #49
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

The author’s confidence is itself an indication that they’re more likely to be wrong. Kidding. Well, half-kidding, I did kind of find the tone a bit biting and dismissive, especially towards one of the commenters that were pointing out exactly what you did. It’s an interesting question to ask whether ask whether the uniformly random data “really” exhibits DK or not, and whether that’s interesting. A world where peopl…

> But I think the author’s right that obviously nothing psychological is happening here. There’s the psychological effect of no one being able to assess themselves, but the fact that unskilled people overestimate themselves in this world has nothing to do with the fact that they are unskilled.

If the results from DK were similar to the random data results, I'd agree. But the DK results do show some correlation between skill and self-assessment ability.

Re: The Dunning-Kruger Effect Is Autocorrelation

#54

Post by alecbz »

geysersam wrote: post #35
omnicognate wrote: post #34

Earlier quoted context omitted.

The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias " in estimating their performance" (remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill). The randomly generated data used is one example of that, albeit an…

Maybe another way of putting it is that the "Dunner-Kruger effect" is simply a tautology. Some formulation of it can still be true - albeit in a rather uninteresting way.

It’s not a complete tautology though, if people’s estimates of their skill were accurate in an unbiased way, we wouldn’t see a DK effect (or we’d only see a slight one, since you can’t really be unbiased at the low and high ends of the spectrum as the other comment pointed out). This isn’t true in the case of uniform random data, or in the real data we see, but it could be true of some data.

Re: The Dunning-Kruger Effect Is Autocorrelation

#55

Post by diwank »

Excerpt from a newer paper by Nuhfer (2017) adds more clarity:

“… Our data show that peoples' self-assessments of competence, in general, reflect a genuine competence that they can demonstrate. That finding contradicts the current consensus about the nature of self-assessment. Our results further confirm that experts are more proficient in self-assessing their abilities than novices and that women, in general, self-assess more accurately than men. The validity of interpretations of data depends strongly upon how carefully the researchers consider the numeracy that underlies graphical presentations and conclusions. Our results indicate that carefully measured self-assessments provide valid, measurable and valuable information about proficiency. …”

https://www.researchgate.net/publication/312107583_How_Rando...

Re: The Dunning-Kruger Effect Is Autocorrelation

#56

Post by ithkuil »

In other words people are quite bad at estimating their skill level. Some people will overestimate, while some other people will underestimate and on average there will be a relatively constant estimated skill level that doesn't change all that much based on the actual abilities.

Given that fact, it logically follows that people who score low ability tests will more often than not have overestimated their ability (and the same on the other end of the spectrum).

You can frame this effect as autocorrelation if you wish or just as a logical consequence. But that's missing the point.

The point is: why on earth are humans so bad at estimating their own competence level as to make it practically indistinguishable from random guesses.

Re: The Dunning-Kruger Effect Is Autocorrelation

#57

Post by notahacker »

andersource wrote: post #26
uldos wrote: post #17

Earlier quoted context omitted.

Unskilled people are more random with their self assessment that skilled. It has nothing to do with unskilled people thinking that they know everything.

> Unskilled people are more random with their self assessment than skilled This is a statistical claim, supported by the DK graph (but not the random data thought experiment from the article). > It has nothing to do with unskilled people thinking that they know everything This reads to me as a claim about the psychological reason for the statistical pattern, which I don't think is either supported nor contradicted by…

"Unskilled people often think they know everything" is pretty much the colloquial use of 'Dunning Krueger' to label people who insist that they know better than the experts from a position of relative ignorance

But I think that's consistent with the statistical pattern: if the distribution of self-assessment [amongst unskilled people] of their relative abilities is random or near random, it logically follows that the set of unskilled people includes a lot of people who significantly overestimate their ability at something. Dunning and Kruger don't really talk about the propensity of excellent test performers to underestimate their skill as much (though the Nuher study results somewhat justify their original focus on the ignorant by finding that more skilled groups like professors and graduate students make smaller average prediction errors of their test scores than undergrads).

Dunning and Krueger's contention in the original article is that "incompetence robs people of their ability to realise they're incompetent". Similarity of the prediction errors to a random walk isn't a rebuttal of that (although it's a fair critique of the presentation) because the null hypothesis is that people who find a test particularly difficult shouldn't be [almost] as likely to believe they achieved above average performance as the people who aced it. There might be other reasons for that (like the test being pretty easy for all participants and raw scores in a fairly narrow range, or test takers wrongly assuming their lack of understanding was being compared against the general population rather than other smart undergraduates) but in general people ought to be able to incorporate knowing that they didn't know how to answer a lot of questions into their self-assessment of how they performed.

Re: The Dunning-Kruger Effect Is Autocorrelation

#58

Post by civilized »

fallingfrog wrote: post #50

OK, I think I understand. What the data from the original experiment actually shows is that people at all skill levels are pretty bad at estimating their skill level- it's just that if you scored well, the errors are likely to be underestimates, and if you scored badly, the errors are likely to be overestimates, by pure chance alone. So it's not that low scoring individuals are particularly overconfident so much as e…

So it seems... but I still don't understand why the author thinks it's helpful to say "autocorrelation" dozens of times when he could have just said this.

Re: The Dunning-Kruger Effect Is Autocorrelation

#60

Post by leto_ii »

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means.

DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedasticity).

> a world in which people are very bad at estimating their own skill, therefore, statistically, people with lower skills tend to overestimate their skills, and experts tend to underestimate it.

Be careful here, the conclusion you drew doesn't actually follow.

> y - x ~ x, this is called the residual plot

You're giving x and y meaning that they don't have. In the article these are uncorrelated random variables - the plot of y-x ~ x will always look that way. That's however not the case if you're plotting y_hat - y ~ y_hat for a y_hat taken out of a model. That won't be a random variable in your setup.

Edit: note on heteroskedasticity

Post reply on HN