Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

Post reply on HN
Print view View original

31–40 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#31

Post by fullshark »

oceliker wrote: post #6

I think the gist of the article is this: Suppose you make 1000 people take a test. Suppose all 1000 of these people are utterly incapable of evaluating themselves, so they just estimate their grade as a uniform random variable between 0-100, with an average of 50. You plot the grades of each of the 4 quartiles and it shows a linear increase as expected. Let's say the bottom quartile had an average of 20, and the top…

Yeah we learned people are bad at giving themselves percentile rankings apparently, especially when the population is illdefined ("your peers").

https://www.avaresearch.com/files/UnskilledAndUnawareOfIt.pd...

Re: The Dunning-Kruger Effect Is Autocorrelation

#32

Post by andersource »

IceDane wrote: post #21
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis. I'll admit that part of it also comes from personal experience, at work and elsewhere. I've met some catastrophically incompetent people were completely oblivious to their own incompetence, and this has very often felt like that the more incompetent they were, the more likely they were to be try to do stuff that…

This is my understanding as well.

Re: The Dunning-Kruger Effect Is Autocorrelation

#34

Post by omnicognate »

andersource wrote: post #30
omnicognate wrote: post #28

Earlier quoted context omitted.

The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed effect. To gather evidence that the effect doesn't actually exist you need a new experiment, not just a new analysis of the same data, because the the different levels of actual skill need to be established separately from establishing the error in skill self-as…

> The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed effect I don't understand this part. "Completely uncorrelated data" is usually taken to represent the null hypothesis, but that's not the case here. In the DK paper, the implicit null hypothesis is "people of all skill levels are good at estimating their perfo…

The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias" in estimating their performance" (remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill). The randomly generated data used is one example of that, albeit an unrealistic one: a world in which all people are completely incapable of estimating their performance at all. In this case all of them have no bias at all in their (totally random) estimates. The fact that the artifact is seen in such data is a powerful demonstration that it is not evidence of the Dunning-Kruger effect.

Re the other experiment, as I say it's just one study and I haven't looked deeply into it or others (and nor do I have a position on whether there is a real effect of this nature, or any great interest in it). The point is the article isn't claiming their randomly generated data example is evidence against the Dunning-Kruger effect itself. That needs further experiments such as the one they showed. The random data example is a demonstration that the original paper's analysis is flawed and doesn't support its conclusions.

Re: The Dunning-Kruger Effect Is Autocorrelation

#35

Post by geysersam »

omnicognate wrote: post #34
andersource wrote: post #30

Earlier quoted context omitted.

> The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed effect I don't understand this part. "Completely uncorrelated data" is usually taken to represent the null hypothesis, but that's not the case here. In the DK paper, the implicit null hypothesis is "people of all skill levels are good at estimating their perfo…

The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias " in estimating their performance" (remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill). The randomly generated data used is one example of that, albeit an…

Maybe another way of putting it is that the "Dunner-Kruger effect" is simply a tautology.

Some formulation of it can still be true - albeit in a rather uninteresting way.

Re: The Dunning-Kruger Effect Is Autocorrelation

#36

Post by bouncycastle »

IceDane wrote: post #21
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis. I'll admit that part of it also comes from personal experience, at work and elsewhere. I've met some catastrophically incompetent people were completely oblivious to their own incompetence, and this has very often felt like that the more incompetent they were, the more likely they were to be try to do stuff that…

The problem is that they calculated each person’s ‘self-assessment error’ with the actual test score. This error is the difference between a person’s self assessment and their test score,

This is like comparing x - y to x, and if you do this, you will get a correlation no matter what.

Re: The Dunning-Kruger Effect Is Autocorrelation

#37

Post by jakear »

Article seems to be saying “DK doesn’t exist because it always exists”. Which is… absurd?

The point of DK is that when you don’t know shit, any non-degenerate self assessment will result in overestimating your ability. In short, “there are more natural numbers above smaller natural numbers than bigger ones”. This doesn’t have to do with psychology, and it’s expected that it appears when evaluating random data. That’s a good thing! It means DK exists even when us pesky humans aren’t involved at all, not that DK doesn’t exist at all.

Re: The Dunning-Kruger Effect Is Autocorrelation

#38

Post by geysersam »

danbruc wrote: post #15

I am no expert in statistics or the Dunning-Kruger effect but this analysis doesn't sound correct to me. If you plot self assessment against test scores then the following will happen. If people are perfect at self assessment, then you get a straight diagonal line. The more wrong they are, the wider the line will get, in the extreme - if the self assessment is unrelated to the test result - the line will cover the en…

I'm not sure what you mean with "the wider the line will get". But here is the issue:

The least competent person cannot underestimate their relative competency. Any not exactly accurate estimate they do is an overestimate.

Correspondingly, the most competent person cannot overestimate their relative competency.

This leads to the perception of bias where there is none, except a trivial tautological one.

Re: The Dunning-Kruger Effect Is Autocorrelation

#39

Post by motoboi »

IceDane wrote: post #21
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis. I'll admit that part of it also comes from personal experience, at work and elsewhere. I've met some catastrophically incompetent people were completely oblivious to their own incompetence, and this has very often felt like that the more incompetent they were, the more likely they were to be try to do stuff that…

The feeling your got from looking at the graph come from the idea that the red line represent absolute values. It’s actually an average of the real value as that is a really big difference and of the reasons why it’s very easy to lie with statistics.

To correct correlate the two variables (assessment and actual-score) you need to correlate the actual data, not measures of its caracteristics (average being one of them).

The actual data is shown. Even by eye it’s possible to see no strong (or significant) correlation exists.

Re: The Dunning-Kruger Effect Is Autocorrelation

#40

Post by andersource »

bouncycastle wrote: post #36
IceDane wrote: post #21

Earlier quoted context omitted.

I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis. I'll admit that part of it also comes from personal experience, at work and elsewhere. I've met some catastrophically incompetent people were completely oblivious to their own incompetence, and this has very often felt like that the more incompetent they were, the more likely they were to be try to do stuff that…

The problem is that they calculated each person’s ‘self-assessment error’ with the actual test score. This error is the difference between a person’s self assessment and their test score, This is like comparing x - y to x, and if you do this, you will get a correlation no matter what.

Not necessarily. If for example y =~ x, comparing x - y to x would yield zero correlation, which to me is what DK is all about - people _aren't_ as good as we would expect at estimating their own skill.
Post reply on HN