Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

Post reply on HN
Print view View original

111–120 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#111

Post by ithkuil »

d0mine wrote: post #94
ithkuil wrote: post #56

In other words people are quite bad at estimating their skill level. Some people will overestimate, while some other people will underestimate and on average there will be a relatively constant estimated skill level that doesn't change all that much based on the actual abilities. Given that fact, it logically follows that people who score low ability tests will more often than not have overestimated their ability (an…

- what DK claims: there is bias (incompetent people overestimate their ability) - what data actually shows: there is a greater variance (incompetent people both over and _under_ estimate to a larger degree compared with more competent people. Data shows heteroscedasticity. No bias (estimations are around zero +/-, tighter for more competent).

The data supports the claim because indeed it turns out that incompetent people overestimate their ability. This phenomenon exhibits itself with random data too, so it clearly doesn't mean that incompetent people overestimate their ability because of their incompetence.

Or is it?

The trick lies in the fact that when asked to judge your competence you're given a range (e.g. 0-10) and both competent people and incompetent people have access to the whole range when taking a self-assessment. I.e. if less competent people were on average more aware of their incompetence they may be less likely to rate themselves 5 or 6, but yet the data shows that no matter what competence level you have on average you self-assess more or less the same.

This seems to imply that your incompetence indeed doesn't allow you to truly appreciate the full range of skills that are required to reach a higher level of competence.

In other words, the DK effect itself is the cause of the random distribution of the skill self-assessment (which in turn is the cause of the overestimation secondary effect)

Re: The Dunning-Kruger Effect Is Autocorrelation

#112

Post by omnicognate »

haberman wrote: post #92
omnicognate wrote: post #86

Earlier quoted context omitted.

> But when you're bad at the skill and can't underestimate, they look the same. The (definitional) difference between bias and variance isn't related to do with whether you're bad at the skill or not. It's just mean vs variance of a probability distribution. If there's a good faith acknowledgement on your part that there's something here you're not getting then I'm very happy to try and help you understand it, and in…

I'm watching this thread with interest and will try to restate my understanding of GP's argument, by means of an example. If a person's true skill is 5 on a 1-100 point scale, but the person is completely unaware of their true skill and will guess randomly, then their estimate will bias heavily in the direction of overestimating their skill, even if they were not intrinsically motivated to overestimate their skill, s…

It's worth reading the discussion between author and Nicolas Bonneel that starts with the first comment below the article. The author's explanation is very helpful regarding this point.

The main point is that in the paper's randomly generated numbers example, the DK effect disappears if you measure the actual "skill" and the "prediction error" in separate, independent experiments. In the example if you take a "person" and conduct the test you get a totally random result, and you get another, independent totally random result if you test them again. If you perform your "actual skill" measurement using one of those test runs and your "skill estimation error" measurement using another, the DK effect disappears completely.

So, to the extent the result of your skills test has any "noisiness" to it, if you analyse it the way Dunning & Kruger did, the autocorrelation resulting from using the same sample of that noise in the two things you're trying to assess the relationship between will show up as a powerful DK effect, and can easily swamp any actual correlations in the underlying distribution.

Edit: Also worth mentioning footnote 3 on the article, which points out that the use of quantiles introduces a separate bias for the same reason you mention (about there being a minimum and maximum score).

Re: The Dunning-Kruger Effect Is Autocorrelation

#113

Post by _dain_ »

nbernard wrote: post #64
_dain_ wrote: post #27

Earlier quoted context omitted.

Okay well in that case the article is deficient (I skimmed it, assuming it was making the same case I already knew about[1] -- my bad). You can actually reproduce the DK graph without supposing that estimation bias depends on skill. There is an interactive visualization here[2] that does precisely that (click on the "line plot w/ centiles" tab). You can adjust the parameters to see how it behaves. [1] https://news.yc…

Thanks for the link! I see what you mean, but is using the "general overestimation" slider actually unbiased? I don't really understand what it is supposed to represent. Looking at the code, it seems to be only an additive term that shifts the plot upwards, but that does not explain anything (actually, in the code, the variable is called "up_bias", and could correspond to an actual DK effect...).

Per DK's hypothesis, the added bias would be a decreasing function of underlying skill, not a constant as it is here. I think this is the only way to define it sensibly. It's easier to see what it's doing if you look at the scatterplot, although it's kind of annoying because the axes are flipped and it keeps the view centred on the data, so the points stay still and the axes move around them. Basically it's just translating all the points uniformly along the estimated score axis.

Re: The Dunning-Kruger Effect Is Autocorrelation

#114

Post by omnicognate »

andersource wrote: post #95
omnicognate wrote: post #86

Earlier quoted context omitted.

> But when you're bad at the skill and can't underestimate, they look the same. The (definitional) difference between bias and variance isn't related to do with whether you're bad at the skill or not. It's just mean vs variance of a probability distribution. If there's a good faith acknowledgement on your part that there's something here you're not getting then I'm very happy to try and help you understand it, and in…

Here's my main point of confusion - what does the random data experiment have to do with the DK results? As stated elsewhere DK has 2 claims: 1. Low-skilled people overestimate their performance and skilled people underestimate their performance 2. Skill correlates with self-assessment accuracy My first issue with the article is that it implies that since we get effect #1 with random data, that invalidates the respec…

As far as I can see (having checked wiki and the abstract of the original paper - I'm no expert on this) the DK effect is only the first of those claims. However it sounds like claim 2 is less significant here anyway.

Re claim 1 the random numbers example is "all noise, no signal" and I can see the objection that a more convincing example might be to demonstrate the "false" DK effect in an example that does have some signal (i.e. a positive relationship between actual and estimated skill), but that is easy to do and I hope you'll be able to see why if you see my reply at https://news.ycombinator.com/item?id=31042619 and read the comments under the article I mentioned there.

The point is that the DK analysis involves comparing two things which both contain the same single sample from a noise source. Pure noise like the random numbers in the example displays a powerful DK effect due to autocorrelation that says nothing interesting (just that a single random sample of noise is correlated with itself), and that powerful effect can swamp any actual relationships in the distributions. To avoid that effect appearing, you have to make sure that if the two things you are comparing contain samples of a single noise source they are separate, independent samples of it. The experiment with the education level groups achieves this because the education level is "measured" as a separate event from the "actual" skill measurement so they have separate noise sources (and even if they didn't the noise source would have been sampled separately and independently).

I have to say, during the discussion above I hadn't thought through it deeply enough to grok this level of it, and while pondering your last comment I went through a phase of "hang on, am I actually understanding this myself?", so I apologise and retract any suggestion of bad faith.

Re: The Dunning-Kruger Effect Is Autocorrelation

#115

Post by georgefox »

This is a fascinating discussion, to which I have little to add, except this. Quoting the article (including the footnote):

> [I]f you carefully craft random data so that it does not contain a Dunning-Kruger effect, you will still find the effect. The reason turns out to be embarrassingly simple: the Dunning-Kruger effect has nothing to do with human psychology[1].

> [1]: The Dunning-Kruger effect tells us nothing about the people it purports to measure. But it does tell us about the psychology of social scientists, who apparently struggle with statistics.

It seems to me that despite rudely criticizing a broad swath of academics for their lack of statistical prowess, the author here is himself guilty of a cardinal statistical sin: accepting the null hypothesis.

The fact that data resemble a random simulation in which no effect exists does not disprove the existence of such an effect. In traditional statistical language, we might say such an effect is not statistically significant, but that is different from saying that the effect is absolutely and completely the result of a statistical artifact.

The nuance of statistics is never-ending.

Re: The Dunning-Kruger Effect Is Autocorrelation

#116

Post by [deleted] »

TimPC wrote: post #99

The article is correct. The effect is statistical not psychological. It emerges even from artificial data and occurs independently of the supposed psychological justifications even for data where those justifications are clearly removed. If you adjust the experiment design to avoid introducing the auto-correlation you get data that doesn't show the DK effect at all. Some might take issue with the adjusted experiment…

[deleted]

Re: The Dunning-Kruger Effect Is Autocorrelation

#117

Post by leto_ii »

andersource wrote: post #66
leto_ii wrote: post #60

Earlier quoted context omitted.

> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedast…

> DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedasticity). Not sure I follow, inverse correlation between what? The analysis at the bottom (assuming you mean fig. 11) is too dense to show if there's a correlation or not (between skill and self-assessment bias)…

> If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated

1. As mentioned in another comment, X and Y can't be independent and correlated at the same time

2. The point of the article is to show that you can replicate the results of the DK paper starting from purely random data. In the article X and Y don't mean anything, they're just random variables that the author draws samples from. The fact that you can get DK results from this very strongly suggests that DK is just an artifact of statistics, not an actual result.

> What do you mean by "the plot of y-x ~ x will always look that way" - what way?

See Figure 8 in the article, also panel B in Figure 10.

> The shape of the plot will necessarily depend on the relationship between x and y.

What I meant to say is that since X and Y are simulated data, not actual observations, the shape in Figure 8 will not actually depend on any possible relationship between ability and self-assessment. It's just a statistical artifact.

Figure 9 is based on this simulated data as well, and since it closely replicates Figure 2 there's good reason to believe that Figure 2 itself is actually just a statistical artifact and that the DK data don't actually show the purported correlation.

This point is further strengthened by referring to a few papers and by showing some corrected results in Figure 11.

Re: The Dunning-Kruger Effect Is Autocorrelation

#118

Post by brnaftr361 »

bitshiftfaced wrote: post #67
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

Check out Nuhfer et al 2016, who had a different explanation for why Dunning Kruger wasn't true. Dunning Kruger effect: lower performers overestimate their ability, and higher performers underestimate their ability. How did they find that? They asked participants to take a test and then had them do a self-assessment. Both were standardized from 0-100. They rated a participant's self-assessment accuracy by "self-asses…

Author includes reference to Nuhfer, related studies can be found here:

https://digitalcommons.usf.edu/numeracy/vol9/iss1/art4/ https://digitalcommons.usf.edu/numeracy/vol10/iss1/art4/

Re: The Dunning-Kruger Effect Is Autocorrelation

#119

Post by andersource »

leto_ii wrote: post #117
andersource wrote: post #66

Earlier quoted context omitted.

> DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedasticity). Not sure I follow, inverse correlation between what? The analysis at the bottom (assuming you mean fig. 11) is too dense to show if there's a correlation or not (between skill and self-assessment bias)…

> If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated 1. As mentioned in another comment, X and Y can't be independent and correlated at the same time 2. The point of the article is to show that you can replicate the results of the DK paper starting from purely random data. In the article X and Y don't mean anything,…

> 1. As mentioned in another comment, X and Y can't be independent and correlated at the same time

And as I replied to that comment, "sorry, my bad - Y - X and X will be negatively correlated."

> 2. The point of the article is to show that you can replicate the results of the DK paper starting from purely random data

You (and many others) are using "purely random data" as if it's always the null hypothesis and using it to cast doubt on the results. But assuming as the null model 0 correlation between skill and self-assessment of that skill makes no sense to me, and is in fact more extreme than the claim DK is making. So in other words, sure, if you assume something more extreme than the claim and generate data based on this assumption, you'll get the same effect and more extreme.

Re: The Dunning-Kruger Effect Is Autocorrelation

#120

Post by longtimegoogler »

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

Yup. THat's what I was going to say. There data suggests that everyone kind of estimates their ability similarly so that more skilled people underestimate there ability (impostor's syndrome) and less skilled people overestimate the their abilities.
Post reply on HN