Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

161–170 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#161
post #129

Earlier quoted context omitted.

> DK doesn't mean no correlation, it means inverse correlation. No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. Which is not what we would expect: we would expect that their predictions of their test scores would be strongly (or at least more st…

>> DK doesn't mean no correlation, it means inverse correlation. > No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. The problem is that in common Internet discussions, there is no fixed meaning of D-K. When things heat up, D-K becomes "his belief…

> in common Internet discussions, there is no fixed meaning of D-K

That's why I focused explicitly on the actual data in the D-K paper and what it actually says.

Re: The Dunning-Kruger Effect Is Autocorrelation

#162
post #37

Article seems to be saying “DK doesn’t exist because it always exists”. Which is… absurd? The point of DK is that when you don’t know shit, any non-degenerate self assessment will result in overestimating your ability. In short, “there are more natural numbers above smaller natural numbers than bigger ones”. This doesn’t have to do with psychology, and it’s expected that it appears when evaluating random data. That’s…

> In short, “there are more natural numbers above smaller natural numbers than bigger ones” I get what you are trying to say, but this isn’t true… every natural number has the same amount of numbers greater and smaller… an infinite number.

Natural numbers are not the same as integers. They’re strictly positive.

(Now rate your confidence in making that assertion.)

Re: The Dunning-Kruger Effect Is Autocorrelation

#163
post #158

Earlier quoted context omitted.

You're changing the definition of the null hypothesis. What you are essentially saying is that there is no need to perform experiments and studies, I can arbitrarily grab data from anywhere and if it fails to support the hypothesis... then the hypothesis is wrong.

if your analysis produces an effect when fed uniform random data, then yes, no need to perform experiments or studies, because all your results are null. right? so the hypothesis would be something like "the analysis shows a relationship between the two variables" and the null hypothesis would be something like "the analysis shows a relationship between two uniform random variables" and in this case that null is show…

note that i have not carefully been through the claimed analysis here (and specifically have doubts about the data v. error plot) but if the claim that the analysis produces the effect with random inputs is assumed, then the whole thing can be rejected at step 0, right?

Re: The Dunning-Kruger Effect Is Autocorrelation

#164

Earlier quoted context omitted.

Here's my main point of confusion - what does the random data experiment have to do with the DK results? As stated elsewhere DK has 2 claims: 1. Low-skilled people overestimate their performance and skilled people underestimate their performance 2. Skill correlates with self-assessment accuracy My first issue with the article is that it implies that since we get effect #1 with random data, that invalidates the respec…

As far as I can see (having checked wiki and the abstract of the original paper - I'm no expert on this) the DK effect is only the first of those claims. However it sounds like claim 2 is less significant here anyway. Re claim 1 the random numbers example is "all noise, no signal" and I can see the objection that a more convincing example might be to demonstrate the "false" DK effect in an example that does have some…

Thanks! No worries, I appreciate you writing this.

> I can see the objection that a more convincing example might be to demonstrate the "false" DK effect in an example that does have some signal

More than that - as it is, the argument is meaningless to me. It states that DK is trivial in a world where all people have no ability whatsoever to assess their own performance. OK, and finding dinosaur bones is uninteresting in a world where dinosaurs roam free. Both are true, but both are irrelevant in our world (considering my priors). To give a less hyperbolic example, suppose I found some population of people whose weight and height correlate much less than we currently measure, through some biological mechanism of very high variance in bone density or something. To me this article is like saying "well yeah, but this finding is uninteresting, for example if you take purely random weight and height you get an even stronger effect of short people with very high bone density and tall people with very low bone density".

Regarding all the rest - I don't really understand all this "comparing things which both contain the same single sample from a noise source". I'm currently willing to bet (albeit not too much) that any synthetic data experiment you'll come up with, that doesn't display an effect through the DK analysis, will turn out to be based on assumptions that strongly align with my prior, which is that subjects' self-assessment of their performance is correlated to their performance, with 0 bias (on average) and noise that is small (but not negligible) compared to the signal. Would be interested to be proven wrong.

> Pure noise like the random numbers in the example displays a powerful DK effect due to autocorrelation that says nothing interesting

On the contrary, finding out that the distribution in the real world is like that ("pure noise") would be very surprising (therefore interesting, in a sense) to me.

Re: The Dunning-Kruger Effect Is Autocorrelation

#165
post #157

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

You've been corrected about "what DK means" in the other comments, but this is not quite the point of the post. This is not about if DK (as expressed in English words) is true or not — in fact, author points out in the beginning that it's one of these "everybody knows it's like that" ideas (as it often is with social psychology). The point is, that the original DK paper is bullshit. At least, this plot is. And people…

I strongly disagree, not necessarily with everything (e.g. I don't have access to the raw data from the DK experiment, don't know how well they performed all the analysis leading to the plot). But the plot itself is not inherently deceptive, and, unlike implied in the article, is not equivalent to "just comparing x to x". The plot essentially shows the actual performance vs. self-assessment of performance, compared to what we would expect if there were perfect correlation.

Re: The Dunning-Kruger Effect Is Autocorrelation

#166
post #126

Earlier quoted context omitted.

Here's my main point of confusion - what does the random data experiment have to do with the DK results? As stated elsewhere DK has 2 claims: 1. Low-skilled people overestimate their performance and skilled people underestimate their performance 2. Skill correlates with self-assessment accuracy My first issue with the article is that it implies that since we get effect #1 with random data, that invalidates the respec…

For 1. Their evidence does not support the argument, since they impose a boundary condition on how low you can estimate and how low you can perform. To show that high performers don't overestimate their skills you have to give them the opportunity to

Sorry, but this doesn't make sense to me. There has to be a boundary - a person who got all the answers wrong can't underestimate their performance, and a person who got all the answers right can't overestimate their performance. You could make the case that boundary effects are all DK is about, but that's not what the article is doing (and also I don't think such a claim is supported by the DK plot).

Re: The Dunning-Kruger Effect Is Autocorrelation

#167
post #158

Earlier quoted context omitted.

You're changing the definition of the null hypothesis. What you are essentially saying is that there is no need to perform experiments and studies, I can arbitrarily grab data from anywhere and if it fails to support the hypothesis... then the hypothesis is wrong.

if your analysis produces an effect when fed uniform random data, then yes, no need to perform experiments or studies, because all your results are null. right? so the hypothesis would be something like "the analysis shows a relationship between the two variables" and the null hypothesis would be something like "the analysis shows a relationship between two uniform random variables" and in this case that null is show…

In the case of DK, the hypothesis is that there is a bizarre relation between perceived skill and actual skill. The null hypothesis is that is there no relation or correlation between perceived skill and actual skill.

A researcher would perform an experiment. If researcher observes statistically significant results. Then the researcher can reject the null hypothesis, and say the theory is valid. If researcher does not observe statistically significant results. They can only say their experiment doesn't support the theory.

Re: The Dunning-Kruger Effect Is Autocorrelation

#168
post #60

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedast…

> DK doesn't mean no correlation, it means inverse correlation

Both are wrong. DK effect describes a weak positive correlation, but weaker than we intuitively expect. The top quartile still estimates their ability better than the bottom - positive correlation. But this is still below their actual ability. The bottom quartile correctly predicts they score worse than the top, but they overestimate their own ability ie they underestimate how much better the top quartile actually performs.

Re: The Dunning-Kruger Effect Is Autocorrelation

#169
post #157

Earlier quoted context omitted.

You've been corrected about "what DK means" in the other comments, but this is not quite the point of the post. This is not about if DK (as expressed in English words) is true or not — in fact, author points out in the beginning that it's one of these "everybody knows it's like that" ideas (as it often is with social psychology). The point is, that the original DK paper is bullshit. At least, this plot is. And people…

I strongly disagree, not necessarily with everything (e.g. I don't have access to the raw data from the DK experiment, don't know how well they performed all the analysis leading to the plot). But the plot itself is not inherently deceptive, and, unlike implied in the article, is not equivalent to "just comparing x to x". The plot essentially shows the actual performance vs. self-assessment of performance, compared t…

Well, I wouldn't say "equivalent" (and I wouldn't say it's implied in the article), but it isn't very far from that. And meaning of "very far" is relative. I mean, f(x,y) ~ x obviously isn't equivalent to x ~ x, but the importance of pointing this out depends on how likely people are to mistake it for x ~ y, and how likely they are to skip that f() is DEFINED by authors and not found in the nature. Same with "misleading": your perspective on if something is misleading obviously depends on where it leads you. But my evaluation of 2 questions above are "very likely" and "it does lead majority of people to the wrong conclusions", so the whole sentiments stands true. I'll get back to it in a moment.

Anyway, my (lesser) point is that a study like that (even in 1999) must contain at least one plot of [actual score] vs [self-assessment score], or it is worthless, and I stand by it. (The original paper doesn't contain such plot.) My more general point is that what is accepted as "sufficient" data provided in overwhelming majority of research is laughable, and I really want people to stop tolerating that.

(I don't want to delve into discussing the particular paper, I'm purposefully trying to keep all my point as general as possible. Some problems are actually addressed in DK-1999, but the whole study makes you smirk at a circus social psychology experiments are. Here's a nice quote for you (being tested is "ability to recognize what's funny"): "To assess joke quality, we contacted several professional comedians … and asked them to rate each joke on a scale ranging from 1 (not at all funny) to 11 (very funny). Eight comedians responded to our request …, an analysis of interrater correlations found that one (and only one) comedian's ratings failed to correlate positively with the others (mean r = -.09). We thus excluded this comedian's ratings in our calculation of the humor value of each joke.")

Now, back to the f(x, y) ~ x. Let's imagine f(x, y) = C. E.g., every respondent evaluated himself as "average". Is it true that less knowledgeable people turned out to overestimate themselves, and more knowledgeable people turned out to underestimate themselves? Well, yeah, I guess. Does x correlate with y? No. So, what's more valuable here, to plot [x ~ y] or [f(x, y) ~ x]? Moreover, this is obviously different from the case where y (the self-assessment score) is uniformly distributed and doesn't correlate with x, yet it will yield the same line-plot as the above, if we construct them like D&K did. And that's skipping the part that there are several viable ways to define "self-assessment of performance" as f(x,y). I think, if I'll try hard enough I would be able to "prove" almost any hypothesis about self-assessment using this method. Of course, I won't really prove anything, but the plots will be convincing enough, and the "results" will be much more demonstrative than the original data is, same as probably is with DK-1999 (but that's impossible to tell without seeing actual source data of DK-1999).

TL;DR: It all boils down to the old adage about "3 kinds of lies". Is the result being discussed an example of auto-correlation? Yes, it is. Is auto-correlation necessarily bad? Not really. Is it important and can it obstruct interpretation in the particular example of DK-1999 paper? It's subjective and hasn't been measured, but my evaluation is — absolutely.

Re: The Dunning-Kruger Effect Is Autocorrelation

#170

Earlier quoted context omitted.

> If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated 1. As mentioned in another comment, X and Y can't be independent and correlated at the same time 2. The point of the article is to show that you can replicate the results of the DK paper starting from purely random data. In the article X and Y don't mean anything,…

> 1. As mentioned in another comment, X and Y can't be independent and correlated at the same time And as I replied to that comment, "sorry, my bad - Y - X and X will be negatively correlated." > 2. The point of the article is to show that you can replicate the results of the DK paper starting from purely random data You (and many others) are using "purely random data" as if it's always the null hypothesis and using…

The hypothesis and distribution of data are irrelevant. The artifact comes from x being on both sides of the regression equation.
Post reply on HN