Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

Post reply on HN
Print view View original

181–190 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#181

Post by saurik »

pdonis wrote: post #159
bumby wrote: post #149

Earlier quoted context omitted.

Thanks for taking the time to help me understand the data. Looking at the graphs, isn't the implication that the distance between the two lines is the metric of interest? In other words, that the distance between real/perceived scores in Q1 & Q2 is "grossly" wider than the distance between Q3 & Q4, with the noted exception that only Q4 underestimates?

> isn't the implication that the distance between the two lines is the metric of interest? That's more or less what D-K are saying, yes. That doesn't necessarily mean it's the best metric to use for understanding what the data is saying. Also, "distance between the two lines" is misleading because the data is bucketed--on the x axis by quartiles, on the y axis by percentiles. In other words, the actual data is the ci…

You seem to be first correcting the statement made by D-K in a way so compatible with this article that it is literally their example and then claiming that this corrected statement is correct so the article is wrong... the actual statement--as you even admit--was wrong, and that their data was fine was not being questioned by the article: it was their interpretation, which is what the D-K effect actually is... an inverse correlation between skill and perceived skill. Yes: if you choose to read their graph in a way which D-K didn't consider you can see the correct statement, but D-K didn't do that, and that isn't what anyone--but I guess you--means when they reference the conclusions of the paper and the resulting "effect", which purports to explain "why people tend to hold overly optimistic and miscalibrated views about themselves".

Re: The Dunning-Kruger Effect Is Autocorrelation

#182

Post by andersource »

uoaei wrote: post #134
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

> the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance That's not really the intended interpretation of "null" in "null hypothesis". "Null" does not mean "contrary to the effect you're testing". "Null" means "do not assume dependencies anywhere" and so your description is b…

"Null hypothesis" comes down to agreeing on a prior that is "reasonable". Most of the time, that indeed means not assuming dependencies, e.g. when testing the outcomes of a medical treatment. But that's not always the case, e.g. does it seem reasonable to you to assume a priori no dependency between a person's age and their height? Does the result "People get taller until the age of 20" merit a journal article? There's a very strong correlation there after all.

As I've written elsewhere this all comes down to the prior, based on my life experience the prior "people are generally capable of self-assessing their performance" is much more likely than "people have absolutely no ability to self-assess their performance."

To state it differently, when you assume no dependencies anywhere, you've already jumped to a conclusion that is more far-fetched to me than the results of DK. Do you really think people have zero ability to self-assess their own performance? In all domains and all contexts, as this is your "null hypothesis"?

Re: The Dunning-Kruger Effect Is Autocorrelation

#183

Post by geysersam »

danbruc wrote: post #98
geysersam wrote: post #38

Earlier quoted context omitted.

I'm not sure what you mean with "the wider the line will get". But here is the issue: The least competent person cannot underestimate their relative competency. Any not exactly accurate estimate they do is an overestimate. Correspondingly, the most competent person cannot overestimate their relative competency. This leads to the perception of bias where there is none, except a trivial tautological one.

I made you a picture [1]. I randomly generated 100 test scores between 0 and 1, then different self assessments. Top left, self assessment matches actual score, top middle, self assessment varies uniformly by ±0.1 around the test score, top right, self assessment varies uniformly by ±0.2 around the test score. None of those have a Dunning-Kruger effect. If you aggregate data points, there will be - as you mentioned -…

I see what you mean. It should be possible to determine if there is an effect using the raw data from the experiment.

Re: The Dunning-Kruger Effect Is Autocorrelation

#184

Post by hgomersall »

Accujack wrote: post #104
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

>the author seems very confident in their conclusion They are, but honestly all that can be concluded safely IMHO is that the original D-K graph doesn't support that the widely discussed "effect" which their conclusion describes exists. Therefore unless there is more evidence from some other subsequent study there may not be any evidence for it at all, and if that's the case then there's potentially no proof it exist…

This is it. You need to write down the model then perform the inference. With some rudimentary munging you can get a super simple linear model.

Re: The Dunning-Kruger Effect Is Autocorrelation

#185

Post by hgomersall »

uoaei wrote: post #134
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

> the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance That's not really the intended interpretation of "null" in "null hypothesis". "Null" does not mean "contrary to the effect you're testing". "Null" means "do not assume dependencies anywhere" and so your description is b…

Null means "that thing which I've decided to attach special significance to". There's in general no reason to prefer a null hypothesis over any other hypothesis.

Re: The Dunning-Kruger Effect Is Autocorrelation

#186

Post by hgomersall »

pdonis wrote: post #127
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

> I really don't see how it concludes that the DK effect is wrong based on the analysis. Neither do I. Basically what the article actually shows is that these two statements are equivalent: (1) People with low test scores tend to overpredict their test scores, while people with high test scores tend to underpredict their test scores. (2) People's predictions of their test scores are uncorrelated (or more precisely ve…

My issue with the thought process is the general picking of uniform distributions to show random data that follows DK. Uniform distributions are kind of uninteresting in this case because they're pretty artificial. It's not like competence is actually measured in the range 0-100 in anything that actually matters.

Re: The Dunning-Kruger Effect Is Autocorrelation

#187

Post by sweezyjeezy »

hungrygs wrote: post #123
TimPC wrote: post #99

The article is correct. The effect is statistical not psychological. It emerges even from artificial data and occurs independently of the supposed psychological justifications even for data where those justifications are clearly removed. If you adjust the experiment design to avoid introducing the auto-correlation you get data that doesn't show the DK effect at all. Some might take issue with the adjusted experiment…

Just anecdotal, but my life observation of DK is often highly intelligent and competent people in a particular field who then generalize that to pontificate and proclaim, directly or indirectly, superior understanding to certified domain experts (e.g., have directly related advanced degree(s), work in the field for decades.) It thus seems more or as much a psychological effect - in short, people with a personality ty…

https://xkcd.com/793/

Re: The Dunning-Kruger Effect Is Autocorrelation

#188

Post by ImaCake »

TimPC wrote: post #99

The article is correct. The effect is statistical not psychological. It emerges even from artificial data and occurs independently of the supposed psychological justifications even for data where those justifications are clearly removed. If you adjust the experiment design to avoid introducing the auto-correlation you get data that doesn't show the DK effect at all. Some might take issue with the adjusted experiment…

Yes you are right and I don't understand how so many commenters in this thread can be so confidently state that the article is wrong. I just went ahead and did the random simulation myself and you get the "Dunning Kruger effect" which is exactly what the author paints it as: autocorrelation.

>If you adjust the experiment design to avoid introducing the auto-correlation you get data that doesn't show the DK effect at all.

This is even in the article! Yet some people are making claims against this despite references to the contrary.

Re: The Dunning-Kruger Effect Is Autocorrelation

#189

Post by ImaCake »

georgefox wrote: post #115

This is a fascinating discussion, to which I have little to add, except this. Quoting the article (including the footnote): > [I]f you carefully craft random data so that it does not contain a Dunning-Kruger effect, you will still find the effect. The reason turns out to be embarrassingly simple: the Dunning-Kruger effect has nothing to do with human psychology[1]. > [1]: The Dunning-Kruger effect tells us nothing ab…

Later in the article the author points to an article which does systematically illustrate that the D-K effect is probably not real. They achieve this by using college education level as an independent proxy for test skill with the Y variable being an unrelated assessment of skill - self-assessment. So we can be pretty confident that the D-K effect is at least very small.

Re: The Dunning-Kruger Effect Is Autocorrelation

#190

Post by pdonis »

saurik wrote: post #181
pdonis wrote: post #159

Earlier quoted context omitted.

> isn't the implication that the distance between the two lines is the metric of interest? That's more or less what D-K are saying, yes. That doesn't necessarily mean it's the best metric to use for understanding what the data is saying. Also, "distance between the two lines" is misleading because the data is bucketed--on the x axis by quartiles, on the y axis by percentiles. In other words, the actual data is the ci…

You seem to be first correcting the statement made by D-K in a way so compatible with this article that it is literally their example and then claiming that this corrected statement is correct so the article is wrong... the actual statement--as you even admit--was wrong, and that their data was fine was not being questioned by the article: it was their interpretation, which is what the D-K effect actually is ... an i…

> it is literally their example

No, it isn't. The article is claiming that the D-K effect doesn't actually exist; actually all that is going on is "autocorrelation". That claim is wrong. The D-K effect does exist, but it isn't what D-K said it was: it's not that there is an inverse correlation between perceived skill and actual skill; it is that there is no correlation (or at best a weak positive correlation) between perceived skill and actual skill (when we would intuitively expect a fairly strong positive correlation between them). That is what D-K's actual data shows; but neither D-K themselves in their paper, nor this article, correctly describe that.

Post reply on HN