Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

Post reply on HN
Print view View original

191–200 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#191

Post by saurik »

pdonis wrote: post #190
saurik wrote: post #181

Earlier quoted context omitted.

You seem to be first correcting the statement made by D-K in a way so compatible with this article that it is literally their example and then claiming that this corrected statement is correct so the article is wrong... the actual statement--as you even admit--was wrong, and that their data was fine was not being questioned by the article: it was their interpretation, which is what the D-K effect actually is ... an i…

> it is literally their example No, it isn't. The article is claiming that the D-K effect doesn't actually exist; actually all that is going on is "autocorrelation". That claim is wrong. The D-K effect does exist, but it isn't what D-K said it was: it's not that there is an inverse correlation between perceived skill and actual skill; it is that there is no correlation (or at best a weak positive correlation) between…

> The D-K effect does exist, but it isn't what D-K said it was...

You are claiming an effect exists, and no one--and certainly not the author of this article, who seem to be fully bought into your very different statement of the effect (and so maybe we shall call it "the pdonis effect")--is disagreeing with you on that point; however, your insistence upon calling this effect "the Dunning-Kruger effect"--while simultaneously admitting that Dunning and Kruger described a different incorrect analysis as their "effect"--is what is unreasonable. Put simply: "the Dunning-Kruger effect" is the thing that Dunning and Kruger claimed to be true, not some other random property of the data that their same experiment could have backed up, if only they had realized; and it is this effect--the actual Dunning-Kruger effect, as described by Dunning and Kruger in no uncertain terms--which does not seem to exist.

Re: The Dunning-Kruger Effect Is Autocorrelation

#192

Post by doetoe »

ImaCake wrote: post #188
TimPC wrote: post #99

The article is correct. The effect is statistical not psychological. It emerges even from artificial data and occurs independently of the supposed psychological justifications even for data where those justifications are clearly removed. If you adjust the experiment design to avoid introducing the auto-correlation you get data that doesn't show the DK effect at all. Some might take issue with the adjusted experiment…

Yes you are right and I don't understand how so many commenters in this thread can be so confidently state that the article is wrong. I just went ahead and did the random simulation myself and you get the "Dunning Kruger effect" which is exactly what the author paints it as: autocorrelation. >If you adjust the experiment design to avoid introducing the auto-correlation you get data that doesn't show the DK effect at…

The article shows that the qualitative claim, that low scoring people tend to overestimate, and high scoring people tend to underestimate their ability, is nothing but a statistical obviety.

For the Dunning-Kruger effect to have psychological significance, you must quantify this, and show that they overestimate their abilities more resp. less than expected.

Regardless of how you set your expectation/null hypothesis, the absence of the effect would mean that the lowest scoring quartile would on average estimate their abilities to lie at 50% or below. It is found however that people in the lowest scoring quartile position themselves in the third or fourth quartile.

I'm not saying that this is necessarily deep or unexpected, just that the article only shows that the qualitative statement is true regardless of any psychological factors, and not that the Dunning-Kruger effect doesn't exist

Re: The Dunning-Kruger Effect Is Autocorrelation

#193

Post by ImaCake »

doetoe wrote: post #192
ImaCake wrote: post #188

Earlier quoted context omitted.

Yes you are right and I don't understand how so many commenters in this thread can be so confidently state that the article is wrong. I just went ahead and did the random simulation myself and you get the "Dunning Kruger effect" which is exactly what the author paints it as: autocorrelation. >If you adjust the experiment design to avoid introducing the auto-correlation you get data that doesn't show the DK effect at…

The article shows that the qualitative claim, that low scoring people tend to overestimate, and high scoring people tend to underestimate their ability, is nothing but a statistical obviety. For the Dunning-Kruger effect to have psychological significance, you must quantify this, and show that they overestimate their abilities more resp. less than expected. Regardless of how you set your expectation/null hypothesis,…

The y-axis is actual_score - self_assessed_score, right?

So we are interested in whether the average of this y-axis value differs between high scoring and low scoring groups. The study that shows there is no D-K effect [1] linked by the OP article shows there isn't a difference in the average value (y-axis) between groups. In fact the mean value was 0 for all quartiles. So we can conclude that we have found no systematatic over/under reporting of actual vs percieved skill. People in the lowest quartile do not appear to position themselves higher than they should. Or at least, we can't reject the null hypothesis

1. See figure 2 on page 15, note that zero means agreement between self-assessed and actual: https://digitalcommons.usf.edu/cgi/viewcontent.cgi?article=1...

Re: The Dunning-Kruger Effect Is Autocorrelation

#194

Post by TimPC »

hungrygs wrote: post #123
TimPC wrote: post #99

The article is correct. The effect is statistical not psychological. It emerges even from artificial data and occurs independently of the supposed psychological justifications even for data where those justifications are clearly removed. If you adjust the experiment design to avoid introducing the auto-correlation you get data that doesn't show the DK effect at all. Some might take issue with the adjusted experiment…

Just anecdotal, but my life observation of DK is often highly intelligent and competent people in a particular field who then generalize that to pontificate and proclaim, directly or indirectly, superior understanding to certified domain experts (e.g., have directly related advanced degree(s), work in the field for decades.) It thus seems more or as much a psychological effect - in short, people with a personality ty…

Anecdotes are not data. Even in plural form. We can witness isolated instances of what seems to be a phenomenon without it actually being part of a phenomenon. Other posters have already established that some experts can overestimate their expertise as well. The study mentioned by the original post and in my comment seems to suggest that the overestimation bias is prevalent across a wide range of cohorts of expertise. Senior students are just as likely to overestimate their talents as freshmen for instance. This effect likely extrapolates to experts as well it's just hard to get good data.

No DK doesn't say no bluster, no proclamations or no artificial assertions of expertise. It doesn't even say that the overestimates are just as prevalent among experts as laypeople. All it says is as near as we can tell the effect size of the overestimation is the statistical autocorrelation and our best efforts to produce the same effect without relying on the autocorrelation have failed.

I think there are a lot of ways to accept the anecdotes you mentioned occur that need much weaker assertions than DK as a psychological phenomenon and would hesitate to jump to DK based on that information.

Re: The Dunning-Kruger Effect Is Autocorrelation

#195

Post by uoaei »

hgomersall wrote: post #185
uoaei wrote: post #134

Earlier quoted context omitted.

> the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance That's not really the intended interpretation of "null" in "null hypothesis". "Null" does not mean "contrary to the effect you're testing". "Null" means "do not assume dependencies anywhere" and so your description is b…

Null means "that thing which I've decided to attach special significance to". There's in general no reason to prefer a null hypothesis over any other hypothesis.

As stated above, it means precisely the opposite of that.

Re: The Dunning-Kruger Effect Is Autocorrelation

#196

Post by uoaei »

andersource wrote: post #182
uoaei wrote: post #134

Earlier quoted context omitted.

> the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance That's not really the intended interpretation of "null" in "null hypothesis". "Null" does not mean "contrary to the effect you're testing". "Null" means "do not assume dependencies anywhere" and so your description is b…

"Null hypothesis" comes down to agreeing on a prior that is "reasonable". Most of the time, that indeed means not assuming dependencies, e.g. when testing the outcomes of a medical treatment. But that's not always the case, e.g. does it seem reasonable to you to assume a priori no dependency between a person's age and their height? Does the result "People get taller until the age of 20" merit a journal article? There…

I invite you to ponder on the etymology and actual definition of the word "null".

It does not refer to some antecedent or to some notion of "reason", it refers as explicitly as possible to "not-x", contra whatever effect generates "x".

You are conflating the construction of the null hypothesis with a more liberal notion of hypothesis testing (science) per se.

Re: The Dunning-Kruger Effect Is Autocorrelation

#197

Post by roguecoder »

The point is that if people estimate their abilities at random, with no information, it will look like people who perform worse over-estimate their performance. But it isn't because people who are bad at a thing are any worse at estimating their performance than people who are good at the thing: they are both potentially equally bad at estimating their performance, and then one group got lucky and the other didn't.

It would require them to be _even worse that random_ for them to be worse at estimating their abilities, rather than simply being judged for being bad at the task. It is only human attribution bias that leads us to assume that people should already know whether they are good or bad at a task without needing to being told.

The study assumed that the results on the task are non-random, performance is objective, and that people should reasonably have been expected to have updated their uniform Bayesian priors before the study began.

If any of those are not true, we would still see the same correlation, but it wouldn't mean anything except that people shared a reasonable prior about their likely performance on the task.

People will nevertheless attribute "accurate" estimates to some kind of skill or ability, when the only thing that happened is that you lucked into scoring an average score. You could ask people how well they would do at predicting a coin flip and after the fact it would look like whoever guessed wrong over-estimated their "ability" and a person who guessed right under-estimated theirs, even though they were both exactly accurate.

This comment section clearly demonstrates the attribution bias that makes this myth appealing, though. And this blog post demonstrates how difficult it is to effectively explain the implications of Bayesian reasoning without using the concept.

Re: The Dunning-Kruger Effect Is Autocorrelation

#198

Post by roguecoder »

roguecoder wrote: post #197

The point is that if people estimate their abilities at random, with no information, it will look like people who perform worse over-estimate their performance. But it isn't because people who are bad at a thing are any worse at estimating their performance than people who are good at the thing: they are both potentially equally bad at estimating their performance, and then one group got lucky and the other didn't. I…

Consider the original study: they used 45 Cornell undergraduate students and asked them about grammar. Grammar isn't objective. Everyone there had performed well on the verbal portion of the SAT, but they weren't studying grammar and hadn't gotten instruction on this particular book of grammar they were judged against. It is very likely that what they were capturing in the "better" or "worse" scores is differences in local dialect.

They then judged people whose beliefs about grammar varied from the one book's beliefs about grammar as having over-estimated their performance. They took people out of one context, asked them how they would behave in a novel context, and everyone made an educated guess. The people who guessed correctly were judged to accurately know their own abilities, when actually they may just have gotten lucky.

Thus what Dunning-Kruger's paper actually says is that if you want people to know how you would like them to perform a task, you can't assume they will read your mind: you have to provide them with actual feedback on their performance.

Re: The Dunning-Kruger Effect Is Autocorrelation

#199

Post by roguecoder »

PaulKeeble wrote: post #13

Modern Psychology is having a lot of these sorts of results over the last decade, none of their methods are holding up under proper scrutiny. They are struggling to reproduce findings but more critically even the reproduced ones are turning out to be statistical and mathematical errors like shown here. Some of the findings have also done severe harm to patients over the decades as well, I can't help but think we need…

Maybe asking 45 Cornell undergraduates and then generalizing that to "people" was a bad idea after all.

Re: The Dunning-Kruger Effect Is Autocorrelation

#200

Post by [deleted] »

Phileosopher wrote: post #29
PaulKeeble wrote: post #13

Modern Psychology is having a lot of these sorts of results over the last decade, none of their methods are holding up under proper scrutiny. They are struggling to reproduce findings but more critically even the reproduced ones are turning out to be statistical and mathematical errors like shown here. Some of the findings have also done severe harm to patients over the decades as well, I can't help but think we need…

I'm convinced it's associated with the methodology of how psychology has approached matters. In the world of biology, you're observing the world around you. Same for physics, chemistry, et al. This means that you can set up proper controls to obscure your own presence from any potential results (e.g., isolate everything in another room, use cameras to avoid being near animals, etc.) Psychology has the same nightmare…

[deleted]
Post reply on HN