Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

Post reply on HN
Print view View original

131–140 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#131

Post by m3047 »

askasp wrote: post #72

If we assume random data then the people at the lower end will over-estimate their own performance the same amount that people on the higher end will under-estimate theirs. However, if the under-performers consistently over-estimate more than the over-performers under-estimate there is still some merit to the effect, isn't there? That is, the interesting number is the difference between integral of y-x on lower half…

Yeah, I think so and we're probably in the minority here. There are a couple of other comments referring to regression to the mean and that the article takes a literalist view which is perhaps unwarranted. You win the followup comment. ;-)

I confess that I've never paid that much attention to the classic D-K graph, and that taking a close look at it, it is most assuredly crap. Now I want to know what the plots of the actual scores for those quartiles look like rather than %ile, or after-the-fact ranking. Yeah, it sure looks like people mostly figure they're in the 55-75 %ile ranking, if that's what that actually is, and that where in that spread they think they are correlates with their actual ranking.

Let's go down a Bayesian rabbit hole. Let's assume, as does the article, that people's self estimations are completely random rubbish: the worst people have nowhere to go but up, the best nowhere but down. Yup, completely agree.

Now let me ask a question: is self-estimation of any use in determining actual ability? The answer in this case is no: knowing one does not inform our ability to know the other in a Bayesian sense, they are not correlated.

D-K sounds valuable as a cautionary tale concerning excessive exuberance and a tendency not to learn well from experience, but aside from child-proof caps and Mr. Yuk stickers where we really want to apply the lesson is at the high-performing end of the scale and here we get into trouble immediately.

It is tempting to say "high-performers have nowhere to go but down" as though maybe we should reject those self-reporting the best performance. The classic chart hints at high performers underestimating their true performance, but it's a crappy chart; maybe they want it to be true.

But in the specific case where there is utterly no correlation and true performance is as evenly distributed as self-assessment, if we chop off the "top X self-reporting" we will chop off just as many poor performers as high performers. Yes, I hear you, and I agree, random is an edge case; I just don't believe that affects its prevalence.

Maybe it is true; alright dust off those priors and have at it.

Re: The Dunning-Kruger Effect Is Autocorrelation

#132

Post by cestith »

IceDane wrote: post #21
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis. I'll admit that part of it also comes from personal experience, at work and elsewhere. I've met some catastrophically incompetent people were completely oblivious to their own incompetence, and this has very often felt like that the more incompetent they were, the more likely they were to be try to do stuff that…

I think part of the issue here is you're talking about people estimating future accomplishment where they have no skill, and all variations of these experiments are people assessing how they performed on a test they just took. There's very likely a difference between someone doing open-ended talking wildly in a field where they don't even know what's possible and an introspective assessment of themselves on a concrete task they just completed.

Re: The Dunning-Kruger Effect Is Autocorrelation

#133

Post by aidenn0 »

PaulKeeble wrote: post #13

Modern Psychology is having a lot of these sorts of results over the last decade, none of their methods are holding up under proper scrutiny. They are struggling to reproduce findings but more critically even the reproduced ones are turning out to be statistical and mathematical errors like shown here. Some of the findings have also done severe harm to patients over the decades as well, I can't help but think we need…

So many psychology experiments are so fantastically underpowered, that if the effect they are attempting to measure were real, odds are that it would have to be of such a large magnitude that it would completely overturn all of our beliefs on how humans function. And they can publish with a P value of 0.049.

So, implicit in the standards for publishing new research in psychology is "We think there is much greater than a 5% chance that our entire field is wrong" which is not a great place to start from.

Re: The Dunning-Kruger Effect Is Autocorrelation

#134

Post by uoaei »

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

> the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance

That's not really the intended interpretation of "null" in "null hypothesis". "Null" does not mean "contrary to the effect you're testing". "Null" means "do not assume dependencies anywhere" and so your description is backwards.

Re: The Dunning-Kruger Effect Is Autocorrelation

#135

Post by playpause »

I’ve always felt the DK effect is cynical pseudoscience for midwit egos. It’s a sophistic statement of the obvious dressed up as an insight. But worse, it serves to obvert something interesting and beautiful about humans - that even very intellectually challenged people sometimes can, over time, develop behaviours and strategies that nobody else would have thought of, and form a kind of background awareness of their shortcomings even if they aren’t equipped to verbalise them, allowing them to manage their differences and rise to challenges and social responsibilities that were assumed to be beyond their potential. Forrest Gump springs to mind as an albeit fictional example of the phenomenon I’m talking about. I think this is a far more interesting area than the vapid tautology known as the DK effect.

Re: The Dunning-Kruger Effect Is Autocorrelation

#136

Post by jollybean »

playpause wrote: post #135

I’ve always felt the DK effect is cynical pseudoscience for midwit egos. It’s a sophistic statement of the obvious dressed up as an insight. But worse, it serves to obvert something interesting and beautiful about humans - that even very intellectually challenged people sometimes can, over time, develop behaviours and strategies that nobody else would have thought of, and form a kind of background awareness of their…

I think there's an easier explanation for the effect, and that is people are just not very good at judging their skill level, and due to reversion to the mean, low-performers probably overestimate and high-performers underestimate.

And also, I think there is actually a tiny bit of DK going on.

And then, as you say, it gets amplified by the pseudo-literati.

Re: The Dunning-Kruger Effect Is Autocorrelation

#138

Post by bumby »

pdonis wrote: post #129
leto_ii wrote: post #60

Earlier quoted context omitted.

> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedast…

> DK doesn't mean no correlation, it means inverse correlation. No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. Which is not what we would expect: we would expect that their predictions of their test scores would be strongly (or at least more st…

>their prediction of what their test scores will be, is uncorrelated

I couldn't access the original 1999 full article because it's behind a paywall, but is this accurate?

The abstract states that people "grossly overestimated their test performance and ability". If it were truly uncorrelated, wouldn't we expect there to be roughly equal number of people who "grossly underestimate" their performance? In any event, I wouldn't expect a weak correlation to be described as "grossly [anything]".

Or maybe this is just the D-K effect in myself trying to understand the abstract :-)

Re: The Dunning-Kruger Effect Is Autocorrelation

#139

Post by pklausler »

pdonis wrote: post #129
leto_ii wrote: post #60

Earlier quoted context omitted.

> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedast…

> DK doesn't mean no correlation, it means inverse correlation. No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. Which is not what we would expect: we would expect that their predictions of their test scores would be strongly (or at least more st…

I think of the DK effect in kind of a recursive way: gauging one's skill level is itself a skill. And if you are bad at that skill, you can't know it.

Re: The Dunning-Kruger Effect Is Autocorrelation

#140

Post by a-dub »

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

i immediately jump to think that replacing all the data with noise is a pretty good null hypothesis (at least for the analysis). is that not true?
Post reply on HN