Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

andersource.dev

Post reply on HN
Print view View original

11–20 of 308 posts

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#11

Post by _0ffh »

blamestross wrote: post #10

Possibility 3 backed up by all the same data: The less you know, the more random your guess at your own knowledge is. The actual value is low and less than zero isn't an option, so this drags the average up consistently. The more you know, the more accurate your guess of your knowledge is. Especially as you hit the limits of the test, this noise can only drag the average down, but less dramatically than the other cas…

I also recently though about this problem and came to the same hypothesis, which fits the Dunning-Kruger data perfectly.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#12

Post by karpierz »

goosedragons wrote: post #4

"The second option conforms with the Research Methods 101 rule-of-thumb “always assume independence.” Until proven otherwise, we should assume people have no ability to self-assess their performance" It's not that at all. The assumption should be that everyone is equally good (or bad) at assessing their performance. Not that they have no ability but that the means between groups is the same vs. not the same. That the…

This confused me at first too. The issue is that "X" is your performance, and "Y" is your perceived performance.

Say that everyone is equally okay at assessing themselves, and get within 0.1 of their actual performance (rated from 0 to 1). Then X and Y are going to be very correlated, as X - 0.1 The only case where X and Y wouldn't correlate at all is if people have no ability to assess their performance (IE, Y isn't sampled around X, but is instead sampled from a fixed range).

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#13

Post by soVeryTired »

That original article was bogus and needlessly combative. I feel like the majority view in the HN comments saw it as such.

Most comments were splitting hairs on what _exactly_ the Dunning-Kruger effect was, plus some general nerd-sniping on how the original article was off base.

IMO it was something that fell flat on its own rather than something that needed a lengthy refutation, but I can understand that sometimes these things get under your skin.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#14

Post by majormajor »

blamestross wrote: post #10

Possibility 3 backed up by all the same data: The less you know, the more random your guess at your own knowledge is. The actual value is low and less than zero isn't an option, so this drags the average up consistently. The more you know, the more accurate your guess of your knowledge is. Especially as you hit the limits of the test, this noise can only drag the average down, but less dramatically than the other cas…

Assuming a world where all the participants understand normal distributions, would this be addressed by asking people to rate how they did in terms of "standard deviations compared to the average" or such?

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#15

Post by h0l0cube »

skybrian wrote: post #5
galaxyLogic wrote: post #3

I think Dunning Krueger makes intuitive sense. When you become skilled in your field you learn from other people in your field, and your assessment of yourself is based on your relation to the skills of those other people. But if you know very little about something, you have no reference point to evaluate yourself against. When you learn something you also learn what are some of the mistakes you can make. You evalua…

I think that's assuming more ignorance than even an unskilled person has. If you had never listened to a professional play piano before then you'd have no idea what level of performance is possible. Similarly, if you had never seen skilled skaters perform on TV. But we have done these things, so it's obvious that they're doing something that's very difficult. Maybe you don't fully appreciate the skill, though. You wo…

The OP article mentions in their rumination that there's some difficulty in generalising DK:

"And maybe there’s no contradiction - there’s always room for nuance, for finding out where the Dunning-Kruger effect is relevant and where it’s not. That can be done with more studies, but only if the authors manage to agree on assumptions and basic statistical practice."

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#17

Post by blamestross »

majormajor wrote: post #14
blamestross wrote: post #10

Possibility 3 backed up by all the same data: The less you know, the more random your guess at your own knowledge is. The actual value is low and less than zero isn't an option, so this drags the average up consistently. The more you know, the more accurate your guess of your knowledge is. Especially as you hit the limits of the test, this noise can only drag the average down, but less dramatically than the other cas…

Assuming a world where all the participants understand normal distributions, would this be addressed by asking people to rate how they did in terms of "standard deviations compared to the average" or such?

That still wouldn't be useful. The root problem is that a scoring system that isn't infinite both ways (or the reasonably achievable scores are significantly farther from the bounds than the variance of guesses) will end up with a "clipping" at the edges of the model.

There are ways to fix this:

- Throw out the extreme high and low ends of the data bc the model breaks down there. (Which results in a very boring result)

- Have people guess their score and a rough level of confidence along side it (just a 0-5 sort of thing) and see what happens.

Note that I actually do think from my own experience that the effect is real, but the arguments presented fail to prove it statistically bc the model breaks down at the extreme where the effect is detected.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#18

Post by bombcar »

kurthr wrote: post #16
lamontcg wrote: post #6

I'm not a practicing statistician, so I'm uncertain how to weigh the two arguments here.

I see what you did and I am completely uninformed in my certainty that Dunning-Kruger is wrong!

I fall back to Sturgeon’s Law and assume 90% of everything, including me, is shit.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#19

Post by waynesonfire »

galaxyLogic wrote: post #3

I think Dunning Krueger makes intuitive sense. When you become skilled in your field you learn from other people in your field, and your assessment of yourself is based on your relation to the skills of those other people. But if you know very little about something, you have no reference point to evaluate yourself against. When you learn something you also learn what are some of the mistakes you can make. You evalua…

Your post reminded me of one of my favorite Adam Savage videos where he touches upon this idea you're exploring. I encourage folks to see it, he articulates it so well.

I linked to the start of the video where he begins to build the idea. TLDR is he mentions Monet painting Impression Sunrise and how it was something that people have never seen before and it took a bit of time for it to blow people away--they needed to develop "new eyes" to see the genius. Adam then dives into this idea of "new eyes". I'm sure many of us have experienced this in our life and it was so nice to hear Adam unpack it.

https://www.youtube.com/watch?v=qE7dYhpI_bI&t=122s

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#20

Post by haberman »

On a pure human level, a large portion of DK discourse seems to be a fight over which people are the "Unskilled and Unaware." Or more bluntly, who gets to call who stupid.

The author says as much in this article:

> Why so angry? [...] [Frankly], for the last few years, the world seems to be accelerating the rate at which it’s going crazy, and it feels to me a lot of that is related to people’s distrust in science (and statistics in particular). Something about the way the author conveniently swapped “purely random” with “null hypothesis” (when it’s inappropriate!) and happily went on to call the authors “unskilled and unaware of it”, and about the ease with which people jumped on to the “lies, damned lies, statistics” wagon but were very stubborn about getting off, got to me. Deeply. I couldn’t let this go.

It's true, the previous article (https://economicsfromthetopdown.com/2022/04/08/the-dunning-k...) was pretty harsh on the authors of the original paper:

> In their seminal paper, Dunning and Kruger are the ones broadcasting their (statistical) incompetence by conflating autocorrelation for a psychological effect. In this light, the paper’s title may still be appropriate. It’s just that it was the authors (not the test subjects) who were ‘unskilled and unaware of it’.

But on some level, the original paper sounds just as condescending and dismissive. It presents a scholarly and statistical framework for looking down on "the incompetent" (a phrase used four times in the original paper). In practice, most of the times I see the DK effect cited, it functions as a highbrow and socially acceptable way of calling someone else stupid, in not so many words.

Cards on the table, I've never liked DK discourse for this reason. It's always easy to imagine others as the "Unskilled and Unaware", and for this reason bringing DK into any discussion rarely generates much insight.

Post reply on HN