The Dunning-Kruger Effect Is Autocorrelation • Hacker News Board

Re: The Dunning-Kruger Effect Is Autocorrelation

#61

Post by omnicognate » Fri, Apr 15, 2022, 1:50 PM UTC

andersource wrote: post #41

omnicognate wrote: post #34

Earlier quoted context omitted.

The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias " in estimating their performance" (remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill). The randomly generated data used is one example of that, albeit an…

> The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias" in estimating their performance" OK, > remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill The way I see it these aren't very different - conditioning on…

> What is the distinction between being bad at the skill and bad at estimation, and being bad at the skill and systematic overestimation?

The difference between bias and variance.

> conditioning on low skill and randomly sampling will tend to give way more overestimates than underestimates

That's the hypothesis that's being tested.

Re: The Dunning-Kruger Effect Is Autocorrelation

#62

Post by civilized » Fri, Apr 15, 2022, 1:53 PM UTC

andersource wrote: post #53

alecbz wrote: post #49

Earlier quoted context omitted.

The author’s confidence is itself an indication that they’re more likely to be wrong. Kidding. Well, half-kidding, I did kind of find the tone a bit biting and dismissive, especially towards one of the commenters that were pointing out exactly what you did. It’s an interesting question to ask whether ask whether the uniformly random data “really” exhibits DK or not, and whether that’s interesting. A world where peopl…

> But I think the author’s right that obviously nothing psychological is happening here. There’s the psychological effect of no one being able to assess themselves, but the fact that unskilled people overestimate themselves in this world has nothing to do with the fact that they are unskilled. If the results from DK were similar to the random data results, I'd agree. But the DK results do show some correlation betwee…

My guess is that the spread in self-evaluation is largest at low skill level and decreases as skill level increases. This would produce results more similar to what D-K actually observed, and is much more plausible than postulating that highly skilled people have no more idea of their skill level than low skilled people.

Re: The Dunning-Kruger Effect Is Autocorrelation

#63

Post by ewzimm » Fri, Apr 15, 2022, 1:53 PM UTC

larwent wrote: post #52

jakear wrote: post #37

Article seems to be saying “DK doesn’t exist because it always exists”. Which is… absurd? The point of DK is that when you don’t know shit, any non-degenerate self assessment will result in overestimating your ability. In short, “there are more natural numbers above smaller natural numbers than bigger ones”. This doesn’t have to do with psychology, and it’s expected that it appears when evaluating random data. That’s…

Seems pretty simple. When we create upper and lower boundaries to some score, people with lower scores have more space to overestimate and those with higher scores more space to underestimate, causing the perceived score to trend towards the mean. I think there's both a component of numbers and psychology here. If the dispersion in perceived score caused by inaccuracy is wide enough to touch the bounds, it will force…

This seems pretty simple to correct, so I'm skeptical that nobody has done so yet in these experiments. If true, it's an equally interesting oversight as the Monty Hall problem. The basic premise is that the structure of an experiment will naturally nudge randomness in a particular direction, and we need to adjust for that in the analysis. Everyone who does this type of work should know this.

In a simplified experiment where we give people a 3 question quiz, those who got 2 questions right have one overestimation option, 3, and two underestimation options, 0 and 1. So it's very easy to adjust for autocorrelation by checking if a large group of 2-scorers underestimate more than twice as often as they overestimate. Then we see how their tendencies compare against 1-scorers and how they deviate from naturally overestimating more than twice as often as underestimating.

I haven't reviewed these types of papers, but if nobody made even that basic adjustment in their analysis, how many others have been missed in experiments like this?

Re: The Dunning-Kruger Effect Is Autocorrelation

#64

Post by nbernard » Fri, Apr 15, 2022, 2:09 PM UTC

_dain_ wrote: post #27

nbernard wrote: post #19

Earlier quoted context omitted.

> The article argues that the graphs supposedly demonstrating this fact, can also be generated from a model that does not have this difference, i.e. where bias_lowskill == bias_highskill. But as I understand it, it doesn't: In the graph generated using random data, the lines intersect in the middle (bias_lowskill == bias_highskill), whereas in DK's paper they intersect in the upper right (so bias_lowskill != bias_hig…

Okay well in that case the article is deficient (I skimmed it, assuming it was making the same case I already knew about[1] -- my bad). You can actually reproduce the DK graph without supposing that estimation bias depends on skill. There is an interactive visualization here[2] that does precisely that (click on the "line plot w/ centiles" tab). You can adjust the parameters to see how it behaves. [1] https://news.yc…

Thanks for the link!

I see what you mean, but is using the "general overestimation" slider actually unbiased? I don't really understand what it is supposed to represent. Looking at the code, it seems to be only an additive term that shifts the plot upwards, but that does not explain anything (actually, in the code, the variable is called "up_bias", and could correspond to an actual DK effect...).

Re: The Dunning-Kruger Effect Is Autocorrelation

#65

Post by ImaCake » Fri, Apr 15, 2022, 2:11 PM UTC

larwent wrote: post #48

Unless I missed something, this article doesn't explain WHY random data can result in a Dunning-Kruger effect. The relationship between the "actual" and "perceived" score is a product of bounding the scores to 0-100. When you generate a random "actual" score near the top, the random "perceived" score has a higher chance of being below the "actual" the numerical below is larger than the one above, and vice-versa. E.g.…

As explained in the article, the reason is autocorrelation. Basically the y axis is correlated to the x axis because the y axis is actually x + random noise. The dunning kruger graph is then a transformation of that data - still subject to autocorrelation.

Re: The Dunning-Kruger Effect Is Autocorrelation

#66

Post by andersource » Fri, Apr 15, 2022, 2:11 PM UTC

leto_ii wrote: post #60

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedast…

> DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedasticity).

Not sure I follow, inverse correlation between what? The analysis at the bottom (assuming you mean fig. 11) is too dense to show if there's a correlation or not (between skill and self-assessment bias), and looking at the relevant figure from the paper itself gives me the impression that there is a correlation.

>> a world in which people are very bad at estimating their own skill, therefore, statistically, people with lower skills tend to overestimate their skills, and experts tend to underestimate it.

> Be careful here, the conclusion you drew doesn't actually follow.

Can you elaborate? If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated - this is exactly what the first part of the article is about, although from my perspective it's the author who's drawing the wrong conclusion.

>> y - x ~ x, this is called the residual plot

> You're giving x and y meaning that they don't have. In the article these are uncorrelated random variables - the plot of y-x ~ x will always look that way. That's however not the case if you're plotting y_hat - y ~ y_hat for a y_hat taken out of a model. That won't be a random variable in your setup.

Not following again. Other than calling x y_hat, and having y_hat be your own estimate vs. x be the subjects' estimate, what is the distinction? What do you mean by "the plot of y-x ~ x will always look that way" - what way? The shape of the plot will necessarily depend on the relationship between x and y.

Re: The Dunning-Kruger Effect Is Autocorrelation

#67

Post by bitshiftfaced » Fri, Apr 15, 2022, 2:12 PM UTC

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

Check out Nuhfer et al 2016, who had a different explanation for why Dunning Kruger wasn't true.

Dunning Kruger effect: lower performers overestimate their ability, and higher performers underestimate their ability.

How did they find that? They asked participants to take a test and then had them do a self-assessment. Both were standardized from 0-100. They rated a participant's self-assessment accuracy by "self-assessment minus test score."

What's wrong with that method? You can't arrogantly self-assess as though you got a 130, and you can't humbly say that you got -50. Because of the standardization, you're bound by 0 and 100. This method makes it almost impossible for higher performers to overestimate their ability and for lower performers to underestimate.

What they actually found was that higher performers tend to be better at self-assessment. Lower performers are less accurate, but in both directions (not just overconfident).

Re: The Dunning-Kruger Effect Is Autocorrelation

#68

Post by kizer » Fri, Apr 15, 2022, 2:14 PM UTC

ithkuil wrote: post #56

In other words people are quite bad at estimating their skill level. Some people will overestimate, while some other people will underestimate and on average there will be a relatively constant estimated skill level that doesn't change all that much based on the actual abilities. Given that fact, it logically follows that people who score low ability tests will more often than not have overestimated their ability (an…

That’s what I was thinking; if the average is about constant all you’ve shown is that everyone is bad at self-assessment (another issue - not fully qualifying a distribution by just using the average loses information).

But a comment above quoting the more recent paper presents a contradictory conclusion: that humans can self-assess with some accuracy. So now I’m confused again.

Re: The Dunning-Kruger Effect Is Autocorrelation

#69

Post by erikmolin » Fri, Apr 15, 2022, 2:16 PM UTC

jakear wrote: post #37

Article seems to be saying “DK doesn’t exist because it always exists”. Which is… absurd? The point of DK is that when you don’t know shit, any non-degenerate self assessment will result in overestimating your ability. In short, “there are more natural numbers above smaller natural numbers than bigger ones”. This doesn’t have to do with psychology, and it’s expected that it appears when evaluating random data. That’s…

I share your sentiment that DK being a statistical phenomenon actually makes the effect more interesting. I also, however, think that you are being too semantically liberal when you insist that this proves the effect exists. The DK effect is a _psychological_ effect, with a whole lot of psychological theories around it's causes. If the cause is statistical, the psychological effect can no longer be said to exist.

The stasticial phenomenon exists, surely - but I think it will be very confusing for everyone to re-use the same name.

Re: The Dunning-Kruger Effect Is Autocorrelation

#70

Post by kizer » Fri, Apr 15, 2022, 2:19 PM UTC

I’m not a scientist, but wouldn’t it make sense for standard practice to be to assume at first that there’s a shared variable (that you have introduced) and to look for it until you’re certain the things you’re plotting are independent? Of course they may not be in the end as that’s the “goal”, but the shared variable if there is indeed causation in that case will be what you’re looking for, not one of the variables you “know”.