Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dunning-Kruger Effect Is Autocorrelation

economicsfromthetopdown.com

Post reply on HN
Print view View original

71–80 of 203 posts

Re: The Dunning-Kruger Effect Is Autocorrelation

#71

Post by BlueTemplar »

Phileosopher wrote: post #29
PaulKeeble wrote: post #13

Modern Psychology is having a lot of these sorts of results over the last decade, none of their methods are holding up under proper scrutiny. They are struggling to reproduce findings but more critically even the reproduced ones are turning out to be statistical and mathematical errors like shown here. Some of the findings have also done severe harm to patients over the decades as well, I can't help but think we need…

I'm convinced it's associated with the methodology of how psychology has approached matters. In the world of biology, you're observing the world around you. Same for physics, chemistry, et al. This means that you can set up proper controls to obscure your own presence from any potential results (e.g., isolate everything in another room, use cameras to avoid being near animals, etc.) Psychology has the same nightmare…

Well, all sciences have this issue : see Kuhn, Feyerabend...

https://samzdat.com/2018/05/19/science-under-high-modernism/

It would be kind of ironic if psychologists were more susceptible to the "Pop-Baconian" simplification of science ?

And damning if what I've heard about psychology going through multiple paradigms during the 20th century alone is true.

But then, indeed, also understandable, due to the "softness" and "paradigmlessness" of the subject matter, as Kuhn had pointedout the later back then ?

It's still sad how we now have detailed theories and histories of science, but in practice scientists show no interest in trying to learn from them, nor the mistakes of their predecessors?

But then maybe that would have too high of a cost. (Consider all those successful projects where the founders later say : "we didn't knew what we were getting into / that this was considered impossible".)

Bonus : Wiseman & Schlitz’s attempts to do an adversarial super-controlled parapsychological experiment : ( IV. )

https://slatestarcodex.com/2014/04/28/the-control-group-is-o...

Re: The Dunning-Kruger Effect Is Autocorrelation

#72

Post by askasp »

If we assume random data then the people at the lower end will over-estimate their own performance the same amount that people on the higher end will under-estimate theirs.

However, if the under-performers consistently over-estimate more than the over-performers under-estimate there is still some merit to the effect, isn't there?

That is, the interesting number is the difference between integral of y-x on lower half vs the integral of y-x on the upper half. Does that make sense to anyone else?

Re: The Dunning-Kruger Effect Is Autocorrelation

#73

Post by once_inc »

larwent wrote: post #48

Unless I missed something, this article doesn't explain WHY random data can result in a Dunning-Kruger effect. The relationship between the "actual" and "perceived" score is a product of bounding the scores to 0-100. When you generate a random "actual" score near the top, the random "perceived" score has a higher chance of being below the "actual" the numerical below is larger than the one above, and vice-versa. E.g.…

A person with an actual score of 80% will probably have enough confidence in his or her abilities due to experience that they will tend not to rate themselves low. Imagine being a graduate student asked how high (s)he would rank. They would not rank themselves as low as they might have when they were sophomore students. They would probably rank within 20% of their actual score, which is what the final graph in the article shows; professors have enough experience to be able to self-assess themselves better than less experienced subjects can.

Re: The Dunning-Kruger Effect Is Autocorrelation

#75

Post by newbamboo »

Autocorrelation is much more interesting, and much more important topic than dk, which mostly seems to be popular concept because it supports biases and other fallacious, ego driven thinking. Autocorrelation is an under-appreciated problem, particularly in the social sciences and Econ. So it’s nice to use dk to catch the attention of the masses to spread the word about autocorrelation.

Re: The Dunning-Kruger Effect Is Autocorrelation

#76

Post by stevage »

larwent wrote: post #52
jakear wrote: post #37

Article seems to be saying “DK doesn’t exist because it always exists”. Which is… absurd? The point of DK is that when you don’t know shit, any non-degenerate self assessment will result in overestimating your ability. In short, “there are more natural numbers above smaller natural numbers than bigger ones”. This doesn’t have to do with psychology, and it’s expected that it appears when evaluating random data. That’s…

Seems pretty simple. When we create upper and lower boundaries to some score, people with lower scores have more space to overestimate and those with higher scores more space to underestimate, causing the perceived score to trend towards the mean. I think there's both a component of numbers and psychology here. If the dispersion in perceived score caused by inaccuracy is wide enough to touch the bounds, it will force…

It would be possible to rule out that effect in an experiment.

Re: The Dunning-Kruger Effect Is Autocorrelation

#77

Post by tacitusarc »

leto_ii wrote: post #60
andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedast…

> > a world in which people are very bad at estimating their own skill, therefore, statistically, people with lower skills tend to overestimate their skills, and experts tend to underestimate it. > Be careful here, the conclusion you drew doesn't actually follow.

How does that not follow? It's just regression to the mean.

Re: The Dunning-Kruger Effect Is Autocorrelation

#78

Post by highfrequency »

The author is onto something that Dunning-Kruger is suspicious, but the argument is wrong. The "statistical noise" plot actually demonstrates a very noteworthy conclusion: that Usain Bolt estimates his own 100m ability as the same as a random child's. This would be a great demonstration of the Dunning-Kruger effect, not a counterargument.

On the other hand, regression to the mean rather than autocorrelation does explain how you could get a spurious Dunning-Kruger effect. Say that 100 people all have some true skill level, and all undergo an assessment. Each person's score will be equal to their true skill level plus some random noise based on how they were performing that day or how the assessment's questions matched their knowledge. There will be a statistical effect where the people who did the worst on the test tend to be people with the most negative idiosyncratic noise term. Even if they have perfect self-knowledge about their true skill, they will tend to overestimate their score on this specific assessment.

Regression to the mean has broad relevance, and explains things like why we tend to be disappointed by the sequel to a great novel.

Re: The Dunning-Kruger Effect Is Autocorrelation

#79

Post by andersource »

omnicognate wrote: post #61
andersource wrote: post #41

Earlier quoted context omitted.

> The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias" in estimating their performance" OK, > remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill The way I see it these aren't very different - conditioning on…

> What is the distinction between being bad at the skill and bad at estimation, and being bad at the skill and systematic overestimation? The difference between bias and variance. > conditioning on low skill and randomly sampling will tend to give way more overestimates than underestimates That's the hypothesis that's being tested.

> The difference between bias and variance.

But when you're bad at the skill and can't underestimate, they look the same.

> That's the hypothesis that's being tested

And evidence from DK supports it.

Re: The Dunning-Kruger Effect Is Autocorrelation

#80

Post by ohwellhere »

andersource wrote: post #7

Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…

It depends on what one means by the "Dunning-Kruger effect."

I had the same impulse that the analysis did not disprove DK, but after sitting with it for overlong I agree with the analysis.

I think there are two competing DK effect definitions that are being conflated, one descriptive and one explanatory:

1. DK shows that less skilled people overestimate their ability, and highly skilled people underestimate it

2. DK shows that people's estimation of their ability is causally determined by their actual ability

I believe you are claiming, correctly, that the article does not disprove the first definition that explains the observation, but I think the article is trying to disprove the second definition that explains why it occurs.

In other words: Yes, there is an observable Dunning-Kruger effect in the sense that we're bad at self evaluation. Is that effect attributable to one's actual level of competence? The evidence for that appears to be a statistical artifact, and further experiments seem to disprove that conjecture.

I'm not a statistician or a psychologist.

Post reply on HN