"Academic rank" is an awfully weird proxy for skill, though.
The Dunning-Kruger Effect Is Autocorrelation
151–160 of 203 posts
Re: The Dunning-Kruger Effect Is Autocorrelation
#152Because of the effect that is actually found (variance is higher the less achievement) it follows that people you encounter who wildly overestimate their ability are more likely to people who are poor performers (the same is true for the inverse, but they obviously don't stand out anecdotally to us). IMO that explains why Dunning Kruger seems intuitively correct even if the conclusion they drew isn't actually correct…
> people you encounter who wildly overestimate their ability are more likely to people who are poor performers
How is this helpful? You won't know whether someone is "overestimating" their ability until you learn both their estimated and actual performance, at which point you don't need to guess whether they're "likely" to have poor actual performance.
Re: The Dunning-Kruger Effect Is Autocorrelation
#153Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…
i immediately jump to think that replacing all the data with noise is a pretty good null hypothesis (at least for the analysis). is that not true?
You're changing the definition of the null hypothesis. What you are essentially saying is that there is no need to perform experiments and studies, I can arbitrarily grab data from anywhere and if it fails to support the hypothesis... then the hypothesis is wrong.
Re: The Dunning-Kruger Effect Is Autocorrelation
#154Earlier quoted context omitted.
> DK doesn't mean no correlation, it means inverse correlation. No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. Which is not what we would expect: we would expect that their predictions of their test scores would be strongly (or at least more st…
> their prediction of what their test scores will be, is uncorrelated I couldn't access the original 1999 full article because it's behind a paywall, but is this accurate? The abstract states that people "grossly overestimated their test performance and ability". If it were truly uncorrelated, wouldn't we expect there to be roughly equal number of people who "grossly underestimate" their performance? In any event, I…
having also not read the original paper recently, I'm in the camp that says the question is interesting and valid, but you've got to be careful about interpretation. how much the scores correlate (or autocorrelate) with self assessment IS an interesting question, and it's not apparent a priori what the answer will be, or whether the real effect we're interested in will differ for different levels of expertise.
but the use of percentile to percentile (or quartiles, but those are just grouped percentiles) to give the impression of a particular kind of effect (lower groups overestimating and higher groups underestimating) is a flaw i think. it's a common one, in my experience, when dealing with percentages.
if you think about it, percentages/percentiles have to be bounded at 0 or 100. for a dunning kruger effect not to appear, participants at both ends of the ability spectrum would have to be eerily accurate in their self-assessments. if they aren't, there's just more space on one side of the measurement scale for each group to make an error (if you score 1 in ability, there's ~99 percentiles available for you to make an overestimate and only 1 to be accurate, ditto for those with high ability. those in the middle of the ability group have equal chance on either side and so appear statistically more accurate even with a purely random distribution of guesses). so if there is any measure of central tendancy towards the middle percentile in the estimations of peoples abilities at all (and i would argue there is a priori reason to believe there would be, as the alternative would require those at both ends of the distribution to be getting increasingly accurate, which would be really weird), then practically any real world graph of percentile performance to percentile estimation will show a dunning kruger effect (with lower ends overestimating and higher ends underestimating). the article does a good job of showing this by plotting just random estimations and observing one appears.
Re: The Dunning-Kruger Effect Is Autocorrelation
#155Earlier quoted context omitted.
> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedast…
> DK doesn't mean no correlation, it means inverse correlation. No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. Which is not what we would expect: we would expect that their predictions of their test scores would be strongly (or at least more st…
>> DK doesn't mean no correlation, it means inverse correlation.
> No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores.
The problem is that in common Internet discussions, there is no fixed meaning of D-K. When things heat up, D-K becomes "his belief in his competent is an indication of his incompetence,ha" but in more calm debate is become "less competent people have a somewhat less accurate understanding of their competence".
In this form, D-K is a classic Motte and Baily device [1]. It's plausible, unsurprising and uninteresting that less competent people in a field self-access somewhat less accurately (the motte). It's completely unsupported but very "juicy" that someone's claim of competence indicates an incompetence (the bailey).
[1] To one more pieces of rhetorical which I'm mention also proves nothing but is illuminating - https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy
Re: The Dunning-Kruger Effect Is Autocorrelation
#156[deleted]
Re: The Dunning-Kruger Effect Is Autocorrelation
#157Very interesting article and statistical analysis, but I really don't see how it concludes that the DK effect is wrong based on the analysis. The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-eval…
You've been corrected about "what DK means" in the other comments, but this is not quite the point of the post. This is not about if DK (as expressed in English words) is true or not — in fact, author points out in the beginning that it's one of these "everybody knows it's like that" ideas (as it often is with social psychology).
The point is, that the original DK paper is bullshit. At least, this plot is. And people tend to miss it, until they start to carefully read the labels and think about the caveats. In fact, as presented here it looks like it shouldn't even be accepted as a valid study, this is outright deceptive, maliciously so. If there is assumed to be a correlation between x & y, how about we start by plotting x against y then? I know, it may be messy. It almost certainly will be. Because of that, I personally won't even be offended (but some people might) by you removing the outliers and producing the unnaturally clean version of the plot in the end to highlight the main idea. Then some statistical tests to make the results quantified. But here we see nothing, it really is just comparing x to x.
IMO, this is pretty much the invariant of most of the problems of academic research in the last God-knows-how-many decades (maybe always was, I don't know). Computer science papers without the code. Data science papers without the data. Yeah-yeah, I've heard hundreds of excuses why researchers do it like that. But it's pointless, such "research" shouldn't be accepted by anybody. Either you make your findings actually public by providing everything to replicate every single step of your study (which is supposed to be the point), or you just don't publish anything and keep the research proprietary (I mean, obviously it's never black and white, there always will be concerns about test-subject anonymity, etc. — but it's ridiculous to discuss that when the accepted standard even in "proper" sciences are 20 pages of dense text which might never even get to the point of the study, i.e., actually showing the data to any extent.)
Re: The Dunning-Kruger Effect Is Autocorrelation
#158Earlier quoted context omitted.
i immediately jump to think that replacing all the data with noise is a pretty good null hypothesis (at least for the analysis). is that not true?
You're changing the definition of the null hypothesis. What you are essentially saying is that there is no need to perform experiments and studies, I can arbitrarily grab data from anywhere and if it fails to support the hypothesis... then the hypothesis is wrong.
if your analysis produces an effect when fed uniform random data, then yes, no need to perform experiments or studies, because all your results are null. right?
so the hypothesis would be something like "the analysis shows a relationship between the two variables" and the null hypothesis would be something like "the analysis shows a relationship between two uniform random variables" and in this case that null is shown and accepted because no such relationship exists by definition. right? (unless it's like, "they have the same entropy", or something)
i'm very rusty with this stuff, so clarification would be much appreciated!
Re: The Dunning-Kruger Effect Is Autocorrelation
#159Earlier quoted context omitted.
> The abstract states that people "grossly overestimated their test performance and ability". I understand that the words D-K used were along those lines; but the actual data shown in their graphs says what I said. Just look at the lines on the graphs. The "actual" lines are 45 degree lines up and to the right--as they must be. But the "perceived" lines are horizontal, or roughly so. That means the two lines are unco…
Thanks for taking the time to help me understand the data. Looking at the graphs, isn't the implication that the distance between the two lines is the metric of interest? In other words, that the distance between real/perceived scores in Q1 & Q2 is "grossly" wider than the distance between Q3 & Q4, with the noted exception that only Q4 underestimates?
> isn't the implication that the distance between the two lines is the metric of interest?
That's more or less what D-K are saying, yes. That doesn't necessarily mean it's the best metric to use for understanding what the data is saying.
Also, "distance between the two lines" is misleading because the data is bucketed--on the x axis by quartiles, on the y axis by percentiles. In other words, the actual data is the circles/triangles/squares, not the lines. When put that way, "distance between the lines" looks a lot less like an actual metric and a lot more like an artificial one.
Re: The Dunning-Kruger Effect Is Autocorrelation
#160Earlier quoted context omitted.
Thanks for taking the time to help me understand the data. Looking at the graphs, isn't the implication that the distance between the two lines is the metric of interest? In other words, that the distance between real/perceived scores in Q1 & Q2 is "grossly" wider than the distance between Q3 & Q4, with the noted exception that only Q4 underestimates?
> isn't the implication that the distance between the two lines is the metric of interest? That's more or less what D-K are saying, yes. That doesn't necessarily mean it's the best metric to use for understanding what the data is saying. Also, "distance between the two lines" is misleading because the data is bucketed--on the x axis by quartiles, on the y axis by percentiles. In other words, the actual data is the ci…
Right, bad wording on my part. Thanks for correcting.