Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

andersource.dev

Post reply on HN
Print view View original

271–280 of 308 posts

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#271

Post by civilized »

bmacho wrote: post #198
civilized wrote: post #52

Earlier quoted context omitted.

> if an effect can be replicated with random noise, then it's not a psychological effect This isn't true either. Statistical dependence does not determine or uniquely identify causal interpretation or system structure. See Judea Pearl's works (e.g. The Book of Why) for more on this. People lacking the ability to self-assess is interesting psychologically. People can learn from experience in many other contexts. Peopl…

> People lacking the ability to self-assess is interesting psychologically. People can learn from experience in many other contexts. People can judge their relative position versus other people in many contexts. Why would they be so bad at this particular task? There could be a psychological underpinning. Is there an existing and relevant phenomenon about people lacking the ability to self-asses, that is true, proven…

I have no idea. I think it would be a fascinating experiment to take people who have never taken an IQ test, ask them how they think they'll do, then compare that against their actual performance.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#272

Post by samhw »

mike_hearn wrote: post #270
samhw wrote: post #265

Earlier quoted context omitted.

> It's about memorization of answers to pass law exams, not an actual test itself designed to verify logical reasoning ability. Why do they think this is a good source of questions for testing logic? I'm not sure what that has to do with anything? The paper doesn't claim to have anything to do with testing logic. It's about people's self-perception in relation to a task at which they are, or are not, competent. That…

> The paper doesn't claim to have anything to do with testing logic. The paper reports on the results of a 'logic' test administered to undergrads and uses this to define competence. It's a key part of their evidence their effect is real. > It's about people's self-perception in relation to a task at which they are, or are not, competent. That task could be juggling watermelons or strangling geese for all it matters.…

> The paper reports on the results of a 'logic' test administered to undergrads and uses this to define competence.

Right, but my point is that 'logic' is simply being used as an example of 'a task'. It's immaterial whether it's actually a good test of logic. As long as you agree that whatever it is is a good example of 'a task', then it's equally probative for the purpose of their argument.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#273

Post by whatshisface »

kortilla wrote: post #142
whatshisface wrote: post #101

Earlier quoted context omitted.

> it's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods). I don't think this is true. It is possible to put a lot of work into unsound statistics and to make a lot of "noise and fury" about how mathematical you are while failing some basic principle, but I don't think sound statistics can mislead. The replication crisis was caused by scientists not being rigorous and journ…

> but I don't think sound statistics can mislead. Of course they can, unless you magically exclude all statistics that made a bad assumption on independence. I plot all the daily high temperatures and the presence of the ice cream cart and it turns out the ice cream cart causes warmer highs! Solid statistics. Turns out the guy that has the ice cream cart has a weather app on his phone though and doesn’t come out on f…

That's not sound statistics. How can you call something sound and then point out a clear problem with it?

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#274

Post by topaz0 »

rafaeltorres wrote: post #8

Yeah, this makes sense to me. Imagine in the Dunning-Kruger chart the second plot (perceived ability) was a horizontal line at 70, which is not true but not far off from the real results. Now imagine I told you "did you know that, regardless of their actual score, everyone thought they got a 70?" That's a surprising fact.

I think the most egregious thing about the original presentation is that it leads you to believe that people with a given skill level all self-assessed similarly. If you plotted the scores and self-assessments of each individual you would see that it's not "everyone [in the first quartile] thought [they were about average]", it's that their self-assessments varied wildly, from low and accurate to high and inaccurate.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#275

Post by lelanthran »

SamoyedFurFluff wrote: post #251
lelanthran wrote: post #194

Earlier quoted context omitted.

>> What's new is that any research that might produce results counter to the what the PC-mob deems acceptable is attacked. > No this is not new. I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently. > You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superio…

> I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently. Are you genuinely serious or were you completely unaware of anti-communist government sanctioned blacklisting of academics suspected of being communist for political clout? Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun? Are you unaware of…

> Are you genuinely serious or were you completely unaware of anti-communist government sanctioned blacklisting of academics suspected of being communist for political clout? Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun? Are you unaware of our own Alan Turing, of the Turing award, literally castrated not for his research but because he was a known gay researcher? Are you unaware of why HBSUs exist(black scholars were segregated for being black, their research dismissed because of the race of the researcher)?

Every single one of those was NOT a mob.

The people in authority, using their authority to push their PoV, is very different to people with no authority forming a mob and demanding that the current authority silence other people from speaking their minds is a very different thing.

Whatever your view of the current authority is, it is infinitely better than mob-justice.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#276

Post by mrguyorama »

SamoyedFurFluff wrote: post #251
lelanthran wrote: post #194

Earlier quoted context omitted.

>> What's new is that any research that might produce results counter to the what the PC-mob deems acceptable is attacked. > No this is not new. I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently. > You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superio…

> I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently. Are you genuinely serious or were you completely unaware of anti-communist government sanctioned blacklisting of academics suspected of being communist for political clout? Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun? Are you unaware of…

>Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun?

Your point is good but this is a pet peeve of mine. Galileo was not punished by the church for saying the earth orbits the sun. Galileo was indicted and punished by the church because he was a local elite with several personal and political enemies within the church, and more directly, because he slighted the pope, his former friend, by taking a philosophical argument made by said pope, and putting it, paraphrased, into the mouth of a character in his book who was named "simplicio" and cast as a moron. That pope literally gave him permission to publish his claim that the earth orbited the sun, a claim which Galileo did not make based on science, but instead made because he felt the resulting (incorrect and based on outdated observations) mathematical model for the orbits of planets was more "elegant".

If the church truly wanted to punish him, they would not have sentenced him to literally stay at home in a beautiful villa and write books all day. His official charge was that lay people are not allowed to interpret the scripture, which he did a bit in his book. The church did not care if you made mathematical or scientific arguments about how the world worked. They only cared that you leave theology to the priests.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#277

Post by mike_hearn »

samhw wrote: post #272
mike_hearn wrote: post #270

Earlier quoted context omitted.

> The paper doesn't claim to have anything to do with testing logic. The paper reports on the results of a 'logic' test administered to undergrads and uses this to define competence. It's a key part of their evidence their effect is real. > It's about people's self-perception in relation to a task at which they are, or are not, competent. That task could be juggling watermelons or strangling geese for all it matters.…

> The paper reports on the results of a 'logic' test administered to undergrads and uses this to define competence. Right, but my point is that 'logic' is simply being used as an example of 'a task'. It's immaterial whether it's actually a good test of logic. As long as you agree that whatever it is is a good example of 'a task', then it's equally probative for the purpose of their argument.

The tasks aren't arbitrary. They're meant to be a proxy for some universal concept of competence. That's why DK is a well known effect, it claims to hold true for anything even though they can't test every possible task.

> we presented participants with tests that assessed their ability in a domain in which knowledge, wisdom, or savvy was crucial: humor (Study 1), logical reasoning (Studies 2-and 4), and English grammar (Study 3).

They picked humor because they think it reflects "competence in a domain that requires sophisticated knowledge and wisdom". They then realized the obvious objection - it's subjective - and decided to do the logical reasoning task to try and rebut those complaints (but then why do the first experiment at all?):

> We conducted Study 2 with three goals in mind. First, we wanted to replicate the results of Study 1 in a different domain, one focusing on intellectual rather than social abilities. We chose logical reasoning, a skill central to the academic careers of the participants we tested and a skill that is called on frequently ... it may have been the tendency to define humor idiosyncratically, and in ways favorable to one's tastes and sensibilities, that produced the miscalibration we observed-not the tendency of the incompetent to miss their own failings. By examining logical reasoning skills, we could circumvent this problem by presenting students with questions for which there is a definitive right answer.

So logical reasoning was chosen because:

1. It's objective.

2. It's an important skill.

3. It's a general "intellectual" skill.

That makes it very important if it's actually a good test of logical reasoning. If it was truly an arbitrary test like an egg-and-spoon-race or something, then there's no reason to believe the results would generalize to other areas of life and nobody would care.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#278

Post by danbruc »

samhw wrote: post #272
mike_hearn wrote: post #270

Earlier quoted context omitted.

> The paper doesn't claim to have anything to do with testing logic. The paper reports on the results of a 'logic' test administered to undergrads and uses this to define competence. It's a key part of their evidence their effect is real. > It's about people's self-perception in relation to a task at which they are, or are not, competent. That task could be juggling watermelons or strangling geese for all it matters.…

> The paper reports on the results of a 'logic' test administered to undergrads and uses this to define competence. Right, but my point is that 'logic' is simply being used as an example of 'a task'. It's immaterial whether it's actually a good test of logic. As long as you agree that whatever it is is a good example of 'a task', then it's equally probative for the purpose of their argument.

For completeness I would add that a good task must allow objectively rating the performance of participants with [much] room for debate. But given that, the whole setup is self-contained and task-independent. Let participants perform the task and establish their competence by rating their performance. Then let participants perform the meta-tasks of rating their performance in absolute and relative terms and finally check how task and meta-task performances are related.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#279

Post by danbruc »

mike_hearn wrote: post #277
samhw wrote: post #272

Earlier quoted context omitted.

> The paper reports on the results of a 'logic' test administered to undergrads and uses this to define competence. Right, but my point is that 'logic' is simply being used as an example of 'a task'. It's immaterial whether it's actually a good test of logic. As long as you agree that whatever it is is a good example of 'a task', then it's equally probative for the purpose of their argument.

The tasks aren't arbitrary. They're meant to be a proxy for some universal concept of competence. That's why DK is a well known effect, it claims to hold true for anything even though they can't test every possible task. > we presented participants with tests that assessed their ability in a domain in which knowledge, wisdom, or savvy was crucial: humor (Study 1), logical reasoning (Studies 2-and 4), and English gram…

The tasks aren't arbitrary. They're meant to be a proxy for some universal concept of competence.

This seems at least somewhat wrong to me - the competence is not universal but task specific. They compare how your competence to perform task X is related to your ability of assessing your performance of task X in absolute terms and relative to the other participants. They repeat this for different tasks and find that for all tested tasks the same pattern emerges - roughly, the better your performance, the better your ability to accurately assess your own performance and the performance of others.

So you can be competent doing task X and provide accurate assessments for task X performances while at the same time being incompetent doing task Y and being less accurate in assessing task Y performances. This essentially means that you can not be universally good at assessing performances of arbitrary tasks, you can only do this well for tasks for which you are yourself competent.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#280

Post by [deleted] »

lelanthran wrote: post #275
SamoyedFurFluff wrote: post #251

Earlier quoted context omitted.

> I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently. Are you genuinely serious or were you completely unaware of anti-communist government sanctioned blacklisting of academics suspected of being communist for political clout? Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun? Are you unaware of…

> Are you genuinely serious or were you completely unaware of anti-communist government sanctioned blacklisting of academics suspected of being communist for political clout? Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun? Are you unaware of our own Alan Turing, of the Turing award, literally castrated not for his research but because he was…

[deleted]
Post reply on HN