Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

andersource.dev

Post reply on HN
Print view View original

221–230 of 308 posts

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#221

Post by andersource »

john_pryan wrote: post #212

For anyone who is interested in playing around with these charts, the various assumptions that under pin them etc. I've thrown together a colab notebook as a starting point. Observation: if you rank via true "skill" and assume for a particular instance the predicted performance and observed performance are independent but both have the true skill as their mean you dont observe the effect. CC of 0.00332755. If you ran…

Thanks John! Very interesting.

What your simulation includes and the original article didn't (and I didn't touch at all in my article) is the statistical reliability of the tests they administered. Where you got a CC of -0.38 you used equal reliability (/ unreliability) of the skill tests and self-assessments. You can see that as you increase the test reliability, the CC shrinks and the effect disappears.

I have no idea what's the actual reliability of the DK tests, they do seem to consider that but maybe not thoroughly enough. In my view it's very fair to criticize DK from that angle. But that would require looking at the actual tests and their data.

My point being, that any purely random analysis is based on assumptions that can easily be tweaked to show the same effect, the opposite effect, or no effect at all.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#222

Post by emsign »

Sounds like the premise is flawed. He's assuming kids are good at getting another 10 minutes before bedtime. All of them? What about those who fail? Those that don't even try?

The issue is not the way our brains generalize, but that you are using just one brain, one life's experience.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#223

Post by NumberCruncher »

Traster wrote: post #177
NumberCruncher wrote: post #172

> We don’t need statistics to learn about the world. A sentence, written by the author on, commented by me on and read by the HN community on devices, which exist only thanks to 80-90 years of rigorous, statistics based QA in engineering, especially in mechanical/hardware engineering. Anyhow, after spending years on a team filled with social science PHDs, I would not waste my time on reading papers about statistical…

I don't think your interpreting the sentence correctly, I think it's more correct to read it as: > We don't [only] need [the formal discipline of mathematics known as] statistics to learn about the world. Sure, there are things you can only functionally ascertain through statistical analysis. But not everything in the world needs rigorous statistics.

> I don't think your interpreting the sentence correctly

And I think you are injecting the words "only", "there are" and "everything" here and there just to change the meaning of the sentences I quoted and I have written...

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#224

Post by john_pryan »

andersource wrote: post #221
john_pryan wrote: post #212

For anyone who is interested in playing around with these charts, the various assumptions that under pin them etc. I've thrown together a colab notebook as a starting point. Observation: if you rank via true "skill" and assume for a particular instance the predicted performance and observed performance are independent but both have the true skill as their mean you dont observe the effect. CC of 0.00332755. If you ran…

Thanks John! Very interesting. What your simulation includes and the original article didn't (and I didn't touch at all in my article) is the statistical reliability of the tests they administered. Where you got a CC of -0.38 you used equal reliability (/ unreliability) of the skill tests and self-assessments. You can see that as you increase the test reliability, the CC shrinks and the effect disappears. I have no i…

That's a nice spot about the decreasing CC as we increase accuracy!

My hypothesis would be that some of the DK effect in the original paper may be down to an effect like this (as suggested in the original article) but that asserting it is completely incorrect because of it is premature. We'd need access to more data to verify that the level of reliability was sufficiently acceptable.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#225

Post by andersource »

john_pryan wrote: post #224
andersource wrote: post #221

Earlier quoted context omitted.

Thanks John! Very interesting. What your simulation includes and the original article didn't (and I didn't touch at all in my article) is the statistical reliability of the tests they administered. Where you got a CC of -0.38 you used equal reliability (/ unreliability) of the skill tests and self-assessments. You can see that as you increase the test reliability, the CC shrinks and the effect disappears. I have no i…

That's a nice spot about the decreasing CC as we increase accuracy! My hypothesis would be that some of the DK effect in the original paper may be down to an effect like this (as suggested in the original article) but that asserting it is completely incorrect because of it is premature. We'd need access to more data to verify that the level of reliability was sufficiently acceptable.

Right. Just to be clear, "an effect like this" is (comparatively) unreliable tests, not some elusive statistical phenomena as implied by the original article. I'd have no issue if the author had called the article "the DK effect is due to poor skill tests", spent 5 minutes showing that the DK results are consistent not only with their claims but also with unreliable tests (like you did), then went on to show data that indicates that the tests indeed are not reliable enough to draw the conclusions that DK did. Instead the author spends a lot of time digging under the wrong tree and no time at all saying anything about the reliability of the tests.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#226

Post by irrational »

> If you tell me you didn’t have a single serious thought of self-assessing today, even semi-conscious, I simply won’t believe you.

I stopped reading at this point. Someone that is so certain that they say “I simply won’t believe you.” is too self-assured to be worth paying much attention to.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#227

Post by Terretta »

Tenoke wrote: post #174
formerly_proven wrote: post #155

Earlier quoted context omitted.

> My understanding is that the hypothesis is "Those who are incompetent overestimate themselves, and experts underestimate themselves". The DK hypothesis is "double burden of the incompetent": "Because incompetent people are incompetent, they fail to comprehend their incompetence and therefore overestimate their abilities more than expertes underestimate theirs" Arguably the hypothesis that matches the data from the…

>Arguably the hypothesis that matches the data from the DK paper best is: "Everyone thinks they're average regardless of skill level" No, if you look at the graph[0] everyone thinks they are above average (over 50). The worst think they are a little above average and everyone else thinks they are better and better but increasing by less than the real difference. At any rate, the issue seems to be with how people imag…

Read through comments to see if this point was made, thank you.

This is so well understood there’s even a joke about it re: drivers that everyone “gets” even while knowing it doesn’t apply to them.

But to your point, look at the slope in the second chart here, “Histogram of subjective ranks”:

https://gottwurfelt.com/2012/02/28/why-everyone-thinks-theyr...

Compare to DK slope…

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#228

Post by pedrosorio »

mike_hearn wrote: post #218
danbruc wrote: post #203

Earlier quoted context omitted.

I could not quickly find the LSAT preparation guide but I found some LSAT sample questions [1] and they seem suitable to assess reasoning abilities. Also I do not think that it really matters which questions you choose as long as they span a wide enough difficulty range so that you are able to separate participants. [1] https://www.petersons.com/blog/sample-lsat-test-questions/

Hmm, do they? The logical reasoning test in that page is a question about lab rat studies on coffee+birth defects, and a hypothetical spokesperson's response that they wouldn't apply a warning label because the government would lose credibility if the study were to be refuted in future. You're then asked a multiple choice question: 1. Which of the following is most strongly suggested by the government’s statement abo…

> Given the structure of this question I assumed there'd be more than one right answer

I did not, since the question is explicit about there being one correct answer only: “Which of the following is most strongly suggested by the government’s statement above?”

> but this doesn't resemble the kind of logic test I'm used to. It's about unstated/assumed implications of natural language statements

Agreed.

> But (C) and (D) appear to just be minor re-phrasings of each other.

I think the key here is “there are doubts”. The government’s position stems from doubts on the conclusive nature of the study, that’s it. The statement doesn’t say anything about how much data studies on rats provide about human birth defects. If we’re being logical, studies on rats provide “no data” on human birth defects. Across many studies with different substances there may be a correlation (p(human birth defect | rat birth defect) = x), but an observation of birth defects on rats for a particular substance gives us data about rat birth defects, not human ones.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#229

Post by sbilstein »

lelanthran wrote: post #194
pastage wrote: post #173

Earlier quoted context omitted.

No this is not new. You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superior. There is still "science" being done to bolster those political views.

>> What's new is that any research that might produce results counter to the what the PC-mob deems acceptable is attacked. > No this is not new. I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently. > You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superio…

the “PC mob” might be new but we’ve had mobs of every political, religious, and cultural motivation pressuring academia since it’s invention.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#230

Post by mike_hearn »

pedrosorio wrote: post #228
mike_hearn wrote: post #218

Earlier quoted context omitted.

Hmm, do they? The logical reasoning test in that page is a question about lab rat studies on coffee+birth defects, and a hypothetical spokesperson's response that they wouldn't apply a warning label because the government would lose credibility if the study were to be refuted in future. You're then asked a multiple choice question: 1. Which of the following is most strongly suggested by the government’s statement abo…

> Given the structure of this question I assumed there'd be more than one right answer I did not, since the question is explicit about there being one correct answer only: “Which of the following is most strongly suggested by the government’s statement above?” > but this doesn't resemble the kind of logic test I'm used to. It's about unstated/assumed implications of natural language statements Agreed. > But (C) and (…

Ah yes - is vs are. You're right. I think I assumed there'd have to be >1 right answer after reading the options.

It's a remarkably poor question, but option (C) isn't about doubts on the conclusive nature of this specific study, but rather the nature of all studies on all animals. You could credibly argue (and I'd hope a lawyer would!) that no government would base policy on doubting all animal studies and that their position in this case must therefore be due to something about this specific study, e.g. the usage of rats, or the topic of birth defects, or both. So they could argue that (D) is the most logical answer.

Not that it really matters. Pretty clearly the LSAT authors are using the word logical in the street sense of "makes sense" or "sounds plausible" rather than meaning "based on an inference process that's free of fallacies". If DK based their test of competence on questions like this then it doesn't mean much, in my view.

Post reply on HN