Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

andersource.dev

Post reply on HN
Print view View original

191–200 of 308 posts

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#191

Post by rkangel »

darawk wrote: post #86
quanto wrote: post #62

The conclusion in the article: > Why so angry? I know I’ve taken this far too personally. I have no illusions that everything I read online should be correct, or about people’s susceptibility to a strong rhetoric cleverly bashing conventional science, even in great communities such as HN. But frankly, for the last few years, the world seems to be accelerating the rate at which it’s going crazy, and it feels to me a l…

Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand math…

One way that journals could actually add value (a concept to which they seem resistant) would be to review the statistical analysis. Statistics is hard and easy to get subtley wrong, and is often an independent skill to the underlying science. If journals had statistics experts to critique the analysis techniques prior to publishing it would be a great improvement in the confidence in which we could read papers.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#192

Post by danbruc »

mike_hearn wrote: post #188
danbruc wrote: post #179

Read the actual paper [1], there is so much more than those charts. They ask for an assessment of the own test score and an assessment of the ranking among the other participants to distinguish between misjudgments of the own abilities and the abilities of others. They give participants access to the tests of other participants and check how this affects self assessments - competent participants realize that they hav…

I think if you wanted to poke holes in the paper you'd start with the generic issues that are typical to much psychological research: 1. It uses a tiny sample size. 2. It assumes American psych undergrads are representative of the entire human race. 3. It uses stupid and incredibly subjective tests, then combines that with cherry picking: "Thus, in Study 1 we presented participants with a series of jokes and asked th…

I totally agree, the first study with the jokes seems silly. But I am also not from the field, maybe it is not actually as silly as it seems to me. But the other studies seem much better to me and removing the first one would not change the conclusions.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#193

Post by [deleted] »

obastani wrote: post #45

I feel like this article is severely over-complicating the analysis. Looking at the original blog post [1], their key claim appears to be that "random data produces the same curves as the DK effect, so the DK effect is a statistical artifact". However, by "random data", the original blog means people and their self-assessments are completely independent! In fact, this is exactly what the DK effect is saying -- people…

[deleted]

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#194

Post by lelanthran »

pastage wrote: post #173
lelanthran wrote: post #135

Earlier quoted context omitted.

> I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. It's more than that, I think. Sibling-thread poster hit the nail on the head when he complained of politicised science. The social sciences have this dominating and silencing effect on the rest of the sciences. There's always been junk science, and when found out it gets discredited. Th…

No this is not new. You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superior. There is still "science" being done to bolster those political views.

>> What's new is that any research that might produce results counter to the what the PC-mob deems acceptable is attacked.

> No this is not new.

I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently.

> You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superior. There is still "science" being done to bolster those political views.

I don't see what that has to do with that I said - that a very vocal bunch of non-science people seem to have successfully lobbied into silencing specific topics.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#195

Post by Pyramus »

mike_hearn wrote: post #188
danbruc wrote: post #179

Read the actual paper [1], there is so much more than those charts. They ask for an assessment of the own test score and an assessment of the ranking among the other participants to distinguish between misjudgments of the own abilities and the abilities of others. They give participants access to the tests of other participants and check how this affects self assessments - competent participants realize that they hav…

I think if you wanted to poke holes in the paper you'd start with the generic issues that are typical to much psychological research: 1. It uses a tiny sample size. 2. It assumes American psych undergrads are representative of the entire human race. 3. It uses stupid and incredibly subjective tests, then combines that with cherry picking: "Thus, in Study 1 we presented participants with a series of jokes and asked th…

> is a great example of why people don't or shouldn't take the social sciences seriously.

Oh the irony in your last statement. Somebody who hasn't done social science research professionally (this is an assumption, let me know if I'm wrong), has difficulty judging what social science research can (and can't) do ...

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#196

Post by abirch »

quanto wrote: post #62

The conclusion in the article: > Why so angry? I know I’ve taken this far too personally. I have no illusions that everything I read online should be correct, or about people’s susceptibility to a strong rhetoric cleverly bashing conventional science, even in great communities such as HN. But frankly, for the last few years, the world seems to be accelerating the rate at which it’s going crazy, and it feels to me a l…

> That said, I give the benefit of doubt to the author of "The DK Effect is Autocorrelation." It is a human error to be overly zealous in some opinions without thinking it through.

If only there were a term for "a cognitive bias whereby people with limited knowledge or competence in a given intellectual or social domain greatly overestimate their own knowledge or competence in that domain relative to objective criteria or to the performance of their peers or of people in general"

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#197

Post by blue_pants »

mike_hearn wrote: post #181
russdill wrote: post #90

Earlier quoted context omitted.

I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. That is a problem and it does erode trust. But in some ways it's also how the sausage gets made, there's going to be room for things to get published they later gets refuted. People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becomi…

The reasonable middle ground is, or probably should be, closer to the latter end than the former end unfortunately. Put simply, science is advertised as self-correcting but in reality it's not. Representative experience documented here: http://crystalprisonzone.blogspot.com/2021/01/i-tried-to-rep... So, the reasons people learn a generalized distrust of science are that often the sausage doesn't get made. Bad science…

Maybe I'm missing something, but 'self-correcting' doesn't necessarily mean 'immediately self-correcting'. I guess it's safe to assume, that incorrect studies are not cementing our world view and entirely stopping us from questioning studied topics again.

The way I see the self-correcting nature of science: the truthiness of our view about specific set of topics increases over time (in some approximation).

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#198

Post by bmacho »

civilized wrote: post #52
nosefrog wrote: post #37

I think the main point of this post is correct -- just because you can find the effect in random noise, doesn't mean it's not real phenomenon that happens in real life. But it's missing a nuance there: if an effect can be replicated with random noise, then it's not a psychological effect (e.g. something that you would explain as a human bias), but a statistical effect. E.g. regression towards the mean is a real effec…

> if an effect can be replicated with random noise, then it's not a psychological effect This isn't true either. Statistical dependence does not determine or uniquely identify causal interpretation or system structure. See Judea Pearl's works (e.g. The Book of Why) for more on this. People lacking the ability to self-assess is interesting psychologically. People can learn from experience in many other contexts. Peopl…

> People lacking the ability to self-assess is interesting psychologically. People can learn from experience in many other contexts. People can judge their relative position versus other people in many contexts. Why would they be so bad at this particular task? There could be a psychological underpinning.

Is there an existing and relevant phenomenon about people lacking the ability to self-asses, that is true, proven, and not just trivia?

I do believe that people understand all the available information about their skills and performance, and they rate themselves according to it.

E.g. if they are asked about whether they perform good on an IQ test against their classmates they will produce noise (see E.g. the article "I can't let go of.."), and if they have the results of the IQ test, they will be able correctly calculate in which quartile they are.

Is there anything against this view?

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#199

Post by TrackerFF »

darawk wrote: post #98

I can't believe nobody has pointed out that the original article debunking the DK effect is in fact an example of the effect. Truly poetic.

Tbh, I only ever hear people reference DK when they're trying to point out that they're "working with morons" (in their opinion, of course).

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#200

Post by mike_hearn »

danbruc wrote: post #192
mike_hearn wrote: post #188

Earlier quoted context omitted.

I think if you wanted to poke holes in the paper you'd start with the generic issues that are typical to much psychological research: 1. It uses a tiny sample size. 2. It assumes American psych undergrads are representative of the entire human race. 3. It uses stupid and incredibly subjective tests, then combines that with cherry picking: "Thus, in Study 1 we presented participants with a series of jokes and asked th…

I totally agree, the first study with the jokes seems silly. But I am also not from the field, maybe it is not actually as silly as it seems to me. But the other studies seem much better to me and removing the first one would not change the conclusions.

Is there supplemental material I didn't notice? I only scan read it after the joke section but I can't find any mention of supplemental data anywhere. That's a problem because although you say the other tests are better, no information appears to be provided on which we can judge that.

Let's look at the second test. It's advertised as a "logic test". The description is:

> Participants then completed a 20-item logical reasoning test that we created using questions taken from a Law School Admissions Test (LSAT) test preparation guide (Orton, 1993).

That's the entire description of their method. So immediately, we can see the following problems:

1. Just like the joke test, there's no way to replicate this given the description in the paper. Which questions did they take and why? In turn this throws all claims that the DK study has been replicated into question.

2. The citation is literally a Cliffs Notes exercise for students. It's about memorization of answers to pass law exams, not an actual test itself designed to verify logical reasoning ability. Why do they think this is a good source of questions for testing logic? Law is not a system of logic, there's even a famous saying about that: "the life of the law is not logic but experience". If you wanted to test logical reasoning a more standard approach would be something like Raven's Matrices.

Putting my two posts together there's a third problem:

3. Putting aside the obvious problems with subjectivity, their joke test is defined in an illogical way. They define a test of expertise (working as a comedian), select some people who pass this test and define them as experts, then discover that one expert would have been ranked by their own test as "incompetent but doesn't know it". Yet this is a contradiction, because this person was selected specifically because the researchers defined them as competent. Rather than deal with this logical contradiction by reframing the question they simply ignore it by discarding that comedian from their expert pool.

This is good evidence that DK themselves weren't particularly logical people, yet, they claim to have designed a test of logic - a bold claim at the best of times. Ironically, it appears DK may be suffering from their own effect. They believe themselves to be competent at designing tests yet the evidence in their paper suggests they aren't.

Post reply on HN