I can't let go of “The Dunning-Kruger Effect is Autocorrelation” • Hacker News Board

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#231

Post by bigfudge » Fri, Apr 22, 2022, 1:12 PM UTC

tsimionescu wrote: post #171

FrenchDevRemote wrote: post #125

Earlier quoted context omitted.

>A problem here is that there are fields of science that are almost certainly bogus in themselves. One very likely candidate is nutrition, which seems to be fumbling in the dark and has a long history of producing worse recommendations than doing nothing (e.g. replace fat with sugar). It's not the field that is bogus in that case. People were quite literally bribed to push this. This could happen anywhere, anytime, i…

That was just an example, but we can go into more detail - at every point, nutrition studies are highly questionable. Sample sizes are usually minuscule (you can find important studies with 5-10 subjects, and even the largest studies rarely have more than a few hundred), they are often biased samples (only overweight people, only people with heart disease or diabetes etc.), they often don't account for likely confoun…

I think commercial operations like Huel and similar supplements will definitely be thought of like that. It just seems crazy, having watched the evolution of "scientific" diet advice over the past 70 years, to now think it's plausible that industrially-produced protein drinks are a suitable food substitute.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#232

Post by tpoacher » Fri, Apr 22, 2022, 1:17 PM UTC

Great article. Very nicely written.

In partial "defense" of the "autocorrelation" article, the author was in fact arguing against their own perceived definition of DK, not what most people consider to be DK. They just didn't realise it.

Which is an all too common thing to begin with. (that particular article pulled the same stunt with the definition of the word 'autocorrelation', after all).

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#233

Post by bigfudge » Fri, Apr 22, 2022, 1:17 PM UTC

rkangel wrote: post #191

darawk wrote: post #86

Earlier quoted context omitted.

Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand math…

One way that journals could actually add value (a concept to which they seem resistant) would be to review the statistical analysis. Statistics is hard and easy to get subtley wrong, and is often an independent skill to the underlying science. If journals had statistics experts to critique the analysis techniques prior to publishing it would be a great improvement in the confidence in which we could read papers.

I think this is a great idea, but in my experience there are surprisingly few such experts available. In practice, most statisticians not doing active methods research (I'm thinking of 'trial statisticians' mostly here, in CTU's) just cargo cult whatever procedures previous trials used. I guess they would pick up issues around sample size, but without also integrating that with some substantive knowledge about plausible effect sizes I'm not sure what value they would have.

Plus, it would reduce the number of publishable papers quite substantially including from high profile authors/groups, so I don't think they want the fight. We should also remember that most journal editors are also involved in publishing this research — they often have no real incentives to make things awkward.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#234

Post by bigfudge » Fri, Apr 22, 2022, 1:19 PM UTC

quanto wrote: post #89

darawk wrote: post #86

Earlier quoted context omitted.

Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand math…

Completely agreed. While I strongly believe in citizen science and people's right (and perhaps obligation) to critique established science, there is just so much poor analyses done by people to criticize some scientific findings they do not approve, motivated emotionally or otherwise. This phenomenon does not bring us closer to finding scientific facts or resolving the replication crisis. People should learn some bas…

We should really just do (and certainly publish) less research.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#235

Post by Tyr42 » Fri, Apr 22, 2022, 1:19 PM UTC

karpierz wrote: post #12

goosedragons wrote: post #4

"The second option conforms with the Research Methods 101 rule-of-thumb “always assume independence.” Until proven otherwise, we should assume people have no ability to self-assess their performance" It's not that at all. The assumption should be that everyone is equally good (or bad) at assessing their performance. Not that they have no ability but that the means between groups is the same vs. not the same. That the…

This confused me at first too. The issue is that "X" is your performance, and "Y" is your perceived performance. Say that everyone is equally okay at assessing themselves, and get within 0.1 of their actual performance (rated from 0 to 1). Then X and Y are going to be very correlated, as X - 0.1 The only case where X and Y wouldn't correlate at all is if people have no ability to assess their performance (IE, Y isn't…

That's exactly the difference this article is driving at!

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#236

Post by danbruc » Fri, Apr 22, 2022, 1:27 PM UTC

mike_hearn wrote: post #218

danbruc wrote: post #203

Earlier quoted context omitted.

I could not quickly find the LSAT preparation guide but I found some LSAT sample questions [1] and they seem suitable to assess reasoning abilities. Also I do not think that it really matters which questions you choose as long as they span a wide enough difficulty range so that you are able to separate participants. [1] https://www.petersons.com/blog/sample-lsat-test-questions/

Hmm, do they? The logical reasoning test in that page is a question about lab rat studies on coffee+birth defects, and a hypothetical spokesperson's response that they wouldn't apply a warning label because the government would lose credibility if the study were to be refuted in future. You're then asked a multiple choice question: 1. Which of the following is most strongly suggested by the government’s statement abo…

The second question labeled analytical reasoning is probably closer to what you consider a logical reasoning question, maybe they picked questions more like those?

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#237

Post by Scea91 » Fri, Apr 22, 2022, 1:27 PM UTC

TrackerFF wrote: post #199

darawk wrote: post #98

I can't believe nobody has pointed out that the original article debunking the DK effect is in fact an example of the effect. Truly poetic.

Tbh, I only ever hear people reference DK when they're trying to point out that they're "working with morons" (in their opinion, of course).

...and claim they suffer from impostor syndrome themself.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#238

Post by Scea91 » Fri, Apr 22, 2022, 1:30 PM UTC

irrational wrote: post #226

> If you tell me you didn’t have a single serious thought of self-assessing today, even semi-conscious, I simply won’t believe you. I stopped reading at this point. Someone that is so certain that they say “I simply won’t believe you.” is too self-assured to be worth paying much attention to.

If you tell me that Earth is flat I simply won't believe you.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#239

Post by john_pryan » Fri, Apr 22, 2022, 1:33 PM UTC

andersource wrote: post #225

john_pryan wrote: post #224

Earlier quoted context omitted.

That's a nice spot about the decreasing CC as we increase accuracy! My hypothesis would be that some of the DK effect in the original paper may be down to an effect like this (as suggested in the original article) but that asserting it is completely incorrect because of it is premature. We'd need access to more data to verify that the level of reliability was sufficiently acceptable.

Right. Just to be clear, "an effect like this" is (comparatively) unreliable tests, not some elusive statistical phenomena as implied by the original article. I'd have no issue if the author had called the article "the DK effect is due to poor skill tests", spent 5 minutes showing that the DK results are consistent not only with their claims but also with unreliable tests (like you did), then went on to show data tha…

I agree, the article seems to imply the plot in the original paper will always be an incorrect thing to do, instead of something which can have some issues in cases when we have inaccurate tests.

I've gone back and updated the colab notebook to use orderings exclusively instead of values and you can see that the auto correlation plot B from the first article exists when the noise is high enough but disappears when you reduce it, definitely not a statistical law.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#240

Post by topaz0 » Fri, Apr 22, 2022, 1:38 PM UTC

andersource wrote: post #146

omnicognate wrote: post #126

Gah, I wish I had time to fully read this and get into it, but I have to spend the next few hours driving. Unfortunately the original article isn't very clearly explained, and it's only on reading the discussion in the comments under it that it becomes clear what it's actually saying. The point is about signal & noise. Say your random variable X contains a signal component and a noise component, the former determinis…

Hey omnicognate, good to see you here, appreciated our previous discussion. What you're saying is that we need to verify the statistical reliability of the skill tests DK gave, and to some extent that we need to scrutinize the assumption that there indeed is such a thing as "skill" to be measured in the first place. I hope we can both agree that skill exists. That leaves the test reliability (technical term from stat…

I think the other piece that has been glossed over a bit is that DK are using quantiles (for both the test and the self-assessment). That means everything is bounded by 0 and 1, and you can't underestimate your performance if it was poor, or overestimate your performance if it was perfect. Or conversely, if you're the most skilled person in the room, your (random) actual performance on the day of the test is bounded above by your true skill, and vice versa for the least skilled. So e.g. we could simulate data with perfect self-assessment of overall skill, add a small amount of noise to actual performance on the day of the test, and get the same results. The bottom quartile (grouped by actual test score) will be a mix of people who are actually in the bottom quartile in skill and some who are in the higher quartiles. The top quartile by actual test score will be a mix of some from the top quartile in skill and some from lower quartiles.