Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

andersource.dev

Post reply on HN
Print view View original

201–210 of 308 posts

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#201

Post by mike_hearn »

blue_pants wrote: post #197
mike_hearn wrote: post #181

Earlier quoted context omitted.

The reasonable middle ground is, or probably should be, closer to the latter end than the former end unfortunately. Put simply, science is advertised as self-correcting but in reality it's not. Representative experience documented here: http://crystalprisonzone.blogspot.com/2021/01/i-tried-to-rep... So, the reasons people learn a generalized distrust of science are that often the sausage doesn't get made. Bad science…

Maybe I'm missing something, but 'self-correcting' doesn't necessarily mean 'immediately self-correcting'. I guess it's safe to assume, that incorrect studies are not cementing our world view and entirely stopping us from questioning studied topics again. The way I see the self-correcting nature of science: the truthiness of our view about specific set of topics increases over time (in some approximation).

Self-correcting doesn't mean immediately self correcting, but it does imply self-correcting in a somewhat reasonable time period, and ideally not needing to self correct too often.

What's reasonable, well, probably not years or decades. Average people cannot make major errors that destroy the value of their job output and then blow it off with "well but the company self corrected eventually so please don't fire me". When they judge science, they will judge it by the standards they are themselves held to in normal jobs.

And what's too often, well, probably papers that don't replicate should be a clear minority instead of (in some fields) the majority. Recall that failure to replicate is only one of many things that can go wrong with a study. Even if the replication rate was 100% many fields would still be filled with unusable papers.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#203

Post by danbruc »

mike_hearn wrote: post #200
danbruc wrote: post #192

Earlier quoted context omitted.

I totally agree, the first study with the jokes seems silly. But I am also not from the field, maybe it is not actually as silly as it seems to me. But the other studies seem much better to me and removing the first one would not change the conclusions.

Is there supplemental material I didn't notice? I only scan read it after the joke section but I can't find any mention of supplemental data anywhere. That's a problem because although you say the other tests are better, no information appears to be provided on which we can judge that. Let's look at the second test. It's advertised as a "logic test". The description is: > Participants then completed a 20-item logical…

I could not quickly find the LSAT preparation guide but I found some LSAT sample questions [1] and they seem suitable to assess reasoning abilities. Also I do not think that it really matters which questions you choose as long as they span a wide enough difficulty range so that you are able to separate participants.

[1] https://www.petersons.com/blog/sample-lsat-test-questions/

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#205

Post by Kye »

lelanthran wrote: post #194
pastage wrote: post #173

Earlier quoted context omitted.

No this is not new. You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superior. There is still "science" being done to bolster those political views.

>> What's new is that any research that might produce results counter to the what the PC-mob deems acceptable is attacked. > No this is not new. I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently. > You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superio…

Good science defines its terms. Can you unroll "PC-mob" so we're all on the same page here? You don't sound like an asshole, so your meaning is probably not the usual "anything that leans a little left."

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#206

Post by ta123457864 »

devit wrote: post #46

The "The Dunning-Kruger Effect is Autocorrelation" article is an example of obvious bullshit. Their claim that "If we have been working with random numbers, how could we possibly have replicated the Dunning-Kruger effect?" is the first blatantly false statement, and then the rest is built upon that so it can be safely disregarded. It's easy to see this because while the effect is present if everyone evaluates themsel…

Your comment amounts to saying that some of the randomly generated data really is consistently over estimating it's performance. How absurd.

Like similar analyses here you don't factor in that DK is about bias. Of course you can't see bias when test score=self assessment. That's because "IF everyone perfectly knows their score then there is no bias in their assessment" is a tautology.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#207

Post by snovv_crash »

Traster wrote: post #177
NumberCruncher wrote: post #172

> We don’t need statistics to learn about the world. A sentence, written by the author on, commented by me on and read by the HN community on devices, which exist only thanks to 80-90 years of rigorous, statistics based QA in engineering, especially in mechanical/hardware engineering. Anyhow, after spending years on a team filled with social science PHDs, I would not waste my time on reading papers about statistical…

I don't think your interpreting the sentence correctly, I think it's more correct to read it as: > We don't [only] need [the formal discipline of mathematics known as] statistics to learn about the world. Sure, there are things you can only functionally ascertain through statistical analysis. But not everything in the world needs rigorous statistics.

We don't need != We don't only need.

First one means "throw it away". Second one means "add other things too".

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#208

Post by Hendrikto »

rkangel wrote: post #191
darawk wrote: post #86

Earlier quoted context omitted.

Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand math…

One way that journals could actually add value (a concept to which they seem resistant) would be to review the statistical analysis. Statistics is hard and easy to get subtley wrong, and is often an independent skill to the underlying science. If journals had statistics experts to critique the analysis techniques prior to publishing it would be a great improvement in the confidence in which we could read papers.

If only journals actually wanted to contribute… They just want to skim.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#209

Post by omginternets »

mike_hearn wrote: post #190
kortilla wrote: post #142

Earlier quoted context omitted.

> but I don't think sound statistics can mislead. Of course they can, unless you magically exclude all statistics that made a bad assumption on independence. I plot all the daily high temperatures and the presence of the ice cream cart and it turns out the ice cream cart causes warmer highs! Solid statistics. Turns out the guy that has the ice cream cart has a weather app on his phone though and doesn’t come out on f…

Is that the fault of statistics though, or the non-statistical implication of causation that was tacked on the end of the statistical detection of correlation? Statistics is pretty explicit that it can't tell you about causality, right?

Yes, the whole point is that you can use sound statistics in bad faith.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#210

Post by jtc331 »

Pyramus wrote: post #195
mike_hearn wrote: post #188

Earlier quoted context omitted.

I think if you wanted to poke holes in the paper you'd start with the generic issues that are typical to much psychological research: 1. It uses a tiny sample size. 2. It assumes American psych undergrads are representative of the entire human race. 3. It uses stupid and incredibly subjective tests, then combines that with cherry picking: "Thus, in Study 1 we presented participants with a series of jokes and asked th…

> is a great example of why people don't or shouldn't take the social sciences seriously. Oh the irony in your last statement. Somebody who hasn't done social science research professionally (this is an assumption, let me know if I'm wrong), has difficulty judging what social science research can (and can't) do ...

One does not need to have done social science research to be able to recognize obvious general philosophy of science level problems with the methods used in much social science.

I’d we take your claim seriously then we have to disallow all critiques of the replicatability crisis in the social sciences that don’t come from social scientists, but that would present an obvious new problem: conflict of interest. It’s also just an absurd requirement.

Post reply on HN