The Irreproducibility Crisis of Modern Science • Hacker News Board

Re: The Irreproducibility Crisis of Modern Science

#211

Post by DanielBMarkham » Tue, Apr 17, 2018, 8:18 PM UTC

jrochkind1 wrote: post #201

Thrymr wrote: post #198

To be clear, the "NAS" (nas.org) that published this study is the National Association of Scholars [0], a political group, not the National Academy of Sciences (nasonline.org) [1], a nongovernmental organization that consists of scientists elected by their peers to provide independent scientific advice to the US government. There was in fact a study published recently in PNAS, the Proceedings of the National Academy…

Good catch thanks! Wikipedia says "The National Association of Scholars (NAS) is an American non-profit politically conservative advocacy group, with a particular interest in education" Thanks for the link to the PNAS -- looks like a whole special issue, not just an article, in fact. Great. I'm gonna ignore the fake-NAS one, and read the real-PNAS one. I wonder what National Association of Scholars' motivation is her…

Isn't the entire purpose of reproducability is to eliminate such questions from discussions about science?

Re: The Irreproducibility Crisis of Modern Science

#212

Post by lisper » Tue, Apr 17, 2018, 8:18 PM UTC

godelski wrote: post #182

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

> 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. This is why you include the error in your results. We don't care if your two experiments result in 100% the same results. We care about if your results line up within the error of your experiment. > 2. Even a completely non-reproducible result can be scientifically significant. For example, celestial event…

> Flipping 10 heads doesn't guarantee 5 tails

I never said it did. But in the long run, flipping a fair coin will generate pretty close to 50% heads and 50% tails. That's what it means to be a fair coin. Likewise, in the long run, using a p threshold of 0.05 (which many journals do) will generate 5% false positive results (that's that the 0.05 means), i.e. on average 1 false positive result for every 20 experiments you do.

Re: The Irreproducibility Crisis of Modern Science

#213

Post by UncleMeat » Tue, Apr 17, 2018, 8:22 PM UTC

rplst8 wrote: post #7

Retric wrote: post #3

Being reproducible is only critically important if people treat individual studies as meaningful. That IMO is a far more dangerous stance. Any study can have hidden flaws, none should be trusted without some form of replication.

How often is a study even repeated in the course of normal scientific research? I think most studies are focused on expanding the research and therefore knowledge about the science being studied. Accepting previous studies as fact is dangerous and could easily lead to a house of cards scenario. I think this is especially true in the very specific, niche studies that are common in today's highly competitive graduate s…

Explicitly replicated, rarely. But other similar studies will either reinforce or call into question the original result. This has a similar effect as replication over time.

Re: The Irreproducibility Crisis of Modern Science

#214

Post by 3JPLW » Tue, Apr 17, 2018, 8:28 PM UTC

lisper wrote: post #210

3JPLW wrote: post #200

Earlier quoted context omitted.

It's different because you're ignoring my second bullet point. There's a very common misconception that p-value is the percent chance the data confirm that the effect is real. That's not at all how p-values work. They are showing the _likelihood_ of generating the data were there no "effect." You make a "null hypothesis" — that is, you assume there is no effect — and you construct a distribution of the results you mi…

I think you're just misinterpreting what I mean by "failure" or "wrong result". I mean a positive result that is due to chance rather than to the null hypothesis actually being false. On that view, statistics alone guarantee a "failure" rate of at least 1 in 20. That's what choosing a threshold of 0.05 means . The on top of that baseline rate of false results you also have self-selection and occasional outright cheat…

Ah, yes, I read you as saying that "1 out of every 20 statistical tests in every journal is guaranteed to be wrong. It's just stats, people."

If what you mean to say is that "1 out of every 20 statistical tests of a false finding will demonstrate a 'significant' p-value and may subsequently get published," then we're in agreement.

Re: The Irreproducibility Crisis of Modern Science

#215

Post by emodendroket » Tue, Apr 17, 2018, 8:30 PM UTC

iguy wrote: post #206

emodendroket wrote: post #30

Earlier quoted context omitted.

Those views really were thought to be rational and scientific at the time they were promulgated, though, and Nazism was only their most extreme expression.

The problem with the Nazis is that they were genocidal maniacs, not that they p-hacked their scientific papers. Genocidal maniacs have spouted many creeds, some of which sound much nicer (like brotherly love & eternal life... or equality of mankind & ownership of the means of production) and they are still evil. Because mass murder is wrong. Full stop. I think it's really dangerous to tie the wrongness of the suppose…

So if the Nazis weren't mass-murders, and instead just had an ideology revolving around the inherent inferiority of anyone other than Aryans (naturally that would probably include bans on intermarriage, restrictions on social race-mixing, and the like), that would have been alright? I don't think I agree.

Frankly, genocide, or at least dehumanization and extreme indifference, seems like the logical endpoint of sincere belief in the concept of a "master race" which is inherently good and is hamstrung by perfidious inferior races. Once you've determined that a member of one race is worth less than another, it's natural to judge that the death of a member of this inferior race would be justifiable to save the race of a member of the superior race, and in this way the concept of genocide becomes, in your mind, a "defensive" action.

Re: The Irreproducibility Crisis of Modern Science

#216

Post by lisper » Tue, Apr 17, 2018, 8:32 PM UTC

3JPLW wrote: post #214

lisper wrote: post #210

Earlier quoted context omitted.

I think you're just misinterpreting what I mean by "failure" or "wrong result". I mean a positive result that is due to chance rather than to the null hypothesis actually being false. On that view, statistics alone guarantee a "failure" rate of at least 1 in 20. That's what choosing a threshold of 0.05 means . The on top of that baseline rate of false results you also have self-selection and occasional outright cheat…

Ah, yes, I read you as saying that "1 out of every 20 statistical tests in every journal is guaranteed to be wrong. It's just stats, people." If what you mean to say is that "1 out of every 20 statistical tests of a false finding will demonstrate a 'significant' p-value and may subsequently get published," then we're in agreement.

I meant to say what I said. Getting a significant p-value when in fact there is no effect is a wrong result, hence, at a minimum, 1 in 20 results (on average) in a journal that uses a p-value threshold of 0.05 will be wrong.

Re: The Irreproducibility Crisis of Modern Science

#217

Post by lotsofpulp » Tue, Apr 17, 2018, 8:33 PM UTC

iguy wrote: post #209

lotsofpulp wrote: post #177

Earlier quoted context omitted.

As I wrote above, studies about people, which the aforementioned Nazi stuff would fall into. I don't have a specific study in mind, but I do know that the further you stray from math/physics/chemistry, creating metrics and isolating variables is very, very difficult. I don't need a specific study to know that. Any sociology/economic/psychology research should be taken with a grain of salt.

It would be helpful if you would name specific examples where you think they were wrong. I mean wrong on empirical questions, not morally wrong. Just stating that all of soft science is dodgy doesn't really distinguish us from them, does it? We do a lot of soft science!

Why would that be helpful? The problem is obvious because of the complexity of the variables and problems measuring them. There might be some problem studies cited in the link below:

https://en.wikipedia.org/wiki/Replication_crisis

The point is that some research is weaker than others, and the reasons why haven't changed since WW2, so if it's a problem now, then it was a problem then.

Re: The Irreproducibility Crisis of Modern Science

#218

Post by iguy » Tue, Apr 17, 2018, 8:49 PM UTC

emodendroket wrote: post #215

iguy wrote: post #206

Earlier quoted context omitted.

The problem with the Nazis is that they were genocidal maniacs, not that they p-hacked their scientific papers. Genocidal maniacs have spouted many creeds, some of which sound much nicer (like brotherly love & eternal life... or equality of mankind & ownership of the means of production) and they are still evil. Because mass murder is wrong. Full stop. I think it's really dangerous to tie the wrongness of the suppose…

So if the Nazis weren't mass-murders, and instead just had an ideology revolving around the inherent inferiority of anyone other than Aryans (naturally that would probably include bans on intermarriage, restrictions on social race-mixing, and the like), that would have been alright? I don't think I agree. Frankly, genocide, or at least dehumanization and extreme indifference, seems like the logical endpoint of sincer…

They would not have acquired much notoriety by believing these things in private. It's their actions which got them into the history books.

If I'm not mistaken, the Brahmins tick most of your boxes. Are they alright? Some would argue that they go in for a bit of dehumanization... but nobody is proposing a world war to stop them.

Actually I'm alright with people believing lots of crazy shit, in private. Because I think intellectual monoculture is pretty dangerous.

Re: The Irreproducibility Crisis of Modern Science

#219

Post by emodendroket » Tue, Apr 17, 2018, 8:53 PM UTC

iguy wrote: post #218

emodendroket wrote: post #215

Earlier quoted context omitted.

So if the Nazis weren't mass-murders, and instead just had an ideology revolving around the inherent inferiority of anyone other than Aryans (naturally that would probably include bans on intermarriage, restrictions on social race-mixing, and the like), that would have been alright? I don't think I agree. Frankly, genocide, or at least dehumanization and extreme indifference, seems like the logical endpoint of sincer…

They would not have acquired much notoriety by believing these things in private. It's their actions which got them into the history books. If I'm not mistaken, the Brahmins tick most of your boxes. Are they alright? Some would argue that they go in for a bit of dehumanization... but nobody is proposing a world war to stop them. Actually I'm alright with people believing lots of crazy shit, in private. Because I thin…

Well, I don't know, the Indian government seems to think caste discrimination is a real problem (c.f. https://en.wikipedia.org/wiki/Reservation_in_India). I certainly would object quite strenuously to a government that operated on caste principles and enforced caste segregation and diminished rights for some castes. World War II did not begin because of Nazis' views on Jews, so I don't know that that part of the question really makes sense.

I do find it a little strange how many people want to turn the concept of "diversity" on its head and tell me the real diversity problem is that we don't let enough people with racist views into power.

Re: The Irreproducibility Crisis of Modern Science

#220

Post by logfromblammo » Tue, Apr 17, 2018, 9:00 PM UTC

lisper wrote: post #212

godelski wrote: post #182

Earlier quoted context omitted.

> 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. This is why you include the error in your results. We don't care if your two experiments result in 100% the same results. We care about if your results line up within the error of your experiment. > 2. Even a completely non-reproducible result can be scientifically significant. For example, celestial event…

> Flipping 10 heads doesn't guarantee 5 tails I never said it did. But in the long run, flipping a fair coin will generate pretty close to 50% heads and 50% tails. That's what it means to be a fair coin. Likewise, in the long run, using a p threshold of 0.05 (which many journals do) will generate 5% false positive results (that's that the 0.05 means ), i.e. on average 1 false positive result for every 20 experiments…

Your experiment shouldn't need to presume the coin is fair.

Flipping an unfair coin twice in a row and discarding the HH and TT doubles will generate close to 50% HT pairs and 50% TH pairs. This only presumes that subsequent flips are independent events.

It's still not going to be exactly 50%, because random events are random. You can get as small an error margin as you need by increasing the number of trials. And in some cases, this property of statistical analysis makes it easier to increase N than reduce the randomness and uncertainty in the experiment.