The Irreproducibility Crisis of Modern Science • Hacker News Board

Re: The Irreproducibility Crisis of Modern Science

#191

Post by TangoTrotFox » Tue, Apr 17, 2018, 6:51 PM UTC

haZard_OS wrote: post #142

TangoTrotFox wrote: post #116

Earlier quoted context omitted.

I cannot speak for the person you are responding to, but in my [agreement] of his critique of statistics, I am implicitly speaking of social statistics. I think there is a vast difference in e.g. a statistical modeling of the behavior of electrons and e.g. the statistical modeling of some sort of human behavior.

...but that just isn't true. Statistical analysis is (among other things) a way to quantify our uncertainty. If done approriately, the statistics simply communicate the role probability played in moving from the experimental premises to the results. The level of uncertainty in most (perhaps all) experiments involving particles in a vacuum is far lower than experiments involving human behavior. Statistics doesn't crea…

I'm not entirely sure what you're trying to say, and your post formatting is not helping.

I would recommend reading the article. The executive summary summarizes things. The issue is not statistics in and of themselves, but how they are used in the overwhelming number of studies - particularly those in the social sciences or related with human physiology.

Re: The Irreproducibility Crisis of Modern Science

#192

Post by wsy » Tue, Apr 17, 2018, 6:56 PM UTC

JumpCrisscross wrote: post #153

danharaj wrote: post #148

Earlier quoted context omitted.

Even if that's true, that only science that can have a profit motive slapped on it can be done effectively in the current system is a problem. If only such science were done, then we would have different crises of science to deal with because profit seeking is at odds with much science, in particular basic research. It's also quite obviously the case that profit seeking research has incentives to be bad science in ot…

> It's also quite obviously the case that profit seeking research has incentives to be bad science in other ways I agree. My point is the Marxist framework is a bad one for modern science. We have endowed academics and ones working for private institutions and getting grants from public bodies run by scientists. Privately-financed biotech companies founded by academics taking moon shots and getting acquired by market…

I do not see much need for delineation. In today's world, every higher-level employee can be analyzed just fine from both sides:

- she is exploited, i.e., forced to act in a profit-maximizing way (in particular, short-term profit). Each other behavior will be penalized (no career, no tenure, ...).

- she is exploiting, i.e., forces her workforce to act in way that they maximize her profit.

Re: The Irreproducibility Crisis of Modern Science

#193

Post by Retric » Tue, Apr 17, 2018, 6:58 PM UTC

quotemstr wrote: post #42

Retric wrote: post #3

Being reproducible is only critically important if people treat individual studies as meaningful. That IMO is a far more dangerous stance. Any study can have hidden flaws, none should be trusted without some form of replication.

Are you claiming that reproductively is unimportant when multiple studies cover the same area? Do you think so because you imagine errors in the studies would be uncorrelated? Not the case: look at the social sciences. Errors are very much coordinated.

There are many examples where multiple experiments moved from an original bad result and slowly converged to a correct value. People will throw out results that where overly far from what was expected, but the drift is in the correct direction.

And really science need not move quickly as long as it continues to converge on accurate answers that's plenty useful.

EX: Nutrition research often get's dumped on, but it's far from useless. There are 41 things we know you need or issues will show up: (http://www.nutrientsreview.com/glossary/essential-nutrients + energy + water) Getting the specifics right is complex and we are working on it. Thiamine deficiency from white rice is the kind of thing you need to get right or die, 'optimal health' is a harder target.

Re: The Irreproducibility Crisis of Modern Science

#194

Post by TangoTrotFox » Tue, Apr 17, 2018, 7:04 PM UTC

3JPLW wrote: post #114

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the…

Are you aware of the scale of failure here? The article (referring to the executive summary here) gives some really incredible figures. In psychology 100 reproducibility studies were only able to produce statistically significant results from 36% of prominent papers - compared to the 97% of the original studies. In biotech, a firm tried to reproduce 53 "landmark" studies in hematology and oncology - only 6 could be replicated.

The reason this is being called a 'crisis' is because it's looking like the vast majority of science, particularly in the social and human physiological sciences, is junk.

Re: The Irreproducibility Crisis of Modern Science

#195

Post by foldr » Tue, Apr 17, 2018, 7:04 PM UTC

lotsofpulp wrote: post #177

foldr wrote: post #146

Earlier quoted context omitted.

Which studies are you referring to? If you don't have anything specific in mind, then you can't really judge the quality or compare it to the quality of comparable research today.

As I wrote above, studies about people, which the aforementioned Nazi stuff would fall into. I don't have a specific study in mind, but I do know that the further you stray from math/physics/chemistry, creating metrics and isolating variables is very, very difficult. I don't need a specific study to know that. Any sociology/economic/psychology research should be taken with a grain of salt.

>Any sociology/economic/psychology research should be taken with a grain of salt.

So how is that not an example of "science going wrong"? Unless you are proposing that science is right by definition, or that governments should ignore most of the branches of science that are actually relevant to government.

There are plenty of branches of physics that are at least as speculative as sociology, economics and psychology (e.g. cosmology). The only difference is that these branches of physics have few political implications.

Re: The Irreproducibility Crisis of Modern Science

#196

Post by coldacid » Tue, Apr 17, 2018, 7:06 PM UTC

makecheck wrote: post #179

There has to be similar prestige/career-building/notoriety/funding for spending time on reproducing the experiments of others. Without that shift there will clearly be a greater tendency to just try something new. Also, when experiments depend on source code, etc. we need real engineering tools/principles applied. (Something like: “you can’t publish paper X if you aren’t including a public repository with build/run i…

So, Jupyter?

Re: The Irreproducibility Crisis of Modern Science

#197

Post by 3JPLW » Tue, Apr 17, 2018, 7:11 PM UTC

TangoTrotFox wrote: post #194

3JPLW wrote: post #114

Earlier quoted context omitted.

> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the…

Are you aware of the scale of failure here? The article (referring to the executive summary here) gives some really incredible figures. In psychology 100 reproducibility studies were only able to produce statistically significant results from 36% of prominent papers - compared to the 97% of the original studies. In biotech, a firm tried to reproduce 53 "landmark" studies in hematology and oncology - only 6 could be r…

Absolutely! I'm only addressing the common misconception in the grandparent that the p-value is the percent likelihood that the effect is real. That's not at all accurate.

There are definitely other reasons that lead us to such horrifying results — especially in fields where experimenters _do_ conduct experiments seemingly at random and without a reasonable pathway of action (I'm looking at you, social psychology and friends).

Re: The Irreproducibility Crisis of Modern Science

#198

Post by Thrymr » Tue, Apr 17, 2018, 7:15 PM UTC

To be clear, the "NAS" (nas.org) that published this study is the National Association of Scholars [0], a political group, not the National Academy of Sciences (nasonline.org) [1], a nongovernmental organization that consists of scientists elected by their peers to provide independent scientific advice to the US government. There was in fact a study published recently in PNAS, the Proceedings of the National Academy of Sciences, on this topic [2].

[0] https://en.wikipedia.org/wiki/National_Association_of_Schola...

[1] https://en.wikipedia.org/wiki/National_Academy_of_Sciences

[2] http://www.pnas.org/content/early/2018/03/08/1802324115

Re: The Irreproducibility Crisis of Modern Science

#199

Post by wrp » Tue, Apr 17, 2018, 7:22 PM UTC

toufka wrote: post #84

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

I'd add a last point that a scientific paper (or a particular experimental result) is not sufficient for 'science'. Citing a paper as either a truth or discovery is a disingenuous. The endeavour of science gets at a kind of consensus by having lots of papers, lots of experiments, lots of additional layers of experiments build on each other. And yes it's annoying to realize a given paper is misleading, has errors, or…

> ...come up with an experimental design that avoided a statistical outcome and instead probed at an either/or mechanism.

That is the idea behind Rutherford's famous motto, If your experiment needs statistics, you ought to have done a better experiment.

Re: The Irreproducibility Crisis of Modern Science

#200

Post by 3JPLW » Tue, Apr 17, 2018, 7:29 PM UTC

lisper wrote: post #135

3JPLW wrote: post #114

Earlier quoted context omitted.

> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the…

> > at least one result in 20 will be due to chance > That's wildly overly pessimistic Why? You yourself said: > The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance) How is that different from what I said? 5% is just another way of writing 1 in 20.

It's different because you're ignoring my second bullet point. There's a very common misconception that p-value is the percent chance the data confirm that the effect is real. That's not at all how p-values work.

They are showing the _likelihood_ of generating the data were there no "effect." You make a "null hypothesis" — that is, you assume there is no effect — and you construct a distribution of the results you might see in such a case. If your results are relatively unlikely, then you say the null hypothesis is rejected at a p-value level, leading support to the fact that there is such an effect.

But there's another bullet in my comment. You could _also_ see a low p-value if the null hypothesis is wrong! That would _also_ lead to a low p-value, but for a completely different reasons — reasons that are typically elucidated in the entire rest of the paper. Thus, you should be most suspicious of papers that do not have a reasonable cause of action, as that should increase your prior that the null hypothesis is actually correct.

If I were to edit that post, I'd add a third bullet — that they cheated or did bad science. My objection is to you stating that the statistical tests alone guarantee at least a 1 in 20 failure rate with a p-value of .05. There are indeed many other reasons to be suspect, but the statistical tests don't guarantee this.