The Irreproducibility Crisis of Modern Science • Hacker News Board

Re: The Irreproducibility Crisis of Modern Science

#101

Post by lisper » Tue, Apr 17, 2018, 4:42 PM UTC

Waterluvian wrote: post #90

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

We need a journal that exclusively publishes papers with results that were totally contrary to the hypothesis. So long as the researchers aren't outright inept or fraudulent, what's there to be ashamed about? Those would be some of the more interesting papers to read, in my opinion.

> We need a journal that exclusively publishes papers with results that were totally contrary to the hypothesis.

No, that won't help. You'd end up with the Journal of Creation Science and things like that.

What we need is pre-registration: https://cos.io/prereg/

We also need to change the culture so that positive results are not disproportionately rewarded.

Re: The Irreproducibility Crisis of Modern Science

#102

Post by throwawaymarx » Tue, Apr 17, 2018, 4:47 PM UTC

TangoTrotFox wrote: post #99

danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just comp…

Small correction. A basic income is guaranteed no matter what; there’s no means testing or anything like that. You can’t game the system because there’s nothing to game: you get the money regardless. You seem to be describing the system we have now, where you’re paid for results.

Re: The Irreproducibility Crisis of Modern Science

#103

Post by mattnewport » Tue, Apr 17, 2018, 4:50 PM UTC

lmm wrote: post #63

mattnewport wrote: post #55

Earlier quoted context omitted.

If a decision has to be made then science as a guide may be the best option but much of the time it would be better for government not to try and make policy at all in complex areas. Using "healthy eating" as an example, governments might try to influence what people eat by taxing or even outlawing "unhealthy" foods or subsidizing "healthy" foods. Instead of arguing over whether and how much the government should be…

> Instead of arguing over whether and how much the government should be imposing sin taxes on fats (and what kind of fats?) or sugars (and what kind of sugars), the better approach would be to do nothing and let people make their own choices based on the best information available to them at the time. Why? Do we believe that individuals are likely to be better informed than governments? (why?) Uncertainty is a fact o…

The history of government attempts to get people to eat "healthily" is a pretty good illustration of the many ways that government policy is a bad tool for addressing these type of issues. Government policy tends to have a lot of inertia and is bad at adapting to changing information. This is particularly evident when it attempts to track evolving scientific understanding of complex topics like nutrition and diet and their relation to health. It is also prone to being overly influenced by special interests and this particular area is full of textbook examples of regulatory capture and public choice theory.

I certainly believe that some individuals will be better informed and better able to translate that information into the best course of action for themselves given their goals than governments, especially "governments" as represented by a patchwork of laws and regulations the impact a complex thing like individual health. This is true even at the time that new policy is enacted and tends to become more true over time due to the inertia of government policy that goes down the wrong path.

Re: The Irreproducibility Crisis of Modern Science

#104

Post by Tenoke » Tue, Apr 17, 2018, 4:50 PM UTC

Erlich_Bachman wrote: post #79

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

4.) You mean that papers are published with P When did this become a "standard"? Most papers I have seen in respectable journals and other sources would not strive for such a high Pvalue, it would be much lower like 0.001.

>When did this become a "standard"?

Most people would say in the 20s and point at Fischer's publications, though you can have arguments for later and earlier. Some of that is mentioned in the Wiki page, and you can see 0.05 all over the article [0], and the other related wiki pages[1]. You can find some key quotes there:

"The significance level for a study is chosen before data collection, and typically set to 5% or much lower, depending on the field of study"

"In 1925, Ronald Fisher advanced the idea of statistical hypothesis testing, which he called "tests of significance", in his publication Statistical Methods for Research Workers. Fisher suggested a probability of one in twenty (0.05) as a convenient cutoff level to reject the null hypothesis.In a 1933 paper, Jerzy Neyman and Egon Pearson called this cutoff the significance level, which they named α."

If you mostly see lower p-values, that most likely means that you usually read non-social science papers. I guarantee that most social science (and related) fields use 0.05 regardless of the respectability of the journal.

As an example of fields with lower p-values - 0.001 is common in medicine, and physics commonly has even lower values.

0. https://en.wikipedia.org/wiki/P-value#History 1. https://en.wikipedia.org/wiki/Statistical_significance#Histo...

Re: The Irreproducibility Crisis of Modern Science

#105

Post by 3JPLW » Tue, Apr 17, 2018, 4:50 PM UTC

Waterluvian wrote: post #90

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

We need a journal that exclusively publishes papers with results that were totally contrary to the hypothesis. So long as the researchers aren't outright inept or fraudulent, what's there to be ashamed about? Those would be some of the more interesting papers to read, in my opinion.

Negative results are really hard to do right, and even harder to do interestingly. There are so many possible confounders in experiment design and execution that they generally aren't worth evaluating.

So we choose to spend our time reviewing and evaluating those experiments that do seem to have at least some chance of saying something interesting. Even then, though, depending on the discipline, if the researcher did everything perfectly, they're still saying that there's a 5% or some% chance that they just "got lucky" and there is no underlying effect at all.

Re: The Irreproducibility Crisis of Modern Science

#106

Post by chrisfosterelli » Tue, Apr 17, 2018, 4:53 PM UTC

toufka wrote: post #84

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

I'd add a last point that a scientific paper (or a particular experimental result) is not sufficient for 'science'. Citing a paper as either a truth or discovery is a disingenuous. The endeavour of science gets at a kind of consensus by having lots of papers, lots of experiments, lots of additional layers of experiments build on each other. And yes it's annoying to realize a given paper is misleading, has errors, or…

Thank you for saying this! This is a common view I see outside academia, where people interpret a single paper to mean that something is definitively shown.

I think news organizations make this worse. Whenever a new paper comes out they'll champion it as fact if doing so leads to clicks.

Peer reviewers can only hope to validate that the science was done well, not that it is inarguably correct. It requires a body of work around a topic before it's fair to say at all that something is "for sure" proven.

Re: The Irreproducibility Crisis of Modern Science

#107

Post by haZard_OS » Tue, Apr 17, 2018, 4:53 PM UTC

buvanshak wrote: post #86

timtadh wrote: post #62

In response to several threads here: it is important to distinguish when scientists are self critical vs. when non-scientists are critical of the scientific method. For instance, there is a long history of scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. That work is sometimes used by non-scientists who question the overall scientific meth…

>scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process. >It does not in itself show that science is inherently untrustworthy. I think when statistics is involved, the results are inherently untrustwo…

>I think when statistics is involved, the results are inherently untrustworthy.

Ummm...are you kidding? Statistically vetted results are inherently UNCERTAIN, but how could they possibly be inherently untrustworthy?

Even if a mechanistic effect is observed, its relationship to a particular cause or influence is only established statistically. In fact, the very observation is often performed u Dee the umbrella of statistical calibration of appropriate instruments.

  As Pearson said, "Statistics is the grammar of science".

Re: The Irreproducibility Crisis of Modern Science

#108

Post by lisper » Tue, Apr 17, 2018, 4:53 PM UTC

buvanshak wrote: post #93

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

>For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science. I am not sure this is correct. The rules that govern celestial bodies is same as the ones that govern objects on earth. So why are they not reproducible? It does not require to measure forces between celestial bodies to measure the value of G. Measuring the forces between two mass…

Yes, of course the rules are the same. Figuring this out was the event that launched the entire modern scientific endeavor. But nonetheless, the experimental data that went into this discovery was largely non-reproducible. The stars and planets go where they go and only very rarely does a given configuration repeat itself. In fact, no configuration ever really repeats itself in every detail.

So, for example, we can predict with ridiculous accuracy where the next total solar eclipse is going to happen. But once it happens, we can't roll back the clock and make it happen again.

Re: The Irreproducibility Crisis of Modern Science

#109

Post by TangoTrotFox » Tue, Apr 17, 2018, 4:55 PM UTC

buvanshak wrote: post #86

timtadh wrote: post #62

In response to several threads here: it is important to distinguish when scientists are self critical vs. when non-scientists are critical of the scientific method. For instance, there is a long history of scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. That work is sometimes used by non-scientists who question the overall scientific meth…

>scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process. >It does not in itself show that science is inherently untrustworthy. I think when statistics is involved, the results are inherently untrustwo…

>"I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong."

Another very real issue here is that malicious use of statistics can be used to show nearly anything in ways that can be extremely difficult to detect, even when the maliciousness is hidden in plain sight. And then going a step beyond that there's plain old number fudging which is almost impossible to prove since variance works as sufficient plausible deniability. And finally there is of course plain old ineptitude. Like you mention even when trying to do things completely by the book, statistics are incredibly difficult to get right.

Something that comes to mind here is the recent MIT study stating that Uber drivers earned $3.37/hour. That study was completely broken. [1] It's debatable whether the cause was maliciousness or ineptitude, but the point is that these problems arise, with a disturbing regularity, even when the most reputable of names are attached to them.

[1] - https://qz.com/1222744/mits-uber-study-couldnt-possibly-have...

Re: The Irreproducibility Crisis of Modern Science

#110

Post by dyarosla » Tue, Apr 17, 2018, 4:55 PM UTC

throwawaymarx wrote: post #102

TangoTrotFox wrote: post #99

Earlier quoted context omitted.

I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just comp…

Small correction. A basic income is guaranteed no matter what; there’s no means testing or anything like that. You can’t game the system because there’s nothing to game: you get the money regardless. You seem to be describing the system we have now, where you’re paid for results.

I think what the parent is saying is that with a basic income only for researchers that creates a scenario where ‘Well you need to ensure there's nobody just completely gaming the system and so inextricably you'll end up with some sort of qualifier for results.’ That you don’t have ‘fake researchers’ gaming the ‘basic income only for researchers’ reward.