The Irreproducibility Crisis of Modern Science • Hacker News Board

Re: The Irreproducibility Crisis of Modern Science

#41

Post by monocasa » Tue, Apr 17, 2018, 3:47 PM UTC

scblock wrote: post #21

genericone wrote: post #14

Earlier quoted context omitted.

Racial 'science' was used by the Nazi's to justify antisemitism, eugenics, and everything they did beyond that. The scientific method is important to be used, but don't trust something just because another person believes it to be science. You need the hypothesis and the verification method. Others (all of us) have the obligation to challenge both the hypothesis as well as the verification methods.

So, 'science' not science? Got it. That was a pretty quick Godwining, by the way.

The issue is that it's not easy for the populace to tell the difference between 'science' and science in a lot of cases. You see this on all parts of the political spectrums. Soviet Lysenkoism, left wing anti vax crap, and neo liberal austerity.

EDIT: Krugman had this to say about the R&R austerity paper "What the Reinhart-Rogoff affair shows is the extent to which austerity has been sold on false pretenses. For three years, the turn to austerity has been presented not as a choice but as a necessity. Economic research, austerity advocates insisted, showed that terrible things happen once debt exceeds 90 percent of G.D.P. But "economic research" showed no such thing; a couple of economists made that assertion, while many others disagreed. Policy makers abandoned the unemployed and turned to austerity because they wanted to, not because they had to."

https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt

Particularly telling: "The paper was published in an annual 'Papers and Proceedings' edition of The American Economic Review that was not subject to the same peer-review standards that other editions use before publication"

Re: The Irreproducibility Crisis of Modern Science

#42

Post by quotemstr » Tue, Apr 17, 2018, 3:47 PM UTC

Retric wrote: post #3

Being reproducible is only critically important if people treat individual studies as meaningful. That IMO is a far more dangerous stance. Any study can have hidden flaws, none should be trusted without some form of replication.

Are you claiming that reproductively is unimportant when multiple studies cover the same area? Do you think so because you imagine errors in the studies would be uncorrelated? Not the case: look at the social sciences. Errors are very much coordinated.

Re: The Irreproducibility Crisis of Modern Science

#43

Post by m-watson » Tue, Apr 17, 2018, 3:47 PM UTC

mistrial9 wrote: post #26

Some of the comments miss an important distinction.. Science in the popular press, often refers to science versus completely non-science ways of forming an opinion or deciding policy. Meanwhile, science within a technical community, is subject to human error and manipulation, and relies on a reproducible result, as well as peer-review, to find answers to conflicting claims. There are certainly non-Science ways of for…

Yes, thank you for saying this. I feel like, often, people use science as some ethereal being. It is a process that has to be used. If that process isn't happening it isn't science. That doesn't mean everything has to be science, but things that are science should be held to a certain standard but should also be taken with the extra weight if that standard is met.

Re: The Irreproducibility Crisis of Modern Science

#44

Post by DanAndersen » Tue, Apr 17, 2018, 3:48 PM UTC

unknown_apostle wrote: post #24

ptero wrote: post #15

Earlier quoted context omitted.

The problem with this approach is that once you toss science there is not much left to inform policy decisions (religion? profit? personal views?). One less drastic solution may be to require a full disclosure of the process for all experimental results (to allow reproduction) and place much higher value in funding decisions on reproducibility. One particularly egregious approach reproducibility requirement can addre…

Heuristics: use what has worked in the past and keeps working well enough today? You don't need to fully understand why it works. Aka "conservatism" in the broadest sense.

Exactly. Traditionalism isn't perfect, but it acts as a sort of long-term experimentation. That which has worked over long periods of time can be a good default to stick with. Environments do change, but the nature of humanity and society also has constants over time that are important to consider.

Re: The Irreproducibility Crisis of Modern Science

#45

Post by gus_massa » Tue, Apr 17, 2018, 3:49 PM UTC

rplst8 wrote: post #7

Retric wrote: post #3

Being reproducible is only critically important if people treat individual studies as meaningful. That IMO is a far more dangerous stance. Any study can have hidden flaws, none should be trusted without some form of replication.

How often is a study even repeated in the course of normal scientific research? I think most studies are focused on expanding the research and therefore knowledge about the science being studied. Accepting previous studies as fact is dangerous and could easily lead to a house of cards scenario. I think this is especially true in the very specific, niche studies that are common in today's highly competitive graduate s…

There are a lot of "new" articles that are just a small tweak of a previous article with a lot of hype to make it appear as a groundbreaking result. So there is a lot of implicit informal reproduction.

You can't trust blindly the previous studies because they may be wrong, or have hidden assumptions. A well known case is the use of a control group. If you want to study a new drug, you must compare the results of the patients with a similar group that is taking the previous drug and/or a group that is taking no drug. You can't just trust that the cure rate of the previous study is 23.573% and compare it with your results. (And there are some trick to get more reliable results, like using a placebo instead of no drug, and doing a double blind study.)

Re: The Irreproducibility Crisis of Modern Science

#46

Post by emodendroket » Tue, Apr 17, 2018, 3:49 PM UTC

quantumwoke wrote: post #39

emodendroket wrote: post #35

Earlier quoted context omitted.

Well, not really.

From the HN guidelines: >Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.

I just don't see what relationship the second thought has with the first.

Re: The Irreproducibility Crisis of Modern Science

#47

Post by tonystubblebine » Tue, Apr 17, 2018, 3:49 PM UTC

We did an article trying to help people be better critical thinkers when they hear about psychology research. I thought the author did a good job getting into how these studies go wrong and also the types of magical results you should be suspicious of. https://medium.com/@jhreha/most-psychology-research-is-bs-73...

Re: The Irreproducibility Crisis of Modern Science

#48

Post by merpnderp » Tue, Apr 17, 2018, 3:50 PM UTC

scblock wrote: post #21

genericone wrote: post #14

Earlier quoted context omitted.

Racial 'science' was used by the Nazi's to justify antisemitism, eugenics, and everything they did beyond that. The scientific method is important to be used, but don't trust something just because another person believes it to be science. You need the hypothesis and the verification method. Others (all of us) have the obligation to challenge both the hypothesis as well as the verification methods.

So, 'science' not science? Got it. That was a pretty quick Godwining, by the way.

So you'd ignore one of the most horrific and important examples of widely accepted science going wrong because of an internet meme?

Re: The Irreproducibility Crisis of Modern Science

#49

Post by emodendroket » Tue, Apr 17, 2018, 3:51 PM UTC

m-watson wrote: post #34

rplst8 wrote: post #4

This is the exact reason I don't believe science should be used blindly to affect government policy. It flies in the face of democracy and is subject to falsehoods just like any other human endeavor.

I agree it shouldn't be used blindly but that seems like the extreme. In fact the science side of things would say don't use anything blindly. Science isn't some infallible entity it is a process that is supposed to be self correcting over periods of time. That includes falsehoods from human bias and input. It doesn't always work and it is a slow process but in science's essence it is supposed to be minimizing those…

Of course science can and should be used to inform government policy, but that's a bit removed from technocracy, which I think is what the OP means to describe.

Re: The Irreproducibility Crisis of Modern Science

#50

Post by jbob2000 » Tue, Apr 17, 2018, 3:51 PM UTC

lotsofpulp wrote: post #36

rplst8 wrote: post #7

Earlier quoted context omitted.

How often is a study even repeated in the course of normal scientific research? I think most studies are focused on expanding the research and therefore knowledge about the science being studied. Accepting previous studies as fact is dangerous and could easily lead to a house of cards scenario. I think this is especially true in the very specific, niche studies that are common in today's highly competitive graduate s…

That is why, as a (global) society, we should spend our resources in making sure the data we have is accurate by funding research and repeated studies, and that data should be open to all to verify and build upon.

I think we're passed the point where studies can be reproduced effectively. I actually think the scientific model is falling apart. We've discovered the easy things, and those studies were easy to reproduce. We built a model of science around the low hanging fruit.

How are you going to reproduce a study about a cancer drug? Synthesizing the drug itself is a huge endeavor, never mind finding a group of participants with the exact cancer you need. How would you reproduce the Large Hadron Collider experiments? You'd need to rebuild one of the most technically challenging, most expensive scientific instruments ever made.