The Irreproducibility Crisis of Modern Science • Hacker News Board

Re: The Irreproducibility Crisis of Modern Science

#81

Post by Bograff » Tue, Apr 17, 2018, 4:24 PM UTC

m-watson wrote: post #43

mistrial9 wrote: post #26

Some of the comments miss an important distinction.. Science in the popular press, often refers to science versus completely non-science ways of forming an opinion or deciding policy. Meanwhile, science within a technical community, is subject to human error and manipulation, and relies on a reproducible result, as well as peer-review, to find answers to conflicting claims. There are certainly non-Science ways of for…

Yes, thank you for saying this. I feel like, often, people use science as some ethereal being. It is a process that has to be used. If that process isn't happening it isn't science. That doesn't mean everything has to be science, but things that are science should be held to a certain standard but should also be taken with the extra weight if that standard is met.

It's important to make the distinction between Pure Science and anything on the sliding scale from technical description to ELI5. There are levels to it, you and I know.

Re: The Irreproducibility Crisis of Modern Science

#82

Post by hnuser1234 » Tue, Apr 17, 2018, 4:25 PM UTC

arafa wrote: post #73

What I'd like to know is if the "irreproducibility crisis" is really some combination of "sample size was too small" and "effect size was too small". When I went through a lot of these studies myself and saw the ones that don't reproduce, I saw this theme over and over. "P-hacking" is less of a concern to me when the effect is real and widespread. It's so bad now that for any article/study, I look at the sample size…

I'm not sure this captures the whole problem. Sure, huge sample sizes are great, but look at the current top comment by lisper - we have gained a boatload (that's a scientific term) of insight from single events. LIGO is currently rocking a sample size of 7 but I doubt you ignored it.

Re: The Irreproducibility Crisis of Modern Science

#83

Post by thedonkeycometh » Tue, Apr 17, 2018, 4:27 PM UTC

quantumwoke wrote: post #13

Recently I have been beginning to question university education. It seems that today's education system has mismatched priorities which may have an impact on the reproducibility crisis. You see, to come up with a fantastic new alloy, you don't have to be a genius. Or even very good. You have to be persistent, and clever enough to realize when you've found something interesting. You will need to understand your field…

Intelligence is such a spectrum that if you select individuals on traits you rot through lack of diversity. A breakthrough will generally come from someone with feet in utterly different fields.

Concentrating on grades in STEM poisons the well of raw thinking that science needs to look at everything in unique ways.

The only way that we can move forward is that everyone moves forward.

Re: The Irreproducibility Crisis of Modern Science

#84

Post by toufka » Tue, Apr 17, 2018, 4:27 PM UTC

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

I'd add a last point that a scientific paper (or a particular experimental result) is not sufficient for 'science'. Citing a paper as either a truth or discovery is a disingenuous. The endeavour of science gets at a kind of consensus by having lots of papers, lots of experiments, lots of additional layers of experiments build on each other. And yes it's annoying to realize a given paper is misleading, has errors, or is even wrong - but that annoyance and workarounds are part of the scientific process. It's actually fairly rare for an experiment or paper to be strong enough to stand completely on its own as a singular, thorough, and completed discovery. If you've been part of the scientific field you know that it yet takes years after even a good publication for the knowledge to accrete to the greater scientific body.

Some of the best advice I got was to spend a lot more time than one might want to really come up with an experimental design that avoided a statistical outcome and instead probed at an either/or mechanism. It's not always possible, but when it is, it can be a powerful reply to fighting statistical battles (see the Michelson-Morely experiment for a classic example).

Further careers, families, reputations, humans are also part of that scientific process, which together with the above require and admit some deviation from perfect efficiency. Which we as a society should demand be okay.

Re: The Irreproducibility Crisis of Modern Science

#85

Post by jnordwick » Tue, Apr 17, 2018, 4:28 PM UTC

Afforess wrote: post #28

HarryHirsch wrote: post #10

Earlier quoted context omitted.

It's a position paper from a conservative think tank and needs to be treated as such.

Ad hominem has never been a valid reason to disbelieve something. I quite enjoyed reading the report - it includes XKCD references and everything.

That you're being voted down shows that HN has really changed.

(I blame all the overtly political articles that I believe have changed the tenor of the site, and politics has infected every article now. Thanks @dang!)

Re: The Irreproducibility Crisis of Modern Science

#86

Post by buvanshak » Tue, Apr 17, 2018, 4:28 PM UTC

timtadh wrote: post #62

In response to several threads here: it is important to distinguish when scientists are self critical vs. when non-scientists are critical of the scientific method. For instance, there is a long history of scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. That work is sometimes used by non-scientists who question the overall scientific meth…

>scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor.

I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process.

>It does not in itself show that science is inherently untrustworthy.

I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong.

Then there are things like publication bias, that takes this to a whole new level. Things like that means that a biased body of journals can project any consensus that it favors just by selecting studies that fit its narrative. The inherent issues with statistics means that you can find studies that shows any possible outcome.

Re: The Irreproducibility Crisis of Modern Science

#87

Post by unabst » Tue, Apr 17, 2018, 4:31 PM UTC

rplst8 wrote: post #4

This is the exact reason I don't believe science should be used blindly to affect government policy. It flies in the face of democracy and is subject to falsehoods just like any other human endeavor.

Of course science flies in the face of democracy. Democracy gives all the power to individual subjective opinion. Hence the battle against emotional bias, fake news, and propaganda tactics which all work wonders in a democracy. This battle is fought with objective facts, which are only attainable by science.

Science is the only tool specifically designed to weed out falsehoods and reach an objective consensus based on how nature works.

How is this "subject to falsehoods just like any other human endeavor"?

There is no substitute for science.

We all seem to agree that democracy only works when the population is informed. Well, are we going to inform ourselves with the beliefs and tweets of those which we grant moral authority? Or science?

Re: The Irreproducibility Crisis of Modern Science

#88

Post by monocasa » Tue, Apr 17, 2018, 4:32 PM UTC

merpnderp wrote: post #54

monocasa wrote: post #41

Earlier quoted context omitted.

The issue is that it's not easy for the populace to tell the difference between 'science' and science in a lot of cases. You see this on all parts of the political spectrums. Soviet Lysenkoism, left wing anti vax crap, and neo liberal austerity. EDIT: Krugman had this to say about the R&R austerity paper "What the Reinhart-Rogoff affair shows is the extent to which austerity has been sold on false pretenses. For thre…

The same Krugman who pronounced the markets would never recover from Trump's election or that since we didn't pass a large enough stimulus we were doomed to low GDP growth rates forever?

Don't just listen to Krugman, read how monetary policy for the better part of a decade was based on a paper that wasn't peer reviewed, but was held up as irrefutable 'science'. Politicians latched on to it because it told them what they wanted to hear.

Re: The Irreproducibility Crisis of Modern Science

#89

Post by jnordwick » Tue, Apr 17, 2018, 4:36 PM UTC

gringoDan wrote: post #69

One of Slate Star Codex's top all-time articles discusses this very issue. Highly recommend: http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...

> On the meta-level, you’re studying some phenomenon and you get some positive findings. That doesn’t tell you much until you take some other researchers who are studying a phenomenon you know doesn’t exist – but which they themselves believe in – and see how many of them get positive findings. That number tells you how many studies will discover positive results whether the phenomenon is real or not. Unless studies of the real phenomenon do significantly better than studies of the placebo phenomenon, you haven’t found anything.

This is such an astute observation that reproduction studies can be used to find a placebo control group for bad scuence. Parapsychology was one such group, but now we can find many others. Brilliant.

Re: The Irreproducibility Crisis of Modern Science

#90

Post by Waterluvian » Tue, Apr 17, 2018, 4:37 PM UTC

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

We need a journal that exclusively publishes papers with results that were totally contrary to the hypothesis.

So long as the researchers aren't outright inept or fraudulent, what's there to be ashamed about? Those would be some of the more interesting papers to read, in my opinion.