The Irreproducibility Crisis of Modern Science • Hacker News Board

Re: The Irreproducibility Crisis of Modern Science

#121

Post by haZard_OS » Tue, Apr 17, 2018, 5:05 PM UTC

jbob2000 wrote: post #50

lotsofpulp wrote: post #36

Earlier quoted context omitted.

That is why, as a (global) society, we should spend our resources in making sure the data we have is accurate by funding research and repeated studies, and that data should be open to all to verify and build upon.

I think we're passed the point where studies can be reproduced effectively. I actually think the scientific model is falling apart. We've discovered the easy things, and those studies were easy to reproduce. We built a model of science around the low hanging fruit. How are you going to reproduce a study about a cancer drug? Synthesizing the drug itself is a huge endeavor, never mind finding a group of participants wi…

> I actually think the scientific model is falling apart.

  You must have a totally different definition of "the scientific model" than I do. I suspect (although I could be wrong) is that you have an oversimplified view of just what the scientific process is.

The scientific model/process is not a linear list of discrete steps moving from "Ask a question" to "Communicate results"; it is not a discrete and static set of criteria we apply to individual experiments or instrument output.

Berkeley University does a good job of providing visualizations on this topic:

http://undsci.berkeley.edu/article/0_0_0/howscienceworks_02

Re: The Irreproducibility Crisis of Modern Science

#122

Post by flamedoge » Tue, Apr 17, 2018, 5:06 PM UTC

There is the problem of conflating science with mathematics.

Re: The Irreproducibility Crisis of Modern Science

#123

Post by lordnacho » Tue, Apr 17, 2018, 5:06 PM UTC

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

> 3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science.

Yes and no. I think you have to have predictions in there somewhere. That's why your celestial mechanics is science: whatever the movement is, you were able to say what it was going to be beforehand, even if it only happened once.

As for explanation, that is not necessarily so simple, either. Sure, a lot of things seem to make more sense with a theory, but not every theory makes sense. A lot of stuff in fundamental physics is predicted by the equations, but in what sense are the equations explanations? "Shut up and calculate" comes to mind.

Re: The Irreproducibility Crisis of Modern Science

#124

Post by stevedonovan » Tue, Apr 17, 2018, 5:06 PM UTC

danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

While we're talking political economy, I was thinking that academia tends towards a degenerate form resembling subsistence agriculture: defending a small patch of turf and making just enough meaning to support a career. Even if the raw economic incentive is removed (e.g "publish or perish") there is sufficient ego investment to preserve this pattern.

Re: The Irreproducibility Crisis of Modern Science

#125

Post by eli_gottlieb » Tue, Apr 17, 2018, 5:07 PM UTC

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

>The end-product of science is not truth, it is explanations of observations.

That sounds a whole lot like truth to me, or at least, like any sane construal of truth.

Re: The Irreproducibility Crisis of Modern Science

#126

Post by lliamander » Tue, Apr 17, 2018, 5:07 PM UTC

lmm wrote: post #12

rplst8 wrote: post #4

This is the exact reason I don't believe science should be used blindly to affect government policy. It flies in the face of democracy and is subject to falsehoods just like any other human endeavor.

If we don't use science to make decisions, what are we supposed to use? Gut instinct? Of course science is imperfect, but it tends towards truth over time, whereas dogma and populism have no way of correcting mistakes that isn't just as likely to introduce new ones.

> If we don't use science to make decisions, what are we supposed to use? Gut instinct?

I thought the parent comment made it clear: the alternative we are supposed to use is "democracy". That is, the people who are ruled should determine the rules that are imposed on them.

Now, democracy does not preclude the use of science to make decisions. The voters may rely on science to inform their decisions (when the science is available). It does however preclude a rule by scientific experts/bureaucrats.

But I also think it's foolish to frame the decision as solely one of "science" vs. "gut instinct". The are many phenomena, especially when it comes to human society, that are too causally dense for us to sufficiently tease out the causal factors to make accurate predictions.

In those conditions, the goal should be to minimize the consequences of being wrong. We do that by decentralizing decisions to the greater extent possible.

Re: The Irreproducibility Crisis of Modern Science

#127

Post by AstralStorm » Tue, Apr 17, 2018, 5:08 PM UTC

DanAndersen wrote: post #44

unknown_apostle wrote: post #24

Earlier quoted context omitted.

Heuristics: use what has worked in the past and keeps working well enough today? You don't need to fully understand why it works. Aka "conservatism" in the broadest sense.

Exactly. Traditionalism isn't perfect, but it acts as a sort of long-term experimentation. That which has worked over long periods of time can be a good default to stick with. Environments do change, but the nature of humanity and society also has constants over time that are important to consider.

The problem is that there is no turning back. Especially considering nutrition, chemistry, material science.

Environment has changed enough - especially number of people - that attempting to use traditional methods en masse could be very risky.

(Much less in nutrition or medicine where we have made major gains on the average.)

Re: The Irreproducibility Crisis of Modern Science

#128

Post by m-watson » Tue, Apr 17, 2018, 5:10 PM UTC

Bograff wrote: post #81

m-watson wrote: post #43

Earlier quoted context omitted.

Yes, thank you for saying this. I feel like, often, people use science as some ethereal being. It is a process that has to be used. If that process isn't happening it isn't science. That doesn't mean everything has to be science, but things that are science should be held to a certain standard but should also be taken with the extra weight if that standard is met.

It's important to make the distinction between Pure Science and anything on the sliding scale from technical description to ELI5. There are levels to it, you and I know.

Oh absolutely there is a sliding scale on both the actual practice of science as well as the reporting. From the practice of it, I know as simple anecdote I have participated in conducting research in hard science (physics), softer hard science (biology), social science (behavioral and economic), as well as straight policy analysis with a "scientific approach" through data science. The findings from those are reported differently and have stronger or weaker words and conclusions. I think that is where the practice sliding scale comes in, is how the researcher decided to write up and report their findings and appropriately identifying their methodology and short-comings.

From the journalism side I think you end up with the problem the wide spread field of journalism has, you have to be read by your audience. That means something with a wider audience is probably going to be more on the ELI5 side which means both the reader and the journalist have to take any reported findings more lightly.

The sliding scale, I believe, comes in with the rigor something is reported. That is both in the practice and the journalistic side. More details can give a more granular understanding showing limitations but some times broad strokes are necessary to get some information out to a wider audience or more quickly.

Re: The Irreproducibility Crisis of Modern Science

#129

Post by AstralStorm » Tue, Apr 17, 2018, 5:12 PM UTC

tlb wrote: post #11

rplst8 wrote: post #4

This is the exact reason I don't believe science should be used blindly to affect government policy. It flies in the face of democracy and is subject to falsehoods just like any other human endeavor.

Who is the strongest proponent of using science blindly to affect government policy? What are their main arguments, and how do you disagree?

What do you mean by using science blindly?

Three is NO way to do that. There are ways to misuse scientific results. Premature generalisation, assumptions of safety. Cherry picking to support preconceptions. Attributing strong support to tentative and/or unchallenged results. And more...

Re: The Irreproducibility Crisis of Modern Science

#130

Post by lisper » Tue, Apr 17, 2018, 5:14 PM UTC

hpcjoe wrote: post #119

lisper wrote: post #94

Earlier quoted context omitted.

> You mean that papers are published with P I'm guessing you meant to say that no papers are published with P at least one paper in 20 will be "wrong". Then add self-suppression of negative results and the actual percentage of "wrong" papers becomes higher, potentially much higher if real positive results are rare. > When did this become a "standard"? I have no idea. You'll have a ask a historian.

Part of the mistake here is in interpreting the p value as being a measure of anything really meaningful. P It is not, in any way a measure of reality. It is a guess at how accurately you've modeled the system, how accurate your priors are, and so forth.

> P> It does mean that ... 1 in 20 would be explainable by the null hypothesis.

That seems like splitting a pretty fine hair to me, particularly since I haven't actually committed to any of the many possible interpretations of the word "wrong".