Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Irreproducibility Crisis of Modern Science

nas.org

Post reply on HN
Print view View original

131–140 of 265 posts

Re: The Irreproducibility Crisis of Modern Science

#131

Post by lotsofpulp »

foldr wrote: post #98
lotsofpulp wrote: post #67

Earlier quoted context omitted.

Science didn't go wrong. The data was bad, the processes were bad, there was no replication. How was that science?

>The data was bad, the processes were bad, there was no replication. People are really quick to say this, but how many people have actually studied the 'racial science' literature of the 20s and 30s and figured out to what extent it meets the methodological standards of today? Yes, we know that they drew incorrect (and reprehensible) conclusions. It doesn't necessarily follow that their methods were any worse (episte…

Studies about people in general are of poor quality, due to difficult to define metrics. Even today, psychology and sociology studies aren't rigorous for the simple reason that it's hard to measure these things. Biological studies are hard too, it's hard to suss out the confounding variables and mechanisms of bodily functions.

Just the premise of the "racial science" studies would render them invalid, how could they even have controlled for the myriad variables that we have no idea how to control today?

Re: The Irreproducibility Crisis of Modern Science

#132

Post by haZard_OS »

Waterluvian wrote: post #90
lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

We need a journal that exclusively publishes papers with results that were totally contrary to the hypothesis. So long as the researchers aren't outright inept or fraudulent, what's there to be ashamed about? Those would be some of the more interesting papers to read, in my opinion.

There are several. To offer but one:

http://www.jnr-eeb.org/index.php/jnr/index

I would actually prefer that each and every journal devote at least half of their space to negative results. I have seen some sublime experimental setups that produced negative results. I can't begin to imagine how instructive it would be to see more such experiments.

Re: The Irreproducibility Crisis of Modern Science

#133

Post by lisper »

eli_gottlieb wrote: post #125
lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

>The end-product of science is not truth, it is explanations of observations. That sounds a whole lot like truth to me, or at least, like any sane construal of truth.

That depends on what you mean by "truth". Newtonian mechanics says that gravity is pulling you down towards the surface of the earth. Is that "true"? It has tremendous explanatory power, but I would argue that it is not actually true. The truth is (as far as we can tell at the moment) that the surface of the earth is pushing you up and accelerating you through curved space-time.

Re: The Irreproducibility Crisis of Modern Science

#134

Post by AstralStorm »

lliamander wrote: post #126
lmm wrote: post #12

Earlier quoted context omitted.

If we don't use science to make decisions, what are we supposed to use? Gut instinct? Of course science is imperfect, but it tends towards truth over time, whereas dogma and populism have no way of correcting mistakes that isn't just as likely to introduce new ones.

> If we don't use science to make decisions, what are we supposed to use? Gut instinct? I thought the parent comment made it clear: the alternative we are supposed to use is "democracy". That is, the people who are ruled should determine the rules that are imposed on them. Now, democracy does not preclude the use of science to make decisions. The voters may rely on science to inform their decisions (when the science…

How do you know democratizing decisions results in minimax optimal decisions?

I would expect the result to be strongly biased due to societal norms, much less direct manipulation.

Suppose a majority wants strong social programmes and basic income. The tiny problem is the economy is nowhere near big enough to afford it. Satisficing (!) executive branch puts some of the policies starving innovation and education budgets. Causing long term harm and stagnation.

Where is the minimax decision in here? Over what timeframe?

(P.s. I could've as easily picked liberal, fascist, military or conservative example. This one is relatively easy to understand.)

Re: The Irreproducibility Crisis of Modern Science

#135

Post by lisper »

3JPLW wrote: post #114
lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the…

> > at least one result in 20 will be due to chance

> That's wildly overly pessimistic

Why? You yourself said:

> The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance)

How is that different from what I said? 5% is just another way of writing 1 in 20.

Re: The Irreproducibility Crisis of Modern Science

#136

Post by JumpCrisscross »

danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

> There's just a very big gap between how science should be done and what capital owners want from science

It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speaking, than academia.

[1] https://www.wired.com/2017/01/fighting-cancers-crisis-confid...

Re: The Irreproducibility Crisis of Modern Science

#137

Post by arafa »

hnuser1234 wrote: post #82
arafa wrote: post #73

What I'd like to know is if the "irreproducibility crisis" is really some combination of "sample size was too small" and "effect size was too small". When I went through a lot of these studies myself and saw the ones that don't reproduce, I saw this theme over and over. "P-hacking" is less of a concern to me when the effect is real and widespread. It's so bad now that for any article/study, I look at the sample size…

I'm not sure this captures the whole problem. Sure, huge sample sizes are great, but look at the current top comment by lisper - we have gained a boatload (that's a scientific term) of insight from single events. LIGO is currently rocking a sample size of 7 but I doubt you ignored it.

Of course there are exceptions and what I said doesn't capture all cases (like with massive fixed costs or unique kinds of studies), you're right. I find it's a pretty good rule of thumb, though.

Re: The Irreproducibility Crisis of Modern Science

#138

Post by jonmc12 »

danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

You might enjoy the book, The Fellowship: Gilbert, Bacon, Harvey, Wren, Newton, and the Story of a Scientific Revolution.

The book is set around 1660, as the center of the scientific universe shifted from Italy to England. Interestingly, the formation of the Royal Society occurred in the midst of the English Civil war. Not only economic forces, but military forces and political forces had a dramatic impact on shaping what became the formalization of of the scientific method, and peer review process.

Consider that the very existence of Oxford (and other universities) hung on the arbitrary conceptions of military generals and the political implications of their decisions.

Re: The Irreproducibility Crisis of Modern Science

#139

Post by bbddg »

rplst8 wrote: post #4

This is the exact reason I don't believe science should be used blindly to affect government policy. It flies in the face of democracy and is subject to falsehoods just like any other human endeavor.

This quote from Einstein on the subject sums up my thoughts on this as well.

"Science, however, cannot create ends and, even less, instill them in human beings; science, at most, can supply the means by which to attain certain ends. But the ends themselves are conceived by personalities with lofty ethical ideals and—if these ends are not stillborn, but vital and vigorous—are adopted and carried forward by those many human beings who, half unconsciously, determine the slow evolution of society.

For these reasons, we should be on our guard not to overestimate science and scientific methods when it is a question of human problems; and we should not assume that experts are the only ones who have a right to express themselves on questions affecting the organization of society."

https://monthlyreview.org/2009/05/01/why-socialism/

Re: The Irreproducibility Crisis of Modern Science

#140

Post by quantumofmalice »

danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

The reproducibility crisis is most severe in the social sciences. Hard sciences like physics are on much firmer ground, and conflating the two is clownish. Social science research is funded almost entirely by the government[1].

It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination. In fact, much of it is due to cultural marxist insistence on outcomes conforming to political correctness, as well as uncritical acceptance of politically conforming work. See the ongoing fight against the idea of a high genetic basis for adult IQ.

All of this is obvious enough: capitalists aren't interested in non-reproducable results unless it is obviating them from blame for a given externality. But that sort of research is a small fraction of overall research (and should absolutely be done by impartial third parties.)

[1] https://en.wikipedia.org/wiki/Funding_of_science

Post reply on HN