The Irreproducibility Crisis of Modern Science • Hacker News Board

Re: The Irreproducibility Crisis of Modern Science

#151

Post by SubiculumCode » Tue, Apr 17, 2018, 5:45 PM UTC

scottfr wrote: post #65

SubiculumCode wrote: post #51

We know that weak classifiers can be bagged to produce a strong classifier al la Adaboost. Each study is a weak classifier and would have a 'reproducibility crisis' if retested on new data. However after lots of studies of similar phenomena, a strong classifier emerges. In the field, we call this converging lines of evidence.

I'm not sure I really agree with this, because in science studies are biased towards aligning with the results of previous studies. This can be due to many factors: authors' expectations, journals like to publish positive over negative results, peer reviewers will look more critically (in an analytical sense) at work that disagrees with accepted research, etc.... It can be very hard for many reasons to stand up and s…

The argument is not that all is perfect in science. It is not. The argument is that strong replication for each study is not required for progress. Back to my analogy to the algorithms like Adaboost. Why ever aggregate over a a bunch of weak classifiers over a single strong one? Well, please correct me if I am wrong (machine learning is not my field), but the primary advantage is computational cost. Sometimes, for the same overall classification performance, training hundreds or thousands of weak classifiers requires fewer computations than training the strong classifier.

To me it brings up an interesting point. If we view experiments as classifiers, then how would a machine learning expert set science policy and practice?

Also viewed this way, it makes me wonder about p-hacking, which increases sensitivity while increasing false positives. Since negative results are not generally reported, I wonder whether p-hacking diminishes replicability at the study level for efficiency at the aggregate level. This is an empirical question, as ethically it is of course a dubious practice under current understanding.

Re: The Irreproducibility Crisis of Modern Science

#152

Post by quantumofmalice » Tue, Apr 17, 2018, 5:48 PM UTC

cfadvan wrote: post #145

quantumofmalice wrote: post #140

Earlier quoted context omitted.

The reproducibility crisis is most severe in the social sciences. Hard sciences like physics are on much firmer ground, and conflating the two is clownish. Social science research is funded almost entirely by the government[1]. It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination. In fact, much of it is du…

Maybe none of this ideological bullshit is the real issue? Maybe the major problem is that hard vs soft sciences are subject to different experimental controls and confounding factors, not to mention standards of rigor. If you claim to discover a new particle your n will be in the billions, and a whole community will analyze your results with a fine comb. Meanwhile if you make a claim about implicit bias or the nutri…

> It’d be cool if your and the other poster’s ideological spat could try to not use science as the battleground, or at least eliminate more likely issues before going there.

I don't disagree. A big part of the problem is the social sciences claiming the mantel of science (often just statistics layered over tiny samples, as you point out), in order to imply some sort of impartiality and precision on par with the hard (i.e. real) sciences.

Unfortunately, and I say this sincerely, I believe the ideological nature of the social sciences since the 60s has been responsible for a huge amount of uncritical acceptance of bad (false) work as well as preventing a lot of unpopular good (true) work. I don't know how to deal with that fact other than stating so forthrightly, which, unfortunately from the perspective of science, carries necessary ideological freight with it.

Re: The Irreproducibility Crisis of Modern Science

#153

Post by JumpCrisscross » Tue, Apr 17, 2018, 5:49 PM UTC

danharaj wrote: post #148

JumpCrisscross wrote: post #136

Earlier quoted context omitted.

> There's just a very big gap between how science should be done and what capital owners want from science It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speakin…

Even if that's true, that only science that can have a profit motive slapped on it can be done effectively in the current system is a problem. If only such science were done, then we would have different crises of science to deal with because profit seeking is at odds with much science, in particular basic research. It's also quite obviously the case that profit seeking research has incentives to be bad science in ot…

> It's also quite obviously the case that profit seeking research has incentives to be bad science in other ways

I agree. My point is the Marxist framework is a bad one for modern science.

We have endowed academics and ones working for private institutions and getting grants from public bodies run by scientists. Privately-financed biotech companies founded by academics taking moon shots and getting acquired by marketing and distribution powerhouses. Even delineating who is "capital" and who is "labour" quickly becomes pedantic more than useful.

Re: The Irreproducibility Crisis of Modern Science

#154

Post by danharaj » Tue, Apr 17, 2018, 5:50 PM UTC

quantumofmalice wrote: post #140

danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

The reproducibility crisis is most severe in the social sciences. Hard sciences like physics are on much firmer ground, and conflating the two is clownish. Social science research is funded almost entirely by the government[1]. It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination. In fact, much of it is du…

> Hard sciences like physics are on much firmer ground, and conflating the two is clownish.

https://en.wikipedia.org/wiki/Bogdanov_affair

> It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination.

You understand literally nothing about Marxism. For example, "capitalist" isn't a moral designation, it is a description of an economic agent's role in an economic relationship. Capital owners have access to resources and they give permission to laborers to use those resources. That's it.

> cultural marxist

You can't be serious.

> See the ongoing fight against the idea of a high genetic basis for adult IQ.

This plays out more in the public sphere than in scientific communities. It's also the case that a few popular intellectuals try to sell psychometric science as saying more than it actually does about how social policies should be designed. This, again, happens moreso outside of scientific circles.

Re: The Irreproducibility Crisis of Modern Science

#155

Post by lliamander » Tue, Apr 17, 2018, 5:50 PM UTC

AstralStorm wrote: post #134

lliamander wrote: post #126

Earlier quoted context omitted.

> If we don't use science to make decisions, what are we supposed to use? Gut instinct? I thought the parent comment made it clear: the alternative we are supposed to use is "democracy". That is, the people who are ruled should determine the rules that are imposed on them. Now, democracy does not preclude the use of science to make decisions. The voters may rely on science to inform their decisions (when the science…

How do you know democratizing decisions results in minimax optimal decisions? I would expect the result to be strongly biased due to societal norms, much less direct manipulation. Suppose a majority wants strong social programmes and basic income. The tiny problem is the economy is nowhere near big enough to afford it. Satisficing (!) executive branch puts some of the policies starving innovation and education budget…

I should clarify that there were two parts to my answer: first was answering your question "on behalf of" the parent poster. It seemed their position was to "let people vote on it". I was somewhat defending that position, but more just clarifying that democracy wasn't necessarily in conflict with using science to make decisions.

The second part of my response was to your framing of "science vs. gut-instinct". My point is that sometimes ignorance is thrust upon us by the complexity of the circumstances. In that case, my point was that decentralized policy making (separate from any notion of voting) was essential to reducing the worst case.

In other words, I am not claiming that democratization "results in minimax optimal decisions". I am claiming that decentralized governance structures do "result in minimax optimal decisions". Does that change your question?

edit: fixed spelling

Re: The Irreproducibility Crisis of Modern Science

#156

Post by PeterisP » Tue, Apr 17, 2018, 5:50 PM UTC

3JPLW wrote: post #114

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the…

"p-hacking" is also a thing, it's a reasonable possibility that they tried 20 false but plausible null hypothesis and published the one that got a significant result.

If there's no pre-registration, my prior of a paper actually being true would be much lower than statistics would say simply because the incentives are aligned so that there's a motivation to exaggerate significance.

Re: The Irreproducibility Crisis of Modern Science

#157

Post by quantumofmalice » Tue, Apr 17, 2018, 5:52 PM UTC

danharaj wrote: post #148

JumpCrisscross wrote: post #136

Earlier quoted context omitted.

> There's just a very big gap between how science should be done and what capital owners want from science It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speakin…

Even if that's true, that only science that can have a profit motive slapped on it can be done effectively in the current system is a problem. If only such science were done, then we would have different crises of science to deal with because profit seeking is at odds with much science, in particular basic research. It's also quite obviously the case that profit seeking research has incentives to be bad science in ot…

Your original thesis just got completely destroyed and now you are bring up unrelated speculative issues not explored in the article.

Just admit that the marxist analysis doesn't hold much water in this case. That doesn't mean it is never useful, just that you might need to think a bit before applying it.

Re: The Irreproducibility Crisis of Modern Science

#158

Post by JackFr » Tue, Apr 17, 2018, 5:53 PM UTC

TangoTrotFox wrote: post #99

danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just comp…

> For whatever reason there seems to be extremely little interest in the private direct funding of science.

Too much income equality. You can't have Rockefeller University without Rockefeller.

Re: The Irreproducibility Crisis of Modern Science

#159

Post by pas » Tue, Apr 17, 2018, 5:54 PM UTC

lordnacho wrote: post #123

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

> 3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science. Yes and no. I think you have to have predictions in there somewhere. That's wh…

Shut up and calculate means that "sure, that's a nice toy model, but show that it reproduces the standard model", and also "sure, nobody really understands why the model is that way, but it works, so shut up and use it".

And also, shut up and come up with something falsifiable (in our lifetime).

Re: The Irreproducibility Crisis of Modern Science

#160

Post by enugu » Tue, Apr 17, 2018, 5:56 PM UTC

quantumofmalice wrote: post #140

danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

The reproducibility crisis is most severe in the social sciences. Hard sciences like physics are on much firmer ground, and conflating the two is clownish. Social science research is funded almost entirely by the government[1]. It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination. In fact, much of it is du…

Pharmaceutical companies influence research according to [1] to gain support for their medicines.

Studying economic factors underlying social activity isnt by itself Marxist. This methodology is adopted in support of different political positions. In fact, many free-market economists also do this and get accused of 'economics imperialism' by social science faculty who want to study other factors.

Another point made in [1] is the blame isnt just on companies but people being influenced.

[1]https://medium.com/@drjasonfung/the-corruption-of-evidence-b...