Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Irreproducibility Crisis of Modern Science

nas.org

Post reply on HN
Print view View original

161–170 of 265 posts

Re: The Irreproducibility Crisis of Modern Science

#161

Post by geofft »

JumpCrisscross wrote: post #136
danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

> There's just a very big gap between how science should be done and what capital owners want from science It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speakin…

It certainly seems true that the owners of capital are suffering from the reproducibility crisis as much as anyone is. What 'danharaj is arguing, I think, is that the owners of capital are not well-placed to solve it.

The very nature of capitalism is that people who produce monetizable results should be rewarded for that work, and that people who don't shouldn't. This is fundamentally incompatible with doing science in a reproducible way: paying for positive results and not negative ones gets you p-hacking as an emergent effect. Of course the capitalists don't want to pay for non-reproducible positive results, hence the firestorm. But they more strongly don't want to pay for negative results.

The capital owners in the science industries will probably end up doing something sort of like what the capital owners in the software engineering industries have done: begrudgingly accepted that a non-capitalist approach (open source development across multiple companies without proprietary advantage) produces better results for everyone, and figured out a way to kind of support it. Much like the capital owners in the software engineering industries, they still won't be very effective at it. Their nature will be to still try to extract proprietary advantage when possible, to cut costs per the tragedy of the comments, etc., and so this model will only work well when funded by companies rich enough that they can allow themselves not to show an immediate return on their sponsorship.

Re: The Irreproducibility Crisis of Modern Science

#162

Post by danharaj »

JumpCrisscross wrote: post #153
danharaj wrote: post #148

Earlier quoted context omitted.

Even if that's true, that only science that can have a profit motive slapped on it can be done effectively in the current system is a problem. If only such science were done, then we would have different crises of science to deal with because profit seeking is at odds with much science, in particular basic research. It's also quite obviously the case that profit seeking research has incentives to be bad science in ot…

> It's also quite obviously the case that profit seeking research has incentives to be bad science in other ways I agree. My point is the Marxist framework is a bad one for modern science. We have endowed academics and ones working for private institutions and getting grants from public bodies run by scientists. Privately-financed biotech companies founded by academics taking moon shots and getting acquired by market…

> My broader point is the Marxian framework is a bad one for modern science.

I will concede that it's reasonable to think it's an incomplete and limited framework, but I have to take issue with calling it bad. For example, I think this starting point is more productive than the premises of this report.

> Privately-financed biotech companies founded by academics taking moon shots and getting acquired by marketing and distribution powerhouses. Even delineating who is "capital" and who is "labour" quickly becomes pedantic more than useful.

In this example it's plain to me that VCs are capital and entrepreneurs are labor. That's usually the game in such moonshots. Here we can analyze what sort of science is being done through this relationship through a Marxian lens too. VC's want to make big bets with potentially astronomical upside. That means only certain kinds of science will ever be done in this way. Yea, this kind of science is probably more replicable on average, but what about something like Theranos? This sort of science funding has its own problems with inefficient and incomplete allocation of resources.

I'm not a hardcore Marxist; I don't think you can reduce the world to one grand theory based on hot takes inspired by Hegel, but I think it's a more powerful tool than most care to admit.

Re: The Irreproducibility Crisis of Modern Science

#163

Post by [deleted] »

danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

[deleted]

Re: The Irreproducibility Crisis of Modern Science

#165

Post by pluto9 »

quantumwoke wrote: post #13

Recently I have been beginning to question university education. It seems that today's education system has mismatched priorities which may have an impact on the reproducibility crisis. You see, to come up with a fantastic new alloy, you don't have to be a genius. Or even very good. You have to be persistent, and clever enough to realize when you've found something interesting. You will need to understand your field…

> Train lots and lots of physicists, chemists, engineers, programmers, materials scientists, biologists, mathematicians. [...] Employ them on promising ideas, and on impossible ideas.

The scientifically minded are not the ones who will be deciding what's worthy of research. Politicians and bureaucrats will, and there are plenty of them who believe in all kinds of pseudo-scientific nonsense. How does your system prevent them from shoveling tax dollars into research on homeopathy and young-Earth creationism? And how does it prevent them from using funding to incentivize scientists to find the "correct" results?

To phrase my question more generally: I would like to know how your system would avoid making science merely a tool of political interests instead of a tool of moneyed interests.

Re: The Irreproducibility Crisis of Modern Science

#166

Post by danharaj »

quantumofmalice wrote: post #157
danharaj wrote: post #148

Earlier quoted context omitted.

Even if that's true, that only science that can have a profit motive slapped on it can be done effectively in the current system is a problem. If only such science were done, then we would have different crises of science to deal with because profit seeking is at odds with much science, in particular basic research. It's also quite obviously the case that profit seeking research has incentives to be bad science in ot…

Your original thesis just got completely destroyed and now you are bring up unrelated speculative issues not explored in the article. Just admit that the marxist analysis doesn't hold much water in this case. That doesn't mean it is never useful, just that you might need to think a bit before applying it.

> Your original thesis just got completely destroyed

You're really set off by the word M__x, aren't you?

Re: The Irreproducibility Crisis of Modern Science

#167

Post by thaumaturgy »

JumpCrisscross wrote: post #136
danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

> There's just a very big gap between how science should be done and what capital owners want from science It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speakin…

I'm pretty sure this is not accurate.

Issues with reproducibility have been discovered independently in a number of different scientific disciplines going back to the 1990s at least. This was then covered in a New Yorker article by Jonah Lehrer, published in 2010 [1]. It included the phrase "decline effect", coined by a researcher studying ESP in the 1930s.

A great deal of the research described in that article is academic in nature. The article also alludes to the contributory influence of funding as one of the causes of this phenomenon:

"The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for." ... "In recent years, publication bias has mostly been seen as a problem for clinical trials, since pharmaceutical companies are less interested in publishing results that aren’t favorable. But it’s becoming increasingly clear that publication bias also produces major distortions in fields without large corporate incentives, such as psychology and ecology."

[1]: https://www.newyorker.com/magazine/2010/12/13/the-truth-wear...

Re: The Irreproducibility Crisis of Modern Science

#168

Post by PeterisP »

SubiculumCode wrote: post #151
scottfr wrote: post #65

Earlier quoted context omitted.

I'm not sure I really agree with this, because in science studies are biased towards aligning with the results of previous studies. This can be due to many factors: authors' expectations, journals like to publish positive over negative results, peer reviewers will look more critically (in an analytical sense) at work that disagrees with accepted research, etc.... It can be very hard for many reasons to stand up and s…

The argument is not that all is perfect in science. It is not. The argument is that strong replication for each study is not required for progress. Back to my analogy to the algorithms like Adaboost. Why ever aggregate over a a bunch of weak classifiers over a single strong one? Well, please correct me if I am wrong (machine learning is not my field), but the primary advantage is computational cost. Sometimes, for th…

The ensemble advantage is completely the other way around - we might choose to use a large ensemble of classifiers because they can get slightly better results than the best single strong classifier we can make; and we might choose not to use an ensemble because of computational cost reasons, especially because you train a model once but infer forever (and likely on more limited hardware) and inference for a hundred classifiers takes a hundred times more computing power even if you skimped on training length. A good example is the Netflix Prize, where the best accuracy was achieved by a large ensemble, but the practical implementation afterwards chose a slightly less accurate non-ensemble approach for performance reasons.

What you describe sometimes happens in a tradeoff between very different ML models (e.g. an ensemble of decision trees versus a single deep neural network has the properties you describe), but within any single paradigm (an ensemble of neural networks versus one NN with more training time) it's the other way around.

Re: The Irreproducibility Crisis of Modern Science

#169

Post by leg100 »

JumpCrisscross wrote: post #136
danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

> There's just a very big gap between how science should be done and what capital owners want from science It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speakin…

Academia too is subject to the laws of capitalism. It is not some siloed-off eden of labour working for itself. Anyone working in academia can attest to the same pressures you see in the free market, whether it be short termism or maddening bureaucracy.

Re: The Irreproducibility Crisis of Modern Science

#170

Post by trhway »

cfadvan wrote: post #145
quantumofmalice wrote: post #140

Earlier quoted context omitted.

The reproducibility crisis is most severe in the social sciences. Hard sciences like physics are on much firmer ground, and conflating the two is clownish. Social science research is funded almost entirely by the government[1]. It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination. In fact, much of it is du…

Maybe none of this ideological bullshit is the real issue? Maybe the major problem is that hard vs soft sciences are subject to different experimental controls and confounding factors, not to mention standards of rigor. If you claim to discover a new particle your n will be in the billions, and a whole community will analyze your results with a fine comb. Meanwhile if you make a claim about implicit bias or the nutri…

>Social sciences are hardly even science from the perspective of a chemist or physicist

A chemist or a physicist still have hard time dealing with particle assemblies of say 100-100000 particles. For several last centuries they learnt to deal with [aggregate properties of] assemblies on the scale of 10e20+ particles. Only during last century they came to pretty good understanding of 1 particle.

In social sciences this is Facebook and Cambridge Analytica (ie. they are real social science today, not some professor in an office). This is the scale where aggregate properties appear, and they start to be able to drill down from it.

Post reply on HN