Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Just 11% of 53 cancer research papers were reproducible

nature.com

Post reply on HN
Print view View original

141–150 of 184 posts

Re: Just 11% of 53 cancer research papers were reproducible

#141

Post by kaoD »

tolmasky wrote: post #137
Avshalom wrote: post #125

Earlier quoted context omitted.

Huge swaths of astronomy are functionally unreproducible. We can argue about the math but many phenomena exist as a single example and/or that's basically static on our time scales. The best we can do is see if the math seems to produce similar looking structures when (sparsely) simulated.

Would there be a problem with just calling it something else other than science then? I don't see the need to bend definitions of words to account for inconvenient circumstances. I am a programmer and do hard stuff, I don't require people to call me a scientist. Mathematicians do hard stuff, they don't complain that they're not called scientists. Engineers, too, are not scientists. It's not pejorative, just a stateme…

I once had a very long and intense argument with a guy who was offended because I thought that I wasn't a scientist even though I studied Software Engineering (not even Computer Science).

Apparently people take that crap seriously.

Re: Just 11% of 53 cancer research papers were reproducible

#142

Post by lesterbuck »

epistasis wrote: post #122
spikels wrote: post #109

Earlier quoted context omitted.

If its not reproducible it is not science.

Define reproducible, in the terms of the precise actions that people take. Is it reproducible if the same scientist repeats the experiment in the same lab and gets the same results? Because that's the current bar of reproducibility, and that 89% that is not "reproducible" certainly passed that bar. What was being tested was a different lab, with different materials, trying to get the "same" results, for some definiti…

>Is it reproducible if the same scientist repeats the >experiment in the same lab and gets the same results? >Because that's the current bar of reproducibility, and that >89% that is not "reproducible" certainly passed that bar.

I don't follow you here. The above does not seem to be the current meaning of "reproducible":

http://en.wikipedia.org/wiki/Reproducibility

The same person doing the same experiment is repeatable, not reproducible. And I don't believe even the repeatable bar has been met, as very few projects have funding to do the same experiment twice.

The fact that a given investigator can "repeat" his experiment have very low weight among professional scientists, because we are all human. Irving Langmuir's famous talk about Pathological Science, and especially the sad story of N-rays, is a warning to every scientist.

http://en.wikipedia.org/wiki/Pathological_science

http://www.cs.princeton.edu/~ken/Langmuir/langB.htm#Nrays

Re: Just 11% of 53 cancer research papers were reproducible

#143

Post by nhebb »

Avshalom wrote: post #125
spikels wrote: post #109

Earlier quoted context omitted.

If its not reproducible it is not science.

Huge swaths of astronomy are functionally unreproducible. We can argue about the math but many phenomena exist as a single example and/or that's basically static on our time scales. The best we can do is see if the math seems to produce similar looking structures when (sparsely) simulated.

There's a difference between an observation and an experiment. Lab experiments should be reproducible. The fact that astronomical events are not reproducible does not make the study of them unscientific, but it also doesn't imply that lab experiments should be one time events.

Re: Just 11% of 53 cancer research papers were reproducible

#144

Post by pcrh »

micro_cam wrote: post #92
RyanMcGreal wrote: post #2

The analysis in question: http://www.nature.com/nature/journal/v483/n7391/full/483531a...

This is a really insightful article that contains a number of points that qualify the result and present lessons for any one interested in data science. A few points that stand out to me: !) The 11% number comes not from a randomized sample of cancer papers but from attempts made by AmGen to reproduce results that might be useful in drug development. This means that they made good effort to reproduce these results bu…

With regards to 1), it would be ironic if this study was itself not statistically valid...

Re: Just 11% of 53 cancer research papers were reproducible

#147

Post by CamperBob2 »

JoachimSchipper wrote: post #11
viggity wrote: post #7

After hearing similar things about psychology papers, this is rather disconcerting. This is why I am a climate change skeptic, I don't know whether it is happening due to CO2 or not, but I am confident that science can't be confident when they can't do control experiments. You can't control for any variable when it comes to the climate, let alone all the reasonable ones. We have infinitely more capabilities to contro…

You took an almost entirely unrelated story and made it about your political hobbyhorse. Please keep politics off HN.

And you just took an entire field off the table for discussion on the grounds that it looks like a "political hobbyhorse." Good job.

Re: Just 11% of 53 cancer research papers were reproducible

#148

Post by steve19 »

spikels wrote: post #109
epistasis wrote: post #73

Earlier quoted context omitted.

You're assuming that only "reproducible" results are valid, which is incorrect. Results that are reproducible within a lab, but not when attempted by other people in different settings indicate that there's an unaccounted for variable, and that falls into the 89%. There's still something valuable there, it's just that we don't yet know all the variables. We could just give up, or we could try to continue study of som…

If its not reproducible it is not science.

What about the experiments run by using the LHC? No other organization has a similarly sized particle accelerator, so by your definition it is not science because it cannot be reproduced elsewhere?

Re: Just 11% of 53 cancer research papers were reproducible

#149

Post by JDDunn9 »

Perhaps part of this is statistical errors and selection bias. Most studies are done with a 95% confidence interval, but only studies that reject the null hypothesis are published. If we assume only 20% of all experiments reject the null hypothesis, then we should expect ~25% of those to be fasle positives. Journals might also be biased to include more sensationalistic results, which are more likely to be statistical errors.

My guess is that if you included all test results, that reject the null hypothesis or not, and are published or not, then the reproducibility rate would be closer to 90%-95%.

Re: Just 11% of 53 cancer research papers were reproducible

#150

Post by ttrreeww »

nogoodnik wrote: post #68

The most interesting thing about all this is the fact that the researchers were required by paper authors to sign non-disclosure agreements and agree not to identify papers with non-reproducible results. This says a lot about the confidence authors have in their papers, doesn't it? Sorry but this isn't science anymore.

It's good old capitalism making money from "treating" sickness. The longer the better. A lifetime of "treatment" means a lifetime of profits.

Who cares about the cure.

Post reply on HN