Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

blog.scienceexchange.com

Post reply on HN
Print view View original

11–20 of 39 posts

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#11

Post by gabemart »

mjn wrote: post #8
gabemart wrote: post #6

I don't know very much about how reproducibility validation works. Is it the case that, if we assume p= ~0.05 and all 50 original studies are perfect, we would expect the first iteration of reproducibility validation to fail for ~2 of the 50 studies?

Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really just by chance. Therefore, if we were now given the real ground truth, it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. But you seem to be positing the reverse: a hypoth…

Thanks, very interesting. It was silly of me to assume that the p-value could be used in that way when, as you explain, it's not measuring or indicating that at all.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#12

Post by bmahmood »

Osmium wrote: post #7

Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course, verification should be cheaper than the original research since you know exactly what to look for and how to find it.

Hi this is Bilal from Science Exchange, and I help with the Reproducibility Initiative.

You are right that the per study costs are cheaper than normal as the original protocol is already developed, and there is less trial and error involved. In certain cases, we may not need to validate every aspect of a study as well, only those specific experiments that warrant replication. And depending on the study, those experiments may vary in cost (i.e. a mouse model component would be expensive, but RNA extraction and western blot testing relatively inexpensive).

Additionally, we will leverage expert labs on Science Exchange which perform experiments regularly on a contract basis, allowing for bulk savings on reagents and supplies.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#13

Post by DaveWalk »

I'd be interested to know which studies they are targeting. Is SciEx testing a key figure from an expansive publication, or the entire methodology from discoveries with few tests? To me, this seems to be a conceptually difficult decision to make...most discoveries do not discuss the number of years (or failed attempts) that goes by before obtaining the quantifiable result.

And where is the peer review in this process? I suppose as soon as something turns up unreproducible we will find out.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#14

Post by 001sky »

Osmium wrote: post #7

Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course, verification should be cheaper than the original research since you know exactly what to look for and how to find it.

I came here to say the same thing. For important research, it would be interesting to see more cost/benefit analysis being put to bear in this way. The cost of replication should be so much lower than original work, in many cases. And it would be interesting to evaluate the "science" of academic "science", especially given the controversy around peer review quality controls in general on academic publishing.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#15

Post by Eliezer »

bmahmood wrote: post #10
gabemart wrote: post #6

I don't know very much about how reproducibility validation works. Is it the case that, if we assume p= ~0.05 and all 50 original studies are perfect, we would expect the first iteration of reproducibility validation to fail for ~2 of the 50 studies?

This is Bilal from Science Exchange, and I help with the Reproducibility Intiative as well. I think that in general you're right, with the assumption that p=0.05. However, each of the studies being replicated is composed of many intricate experiments. We will be reproducing most of those experiments, so while the odds of a single validation experiment failing are certainly not zero, I think the chances of an entire p…

> I think that in general you're right

Oh, dear. This is very much not a good sign, since answer to grandparent's question is "no" as pointed out above and anyone who knows statistics should see that immediately. (Checks Science Exchange site.) Okay, Bilal Mahmood is "customer support and outreach" so this is not necessarily fatal. Still worrisome.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#16

Post by napoleoncomplex »

Reproducibility in science is something that badly needs this push. It's an incredibly difficult "sell" to anyone with funds for research, and I'm extremely happy that they've found capital for it.

The foundations of our scientific knowledge need to be solidified, and from all the science news and developments, this one is the one that makes me by far the most excited for the future of science.

Next on the list, open source repositories for protocols of experiments! Maybe someone surprises me with a link to an existing solution :).

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#17

Post by loceng »

bmahmood wrote: post #12
Osmium wrote: post #7

Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course, verification should be cheaper than the original research since you know exactly what to look for and how to find it.

Hi this is Bilal from Science Exchange, and I help with the Reproducibility Initiative. You are right that the per study costs are cheaper than normal as the original protocol is already developed, and there is less trial and error involved. In certain cases, we may not need to validate every aspect of a study as well, only those specific experiments that warrant replication. And depending on the study, those experim…

Bulk Science. I like it.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#18

Post by [deleted] »

Eliezer wrote: post #15
bmahmood wrote: post #10

Earlier quoted context omitted.

This is Bilal from Science Exchange, and I help with the Reproducibility Intiative as well. I think that in general you're right, with the assumption that p=0.05. However, each of the studies being replicated is composed of many intricate experiments. We will be reproducing most of those experiments, so while the odds of a single validation experiment failing are certainly not zero, I think the chances of an entire p…

> I think that in general you're right Oh, dear. This is very much not a good sign, since answer to grandparent's question is "no" as pointed out above and anyone who knows statistics should see that immediately. (Checks Science Exchange site.) Okay, Bilal Mahmood is "customer support and outreach" so this is not necessarily fatal. Still worrisome.

[deleted]

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#19

Post by Osmium »

mathattack wrote: post #9
Osmium wrote: post #7

Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course, verification should be cheaper than the original research since you know exactly what to look for and how to find it.

If $1.3M can cover 50, they should fund reproducing 500, or even 5000. I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.

> I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties.

Not necessarily true at all actually. To use an example, the field I work in requires that we grow our own semiconductor crystals. Different growth reactors in different universities producing nominally the exact same substance will result in sometimes vastly different properties of the end result when studied (usually due to minute trace impurities, or marginally different temperatures or flow rates during growth, or a host of other things–all of which need to be analysed and vetted–and this happens even if they're using growth reactors made by the same manufacturer!). It makes replication very difficult and often results in slow progress. And it's expensive too. So it's not just medicine that has that problem.

Edit: I should add that obviously in the end it works out because these materials eventually make it into industry and into millions of devices. So it's ultimately reproducible. But that gap between first discovering a process or a mechanism or just something interesting and making it reproducible consistently enough and at a large enough scale for industry is just huge.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#20

Post by Alex3917 »

napoleoncomplex wrote: post #16

Reproducibility in science is something that badly needs this push. It's an incredibly difficult "sell" to anyone with funds for research, and I'm extremely happy that they've found capital for it. The foundations of our scientific knowledge need to be solidified, and from all the science news and developments, this one is the one that makes me by far the most excited for the future of science. Next on the list, open…

Assuming there are about 3 dozen common scientific mistakes in medical journal articles, each appearing in on average 10-20% of papers with a relatively high degree of independence, there isn't really much foundation for medical science to begin with. (And I think these assumptions are probably fairly accurate.)
Post reply on HN