Most scientists 'can't replicate studies by their peers' • Hacker News Board

Re: Most scientists 'can't replicate studies by their peers'

#91

Post by return0 » Thu, Feb 23, 2017, 4:12 PM UTC

skosuri wrote: post #79

thehardsphere wrote: post #74

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how generalizable such results are. If the results don't hold in slightly altered systems, it falls into the wastebin of ideas.

> No scientist treats published studies as gospel.

Unfortunately that is a problem. Imagine yourself trying to create a simulation of a biological system, which has to rely on experiments. You may come up with a plan, but every little line on the plan will be either very doubtful or outright false. The problem is that many of these doubts could be dispelled if the experiment was a lot more stringent (much larger sample size, much more controlled conditions etc). That would cost a hell lot more, but it would give you one answer you can rely on.

Re: Most scientists 'can't replicate studies by their peers'

#92

Post by dekhn » Thu, Feb 23, 2017, 4:12 PM UTC

seszett wrote: post #57

dekhn wrote: post #53

Earlier quoted context omitted.

I published in nature. The results were based on a computation that was so large, no other entity in the world could have reproduced it at the time (I work for Google, we built a system that used 600+Kcores continuously for years). I don't think ,at the time, that anybody other than Amazon or Microsoft could have run the same calculation as us. Should our publication have been prevented simply because nobody else had…

> Should our publication have been prevented simply because nobody else had the resources to replicate? Well... maybe, yes? Just because you are the only ones able to do the computation doesn't mean your methods or your results are correct.

We share all the results from our research, so people may inspect them in detail. We also share our methods in enough detail that if somebody was sufficiently motivated, they could reproduce the work by spending time and money

however, If you follow your logic, you would prevent CERN from publishing the discovery of the Higgs Boson, because nobody else has a particle accelerator powerful enough to detect them. You would prevent the LIGO people from publishing because nobody else has a multi-mile-long interferometer that can detect gravity waves. There are many unique resources that would be prevented from publishing under your idea.

Contrary to what most non-scientists claim about science, reproducibility is not a requirement for discovery science. What is import is scrutiny- ensuring that when people discover something using a unique resource, other people can inspect the methods and the data and decide whether to believe the results.

Re: Most scientists 'can't replicate studies by their peers'

#93

Post by LordKano » Thu, Feb 23, 2017, 4:13 PM UTC

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

Don't forget to make sure that none of the people who will be near the area of the experiment live with and thereby smell like a cat.

Re: Most scientists 'can't replicate studies by their peers'

#94

Post by benmmurphy » Thu, Feb 23, 2017, 4:14 PM UTC

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

a load of variables that are needed to be controlled in order to get the original experiment to work is kind of a red flag tho. like someone could just keep trying the experiment and then adding another variable they are controlling in order to get the experiment to work. but really they are just running a bunch of experiments and getting lucky at some point. assuming they stop trying to control variables or try and replicate themselves as soon as they get a positive result :)

Re: Most scientists 'can't replicate studies by their peers'

#95

Post by pmontra » Thu, Feb 23, 2017, 4:15 PM UTC

ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

I agree that there are no test suites to provide experiments to run and no make test to repeat an experiment with little effort. This accounts for the "too complex" thing.

However many experiments should be reproducible. Not making results testable is against the goal of sharing knowledge. But I understand that's an extra effort compared the current state of the art, and that must be rewarded and acknowledged. In another comment I proposed to include reproducibility in the h-index.

Re: Most scientists 'can't replicate studies by their peers'

#96

Post by dekhn » Thu, Feb 23, 2017, 4:15 PM UTC

tinco wrote: post #70

dekhn wrote: post #53

Earlier quoted context omitted.

I published in nature. The results were based on a computation that was so large, no other entity in the world could have reproduced it at the time (I work for Google, we built a system that used 600+Kcores continuously for years). I don't think ,at the time, that anybody other than Amazon or Microsoft could have run the same calculation as us. Should our publication have been prevented simply because nobody else had…

Well, maybe they could not run the full experiment, but maybe they could audit the source code of the computation you ran, maybe just a small subset of the computation on for example a synthetic smaller dataset, that could maybe be used to test whether your computation works like it should. Did you think about the replicability of your research while you were working on it?

Gromacs (the code we used) is open source. MSM, the code we used to analyze the data, is open source. We ported MSM to an internal Google framework (Flume), so the resulting code could be open sourced, but wold be lacking the Flume framework to run inside.

As for thinking about replicability, sure I thought about it. That's why we released all the data we generated and described our methods in detail- so pharma and academics could use the data with confidence to design new drugs that target GPCRs. I also decided that rather than focus on replicability, I would focus on discovery and sharing, and I think that actually is a better use of my time as well as the time of other scientists.

Re: Most scientists 'can't replicate studies by their peers'

#97

Post by BickNowstrom » Thu, Feb 23, 2017, 4:17 PM UTC

I am somewhat disappointed with the lack of replicability in the field of machine learning and computer science. I think there is not much excuse for releasing a ML paper on a new algorithm or modeling technique without a link to a source code repo. Sure, your research code may not be pretty, but that should not be a deal-breaker. I hope reviewers start rewarding papers with links to source code. This should also stimulate refactoring, documenting, and cleaning up the linked source code.

Also a standard unified process for replicability, reproducibility, and reuse is needed. Dock points for not stating random seeds, hardware used, metadata, etc.

Re: Most scientists 'can't replicate studies by their peers'

#98

Post by rtkwe » Thu, Feb 23, 2017, 4:18 PM UTC

angry-hacker wrote: post #71

fsloth wrote: post #47

Earlier quoted context omitted.

So why aren't mouse studies done in acoustically isolated rooms? Too expensive or lack of forethought? I know metrological laboratories that are mechanically insulated with all sorts of contraptions making the space vibration free so it'd not as if buildings could not be adapted for their intended scientific use as a matter of principle.

This sounds a reasonable answer. If you can't do science in that environment, then it's not scientific because you can't control all the variables.

The classic ideal of controlling a single variable at a time is nice but largely impossible. Especially in biological sciences but that's true even down in chemistry and physics the systems scientists are trying to control are complex and it's not always possible to modify one variable without affecting others.

Re: Most scientists 'can't replicate studies by their peers'

#99

Post by lngnmn » Thu, Feb 23, 2017, 4:22 PM UTC

Publications, funding and status are the goals, not validity. Science became theology and metaphysics - joggling with fancy concepts and theories, producing chimeras from "advanced statistics" and probabilistic models, etc. The main assumption is that no one would even try to validate or reproduce the results, because they are too messy and too costly (labor-intensive).

"Scientist" and "researches" do not even realize that things like "dimensions" are mere concepts and does not exist in reality. Most of "scientists" could not tell an abstract concept from an existent phenomena. Most of them cannot explain the Correlation Is Not Causation principle. They believe that time really exist and could be slowed down or accelerated. Or it could be part of a crystal.)

The "studies" contains naive logical flaws, non-validated references, use of statistical inferences as logical inferences, use of statistics to establish causality and even Excel errors (unwanted auto-correction, rounding etc.)

This is only the tip of the iceberg - there will be turtles^W unreplicable metaphysics-like crap all the way down.

Re: Most scientists 'can't replicate studies by their peers'

#100

Post by skosuri » Thu, Feb 23, 2017, 4:23 PM UTC

return0 wrote: post #91

skosuri wrote: post #79

Earlier quoted context omitted.

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how generalizable such results are. If the results don't hold in slightly altered systems, it falls into the wastebin of ideas.

> No scientist treats published studies as gospel. Unfortunately that is a problem. Imagine yourself trying to create a simulation of a biological system, which has to rely on experiments. You may come up with a plan, but every little line on the plan will be either very doubtful or outright false. The problem is that many of these doubts could be dispelled if the experiment was a lot more stringent (much larger samp…

I don't think making experiments more stringent is the answer here (btw, I've spent a lot of time trying to build simulations of biological systems). Usually, we aren't doing the right experiment in the first place; this is hard to figure out ahead of time. Again, read the ACSB that I linked to below... it's probably the most nuanced and interesting discussion on the subject