Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

291–300 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#291

Post by Svezik »

skosuri wrote: post #79
thehardsphere wrote: post #74

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how generalizable such results are. If the results don't hold in slightly altered systems, it falls into the wastebin of ideas.

> No scientist treats published studies as gospel.

Tell that to the climatologists burning their peers at the stake for heresy.

Re: Most scientists 'can't replicate studies by their peers'

#292

Post by SubiculumCode »

hackuser wrote: post #284
SubiculumCode wrote: post #255

Earlier quoted context omitted.

That is frustrating. In my field of cognitive neuroscience, there is often little incentive for a researcher to hide parts of their methods from their 'competitors' since, for example, my memory study is not about reporting but protecting a new technology. Indeed, prestige is often a consequence of others adopting your methods, so researchers are motivated to share scripts, and report the methods fully. HOWEVER, cert…

> there is often little incentive for a researcher to hide parts of their methods from their 'competitors' since, for example, my memory study is not about reporting but protecting a new technology If you are protecting, not reporting information (e.g., about a new technology), why would you be incentivized to share it?

yeah it seems funny, but it happens all the time.Perhaps there are reporting requirements from the funding?

Re: Most scientists 'can't replicate studies by their peers'

#293

Post by petra »

escot wrote: post #271
petra wrote: post #193

Earlier quoted context omitted.

Wouldn't it be hard to achieve deep consistency between experiments, in so many labs around the world, with so different conditions/cultures/etc , and when the experimenters aren't experts in consistency , but in their science ? Wouldn't it be better to use something like a cloud biology model - where you define experiments via code, CRO's compete on consistency(and efficiency and automation) and since they probably…

I work at a cloud bio lab. We run all of our experiments on automation, and all protocols must be defined in code. The latter is both the power and the difficulty -- when your protocol is defined in code it is explicit. However, writing code is both new and sometimes difficult for the scientists that we currently work with (molecular biology, drug discovery). I believe what we are doing is the right model. But it com…

I do agree, it seems like right model, and will have a large impact.

Between automating labor, economies of scale in purchasing, and access to more efficient technology(like acoustic liquid handling) ,etc - isn't it just a matter of time before cloud biology becomes quite cost effective and combined with other benefits - it would be the only way that makes sense to do research, so funding will naturally go there?

Also - do you see a way to add the extreme versatility of the biology lab into a cloud service ?

Re: Most scientists 'can't replicate studies by their peers'

#294

Post by Tsagadai »

BickNowstrom wrote: post #97

I am somewhat disappointed with the lack of replicability in the field of machine learning and computer science. I think there is not much excuse for releasing a ML paper on a new algorithm or modeling technique without a link to a source code repo. Sure, your research code may not be pretty, but that should not be a deal-breaker. I hope reviewers start rewarding papers with links to source code. This should also sti…

I have tried and failed to reproduce some findings on ML papers. Sometimes graphs are being significantly smoothed or filtered which makes results look better, other times core components of algorithms are not described and the findings cannot be reproduced at all.

Source code, or at the very least proper pseudocode, should be mandatory for all published computer science research.

Re: Most scientists 'can't replicate studies by their peers'

#295

Post by jonlucc »

hackuser wrote: post #283
jonlucc wrote: post #132

Earlier quoted context omitted.

Additionally, not everything appears on video. We have found major differences between identical studies run in rooms set at different temperatures. Also, many of the procedures I do would essentially require a camera operator to capture all of the movement. I take a cage out of a rack, take a mouse out of the cage, weigh it, dose it, etc.

> not everything appears on video Sure, but a lot of information does appear. Not everything appears in papers; should we stop publishing them?

No, but the burden of videography is a lot higher than the burden of writing a paper. In fact, a paper isn't written for every study, so in order to write a paper, you'd have to take video of every study just in case one of them is used in a paper in the future. It's a much higher burden than people want to believe. Add to that the fact that most animal facilities won't add cameras unless you force them at gunpoint because historically, videos of that sort make for targeting by protest groups.

Re: Most scientists 'can't replicate studies by their peers'

#296

Post by jamesash »

I wish they used more specific language than the blanket term "scientists". Reproducibility is highly field-dependent. In organic chemistry, for example, the journal Organic Syntheses only publishes procedures that have been checked for reproducibility in the lab of a member of the board of directors. http://www.orgsyn.org A few years ago someone published a paper in a highly esteemed journal that was largely thought to be bunk (oxidation of alcohols by sodium hydride). A prominent blogger live-blogged his (unsuccessful) attempt to reproduce the experiment, as did others. The oxidation was in fact due to the presence of oxygen, not due to sodium hydride. The paper was retracted.

Re: Most scientists 'can't replicate studies by their peers'

#297

Post by mtdewcmu »

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

What if you took the information that doesn't fit in the paper and publish it on the web?

Re: Most scientists 'can't replicate studies by their peers'

#298

Post by Tsagadai »

virusduck wrote: post #46
tinco wrote: post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

I think you vastly underestimate how much a lot of scientific studies cost. It would be ideal to be able to have studies replicated by a separate group, but there are many times labs that have specialized equipment and engineering that make replicating studies by some random group unfeasible. In addition, if you spend too much time trying to replicate others' work, you have no time to work on the things that will act…

If it only works on a Wizbang 200 and cannot be reproduced with the Doodad 3000 then it probably doesn't work or there are other variables missing from their results. Specialised equipment should not affect results (proving that equipment is broken has some value though).

Re: Most scientists 'can't replicate studies by their peers'

#299

Post by hyperion2010 »

hackuser wrote: post #287
hyperion2010 wrote: post #281

Earlier quoted context omitted.

Video and photographic evidence can play a big role and when we have the extra bandwidth to process such a dataset. Right now we barely have time to do the experiments, much less 'watch tape' to see how we did (maybe if scientists were paid like professional sports players...). In an ideal world we would be collecting a much data as we possibly could about the whole state of the universe surrounding the 'controlled'…

I didn't mean record it to use the data in your analysis, but record it to preserve the methods for others. If they have trouble getting part of the experiment to work, they can pull up the video and see how you did it (at least to a degree; I'm not expecting 360 video. You can't possible record in text everything a video could capture. Thanks for sharing your knowledge and experience in this discussion, by the way.…

Ah, yes, things like JOVE [0] are definitely useful but they don't seem to scale to the sheer number of protocols that need to be documented (eg a single JOVE publication is exceedingly expensive). I have also heard from people who have tried to record video of themselves doing a protocol is that it is very hard to make them understandable for someone else. That said if the 'viewer' is highly motivated videos of any quality could be invaluable. Sometimes it is just better to buy the plane tickets and go directly to the lab of the person who can teach you (if they are still around).

0. https://www.jove.com/

Re: Most scientists 'can't replicate studies by their peers'

#300

Post by quebien »

BeetleB wrote: post #209
thehardsphere wrote: post #167

Earlier quoted context omitted.

>>That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. >No, no, a thousand times no! Most studies do not have follow up studies that confirm/refute the original. Often such a followup study is hard to publish. If you manage to reproduce it, you cannot publish unless it presents a new finding. If you fail to reproduce it, it often doesn't get publis…

>Sorry, let me be clear: If an interesting result is published, people will go to the trouble. Most results are of limited interest and mediocre. If it's in a journal, it is interesting. Journal editors will require "interesting" as a prerequisite to publishing a paper. Papers do get rejected for "valid work but not interesting". If journals are publishing papers that are of limited interest, then there is a serious…

"Interesting" is vague and subjective. Some work is boring as hell but the results provide the foundation for things that are truly "interesting".
Post reply on HN