Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

281–290 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#281

Post by hyperion2010 »

hackuser wrote: post #278
hyperion2010 wrote: post #183

Earlier quoted context omitted.

My PhD work is trying to address this by developing better ways for scientists to record and communicate their methods/protocols. Methods sections are NOT about providing the information needed to reproduce the work, often even the supplemental information is insufficient. They are a best guess at what needs to be done, and the methods section is often massively condensed due to editor demands and cites other papers…

How about recording videos of the lab? People trying to reproduce the experiment can just sift through the video. That may be tedious but it's far better than nothing. Just a little metadata would help: Experiment A, Phase N, Day X

Video and photographic evidence can play a big role and when we have the extra bandwidth to process such a dataset. Right now we barely have time to do the experiments, much less 'watch tape' to see how we did (maybe if scientists were paid like professional sports players...). In an ideal world we would be collecting a much data as we possibly could about the whole state of the universe surrounding the 'controlled' experiment. That said video and photographs are very bad a communicating important parameters in an efficient way. Think about how hard it is to get information out of a youtube video if you need something like a part number. Photos do better, but if you need to copy and paste out of a photo we will need a bit more heavy lifting to translate that into some actionable format (eg ASCII).

Re: Most scientists 'can't replicate studies by their peers'

#282

Post by hackuser »

bbctol wrote: post #119
JumpCrisscross wrote: post #40

Earlier quoted context omitted.

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

The journal I work for just rolled out a new methods format; the big change is requiring all resources to be listed, with their origin (a major problem is that Chemical A or lab rat subspecies B might be very different from one company or another, even if they're theoretically identical.) What's really needed is broader standardization of all practices; requiring video wouldn't solve the issue that different labs mig…

> No one is going to document and record the entire course of an experiment; that would literally be months of footage.

Why not? Just setup a permanent camera and go. Storage is cheap. Organizations with security cameras often have multiple streams to record.

Re: Most scientists 'can't replicate studies by their peers'

#283

Post by hackuser »

jonlucc wrote: post #132
bbctol wrote: post #119

Earlier quoted context omitted.

The journal I work for just rolled out a new methods format; the big change is requiring all resources to be listed, with their origin (a major problem is that Chemical A or lab rat subspecies B might be very different from one company or another, even if they're theoretically identical.) What's really needed is broader standardization of all practices; requiring video wouldn't solve the issue that different labs mig…

Additionally, not everything appears on video. We have found major differences between identical studies run in rooms set at different temperatures. Also, many of the procedures I do would essentially require a camera operator to capture all of the movement. I take a cage out of a rack, take a mouse out of the cage, weigh it, dose it, etc.

> not everything appears on video

Sure, but a lot of information does appear. Not everything appears in papers; should we stop publishing them?

Re: Most scientists 'can't replicate studies by their peers'

#284

Post by hackuser »

SubiculumCode wrote: post #255
busyant wrote: post #185

Earlier quoted context omitted.

I can't tell you how many times I tried (unsuccessfully) to replicate a procedure using the methods section in a research article. Then I contact the author and they send me a protocol that is a 'distant cousin' to what I read in the published article. This caused me to assiduously report methods when I wrote papers.

That is frustrating. In my field of cognitive neuroscience, there is often little incentive for a researcher to hide parts of their methods from their 'competitors' since, for example, my memory study is not about reporting but protecting a new technology. Indeed, prestige is often a consequence of others adopting your methods, so researchers are motivated to share scripts, and report the methods fully. HOWEVER, cert…

> there is often little incentive for a researcher to hide parts of their methods from their 'competitors' since, for example, my memory study is not about reporting but protecting a new technology

If you are protecting, not reporting information (e.g., about a new technology), why would you be incentivized to share it?

Re: Most scientists 'can't replicate studies by their peers'

#285

Post by hackuser »

glibgil wrote: post #88
aaronchall wrote: post #77

Earlier quoted context omitted.

> Make sure the computer don't start to fucking updating windows half way during experiment. Solution? Linux. Also, use proper tense and grammar when describing methods, else others may find the instructions ambiguous. I do appreciate this response's level of detail... +1.

> use proper tense and grammar when describing methods What a nauseatingly unnecessary comment. Boo

I disagree. Grammar rules are a very effective checklist for clarity and precision in communication, IME. Does the verb have a subject and object? Otherwise, it's not clear who should do which to what. The instructions say, 'turn it on'; does "it" have a clear antecedent? Otherwise, what are they turning on?

I would guess that the practical benefits are a big reason that they are the 'rules'.

Re: Most scientists 'can't replicate studies by their peers'

#286

Post by pvaldes »

eclee2 wrote: post #16
ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

If there are errors in a study's methods that make it unreplicable, then it shouldn't have passed peer review or been published.

Then you risk to lose all einstein work unless you have two einsteins at the same time in fact. Genious are scarce. And you could not publish nothing about comets for example, because this would be unreplicable until the next 20 years. Is not so simple as that.

Re: Most scientists 'can't replicate studies by their peers'

#287

Post by hackuser »

hyperion2010 wrote: post #281
hackuser wrote: post #278

Earlier quoted context omitted.

How about recording videos of the lab? People trying to reproduce the experiment can just sift through the video. That may be tedious but it's far better than nothing. Just a little metadata would help: Experiment A, Phase N, Day X

Video and photographic evidence can play a big role and when we have the extra bandwidth to process such a dataset. Right now we barely have time to do the experiments, much less 'watch tape' to see how we did (maybe if scientists were paid like professional sports players...). In an ideal world we would be collecting a much data as we possibly could about the whole state of the universe surrounding the 'controlled'…

I didn't mean record it to use the data in your analysis, but record it to preserve the methods for others. If they have trouble getting part of the experiment to work, they can pull up the video and see how you did it (at least to a degree; I'm not expecting 360 video. You can't possible record in text everything a video could capture.

Thanks for sharing your knowledge and experience in this discussion, by the way. It's what makes HN great.

Re: Most scientists 'can't replicate studies by their peers'

#288

Post by pvaldes »

ZeroGravitas wrote: post #29
gcp wrote: post #22

Earlier quoted context omitted.

Peer review does not imply reproducibility, and it's the latter that is the problem. I can confirm, as a reviewer, that your methodology and analysis looks sensible, but the flaws may be deeper, and the fact that you didn't publish the 19 other studies that failed, but that this is the "lucky one", or that you simply cherry picked the data, is not something I can see as a reviewer. This is especially true if the expe…

" Peer review does not imply reproducibility, and it's the latter that is the problem." I think this is the key to it, I'm suggesting that reproducibility should be part of considering something peer-reviewed, but of course as currently practised, that isn't true. Of course in a software metaphor, that would probably cover both code review and QA, which is sometimes done by a different job role which further muddies…

This would be like expecting a car brand to open its code to its competitors before to release a new product. Will lead typically also to the peers rushing to publish the same discovering on disguise before the original.

Re: Most scientists 'can't replicate studies by their peers'

#289

Post by Svezik »

thehardsphere wrote: post #74

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

> But if you're actually a working scientist, you should understand that already and not take everything you see in a journal as actual truth, but as something maybe might be true.

I believe that's called "skepticism" which makes you a heretic and "anti-science" in certain fields.

Re: Most scientists 'can't replicate studies by their peers'

#290

Post by [deleted] »

hackuser wrote: post #285
glibgil wrote: post #88

Earlier quoted context omitted.

> use proper tense and grammar when describing methods What a nauseatingly unnecessary comment. Boo

I disagree. Grammar rules are a very effective checklist for clarity and precision in communication, IME. Does the verb have a subject and object? Otherwise, it's not clear who should do which to what. The instructions say, 'turn it on'; does "it" have a clear antecedent? Otherwise, what are they turning on? I would guess that the practical benefits are a big reason that they are the 'rules'.

[deleted]
Post reply on HN