Most scientists 'can't replicate studies by their peers' • Hacker News Board

Re: Most scientists 'can't replicate studies by their peers'

#171

Post by thehardsphere » Thu, Feb 23, 2017, 6:12 PM UTC

BeetleB wrote: post #104

thehardsphere wrote: post #74

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

>That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. No, no, a thousand times no! Most studies do not have follow up studies that confirm/refute the original. Often such a followup study is hard to publish. If you manage to reproduce it, you cannot publish unless it presents a new finding. If you fail to reproduce it, it often doesn't get publishe…

> When I highlighted this to people in the field, they were quite open with another explanation: It helps them because they do not want their peers to know all the details. That allows them to have an edge over their peers and they do not need to race with them to publish further studies.

I have always suspected that, but I've never heard anyone be that open about it.

At a previous job I had to implement various algorithms described in research papers, and in every case except one, the authors left out a key part of the algorithm by glossing over it in the laziest way possible. My favorite one cited an entire linear algebra textbook as "modern optimization techniques."

Re: Most scientists 'can't replicate studies by their peers'

#172

Post by artimaeis » Thu, Feb 23, 2017, 6:14 PM UTC

cwyers wrote: post #150

jpfed wrote: post #141

Earlier quoted context omitted.

If some mice are housed near rats and others are not (or simply vary in how far they are from the nearest rat), that will introduce variation into your measurements. For experiments not specifically about mouse/rat interactions, this variation is irrelevant noise that makes it harder to detect or characterize the effects the experimenters actually care about.

That's not my point, though. Like, if you conduct physics experiments in a vacuum because interactions between objects and atmosphere isn't part of what you're studying, that's fine. If everyone in physics is conducting experiments that way, then suddenly you're left with the question of, it turns out that in real life I encounter atmosphere all the time , and how much can these experimental results tell me about the…

> "if you aren't comprehensively listing all the variables you're controlling for [...] then are you even really doing science?"

Mate I think that's the point of this whole thread that you're commenting in. And the tangential point to the article posted.

Science isn't some binary thing. You can do poor science, and you can do great science. Some variables are hard or impossible to control for. Some fields make this simpler than others. I'd say that as we've continually endeavored with the sciences we're probably better at it now than we've ever been before.

Synthetic conditions are absolutely critical to science. Typically, the more conditions you can specify in the experiment, the more reproducible it should be. Some of these are very difficult, and others in the thread have pointed out that some don't get labeled in the journals.

If we ran such experiments in the wild, completely outside of control, then we can never know what we're really observing. By controlling the environmental variables your observations gain meaning.

Re: Most scientists 'can't replicate studies by their peers'

#173

Post by epistasis » Thu, Feb 23, 2017, 6:16 PM UTC

Balgair wrote: post #153

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

A lot of the commenters are trying to come up with technical solutions to the issue. It belies their incomplete study of the 'softer' sciences and the difficulty of biological studies. Tech solutions won't work, bureaucratic solutions wont work, more data wont work. Guys, the bio side of things is incredibly complicated and trying to set controls that are achievable in your time and budget is the heart of these field…

I think there's also a lot of confusion about what the scientific literature should be.

Outsiders think of journals as pre-packaged nuggets of science fact, with conclusions that you can read and trust.

Scientists who publish in the journals view them as a way to communicate a body of work of "I did this, here is the result I saw. I think it might mean this."

The difference between these two views can not be understated. Each group wants the literature to be useful to them, so that they can use it, and that's understandable.

For most areas of science, especially in the early days of that science, it's absolutely essential that scientist's view be allowed to persist. Is it better to share early, or to wait to publish until you've tried all possible things that could potentially go wrong? I think the answer is obviously that you share the early data, and what you think it means, even if you may be wrong about it.

If the goal is to advance knowledge as quickly as possible, I think that the scientist's view is probably a better idea of what a journal needs to be than what the outsiders' view is. In some fields, like fMRI studies, the field is realizing that they may need to go about things differently. And that means that a lot of the interpretations that were published earlier are incorrect. But that process of incorrect interpretation to corrected interpretation is an essential part of science.

Re: Most scientists 'can't replicate studies by their peers'

#174

Post by IndianAstronaut » Thu, Feb 23, 2017, 6:18 PM UTC

sndean wrote: post #41

JumpCrisscross wrote: post #40

Earlier quoted context omitted.

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

This exists ( https://www.jove.com/ ), but, in my experience, even having that doesn't necessarily allow you to replicate the experiment. Though I should say that I really like JOVE. You can learn a lot.

JOVE is really nice. My old lab posted some videos on there of how to for experiments. It is a huge step up from just reading methods and materials.

Re: Most scientists 'can't replicate studies by their peers'

#175

Post by intrasight » Thu, Feb 23, 2017, 6:24 PM UTC

Important scientific results should be replicated. My understanding is that the scientific community now recognizes this and is trying to put in place the institutional frameworks to make this possible. Assuming that gets put in place, the next task is obviously choosing which experiments are important enough to justify replicating. Would be cool to apply some AI so as to remove the politics from that decision.

Re: Most scientists 'can't replicate studies by their peers'

#176

Post by jedberg » Thu, Feb 23, 2017, 6:27 PM UTC

binaryzeitgeist wrote: post #116

Specifically to the CS context, I think some version of double blinded peer code-review should be made mandatory for a publication. I've seen authors skip quite a many details that are quintessential to the replication process. In short, if research is not replicable by the peer community it just useless, that's what it is.

CS is the craziest of them all. Those should be the easiest to replicate. "Here is the code, here is a manifest of the environment/container/disk image/etc." You should be able to take that and run it and get the same results.

Or are you saying that the code itself is the problem and that they've done the equivalent of "return True" to get the result they want?

Re: Most scientists 'can't replicate studies by their peers'

#177

Post by lr4444lr » Thu, Feb 23, 2017, 6:32 PM UTC

cwyers wrote: post #112

carsongross wrote: post #72

Earlier quoted context omitted.

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Think about the epistemological ramifications of this. This is why trust is so important in social contexts, even in science.

> Think about the epistemological ramifications of this. This this this this this a million times this. If your experiment with lab rats needs to be done this finely to be replicated, is it really telling you anything about the real world? Because of course, we don't really care about the behavior of laboratory rats, right? Not in proportion to how often we do studies with laboratory rats versus other species of anim…

Exactly. The grandparent comment begs even more terrible questions about this whole enterprise of our species we call science than even the original article.

Re: Most scientists 'can't replicate studies by their peers'

#178

Post by UncleMeat » Thu, Feb 23, 2017, 6:34 PM UTC

godelski wrote: post #67

So I can't be the only one who has noticed the correlation between this and the field at question. As soft sciences like soc, psych, and medicine seem to have the most problems with it. I'm not saying hard sciences like physics don't, but it is less common. The math for the soft sciences isn't as concrete and doesn't provide a good foundation. I think there are also major problems with the use of p values. It is too…

Soft sciences actually dont have a greater problem with replication. It is just more publicized, in part because those researchers are actually addressing the problem and making large scale replication attempts.

Re: Most scientists 'can't replicate studies by their peers'

#179

Post by lr4444lr » Thu, Feb 23, 2017, 6:35 PM UTC

skosuri wrote: post #79

thehardsphere wrote: post #74

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how generalizable such results are. If the results don't hold in slightly altered systems, it falls into the wastebin of ideas.

Maybe not, but these studies get swept up into meta-studies, books, think tanks and special interest groups write policy papers based on that secondary layer, which then become position papers that inform policy makers about Big Decisions. There's a lot at stake when only scientists are aware of the fallacy.

Re: Most scientists 'can't replicate studies by their peers'

#180

Post by SilasX » Thu, Feb 23, 2017, 6:38 PM UTC

jonlucc wrote: post #163

SilasX wrote: post #120

Earlier quoted context omitted.

Or require the author to only communicate to the technicians via the same means that will be published with the paper -- i.e. Make sure anyone reading the paper has the same spec for the experiment as those who performed the original. [1] That would help prevent such "hidden specs" from entering the experiment. [1] Note: this implies that authors cannot be part of the experiment or conduct it themselves, because they…

In my job, I work with an outside contract research organization. The way this works is pretty similar to what you're describing, and its a living hell. For starters, it takes 6-10 drafts back and forth to get a protocol to start with, and the resulting document is never shorter than 12 pages. Then, once they start a study, if we get any aberrant data, the question becomes "did they mess that up, or is that data real…

Well, the burden of a policy has to be judged relative to what you're trying to accomplish. If the policy is ensuring that you achieve your ostensible goals, then that burden is justified.

Based on your description, it actually sounds like it's making you do things exactly the way science is supposed to work! You quickly identify issues of "real effect or experimenter error or undocumented protocol?" -- and you prevent any ambiguous case from "infecting" the literature.

Those are the same objectives modern science is currently failing at with it's "publish but never replicate" incentives.

> Lastly, you can tell a lot by being hands-on with a study; there's a lot you can miss if you aren't in the room with the study.

I wasn't saying that you can't be there in the lab and do that kind of experimentation, just that scientists shouldn't represent this kind of ad hoc work as the repeatable part that merits a scientific conclusion. The process should be: if you find something interesting that way, see if you can identify a repeatable, articulable procedure by which others can see the same thing, and publish that.