Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

81–90 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#81
post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

I think you vastly underestimate how much a lot of scientific studies cost. It would be ideal to be able to have studies replicated by a separate group, but there are many times labs that have specialized equipment and engineering that make replicating studies by some random group unfeasible. In addition, if you spend too much time trying to replicate others' work, you have no time to work on the things that will act…

> I think you vastly underestimate how much a lot of scientific studies cost.

On the other hand, the money currently being invested seems to produce mostly garbage. "Fake science", just to rustle some feathers.

Re: Most scientists 'can't replicate studies by their peers'

#82
post #79

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how generalizable such results are. If the results don't hold in slightly altered systems, it falls into the wastebin of ideas.

> Our focus shouldn't be on exact replication, and should be on how generalizable such results are.

This is how I normally look at things. If I can't easily replicate an experiment, then it's very likely to be wrong.

Sadly, it's pretty rare (and exciting) when you can easily do something based on the methods in someone's paper

Re: Most scientists 'can't replicate studies by their peers'

#83
I think this can be very field dependent. I am a physicist and perhaps I am overly optimistic. However, I find that the existence of supplemental materials is helping things a lot. Generally, in an paper (in experimental physics), you are trying to tell a story about your measurements and the understanding that you gain from them. So, it's nice to be able to tell that story, where you focus on the most relevant data. However, with supplemental materials, you can add a lot of the sanity checks that you did (sometimes at the prompting of referees) in the supplemental materials. For example, maybe I want to tell a story about how in some material that by applying a voltage, I can influence the magnetic properties of that material. There are a lot of baseline measurements that you perform these days to determine that. In the past, with page limits, you couldn't include all of that information, but now, you can place a great deal more in.

In my particular field, my raw data is online, so that can be checked--though without more meta data, it would be of limited use to someone without my notebook and annotations. A lot of the code for the reduction tools that we use is also open sourced as well. There have been some moves in science towards reproducible data analysis, but the problem is infrastructure/ecosystem. For example, let's suppose that I use a python code to do some statistical analysis on my data, it could be hard for someone else to try to reproduce that say 20 years from now--because they won't just need the library I wrote, but the entire ecosystem that it lives in--I don't have a good answer for that.

But, I think that for high profile results (again, I'm optimistic) in physics, there's an incentive for other groups to try to reproduce them to see if they hold or not. There have been cases where someone honestly thought that an effect was there (and it was reproducible), but it was found out later that it was due to something extrinsic--for example, many sources of Boron that you buy commercially have trace amounts of magnetic impurities, so it took some time before someone realized that this was the cause for a weak signal that people were seeing.

In some communities, such as crystallography, you have to submit a file which shows your results and it is automatically checked for chemical consistency. I think this can help weed out some sloppy errors. But, it is still possible to make mistakes.

Also, with journals like Nature Scientific Reports, it becomes feasible to publish results that aren't so exciting, but are technically correct (it takes a lot of time to write a paper and an even longer time to publish it and the cost benefit analysis makes it difficult at times to publish everything, so lowering the barrier to publication to technical correctness rather than excitement helps people to publish null results so other people don't waste their time).

There's also the question of where to draw the line as a referee. If someone is publishing a methods paper where they have made a new analysis technique that is going to be implemented in software that most people are not going to check, then I feel obligated to check their derivations, reproduce integrals, etc. For other papers, looking for consistency between different measurements and the literature is probably sufficient.

There's still a lot of work to do, but I don't think things are completely disastrous in physics. I recently had a colleague retire and he was very proud of the fact that his data had never been wrong. The interpretation might have changed with time, but the data itself was right. I think that's the best we can hope for...

Re: Most scientists 'can't replicate studies by their peers'

#84
post #79

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how generalizable such results are. If the results don't hold in slightly altered systems, it falls into the wastebin of ideas.

Also, you should read the Peter Walter piece that I think is the best discussion of the topic: https://www.ascb.org/newsletter/2016-marchapril-newsletter/o...

Re: Most scientists 'can't replicate studies by their peers'

#85
This is a very unfortunate state of affairs for science in general, giving the amount of tools we have to make an experiment more replicable.

Replication is perhaps simplest in the field of computer science, yet many papers do not release the associated source code and configuration used in experiments. It's very easy to make the code available to all so I find it a bit dishonest and unscientific not to share an experiment.

There may be other things at play which prevent methodology being fully shared, it may get too personal, for example experimental code is sometimes developed quickly, and people may be reluctant to share their own messy code for fear it might reflect badly.

Re: Most scientists 'can't replicate studies by their peers'

#87
How about adding reproducibility to the h-index? Let's say that a reproducible paper is worth at least three times as much as a non reproducible one, maybe more (because it's going to take more time to write.) Would that give scientists an incentive to detail their methods so other scientists can reproduce their findings? Would that give rise to reproducibility rings to game the system?

Re: Most scientists 'can't replicate studies by their peers'

#88
post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> Make sure the computer don't start to fucking updating windows half way during experiment. Solution? Linux. Also, use proper tense and grammar when describing methods, else others may find the instructions ambiguous. I do appreciate this response's level of detail... +1.

> use proper tense and grammar when describing methods

What a nauseatingly unnecessary comment. Boo

Re: Most scientists 'can't replicate studies by their peers'

#89

This is ego, politics and career ambitions undermining modern science. Unfortunately, the fact that this is occurring so rampantly will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts. This is a systemic failure of basic ethics that will hurt us all. The success-at-all-cost career mindset is toxic in all tracks, but this is one of the most dangerous for it to take…

> Unfortunately, the fact that this is occurring so rampantly will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts.

Shouldn't you be just as well happy that these "anti-intellectuals" are pointing out flaws in something you thought was true?

Re: Most scientists 'can't replicate studies by their peers'

#90
post #78

Earlier quoted context omitted.

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

I was chatting with a colleague last summer, and asking him for more details about a method he had published that I was having trouble reproducing. After pointing me to a few papers that didn't include the relevant details, hefinally told me that "if we told everyone about how it was done, anyone could do it." While at the time I was pretty upset with him, perhaps it's the competitive nature of science and funding th…

> I was chatting with a colleague last summer, and asking > him for more details about a method he had published that > I was having trouble reproducing. After pointing me to a > few papers that didn't include the relevant details, > hefinally told me that "if we told everyone about how it > was done, anyone could do it."

That's not science, that's bullshit. Can you please expose that? Scientists shouldn't simply get away with such malicious behavior.

Post reply on HN