Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

71–80 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#71

Post by angry-hacker »

fsloth wrote: post #47
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

So why aren't mouse studies done in acoustically isolated rooms? Too expensive or lack of forethought? I know metrological laboratories that are mechanically insulated with all sorts of contraptions making the space vibration free so it'd not as if buildings could not be adapted for their intended scientific use as a matter of principle.

This sounds a reasonable answer.

If you can't do science in that environment, then it's not scientific because you can't control all the variables.

Re: Most scientists 'can't replicate studies by their peers'

#72

Post by carsongross »

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know.

Think about the epistemological ramifications of this.

This is why trust is so important in social contexts, even in science.

Re: Most scientists 'can't replicate studies by their peers'

#73

Post by angry-hacker »

benrawk wrote: post #38

I am a social scientist studying human behavior, and this is a huge problem in the field. Myself and my statistician friends who analyze the literature have basically concluded that most extremely "novel" and "surprising" findings in the literature aren't even worth trying to replicate (remember, replications cost money to run, so before you start you have to make some judgment about the likelihood of success.) This…

Thank you. At first I thought puzzagate?! - I don't need that bs, but this is very different pizzagate.

I once found a good blog about mental health and science, a lot of snakeoil shown about srris, adhd etc. but I'm unable to find it now. Can anyone help me out?

Re: Most scientists 'can't replicate studies by their peers'

#74

Post by thehardsphere »

An alternative headline would be "Most Published Studies Are Wrong"

Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow from the original finding. But either way, people find out.

I mean, I guess the real problem is that lots of sloppy studies get published, and a lot of scientists are incentivized to write sloppy studies. But if you're actually a working scientist, you should understand that already and not take everything you see in a journal as actual truth, but as something maybe might be true.

Re: Most scientists 'can't replicate studies by their peers'

#75

Post by charles-salvia »

TheAceOfHearts wrote: post #65

This isn't that surprising, at least based on my limited experience from reading computer science research papers. My experience has been that there's usually not enough information for you to implement something. Am I an outlier, or have others experienced the same?

I've spent hours/days/weeks implementing algorithms or data structures from journal articles. No, you're not an outlier. A lot of times they don't include source code, even in the age of Github. Certain details or assumptions are often glossed over, and often the wording of a crucial technical detail is extremely ambiguous. Sometimes the only way I'm able to successfully implement the algorithm/data structure is through sheer luck - i.e. stumbling upon some piece of information elsewhere that gave me the necessary insight to understand what the author of the journal article meant.

I mean, I can understand academic time pressure and everything, but not providing a link to source code in this day and age is almost absurd. At the very least, it certainly doesn't encourage anyone to actually use your research in industry.

Re: Most scientists 'can't replicate studies by their peers'

#76

Post by tgb »

ZeroGravitas wrote: post #18
tgb wrote: post #12

Earlier quoted context omitted.

I don't think that analogy is at all accurate and I think the conclusion that you reach from it is completely incorrect.

I think I must have expressed myself poorly, as I think my conclusion is the same as the article suggests i.e. that the science/code shouldn't be considered "done", until it's been peer-reviewed, since it's easy to fool yourself and others if you're not actually reviewing and testing your code. What did you take away from my comment?

After reading your explanations elsewhere, I take back my statements. Your statements are literally correct, although easy to take incorrectly, and I support them: in particular you seem to be supporting that 100% of researchers should attempt to replicate studies and do so sufficiently that all of them will eventually fail to replicate at least some studies. I think most of us took this to mean that you thought every study should fail to replicate (by analogy that every software has at least some bugs), but I see now your intent and that your original wording backed that up.

Re: Most scientists 'can't replicate studies by their peers'

#77

Post by aaronchall »

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> Make sure the computer don't start to fucking updating windows half way during experiment.

Solution? Linux.

Also, use proper tense and grammar when describing methods, else others may find the instructions ambiguous.

I do appreciate this response's level of detail... +1.

Re: Most scientists 'can't replicate studies by their peers'

#78

Post by ovis »

JumpCrisscross wrote: post #40
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

I was chatting with a colleague last summer, and asking him for more details about a method he had published that I was having trouble reproducing. After pointing me to a few papers that didn't include the relevant details, hefinally told me that "if we told everyone about how it was done, anyone could do it."

While at the time I was pretty upset with him, perhaps it's the competitive nature of science and funding that also gives people a mild incentive to be secretive.

In this case, simply publishing code would have resolved the questions.

Re: Most scientists 'can't replicate studies by their peers'

#79

Post by skosuri »

thehardsphere wrote: post #74

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how generalizable such results are. If the results don't hold in slightly altered systems, it falls into the wastebin of ideas.

Re: Most scientists 'can't replicate studies by their peers'

#80

Post by ibgib »

zeristor wrote: post #3

Hmmm. Software Engineering has Continuous Integration, since it is so expensive to fix software later in the day. Is there any such thing as Continuous Reproducibility? Constantly checking that the science can be reproduced? How prevalent is this in different branches of Science?

I completely agree with applying software engineering skills to this. Not just CI for deployment, but also open-source collaboration. I see research and development in other fields as largely like closed-source silos, and a centralized but evolving ecosystem for information collaboration in any topic is in the works.

I know that is definitely one of my driving visions for ibGib, and I can't be the only one. Open source, open data, open collaboration.

Post reply on HN