Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

121–130 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#121

Post by cowsandmilk »

mattnewton wrote: post #59
dekhn wrote: post #50

So. Having been a scientist, I observed an interesting phenomenon, several times. It's almost as if scientists enjoy leaving out critical details from the methods section, and other scientists enjoy puzzling out what the missing details are. I think there's this sort-of assumption of competence that for any reasonably interesting paper, the people in the field who are reading it, they have the level of skill to repro…

Do you think that has anything to do with the incentives for reviewers, who want shorter papers with a length limit and prefer more details on the impact/why of the experiment? Does the same method-hiding hold true in journals without length limits or different review processes?

I think length limits cause problems.

In the software world, Bioinformatics has 2 page application notes[1]. That is nowhere near enough room to have a figure, describe an algorithm, and describe results. In cases where the source code is available, I've found the actual implementation often has steps not described at all in the algorithm. And these differences make a clean room implementation difficult to impossible if you want to avoid certain license restrictions.

Since it has been a decade since I worked in a wet lab, I'm less familiar with examples in that world, but I know not offending chemical vendors is a concern for some people in the synthetic chemistry world. At a poster session, they'll tell you that you shouldn't buy a reagent from a particular vendor because impurities in their formulation kill the described reaction. They won't put that in a paper though.

[1] https://academic.oup.com/bioinformatics/pages/instructions_f...

Re: Most scientists 'can't replicate studies by their peers'

#122

Post by bbctol »

blakesterz wrote: post #20

I have a high level question that I don't see answered, but maybe I missed it? Are most studies conducted by folks in academia before they get tenure? That is, are most of these results that they're trying to replicate or study done by people who are rather new to doing studies? Is this even possible to know? My guess would be yes , but really I don't have much to base that on. And if that is a yes could that have so…

Well... most studies are conducted by a lab run by someone with tenure. Most people doing the experimental design and research do not have tenure, but are trying to get there. Most people doing the actual grunt work of running tests, drawing blood, dissecting mice, wrangling data, and so on, are grad students or undergrads there to learn. And if that's a problem, it could only be solved by a serious overhaul of the funding model for research.

Re: Most scientists 'can't replicate studies by their peers'

#123

Post by [deleted] »

tinco wrote: post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

[deleted]

Re: Most scientists 'can't replicate studies by their peers'

#124

Post by marknutter »

This is an especially troubling problem in the soft-sciences. I recently learned about [post-treatment bias](http://gking.harvard.edu/files/bigprobP.pdf) which is one of the bigger problems plaguing the social sciences. Avoiding biasing an experiment when choosing the variables you control for is insanely difficult to the point where I wonder if it's even possible.

The scary thing is that people will regularly cite soft-science publications that align with whatever political agenda they may have, and anyone who dares contradict the authority of those studies are shouted down as "anti-intellectual" or other such nonsense.

I used to abhor people for leaning on their bibles to push their agendas but I'm starting to see how secular people are basically doing the same thing.

Re: Most scientists 'can't replicate studies by their peers'

#125

Post by endorphone »

ZeroGravitas wrote: post #33
endorphone wrote: post #15

Earlier quoted context omitted.

It is a terrible thing, and it is absolutely nothing like finding bugs. Reproducibility is a core requirement of good science, and if we need to compare it to software engineering, the reproducibility crisis is like the adage "many eyes make all bugs shallow", when the assumption that there is many eyes even looking is often untrue. Most studies are never reproduced, but are held as true under the belief that if some…

No, I claimed that all scientist would have had the experience of not reproducing something. Because if they do it a lot, as part of a regular process then they will eventually find something that doesn't work because the original scientist didn't document a step correctly or misread the results or just got lucky due to random chance. Just like all developers will eventually find a bug in code they code review. This…

While the wording may be vague, they aren't talking about the experiences of a subset of researchers -- they are saying that of the experiments they tried to replicate, 2/3rds weren't reproduceable. That is terrible, and has absolutely nothing to do with finding bugs.

Re: Most scientists 'can't replicate studies by their peers'

#126

Post by bbctol »

droopyEyelids wrote: post #5

>The reproducibility difficulties are not about fraud, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge. That would be relatively easy to stamp out. Instead, she says: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about." Well, maybe I'm too much of a layman, but that doesn't qui…

It's more like if a guy went to Vegas and came back with thousands of dollars, and decided to tell his friends he's a master poker player, not that he got lucky. Interesting findings happen in scientific experiments, and it's hard to say why, but we should stop acting as if they're always true (and stop just funding quests for even more interesting findings.)

Re: Most scientists 'can't replicate studies by their peers'

#127

Post by DoofusOfDeath »

AlexB138 wrote: post #23

This is ego, politics and career ambitions undermining modern science. Unfortunately, the fact that this is occurring so rampantly will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts. This is a systemic failure of basic ethics that will hurt us all. The success-at-all-cost career mindset is toxic in all tracks, but this is one of the most dangerous for it to take…

> Unfortunately, the fact that this is occurring so rampantly will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts.

I agree with your point, but I suspect your choice of wording tells a little more of the story.

You said "facts", but given your skepticism of their veracity, I think "claims" or "propositions" is more apropos.

Consider the rhetorical difference of these two statements:

"Anti-intellectuals don't accept the facts."

vs.

"Anti-intellectuals don't accept the claims."

I would also question your use of the term "anti-intellectuals". I wonder if, at least for some of the persons to whom you apply that label, a better term is "skeptic".

Re: Most scientists 'can't replicate studies by their peers'

#128

Post by wycx »

blakesterz wrote: post #20

I have a high level question that I don't see answered, but maybe I missed it? Are most studies conducted by folks in academia before they get tenure? That is, are most of these results that they're trying to replicate or study done by people who are rather new to doing studies? Is this even possible to know? My guess would be yes , but really I don't have much to base that on. And if that is a yes could that have so…

Most studies are done by people who need high impact publications to secure funding to do more studies to produce high impact publications to secure funding to do more studies...

Not having a permanent job just serves as additional motivation to get high impact publications to secure a job.

Re: Most scientists 'can't replicate studies by their peers'

#129

Post by jonlucc »

LordKano wrote: post #93
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

Don't forget to make sure that none of the people who will be near the area of the experiment live with and thereby smell like a cat.

You can add to this that if you are in a facility with both rats and mice, you should never house them in the same room, and you should probably not enter a mouse room after being in a rat room. In the wild, rats are predators of mice.

Re: Most scientists 'can't replicate studies by their peers'

#130

Post by logfromblammo »

marcosdumay wrote: post #45
droopyEyelids wrote: post #5

>The reproducibility difficulties are not about fraud, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge. That would be relatively easy to stamp out. Instead, she says: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about." Well, maybe I'm too much of a layman, but that doesn't qui…

That text is describing this situation: Context: You are a young scientist, under constant evaluation that may not only fire you, but also invalidate lots of your previous work. You have to produce flashy results to grow up on your career. You _really_ do not desire to get stuck where you are now. Now, let's decide on a new experiment to proceed. Do you choose: a - Boring important experiment that you can't hype but…

This quandary is a great argument for never becoming an academic researcher.

It would seem to me that science needs to eliminate "career bias" or "mortgage bias" or "ramen bias" from its results. Negative results need to be just as publishable as significant results.

If I were tyrant of the Ivory Tower, I would decree that results be blinded until after acceptance for publication. I would further decree that prestige be allocated such that the first and second replications have equal prestige to an original publication, and successive replications are worth less prestige, but are still worth attempting.

The only failed experiment is one that does not advance the body of human knowledge. Negative results still let us know that one thing was tried, and it didn't work--it crosses off one line in a process of elimination. And those that fudge data and methods to produce the appearance of a result, and those that are so flawed as to be non-reproducible, are failures in that sense, even if they still allow for the career advancement of a few people.

Post reply on HN