Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

51–60 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#51

Post by sambe »

mistermann wrote: post #30
AlexB138 wrote: post #23

This is ego, politics and career ambitions undermining modern science. Unfortunately, the fact that this is occurring so rampantly will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts. This is a systemic failure of basic ethics that will hurt us all. The success-at-all-cost career mindset is toxic in all tracks, but this is one of the most dangerous for it to take…

> will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts Case in point: the very first thing I thought of is, does this have any relevance to the field of climate science! So....does it? Because we're told the reason we have to get on board with the program is because the people telling us the facts are scientists , and scientists are smart and trustworthy . However…

Is this an intentional straw man? No, you're not told to trust the people because they have a certain job title and trustworthiness. You're told to look at the data, which is overwhelming and consistent across multiple years and teams. It is the exact opposite of lack of reproducibility.

Re: Most scientists 'can't replicate studies by their peers'

#52

Post by neffy »

mistermann wrote: post #30
AlexB138 wrote: post #23

This is ego, politics and career ambitions undermining modern science. Unfortunately, the fact that this is occurring so rampantly will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts. This is a systemic failure of basic ethics that will hurt us all. The success-at-all-cost career mindset is toxic in all tracks, but this is one of the most dangerous for it to take…

> will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts Case in point: the very first thing I thought of is, does this have any relevance to the field of climate science! So....does it? Because we're told the reason we have to get on board with the program is because the people telling us the facts are scientists , and scientists are smart and trustworthy . However…

A deliberately sceptical person goes and reads the climate literature, and forms their own opinion based on all the information available.

... and then they move to high ground.

Re: Most scientists 'can't replicate studies by their peers'

#53

Post by dekhn »

tinco wrote: post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

I published in nature. The results were based on a computation that was so large, no other entity in the world could have reproduced it at the time (I work for Google, we built a system that used 600+Kcores continuously for years). I don't think ,at the time, that anybody other than Amazon or Microsoft could have run the same calculation as us.

Should our publication have been prevented simply because nobody else had the resources to replicate?

Re: Most scientists 'can't replicate studies by their peers'

#54

Post by sametmax »

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

Even if you do know all that, it means reproducing the experiment is costly. Controlling properly an environment take times and resources, and it's not like it's easy to get those for anything, let alone for something like peer reviewing.

Re: Most scientists 'can't replicate studies by their peers'

#57

Post by seszett »

dekhn wrote: post #53
tinco wrote: post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

I published in nature. The results were based on a computation that was so large, no other entity in the world could have reproduced it at the time (I work for Google, we built a system that used 600+Kcores continuously for years). I don't think ,at the time, that anybody other than Amazon or Microsoft could have run the same calculation as us. Should our publication have been prevented simply because nobody else had…

> Should our publication have been prevented simply because nobody else had the resources to replicate?

Well... maybe, yes?

Just because you are the only ones able to do the computation doesn't mean your methods or your results are correct.

Re: Most scientists 'can't replicate studies by their peers'

#58

Post by jamessb »

JumpCrisscross wrote: post #40
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

> Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

A 'world view' column in Nature suggested the same things last week [1]; the author described a paper of theirs [2]:

> Yes, visual evidence can be faked, but a few simple safeguards should be enough to prevent that. Take a typical experiment in my field: using a tank of flowing water to expose fish to environmental perturbations and looking for shifts in behaviour. It is trivial to set up a camera, and equally simple to begin each recorded exposure with a note that details, for example, the trial number and treatment history of the organism. (Think of how film directors use clapper boards to keep records of the sequence of numerous takes.) This simple measure would make it much more difficult to fabricate data and ‘assign’ animals to desired treatment groups after the results are known.

[1]: http://www.nature.com/news/science-lies-and-video-taped-expe... [2]: http://onlinelibrary.wiley.com/doi/10.1111/2041-210X.12668/f...

Re: Most scientists 'can't replicate studies by their peers'

#59

Post by mattnewton »

dekhn wrote: post #50

So. Having been a scientist, I observed an interesting phenomenon, several times. It's almost as if scientists enjoy leaving out critical details from the methods section, and other scientists enjoy puzzling out what the missing details are. I think there's this sort-of assumption of competence that for any reasonably interesting paper, the people in the field who are reading it, they have the level of skill to repro…

Do you think that has anything to do with the incentives for reviewers, who want shorter papers with a length limit and prefer more details on the impact/why of the experiment?

Does the same method-hiding hold true in journals without length limits or different review processes?

Re: Most scientists 'can't replicate studies by their peers'

#60

Post by codr4life »

sambe wrote: post #51
mistermann wrote: post #30

Earlier quoted context omitted.

> will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts Case in point: the very first thing I thought of is, does this have any relevance to the field of climate science! So....does it? Because we're told the reason we have to get on board with the program is because the people telling us the facts are scientists , and scientists are smart and trustworthy . However…

Is this an intentional straw man? No, you're not told to trust the people because they have a certain job title and trustworthiness. You're told to look at the data, which is overwhelming and consistent across multiple years and teams. It is the exact opposite of lack of reproducibility.

The only thing that is consistent and overwhelming about that data is the amount of energy put into falsifying and massaging it to support the chosen narrative.
Post reply on HN