Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

111–120 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#111

Post by wycx »

hannob wrote: post #90
ovis wrote: post #78

Earlier quoted context omitted.

I was chatting with a colleague last summer, and asking him for more details about a method he had published that I was having trouble reproducing. After pointing me to a few papers that didn't include the relevant details, hefinally told me that "if we told everyone about how it was done, anyone could do it." While at the time I was pretty upset with him, perhaps it's the competitive nature of science and funding th…

> I was chatting with a colleague last summer, and asking > him for more details about a method he had published that > I was having trouble reproducing. After pointing me to a > few papers that didn't include the relevant details, > hefinally told me that "if we told everyone about how it > was done, anyone could do it." That's not science, that's bullshit. Can you please expose that? Scientists shouldn't simply get…

That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others.

People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...

Re: Most scientists 'can't replicate studies by their peers'

#112

Post by cwyers »

carsongross wrote: post #72
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Think about the epistemological ramifications of this. This is why trust is so important in social contexts, even in science.

> Think about the epistemological ramifications of this.

This this this this this a million times this. If your experiment with lab rats needs to be done this finely to be replicated, is it really telling you anything about the real world? Because of course, we don't really care about the behavior of laboratory rats, right? Not in proportion to how often we do studies with laboratory rats versus other species of animals. We care because of what it can tell us about biology in general. And if the finding can't even generalize to "laboratory rats whose cages have been cleaned recently," does the study really say what it seems to?

Re: Most scientists 'can't replicate studies by their peers'

#113

Post by marknutter »

neffy wrote: post #52
mistermann wrote: post #30

Earlier quoted context omitted.

> will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts Case in point: the very first thing I thought of is, does this have any relevance to the field of climate science! So....does it? Because we're told the reason we have to get on board with the program is because the people telling us the facts are scientists , and scientists are smart and trustworthy . However…

A deliberately sceptical person goes and reads the climate literature, and forms their own opinion based on all the information available. ... and then they move to high ground.

By reproducing the results? Because I'm quite certain most, if not all people do not do that.

Re: Most scientists 'can't replicate studies by their peers'

#114

Post by edblarney »

codr4life wrote: post #105
tunap wrote: post #69

Earlier quoted context omitted.

If you can still, edit/remove the presumption of falsification and your argument has merit. I too percieve group think in climate change narratives, while not necessarily disbelieving the data. We are wrecking our environment, exact detrimental effects are still uncertain.

I'm sorry, but given that we're gold fish looking out of the bowl from the inside; trying to predict the future from vague memories of the immediate past; we have no clue what's causing what in this mess. Any one who claims otherwise is an ignorant prostitute, doesn't matter if they're aware or not. There are huge monetary incentives to pushing this agenda, and plenty of bullying to get people to walk in line. Have a…

I see your skepticism - at the same time - I take the data as 'crudely directional'.

'On the whole', it would rather seem we are doing some damage. We don't need perfect predictions to get that.

Second - is the issue of 'risk'.

If there were a 1% chance that your child would be kidnapped if you let them play at the park past 11pm, would you let them do it? No.

Given the level of existential risk inherent in climate change, even if there is a small chance that the climate-alarmists are correct, we basically have to confront the challenge.

Rationally - we should have a very low risk threshold for activities that constitute existential problems for us all.

I'm hugely skeptical of so many specific statements about climate change, especially the politicization and obvious 'group think' - it drives me nuts.

But at the end of the day - 'it looks like in general' there is a problem, and 'even if there is a small risk of it' - we have to do something about it.

Which is how I manage to swallow it all.

So we should take it 'with a grain of salt' but we have to take it, kind of thing.

Re: Most scientists 'can't replicate studies by their peers'

#115

Post by michaelmrose »

wycx wrote: post #111
hannob wrote: post #90

Earlier quoted context omitted.

> I was chatting with a colleague last summer, and asking > him for more details about a method he had published that > I was having trouble reproducing. After pointing me to a > few papers that didn't include the relevant details, > hefinally told me that "if we told everyone about how it > was done, anyone could do it." That's not science, that's bullshit. Can you please expose that? Scientists shouldn't simply get…

That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others. People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...

At that point its actually not even science and shouldn't be funded

Re: Most scientists 'can't replicate studies by their peers'

#116

Post by binaryzeitgeist »

Specifically to the CS context, I think some version of double blinded peer code-review should be made mandatory for a publication.

I've seen authors skip quite a many details that are quintessential to the replication process.

In short, if research is not replicable by the peer community it just useless, that's what it is.

Re: Most scientists 'can't replicate studies by their peers'

#117

Post by Fenrisulfr »

angry-hacker wrote: post #73
benrawk wrote: post #38

I am a social scientist studying human behavior, and this is a huge problem in the field. Myself and my statistician friends who analyze the literature have basically concluded that most extremely "novel" and "surprising" findings in the literature aren't even worth trying to replicate (remember, replications cost money to run, so before you start you have to make some judgment about the likelihood of success.) This…

Thank you. At first I thought puzzagate?! - I don't need that bs, but this is very different pizzagate. I once found a good blog about mental health and science, a lot of snakeoil shown about srris, adhd etc. but I'm unable to find it now. Can anyone help me out?

Is it thelastpsychiatrist.com ?

Re: Most scientists 'can't replicate studies by their peers'

#118

Post by baxman22 »

alexw1 wrote: post #107
AlexB138 wrote: post #23

This is ego, politics and career ambitions undermining modern science. Unfortunately, the fact that this is occurring so rampantly will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts. This is a systemic failure of basic ethics that will hurt us all. The success-at-all-cost career mindset is toxic in all tracks, but this is one of the most dangerous for it to take…

> This is ego, politics and career ambitions undermining modern science. When has this never been the case?

I totally agree. The third constant of life, after Death and Taxes, is Bullshit. It is preciciely why the scientific method was conceived, as a way to filter out the progress hindering bullshit that laces all passed-on information due to emebelishments brought on by ambition.

Re: Most scientists 'can't replicate studies by their peers'

#119

Post by bbctol »

JumpCrisscross wrote: post #40
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

The journal I work for just rolled out a new methods format; the big change is requiring all resources to be listed, with their origin (a major problem is that Chemical A or lab rat subspecies B might be very different from one company or another, even if they're theoretically identical.) What's really needed is broader standardization of all practices; requiring video wouldn't solve the issue that different labs might not know what another would think needed to be videotaped. No one is going to document and record the entire course of an experiment; that would literally be months of footage. We'd like to make sure everyone has the same understanding of procedures and techniques, but that requires communication not just through journals, but between scientists and academic institutions.

Re: Most scientists 'can't replicate studies by their peers'

#120

Post by SilasX »

JumpCrisscross wrote: post #40
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

Or require the author to only communicate to the technicians via the same means that will be published with the paper -- i.e. Make sure anyone reading the paper has the same spec for the experiment as those who performed the original. [1]

That would help prevent such "hidden specs" from entering the experiment.

[1] Note: this implies that authors cannot be part of the experiment or conduct it themselves, because they can't "pass on their identity" in a research paper, as would be necessary to put readers on par with the technicians.

Post reply on HN