Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

251–260 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#251

Post by SubiculumCode »

jamessb wrote: post #58
JumpCrisscross wrote: post #40

Earlier quoted context omitted.

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

> Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication. A 'world view' column in Nature suggested the same things last week [1]; the author described a paper of theirs [2]: > Yes, visual evidence can be faked, but a few simple safeguards should be enough to prevent that. Take a typical experiment in my field: using a tank of flowing water t…

For those that study human subjects, releasing video's of the subject is not going to happen any time soon. Participants have rights, and anomyity is an important one.

Re: Most scientists 'can't replicate studies by their peers'

#252

Post by SubiculumCode »

TheSpiceIsLife wrote: post #181
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

Could this be addressed by referencing a standard mouse handling document, something similar to: Pre-experiment mouse handling procedures as outlined in section M.1.2 and M.2.4 of the Mouse Handling Procedure Standards Body Publication of 2014 Which document could have any number of sections detailing the finer points variable control for the relevant experiment.

There is continual progress on standardization of methodology, but it takes a while. People have vested interests in using their protocols, and there often valid disagreements about what is actually the best protocol.

Nevertheless, these kinds of standardization documents are important. But only as long as deviations from these standards are not discounted when justified theoretically.

Re: Most scientists 'can't replicate studies by their peers'

#253

Post by rokosbasilisk »

DanBC wrote: post #42
eclee2 wrote: post #16

Earlier quoted context omitted.

If there are errors in a study's methods that make it unreplicable, then it shouldn't have passed peer review or been published.

How do you know it's unreplicable until someone tries to replicate it?

I work with principal investigators phd/mds at upenn automating some of their data analysis pipelines.

They have all have secret checklists for bs detection in papers they read. Certain labs set off red flags to them or certain techniques being too fuzzy or easy to mess up.

Every one seems to have their own heuristics, and no one seems to take any article at face value anymore.

I hear PI's say stuff is unreplicable all the time.

Re: Most scientists 'can't replicate studies by their peers'

#254

Post by IanCal »

cwyers wrote: post #164
IanCal wrote: post #137

Earlier quoted context omitted.

The point, I think, is that you want to control for things that really alter the outcome. It's not that a clean cage means that the end-product no longer works, but you want to compare "didn't have drug" vs "did have drug" not "didn't have drug" vs "did have drug but the fire alarm went off this morning". Unless your effect size is much larger than all the noise, you could easily miss a true result. Worse, you could…

So, let's continue with your example of a drug being given to the rodents. Now, you want to control diet -- you don't want your control group eating differently than your treatment group. This ends up being relatively easy to do with laboratory rodents, they aren't popping down to McDonalds for lunch or lying in their food diaries, like human subjects might. You buy your rodent food from one rodent pellet supplier, a…

Yes but the things they mentioned would be bizarre to control for the other way.

Would you expect the tests to be done, then again but with both rats hearing the person upstairs installing a new microscope, then with a regular fire alarm, etc?

Describing the food given is quite a step away from describing how frequently the fire alarm goes off.

Re: Most scientists 'can't replicate studies by their peers'

#255

Post by SubiculumCode »

busyant wrote: post #185
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

I can't tell you how many times I tried (unsuccessfully) to replicate a procedure using the methods section in a research article. Then I contact the author and they send me a protocol that is a 'distant cousin' to what I read in the published article. This caused me to assiduously report methods when I wrote papers.

That is frustrating. In my field of cognitive neuroscience, there is often little incentive for a researcher to hide parts of their methods from their 'competitors' since, for example, my memory study is not about reporting but protecting a new technology. Indeed, prestige is often a consequence of others adopting your methods, so researchers are motivated to share scripts, and report the methods fully. HOWEVER, certain journals can place limits on the length of the methods section, which is a damn shame.

Re: Most scientists 'can't replicate studies by their peers'

#256

Post by dorgo »

entee wrote: post #230
ivan_gammel wrote: post #194

Earlier quoted context omitted.

Probably, the real problem is that "the scientific literature should be". It's not the right question to discuss today, because scientific literature is just a brief approximation of scientific knowledge and this is not what we need. Whole modern scientific system is outdated and has to be upgraded: journals, citation indexes, degrees should go away and be replaced. Science should become more formal and more digital,…

Respectfully, I think you're missing the complications outlined in the rest of the thread. The difficulty in the biosciences and even in chemistry and some physics is that they can't really be formalized in the way that is necessary to construct these "chains of proof". The scientific literature is an ongoing conversation anchored by rigorous experimental facts and data. But rigorous doesn't mean it's clean like a ma…

I thought that statistics is the right tool to handle uncertainity? I had the impression that all the "soft" science is based on statistics. We can't prove that smoking kills people. But, given a certain confidance interval (or whatever equivalent measure for baysian statistics?), we can state that smoking is not a good idea if you want to live longer than mean expected years... Sure, some unforseen event may interfere with your experiment. But statistics should account for that, shouldn't it? Just wondering..

Re: Most scientists 'can't replicate studies by their peers'

#257

Post by coldtea »

michaelmrose wrote: post #115
wycx wrote: post #111

Earlier quoted context omitted.

That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others. People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...

At that point its actually not even science and shouldn't be funded

Lots of insights/technologies you use today were produced by scientists working in such and even more secretive conditions, even with public funding involved (e.g. nobody at the Manhattan project would broadcast their discoveries to the world and same for many other fields, not necessarily war related).

Re: Most scientists 'can't replicate studies by their peers'

#258

Post by Obi_Juan_Kenobi »

JumpCrisscross wrote: post #40
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

The 'unknown unknowns' will never be eliminated. Certainly it's worthwhile to try to improve methods communication, but there are limits.

You really have to try it yourself before you can understand the degree of troubleshooting that's required of a good experimentalist. You could have scientists live-streaming all their work, and you'd still have the same issues you do now. Even a simple experiment, something most wouldn't blink an eye at, has dozens and dozens of variables that could influence the result. The combinatoric complexity is staggering. The reality is that you try things, find some that work, and then convince yourself that it's a real result with some further work.

The methods that endure are the ones that replicate well and work robustly. Molecular biology is still built on Sanger sequencing, electrophoresis, and blotting, all in the context of good crossing and genetics, because that's what works. Some of the genomic tools are starting to get there, I'd venture to say that RNAseq is reasonably standardized and robust at this point. Interpreting genomic data is another story...

Re: Most scientists 'can't replicate studies by their peers'

#259

Post by SubiculumCode »

zeristor wrote: post #3

Hmmm. Software Engineering has Continuous Integration, since it is so expensive to fix software later in the day. Is there any such thing as Continuous Reproducibility? Constantly checking that the science can be reproduced? How prevalent is this in different branches of Science?

Well often in behavioral sciences, the alternative hypothesis in experiment A is the null hypothesis in the follow-up, experiment B. If the results in alternative A do not act like the results in null in experiment B, then there is a failed replication. The researcher observing this, begins to lose confidence in the earlier finding, and pursues other ideas. This is the kind of implicit replications currently supported by funding agencies.
Post reply on HN