Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

271–280 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#271

Post by escot »

petra wrote: post #193
hyperion2010 wrote: post #183

Earlier quoted context omitted.

My PhD work is trying to address this by developing better ways for scientists to record and communicate their methods/protocols. Methods sections are NOT about providing the information needed to reproduce the work, often even the supplemental information is insufficient. They are a best guess at what needs to be done, and the methods section is often massively condensed due to editor demands and cites other papers…

Wouldn't it be hard to achieve deep consistency between experiments, in so many labs around the world, with so different conditions/cultures/etc , and when the experimenters aren't experts in consistency , but in their science ? Wouldn't it be better to use something like a cloud biology model - where you define experiments via code, CRO's compete on consistency(and efficiency and automation) and since they probably…

I work at a cloud bio lab. We run all of our experiments on automation, and all protocols must be defined in code. The latter is both the power and the difficulty -- when your protocol is defined in code it is explicit. However, writing code is both new and sometimes difficult for the scientists that we currently work with (molecular biology, drug discovery). I believe what we are doing is the right model. But it comes with this overhead of transitioning assays to code, so there is that against it. This is mostly just a matter of time though. Another nice thing about code is that you can't tweak it once it's running. You can define your execution and analysis up front to guard against playing with results down the road. Now that being said, there still needs to be a significant change to how research is funded and viewed by the public because pure tech solutions can't solve everything. Our tech can't decide what you pick yo research. It can't dish out grants to the truly important research. So it will take many angles to really solve any portion of this problem.

Re: Most scientists 'can't replicate studies by their peers'

#272

Post by jsweojtj »

dekhn wrote: post #53
tinco wrote: post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

I published in nature. The results were based on a computation that was so large, no other entity in the world could have reproduced it at the time (I work for Google, we built a system that used 600+Kcores continuously for years). I don't think ,at the time, that anybody other than Amazon or Microsoft could have run the same calculation as us. Should our publication have been prevented simply because nobody else had…

There are two ways that things can be reproducible: reproducible in practice and reproducible in principle. You are stating that you have a computation that is reproducible in principle (just in practice, unfeasible).

I find this an easy and useful distinction and your publication should NOT be prevented from being published by this measure.

Re: Most scientists 'can't replicate studies by their peers'

#273

Post by marcosdumay »

dwringer wrote: post #244
logfromblammo wrote: post #130

Earlier quoted context omitted.

This quandary is a great argument for never becoming an academic researcher. It would seem to me that science needs to eliminate "career bias" or "mortgage bias" or "ramen bias" from its results. Negative results need to be just as publishable as significant results. If I were tyrant of the Ivory Tower, I would decree that results be blinded until after acceptance for publication. I would further decree that prestige…

Prestige ... points?

The promise of doing really cool work.

What also increases the rate of people choosing a at the first question.

Re: Most scientists 'can't replicate studies by their peers'

#274

Post by SFJulie »

Why are they called scientists then?

If you can't write a good modus operandi you are just a fraud.

If you cannot reproduce a well writen modus operandi, you are a fraud.

Thus take the failures divide it by 2 (odds wrong articles are as much as wrong manipulators) and you got a 33% of lousy scientists

Re: Most scientists 'can't replicate studies by their peers'

#275

Post by vacri »

Balgair wrote: post #153
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

A lot of the commenters are trying to come up with technical solutions to the issue. It belies their incomplete study of the 'softer' sciences and the difficulty of biological studies. Tech solutions won't work, bureaucratic solutions wont work, more data wont work. Guys, the bio side of things is incredibly complicated and trying to set controls that are achievable in your time and budget is the heart of these field…

> Guys, the bio side of things is incredibly complicated

I always find it amusing when physicists talk about how mind-blowing it is that at the quantum level, things aren't entirely predictable. Over in biology, that's the starting point for everything rather than the final frontier - you don't need ridiculously expensive tools to get to the point where you're finding unpredictable stuff.

Re: Most scientists 'can't replicate studies by their peers'

#276

Post by vacri »

dorgo wrote: post #256
entee wrote: post #230

Earlier quoted context omitted.

Respectfully, I think you're missing the complications outlined in the rest of the thread. The difficulty in the biosciences and even in chemistry and some physics is that they can't really be formalized in the way that is necessary to construct these "chains of proof". The scientific literature is an ongoing conversation anchored by rigorous experimental facts and data. But rigorous doesn't mean it's clean like a ma…

I thought that statistics is the right tool to handle uncertainity? I had the impression that all the "soft" science is based on statistics. We can't prove that smoking kills people. But, given a certain confidance interval (or whatever equivalent measure for baysian statistics?), we can state that smoking is not a good idea if you want to live longer than mean expected years... Sure, some unforseen event may interfe…

> I had the impression that all the "soft" science is based on statistics.

In medicine, case studies are often used for low n issues. There are too many variables for meaningful statistics to be pulled out, but a "this patient had X, we did Y, Z happened" is still a way to pass on observational information. It's recognised that case studies aren't ideal, but it's still better than not passing on information at all.

Re: Most scientists 'can't replicate studies by their peers'

#277

Post by vacri »

cwyers wrote: post #133
jonlucc wrote: post #129

Earlier quoted context omitted.

You can add to this that if you are in a facility with both rats and mice, you should never house them in the same room, and you should probably not enter a mouse room after being in a rat room. In the wild, rats are predators of mice.

But in the wild, there aren't separate rooms for predators and prey, either. If laboratory rodent experiments are so reliant on such synthetic conditions, what's the point?

> But in the wild, there aren't separate rooms for predators and prey, either.

In the wild, there are plenty of mice that aren't in the vicinity of rats.

Re: Most scientists 'can't replicate studies by their peers'

#278

Post by hackuser »

hyperion2010 wrote: post #183
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

My PhD work is trying to address this by developing better ways for scientists to record and communicate their methods/protocols. Methods sections are NOT about providing the information needed to reproduce the work, often even the supplemental information is insufficient. They are a best guess at what needs to be done, and the methods section is often massively condensed due to editor demands and cites other papers…

How about recording videos of the lab? People trying to reproduce the experiment can just sift through the video. That may be tedious but it's far better than nothing.

Just a little metadata would help: Experiment A, Phase N, Day X

Re: Most scientists 'can't replicate studies by their peers'

#279

Post by hackuser »

mattkrause wrote: post #242
jamessb wrote: post #58

Earlier quoted context omitted.

> Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication. A 'world view' column in Nature suggested the same things last week [1]; the author described a paper of theirs [2]: > Yes, visual evidence can be faked, but a few simple safeguards should be enough to prevent that. Take a typical experiment in my field: using a tank of flowing water t…

I don't think this is particularly practical. Most experiments run for years (literally) and no one is going to record or archive, let alone watch, years of footage to confirm that one paper is legit. A brief experiment showing the apparatus and the collection of a few data points might be helpful for understanding the paper, but I can't see using it to verify a non-trivial experiment.

> no one is going to record or archive, let alone watch, years of footage

Recording and storing years of footage shouldn't be a significant problem with modern tech.

Nobody has to watch years of it; they can watch the parts they are interested in. They also can watch at 4x and search, as needed.

Re: Most scientists 'can't replicate studies by their peers'

#280

Post by hackuser »

SubiculumCode wrote: post #251
jamessb wrote: post #58

Earlier quoted context omitted.

> Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication. A 'world view' column in Nature suggested the same things last week [1]; the author described a paper of theirs [2]: > Yes, visual evidence can be faked, but a few simple safeguards should be enough to prevent that. Take a typical experiment in my field: using a tank of flowing water t…

For those that study human subjects, releasing video's of the subject is not going to happen any time soon. Participants have rights, and anomyity is an important one.

Blur their faces? Some experiments might depend on seeing faces, but not all. Plus, you would at least have video of everything the experimenters do, if not the results.
Post reply on HN