Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

301–310 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#301

Post by jszymborski »

Balgair wrote: post #153
nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

A lot of the commenters are trying to come up with technical solutions to the issue. It belies their incomplete study of the 'softer' sciences and the difficulty of biological studies. Tech solutions won't work, bureaucratic solutions wont work, more data wont work. Guys, the bio side of things is incredibly complicated and trying to set controls that are achievable in your time and budget is the heart of these field…

While we're sharing confusing pathways, here's a visualisation of _some_ of the known metabolic pathways:

http://biochemical-pathways.com/#/map/1

Re: Most scientists 'can't replicate studies by their peers'

#302

Post by jonlucc »

SilasX wrote: post #245
PeterisP wrote: post #217

Earlier quoted context omitted.

The proposal has a problem in that it will increase the quality of results but will enormously slow down progress. E.g., looking from the perspective of a scientist-reader, if someone has spent a few months doing the ad hoc hands on experiments and achieved interesting observations, then I would want them to publish that research now, instead of spending another half a year to make the repeatable procedure or possibl…

Then maybe the problem is that the public is expecting results to be actually true, exacerbated by "hey what peer-reviewed literature are you supporting your argument with". If peer-reviewed literature is just a scratchpad for ad hoc ideas that may turn into something legit later, then the current standard is good enough, and we shouldn't be worrying that most of them are false. OTOH, it's a problem if people are bas…

I think this is the case, and it has been mentioned elsewhere in this thread. When I see a paper published that I am interested in, I have to fit it into the context what I already know about a field, the standards of the the journal its published in, sometimes the origin of the paper (some labs are much less sloppy than others), and other factors.

For a recent personal example, a company published a paper saying that if you give a pre-treat with 2 doses of a particular drug, you can avoid some genetic markers of inflammation that are in the bloodstream and kidneys. Well, I looked at the stimulus and ordered some of my own from a different manufacturer that was easier to obtain and gave it to some mice with and without pretreatment by their compound. Instead of looking at the genes they looked at, I looked at an uptick in a protein expected to be one step removed from the genes they showed a change in. Well I haven't exactly replicated their study, but I've replicated the core points: stimulus with a the same cytokine gives a response in a particular pathway and it either is or isn't mitigated by the drug or class of drugs they showed. Now, my study took 2 days less than theirs, but it worked well enough that I don't need to fret the particular details I did differently from them. If my study didn't work, I could either decide that the study isn't important to me if it didn't work my way or go back a step and try to match their exact reagents and methods.

So yes, I do think the news industry picks up stuff too quickly sometimes, but depending on the outlet, they tend to couch things in appropriate wiggle words (may show, might prove, could lead to, add evidence, etc).

Re: Most scientists 'can't replicate studies by their peers'

#303

Post by CodeWriter23 »

skosuri wrote: post #79
thehardsphere wrote: post #74

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how generalizable such results are. If the results don't hold in slightly altered systems, it falls into the wastebin of ideas.

> No scientist treats published studies as gospel.

Why then do so many scientists in so many different fields insist "the science is settled"?

Re: Most scientists 'can't replicate studies by their peers'

#304

Post by wycx »

comicjk wrote: post #240
wycx wrote: post #111

Earlier quoted context omitted.

That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others. People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...

That's an absurd degree of cynicism. I just finished my PhD in chemical engineering, and I did the reverse of this. In fact, I successfully migrated the younger members of my research group to a completely open-source software stack for chemical simulations, so that in principle anyone with a Linux box could reproduce our results. We published all our code and plain text versions of the data. I'm not saying you're wr…

I am writing a grant proposal right now. The success rate is ~19%. Personnel (i.e. publication record) of the team is weighted at 40% of the proposal. If you don't have an H-index of >50, there is no incentive to share with other that which gives you an advantage in the science section.

Re: Most scientists 'can't replicate studies by their peers'

#305

Post by averagewall »

panic wrote: post #227
epistasis wrote: post #173

Earlier quoted context omitted.

I think there's also a lot of confusion about what the scientific literature should be. Outsiders think of journals as pre-packaged nuggets of science fact, with conclusions that you can read and trust. Scientists who publish in the journals view them as a way to communicate a body of work of "I did this, here is the result I saw. I think it might mean this." The difference between these two views can not be understa…

Scientists and academic institutions add to the confusion with the way they promote their work. Nobody puts language like "I did this, here is the result I saw. I think it might mean this" in a press release. It's always "Breakthrough research! Is the end of cancer in sight??"

It doesn't help when teachers tell their students they can reference academic papers from peer reviewed journals but can't reference Wikipedia. The former is tentative and the latter is established knowledge from text books, more or less.

Re: Most scientists 'can't replicate studies by their peers'

#306

Post by wycx »

wycx wrote: post #149
michaelmrose wrote: post #115

Earlier quoted context omitted.

At that point its actually not even science and shouldn't be funded

The way you get funded is by being able to do things that others can't do, or by being better at some technique than everyone else. Giving away all your hard won tricks by putting all of them in a methods section takes away your advantage come funding time.

It is more subtle than you think. It is not that you give no information about how to do things, you lay out all the steps that you took in your methods section. An example of keeping all tricks to yourself is that you do not tell others about the 100 small thing you found out, the hard way, that you should avoid doing. i.e. You explicitly say in the methods section these are the steps I took, but what others really need to know is why you ended up doing all the tiny tiny things it the particular way that you did. In many cases, you could write pages and pages about all these reasons. All these little tricks add up to much greater efficiency. Good experimentalists are the ones that have already made all the mistakes.

Edit: This is additional context for the commenters below.

Re: Most scientists 'can't replicate studies by their peers'

#307

Post by apathy »

vacri wrote: post #275
Balgair wrote: post #153

Earlier quoted context omitted.

A lot of the commenters are trying to come up with technical solutions to the issue. It belies their incomplete study of the 'softer' sciences and the difficulty of biological studies. Tech solutions won't work, bureaucratic solutions wont work, more data wont work. Guys, the bio side of things is incredibly complicated and trying to set controls that are achievable in your time and budget is the heart of these field…

> Guys, the bio side of things is incredibly complicated I always find it amusing when physicists talk about how mind-blowing it is that at the quantum level, things aren't entirely predictable. Over in biology, that's the starting point for everything rather than the final frontier - you don't need ridiculously expensive tools to get to the point where you're finding unpredictable stuff.

You don't need a ridiculously expensive rig to find that in physics, either: https://en.wikipedia.org/wiki/Double-slit_experiment

laser pointer plus a sharp enough carbide wheel

(trained as a chemist, grad school in biostats & genetics, now mostly design experiments & clinical trials... I have physics envy, except I don't envy their funding models!)

Re: Most scientists 'can't replicate studies by their peers'

#308

Post by vivekchandsrc »

As a experimental researcher for the last 16 years, I don't find this surprising. Publishing industry has becoming a scam. Wherein journals like Nature or its sister journals are charging $1000s to be open access. They don't care whether it is reproducible....

Re: Most scientists 'can't replicate studies by their peers'

#309

Post by escot »

petra wrote: post #293
escot wrote: post #271

Earlier quoted context omitted.

I work at a cloud bio lab. We run all of our experiments on automation, and all protocols must be defined in code. The latter is both the power and the difficulty -- when your protocol is defined in code it is explicit. However, writing code is both new and sometimes difficult for the scientists that we currently work with (molecular biology, drug discovery). I believe what we are doing is the right model. But it com…

I do agree, it seems like right model, and will have a large impact. Between automating labor, economies of scale in purchasing, and access to more efficient technology(like acoustic liquid handling) ,etc - isn't it just a matter of time before cloud biology becomes quite cost effective and combined with other benefits - it would be the only way that makes sense to do research, so funding will naturally go there? Als…

> it would be the only way that makes sense to do research There will certainly be more than just one way, although I hope cloud labs are the front runner. Also cloud and automated are two separate concepts. We do both, but there's no reason that you can't just do one or the other. The automation is critical for reproducibility for many reasons. But I think the cloud aspect is mostly helpful from a business perspective -- it makes it easier on everyone to get up and running on our system. But there are many in lab automation solutions that are helping fight the reproducibility crisis. And on the flip side, there are cloud labs that aren't automated.

> do you see a way to add the extreme versatility of the biology lab into a cloud service We let you run any assay that can be executed on the set of the devices that we have in our automated lab. So in that sense, yes its very flexible. Also, there's no need to run your entire workflow in the cloud. You can do some at home, some in the cloud. Some people even string together multiple cloud services into a workflow. See https://www.youtube.com/watch?v=bIQ-fi3KoDg&t=1682s

That being said, biology labs can be crazy places. Part of what we do is put constraints on what can be encoded in each protocol to reduce the number of hidden variables. Every parameter that counts must be encoded in the protocol, because once you hit "go" on the protocol, it could run possibly on any number of different devices each time it runs. The only constant is that the exact instructions specified in the protocol will be run on the correct device set.

Re: Most scientists 'can't replicate studies by their peers'

#310

Post by hackuser »

hyperion2010 wrote: post #299
hackuser wrote: post #287

Earlier quoted context omitted.

I didn't mean record it to use the data in your analysis, but record it to preserve the methods for others. If they have trouble getting part of the experiment to work, they can pull up the video and see how you did it (at least to a degree; I'm not expecting 360 video. You can't possible record in text everything a video could capture. Thanks for sharing your knowledge and experience in this discussion, by the way.…

Ah, yes, things like JOVE [0] are definitely useful but they don't seem to scale to the sheer number of protocols that need to be documented (eg a single JOVE publication is exceedingly expensive). I have also heard from people who have tried to record video of themselves doing a protocol is that it is very hard to make them understandable for someone else. That said if the 'viewer' is highly motivated videos of any…

> if the 'viewer' is highly motivated videos of any quality could be invaluable

That's what I meant. Just stick some cameras in the ceiling (or wherever is best) and capture what you can. It seems cheap and better than nothing, but I know nothing about biological research.

Post reply on HN