Most scientists 'can't replicate studies by their peers' • Hacker News Board

Re: Most scientists 'can't replicate studies by their peers'

#201

Post by 2xlbuds » Thu, Feb 23, 2017, 7:30 PM UTC

bradjohnson wrote: post #184

wycx wrote: post #149

Earlier quoted context omitted.

The way you get funded is by being able to do things that others can't do, or by being better at some technique than everyone else. Giving away all your hard won tricks by putting all of them in a methods section takes away your advantage come funding time.

What's the point of funding something that won't help anyone because the author won't tell you how they did it? How is any conclusion that they've come to useful if nobody can verify if it's correct?

At that point it's just like funding magicians to come up with new tricks.

Re: Most scientists 'can't replicate studies by their peers'

#202

Post by tnecniv » Thu, Feb 23, 2017, 7:33 PM UTC

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

Heck, I've tried to reproduce CS papers with incomplete methodologies. There's always enough that that I know it will work, but they normally don't include all the tricks you need to get it to work easily or efficiently. Stuff like "we used an SMT solver with this important feature" but no mention of which solver or "a key part of this algorithm is factoring the sparse matrix with this non-standard technique that you'll have to implement yourself" but don't tell you the number of tricks they used to make it as fast as you need.

Re: Most scientists 'can't replicate studies by their peers'

#203

Post by maverick_iceman » Thu, Feb 23, 2017, 7:34 PM UTC

I'm not sure that this is necessarily a problem. Most working scientists know that preliminary studies are often wrong and just because it is published in some journal doesn't mean it's true. If there's merit to it then the scientific method will eventually settle upon the correct answer. This looks like someone outside the field just came across this news and had a visceral reaction, "Gasp! Most journal articles are wrong! We can't trust science!"

(Note that I'm talking only about hard natural sciences. Social sciences are another whole can of worms.)

Re: Most scientists 'can't replicate studies by their peers'

#204

Post by mistermann » Thu, Feb 23, 2017, 7:35 PM UTC

sambe wrote: post #51

mistermann wrote: post #30

Earlier quoted context omitted.

> will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts Case in point: the very first thing I thought of is, does this have any relevance to the field of climate science! So....does it? Because we're told the reason we have to get on board with the program is because the people telling us the facts are scientists , and scientists are smart and trustworthy . However…

Is this an intentional straw man? No, you're not told to trust the people because they have a certain job title and trustworthiness. You're told to look at the data, which is overwhelming and consistent across multiple years and teams. It is the exact opposite of lack of reproducibility.

> Is this an intentional straw man? No, you're not told to trust the people because they have a certain job title and trustworthiness.

Are you joking? Are you seriously making the claim that one of the persuasive approaches used in the "public realm" (media, discussions, etc) isn't that we should fight climate change because scientists have almost unanimously decided it is a real thing and we must do something?

If scientists are telling us something, we sure as hell should listen, at least two reasons being they are the experts on the subject (why wouldn't you listen to experts), and the subject is so immensely complicated that an average non-scientist person wouldn't have a chance of "looking at the data" and forming a reasonably correct opinion.

But now you are telling me no one is suggesting I listen to scientists? I could easily google thousands of articles/papers/blog posts/internet discussions where people are doing just that, but you are telling me no, that content does not exist.

What is it about this topic where otherwise reasonable people seem to go off the rails?

Re: Most scientists 'can't replicate studies by their peers'

#205

Post by hyperion2010 » Thu, Feb 23, 2017, 7:36 PM UTC

petra wrote: post #193

hyperion2010 wrote: post #183

Earlier quoted context omitted.

My PhD work is trying to address this by developing better ways for scientists to record and communicate their methods/protocols. Methods sections are NOT about providing the information needed to reproduce the work, often even the supplemental information is insufficient. They are a best guess at what needs to be done, and the methods section is often massively condensed due to editor demands and cites other papers…

Wouldn't it be hard to achieve deep consistency between experiments, in so many labs around the world, with so different conditions/cultures/etc , and when the experimenters aren't experts in consistency , but in their science ? Wouldn't it be better to use something like a cloud biology model - where you define experiments via code, CRO's compete on consistency(and efficiency and automation) and since they probably…

1. Yes, but the idea would be that if you provide a way to communicate the variables that actually matter for consistency then you can increase the robustness of a finding. If you have one lab that can _always_ produce a result, but no one else can, then clearly we do not really understand what is going on and one might not even be willing to call the result scientific.

2. Maybe not better, but certainly more result oriented. Core facilities do exist right now for things like viral vectors and microscopy (often because you do need levels of technical expertise that are simply not affordable in single labs). If there were a way to communicate how to do experiments more formally then the core facilities could expand to cover a much wider array of experiment types. You still have to worry about robustness, but if you have multiple 'core' facilities that can execute any experiment then that issue goes away as well. The hope of course is that individual labs as they exist today (perhaps with an additional computational tint) would be able to actually replicate each other's result, because we will probably end up needing nearly as many 'core' facilities as we have labs right now, simply because the diversity of phenomena that we need to study in biology is so high.

Re: Most scientists 'can't replicate studies by their peers'

#206

Post by mistermann » Thu, Feb 23, 2017, 7:37 PM UTC

codr4life wrote: post #105

tunap wrote: post #69

Earlier quoted context omitted.

If you can still, edit/remove the presumption of falsification and your argument has merit. I too percieve group think in climate change narratives, while not necessarily disbelieving the data. We are wrecking our environment, exact detrimental effects are still uncertain.

I'm sorry, but given that we're gold fish looking out of the bowl from the inside; trying to predict the future from vague memories of the immediate past; we have no clue what's causing what in this mess. Any one who claims otherwise is an ignorant prostitute, doesn't matter if they're aware or not. There are huge monetary incentives to pushing this agenda, and plenty of bullying to get people to walk in line. Have a…

We have no clue whatsoever? Hmmmmmm.

Re: Most scientists 'can't replicate studies by their peers'

#207

Post by wuschel » Thu, Feb 23, 2017, 7:37 PM UTC

petra wrote: post #193

hyperion2010 wrote: post #183

Earlier quoted context omitted.

My PhD work is trying to address this by developing better ways for scientists to record and communicate their methods/protocols. Methods sections are NOT about providing the information needed to reproduce the work, often even the supplemental information is insufficient. They are a best guess at what needs to be done, and the methods section is often massively condensed due to editor demands and cites other papers…

Wouldn't it be hard to achieve deep consistency between experiments, in so many labs around the world, with so different conditions/cultures/etc , and when the experimenters aren't experts in consistency , but in their science ? Wouldn't it be better to use something like a cloud biology model - where you define experiments via code, CRO's compete on consistency(and efficiency and automation) and since they probably…

There are already approaches in this direction e.g. such as providing a standardized experimental hardware-software interface with Antha [1]. The complexity of the problems in questions (biological domain, biophysical, biochemical) is daunting - we do not understand many things, "there is plenty of room at the bottom".

[1] https://www.antha-lang.org/

Re: Most scientists 'can't replicate studies by their peers'

#208

Post by hellofunk » Thu, Feb 23, 2017, 7:40 PM UTC

I noticed this phenomenon recently when reading a variety of articles on evolutionary programming techniques. Each paper would present its benchmarks in comparison to other techniques, which were also described in other papers. Often, a paper would say "we could not reproduce the results in X paper."

Re: Most scientists 'can't replicate studies by their peers'

#209

Post by BeetleB » Thu, Feb 23, 2017, 7:41 PM UTC

thehardsphere wrote: post #167

BeetleB wrote: post #104

Earlier quoted context omitted.

>That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. No, no, a thousand times no! Most studies do not have follow up studies that confirm/refute the original. Often such a followup study is hard to publish. If you manage to reproduce it, you cannot publish unless it presents a new finding. If you fail to reproduce it, it often doesn't get publishe…

>>That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. >No, no, a thousand times no! Most studies do not have follow up studies that confirm/refute the original. Often such a followup study is hard to publish. If you manage to reproduce it, you cannot publish unless it presents a new finding. If you fail to reproduce it, it often doesn't get publis…

>Sorry, let me be clear: If an interesting result is published, people will go to the trouble. Most results are of limited interest and mediocre.

If it's in a journal, it is interesting. Journal editors will require "interesting" as a prerequisite to publishing a paper. Papers do get rejected for "valid work but not interesting".

If journals are publishing papers that are of limited interest, then there is a serious problem with the state of science.

I'm not trying to pick hairs. One way or other, there is a real problem - either journals are not being the appropriate gatekeepers (by allowing uninteresting studies), or most interesting studies are not being replicated.

Re: Most scientists 'can't replicate studies by their peers'

#210

Post by BeetleB » Thu, Feb 23, 2017, 7:44 PM UTC

thehardsphere wrote: post #171

BeetleB wrote: post #104

Earlier quoted context omitted.

>That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. No, no, a thousand times no! Most studies do not have follow up studies that confirm/refute the original. Often such a followup study is hard to publish. If you manage to reproduce it, you cannot publish unless it presents a new finding. If you fail to reproduce it, it often doesn't get publishe…

> When I highlighted this to people in the field, they were quite open with another explanation: It helps them because they do not want their peers to know all the details. That allows them to have an edge over their peers and they do not need to race with them to publish further studies. I have always suspected that, but I've never heard anyone be that open about it. At a previous job I had to implement various algo…

>I have always suspected that, but I've never heard anyone be that open about it.

Yes, they'll just say they expect their peers to be competent enough to reproduce, and a paper shouldn't be filled with such trivialities.

To get the real answer, talk to their grad students. Especially those who are aspiring for academic careers. They tend to be quite frank on why they will act like their advisors.

Oh, and citing a whole book for optimization - that kind of thing is quite common. "We numerically solved this complex PDE using the method of X" and then just give a reference to a textbook. But usually the algorithm to implement is sensitive to tiny details (e.g. techniques to ensure tiny errors don't grow to big ones, etc).