Most scientists 'can't replicate studies by their peers' • Hacker News Board

Re: Most scientists 'can't replicate studies by their peers'

#341

Post by j7ake » Sat, Feb 25, 2017, 7:43 AM UTC

dragonwriter wrote: post #169

thehardsphere wrote: post #167

Earlier quoted context omitted.

>>That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. >No, no, a thousand times no! Most studies do not have follow up studies that confirm/refute the original. Often such a followup study is hard to publish. If you manage to reproduce it, you cannot publish unless it presents a new finding. If you fail to reproduce it, it often doesn't get publis…

> Sorry, let me be clear: If an interesting result is published, people will go to the trouble. That's only true for a definition of "interesting" that is more like the sense most people assign to "astounding" or "groundbreaking", and even then it's not guaranteed, just somewhat probable. If it's both groundbreaking and controversial (in the sense of "immediately implausible to lots of people in the domain, but still…

If a result opens up an entirely new paradigm, then you can bet there will be people trying to replicate the experiments.

Re: Most scientists 'can't replicate studies by their peers'

#342

Post by j7ake » Sat, Feb 25, 2017, 8:02 AM UTC

dekhn wrote: post #241

dekhn wrote: post #238

Earlier quoted context omitted.

I would say that most scientists, at the top of their game, who have years of trusted publications behind them, should be given some latitude to publish discovery science without the requirement of reproducibility, so long as they share their results in detail, and accept postdocs who then take the resulting research materials to their new labs. I didn't always believe this, but after spending a bunch of time reading…

For those interested in learning more about excellent scientists whose work could not be reproduced: Tom Cech and his grad students discovered that RNA can have enzymatic activity. They proved this (with excellent control experiments eliminating alternative hypotheses) and yet, the community completely denied this for years and reported failed replication, when in fact, the deniers were messing up their experiments b…

Thanks for the lisr. I think negative effects of labeling a paper irreproducible when in fact it is not is greater than the positive effect of correctly labeling a paper correct.

Because if a piece of work is important enough. People will find the truth. If it is not important enough.. then it does not matter.

Re: Most scientists 'can't replicate studies by their peers'

#343

Post by Too » Sat, Feb 25, 2017, 8:07 AM UTC

notzorbo3 wrote: post #218

bradjohnson wrote: post #184

Earlier quoted context omitted.

What's the point of funding something that won't help anyone because the author won't tell you how they did it? How is any conclusion that they've come to useful if nobody can verify if it's correct?

Who says the author(s) won't tell anybody? They only need to keep it secret from the competition, not the hand that feeds them.

This can definitely be the problem. I've seen studies where 50% of funding came from university and 50% from a private company. University required study to be published in their database, company wants the end result and don't want media to know what they are working with. So the report is obfuscated just enough to be legible for publication without giving away too much data from the company.

Especially for final masters degree projects this is very common as the students don't get paid by the university at all, so many try to find a company to sponsor. But the students still need the uni to publish the report for them to get their final degree so you get this conflict of interest again. Most of these reports are just written with the end goal of getting a degree, not of creating solid research, this really needs the stricter universities not letting through all that crap, for now they shouldn't really be trusted the same way as proper research papers.

Re: Most scientists 'can't replicate studies by their peers'

#344

Post by j7ake » Sat, Feb 25, 2017, 8:16 AM UTC

Tloewald wrote: post #110

Click bait from the BBC. 2/3 of scientists have failed to replicAte a study is not a statement that reinforces the headline. If 2/3 of people have had trouble sleeping that does not mean most people can't sleep. If you've ever tried a simple science experiment out of a book, it's easy to screw one up no matter how sound the underlying science because you messed something up. A bleeding edge study is several degrees o…

This comment is insightful and is in line with Mina bissels views.

The immediate questions when someone tries to replicate a study is... How hard did they try? Did they go to the lab and spend one year of their life learning the technique? Or did they try it once and didn't think hard enough about being careful?

Re: Most scientists 'can't replicate studies by their peers'

#345

Post by PeterisP » Sat, Feb 25, 2017, 11:38 AM UTC

SilasX wrote: post #245

PeterisP wrote: post #217

Earlier quoted context omitted.

The proposal has a problem in that it will increase the quality of results but will enormously slow down progress. E.g., looking from the perspective of a scientist-reader, if someone has spent a few months doing the ad hoc hands on experiments and achieved interesting observations, then I would want them to publish that research now, instead of spending another half a year to make the repeatable procedure or possibl…

Then maybe the problem is that the public is expecting results to be actually true, exacerbated by "hey what peer-reviewed literature are you supporting your argument with". If peer-reviewed literature is just a scratchpad for ad hoc ideas that may turn into something legit later, then the current standard is good enough, and we shouldn't be worrying that most of them are false. OTOH, it's a problem if people are bas…

Yes, the problem seems to be that the general public is expecting journals/conferences that are essentially implemented by a research community as a tool for their ongoing research workflow, to fit to the goals of informing the general public - but there reasonably would/should/must be a gap of something like a year (or many years) between the finding must be initially published so that others can work on that research, and the time when the finding is reasonably verified by other teams (which necessarily happens a significant time after it's been published) and thus is ready to be used for informing public policy.

It's like the stages of clinical research - there we have general standards on when it is considered acceptable to use findings for actually treating people (e.g. phase 3 studies), but it's obviously clear that we need to publish and discuss the initial findings since that's required to actually get to the phase 3 studies. However, the effects seen in phase 1 studies often won't generalize to something that actually works in clinical practice, so if general public reads them then often they'll assume predictions that won't come true.

Re: Most scientists 'can't replicate studies by their peers'

#346

Post by SilasX » Sat, Feb 25, 2017, 7:26 PM UTC

dTal wrote: post #338

SilasX wrote: post #225

Earlier quoted context omitted.

Hold on -- there's a category confusion here: "check your blindspots" isn't a comforting lie; it's a command. Converting it to "checking your blindspots will avoid collisions with careless drivers" would make it no longer a lie.

I was pointing out the category confusion. Science teachers don't purport to tell you what scientists actually do (history or sociology teachers might, I suppose, but they don't usually "make such a big deal" out of the scientific method). They purport to teach science. As you say - it's not a comforting lie, it's a command.

The "comforting lie" I was referring to was "this is how scientists actually do it and why you can trust the results".

Re: Most scientists 'can't replicate studies by their peers'

#347

Post by fuzzfactor » Sat, Feb 25, 2017, 8:01 PM UTC

dekhn wrote: post #238

RodericDay wrote: post #212

Earlier quoted context omitted.

> Actually, reproducibility is not considered a necessity in science. By whom? I think you're using as a premise the very matter that is being discussed.

I would say that most scientists, at the top of their game, who have years of trusted publications behind them, should be given some latitude to publish discovery science without the requirement of reproducibility, so long as they share their results in detail, and accept postdocs who then take the resulting research materials to their new labs. I didn't always believe this, but after spending a bunch of time reading…

Many times it requires years to build a laboratory which can even begin to replicate the work of an established operation.

And sometimes it is impossible to solve a difficult problem using the first lab one builds in pursuit of that concept.

In these cases the true type of laboratory needed only comes within focus after completing the first attempt by the book and having it prove inadequate.

Proving things is difficult not only in the laboratory, but in hard science it still usually has to be possible to fully support your findings, especially if you are overturning the status quo in some way. That still doesn't require findings to actually be reproduced in a separate lab if the original researchers can demonstrate satisfactory proof in their own unique lab. This may require outside materials or data sets to be satisfactory. Science is not easy.

I think all kinds of scientists should be supported in all kinds of ways.

Fundamentally, there are simply some scientists without peers, always have been.

For unique breakthroughs this might be a most likely source, so they should be leveraged by capitalists using terms without peer.

Re: Most scientists 'can't replicate studies by their peers'

#348

Post by ktamiola » Sat, Feb 25, 2017, 10:04 PM UTC

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

Nothing to add!

Re: Most scientists 'can't replicate studies by their peers'

#349

Post by FabHK » Sat, Feb 25, 2017, 10:13 PM UTC

stult wrote: post #327

chriswarbo wrote: post #322

Earlier quoted context omitted.

Whilst the experimental subjects and data collection are inherently fraught with difficulty, there's still a LOT of low-hanging fruit regarding things like automation. Many scientists use computers to write up results, to store data and perform calculations, but there's often a lot of manual, undocumented work which could easily be scripted to help those re-running the experiment. For example, running some program to…

I imagine it would be easy to have a git-like storage system for this information, where reproduction experiments would be a branch without the actual measurement data.

Check out Common Workflow Language (CWL),

> a specification for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments, from workstations to cluster, cloud, and high performance computing (HPC) environments. CWL is designed to meet the needs of data-intensive science, such as Bioinformatics, Medical Imaging, Astronomy, Physics, and Chemistry.

http://www.commonwl.org

Re: Most scientists 'can't replicate studies by their peers'

#350

Post by FabHK » Sat, Feb 25, 2017, 10:15 PM UTC

Balgair wrote: post #269

epistasis wrote: post #265

Earlier quoted context omitted.

Statistics is also the way to look at data in the "hard" sciences. Every measurement has error, and that's how you deal with it. However it's not a simple thing that automatically combines different studies. It takes skilled application to understand how data connects, what's comparable, what's not, etc. Traditionally, "meta-analysis" is the sub-field of statistics that combines studies. But that only combines extrem…

Amen. You can google the controversy around the StatCheck program to really dive into why stats and their applications are beyond lies and damned-lies in their falsehoods (because you can prove the lie is right). Doing something as simple as smoking causes cancer is a very very simple experiment to interpret (hard to fund/preform though). It's the pernicious little studies on 8 cells total that get messy. Preforming…

What a crazy story. I want a beer from that dude.