Most scientists 'can't replicate studies by their peers' • Hacker News Board

Re: Most scientists 'can't replicate studies by their peers'

#131

Post by wespisea » Thu, Feb 23, 2017, 5:06 PM UTC

cowsandmilk wrote: post #121

mattnewton wrote: post #59

Earlier quoted context omitted.

Do you think that has anything to do with the incentives for reviewers, who want shorter papers with a length limit and prefer more details on the impact/why of the experiment? Does the same method-hiding hold true in journals without length limits or different review processes?

I think length limits cause problems. In the software world, Bioinformatics has 2 page application notes[1]. That is nowhere near enough room to have a figure, describe an algorithm, and describe results. In cases where the source code is available, I've found the actual implementation often has steps not described at all in the algorithm. And these differences make a clean room implementation difficult to impossible…

I just heard of a new NGS file format that should fix this

Re: Most scientists 'can't replicate studies by their peers'

#132

Post by jonlucc » Thu, Feb 23, 2017, 5:06 PM UTC

bbctol wrote: post #119

JumpCrisscross wrote: post #40

Earlier quoted context omitted.

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

The journal I work for just rolled out a new methods format; the big change is requiring all resources to be listed, with their origin (a major problem is that Chemical A or lab rat subspecies B might be very different from one company or another, even if they're theoretically identical.) What's really needed is broader standardization of all practices; requiring video wouldn't solve the issue that different labs mig…

Additionally, not everything appears on video. We have found major differences between identical studies run in rooms set at different temperatures. Also, many of the procedures I do would essentially require a camera operator to capture all of the movement. I take a cage out of a rack, take a mouse out of the cage, weigh it, dose it, etc.

Re: Most scientists 'can't replicate studies by their peers'

#133

Post by cwyers » Thu, Feb 23, 2017, 5:08 PM UTC

jonlucc wrote: post #129

LordKano wrote: post #93

Earlier quoted context omitted.

Don't forget to make sure that none of the people who will be near the area of the experiment live with and thereby smell like a cat.

You can add to this that if you are in a facility with both rats and mice, you should never house them in the same room, and you should probably not enter a mouse room after being in a rat room. In the wild, rats are predators of mice.

But in the wild, there aren't separate rooms for predators and prey, either. If laboratory rodent experiments are so reliant on such synthetic conditions, what's the point?

Re: Most scientists 'can't replicate studies by their peers'

#134

Post by wycx » Thu, Feb 23, 2017, 5:09 PM UTC

j_m_b wrote: post #39

One heuristic I use for scientific papers is "Does it have supplementary materials and what are their quality?" Supplementary materials is where you put raw data and the 'howtos'. It is not just a place to cram in extra figures that wouldn't fit.

Why is it that journals find it impossible to append the pdf supplement to the pdf of the paper?

I recall seeing only a couple of journals than manage to do this for errata.

Re: Most scientists 'can't replicate studies by their peers'

#135

Post by DanBC » Thu, Feb 23, 2017, 5:10 PM UTC

droopyEyelids wrote: post #61

Thriptic wrote: post #25

Earlier quoted context omitted.

A lot of it is people over hyping their results and cherry picking their data to fit a narrative. Can you blame them? You can literally build a career off a paper or two published in Science or Nature. Meanwhile, no one controlling funding sources or faculty appointments cares that you did amazing, rigorous work if it leads to less interesting conclusions. This is especially true if you generate null results, even th…

Do your motivations matter if you do incorrect work? In the example I gave, an accountant can definitely face _heavy_ pressure from his employer to "make the numbers work". But if the numbers superficially "work" without adding up, who cares what the motivation was? That is buffoonery or fraud.

> Do your motivations matter if you do incorrect work?

YES! If I'm careless my results don't match the data and someone can catch my mistake. If I'm trying to defraud you I'll fix this problem by making the data fit, and it's much harder for people to find the mistake.

If I'm a careless accountant we can audit my spreadsheets and find the errors.

If I'm a crooked accountant I'll have deliberately hidden the "error" in shell companies or offshore accounts, and this will be resistant to lower levels of scrutiny.

Re: Most scientists 'can't replicate studies by their peers'

#136

Post by toss1941 » Thu, Feb 23, 2017, 5:11 PM UTC

michaelmrose wrote: post #115

wycx wrote: post #111

Earlier quoted context omitted.

That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others. People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...

At that point its actually not even science and shouldn't be funded

I remember my teachers in high school making such a big deal about the scientific method on how important it was, how experiments must be reproducible to be useful, but today you barely hear it mentioned one way or another.

Re: Most scientists 'can't replicate studies by their peers'

#137

Post by IanCal » Thu, Feb 23, 2017, 5:11 PM UTC

cwyers wrote: post #112

carsongross wrote: post #72

Earlier quoted context omitted.

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Think about the epistemological ramifications of this. This is why trust is so important in social contexts, even in science.

> Think about the epistemological ramifications of this. This this this this this a million times this. If your experiment with lab rats needs to be done this finely to be replicated, is it really telling you anything about the real world? Because of course, we don't really care about the behavior of laboratory rats, right? Not in proportion to how often we do studies with laboratory rats versus other species of anim…

The point, I think, is that you want to control for things that really alter the outcome. It's not that a clean cage means that the end-product no longer works, but you want to compare "didn't have drug" vs "did have drug" not "didn't have drug" vs "did have drug but the fire alarm went off this morning". Unless your effect size is much larger than all the noise, you could easily miss a true result.

Worse, you could get a false result the other way. What if your control group had all been spooked before they were measured a few times? That could make the control group seem worse than they are!

Re: Most scientists 'can't replicate studies by their peers'

#138

Post by tinco » Thu, Feb 23, 2017, 5:12 PM UTC

dekhn wrote: post #92

seszett wrote: post #57

Earlier quoted context omitted.

> Should our publication have been prevented simply because nobody else had the resources to replicate? Well... maybe, yes? Just because you are the only ones able to do the computation doesn't mean your methods or your results are correct.

We share all the results from our research, so people may inspect them in detail. We also share our methods in enough detail that if somebody was sufficiently motivated, they could reproduce the work by spending time and money however, If you follow your logic, you would prevent CERN from publishing the discovery of the Higgs Boson, because nobody else has a particle accelerator powerful enough to detect them. You wo…

Though I agree for the most part, isn't CERN using multiple teams on multiple detectors, even going so far as having multiple implementations of data analyses and such, just to make sure any result they find is never a product of one person or team making a mistake in their process?

Re: Most scientists 'can't replicate studies by their peers'

#139

Post by godelski » Thu, Feb 23, 2017, 5:13 PM UTC

BeetleB wrote: post #106

godelski wrote: post #67

So I can't be the only one who has noticed the correlation between this and the field at question. As soft sciences like soc, psych, and medicine seem to have the most problems with it. I'm not saying hard sciences like physics don't, but it is less common. The math for the soft sciences isn't as concrete and doesn't provide a good foundation. I think there are also major problems with the use of p values. It is too…

>So I can't be the only one who has noticed the correlation between this and the field at question. As soft sciences like soc, psych, and medicine seem to have the most problems with it. I'm not saying hard sciences like physics don't, but it is less common. It is a problem in physics, although a "different" problem. See my comment: https://news.ycombinator.com/item?id=13715197

Yes, but there is a huge difference in degree of problem. That's what I'm getting at. It exists, but in the soft sciences it is much more rampant. Compound that with the weaker analysis and the problem starts becoming that you have to become skeptic of any result from the field.

Different degrees of the same problem.

Re: Most scientists 'can't replicate studies by their peers'

#140

Post by mdpopescu » Thu, Feb 23, 2017, 5:14 PM UTC

dekhn wrote: post #53

tinco wrote: post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

I published in nature. The results were based on a computation that was so large, no other entity in the world could have reproduced it at the time (I work for Google, we built a system that used 600+Kcores continuously for years). I don't think ,at the time, that anybody other than Amazon or Microsoft could have run the same calculation as us. Should our publication have been prevented simply because nobody else had…

How can there be any other answer than "yes"? The point of science was that it's reproducible; you are saying that what you did was not; therefore, it wasn't science.

What is the difference between your paper and an Uri Geller "experiment"? Both are extremely hard to duplicate for objective reasons so their results have to be accepted based on reputation. (Imagine someone trying to publish the same thing but using "Startup LLC" instead of "Google".)