Most scientists 'can't replicate studies by their peers' • Hacker News Board

Re: Most scientists 'can't replicate studies by their peers'

#61

Post by droopyEyelids » Thu, Feb 23, 2017, 3:28 PM UTC

Thriptic wrote: post #25

droopyEyelids wrote: post #5

>The reproducibility difficulties are not about fraud, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge. That would be relatively easy to stamp out. Instead, she says: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about." Well, maybe I'm too much of a layman, but that doesn't qui…

A lot of it is people over hyping their results and cherry picking their data to fit a narrative. Can you blame them? You can literally build a career off a paper or two published in Science or Nature. Meanwhile, no one controlling funding sources or faculty appointments cares that you did amazing, rigorous work if it leads to less interesting conclusions. This is especially true if you generate null results, even th…

Do your motivations matter if you do incorrect work? In the example I gave, an accountant can definitely face _heavy_ pressure from his employer to "make the numbers work".

But if the numbers superficially "work" without adding up, who cares what the motivation was? That is buffoonery or fraud.

Re: Most scientists 'can't replicate studies by their peers'

#62

Post by [deleted] » Thu, Feb 23, 2017, 3:28 PM UTC

Thriptic wrote: post #25

droopyEyelids wrote: post #5

>The reproducibility difficulties are not about fraud, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge. That would be relatively easy to stamp out. Instead, she says: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about." Well, maybe I'm too much of a layman, but that doesn't qui…

A lot of it is people over hyping their results and cherry picking their data to fit a narrative. Can you blame them? You can literally build a career off a paper or two published in Science or Nature. Meanwhile, no one controlling funding sources or faculty appointments cares that you did amazing, rigorous work if it leads to less interesting conclusions. This is especially true if you generate null results, even th…

[deleted]

Re: Most scientists 'can't replicate studies by their peers'

#63

Post by danjoc » Thu, Feb 23, 2017, 3:29 PM UTC

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

>most of the time, when you are doing an experiment you never done, you just fuck it up.

"Experiments are supposed to be replicable. The authors should have done it themselves before publication"

That seems to imply the experiments aren't even replicated by the people who are supposed to know how to get it right.

Re: Most scientists 'can't replicate studies by their peers'

#64

Post by leereeves » Thu, Feb 23, 2017, 3:30 PM UTC

jobvandervoort wrote: post #7

> The problem, it turned out, was not with Marcus Munafo's science, but with the way the scientific literature had been "tidied up" to present a much clearer, more robust outcome. I've seen this time and time again while working in neuroscience and hearing the same from friends that are still in those fields. Data is often thoroughly massaged, outliers left out of reporting and methods tuned to confirm, rather than f…

Science would be much more accurate, I think, if negative results were reported.

Even if everyone involved is perfectly honest, you still have the green jelly beans problem.

Re: Most scientists 'can't replicate studies by their peers'

#65

Post by TheAceOfHearts » Thu, Feb 23, 2017, 3:31 PM UTC

This isn't that surprising, at least based on my limited experience from reading computer science research papers. My experience has been that there's usually not enough information for you to implement something. Am I an outlier, or have others experienced the same?

Re: Most scientists 'can't replicate studies by their peers'

#66

Post by lightedman » Thu, Feb 23, 2017, 3:31 PM UTC

dekhn wrote: post #50

So. Having been a scientist, I observed an interesting phenomenon, several times. It's almost as if scientists enjoy leaving out critical details from the methods section, and other scientists enjoy puzzling out what the missing details are. I think there's this sort-of assumption of competence that for any reasonably interesting paper, the people in the field who are reading it, they have the level of skill to repro…

"It's almost as if scientists enjoy leaving out critical details from the methods section, and other scientists enjoy puzzling out what the missing details are."

From my experience, at least in horticulture, scientists leave out a few methods because there's some sort of potential patent or marketing process in the works and they don't want to reveal too much and be beaten to the punch.

Re: Most scientists 'can't replicate studies by their peers'

#67

Post by godelski » Thu, Feb 23, 2017, 3:40 PM UTC

So I can't be the only one who has noticed the correlation between this and the field at question. As soft sciences like soc, psych, and medicine seem to have the most problems with it. I'm not saying hard sciences like physics don't, but it is less common.

The math for the soft sciences isn't as concrete and doesn't provide a good foundation. I think there are also major problems with the use of p values. It is too easy to manipulate and a lot of incentive to do so. Teach a science class (even the hard sciences) and you'll see how quickly students try to fudge their data to match the expected result. I've seen even professionals do this. I once talked to a NASA biologist who I was trying to get his chi-square value and took a little bit of pressing because he was embarrassed that it didn't confirm his thesis (it didn't disprove it though. Just error was large enough to allow for the other prevailing theory). As scientists we have to be okay with a negative result. It is still useful. That's how we figure things out. A bunch of negatives narrows the problem. A reduced search space is extremely important in science.

The other problem is incentives in funding. There is little funding to reproduce experiments. It isn't as glorious, but it is just as important.

Re: Most scientists 'can't replicate studies by their peers'

#68

Post by 6d6b73 » Thu, Feb 23, 2017, 3:40 PM UTC

YCode wrote: post #26

jobvandervoort wrote: post #7

> The problem, it turned out, was not with Marcus Munafo's science, but with the way the scientific literature had been "tidied up" to present a much clearer, more robust outcome. I've seen this time and time again while working in neuroscience and hearing the same from friends that are still in those fields. Data is often thoroughly massaged, outliers left out of reporting and methods tuned to confirm, rather than f…

For what it's worth I see the same thing in enterprise app development. We've been doing a lot of data visualization and it often happens that someone comes to me with a thinly veiled task that's really to prove this or that person/process is at fault for delaying a project or something. Sometimes though the numbers either don't support their opinion or even show a result they don't like and so inevitably they have m…

I've seen something similar in my field - industrial automation and testing. When a company wants to upgrade their testers, the testers we create are usually much more precise when compared to something created 20-30 years earlier. Often we have to modify our testers to match the results generated by these old, barely working testers.These companies request us to do it simply because otherwise they would need to change all of its literature, and explain to their customers why the products have slightly different specs then what they delivered last quarter.

Unfortunately, Our society is built on rotten foundations.

Re: Most scientists 'can't replicate studies by their peers'

#69

Post by tunap » Thu, Feb 23, 2017, 3:43 PM UTC

codr4life wrote: post #60

sambe wrote: post #51

Earlier quoted context omitted.

Is this an intentional straw man? No, you're not told to trust the people because they have a certain job title and trustworthiness. You're told to look at the data, which is overwhelming and consistent across multiple years and teams. It is the exact opposite of lack of reproducibility.

The only thing that is consistent and overwhelming about that data is the amount of energy put into falsifying and massaging it to support the chosen narrative.

If you can still, edit/remove the presumption of falsification and your argument has merit. I too percieve group think in climate change narratives, while not necessarily disbelieving the data. We are wrecking our environment, exact detrimental effects are still uncertain.

Re: Most scientists 'can't replicate studies by their peers'

#70

Post by tinco » Thu, Feb 23, 2017, 3:45 PM UTC

dekhn wrote: post #53

tinco wrote: post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

I published in nature. The results were based on a computation that was so large, no other entity in the world could have reproduced it at the time (I work for Google, we built a system that used 600+Kcores continuously for years). I don't think ,at the time, that anybody other than Amazon or Microsoft could have run the same calculation as us. Should our publication have been prevented simply because nobody else had…

Well, maybe they could not run the full experiment, but maybe they could audit the source code of the computation you ran, maybe just a small subset of the computation on for example a synthetic smaller dataset, that could maybe be used to test whether your computation works like it should.

Did you think about the replicability of your research while you were working on it?