Most scientists 'can't replicate studies by their peers'

Peer review does not imply reproducibility, and it's the latter that is the problem.

I can confirm, as a reviewer, that your methodology and analysis looks sensible, but the flaws may be deeper, and the fact that you didn't publish the 19 other studies that failed, but that this is the "lucky one", or that you simply cherry picked the data, is not something I can see as a reviewer.

This is especially true if the experiment is nontrivial to re-do.

I wonder if paying grad students to write a more full paper that includes all the steps and the negative results would help. It wouldn't be something that is published right away, and perhaps it wouldn't need to be published. Maybe it would simply be a follow-up to the original paper. It would be a "proof" of sorts, provided by the authors. There are many students out there that would happily do this, I think. I know so many that clamor for even the slightest bit of work in their departments. I think it would also be beneficial to their future, teaching them about reproducibility and impressing upon them to continue this practice down the road. The current climate of publish-or-perish isn't going away anytime soon, and neither are the clean, pretty papers with only positive results. And that's fine. Those are the quick highlights. But the full studies still need to be out there, and I think this could potentially be a way to approach that necessity.

For what it's worth I see the same thing in enterprise app development.

We've been doing a lot of data visualization and it often happens that someone comes to me with a thinly veiled task that's really to prove this or that person/process is at fault for delaying a project or something.

Sometimes though the numbers either don't support their opinion or even show a result they don't like and so inevitably they have me massage the graphs and filters until they see a result that looks how they want it to and that's what gets presented at various meetings and email chains.

The information at that point isn't wrong per se, just taken out of context and shown in a persuasive (read: propaganda) rather than informative way.

"all you have to do is read the methods section in the paper and follow the instructions."

I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know.

Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the time, when you are doing an experiment you never done, you just fuck it up.

Here is an example: we are studying memory using mice. Mice don't remember that well if they are anxious. Here are variables we have to take care of to keep the mice happy, but they are never going to go to the methods section:

Make sure the animal facility haven't cleaned their cages.

But make sure the cage is otherwise relative clean.

Make sure they don't fight each other.

Make sure the (usually false) fire alarm hasn't sound for 24 hours.

Make sure the guy who was installing microscope upstairs has finished producing noise.

Make sure there is no irrelevant people talking/laughing loudly outside the behaviour space.

Make sure the finicky equipment works.

Make sure the animals love you.

The list can go on.

Because if one of this happens, the animals are anxious, then they don't remember, and you got a negative result which have nothing to do with your experiment (although you may not notice this). That's why if your lab just start to do something you haven't done for years, you fail. And replicating other people's experiment is hard.

A positive control would help to make sure your negative result is real, but for some experiments a good positive control can be a luxury.

> will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts

Case in point: the very first thing I thought of is, does this have any relevance to the field of climate science!

So....does it? Because we're told the reason we have to get on board with the program is because the people telling us the facts are scientists, and scientists are smart and trustworthy. However, we know this is not always true, don't we.

So what is a deliberately skeptical person to think?