Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

151–160 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#151

Post by TorKlingberg »

pvaldes wrote: post #19
ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

Science publishing is based in a review by peers system. All evident bugs (and many non so evident bugs) should be catched before to appear in a journal. Is totally different to standard journalism.

Replication happens after publication. Why is everyone misunderstanding ZeroGravitas' point?

Re: Most scientists 'can't replicate studies by their peers'

#152

Post by lutusp »

godelski wrote: post #139
BeetleB wrote: post #106

Earlier quoted context omitted.

>So I can't be the only one who has noticed the correlation between this and the field at question. As soft sciences like soc, psych, and medicine seem to have the most problems with it. I'm not saying hard sciences like physics don't, but it is less common. It is a problem in physics, although a "different" problem. See my comment: https://news.ycombinator.com/item?id=13715197

Yes, but there is a huge difference in degree of problem. That's what I'm getting at. It exists, but in the soft sciences it is much more rampant. Compound that with the weaker analysis and the problem starts becoming that you have to become skeptic of any result from the field. Different degrees of the same problem.

Another important distinction between physics and, say, psychology is that the latter studies aren't testing a theory, they're testing an observation. A particular observation sometimes leads to the widespread assumption that a particular effect exists, but without anyone trying to shape a theory about its cause, only that it exists. In physics by contrast, it's all about fitting an observation into existing theory.

Re: Most scientists 'can't replicate studies by their peers'

#153

Post by Balgair »

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

A lot of the commenters are trying to come up with technical solutions to the issue. It belies their incomplete study of the 'softer' sciences and the difficulty of biological studies. Tech solutions won't work, bureaucratic solutions wont work, more data wont work.

Guys, the bio side of things is incredibly complicated and trying to set controls that are achievable in your time and budget is the heart of these fields. The thing you are trying to study in the bio side of science is actually alive and trying to study you right on back. If you are going to kill the thingys, they really do want to kill you too. Look at this diagram of mitochondrial phospholipid gene/protein interactions (https://www.frontiersin.org/files/Articles/128067/fgene-06-0...). That is very complicated and that is for one of the best studied organelles in the Animalia family. There are an uncountable and evolving set of other proteins that then interact with that diagram in different ways depending on the cell-type, species, and developmental history of the organelle (to start with). All of which are totally unknown to you and will likely forever remain unknown to you up until your death. Hell, we are still figuring out the shapes of organs in our own bodies. People that have studied areas for decades spending untold millions of dollars on some of the most central questions of life have essentially nothing to say for themselves and the money spent('We dream because we get sleepy'). Trying to tease out a system that has been evolving for 4.5 billion years and makes a new generation (on average) every 20 minutes is going to be just insanely difficult.

This stuff is hard, the fields know it, and we don't believe anything that someone else says, let alone what our 'facts' and experiments tell us. But we solider on because we love it and because we want to help the world.

Re: Most scientists 'can't replicate studies by their peers'

#154

Post by return0 »

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

A good way to avoid this is using robots wherever possible.

Re: Most scientists 'can't replicate studies by their peers'

#155

Post by lutusp »

zeristor wrote: post #3

Hmmm. Software Engineering has Continuous Integration, since it is so expensive to fix software later in the day. Is there any such thing as Continuous Reproducibility? Constantly checking that the science can be reproduced? How prevalent is this in different branches of Science?

> Is there any such thing as Continuous Reproducibility?

Only in principle. Most studies are episodic -- they have a beginning, an end, and publication of results only after the end. For continuous replication to exist, laboratories would have to be much more open about what they're working on, less competitive as to ideas and funding than they actually are.

> How prevalent is this in different branches of Science?

It's nonexistent everywhere but in "big" physics, where (because of the large number of people involved) people tend to know in advance what's being worked on. But I'm only saying physics opens the possibility for continuous replication, not that it actually exists, mostly because of cost.

Re: Most scientists 'can't replicate studies by their peers'

#156

Post by makecheck »

Replication needs more incentive, too: considered on par with, or even more significant than, publishing any new results in a field. The incentive should come on both sides (rewards for labs that reproduce results of other labs, and rewards for scientists that publish results that are paired with clear methods).

I’m afraid that research is starting to descend into a fight for a few measly dollars, at any cost. If the results don’t really matter, you start seeing far less important measurements like “number of publications” taking precedence, which is a huge problem. At some point, if your lifeline depends on bogus metrics and all the competing labs are publishing crap that no one reads and no one can reproduce, are you forced to also publish the hell out of everything you can think of just to “compete” and stay funded? And at some point, are you spending more time publishing useless papers and writing grants begging for money, than time spent doing useful research? It’s a race to the bottom that will harm the world’s library of scientific data.

Re: Most scientists 'can't replicate studies by their peers'

#157

Post by FeepingCreature »

wycx wrote: post #149
michaelmrose wrote: post #115

Earlier quoted context omitted.

At that point its actually not even science and shouldn't be funded

The way you get funded is by being able to do things that others can't do, or by being better at some technique than everyone else. Giving away all your hard won tricks by putting all of them in a methods section takes away your advantage come funding time.

So what you're saying is science is dead and we're going back to mysticism and alchemy.

Re: Most scientists 'can't replicate studies by their peers'

#159

Post by jonlucc »

Declanomous wrote: post #142
LordKano wrote: post #93

Earlier quoted context omitted.

Don't forget to make sure that none of the people who will be near the area of the experiment live with and thereby smell like a cat.

Yeah I did an experiment with metabolism in relation to temperature with rodents in college. Our results were just absolute garbage for the most part. I think two of us owned cats, walk in temperature controlled fridge was incredibly loud and the light never turned off, and the fridge was on the opposite side of the building from the only lab with the correct equipment to run the experiment. Plus the enclosure for me…

There's some neat software now (not sure if it was around when you were in college) to translate video into reliable data about the movement of animals. I've used it for fish, but the same software can be used to track people around a room or to track mice in a cage. I think that might be much more useful than, say, how many backflips a mouse does in their cage.

Re: Most scientists 'can't replicate studies by their peers'

#160

Post by lutusp »

Tloewald wrote: post #110

Click bait from the BBC. 2/3 of scientists have failed to replicAte a study is not a statement that reinforces the headline. If 2/3 of people have had trouble sleeping that does not mean most people can't sleep. If you've ever tried a simple science experiment out of a book, it's easy to screw one up no matter how sound the underlying science because you messed something up. A bleeding edge study is several degrees o…

> ... it's easy to screw one up no matter how sound the underlying science because you messed something up.

Yes, but when a replication fails, the question one wants to ask is whether it was the original study or the replication effort that caused the problem. J. B. Rhine's seemingly solid psychic-abilities studies couldn't be replicated, but this was because he kept tossing results that didn't fit his expectations, something that didn't come out until after he passed on.

Post reply on HN