Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

191–200 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#191
post #38

I am a social scientist studying human behavior, and this is a huge problem in the field. Myself and my statistician friends who analyze the literature have basically concluded that most extremely "novel" and "surprising" findings in the literature aren't even worth trying to replicate (remember, replications cost money to run, so before you start you have to make some judgment about the likelihood of success.) This…

Thank you. At first I thought puzzagate?! - I don't need that bs, but this is very different pizzagate. I once found a good blog about mental health and science, a lot of snakeoil shown about srris, adhd etc. but I'm unable to find it now. Can anyone help me out?

www.slatestarcodex.com ?

Re: Most scientists 'can't replicate studies by their peers'

#192
post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

My PhD work is trying to address this by developing better ways for scientists to record and communicate their methods/protocols. Methods sections are NOT about providing the information needed to reproduce the work, often even the supplemental information is insufficient. They are a best guess at what needs to be done, and the methods section is often massively condensed due to editor demands and cites other papers…

True, biology is a horror show full of surprises. Many experimental instructions are as reliable as astrologic forecasts. I guess that is the price of complexity and the human factor.

I would love to hear more about your work, and the strategies you propose to improve the reproducibility of scientific experiments. My email can be found in my user description.

Cheers!

Re: Most scientists 'can't replicate studies by their peers'

#193
post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

My PhD work is trying to address this by developing better ways for scientists to record and communicate their methods/protocols. Methods sections are NOT about providing the information needed to reproduce the work, often even the supplemental information is insufficient. They are a best guess at what needs to be done, and the methods section is often massively condensed due to editor demands and cites other papers…

Wouldn't it be hard to achieve deep consistency between experiments, in so many labs around the world, with so different conditions/cultures/etc , and when the experimenters aren't experts in consistency , but in their science ?

Wouldn't it be better to use something like a cloud biology model - where you define experiments via code, CRO's compete on consistency(and efficiency and automation) and since they probably do a much larger volume of experiments than the regular lab, they would have stronger incentives to develop better processes and technologies ?

Re: Most scientists 'can't replicate studies by their peers'

#194

Earlier quoted context omitted.

A lot of the commenters are trying to come up with technical solutions to the issue. It belies their incomplete study of the 'softer' sciences and the difficulty of biological studies. Tech solutions won't work, bureaucratic solutions wont work, more data wont work. Guys, the bio side of things is incredibly complicated and trying to set controls that are achievable in your time and budget is the heart of these field…

I think there's also a lot of confusion about what the scientific literature should be. Outsiders think of journals as pre-packaged nuggets of science fact, with conclusions that you can read and trust. Scientists who publish in the journals view them as a way to communicate a body of work of "I did this, here is the result I saw. I think it might mean this." The difference between these two views can not be understa…

Probably, the real problem is that "the scientific literature should be". It's not the right question to discuss today, because scientific literature is just a brief approximation of scientific knowledge and this is not what we need. Whole modern scientific system is outdated and has to be upgraded: journals, citation indexes, degrees should go away and be replaced. Science should become more formal and more digital, so that it will be easier to find and validate the results. There should be databases, search engines and digital signatures of the raw data and chains of proofs. Counting citations should be replaced with counting chains of proof produced by the scientist and included in others' works. Reliability of proof and contribution to ratings should be based on independent confirmations, which should be counted too - replication of new results should be considered an achievement too. Journals may remain, but not as a primary way to exchange scientific information - more likely, they should become the portals to the data which will link the results in databases to their human-readable interpretations.

Re: Most scientists 'can't replicate studies by their peers'

#195
post #98

Earlier quoted context omitted.

The classic ideal of controlling a single variable at a time is nice but largely impossible. Especially in biological sciences but that's true even down in chemistry and physics the systems scientists are trying to control are complex and it's not always possible to modify one variable without affecting others.

How can you possibly draw any useful conclusions when your experiment's parameters are so fluid as to be uncontrollable?

In some cases you use various statistical tools over many trials or in others you measure the effects of the variables in other experiments. Or you can control the important variables and allow those with small or no effect on what you're measuring to vary slightly.

Re: Most scientists 'can't replicate studies by their peers'

#196
post #147

Quote: "The issue of replication goes to the heart of the scientific process." Indeed it does, but some fields don't think it's very important. For example, there's a surprisingly widespread attitude among psychologists that it's either a waste of time or an attack on the integrity of one's colleagues. Example: http://jasonmitchell.fas.harvard.edu/Papers/Mitchell_failed_... Author: Professor Jason Mitchell, Harvard U…

That's because Jason Mitchell is largely right and I suspect most scientists would, if pressed, agree with him.

The reality is that science is a largely social process. It "works" because institutions have come together to devise cultures and processes that produce useful new knowledge. Now it may be that many people would rather not know how the sausage is made (eg the level of histrionics in this thread)but nobody but the very foolish would argue with the outputs of this very unique social process. And while it may provide some small measure of comfort to insist that this is a problem with just the "social" sciences, this comfort is illusory. Even physics, as it has become increasingly expensive and abstract, has entered this domain. But I don't foresee people abandoning physics because of non-reproducibility as long as useful new knowledge is produced. To that extent all the concern over failed replications is just that, hand-wringing.

The underlying problem which Mitchell hints at is that even if you replicated every experiment a million times this would not necessarily prove anything unless you have somehow managed to find "flawless experimenters." The logical conclusion is that even successful replications produce little new knowledge and there are indeed experiments that have been successfully replicated... until they're not.

A person with a grade school level of understanding of science might declare the whole enterprise flawed and call for it abandoned but I think most adults would recognize the flaws, work to improve them, and keep forward. In that light the "reproducibility crisis" is for the most part a kind of media phenomenon -- nothing in reality has changed (and it's certainly not the case that experiments in the past were reproducible and science needs to be made great again) -- it's just that more people are understanding what science really is and how it can be improved.

Re: Most scientists 'can't replicate studies by their peers'

#197
post #150

Earlier quoted context omitted.

That's not my point, though. Like, if you conduct physics experiments in a vacuum because interactions between objects and atmosphere isn't part of what you're studying, that's fine. If everyone in physics is conducting experiments that way, then suddenly you're left with the question of, it turns out that in real life I encounter atmosphere all the time , and how much can these experimental results tell me about the…

Yes, because you are trying to control your variates? Just because physics is the most amenable to experimental control does not mean that the only real science is physics. I mean by that measure, medicine is not a science either because we don't know most of the possible confounding variables. That doesn't mean that attempting to use the scientific method still isn't the correct choice.

> Just because physics is the most amenable to experimental control does not mean that the only real science is physics.

That's not what I'm trying to say here at all. The point isn't about how amenable you are to experimental control. The point is that even when experimental control is easy to isolate out, like in the physics example above (which I don't think is true in all of physics, by the way), it's not free. You're trying to compensate for the lack of available statistical power to measure an effect in noisy data by cutting down on the noise in the data. But you're doing it by generating the data in an environment that doesn't exist outside of laboratory conditions. Writing off replication failure as not being a problem because lab conditions are difficult to reproduce misses this; if the findings are difficult to replicate in other conditions, that could indicate that the findings are more narrow in scope than the study suggests. As I pointed out downthread, for example, if all the rodents in an experiment on a drug are on the same diet, all the experiment proves (assuming it's otherwise well run) is that the drug works in combination with this diet. If the drug works independently of diet, then the findings on the drug are generalizable. If it doesn't, though, they aren't. And if you have 60 years of medical research based in part on studies with rodents who eat diets very differently than what rodents eat in the wild, or what people eat, then it raises all sorts of questions about the state of medical research. That doesn't mean that medicine isn't a real science, it just raises questions about how well it tells us what we think it's telling us.

Re: Most scientists 'can't replicate studies by their peers'

#198
post #79

Earlier quoted context omitted.

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how generalizable such results are. If the results don't hold in slightly altered systems, it falls into the wastebin of ideas.

Maybe not, but these studies get swept up into meta-studies, books, think tanks and special interest groups write policy papers based on that secondary layer, which then become position papers that inform policy makers about Big Decisions. There's a lot at stake when only scientists are aware of the fallacy.

The reason why meta studies get done is precisely because individual studies aren't reliable.

Re: Most scientists 'can't replicate studies by their peers'

#199

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

Yes, this is well known among working scientists. Even in my theoretical field it was well known that at least 50% articles contain some mistake.

Re: Most scientists 'can't replicate studies by their peers'

#200

This is ego, politics and career ambitions undermining modern science. Unfortunately, the fact that this is occurring so rampantly will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts. This is a systemic failure of basic ethics that will hurt us all. The success-at-all-cost career mindset is toxic in all tracks, but this is one of the most dangerous for it to take…

> Unfortunately, the fact that this is occurring so rampantly will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts. I agree with your point, but I suspect your choice of wording tells a little more of the story. You said "facts", but given your skepticism of their veracity, I think "claims" or "propositions" is more apropos. Consider the rhetorical difference of t…

Very good points. My own bias is to believe the claims of the scientific community and is thoroughly revealed in my language. Perhaps I'm the one being bilked.
Post reply on HN