Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

221–230 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#221
post #188

Earlier quoted context omitted.

I remember my teachers in high school making such a big deal about the scientific method on how important it was, how experiments must be reproducible to be useful, but today you barely hear it mentioned one way or another.

Wait, you're telling me that schools teach comforting lies about society that don't match how the real world actually works?

It's no more a "comforting lie" than your driving instructor telling you to check your blind spot. Not everyone does it, and bad things happen as a result, but all the more important to teach it.

I'm actually surprised to hear of high school teaching good scientific practice. I don't remember ever being taught that. Widely may it spread.

Re: Most scientists 'can't replicate studies by their peers'

#222
So these results can't be replicated in controlled situations. Some of this applies to how our bodies work, with many more "uncontrolled variables" than experimenters even thought of.

In which universe should we guide our behavior and believe things based on "science" like that.

Thought Experiment: Compare a mass murderer to a scientist in cases like this.

Re: Most scientists 'can't replicate studies by their peers'

#223
Simple. You can only publish to Nature etc. if your results are replicated twice by others and you open up all your raw data.

Short time later only real science will be published.

The amount of published results will massively go down and Nature and other will not makes as much money.

Now you know the reason why publishers publish results that can't be replicated.

Or listen to anything from Feynman about real science e.g.

https://www.youtube.com/watch?v=tWr39Q9vBgo

Re: Most scientists 'can't replicate studies by their peers'

#224

Specifically to the CS context, I think some version of double blinded peer code-review should be made mandatory for a publication. I've seen authors skip quite a many details that are quintessential to the replication process. In short, if research is not replicable by the peer community it just useless, that's what it is.

CS is the craziest of them all. Those should be the easiest to replicate. "Here is the code, here is a manifest of the environment/container/disk image/etc." You should be able to take that and run it and get the same results. Or are you saying that the code itself is the problem and that they've done the equivalent of "return True" to get the result they want?

In my other comment I mentioned the CS results I've largely struggled to reproduce is because they include enough detail for you to get the gist of how it works, but not enough to avoid going down some rabbit holes. Also, not all publications include code. Many venues don't require it.

Re: Most scientists 'can't replicate studies by their peers'

#225
post #221
post #188

Earlier quoted context omitted.

Wait, you're telling me that schools teach comforting lies about society that don't match how the real world actually works?

It's no more a "comforting lie" than your driving instructor telling you to check your blind spot. Not everyone does it, and bad things happen as a result, but all the more important to teach it. I'm actually surprised to hear of high school teaching good scientific practice. I don't remember ever being taught that. Widely may it spread.

Hold on -- there's a category confusion here: "check your blindspots" isn't a comforting lie; it's a command. Converting it to "checking your blindspots will avoid collisions with careless drivers" would make it no longer a lie.

Re: Most scientists 'can't replicate studies by their peers'

#226
This is what happens when you ignore the incentives. I think forcing professionals in any field to "just be moral" -- "just do the right thing" -- "just ignore the inverse financial incentive"... I think this expectation is in itself immoral; human will power has its limitations and just like everything else, we are very good a justify immoral behavior when being moral contradicts our short term needs.

Regulation is indeed complicated and imperfect.. but the solution isn't a simple one -- its certainly not complete deregulation.

Re: Most scientists 'can't replicate studies by their peers'

#227

Earlier quoted context omitted.

A lot of the commenters are trying to come up with technical solutions to the issue. It belies their incomplete study of the 'softer' sciences and the difficulty of biological studies. Tech solutions won't work, bureaucratic solutions wont work, more data wont work. Guys, the bio side of things is incredibly complicated and trying to set controls that are achievable in your time and budget is the heart of these field…

I think there's also a lot of confusion about what the scientific literature should be. Outsiders think of journals as pre-packaged nuggets of science fact, with conclusions that you can read and trust. Scientists who publish in the journals view them as a way to communicate a body of work of "I did this, here is the result I saw. I think it might mean this." The difference between these two views can not be understa…

Scientists and academic institutions add to the confusion with the way they promote their work. Nobody puts language like "I did this, here is the result I saw. I think it might mean this" in a press release. It's always "Breakthrough research! Is the end of cancer in sight??"

Re: Most scientists 'can't replicate studies by their peers'

#228
post #219
post #17

Earlier quoted context omitted.

In applied mathematics, the idea of having a standard platform for releasing numerical experiments and standard datasets have come and gone over the years. My advisor said that in the early 2000s, there was a push in some areas to standardize around Java applets for this in a few journals, but never really took hold. Nowadays I would think some form of VM or container technology could probably do the trick while avoi…

>The good news is that you can't really fake proofs or formal analysis. It's my understanding that most published mathematical proofs aren't "hey look at this theorem in first order logic that we reduced to symbol manipulation"; rather, they present enough evidence that other mathematicians are convinced that such a proof could be constructed. Is that incorrect?

I'm not a mathematician, but I had a math prof tell me that most publications only contain proof sketches, not full proofs.

Can someone in the field comment?

Re: Most scientists 'can't replicate studies by their peers'

#229
post #227

Earlier quoted context omitted.

I think there's also a lot of confusion about what the scientific literature should be. Outsiders think of journals as pre-packaged nuggets of science fact, with conclusions that you can read and trust. Scientists who publish in the journals view them as a way to communicate a body of work of "I did this, here is the result I saw. I think it might mean this." The difference between these two views can not be understa…

Scientists and academic institutions add to the confusion with the way they promote their work. Nobody puts language like "I did this, here is the result I saw. I think it might mean this" in a press release. It's always "Breakthrough research! Is the end of cancer in sight??"

Absolutely true, though I know of very few scientists that have been able to tamp down the game of telephone that the university's press office starts.

http://www.phdcomics.com/comics.php?n=1174

Re: Most scientists 'can't replicate studies by their peers'

#230

Earlier quoted context omitted.

I think there's also a lot of confusion about what the scientific literature should be. Outsiders think of journals as pre-packaged nuggets of science fact, with conclusions that you can read and trust. Scientists who publish in the journals view them as a way to communicate a body of work of "I did this, here is the result I saw. I think it might mean this." The difference between these two views can not be understa…

Probably, the real problem is that "the scientific literature should be". It's not the right question to discuss today, because scientific literature is just a brief approximation of scientific knowledge and this is not what we need. Whole modern scientific system is outdated and has to be upgraded: journals, citation indexes, degrees should go away and be replaced. Science should become more formal and more digital,…

Respectfully, I think you're missing the complications outlined in the rest of the thread. The difficulty in the biosciences and even in chemistry and some physics is that they can't really be formalized in the way that is necessary to construct these "chains of proof".

The scientific literature is an ongoing conversation anchored by rigorous experimental facts and data. But rigorous doesn't mean it's clean like a mathematical proof. In fact, most science approaches its "proofs" in quite a different way than math. For example, as far as science would be concerned P!=NP in complexity theory. We've done the experiment many times, tried different things, it's pretty much true. But it's still not mathematically proven because there isn't a formal proof.

That's not to say it's invalid to expect more rigor, or that we wouldn't all love to have "chains of proof" and databases and signatures for data etc. It's that it's simply not practical given how noisy and complex biological systems are. In contrast to math, you pretty much never know the full complement of objects/chemicals/parameters in your experimental space. You try to do the right controls to eliminate the confounding variables, but you're still never fully in control of all the nobs and switches in your system. That's why usually you need multiple different experiments tackling a problem from multiple different approaches for a result to be convincing.

Formalized systems would be great, but I don't think we're even close to understanding how to properly formalize all of those difficulties and variables in a useful way. And it may not even be possible.

Post reply on HN