Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

Post reply on HN
Print view View original

111–120 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#111

Post by dahart »

cs702 wrote: post #21

My first reaction to this headline was "duh." Of course we should hold off on accepting scientific claims (i.e., predictions about the natural world) that to date have been verified only by the same person making those claims! My next reaction was, "wow, it's a sad state of affairs when a postdoctoral research fellow at Harvard Medical School feels he has to spell this out in a blog post." It implies that even at the…

Pretty much everyone in science and academia is saying we need to reproduce results. It is a sad state of affairs, but I'm not sure why having someone from Harvard say it is surprising or different than anywhere else.

The real issue is that there are anti-incentives to reproducing other people's results. All scientists want to see it, and nobody is able to actually do it, because they'll lose status, publication opportunities, and funding. It's viewed as career suicide. Unfortunately, this article doesn't suggest any solutions to that problem.

It would be best to not suggest this problem is somehow due to "people who treat science like religion". That association isn't called for, nor is it largely true or applicable here. Ahmed even said very early in the article "the majority of irreproducible research stems from a complex matrix of statistical, technical, and psychological biases that are rampant within the scientific community."

Statistical and technical biases are common human errors that affect everyone equally and don't amount to religion. Even beliefs don't amount to religion. You and I both believe electrons run through our computers, yet I haven't verified electrons exist and I've never seen one - have you? Almost everything we know is belief based on what others have told us, whether it's science or not. Even if scientific results are reproduced, unless you're doing the reproducing, you're still subject to believing the results. It's a more believable story when two independent people verify some result, but that doesn't mean that believing one story or one paper demonstrating a result is somehow akin to blind faith, ritual, and deity worship.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#112

Post by leecarraher »

It's not just money that prevents people from repeating experiments, it's recognition.

The general idea for research to be accepted is that it makes some novel, albeit small, impact on the field, acceptable for publication in a peer reviewed journal or proceeding. Repeating someone else's experiments wont get you that, so in general it wont help you graduate or move you toward a higher position at a university or in your profession, meaning there is very little motivation for researchers to pursue such endeavors.

So instead of just throwing money at the problem, we may need to entirely revamp how we recognize the pursuits of researchers.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#113

Post by wrsh07 »

jbb555 wrote: post #84

We shouldn't "accept" or "reject" results at all. It's not a binary option. One poor experiment might give us some evidence something is true. A single well reviewed experiment gives us more confidence. Repeating the results similarly does. As does the reputation of the person conducting the experiment and the way in which it was conducted. It's not a binary thing where we decide something is accepted or rejected, we…

The arXiv, but instead of saying "published" or "not published" we put a score on the paper suggesting the probability one should believe its thesis.

Kinda like Rotten Tomatoes, but... for science?

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#114

Post by dschiptsov »

misnome wrote: post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

What has been seen is reading of instruments which has been built according to some mathematical model to register conditions which supposed to happen according to bunch of theories and then a lot of calculations follows which presumably conform the assumptions according to the reading.

The simple analogy is pictures one has on fMRI. These pictures are, well, pictures based on approximations, not the mind. Not even close.

There is a paradox - one cannot make an instrument out of atoms to "see" what is going on inside these atoms. What they saw are pictures created out of mathematical models, not reality as it is.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#115

Post by dahart »

It's unfortunate that the suggestions at the end don't seem to offer a realistic attack vector.

> First, scientists would need to be incentivized to perform replication studies, through recognition and career advancement. Second, a database of replication studies would need to be curated by the scientific community. Third, mathematical derivations of replication-based metrics would need to be developed and tested. Fourth, the new metrics would need to be integrated into the scientific process without disrupting its flow.

Yes, absolutely those things need to happen, but the problem is how to get this funded, how to get people to not see reproducing results as career suicide, right? Items 2-4 will fall out as soon as item #1 happens.

How do we make item #1 happen? What things could be done to make reproducing results actually an attractive activity to scientists?

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#116

Post by teddyh »

sidarape wrote: post #107
studentrob wrote: post #19

Earlier quoted context omitted.

For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared

I mean, if a paper gives an algorithm, proves its correctness, and you are convinced by the proofs, then you're done. I don't see how implementing the algorithm gives you more insight. I'm doing my master degree in computational geometry and most people in my lab don't even implement their algorithms. They just know they are correct from their proofs.

“Beware of bugs in the above code; I have only proved it correct, not tried it.”

— Donald Knuth

(https://staff.fnwi.uva.nl/p.vanemdeboas/knuthnote.pdf)

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#117

Post by dschiptsov »

According to old-school philosophy of science truth could be discovered only by removing all the nonsense, as a remainder, not by pilling up nonsense on top of nonsense out of math and probabilities.

Probabilities, for example, are not applicable to partially observed, guessed and modeled phenomena. It should be a type-error.

As for math - existence of a concept as a mathematical abstraction does not imply its existence outside the realms of so-called collective consciousness. Projecting mathematical concepts onto physical phenomena which could not be observed is a way to create chimeras and to get lost in them.

Read some Hegel to see how it works.)

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#118

Post by dragandj »

dahart wrote: post #115

It's unfortunate that the suggestions at the end don't seem to offer a realistic attack vector. > First, scientists would need to be incentivized to perform replication studies, through recognition and career advancement. Second, a database of replication studies would need to be curated by the scientific community. Third, mathematical derivations of replication-based metrics would need to be developed and tested. Fo…

The problem is that, if you put mere reproduction as a goal, many scientists would see that as low hanging fruit to beef up the resume, so we'd get countless unnecessary "experiments".

I'd say the goal that gets credited should not be merely reproducing the results, but finding errors in the previous research. That would count as novel, and is something that is presently recognized as contribution. The only problem is that journals or conferences treat it as unattractive, so good luck publishing something of the kind...

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#120

Post by mmierz »

I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important.

At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research project, you need to make sure that the basic premise is sound, so it's not uncommon that the first step is to confirm the previous published result before continuing. And if the replication fails, it's obviously not a wise idea to proceed with a project that relies on the prior result. But that work never makes it into a paper.

If the replication succeeds, great! Proceed with the project. But it's time-consuming and expensive to make the reproduction publication worthy, so it will probably get buried in a data supplement if it's published at all.

If the replication fails, it's even more time-consuming and expensive to convincingly demonstrate the negative result. Moreover, the work is being done by an ambitious student or postdoc who is staring down a horrible job market and needs novel results and interesting publications in order to have a future in science. Why would someone like that spend a year attacking the work of an established scientist over an uninteresting and possibly wrong negative result, and getting a crappy paper and an enemy out of it in the end, instead of planning for their own future?

If enough people fail to replicate a result, it becomes "common knowledge" in the field that the result is wrong, and it kind of fades away. But it's not really in anyone's interest to write an explicit rebuttal, so it never happens.

Post reply on HN