Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

281–282 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#281

I have an alternative proposal: do a study right the first time. That means: A) Pre-registering the study design, including the statistical analysis. Otherwise, attaching a big label "Exploratory! Additional confirmation needed!" B) Properly powering the study. That means gathering a sample large enough that the chances of a false negative aren't just a coin flip. C) Making the data and analysis (scripts, etc.) publi…

Your proposal is comparable to saying that checks and balances are not needed in a democracy, politicians just need to govern "right". This is about incentivising scientists to do the right thing instead of merely demanding it, like you do.

How is advocating for a new set of best practices any more "demanding" or wishful than a regime of obligatory replication? And how is this categorically different from current practices such as peer review, disclosing conflicts of interest, an IRB, etc.?

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#282
post #232

Earlier quoted context omitted.

Does it not? If the experimental design is not preserved, don't you open yourself to inconsistent controls between the experiments? Where do you draw the line?

If you think about it, complete replication of any experiment is useless, and impossible. But that doesn't matter, because it is not the experiment you are trying to replicate, is the effect or the observation. Carefully changing the design of the experiment can allow to verify that your explanation of the effect or observation is accurate. Of course, if the change the experiment too much, a failure to replicated wil…

In hindsight I do think it's true that you want numerous experiments with possibly different controls with the exception of whatever is important for the effect. In that sense you want your experiment to "discover the right abstraction" of details that your principle works over.

So I think what has to occur is a gradual "loosening up" of the controls from strict replication to weak replication, with both types of replication giving information about the effect you are testing: its validity and its generality, respectively.

Post reply on HN