Could It Be? Spooky Experiments That 'See' The Future

True. But on the other hand, publishing ridiculous claims and incorrect results is a necessary part of science.

When we publish only results we know to be correct, because they agree with mainstream beliefs, we introduce a bias into the scientific process. In reality, if you publish 20 experiments with p=0.05 [1], 1 of them should be incorrect. If less than 1 in 20 of your papers isn't wrong (assuming p=0.05 is the gold standard), you are not doing science.

You can see a perfect illustration of this when people tried to reproduce Millikan's oil drop experiment. I'll quote Feynman: Millikan measured the charge on an electron...got an answer which we now know not to be quite right...It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.

Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of - this history - because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong - and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that...

This is why I'm an advocate of accepting/rejecting scientific papers based solely on methodology, with referees being given no information about the conclusions and with authors being forbidden from post-hoc tweaks. You do your experiment, and if you disagree with Millikan/conclude that ESP exists, so be it. Everyone is allowed to be wrong 5% of the time.

[1] I'm wearing my frequentist hat for the purposes of this post. Even if you are a Bayesian, you should still publish, however.

Specs: Two buttons - ESP mode or non-ESP mode. In non-ESP mode, 60 random pictures are shown and we are to guess. Then it gives the correct one. In ESP mode, add some porn. If the results are different, then we have ESP. Use Javascript for the randomisation algorithm so that we can be sure there is no server trickery being done.

But to be fair, we should take things on their merits. Contrasting the two papers, Bem's paper sticks very closely to the standard form and 'rules of engagement' (if you will) of published academic research. This counter paper on the other hand, has only the surface appearance of proper scientific method. For example, in making one if its main points it refers to a hypothetical casino where Bem could've made an 'infinite' amount of money, and to the one million dollar Randi prize, neither of which reach the standard of proper experimentalist scientific evidence. It does this by way of justification to one of their main points which is that because 'extraordinary claims require extraordinary evidence' we should be able to set the prior expectation of H1 - in their words 'for illustrative purposes' - to .00000000000000000001, which they then go on to demonstrate, makes the results non-significant.

But where does the 0.00000000000000000001 come from? It could, just as easily, be twice or half that figure. Thus not falsifiable and therefore not justifiable as an extra bit of arithmetic that Bem's paper must qualify.

To put this in terms this audience will understand, that's kind of like saying well no, because you are using Java and 'everybody knows' Java is slow I think we should multiply your benchmark figures by, oh, let's say, one half. And then we see that java throughput is quite poor, as expected. In fact, we should multiply all java benchmarks by some number like a half but I'm not going to be specific about it because actually its pretty much just an arbitrary number I made up. So, reading past the reference to Bayes and some nice formulas that's just arithmetic in my book.

Note, I'm not saying that there are no flaws in the Bem paper. Everybody can see that it's very likely they'll be something wrong with it (and the desk drawer problem to which the above paper also makes extensive evidence is a likely though not conclusive contender) but I think its only reasonable to hold up the criticism's to the same standard as what they are criticizing. Perhaps that way you'll be more likely to find the actual truth of the matter.