Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Could It Be? Spooky Experiments That 'See' The Future

npr.org

Post reply on HN
Print view View original

51–60 of 123 posts

Re: Could It Be? Spooky Experiments That 'See' The Future

#51

Post by Eliezer »

CWuestefeld wrote: post #24
nervechannel wrote: post #12

Earlier quoted context omitted.

Just reading it now -- seems like a very thorough takedown of the whole thing, in fairly non-technical language. It makes the very good point that this paper makes lots of statistical tests, and then bases big claims on the small minority that showed a significant effect. This is in no way restricted to psychology, drugs companies for example do this all the time. It's cheating, whether you realize it or not. Statist…

the more tests you do, the sooner you'll see a fluke on average In other words, if you toss a coin 1,000 times, then it's hideously unlikely that you'll see a run of 100 consecutive heads. But if you toss the coin 100,000,000 times, you shouldn't be too surprised to see that 100-toss run buried in there somewhere, even though the odds of getting 100 in a row are so small. Right?

Bad example. You just never get 100 heads in a row on a fair coin, period.

Re: Could It Be? Spooky Experiments That 'See' The Future

#52

Post by 3am »

nostromo wrote: post #19

"Extraordinary claims require extraordinary evidence" -- and I wouldn't count these small studies as extraordinary. Unfortunatley, publishing these kind of claims prematurely help the more gullible among us to fall for ridiculous claims from psychics and others who would take advantage of them. (The authors of "The Secret", I'm looking at you.) Commenters on NPR's website (not exactly the dumbest audience online) hav…

I have don't have any interest in the subject of ESP, but you are completely incorrect. Statistical significance isn't hard to calculate and the number of data points here is fairly large, so there is a measurable effect by reasonable standards. The actual statistics and calculations are right there in the paper.

The idea that the effect could be the result of a programming error or a small amount of light leaking through/around the screen is completely plausible. Or it could be dumb luck as you suggest, but it's extraordinarily unlikely.

Re: Could It Be? Spooky Experiments That 'See' The Future

#54

Post by beagle3 »

CWuestefeld wrote: post #24
nervechannel wrote: post #12

Earlier quoted context omitted.

Just reading it now -- seems like a very thorough takedown of the whole thing, in fairly non-technical language. It makes the very good point that this paper makes lots of statistical tests, and then bases big claims on the small minority that showed a significant effect. This is in no way restricted to psychology, drugs companies for example do this all the time. It's cheating, whether you realize it or not. Statist…

the more tests you do, the sooner you'll see a fluke on average In other words, if you toss a coin 1,000 times, then it's hideously unlikely that you'll see a run of 100 consecutive heads. But if you toss the coin 100,000,000 times, you shouldn't be too surprised to see that 100-toss run buried in there somewhere, even though the odds of getting 100 in a row are so small. Right?

Not really.

If you toss 100,000,000 == 2 ^ 27, you should only expect around 30 in a row. To have a good chance of getting 100 in a row, you need about a billion squared times more.

And the problem is MUCH worse than described above: Let's say you test 1000 wrong hypothesises with p=0.05; 50 of those will be accepted as true, even though all are wrong. If you test 980 wrong hypothesises and 20 right ones, more than half of those that pass the p=0.05 "golden" significance test will in fact be wrong.

Now, when you see a medical journal with 20 articles using p=0.05, which do you think is more probable - that 19 are right and one is wrong, or 19 are wrong and one is right? The latter has a much higher likelihood.

Re: Could It Be? Spooky Experiments That 'See' The Future

#56

Post by nostromo »

yummyfajitas wrote: post #31
nostromo wrote: post #19

"Extraordinary claims require extraordinary evidence" -- and I wouldn't count these small studies as extraordinary. Unfortunatley, publishing these kind of claims prematurely help the more gullible among us to fall for ridiculous claims from psychics and others who would take advantage of them. (The authors of "The Secret", I'm looking at you.) Commenters on NPR's website (not exactly the dumbest audience online) hav…

Unfortunatley, publishing these kind of claims prematurely help the more gullible among us to fall for ridiculous claims from psychics and others who would take advantage of them. True. But on the other hand, publishing ridiculous claims and incorrect results is a necessary part of science. When we publish only results we know to be correct, because they agree with mainstream beliefs, we introduce a bias into the sci…

Good point. I shouldn't have blamed the researcher -- I read the paper and it seems straight forward enough with a number of controls in place. (For example, running a second experiment using only random number generators that showed no such results.)

Instead, I should have focused on science journalists, who should be extra diligent when reporting these sorts of stories to point out the possibility that this is a false positive.

Re: Could It Be? Spooky Experiments That 'See' The Future

#57

Post by pbhjpbhj »

danparsonson wrote: post #15
garyrob wrote: post #4

See http://commonsenseatheism.com/wp-content/uploads/2010/11/Wag... ("Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi".) It demolishes the study referred to in the npr article.

That was a refreshing read, thanks for posting it. It took me a moment to understand the important difference between exploratory and confirmatory experiments - that you can find patterns in anything if you look hard enough, but that doesn't necessarily mean anything until you repeat the process but look for the specific pattern you noticed previously. Reminded me of the Bible Code theory from the 90's ( http://en.wi…

>It took me a moment to understand the important difference between exploratory and confirmatory experiments - that you can find patterns in anything if you look hard enough, but that doesn't necessarily mean anything until you repeat the process but look for the specific pattern you noticed previously

It's when the hypothesis predicts a pattern that you haven't noticed yet and the pattern is confirmed by experiment, that's when you know you have something [a theory].

Re: Could It Be? Spooky Experiments That 'See' The Future

#58

Post by mbrubeck »

Here is some very good discussion of (failed) attempts to replicate the study, and at least one possible methodological flaw that could invalidate some of the results:

http://psychsciencenotes.blogspot.com/2010/11/brief-note-dar...

"The real lesson? This is the level of methodological scrutiny every paper should receive, and not just the ones you think are crazy: the ones you like and rely on for your own work should get a good working over like this too (especially these ones; and I'm as guilty on this as everyone else)."

Re: Could It Be? Spooky Experiments That 'See' The Future

#59

Post by Fargren »

endtime wrote: post #48
Fargren wrote: post #40

Earlier quoted context omitted.

Javascript random is only pseudoalleatory. I think you need a true random number generator to truly test this.

Why would psuedorandomness be insufficient? Don't try to hack the JS and the effect should be the same.

If it's pseudorandom, that means there's some form of pattern in the numbers it generates. It's impossible (or at least unreasonably hard) to know whether the person doing the test may subconciously guess or estimate the pattern. That invalidates the whole test.

Re: Could It Be? Spooky Experiments That 'See' The Future

#60

Post by endtime »

Fargren wrote: post #59
endtime wrote: post #48

Earlier quoted context omitted.

Why would psuedorandomness be insufficient? Don't try to hack the JS and the effect should be the same.

If it's pseudorandom, that means there's some form of pattern in the numbers it generates. It's impossible (or at least unreasonably hard) to know whether the person doing the test may subconciously guess or estimate the pattern. That invalidates the whole test.

I suppose that's true, though if someone can intuit a crypto-strong PRNG then that's an interesting and significant result in itself.
Post reply on HN