Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We think this cool study we found is flawed. Help us reproduce it

pudding.cool

Post reply on HN
Print view View original

121–130 of 361 posts

Re: We think this cool study we found is flawed. Help us reproduce it

#121

Post by waqf »

searchableguy wrote: post #7

> Do you think the questionable response looks genuinely random and satisfies the instructions? We don’t. Could someone explain why putting sequences of same choice is invalid? Sure, a probability of having many tails continously is low but how is it not random?

This can be explained using Bayesian reasoning.

P(all heads | bad faith) ≫ P(any given "random"-looking string | bad faith)

therefore

P(bad faith | all heads) ≫ P(bad faith | any given "random"-looking string)

So if you want to exclude bad faith responses, the best strategy (by Neyman–Pearson, if you want to think of it that way) is to remove "all heads" responses (and similarly for "all tails").

Re: We think this cool study we found is flawed. Help us reproduce it

#123

Post by op00to »

bombcar wrote: post #114
icambron wrote: post #113

I agree with the article that the study is flawed in its unwillingness to exclude the all-H and all-T answers. But I’ll go further: the original study is just silly. “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To our algorithm, is what they meant. There’s no such thing as a randomness test that can look at a sequence and decide “is it random?”, so this algo measures somethi…

I suspect many of the “bad” responses are smart aleks saying “11111111” is just as likely a “62536164”.

Is 11111111 less likely than another sequence?

Re: We think this cool study we found is flawed. Help us reproduce it

#124

Post by ascar »

dtech wrote: post #108
Danieru wrote: post #103

Earlier quoted context omitted.

Those are the instructions from the original survey. Those if destructions being under defined, yes that is part of thr entire point.

They are not, these are the instructions from the reproduction > Tap a sequence of 10 dice rolls. Make it look as random as possible; another person should not be able to tell if you made it up or if it was from real dice rolls. And this is the excerpt from the study they mention > Click on a number between one and six as randomly as possible to produce the kind of sequence you'd get if you really rolled a die [...]…

The [...] in your quote reads:

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody.

I have a really hard time rationalizing why you would leave that part out of your quote and drew the conclusion you did. The original task was clearly also about creating patterns that a human would recognize as random.

Re: We think this cool study we found is flawed. Help us reproduce it

#125

Post by Hbruz0 »

plorntus wrote: post #67

I feel like a lot of the comments here are written after only taking the test and many are not reading the rest of the article. The authors of the website are stating that they believe the study is wrong. The below/above 60 answer is showing you it’s incorrect half of the time along with data backing up the claim.

I lol'd at the "Trend line": https://imgur.com/a/ohYbcLL

Re: We think this cool study we found is flawed. Help us reproduce it

#126

Post by Out_of_Characte »

tomsmeding wrote: post #112
sjmm1989 wrote: post #43

Earlier quoted context omitted.

Which to a machine may as well be the same thing in either phrasing. You want something different from what you just listened to. To it, anything not 'that song' is different and 'new' potentially if also not 'just listened to' within a certain set amount of songs. Even without that certain set of songs being logged and considered; any picking of a different song from the last is verifiably random. Think of it all li…

If I understand you correctly, I think you missed my point. You're explaining how with true randomness, you get different stuff most of the time and the same stuff some of the time. That is true. But it's not what people want when they press shuffle. What people want is something _different_, and giving the same song twice is not something different. As another commenter wrote, giving multiple (different) songs after…

This is true for many games as well, their "1%" chance usually means you'll always get lucky twice in a series of 200 attempts

Re: We think this cool study we found is flawed. Help us reproduce it

#127

Post by bombcar »

op00to wrote: post #123
bombcar wrote: post #114

Earlier quoted context omitted.

I suspect many of the “bad” responses are smart aleks saying “11111111” is just as likely a “62536164”.

Is 11111111 less likely than another sequence?

It’s just as likely (or unlikely) as any particular sequence.

Re: We think this cool study we found is flawed. Help us reproduce it

#128

Post by moconnor »

icambron wrote: post #113

I agree with the article that the study is flawed in its unwillingness to exclude the all-H and all-T answers. But I’ll go further: the original study is just silly. “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To our algorithm, is what they meant. There’s no such thing as a randomness test that can look at a sequence and decide “is it random?”, so this algo measures somethi…

“Sequence looks random” is not nonsensical. The authors say your sequence should be indistinguishable from e.g. 12 coin tosses. This would have a uniform distribution.

One approach would be estimate the probability distribution from the input sequence and calculate the KL-divergence [1] of that to the uniform distribution. This gives one objective measure of randomness. There are many others!

TL;DR: There are definitions of randomness that can be tested against.

[1] https://en.m.wikipedia.org/wiki/Kullback–Leibler_divergence

Re: We think this cool study we found is flawed. Help us reproduce it

#129

Post by ricardobeat »

op00to wrote: post #123
bombcar wrote: post #114

Earlier quoted context omitted.

I suspect many of the “bad” responses are smart aleks saying “11111111” is just as likely a “62536164”.

Is 11111111 less likely than another sequence?

The fact it’s equally likely as any other sequence means it should be very unlikely to appear in the study’s 3429 samples, and extremely unlikely to show up more than once.

Re: We think this cool study we found is flawed. Help us reproduce it

#130

Post by denton-scratch »

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody.

That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible.

> the kind of sequence you’d get if you really rolled a die

Well, there is no such sequence. The instruction is incoherent.

You could take it as meaning "Construct a sequence that you think will convince others that it came from a random source". That would be coherent. And then it would be legitimate to eliminate responses that were all-heads ("clearly didn't even try"). But then what are you measuring? The comparative understanding of older and youger people concerning random sources, or the Gambler's Paradox? Their comparative expertise in human psychology? Their comparative willingness to move the mouse-pointer over the screen?

Post reply on HN