We think this cool study we found is flawed. Help us reproduce it • Hacker News Board

Re: We think this cool study we found is flawed. Help us reproduce it

#221

Post by prolyx » Sun, May 01, 2022, 3:00 PM UTC

elif wrote: post #171

prolyx wrote: post #152

Earlier quoted context omitted.

111111 is just as likely as any other number. However, in practice, humans are far more likely to think of 111111 than other numbers, so we exploit the difference between the probability a human guesses a number vs. the probability of rolling a number on a fair dice. "Did I just roll a sequence X or am I lying?" vs. "What is the probability I roll sequence X next?" are quite different questions, if by "lying" you mea…

111111 is just as likely as any other *SEQUENCE* of numbers. This is a little confusing because the "sequential" requirement is somewhat masked by the repeating sequence used by the example. however, the odds of rolling six 1's is 0.00002143347. considering the non-sequential set of rolls 625631 you have the odds of exactly two 6's at 0.201 and also one 2 5 3 and 1 each at 0.402 0.402^4 * 0.201 = 0.00524928641, or ~2…

There are many ways of slicing and dicing things so that you group different sequences, e.g. group permutations as the same as you have done. But by "number" I was referring to a sequence of symbols drawn from a uniform distribution, so order matters. (On the website, you also choose options sequentially, anyways.)

Re: We think this cool study we found is flawed. Help us reproduce it

#222

Post by baobabKoodaa » Sun, May 01, 2022, 3:01 PM UTC

denton-scratch wrote: post #188

dtech wrote: post #166

Earlier quoted context omitted.

It would be incredibly stupid to take the bet, as it's way more likely that the sequence was not produced by a fair die than that it was (i.e. the dice is rigged in the example) Just because it's impossible to know for certain, doesn't mean you can't make a prediction with very high chance of being correct.

I'm not really a gambling man, but I'd expect a crooked 20-sided die to produce a biased sequence, not a running straight. I don't know if it's possible to make a die that always rolls the same, and I'd expect any such die to fail a superficial inspection (all sides but one bulge; one side is larger than the others; the die has a weird magnetic field; the die is heavily weighted on one side). So I'd still expect a ru…

> All this is beside the point; the instructions invite the subject to produce a sequence that they think will convince people it was produced by a roll of dice. But there is no sequence that SHOULD have that power to convince.

If you genuinely believe that, we can easily set up a sequence of bets where you will win infinite amounts of money from me. But of course, you don't genuinely believe that, so you aren't interested in making bets around your supposed "beliefs".

Re: We think this cool study we found is flawed. Help us reproduce it

#223

Post by mmoll » Sun, May 01, 2022, 3:03 PM UTC

HWR_14 wrote: post #150

denton-scratch wrote: post #130

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > t…

> It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. That's just not true. Or feel free to play a game with me. We'll roll a 20 sided die. If it comes up 20, you give me a ten. If it comes up any other number, I'll get you a dollar. Nice EV on that! Oh, the die has come up 20, 20, 20, 20,…

You’re comparing apples to oranges. The correct analogy is: you pick any other sequence of numbers between 1 and 20 and then tell me you’re more likely to win because your sequence is more random.

Re: We think this cool study we found is flawed. Help us reproduce it

#224

Post by baobabKoodaa » Sun, May 01, 2022, 3:03 PM UTC

icambron wrote: post #113

I agree with the article that the study is flawed in its unwillingness to exclude the all-H and all-T answers. But I’ll go further: the original study is just silly. “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To our algorithm, is what they meant. There’s no such thing as a randomness test that can look at a sequence and decide “is it random?”, so this algo measures somethi…

> “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom?

To other people. They specified this in the study instructions. (Not defending the study, I think it's flawed too.)

Re: We think this cool study we found is flawed. Help us reproduce it

#225

Post by [deleted] » Sun, May 01, 2022, 3:06 PM UTC

posix86 wrote: post #189

denton-scratch wrote: post #130

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > t…

You're wrong. Mathematically. Here's why. When you throw a coin 100 times, each sequence you get is equally likely. However. You can look at properties of the sequence which are more likely to be one way than the other. For instance, it's more likely that the number of heads and tails are about equal than not. The reason is that there are more sequences, in general, where that is true, than those where heads or tails…

[deleted]

Re: We think this cool study we found is flawed. Help us reproduce it

#226

Post by mproud » Sun, May 01, 2022, 3:06 PM UTC

The thing I’ve learned early on is, with truly random sets, is they are not evenly distributed. You will at times have clumps of repeating numbers.

Re: We think this cool study we found is flawed. Help us reproduce it

#227

Post by nnoitra » Sun, May 01, 2022, 3:07 PM UTC

pessimizer wrote: post #201

nnoitra wrote: post #191

Earlier quoted context omitted.

No, it is not. On average you expect equal number of 1s and 0s. All 1s won't be "equally as likely" .

All 1s isn't equally as likely as the entire class of outcomes that are not all 1s , but you can also say that about every other outcome (as long as we're talking about distinguishable dice or a sequence.)

You are really confused about this. In general a randomly generated sequence on average will have equal 1s and 0s. Try generating 100K sequences and then count the sequences with:

1) equal 1s and 0s

2) all 1s

3) all 0s

See which one is more likely.

Re: We think this cool study we found is flawed. Help us reproduce it

#228

Post by JuettnerDistrib » Sun, May 01, 2022, 3:08 PM UTC

icambron wrote: post #113

I agree with the article that the study is flawed in its unwillingness to exclude the all-H and all-T answers. But I’ll go further: the original study is just silly. “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To our algorithm, is what they meant. There’s no such thing as a randomness test that can look at a sequence and decide “is it random?”, so this algo measures somethi…

> we only learned that older people pick all-H or all-T more.

I think we learn that older people are less willing to put up with silly instructions.

Re: We think this cool study we found is flawed. Help us reproduce it

#229

Post by kevinpet » Sun, May 01, 2022, 3:10 PM UTC

denton-scratch wrote: post #130

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > t…

Here are some numbers:

123456123456

Did I get them by rolling a die?

Re: We think this cool study we found is flawed. Help us reproduce it

#230

Post by icambron » Sun, May 01, 2022, 3:10 PM UTC

moconnor wrote: post #128

icambron wrote: post #113

I agree with the article that the study is flawed in its unwillingness to exclude the all-H and all-T answers. But I’ll go further: the original study is just silly. “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To our algorithm, is what they meant. There’s no such thing as a randomness test that can look at a sequence and decide “is it random?”, so this algo measures somethi…

“Sequence looks random” is not nonsensical. The authors say your sequence should be indistinguishable from e.g. 12 coin tosses. This would have a uniform distribution. One approach would be estimate the probability distribution from the input sequence and calculate the KL-divergence [1] of that to the uniform distribution. This gives one objective measure of randomness. There are many others! TL;DR: There are definit…

You can define terms however you want, and thus devise any measure you want. But that’s not a definition of randomness I recognize. Perhaps entropy. Randomness is not a property of the result; it’s a property of the process used to generate it.

Let’s imagine applying your measure to a whole bunch of sequences, each of length N, with the elements of each actually drawn from a uniform distribution. You measure the randomness of each one. You’ll get a range of DLK results, distributed from, in your interpretation, “very random” to “not that random”. All-H or mostly-H will come up sometimes, the distribution estimator will return a skewed result, and it will get a divergent “score”. But everything came from the same distribution. So we’re now measuring the output of an actually random process and saying “we’ll, it’s usually random but not quite always”

In contrast, let’s try your method on a different population of sequences, where instead of pulling the sequence elements from a distribution, every sequence is a hardcoded copy of HTHTHT… That gives “perfectly random”, even though it was very far from a random process.

That’s close to what the study authors are doing here, except with a different definition of “random looking”. It’s measuring a property of a sequence, but it isn’t whether it was randomly generated.

We could debate about whether this is a good measure of “random looking” and there could be lots of alternatives with no objectively best. But that is my point: if I ask “make me something random-looking”, I am only asking “how closely does your measure of post-facto randomness match mine?” An actual random sequence would be, well, random.