Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We think this cool study we found is flawed. Help us reproduce it

pudding.cool

Post reply on HN
Print view View original

241–250 of 361 posts

Re: We think this cool study we found is flawed. Help us reproduce it

#241

Post by icambron »

nnoitra wrote: post #227
pessimizer wrote: post #201

Earlier quoted context omitted.

All 1s isn't equally as likely as the entire class of outcomes that are not all 1s , but you can also say that about every other outcome (as long as we're talking about distinguishable dice or a sequence.)

You are really confused about this. In general a randomly generated sequence on average will have equal 1s and 0s. Try generating 100K sequences and then count the sequences with: 1) equal 1s and 0s 2) all 1s 3) all 0s See which one is more likely.

You missed the parent’s point. “Equal 1s and 0s” is not a sequence; it’s a class of sequences. So the fact that sequences with that property are more common than the specific sequence “all 1s” is true but doesn’t answer the question. 11110000 is a different sequence than 10101010. The question this thread is exploring is whether 1111111 is more likely than any other sequence, e.g. than 1111000 in particular (or insert any other sequence). And of course the answer is no.

Re: We think this cool study we found is flawed. Help us reproduce it

#242

Post by skykooler »

nnoitra wrote: post #227
pessimizer wrote: post #201

Earlier quoted context omitted.

All 1s isn't equally as likely as the entire class of outcomes that are not all 1s , but you can also say that about every other outcome (as long as we're talking about distinguishable dice or a sequence.)

You are really confused about this. In general a randomly generated sequence on average will have equal 1s and 0s. Try generating 100K sequences and then count the sequences with: 1) equal 1s and 0s 2) all 1s 3) all 0s See which one is more likely.

I believe what pessimizer means is that while "all 1s" is not as likely as "equal 1s and 0s", it is just as likely as any individual string of 1s and 0s - for example, 111111 is just as likely as 100110.

Re: We think this cool study we found is flawed. Help us reproduce it

#243

Post by xwowsersx »

baobabKoodaa wrote: post #220
denton-scratch wrote: post #205

Earlier quoted context omitted.

"he/she should not be able to tell" isn't the same as "he/she should not be able to make a statistically-probable guess".

By this logic the expression "being able to tell" should be banned from the English vocabulary, because no-one is able to tell anything with 100% certainty. Requiring 100% certainty as a precondition of using this expression is silly.

> no-one is able to tell anything with 100% certainty

Including this very assertion? So it's possible that _someone_ could tell _something_ with 100% certainty?

Re: We think this cool study we found is flawed. Help us reproduce it

#244

Post by danparsonson »

nnoitra wrote: post #227
pessimizer wrote: post #201

Earlier quoted context omitted.

All 1s isn't equally as likely as the entire class of outcomes that are not all 1s , but you can also say that about every other outcome (as long as we're talking about distinguishable dice or a sequence.)

You are really confused about this. In general a randomly generated sequence on average will have equal 1s and 0s. Try generating 100K sequences and then count the sequences with: 1) equal 1s and 0s 2) all 1s 3) all 0s See which one is more likely.

The point the parent is making is that 1) represents a class of results rather than a single result, and any single member of that class is equally as likely as 2) or 3). Obviously the class as a whole is more likely than any other single result, but that's a different assertion.

Re: We think this cool study we found is flawed. Help us reproduce it

#245

Post by travisjungroth »

macksd wrote: post #238
RuggedPineapple wrote: post #163

Earlier quoted context omitted.

>Can you roll a fair die a thousand times and only get 6s? Well yes of course. It'll never happen though. Somewhat related, but humans are also REALLY bad at generating randomness and one of our big tells is an aversion to repeats. If you ask someone to pick 0-9 randomly, repeatedly, they will rarely repeat numbers. But in a truly random sample a repeat is likely 10% of the time, and a three-peat will happen roughly…

And funnily enough, you'll often hear this trait as being desirable in a pseudo-random number generator. People often want something that will jump around fairly unpredictably but that will come close to outputting all possible numbers once before getting into re-runs.

I’ve coded up this exact algorithm, it’s really fun. It’s useful for “shuffling” in the music sense (not the cards sense).

I think it’s actually the prototypical real-world software engineering problem. User says they want X (random music). X is a term in software, so you give them that (you get a random song). They’re not happy. You dig and find out they really want A, B, and C (next song is unknown, songs don’t repeat too soon or too infrequently). This new problem is harder to verify (how soon is too soon?).

Editing in tips on solving this sort of problem. You can turn vague requirements into precise requirements. Rather than make the precise requirements exactly equivalent to the vague ones, it's easier to make them more restrictive. Is playing a song again within 50% of the length of the playlist "too soon"? Maybe. How about within 80% of the length of the playlist? Definitely not. We can give ourselves the requirement "Songs must always play again between 80% and 125% of the length of the playlist." Much easier to solve, much easier to test.

Sometimes the extra restriction make the problem harder (not usually I've found). Still, this is a great trade because understanding requirements is harder than solving well defined problems.

[To the point of this whole post] Requirements can be turned into testable properties even if it's not programmatic. "When I look at a list of chosen songs, there must be no obvious patterns." Who says what's obvious? You do! Then, have someone else do the same.

Consider extreme cases. Extreme cases tend to be the most or least important. If they're least important, create a new set of easier requirements or drop it all together. "If 3 - 10 songs, always play within double the playlist, no obvious patterns, never twice in a row. If 2 alternate, if 1 repeat."

Re: We think this cool study we found is flawed. Help us reproduce it

#246

Post by scraptor »

denton-scratch wrote: post #130

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > t…

All sequences are equally likely to be produced by a fair die but humans are very biased in the kinds of sequences they produce. It might be impossible to ever be certain but you can certainly look at a sequence of all sixes (or that contains more complicated patterns) and estimate that it was much more likely to be produced by a human than a die.

Re: We think this cool study we found is flawed. Help us reproduce it

#247

Post by travisjungroth »

kevinpet wrote: post #229
denton-scratch wrote: post #130

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > t…

Here are some numbers: 123456123456 Did I get them by rolling a die?

Almost certainly not.

Given a circle with diameter AB and a point on the perimeter C, is ABC an equilateral triangle?

Re: We think this cool study we found is flawed. Help us reproduce it

#248

Post by johnnymellor »

macksd wrote: post #238
RuggedPineapple wrote: post #163

Earlier quoted context omitted.

>Can you roll a fair die a thousand times and only get 6s? Well yes of course. It'll never happen though. Somewhat related, but humans are also REALLY bad at generating randomness and one of our big tells is an aversion to repeats. If you ask someone to pick 0-9 randomly, repeatedly, they will rarely repeat numbers. But in a truly random sample a repeat is likely 10% of the time, and a three-peat will happen roughly…

And funnily enough, you'll often hear this trait as being desirable in a pseudo-random number generator. People often want something that will jump around fairly unpredictably but that will come close to outputting all possible numbers once before getting into re-runs.

Yes, it's a very desirable trait in https://en.m.wikipedia.org/wiki/Quasi-Monte_Carlo_method

Quasi-Monte Carlo has a rate of convergence close to O(1/N), whereas the rate for the Monte Carlo method is O(N^(−0.5))

For such applications it's best to use quasi-random numbers (a.k.a. low-discrepancy sequences) such as the Halton sequence or the Sobol sequence instead of pseudorandom numbers.

Re: We think this cool study we found is flawed. Help us reproduce it

#249

Post by penteract »

cheschire wrote: post #231
penteract wrote: post #216

Earlier quoted context omitted.

If it's light when I wake up, I would say that I can tell it's daytime, despite the possibility that it's still nighttime but a sufficiently near star has gone supernova or that the house next door is on fire.

I sense the sarcasm but I'm not sure which way you're intending it to go. What if you live in the arctic circle?

I wasn't trying to be sarcastic, just to give evidence against the statement

> "he/she should not be able to tell" isn't the same as "he/she should not be able to make a statistically-probable guess".

I'm in agreement with the sibling comment by baobabKoodaa.

Re: We think this cool study we found is flawed. Help us reproduce it

#250

Post by icambron »

baobabKoodaa wrote: post #224
icambron wrote: post #113

I agree with the article that the study is flawed in its unwillingness to exclude the all-H and all-T answers. But I’ll go further: the original study is just silly. “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To our algorithm, is what they meant. There’s no such thing as a randomness test that can look at a sequence and decide “is it random?”, so this algo measures somethi…

> “Make a sequence that looks random” is sort of a nonsensical ask. Looks random to whom? To other people. They specified this in the study instructions. (Not defending the study, I think it's flawed too.)

But they didn’t measure that. They’d have to measure that by showing a bunch of people each sequence and asking “is this random?” Instead, they threw a complexity formula at it, which measures some specific thing, but not “other people”

Though perhaps there is a body of existing literature showing that their complexity estimator matches people’s assessment of “randomness”? If so, does it include people over 60?

Post reply on HN