We think this cool study we found is flawed. Help us reproduce it • Hacker News Board

Re: We think this cool study we found is flawed. Help us reproduce it

#351

Post by posix86 » Wed, May 04, 2022, 12:25 PM UTC

roofone wrote: post #235

posix86 wrote: post #189

Earlier quoted context omitted.

You're wrong. Mathematically. Here's why. When you throw a coin 100 times, each sequence you get is equally likely. However. You can look at properties of the sequence which are more likely to be one way than the other. For instance, it's more likely that the number of heads and tails are about equal than not. The reason is that there are more sequences, in general, where that is true, than those where heads or tails…

“Statistically likelier,” but still mathematically possible confirms the OP’s point. This seems to be really testing for pseudo-random numbers. Relevant Dilbert (and article): https://www.lancaster.ac.uk/~blackb/RNG.html .

I don't think this has anything to do with pseudo random numbers :) PRNG are not perfect but their imperfections are impossible to detect by hand.

Re: We think this cool study we found is flawed. Help us reproduce it

#352

Post by posix86 » Wed, May 04, 2022, 12:27 PM UTC

sullyj3 wrote: post #321

posix86 wrote: post #189

Earlier quoted context omitted.

You're wrong. Mathematically. Here's why. When you throw a coin 100 times, each sequence you get is equally likely. However. You can look at properties of the sequence which are more likely to be one way than the other. For instance, it's more likely that the number of heads and tails are about equal than not. The reason is that there are more sequences, in general, where that is true, than those where heads or tails…

This is a good illustration of a binary epistemology vs a continuous one. > It's always impossible to tell, for any given sequence, whether it was produced by a fair die. Something like "you can't make any determination, because it's random". Whereas under the second worldview you can make statements about how likely things are, despite uncertainty. For some reason the binary worldview seems to be incredibly common.…

Could be. Though if you think long enough, with this worldview you can't decide anything, ever. And it's irrational. You can't say for sure which is random, but you can say for sure on which you should bet your money if you have to.

Re: We think this cool study we found is flawed. Help us reproduce it

#353

Post by posix86 » Wed, May 04, 2022, 2:05 PM UTC

jollybean wrote: post #334

posix86 wrote: post #189

Earlier quoted context omitted.

You're wrong. Mathematically. Here's why. When you throw a coin 100 times, each sequence you get is equally likely. However. You can look at properties of the sequence which are more likely to be one way than the other. For instance, it's more likely that the number of heads and tails are about equal than not. The reason is that there are more sequences, in general, where that is true, than those where heads or tails…

The OP is essentially correct. I was definitely confused and assumed it was about 'looking like' randomness. But I did a lot of double clicking of things, because I felt that in 'real life' you're not going to get 1 roll of each number, but odd things happen. But this is a bit moot - the people clicking 'all the same number' have obviously come to some different conclusion as the others - i.e. 'all possible values ar…

I disagree. You want people to click numbers s.t. if you asked them 10k times, a uniform distribution would emerge. But that is not what's happening. They think all numbers have the same probablity, but if you click only 1, then the probability of your choice being random is low.

Edit:

Maybe this will convince you: You said each sequence of numbers is equally likely, hence, we can't tell. I'm going to disagree with that statement.

Let's say I give you a coin, and tell you: I've flipped this coin 100 times in a row, 10k times. And you look at the flips, and each flip result is 1111...111. Would you guess it's random, or biased? The probability of that happening is as high as any other sequence, but clearly, if you'd guess it was random, you'd be a fool. This is exactly what is happening here, just on a smaller scale: 111111111 being the result of the coinflip has a lower probability of being random than the result 100101101110.

11111 has to happen at some point if the experiment is really random. The probability that it happens with YOU is low, however. Thus, it is rational to decide that the sequence is not random. Because it most cases, it won't be.

Re: We think this cool study we found is flawed. Help us reproduce it

#354

Post by posix86 » Wed, May 04, 2022, 2:14 PM UTC

pcthrowaway wrote: post #327

posix86 wrote: post #189

Earlier quoted context omitted.

You're wrong. Mathematically. Here's why. When you throw a coin 100 times, each sequence you get is equally likely. However. You can look at properties of the sequence which are more likely to be one way than the other. For instance, it's more likely that the number of heads and tails are about equal than not. The reason is that there are more sequences, in general, where that is true, than those where heads or tails…

> For instance, it's more likely that the number of heads and tails are about equal than not This isn't even true. Heads and tails being equal over 100 flips is something you'll see something like 8.33% of the time (not based on probability, I just ran a simulation of 100 flips 10,000 times and got 833 instances of them being equal) edit: I missed the key word, "about". Sure, they're more likely within maybe 5-6 of o…

Yes :) The exact probability for having 50%/50% is (100 nCr 50)/2^100, or about (as you found through experiemtation) 0.08.

The more often you run the experiement, the more likely you'll get a result close to 50%/50%, by the way. In the limit, you have a variance (i.e. spread of results away from the expected value, which is 50%) of 0. This is called the law of large numbers. As the generic name suggest, it's pretty central to mathematics haha.

Re: We think this cool study we found is flawed. Help us reproduce it

#355

Post by denton-scratch » Wed, May 04, 2022, 5:19 PM UTC

posix86 wrote: post #350

tidenly wrote: post #203

Earlier quoted context omitted.

Your point doesnt refute OPs argument. Your final statement "you can say the latter is more likely random" is not the same as "you can say this sequence is not random". I think lots of people (especially programmers) who know about true RNG vs expectations of RNG might intentionally put in strings of same numbers, or not include the full set, because we know its what often happens during plain RNG. It isnt clear what…

What does that mean, "you can tell this sequence is not random"? If you show me a blue hat, and ask me if it's blue, I'll say yes, I can say it's blue. But there is always a chance it's not actually blue. It's very conceivable that I'm in a situation where I say with confidence that something is blue, but it isn't. You always only ever speak in probability. Of course you can't say the sequence isn't random, because e…

For a blue hat, it either is or isn't blue (and there's some rather strong evidence - whether or not it looks blue). Like, with a sequence of 6 digits, if you don't know whether the source was random or not, then that's like NOT showing me your hat, and asking me whether it's red or blue.

For a single sequence of six digits, it might or might not have come from a random source. You can't get any edge on that judgement by just inspecting the sequence. Only inspecting the source (the hat, if you like) can give you an advantage. Perhaps you're colour-blind, or the lighting is weird; so there's still uncertainty. But that's equivalent to examining the source of the digits, determining that it's really a random source, but making a mistake in your determination. That's uncertainty on a different level.

Red-pill blue-pill is a sort of meta-uncertainty.

> "is one of the sequences such that it is rational to choose it over the other as being random"

Most people don't care about this shit; it doesn't matter to them what random means, nor whether it's sequences or sources that can be said to be random. But for some people it does matter, and they have to try to use language precisely.

All [red|blue] hats are either red or blue. But no sequence is random or non-random; it's the source of the sequence (the process, if you like) than can be random or non-random.

If 111111 is emitted by a random process, then you can call that a "random sequence" if you like. If I emit 126692 from my ass (not a random process), that's not a "random sequence" in any sense, whatever statistical properties it has. You can't tell which is of random origin by inspection. The experimental subjects face an impossible challenge, and I can't see what conclusions you can draw from their responses.

Re: We think this cool study we found is flawed. Help us reproduce it

#356

Post by denton-scratch » Wed, May 04, 2022, 5:33 PM UTC

sullyj3 wrote: post #321

posix86 wrote: post #189

Earlier quoted context omitted.

You're wrong. Mathematically. Here's why. When you throw a coin 100 times, each sequence you get is equally likely. However. You can look at properties of the sequence which are more likely to be one way than the other. For instance, it's more likely that the number of heads and tails are about equal than not. The reason is that there are more sequences, in general, where that is true, than those where heads or tails…

This is a good illustration of a binary epistemology vs a continuous one. > It's always impossible to tell, for any given sequence, whether it was produced by a fair die. Something like "you can't make any determination, because it's random". Whereas under the second worldview you can make statements about how likely things are, despite uncertainty. For some reason the binary worldview seems to be incredibly common.…

> you can make statements about how likely things are

Sure. And it's true that some sequences are more likely than others to have been emitted by a random process. [Edit] All sequences from a random process are equally likely. It's still true that some sequences are more-likely to have come from non-random processes.

The point is that randomness isn't a property of the sequence; it's a property of the process.

Re: We think this cool study we found is flawed. Help us reproduce it

#357

Post by denton-scratch » Wed, May 04, 2022, 5:54 PM UTC

denton-scratch wrote: post #355

posix86 wrote: post #350

Earlier quoted context omitted.

What does that mean, "you can tell this sequence is not random"? If you show me a blue hat, and ask me if it's blue, I'll say yes, I can say it's blue. But there is always a chance it's not actually blue. It's very conceivable that I'm in a situation where I say with confidence that something is blue, but it isn't. You always only ever speak in probability. Of course you can't say the sequence isn't random, because e…

For a blue hat, it either is or isn't blue (and there's some rather strong evidence - whether or not it looks blue). Like, with a sequence of 6 digits, if you don't know whether the source was random or not, then that's like NOT showing me your hat, and asking me whether it's red or blue. For a single sequence of six digits, it might or might not have come from a random source. You can't get any edge on that judgemen…

Regarding "random process": (and sorry for commenting to myself)

I'm not taking a position on what a "random process" is; for these purposes, a PRNG, a LFSR or even the last three bits of the system-clock would do as well as radioactive decay.

Re: We think this cool study we found is flawed. Help us reproduce it

#358

Post by denton-scratch » Wed, May 04, 2022, 6:15 PM UTC

baobabKoodaa wrote: post #342

denton-scratch wrote: post #340

Earlier quoted context omitted.

It's related only to your challenge with a contrived and complicated betting scenario. "The topic" is whether the challenge faced by the experimental subjects makes any sense. It doesn't; the challenge is to produce a string of six symbols that others can't distinguish from randomness. There is no such string. Your contrived betting scenario doesn't illuminate the issue; it's an attempt to distract attention, and IMO…

> "The topic" is whether the challenge faced by the experimental subjects makes any sense. It doesn't; the challenge is to produce a string of six symbols that others can't distinguish from randomness. There is no such string. I strongly disagree with that statement. On a surface-level inspection, some strings appear to have more entropy than others ("can be distinguished from randomly generated strings"). This absol…

I withdraw my accusation of bad faith, and I apologise for that remark.

I call on you to withdraw your claim that my remarks were made in bad faith.

I still don't know why you felt the need to contrive a complicated betting scenario, when we were discussing a "simpler" scenario that already involved an icosohedral die. If we're using betting scenarios to model good faith, then aren't simple scenarios more useful than complex ones? Ergo, coin-toss is the most appropriate.

But trying to talk about this stuff in terms of physical things like coins or dice inevitably turns into discussion about unfair coins and crooked dice, or whether the caster can influence the outcome; so argument by analogy quickly leads to dead ends, in this area.

Re: We think this cool study we found is flawed. Help us reproduce it

#359

Post by denton-scratch » Thu, May 05, 2022, 6:06 PM UTC

bee_rider wrote: post #313

hakre wrote: post #306

Earlier quoted context omitted.

> I've looked at a lot of random number sequences in my life and I was trying to make it look like one of those. This is perhaps the difference between pseudo and statistically random. No idea which of those the study or the experiment is trying to validate btw. And IIRC, interestingly they write that human capacity to create random numbers declines 25+. I can imagine that the older we are the more we look for someth…

> And IIRC, interestingly they write that human capacity to create random numbers declines 25+. I can imagine that the older we are the more we look for something to make our decisions look more random based on what we've learned so far - more time, there was more time to look at more random number sequences - and the less random the outcome will be. This is what they are testing, and at least based on the data they'…

> This is what they are testing

(human capacity to create random numbers declines 25+)

Not really; what they're testing is what kinds of response differently-aged people give to their question. So it's important what the question actually is; and it's important if the question might, for example, seem to older people to be a waste of their time.

They're not measuring what they claim to be measuring.

Re: We think this cool study we found is flawed. Help us reproduce it

#360

Post by denton-scratch » Thu, May 05, 2022, 6:20 PM UTC

posix86 wrote: post #353

jollybean wrote: post #334

Earlier quoted context omitted.

The OP is essentially correct. I was definitely confused and assumed it was about 'looking like' randomness. But I did a lot of double clicking of things, because I felt that in 'real life' you're not going to get 1 roll of each number, but odd things happen. But this is a bit moot - the people clicking 'all the same number' have obviously come to some different conclusion as the others - i.e. 'all possible values ar…

I disagree. You want people to click numbers s.t. if you asked them 10k times, a uniform distribution would emerge. But that is not what's happening. They think all numbers have the same probablity, but if you click only 1, then the probability of your choice being random is low. Edit: Maybe this will convince you: You said each sequence of numbers is equally likely, hence, we can't tell. I'm going to disagree with t…

> Would you guess it's random, or biased?

Well, I'd guess that it's not a coin-flip at all; even a biased coin won't produce 10,000 heads and no tails, unless it's a two-headed coin.

Let's go back to the actual case in hand: suppose you provide me with "111111", and not 10,000 1s. I simply have no way at all of determining whether that is more or less likely to have come from a random source. So I would decline your bet. If it was 10,000 1s, then maybe I'd be a fool to bet it was of random origin; but you can't convince me that a string of 6 1s is or isn't of random origin. So no bet.

This is all irrelevant. We're discussing a single sequence of 6 digits. There are not enough samples to perform statistical analysis. Probability doesn't come into it.