We think this cool study we found is flawed. Help us reproduce it • Hacker News Board

Re: We think this cool study we found is flawed. Help us reproduce it

#261

Post by [deleted] » Sun, May 01, 2022, 4:21 PM UTC

posix86 wrote: post #189

denton-scratch wrote: post #130

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > t…

You're wrong. Mathematically. Here's why. When you throw a coin 100 times, each sequence you get is equally likely. However. You can look at properties of the sequence which are more likely to be one way than the other. For instance, it's more likely that the number of heads and tails are about equal than not. The reason is that there are more sequences, in general, where that is true, than those where heads or tails…

[deleted]

Re: We think this cool study we found is flawed. Help us reproduce it

#262

Post by sohdas » Sun, May 01, 2022, 4:22 PM UTC

tomsmeding wrote: post #26

https://xkcd.com/1725/ . Those regression lines are absolutely laughable. Point cloud has line going through it, and beforehand I have no clue where the line will be (other than that it will go through the center, presumably). Even if there is a statistically significant effect, its meaning will be a rounding error.

Isn't that the point, though? Their argument is that there is no correlation, in contrast to what the original study claimed

Re: We think this cool study we found is flawed. Help us reproduce it

#263

Post by sohdas » Sun, May 01, 2022, 4:25 PM UTC

Closi wrote: post #119

plorntus wrote: post #67

I feel like a lot of the comments here are written after only taking the test and many are not reading the rest of the article. The authors of the website are stating that they believe the study is wrong. The below/above 60 answer is showing you it’s incorrect half of the time along with data backing up the claim.

But their data doesn't make sense to be personally... Only 5% of their dataset is above the age of 60, making their claim that they are getting 50% of their guesses wrong seem like they are calculating it wrong. Surely their cut-off should be at the 95th percentile of the data? They shouldn't be guessing 'under 60' the same proportion of times as 'over 60', because their population is mostly under 60.

Again though, they are arguing that there is no correlation between randomness and age. This was just a demonstration that when they use randomness to predict age, the results are wrong 50% of the time-- which is precisely in accordance with their hypothesis

Re: We think this cool study we found is flawed. Help us reproduce it

#264

Post by travisjungroth » Sun, May 01, 2022, 4:28 PM UTC

baobabKoodaa wrote: post #252

travisjungroth wrote: post #240

Earlier quoted context omitted.

It depends on the framework. I can tell a geometric figure is a square because it’s a quadrilateral with right angles and sides of equal length. You could ask me a question like “Is a rectangle with a side of length 1 and a diagonal of root 2 a square?” and I can tell it is. Ask me “Was 1 1 1 1 produced by a random process?” and it’s impossible to tell in the way I did with the square.

> It depends on the framework. I can tell a geometric figure is a square because it’s a quadrilateral with right angles and sides of equal length. You're claiming to be able to craft a mathematical proof with 100% certainty. Although the thing you are proving appears to be obviously true (assuming a certain mathematical framework), the probability that you made a mistake is not 0%. You might falsely believe that the…

You’re mixing up two layers of uncertainty. There’s an outer uncertainty. This would include things like I made a mistake, this is all a dream, etc. This outer uncertainty pervades all problems.

It’s often useful to ignore that outer uncertainty. We create a framework where we take certain things as true (shared reality, mathematical axioms). This framework may or may not have uncertainty inside of it, which we could call inner uncertainty.

Questions of probability have inner uncertainty. Questions of geometry do not. This makes them qualitatively different.

If you frame the initial task as something like “do your best to lead people to believe your sequence is random”, that makes sense. If the task is “make it so they can’t tell if it’s random”, that’s a bit off in some way. At the very least, it’s because you’ve presented the spotting of randomness as something that can truly be done to a logical conclusion (random/not or true/false). This violates both the outer and inner uncertainties of randomness.

Re: We think this cool study we found is flawed. Help us reproduce it

#265

Post by denton-scratch » Sun, May 01, 2022, 4:33 PM UTC

HWR_14 wrote: post #150

denton-scratch wrote: post #130

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > t…

> It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. That's just not true. Or feel free to play a game with me. We'll roll a 20 sided die. If it comes up 20, you give me a ten. If it comes up any other number, I'll get you a dollar. Nice EV on that! Oh, the die has come up 20, 20, 20, 20,…

Hey, out of curiosity, why did you choose a 20-sided die? I assume it was just to match the "ten bucks" bit. I'm not snarking, I just began thinking about 20-sided dice.

I do not believe there is a solid with 20 faces, and all faces congruent (I haven't checked). So we end up with a solid that is roughly ball-shaped, with faces of different sizes and shapes.

[Edit] I checked; I suspected I'd made a fool of myself. An icosahedron has 20 congruent faces, of course.

A ball-shaped die is much more likely to topple, and so more likely to be influenced by small differences in weight distribution, selective corner-shaving, whatever.

I can't think of a way of judging the fairness of a 20-sided die other than casting it many times, and analysing the results. I'd be much more confident in my ability to judge by inspection the fairness of a 6-sided die.

Re: We think this cool study we found is flawed. Help us reproduce it

#266

Post by icambron » Sun, May 01, 2022, 4:33 PM UTC

baobabKoodaa wrote: post #253

icambron wrote: post #250

Earlier quoted context omitted.

But they didn’t measure that. They’d have to measure that by showing a bunch of people each sequence and asking “is this random?” Instead, they threw a complexity formula at it, which measures some specific thing, but not “other people” Though perhaps there is a body of existing literature showing that their complexity estimator matches people’s assessment of “randomness”? If so, does it include people over 60?

> But they didn’t measure that. That's correct. > They’d have to measure that by showing a bunch of people each sequence and asking “is this random?” Agreed! > Instead, they threw a complexity formula at it, which measures some specific thing, but not “other people” Yep, they botched the study. Nonetheless, the "ask" that you criticized was fine. It explained that the subject should craft a pattern which would appear…

Alright, fair enough

Re: We think this cool study we found is flawed. Help us reproduce it

#267

Post by sacrosancty » Sun, May 01, 2022, 4:35 PM UTC

graderjs wrote: post #146

denton-scratch wrote: post #130

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > t…

Too deep topic and maybe I don't understand it compared to the megaminds here on hacker News but as far as I get it when it comes to randomness every sequence has the same probability which is simply the inverse of the number of sequences for any given length. In that case it seems to me impossible to tell from only the information of a single sequence. But if you have multiple sequences where one is the binary codin…

I think your way of looking at it requires an explosion of alien civilisations to find one in which our random 1000 digits is their special number. If you generate 1 extra digit, you'd need 10x as many civilizations to expect a match.

Getting back to the experiment, obviously it's measuring the human mind so the context is the person's mind. They might enter their bank account number which looks random to the researchers but is actually cheating because they're not using their brain as the source of randomness.

Re: We think this cool study we found is flawed. Help us reproduce it

#268

Post by sacrosancty » Sun, May 01, 2022, 4:36 PM UTC

bandushrew wrote: post #50

Over 60 according to something, but I’m not. Hypothesis: the older a person is, the more likely they are to understand that HHHHHHHHH is a statistically likely outcome from true randomness.

I suspected it was the older a person is, the less of an eff they give about psychology researchers' questions :P

Re: We think this cool study we found is flawed. Help us reproduce it

#269

Post by justinpombrio » Sun, May 01, 2022, 4:40 PM UTC

denton-scratch wrote: post #130

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > t…

> That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible.

Baloney!

Say you measure the traffic to your website in the morning and the evening, every day for a week. And this is what you see:

    day:      1  1  2  2  3  3  4  4  5  5  6  6  7  7
    time:     M  E  M  E  M  E  M  E  M  E  M  E  M  E
    visitors: 51 73 58 72 50 78 55 74 55 77 52 73 55 76

It could be that the traffic you get is uniformly random, and each day the number of visitors you get is a uniformly random number between 50 and 80. Sure, all the morning numbers are less and the evening numbers are more and there are suspiciously no numbers in the 60s. All that could be a coincidence.

You know perfectly well, though, that you're getting more traffic in the evenings.

Re: We think this cool study we found is flawed. Help us reproduce it

#270

Post by furyofantares » Sun, May 01, 2022, 4:47 PM UTC

A mathematical model sometimes used for shuffling cards is that after the cut, as you joke N cards in one hand and M in the other, the next card to fall has N/(N+M) chance of calling from the hand with N, and M/(N+M) chance from the band with M. This is used to come up with results like 8 riffle shuffles being needed to randomize a deck of cards.

I sometimes thing about how this model gives an extremely small but non-zero chance that you drop all the cards from the left hand before the right and end up with no shuffle at all. Or just one or two cards fall before the other hand. And I'm sure this has actually happened to people!

But if you're counting your shuffles trying to get to 8 to call it randomized, nobody would count that shuffle. They'd redo it.

I don't think anyone submitting HHHHHHHHHH or TTTTTTTTTT does so thinking they've generated a random sequence.