Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We think this cool study we found is flawed. Help us reproduce it

pudding.cool

Post reply on HN
Print view View original

311–320 of 361 posts

Re: We think this cool study we found is flawed. Help us reproduce it

#311

Post by Strilanc »

denton-scratch wrote: post #130

> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody. That instruction is a flaw in the experiment. It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. > t…

Let's not confuse P(observed_rolls|used_dice) with P(used_dice|observed_rolls). P(observed_rolls|used_dice) is always the same, independent of observed rolls, assuming the dice are fair. But P(used_dice|observed_rolls) can vary, because other ways of generating rolls which are under consideration may be biased towards certain answers, and this allows you to perform inference.

For example, suppose the rolls you will be shown were either generated by fair dice or by the program "always return 4" [4]. The rolls you are given are "4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4". Are you really thinking you'd make the same prediction for this sequence of rolls as you would for the sequence "1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1", or is there perhaps some SMALL INKLING OF A HINT as to which answer is correct?

The simple fact of the matter is that, in the real world, if you see a sequence like "4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4" you can be surprisingly confident that it was not generated by die rolls. This is because there are plenty of other ways to get sequences and those other hypotheses didn't just pay a Bayes factor penalty of a trillion. Seeing the instructions as incoherent is the mistake of trying to over-isolate the study to the abstract mathematical realm, instead of the world people actually operate in. If someone tells you their luggage combination is 1234, do you really think it's meaningless to guess that it was the default combination as opposed to being generated by secure die rolls? Do you not form opinions about whether or not someone is using secure randomly generated passwords when you find out their password is "password2"?

4: https://xkcd.com/221/

Re: We think this cool study we found is flawed. Help us reproduce it

#313

Post by bee_rider »

hakre wrote: post #306
zeven7 wrote: post #297

Earlier quoted context omitted.

I also scored 60+ (actual age is in my 30s). I had similar thoughts and also did things like not use up all the numbers and repeat numbers more than twice exactly because I've looked at a lot of random number sequences in my life and I was trying to make it look like one of those.

> I've looked at a lot of random number sequences in my life and I was trying to make it look like one of those. This is perhaps the difference between pseudo and statistically random. No idea which of those the study or the experiment is trying to validate btw. And IIRC, interestingly they write that human capacity to create random numbers declines 25+. I can imagine that the older we are the more we look for someth…

> And IIRC, interestingly they write that human capacity to create random numbers declines 25+. I can imagine that the older we are the more we look for something to make our decisions look more random based on what we've learned so far - more time, there was more time to look at more random number sequences - and the less random the outcome will be.

This is what they are testing, and at least based on the data they've got so far, it looks like it increases up to 25-ish and then stays pretty flat.

Another possibly interesting observation is that their preliminary data set (just eyeballing it, but) looks to have gotten

1) a flatter response

2) generally, less random responses

Which leads me to wonder if the live stats have been skewed more random as there might be some correlation between "interested in this sort of thing" and "has some idea what a random distribution ought to look like," and possibly this knowledge doesn't go away with age.

Re: We think this cool study we found is flawed. Help us reproduce it

#315

Post by [deleted] »

runlevel1 wrote: post #310

After doing it for real, I tried it a few more times with a cryptographically random shuffle. Interestingly, the computer got a worse score on average. Method: I started `irb`, and picked like so: require 'securerandom' # Coin flip 12.times { p [:heads, :tails].shuffle(random: SecureRandom)[0] } # Dice roll 10.times { p (1..6).to_a.shuffle(random: SecureRandom)[0] } # 10 dots 10.times { p (1..9).to_a.shuffle(random:…

[deleted]

Re: We think this cool study we found is flawed. Help us reproduce it

#316

Post by alex_smart »

native_samples wrote: post #116
alex_smart wrote: post #95

Earlier quoted context omitted.

You don't have to view it as "throwing out the data". You can just think of it as an alternative explanation for the data. Original hypothesis: Old people are worse at giving random responses. Alternative hypothesis: Old people are more likely give bad faith responses. This review is suggesting the AH is equally good at explaining the data as the OH.

Although technically, this would be P-hacking. You aren't meant to change your hypothesis post-facto to fit the data. You'd have to conclude no effect, and then design a separate study to determine if age differences correlate with bad faith answers.

It would be p-hacking if we just took the same data to conclude that old people are more likely to give bad faith responses. That is just a possible explanation for the data being offered to reject the original hypothesis.

At the very least, it is an interesting observation that the entire trend line disappears on removing data points where people guess all same coin toss results.

Re: We think this cool study we found is flawed. Help us reproduce it

#317

Post by doetoe »

andi999 wrote: post #301
travisjungroth wrote: post #240

Earlier quoted context omitted.

It depends on the framework. I can tell a geometric figure is a square because it’s a quadrilateral with right angles and sides of equal length. You could ask me a question like “Is a rectangle with a side of length 1 and a diagonal of root 2 a square?” and I can tell it is. Ask me “Was 1 1 1 1 produced by a random process?” and it’s impossible to tell in the way I did with the square.

Ask me "Was 19 19 19 19 19 19 19 produced by a random process" and I can say 'most likely not!'. But then: https://www.dailymail.co.uk/news/article-2162190/What-odds-R...

Interestingly, the article computes the odds incorrectly: "... hit the same number on seven consecutive spins [...] the odds of which happening are 114billion to one...", which actually are the odds of having 7 consecutive 19s or the same (unspecified) number on 8 consecutive spins.

Re: We think this cool study we found is flawed. Help us reproduce it

#318

Post by Groxx »

I decided to chase "how are they measuring randomness?" down, and:

- the study mentions using the "acss" R package in the Methods section: https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...

- acss's documentation directs you to another site for describing the metric: https://www.rdocumentation.org/packages/acss/versions/0.2-5/...

- that site links to some other articles in its bibliography at the bottom, and mentions using "algorithmic probability" to approximate complexity: https://complexitycalculator.com/methodology.html

- the first of which is this, which describes algorithmic probability over halting 2-symbol 4-state Turing machines by exhaustive execution: https://www.sciencedirect.com/science/article/abs/pii/S00963...

So assuming I followed all that correctly:

Your strings are more complex if there are fewer Turing machines that produce it, which is then normalized to the average of all strings to become "randomness". Too few or too many ways to create a string means less random, within this category of simple and very small Turing machines (otherwise it'd be impractical to compute).

I have no idea how broadly applicable that metric is though. It seems fairly niche to my complete amateur reading... but AFAICT all Kolmogorov complexity measures are extremely niche, as it's extremely sensitive to what the execution environment is.

But this was still an interesting rabbit hole. Figured I'd share:)

Re: We think this cool study we found is flawed. Help us reproduce it

#319

Post by denton-scratch »

baobabKoodaa wrote: post #294
denton-scratch wrote: post #290

Earlier quoted context omitted.

I guess my "supposed" beliefs must be the beliefs you suppose I have. Whatever. If you're offering me a bet, and you can easily set it up, then what bet are you proposing? You haven't been very specific. I'm no Turf Accountant[0], but I can spot a three-card-trick when I see one. [0] https://en.wikipedia.org/w/index.php?title=Turf_accountant

Fine. You made this statement: > the instructions invite the subject to produce a sequence that they think will convince people it was produced by a roll of dice. But there is no sequence that SHOULD have that power to convince. Let's gather a random sample of people 20 people. I will produce 10 manually generated sequences of dice rolls and 10 actual dice roll sequences. The sequences are added to a list and the lis…

No bet!

You promised me "infinite amounts of money", I only stand to win 20 bucks.

Also, you have specified that these are 20 random people; so I guess I don't get to brief them in advance that they MUST say manual each time. So you have replaced me, the bettor, with a panel of 20 people whose average IQ is 100, and who don't have my interests at heart. Why would I take that bet?

If my random panel say manual each time, you stand to lose.

But as I say, I don't bet often. Only once a year, only on gee-gees, and only as much as I'm willing to lose (because I always lose).

Re: We think this cool study we found is flawed. Help us reproduce it

#320

Post by denton-scratch »

a1369209993 wrote: post #304
denton-scratch wrote: post #284

Earlier quoted context omitted.

You've answered a different question. The question is: can you construct a sequence of dice-casts that an adversary can't distinguish from a real dice-cast? Answer: you can't.

> The question is: can you construct a sequence of dice-casts that an adversary can't distinguish from a real dice-cast? Yes, obviously: A: 2 4 4 3 5 2 3 6 4 2 B: 5 5 5 6 4 1 1 2 4 5 One is a real dice-cast with a d6 I had lying around. The other is the (100% deterministic) output of: echo 1651434259 $X | md5sum | grep -o '[1-6]' | paste -sd' ' for some X. Feel free to explain[0] how you are able to distinguish which…

Not sure that I'm disagreeing with you; but:

Can you contruct a sequence that an adversary CAN distinguish from a real dice-cast?

If you can't distinguish a spoof from the real article, then that blade has two edges. It's impossible to distinguish them, so the instruction in the pudding test that you are to make a sequence that is indistinguishable from a dice-cast is meaningless, because any sequence is indistinguishable from a dice-cast.

If you ask people to do impossible things before breakfast, then it's not sensible to do an analysis of what they end up doing. It's a waste of time.

Post reply on HN