We think this cool study we found is flawed. Help us reproduce it

absolutely, probably you just did.

> The question is: can you construct a sequence of dice-casts that an adversary can't distinguish from a real dice-cast?

Yes, obviously:

A: 2 4 4 3 5 2 3 6 4 2

B: 5 5 5 6 4 1 1 2 4 5

One is a real dice-cast with a d6 I had lying around. The other is the (100% deterministic) output of:

  echo 1651434259 $X | md5sum | grep -o '[1-6]' | paste -sd' '

for some X. Feel free to explain[0] how you are able to distinguish which is which.

The real issue is that most people don't bother to produce random numbers in a way that's actually secure (which, to be fair, is rather tedious if you don't have a computer handy, and downright prohibitively impractical if you want to do it all in your head, so why would you bother?), either in the study or in general.

0: If you'd like a more black-box distinguishment, I can provide a longer list; obviously a adversary can get the right answer 50% of the time just by chance.

> I've looked at a lot of random number sequences in my life and I was trying to make it look like one of those.

This is perhaps the difference between pseudo and statistically random. No idea which of those the study or the experiment is trying to validate btw.

And IIRC, interestingly they write that human capacity to create random numbers declines 25+. I can imagine that the older we are the more we look for something to make our decisions look more random based on what we've learned so far - more time, there was more time to look at more random number sequences - and the less random the outcome will be.

Yeah but their guess shouldn't be wrong 50% of the time as again that means that they can’t have picked the 95th percentile result! Because it’s 50:50 I’ll assume that they are assigning people scoring higher than average the “under 60” category - which is obviously incorrect. Otherwise how do they pick the cut off?

To explain with another example - let's say that I have a dataset of 100 people's scores at golf (no handicaps) and I know that 5% of them are pro-players and others are 'advanced amateurs'. Because of this I might take the top 5 scores and guess that they are pro's and assign the others the guess of 'advanced amateur'.

Now let's say that there was actually no correlation between people's scores at golf and their 'pro' status - what accuracy would I expect in the above experiment? The answer is actually closer to 90% 'accurate guesses' than 50%! (Although obviously - that's 90% accurate based on random chance).

Now if someone told me they got 50% of the guesses wrong at this task, that implies that they guessed that the top 50% of those golfers were pro rather than picking the top 5% of scores, and I would question the methodology.

This % is similar to the dataset in the webpage - I downloaded it, filtered out exclusions and c4% of the valid responses are 60 or over.

If I inherently pick a small population (i.e. over 60's are c4% in this dataset) and I am guessing wrong 50% of the time, it means that my cut-off is incorrectly calibrated. Their score cut-off should, at worst, be picking the wrong 4% and missing another 4%.

Am I going crazy? It seems logical to me, but to be open maths isn't my strong point. I just know that if I designed the guessing rule, I would be getting more than 50% (my algorithm would be 'if the users average score across the three tests is less than -1.5, assign 'over 60' and that would get c95% accurate guesses, albeit it would still not prove anything and I agree with the authors overall premise!).

After doing it for real, I tried it a few more times with a cryptographically random shuffle. Interestingly, the computer got a worse score on average.

Method:

I started `irb`, and picked like so:

  require 'securerandom'
  # Coin flip 
  12.times { p [:heads, :tails].shuffle(random: SecureRandom)[0] }
  # Dice roll
  10.times { p (1..6).to_a.shuffle(random: SecureRandom)[0] }
  # 10 dots
  10.times { p (1..9).to_a.shuffle(random: SecureRandom)[0] }

I disconnected from the internet before the scores were submitted so as not to taint the survey. (Ranking calculation happens offline based on a CSV of scores loaded into the browser early on.)

Results:

For: "Your answer got a higher random score than X% of people in the study."

Me: 46%, 71%, 59%, 72%

Computer: 32%, 20%, 47%, 31%

My sample size is obviously too small for anything conclusive. However, I'll admit it makes me a little suspicious that something else is amiss.