Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We think this cool study we found is flawed. Help us reproduce it

pudding.cool

Post reply on HN
Print view View original

211–220 of 361 posts

Re: We think this cool study we found is flawed. Help us reproduce it

#211

Post by ajmurmann »

bogwog wrote: post #208

Wouldn't a better prompt be something like > Click the button to roll the die > (plausible dice roll animation is played) > Try to guess the result > (buttons labeled 1-6) That way there's no confusion about how to interpret the instructions, it's just people trying to predict the results of an event they perceive to be random and fair. When it asked me to create a "good" outcome, I didn't really know what it was exp…

I'm not sure if guessing the result would be any better. Knowing anything about a fair dice, you might as well just stick to the same result for all tosses.

However, I approached the exercise in the same way as you did, as an adversarial game. I tried to generate numbers that would trick a human who is tasked with filtering out the series as random. No results repeating stuck me as a tell that the series was done by a human, so I included a fair number of repetitions of the same result that's still statistically plausible. Not sure how to work around that in the instructions.

Edit: I think the study might say more about what the player thinks about other's expectation of randomness than about the player's own understanding of randomness.

Re: We think this cool study we found is flawed. Help us reproduce it

#212

Post by d33k4y »

apwheele wrote: post #193

Not about randomness - but about curve fitting. It is actually very difficult to verify non-linear effects -- or maybe I should say the opposite. The statistical tools we use to identify non-linearities are prone to be very noisy, so even in the subset of data including the no-variation responses I am quite skeptical that downward increase is real, or just due to variance in the tails of the data. So a common social…

I'm surprised to see this comment not echoed by more people. Besides the inadequate sample size of older participants, my immediate response to the variance in the data was that I doubt the slope could be distinguished from zero.

Re: We think this cool study we found is flawed. Help us reproduce it

#213

Post by JoeAltmaier »

All sequences are equally probably. But that's not the thing being asked to distinguish. It's whether a human being produced the sequence. Humans choose patterns. Distinguishing between 'random chance' and 'patterned response' is much, much easier statistically.

Re: We think this cool study we found is flawed. Help us reproduce it

#214

Post by d33k4y »

Hbruz0 wrote: post #125
plorntus wrote: post #67

I feel like a lot of the comments here are written after only taking the test and many are not reading the rest of the article. The authors of the website are stating that they believe the study is wrong. The below/above 60 answer is showing you it’s incorrect half of the time along with data backing up the claim.

I lol'd at the "Trend line": https://imgur.com/a/ohYbcLL

https://xkcd.com/2048/

Re: We think this cool study we found is flawed. Help us reproduce it

#215

Post by blackboxlogic »

ryukoposting wrote: post #206
blackboxlogic wrote: post #94

Earlier quoted context omitted.

Look around the room for objects in sight. For each object, take its common name and count how many of the letters are "odd" letters acegikmoqsuwy, then mod 2. "Window" -> "wiow" -> 4 -> 0. Each word yields a single bit of very slow, pretty good entropy. Don't do this in the same room twice.

I wonder how effective it would be simply to count vowels mod 2. Much faster to calculate, at least for me.

I suspect there's a strong bias in vowelCount % 2. A quick look at English 100 most common [1] has 1's at 73% and 0's at 27%. That would even out with longer words but I wonder how much. Maybe there's something else that's just as easy with less bias?

[1]https://www.englishclub.com/vocabulary/common-words-100.htm

Re: We think this cool study we found is flawed. Help us reproduce it

#216

Post by penteract »

denton-scratch wrote: post #205
posix86 wrote: post #189

Earlier quoted context omitted.

You're wrong. Mathematically. Here's why. When you throw a coin 100 times, each sequence you get is equally likely. However. You can look at properties of the sequence which are more likely to be one way than the other. For instance, it's more likely that the number of heads and tails are about equal than not. The reason is that there are more sequences, in general, where that is true, than those where heads or tails…

"he/she should not be able to tell" isn't the same as "he/she should not be able to make a statistically-probable guess".

If it's light when I wake up, I would say that I can tell it's daytime, despite the possibility that it's still nighttime but a sufficiently near star has gone supernova or that the house next door is on fire.

Re: We think this cool study we found is flawed. Help us reproduce it

#217

Post by eterevsky »

I think you are conflating a little bit randomness and complexity.

Scott Aaronson in "Quantum Computing since Democritus" cites a study that shows that when people are asked to generate a random sequence the sequence that they write looks _more complex_ than a truly random sequence. For example, when generating a sequence of coin tosses people would try to avoid long sequences of heads of tails, making the probability of those sequences lower than in a truly random sequence.

I decided to try and test this using your game. I generated the results for all three tests using random.org, and ended up somewhere between 10th and 20th percentile of complexity.

Re: We think this cool study we found is flawed. Help us reproduce it

#218

Post by dorgo »

geysersam wrote: post #164
HWR_14 wrote: post #150

Earlier quoted context omitted.

> It's always impossible to tell, for any given sequence, whether it was produced by a fair die. There's nothing an experimental subject can do to make the impossible more impossible. That's just not true. Or feel free to play a game with me. We'll roll a 20 sided die. If it comes up 20, you give me a ten. If it comes up any other number, I'll get you a dollar. Nice EV on that! Oh, the die has come up 20, 20, 20, 20,…

It's just a question about what we perceive as random. It has nothing to do with the probability of the sequence being produced by a die, only with the probability of the sequence being produced by a human. A 20 20 20 sequence is not less random, it's just more likely to be produced by someone with incentive to cheat. How did the researchers measure the "randomness" of a particular sequence in this experiment? > Form…

I think we want (and expect) RNGs to produce sequences with high algorithmic complexity (which we regard as "random") and ignore the (almost impossible) possibility that they fail to do so.

An impossible to create modified RNG which always produces sequences with high algorithmic complexity would better meet our expectations of randomness, but would be less random because it could not produce uniform sequences (among others).

Re: We think this cool study we found is flawed. Help us reproduce it

#219

Post by stingraycharles »

Shaanie wrote: post #85
Hendrikto wrote: post #73

Earlier quoted context omitted.

But how can you go from a random sequence to the intent of a subject? I don’t think you really can.

One thing many surveys/studies do is to include "trap" questions (I'm sure there's a real name for 'em) which disqualifies any participants that answer them incorrectly.

I think they’re called “control questions”, as their purpose isn’t related to the study, but rather to control for BS answers.

Re: We think this cool study we found is flawed. Help us reproduce it

#220

Post by baobabKoodaa »

denton-scratch wrote: post #205
posix86 wrote: post #189

Earlier quoted context omitted.

You're wrong. Mathematically. Here's why. When you throw a coin 100 times, each sequence you get is equally likely. However. You can look at properties of the sequence which are more likely to be one way than the other. For instance, it's more likely that the number of heads and tails are about equal than not. The reason is that there are more sequences, in general, where that is true, than those where heads or tails…

"he/she should not be able to tell" isn't the same as "he/she should not be able to make a statistically-probable guess".

By this logic the expression "being able to tell" should be banned from the English vocabulary, because no-one is able to tell anything with 100% certainty. Requiring 100% certainty as a precondition of using this expression is silly.
Post reply on HN