Live data from Hacker News

We think this cool study we found is flawed. Help us reproduce it

pudding.cool

81–90 of 361 posts

Re: We think this cool study we found is flawed. Help us reproduce it

#81

I understood my task as convincing another _human_ that it was randomly generated. Since N was low for each of the tasks, I was deliberate about sometimes having repeated values, and not ensuring that every option was picked an equal number of times, since that looks suspiciously algorithmic. Apparently I'm over 60.

Same reasoning and I got under 60, more random than 74% of the responses.

Re: We think this cool study we found is flawed. Help us reproduce it

#82

I understood my task as convincing another _human_ that it was randomly generated. Since N was low for each of the tasks, I was deliberate about sometimes having repeated values, and not ensuring that every option was picked an equal number of times, since that looks suspiciously algorithmic. Apparently I'm over 60.

Exactly, and I guess my days are numbered.

Re: We think this cool study we found is flawed. Help us reproduce it

#83

The more surprising part to me was that quote about the 2015 study that tried to reproduce 100 studies and managed to replicate 39. That's honestly way better than I expected for psychology. Here is the paper: https://www.nature.com/articles/nature.2015.18248

No post body was provided.

Re: We think this cool study we found is flawed. Help us reproduce it

#84
post #38

Beautifully crafted web page. For any experimental science, the integrity of experiment (thus reliability of data) is important. For experiments with human subjects, the question is whether the subjects answered the questions in good faith. A sequence like 'HHHHHHHHHH' for the coin toss experiment looks like an answer in bad faith; it is mechanically easy to keep pressing the same button, and a subject is unlikely to…

If you're going to exclude bad faith answers, I think you should exclude all of them. But I don't think you can do that. Is HTHTHTHTHT a bad faith answer? Always or only sometimes? We're trying to infer the test subject's intent from their answer, and that's fundamentally impossible I think.

I think including all answers is a solid approach. If test subjects have bad faith, I think that can be filed under 'less random'. If old test subjects show more bad faith, I think it's not really wrong to say older people are less random. And it does have predictive power.

Arbitrarily (because they is no way to do it subjectively) excluding some answers and not others has, I think, a greater risk of skewing the results.

Re: We think this cool study we found is flawed. Help us reproduce it

#85

Very interesting, and I think this part sums up the crux: “The researchers believe that you can only analyze the raw responses because, statistically, any sequence is equally likely to occur, so where do you draw the line?” I’d say that, as it is a psychological study, making claims about a human behavior, treating humans as pure random number generators without considering _intent_ is a mistake. It is entirely possi…

But how can you go from a random sequence to the intent of a subject? I don’t think you really can.

One thing many surveys/studies do is to include "trap" questions (I'm sure there's a real name for 'em) which disqualifies any participants that answer them incorrectly.

Re: We think this cool study we found is flawed. Help us reproduce it

#86
post #78

I was trying to find a mental way to do a true coin toss. Anyone has ideas for how to truly dig into some randomness? Maybe most would think it's impossible, but aren't we at least better placed to do this better than deterministic machines (or maybe we are not - the true free will debate :)?

If you have two people, you can have person A ask person B to “pick a random number”, and use heads for odd, tails for even. Don’t tell person B why you’re asking and my guess is you’re relatively random heads/tails. No studies that I am aware of to back this up, so people could be biased towards odd/even, but a bias correction could correct that too.

Re: We think this cool study we found is flawed. Help us reproduce it

#87
post #56

Earlier quoted context omitted.

The measure of randomness chosen in this particular paper appears to be an approximation of the Kolmogorov-Chaitin complexity adapted for small integer/binary sequences. This effectively looks at how easy it would theoretically be to compress/describe the data, for instance HHHHHHHHHH would be low complexity as it could be encoded as '10 H's'. If something is truly random, it shouldn't be possible to encode it due to…

> If something is truly random, it shouldn't be possible to encode it due to the pigeon-hole principle. This statement is obviously untrue. “Random numbers” don’t really exists. The original authors were right about that. Every number/sequence is equally likely to occur. There’s even an XKCD about this [0]. I guess what you mean is: If you have a process that generates sequences randomly, most of those sequences are…

Yes, you are right on this one - I was 100% wrong and your corrected statement is right

Re: We think this cool study we found is flawed. Help us reproduce it

#88
post #38

Beautifully crafted web page. For any experimental science, the integrity of experiment (thus reliability of data) is important. For experiments with human subjects, the question is whether the subjects answered the questions in good faith. A sequence like 'HHHHHHHHHH' for the coin toss experiment looks like an answer in bad faith; it is mechanically easy to keep pressing the same button, and a subject is unlikely to…

If you're going to exclude bad faith answers, I think you should exclude all of them. But I don't think you can do that. Is HTHTHTHTHT a bad faith answer? Always or only sometimes? We're trying to infer the test subject's intent from their answer, and that's fundamentally impossible I think. I think including all answers is a solid approach. If test subjects have bad faith, I think that can be filed under 'less rando…

However, the study concluded that person's ability to produce randomness peaks at 25. An increase in showing bad faith doesn't tell us anything about the ability to produce randomness if desired. Thus, if we accept the bad faith answers as part of the data, the conclusion of the study becomes incorrect, at least in wording.

Re: We think this cool study we found is flawed. Help us reproduce it

#89

I understood my task as convincing another _human_ that it was randomly generated. Since N was low for each of the tasks, I was deliberate about sometimes having repeated values, and not ensuring that every option was picked an equal number of times, since that looks suspiciously algorithmic. Apparently I'm over 60.

As I understood it, their entire point is that younger people are not better at generating random sequences than older people, so guessing someone's age based on their complexity (randomness) score is completely unreliable. Towards the bottom of the page they said they've only guessed age correctly 51% of the time, which lines up with there being no correlation between age and ability to generate random sequences

My point isn't the age result that I mentioned. (I believe their claim that it's bogus.) It's that the instruction to click "as randomly as possible" is ambiguous so at best they're measuring an average of the behaviours they think they are.

Re: We think this cool study we found is flawed. Help us reproduce it

#90
post #38

Beautifully crafted web page. For any experimental science, the integrity of experiment (thus reliability of data) is important. For experiments with human subjects, the question is whether the subjects answered the questions in good faith. A sequence like 'HHHHHHHHHH' for the coin toss experiment looks like an answer in bad faith; it is mechanically easy to keep pressing the same button, and a subject is unlikely to…

This is not a sound approach. You're declaring what humans think random is first, and then throwing out any data that doesn't match your declaration. There is no way to learn anything from this. I also think 'HHHHHHHHHH' is unlikely to be a good faith response, but if the goal is to actually learn anything instead of merely reinforcing my prior beliefs, it doesn't matter. You need to find a way to design the experime…

It seems to me that if we reject a subset of experimental samples because they look like bad data (e.g. extreme outlier caused by sensor malfunction) we are still keeping all the bad data we are unable to recognize as such (e.g. sensor malfunctions producing less extreme data), which introduces a bias.
Post reply on HN