Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We think this cool study we found is flawed. Help us reproduce it

pudding.cool

Post reply on HN
Print view View original

41–50 of 361 posts

Re: We think this cool study we found is flawed. Help us reproduce it

#41

Post by labster »

AnnikaL wrote: post #19
Closi wrote: post #3

> As for our initial idea to make an age-guessing game, we have guessed right 51% of the time. Pretty much what we had expected . They guessed I was 'Under 60', which I am, but over 50% of people fit into the category of under 60... so the fact they can guess this with only 50% of accuracy doesn't really feel right?

I also guess that the segment of the population interested in a self-described "digital publication that makes data fun" probably skews even younger than the population. FWIW I was guessed to be over 60 but am not.

Yeah, most of us here got “over 60” because we inputted data with repeated values, because we know real random data has repeated values. I’m not sure if we overdid it or the study has a weird metric of “looks random”.

Re: We think this cool study we found is flawed. Help us reproduce it

#42

Post by telesilla »

thombles wrote: post #27

I understood my task as convincing another _human_ that it was randomly generated. Since N was low for each of the tasks, I was deliberate about sometimes having repeated values, and not ensuring that every option was picked an equal number of times, since that looks suspiciously algorithmic. Apparently I'm over 60.

I got the same answer as you, over 60, the first time as I was also very deliberate then went back and did it again like a 3 year old, jabbed anywhere and got a higher random result. Maybe there is something to the study?

Re: We think this cool study we found is flawed. Help us reproduce it

#43

Post by sjmm1989 »

tomsmeding wrote: post #29
jonplackett wrote: post #24

My idea of what random actually looks like has been affected a lot by generating random numbers with a computer. They just don’t actually look that random. I read an anecdote about the iPod shuffle (hey kids - it was a music player with no screen so you could not choose songs directly) - they initially set it to be genuinely random in the way it chose the next song - people didn’t like it. It didn’t _feel_ random to…

I mean, that makes sense. What I want when putting a music player to "shuffle" is not "give me something unpredictable" -- that's fundamentally what randomness means. What I want is "give me something new". Something new is not something random, it's something _different_ from before, if reasonably possible.

Which to a machine may as well be the same thing in either phrasing. You want something different from what you just listened to. To it, anything not 'that song' is different and 'new' potentially if also not 'just listened to' within a certain set amount of songs. Even without that certain set of songs being logged and considered; any picking of a different song from the last is verifiably random.

Think of it all like a deck of cards. Shuffle is apt in that sense. You don't expect to see double aces each time you pick through the shuffled deck of cards, but sometimes you do. Sometimes, you also find double jacks, queens and kings; in a row. Sometimes you don't. That deck could be shuffled by the worlds best trick shufflers. Still gonna get doubles now and then.

True Randomness is not really technically possible. At least, not with our current technologies available; and we have a lot of aces up our sleeves.

The best we can manage for randomness right now, is creating random strings of numbers to serve as the seed for new randomness. At least, if I understand correctly. If I do, then this is why cryptography is so damn important for us in the computational side of things. Network Security requires randomness.

Re: We think this cool study we found is flawed. Help us reproduce it

#44

Post by isitmadeofglass »

> As for our initial idea to make an age-guessing game, we have guessed right 51% of the time. Pretty much what we had expected .

I don’t know what the “guess” was for others. But for me it guessed “are you under 60?” If that’s what it’s doing guessing above or below 60, then I think it’s amazing they, only getting 51% right. I would expect that a strategy of ignoring data completely and always guessing under 60 would be significantly better.

Re: We think this cool study we found is flawed. Help us reproduce it

#45

Post by foerbert »

quanto wrote: post #38

Beautifully crafted web page. For any experimental science, the integrity of experiment (thus reliability of data) is important. For experiments with human subjects, the question is whether the subjects answered the questions in good faith. A sequence like 'HHHHHHHHHH' for the coin toss experiment looks like an answer in bad faith; it is mechanically easy to keep pressing the same button, and a subject is unlikely to…

This is not a sound approach. You're declaring what humans think random is first, and then throwing out any data that doesn't match your declaration. There is no way to learn anything from this.

I also think 'HHHHHHHHHH' is unlikely to be a good faith response, but if the goal is to actually learn anything instead of merely reinforcing my prior beliefs, it doesn't matter.

You need to find a way to design the experiment that discourages bad faith answers or let's you judge them objectively. Alternatively if you have some outside knowledge about the 'shape' of bad faith answers for your kind of experiment, you may be able to use that to properly adjust your data.

But 'nah I don't think so' isn't an acceptable reason to throw out data. It's especially egregious to do so when the data is answers that are, at a bare minimum, technically correct.

Re: We think this cool study we found is flawed. Help us reproduce it

#46

Post by Aethylia »

sjmm1989 wrote: post #39

> As for our initial idea to make an age-guessing game, we have guessed right 51% of the time. Pretty much what we had expected . Yeah... you thought I was 60. Seems from the comments this is a common thing. You might want to check your algorithms. But then again, you do say in the end of the results that you need more 60+ year olds to help make this more accurate. Also, a bone to pick. You claim that people get less…

Did we read the same article? They aren't claiming those things at all. Those are the claims of the original study, which are being disputed by this attempt at reproduction. The writers suspect those claims to be false due to the choice of the original study to not remove likely intentionally non-random data.

I believe the 13% stat you saw is that your score had a higher random score than than 13% of other participants, so not very random.

Re: We think this cool study we found is flawed. Help us reproduce it

#47

Post by CipherThrowaway »

thombles wrote: post #27

I understood my task as convincing another _human_ that it was randomly generated. Since N was low for each of the tasks, I was deliberate about sometimes having repeated values, and not ensuring that every option was picked an equal number of times, since that looks suspiciously algorithmic. Apparently I'm over 60.

Picking evenly seems to consistently produce higher "randomness" scores than picking unevenly or using an RNG. I wonder how this algorithm would rank random sequences vs shuffled linear sequences.

The fundamental issue with a randomness metric for sequences is that an idealized independent generator will under-perform vs a constrained generator that excludes low scoring sequences.

Re: We think this cool study we found is flawed. Help us reproduce it

#48

Post by isitmadeofglass »

tomsmeding wrote: post #29
jonplackett wrote: post #24

My idea of what random actually looks like has been affected a lot by generating random numbers with a computer. They just don’t actually look that random. I read an anecdote about the iPod shuffle (hey kids - it was a music player with no screen so you could not choose songs directly) - they initially set it to be genuinely random in the way it chose the next song - people didn’t like it. It didn’t _feel_ random to…

I mean, that makes sense. What I want when putting a music player to "shuffle" is not "give me something unpredictable" -- that's fundamentally what randomness means. What I want is "give me something new". Something new is not something random, it's something _different_ from before, if reasonably possible.

Not just that though. If you have an iPod filled with 20 albums from your favorite artist and 1 album from 5 others, you wouldn’t be happy even with random excluding previous.

Re: We think this cool study we found is flawed. Help us reproduce it

#49

Post by indecisive_user »

thombles wrote: post #27

I understood my task as convincing another _human_ that it was randomly generated. Since N was low for each of the tasks, I was deliberate about sometimes having repeated values, and not ensuring that every option was picked an equal number of times, since that looks suspiciously algorithmic. Apparently I'm over 60.

As I understood it, their entire point is that younger people are not better at generating random sequences than older people, so guessing someone's age based on their complexity (randomness) score is completely unreliable.

Towards the bottom of the page they said they've only guessed age correctly 51% of the time, which lines up with there being no correlation between age and ability to generate random sequences

Post reply on HN