Poll: HN readers, where's your residence?

Only countries that qualify for the eurovision contest count ;-)

Even if the poll is posted for many hours on HN's front page, we shouldn't rely on it for an inference of where HN participants live, in the level of detail shown in the poll. Just three days ago HN had a good thread about gaps in statistics knowledge even among persons with a university education,

http://news.ycombinator.com/item?id=3285742

and polls like this on HN are a good illustration (in the comments) about what people think they know about statistics that just isn't so. I've even prepared a FAQ on this issue, especially for HN polls, and here it is again for anyone who didn't see it last time.

VOLUNTARY RESPONSE POLLS

As I commented previously when we had a poll on the ages of HNers, the data can't be relied on to make an inference about the age distribution of HN participants, and similarly here we can't make a valid inference about the geographical distribution of HN participants. That's because the data are not from a random sample of the relevant population. One professor of statistics, who is a co-author of a highly regarded AP statistics textbook, has tried to popularize the phrase that "voluntary response data are worthless" to go along with the phrase "correlation does not imply causation." Other statistics teachers are gradually picking up this phrase.

-----Original Message----- From: Paul Velleman [SMTPfv2@cornell.edu] Sent: Wednesday, January 14, 1998 5:10 PM To: apstat-l@etc.bc.ca; Kim Robinson Cc: mmbalach@mtu.edu Subject: Re: qualtiative study

Sorry Kim, but it just aint so. Voluntary response data are worthless. One excellent example is the books by Shere Hite. She collected many responses from biased lists with voluntary response and drew conclusions that are roundly contradicted by all responsible studies. She claimed to be doing only qualitative work, but what she got was just plain garbage. Another famous example is the Literary Digest "poll". All you learn from voluntary response is what is said by those who choose to respond. Unless the respondents are a substantially large fraction of the population, they are very likely to be a biased -- possibly a very biased -- subset. Anecdotes tell you nothing at all about the state of the world. They can't be "used only as a description" because they describe nothing but themselves.

http://mathforum.org/kb/thread.jspa?threadID=194473&tsta...

For more on the distinction between statistics and mathematics, see

http://statland.org/MAAFIXED.PDF

and

http://escholarship.org/uc/item/6hb3k0nz

I think Professor Velleman promotes "Voluntary response data are worthless" as a slogan for the same reason an earlier generation of statisticians taught their students the slogan "correlation does not imply causation." That's because common human cognitive errors run strongly in one direction on each issue, so the slogan has take the cognitive error head-on. Of course, a distinct pattern in voluntary responses tells us SOMETHING (maybe about what kind of people come forward to respond), just as a correlation tells us SOMETHING (maybe about a lurking variable correlated with both things we observe), but it doesn't tell us enough to warrant a firm conclusion about facts of the world. The Literary Digest poll

http://historymatters.gmu.edu/d/5168/

http://www.math.uah.edu/stat/data/LiteraryDigest.pdf

is a spectacular historical example of a voluntary response poll with a HUGE sample size and high response rate that didn't give a correct picture of reality at all.

When I have brought up this issue before, some other HNers have replied that there are some statistical tools for correcting for response-bias effects, IF one can obtain a simple random sample of the population of interest and evaluate what kinds of people respond. But we can't do that here on HN.

Another reply I frequently see when I bring up this issue is that the public relies on voluntary response data all the time to make conclusions about reality. To that I refer careful readers to what Professor Velleman is quoted as saying above (the general public often believes statements that are baloney) and to what Google's director of research, Peter Norvig, says about research conducted with better data,

http://norvig.com/experiment-design.html

that even good data (and Norvig would not generally characterize voluntary response data as good data) can lead to wrong conclusions if there isn't careful thinking behind a study design. Again, human beings have strong predilections to believe certain kinds of wrong data and wrong conclusions. We are not neutral evaluators of data and conclusions, but have predispositions (cognitive illusions) that lead to making mistakes without careful training and thought.

Another frequently seen reply is that sometimes a "convenience sample" (this is a common term among statisticians for a sample that can't be counted on to be a random sample) of a population offers just that, convenience, and should not be rejected on that basis alone. But the most thoughtful version of that frequent reply I previously saw here on HN did correctly point out that if we know from the get-go that the sample was not done statistically correctly, then even if we are confident (enough) that HN participants are young (in the case of polls on age) and American (in the case of polls on location), we wouldn't want to extrapolate from that to conclude that the users of any technology site are young, or that start-up founders in general are American.

On my part, I wildly guess that most HNers are residents in the English-speaking developed world, because the concerns of Hacker News are concerns of people living in developed countries and the language of discussion here is English. Moreover, Hacker News was founded by people in the United States. So it wouldn't be any surprise if a plurality of participants on Hacker News live in the United States, and a majority live in countries where English is a routine language learned in secondary education. But we can't conclude anything from this poll, whether it is on the HN front page for one hour, one day, or one week, about what the overall distribution is on HN of participants residing in "United States - NYC and SF area" as contrasted with "Australia/Oceania," or any other bin the data are sorted into, because the data are voluntary response data and are useless for modeling reality.

I think there should be a log softener applied to submission karma.

In Singapore as well, originally from Malaysia.

I would imagine that the Washington DC Metro area (DC, Northern Virginia, Maryland) has a reasonably sized proportion of that "(Rest of the) United States" total. The startup scene, and tech scene in general (largely thanks to defense contractors), are quite active here.