Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

87% of the U.S. Population are uniquely identified by {DOB, gender, zip}

latanyasweeney.org

Post reply on HN
Print view View original

41–50 of 102 posts

Re: 87% of the U.S. Population are uniquely identified by {DOB, gender, zip}

#41

Post by jerf »

mikey_p wrote: post #2

While I find this fascinating, I'm sure this is old news to people in the business, like the people that run the grocery/chain store savings programs. What I'd be really interested to see is why this works, and what it tells us about distribution of population by zip code. I'd imagine the places where this doesn't work as well, are the most densely populated zip codes, where the likely hood of duplicates on the given…

Simple information theory; in a world of 6 billion people, 33 bits serves to uniquely identify anyone. 33 bits is a really low bar. It works because it can't hardly help but to work, sort of like the birthday paradox. In the US with approx 312 million (wikipedia) that's approximately 28.2 bits.

It doesn't tell us much about zip code distribution because zip codes are chosen to have approx the same number of people in each. As it turns out, that's exactly how you'd go about maximizing the amount of information the zip code carries... which is unsurprising since that's the entire purpose of a zip code. Gender is almost exactly one bit, and date of birth is 15ish bits with some bad uniformity assumptions, zip is another 15ish with bad uniformity assumptions[1], that's 31-ish total, subtract off 3-ish for the bad assumptions and you get 28, which covers 2^28 = 268,435,456 people, which is pretty close to the number cited (.87 times 312,000,000 = 271,440,000). I'll cop to tuning the fudge factor of three bits to nicely match the number given, but the bit count itself just comes from the space of possibilities.

Lots more info on the topic can be found here: http://33bits.org/about/

[1]: http://www.carrierroutes.com/ZIPCodes.html 43,000 ZIP codes => log2 43000 is approx 15.4 bits assuming perfect uniformity.

Re: 87% of the U.S. Population are uniquely identified by {DOB, gender, zip}

#42

Post by esrauch »

ComputerGuru wrote: post #34
pyre wrote: post #31

If we want more anonymized datasets, maybe we should be asking for approx. age or year of birth instead. I doubt that most things that ask for DOB need anything more than granular than that.

I wish. Unfortunately, most sites need this for legal reasons, to determine whether or not the user is above or below a certain age. One could theoretically ask "how old are you" but then you don't know when to lift limitations. Say a user joins six months before their 18th birthday. Are you going to block 18+ content for a whole year from then?

Your point really does not apply anonymized datasets which are released and used to do statistical analysis on a population.

It may be relevant if you are trying to examine differences immediately before and after your 18th or 21st birthdays, but that would be pretty rare.

Re: 87% of the U.S. Population are uniquely identified by {DOB, gender, zip}

#43

Post by phillmv »

brass_cannon wrote: post #35
Zimahl wrote: post #18

Earlier quoted context omitted.

October 8th is an odd date. Since it's my birthday as well (long before 1990 though) I have taken note that I run across very few with that birthday, let alone early October birthdays. Is it statistically a low-birth month? I'd love to know. On a side note, Happy 21st Birthday in just over a month!

As another October 8th birthdate with the opposite impression of how many October kids there are, I had to look this up. 8.62% of all live births in the US during 2006 occurred during October [1]. No single month deviated significantly from it's expected percentage. [1] http://data.un.org/Data.aspx?d=POP&f=tableCode:55

I guess people be fuckin' all of the damn time.

Re: 87% of the U.S. Population are uniquely identified by {DOB, gender, zip}

#44

Post by russell »

jdp23 wrote: post #5

Here's some backstory, from the 1990s when the Massachusetts' Group Insurance Commission released "anonymized" health data: ''At the time GIC released the data, William Weld, then Governor of Massachusetts, assured the public that GIC had protected patient privacy by deleting identifiers. In response, then-graduate student Sweeney started hunting for the Governor’s hospital records in the GIC data. She knew that Gove…

I had to deal with the converse of the problem when I was working for a company that ran the vehicle smog checking for the state of Illinois. A driver's license id was composed of birthdate, sex, and some location information (I dont remember exactly what) for the reasons mentioned by jdp23 it worked quite well, except for identical twins, who had the same driver's license id. (Registrations ofter were processed by hand at drug stores. The driver's license was used to disambiguate ambiguous renewals.)

The state of Texas complicated things by allowing a family to have the same license plate for all their cars. One family had seven vehicles, all with the same license plate number. Florida would issue the same plate to complete strangers. One man arrested with great drama, because a bank robber (or some such undesirable) had the same (legal) plates.

My GF has such a common name that her doctor has to check which one of three she is. The last time she tried to get her driver's license renewed online, she was told she had to go to the DMV in person. She asked why and was told she was a "Code 5". She reported to the DMV and told the woman that she was a Code 5. There dropped jaws and a hurried conference with a supervisor who called Sacramento and had a lengthy conversation with comments like, "Yes I'm sure. She's right here." Finally she got impatient and asked, "What's Code 5?"

The clerk said, "You're dead and it was verified by medical personnel." She did eventually get her license renewed. Officially I live with a zombie. It isnt as bad as the movies portray.

Re: 87% of the U.S. Population are uniquely identified by {DOB, gender, zip}

#46

Post by Semiapies »

ComputerGuru wrote: post #34
pyre wrote: post #31

If we want more anonymized datasets, maybe we should be asking for approx. age or year of birth instead. I doubt that most things that ask for DOB need anything more than granular than that.

I wish. Unfortunately, most sites need this for legal reasons, to determine whether or not the user is above or below a certain age. One could theoretically ask "how old are you" but then you don't know when to lift limitations. Say a user joins six months before their 18th birthday. Are you going to block 18+ content for a whole year from then?

How many sites filter content based on age and really do so selectively based on a birthdate? Most age verifications, even ones that request a date, seem to be just asking the question, "Are you of legal age?"

(I prefer websites just ask that binary question, like this brewery - http://www.newbelgium.com - instead of making me waste time with a date input.)

Re: 87% of the U.S. Population are uniquely identified by {DOB, gender, zip}

#47

Post by robrenaud »

jerf wrote: post #41
mikey_p wrote: post #2

While I find this fascinating, I'm sure this is old news to people in the business, like the people that run the grocery/chain store savings programs. What I'd be really interested to see is why this works, and what it tells us about distribution of population by zip code. I'd imagine the places where this doesn't work as well, are the most densely populated zip codes, where the likely hood of duplicates on the given…

Simple information theory; in a world of 6 billion people, 33 bits serves to uniquely identify anyone. 33 bits is a really low bar. It works because it can't hardly help but to work, sort of like the birthday paradox. In the US with approx 312 million (wikipedia) that's approximately 28.2 bits. It doesn't tell us much about zip code distribution because zip codes are chosen to have approx the same number of people in…

1-.5^3 = .875, eerily close to .87.

In general, if you model this as an N balls in M bins problem, then even when N == M, you'd expect a fair amount of anonyonomity preserving collisions. Maybe 1/2 of people would collide. As we then double the number of bins, we'd roughly expect the number of collisions to be cut in half.

If you imagine putting 100 balls (people) into 800 = 100 * 2^3 bins (number of different birthday-zip-gender encodings) at random, about 1/8 of the bins will have more than one ball (person) [okay, this estimate is somewhat off by a smallish constant factor, it's only true that the 100th ball tossed will collide with probability 99/800 ~= 1/8 if there were no existing collisions, and the earlier balls thrown have less to collide with], and not be uniquely identifiable.

Re: 87% of the U.S. Population are uniquely identified by {DOB, gender, zip}

#49

Post by blahedo »

russell wrote: post #44
jdp23 wrote: post #5

Here's some backstory, from the 1990s when the Massachusetts' Group Insurance Commission released "anonymized" health data: ''At the time GIC released the data, William Weld, then Governor of Massachusetts, assured the public that GIC had protected patient privacy by deleting identifiers. In response, then-graduate student Sweeney started hunting for the Governor’s hospital records in the GIC data. She knew that Gove…

I had to deal with the converse of the problem when I was working for a company that ran the vehicle smog checking for the state of Illinois. A driver's license id was composed of birthdate, sex, and some location information (I dont remember exactly what) for the reasons mentioned by jdp23 it worked quite well, except for identical twins, who had the same driver's license id. (Registrations ofter were processed by h…

> ...for the state of Illinois. A driver's license id was composed of birthdate, sex, and some location information....

It's worse than that: in Illinois the first four characters of your DL ID are your last name's soundex code.

Re: 87% of the U.S. Population are uniquely identified by {DOB, gender, zip}

#50

Post by hvs »

blahedo wrote: post #49
russell wrote: post #44

Earlier quoted context omitted.

I had to deal with the converse of the problem when I was working for a company that ran the vehicle smog checking for the state of Illinois. A driver's license id was composed of birthdate, sex, and some location information (I dont remember exactly what) for the reasons mentioned by jdp23 it worked quite well, except for identical twins, who had the same driver's license id. (Registrations ofter were processed by h…

> ...for the state of Illinois. A driver's license id was composed of birthdate, sex, and some location information.... It's worse than that: in Illinois the first four characters of your DL ID are your last name's soundex code.

That used to be true for MN as well, but in the time that I moved from MN to IL back to MN (about 13 years), MN dropped that.
Post reply on HN