Live data from Hacker News

The False Allure of Hashing for Anonymization

gravitational.com

41–50 of 106 posts

Re: The False Allure of Hashing for Anonymization

#41

Any thoughts on using a UUID instead of a username hash?

A hash is (theoretically) an anonymous function.

How do you anonymously map your UUID to the anonymised signifier? Such that you cannot back it out yourself?

The properties that ensure this make the UUID useless.

Re: The False Allure of Hashing for Anonymization

#42

Where I work we've been debating about this a lot. I work with log data from CDNs, so user IP addresses get ingested. We use that information and correlate it with geoip services to determine stuff like the ISP being used. This is so we can evaluate CDN performance and also see how well ISPs are doing in serving content to the user. So it's essentially asking questions about network performance rather than at a macro…

How are you going to ask user for consent to process their IP this way?

Consent is not the only basis for legally processing data. There is not enough information in the above comment to determine which basis this company has determined their processing falls under.

Re: The False Allure of Hashing for Anonymization

#43

Earlier quoted context omitted.

So one of the issues here is using an externally visible ID (or a transformation of such) as an internal ID. Why not create a random int64 at account creation time which is invisibly linked to the public username (eg, email address). So now you've got a proper join key, you can restrict access to the map, and it's easy to delete the map entry when the user unsubscribes. (There can still be good reasons to apply one-w…

Then user emails you to ask what personal data of his you have on the server. Now you don't have a connection so you can't find it, but you have it. GDPR non compliance.

If you can't connect it to the user in any way, it's no longer personal information. Expect the data protection agency to compliment you.

Re: The False Allure of Hashing for Anonymization

#44
post #31

Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it. Let me know if you have any questions.

Technically a substitution lookup table (like is proposed at the end) is analogous to one-time-pad 'encryption'. In this case the 'pad' is only used within a single (extended time domain) context and is presumably selectively exposed and used only in a contexts where intercepting the context already reveals that data anyway. Additional security could be added by making a session-unique identifier (not based on user,…

Yeah, it sounds a lot like deterministic encryption. Which is OK if your input data is essentially uniform random.

But if it's not, and your adversary knows the distribution of the input data, then the protection level is pretty close to zero.

Re: The False Allure of Hashing for Anonymization

#45
post #43

Earlier quoted context omitted.

Then user emails you to ask what personal data of his you have on the server. Now you don't have a connection so you can't find it, but you have it. GDPR non compliance.

If you can't connect it to the user in any way, it's no longer personal information. Expect the data protection agency to compliment you.

> If you can't connect it to the user in any way, it's no longer personal information

Just because you can't connect it doesn't mean nobody else can.

Re: The False Allure of Hashing for Anonymization

#46

Where I work we've been debating about this a lot. I work with log data from CDNs, so user IP addresses get ingested. We use that information and correlate it with geoip services to determine stuff like the ISP being used. This is so we can evaluate CDN performance and also see how well ISPs are doing in serving content to the user. So it's essentially asking questions about network performance rather than at a macro…

Wouldn't storing the first three octets of an IP address be enough for this kind of analysis? Or use the whois database and reduce the data to the first IP address of the network ?

Re: The False Allure of Hashing for Anonymization

#47
post #46

Where I work we've been debating about this a lot. I work with log data from CDNs, so user IP addresses get ingested. We use that information and correlate it with geoip services to determine stuff like the ISP being used. This is so we can evaluate CDN performance and also see how well ISPs are doing in serving content to the user. So it's essentially asking questions about network performance rather than at a macro…

Wouldn't storing the first three octets of an IP address be enough for this kind of analysis? Or use the whois database and reduce the data to the first IP address of the network ?

I personally think just storing the autonomous system the IP originates from and never writing the IPs to disk at all would be advisable if the goal is purely which ISPs are delivering how many bytes to end users. Another benefit is the AS to IP mapping database is small enough to fit in memory without issue.

Re: The False Allure of Hashing for Anonymization

#48
post #43

Earlier quoted context omitted.

If you can't connect it to the user in any way, it's no longer personal information. Expect the data protection agency to compliment you.

> If you can't connect it to the user in any way, it's no longer personal information Just because you can't connect it doesn't mean nobody else can.

For all practical purposes, a secure, one-way cryptographic hash is irreversible.

Re: The False Allure of Hashing for Anonymization

#49

This is a question that I've thought of recently, as I am going to be working with a set of data that is the kind of data that may have damaging personal repercussions if identified with you but is good for society as a whole to be tracking, but that tracking doesn't have to be personally identifiable. Something like, it could be bad for me if it was revealed to my insurance company that I drove more than 5000 miles…

Differential privacy and other formalized systems are a good choice, but if you never need to give the data back or present it as-such to the customer/inputer, you can get heuristic Pretty Good Anonymization if you understand the structure of your problem and how you're going to use it.

For example taking your example of motor vehicle trips off the top of my head, in order the things that can ID you are:

  Driver's License
  Name
  Vehicle License Plate
  Time, Location of trip
  Trip Distance
  Location of driver residence
  Location of driver workplace
If you had a database of these things, you could apply some of the strategies in the article, and a few others to ensure no collisions.

  Driver's License: Ditch it, 
  hash it with private key or have a lookup table
  somewhere. I'd favor ditching it.
  Name: Same as DL number
  Vehicle License Plate: Same as DL number
For the above 3, you really may only need a few variables that are less constrained: gender, approximate age, type of vehicle so you could just compute out to those and store only that result.

  Time, Location of trip: Fudge these +- random time, or +- random distance from start/finish. 
  Careful not to have it be a dumb random circle, Strava does this, given enough public rides I'm sure people 
  could figure out where I live. (maybe do this as function of population density?)
  Trip Distance: Fudge +- random distance
  Location of driver residence: Fudge to begin with, probably ditch if possible
  Location of driver workplace: Ditto
The point is think about what you need from the dataset and deliberately mess it up so that you'd have to have the original to piece it together. Often, you don't need the exact input data, but something within a random delta of it, so just keep the stuff within a random delta.

Re: The False Allure of Hashing for Anonymization

#50
post #43

Earlier quoted context omitted.

If you can't connect it to the user in any way, it's no longer personal information. Expect the data protection agency to compliment you.

> If you can't connect it to the user in any way, it's no longer personal information Just because you can't connect it doesn't mean nobody else can.

http://www.privacy-regulation.eu/en/r26.htm

... account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments.

The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an identified or identifiable natural person or to personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable. ...

It's sufficient if one can't reasonably reconnect the data back to the user. It doesn't need to be NSA-proof.

Post reply on HN