This is a question that I've thought of recently, as I am going to be working with a set of data that is the kind of data that may have damaging personal repercussions if identified with you but is good for society as a whole to be tracking, but that tracking doesn't have to be personally identifiable. Something like, it could be bad for me if it was revealed to my insurance company that I drove more than 5000 miles a year on a motorcycle, but beneficial for society as a whole to understand accident rates for high mileage motorcycle drivers. Do you have any thoughts/resources on how one could go about creating a privacy environment where users could input how many miles they drove, and where we have reporting that analyzes that information they put in? My first thought had been hashing primary keys, but as you point out in your article, that obviously isn't the best answer.
The False Allure of Hashing for Anonymization
11–20 of 106 posts
Re: The False Allure of Hashing for Anonymization
#12Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it. Let me know if you have any questions.
I think this is the best advice:
“And most importantly, we only collect enough data to fulfill our stated purpose. The fewer data points that we collect, the less opportunity that someone can correlate the data.”
The smaller the domain, the less anonymization works to conceal which user id did an action. However, if you think about it, identity is far more than user id. That is why real anonymity means not storing any unnecessary information from other domains. We have a technique where we use iframes to display a person’s name, friends etc. back to them based on user ids, but the enclosing domain knows only the user ids and their connections.
Re: The False Allure of Hashing for Anonymization
#13If you don’t require deterministic hashes (and deterministic hashes are bad for anonymization anyway) just hash data+randomBytes(16) (obviously, don't save randomBytes(16) anywhere). There you are, nobody can bruteforce your hashes. Even better, just replace your data with H(randomBytes(16)). Or a random UUID.
Umm what good is the string of random bytes if you don’t store it anywhere? The point of a one-way function is that its output is verifiable given the input.
Re: The False Allure of Hashing for Anonymization
#14Surprising that no mention is made of rainbow tables or lookup tables. If you hash something that can easily be looked up in a table, it's obviously not anonymous.
Passwords are stored as salted hashes for these obvious reasons...
Re: The False Allure of Hashing for Anonymization
#15The trouble is when we're holding on to the original data because we want the option to process it in new ways later on. The fundamental problem is that data correlates facts. Thus - as the article rightly points out - if you know some of the facts you can reconstruct identities.
I find the distinction between information and exformation revealing: Information is the bits we gleaned from the data, exformation is the bits we discarded while reducing the data. The efficacy of an information processing system is in how much it discards while extracting the information we need. The expensive operation is not the recording but the forgetting.
If you want to protect data from being stolen, distill it as soon as possible into the information you need. And destroy the rest. It comes down to the value of being able to re-run the analysis versus the effort to guard the data.
Re: The False Allure of Hashing for Anonymization
#16Really good article. One of those things that are beyond obvious to those of us close to this field, but not at all obvious to the general software dev (who also mightn't know the difference between a good cryptographic hash and a good password hash).
What would make this article great is general ideas on what is a good way to anonymize data. I'm surprised that info is missing, actually.
What would make it world class great is discussion about GDPR ramifications, keeping in mind that one need not necessarily be perfect for GDPR, even if you're FB/Google.
Re: The False Allure of Hashing for Anonymization
#17The trouble is when we're holding on to the original data because we want the option to process it in new ways later on. The fundamental problem is that data correlates facts. Thus - as the article rightly points out - if you know some of the facts you can reconstruct identities. I find the distinction between information and exformation revealing: Information is the bits we gleaned from the data, exformation is the…
As someone who's had to work through the implications of GDPR lately, I think the future of user data is that you can't keep the option to "process it in new ways" later. Permissions are becoming opt-in instead of opt-out.
Re: The False Allure of Hashing for Anonymization
#18Another recent writeup https://freedom-to-tinker.com/2018/04/09/four-cents-to-deano...
Re: The False Allure of Hashing for Anonymization
#19Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it. Let me know if you have any questions.
So one of the issues here is using an externally visible ID (or a transformation of such) as an internal ID. Why not create a random int64 at account creation time which is invisibly linked to the public username (eg, email address). So now you've got a proper join key, you can restrict access to the map, and it's easy to delete the map entry when the user unsubscribes. (There can still be good reasons to apply one-w…
This is along the lines of where I was going with the alternative approach, I just simplified it for brevity. :)
In the case of teleport, I think this is a bit more difficult to achieve, because we don't necessarily have our own account database, our common commercial use case is integrated to an identity provider through SAML/OIDC, which I'm not sure would consistently offer a random id per account to use.
While there are many way's we could generate and store the username random id mappings, this adds a certain amount of complexity to get right on a distributed system.
If building a system from scratch with end to end control, I do prefer the random identifier approach.
Re: The False Allure of Hashing for Anonymization
#20Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it. Let me know if you have any questions.
Have you looked into VRFs (verifiable random functions)? This is essentially a public key crypto version of HMACs. You have a piece of data and a blinding factor, both are committed into a single point on an elliptic curve, and blinding factor is not exposed as-is (as in case with salt for hashed content), but instead committed into a public key (a point). The resulting point preserves the algebraic structure, which…
No I haven't, this sounds very intriguing though, I'll have to do some research.