21–30 of 106 posts
Surprising that no mention is made of rainbow tables or lookup tables. If you hash something that can easily be looked up in a table, it's obviously not anonymous. Passwords are stored as salted hashes for these obvious reasons...
Differential privacy seems like a pretty good approach to this problem. https://machinelearning.apple.com/2017/12/06/learning-with-p...
https://www.privitar.com/listing/k-anonymity-an-introduction
Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it. Let me know if you have any questions.
This is so we can evaluate CDN performance and also see how well ISPs are doing in serving content to the user. So it's essentially asking questions about network performance rather than at a macro level of individual users.
As far as IPs are concerned we don't care much after that, other than maybe the odd "how many unique IP addresses were served today" type queries.
We've talked about doing the secret/salt that is rotated periodically, but to be safe you would definitely need to ensure previous salts are destroyed, and not even let people view them or access them when they are live.
Really good article. One of those things that are beyond obvious to those of us close to this field, but not at all obvious to the general software dev (who also mightn't know the difference between a good cryptographic hash and a good password hash). What would make this article great is general ideas on what is a good way to anonymize data. I'm surprised that info is missing, actually. What would make it world clas…
I was trying to avoid the general ideas on what is a good way to anonymize data, because I don't think there are general rules that apply, and I'm not in a position to give authoritative advice on this. The more I dug in, the more I realized this is probably one of the hardest technical problems that exists right now, and there isn't yet a right answer that works (like use scrypt for passwords).
As for GDPR, I think digging into this in more detail would be a great follow up.
I'm surprised that there was no mention of a salt used in a secure server to generate the hashes and act as an oracle. Adding pepper at the customer site already seemed like a good idea. Of course this is still hard and requires diligence for those who care about their customers and data security.
A good rule of thumb with these things is to assume that if there's any sort of indirect link between some person and that server (even if it involves multiple hops across security boundaries - e.g a web request invoking a backend service querying a database that accesses the hash from a stored procedure), it can potentially be compromised. You never know when another Meltdown happens, and what it'll look like.
Really good article. One of those things that are beyond obvious to those of us close to this field, but not at all obvious to the general software dev (who also mightn't know the difference between a good cryptographic hash and a good password hash). What would make this article great is general ideas on what is a good way to anonymize data. I'm surprised that info is missing, actually. What would make it world clas…
Thanks for the feedback. I was trying to avoid the general ideas on what is a good way to anonymize data, because I don't think there are general rules that apply, and I'm not in a position to give authoritative advice on this. The more I dug in, the more I realized this is probably one of the hardest technical problems that exists right now, and there isn't yet a right answer that works (like use scrypt for password…
If you don’t require deterministic hashes (and deterministic hashes are bad for anonymization anyway) just hash data+randomBytes(16) (obviously, don't save randomBytes(16) anywhere). There you are, nobody can bruteforce your hashes. Even better, just replace your data with H(randomBytes(16)). Or a random UUID.