Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The False Allure of Hashing for Anonymization

gravitational.com

Post reply on HN
Print view View original

71–80 of 106 posts

Re: The False Allure of Hashing for Anonymization

#71

Post by WalterBright »

michaelbuckbee wrote: post #55

In digital security there is the concept of "defense in depth", that no one product, feature, approach or safeguard is going to magically make you protected from attacks. What's required are multiple overlapping layers of protection that collectively work together to create a more protected whole. We're seeing more of this with privacy and user data. The author very correctly points out some issues with hashing and "…

Defense in depth is a lesson other industries have learned - that's why airliners are incredibly safe these days. It's not safe because parts don't fail - they do fail, as the recent engine compressor failure showed. But the airliner is designed to withstand those failures, the pilot is trained to deal with them, and the process is designed to prevent them from happening again.

Notably the Fukushima Nuke plant and Deepwater Horizon disasters did not have defense in depth. One failure each had a zipper effect.

(Of course, defense in depth is a concept from the military, look how medieval castles are constructed for a very visible implementation of it.)

Re: The False Allure of Hashing for Anonymization

#72

Post by r00fus »

sdenton4 wrote: post #10
kevin_nisbet wrote: post #2

Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it. Let me know if you have any questions.

So one of the issues here is using an externally visible ID (or a transformation of such) as an internal ID. Why not create a random int64 at account creation time which is invisibly linked to the public username (eg, email address). So now you've got a proper join key, you can restrict access to the map, and it's easy to delete the map entry when the user unsubscribes. (There can still be good reasons to apply one-w…

Isn't this essentially the uuid() function that many databases support natively (even the black sheep MySQL)?

Re: The False Allure of Hashing for Anonymization

#73

Post by robbiemitchell »

sdenton4 wrote: post #67
merinowool wrote: post #60

Earlier quoted context omitted.

If you throw away the key you still have the data but encrypted. There is no guarantee that in 5 years user data could be easily decrypted.

'Key' here refers to the key in the mapping from external to internal userID. The whole point is that (as mentioned in a sibling comment) choosing an internal user ID uniformly at random is equivalent to a one-time pad; it's guaranteed non-decryptable, unless you invent a time machine...

Isn't there a distinction here, though? While they might result in a similar outcome, deletion is different from de-identification.

Re: The False Allure of Hashing for Anonymization

#74

Post by Area12 »

I am not a crypto expert, but I thought that the idea was to produce a new more or less random salt for EACH password, store the salt with the hashed password, hashing using an expensive algorithm. Yes the hacker steals the salt with the hash, but now has to go to the trouble of brute forcing that ONE password with its UNIQUE (or almost unique) salt. In other words, the hacker can crack it, but the process is so expensive for ONE password that cracking an entire database of passwords is a nightmare. Of course, the hacker just focuses on the most privileged accounts I guess, but the idea is to make the hackers life as unpleasant as possible, and to catch the hacker while they are coming back in. Am I missing the point? I do see that if the hacker wants one password, they can do with effort even with unique salts.

Re: The False Allure of Hashing for Anonymization

#75

Post by unilynx »

merinowool wrote: post #61
unilynx wrote: post #50

Earlier quoted context omitted.

http://www.privacy-regulation.eu/en/r26.htm ... account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments. The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an id…

It doesn't say that information cannot be _reasonably_ reconnected, but that you shouldn't be able to reconnect it at all. I don't know how you have drawn that it shouldn't be NSA-proof from this text if it literally says "in such a manner that the data subject is not or no longer identifiable."

Its in the original link, I may have limited the quote too much:

... To determine whether a natural person is identifiable, account should be taken of all the means reasonably likely to be used ...

Re: The False Allure of Hashing for Anonymization

#76

Post by busterarm »

Area12 wrote: post #74

I am not a crypto expert, but I thought that the idea was to produce a new more or less random salt for EACH password, store the salt with the hashed password, hashing using an expensive algorithm. Yes the hacker steals the salt with the hash, but now has to go to the trouble of brute forcing that ONE password with its UNIQUE (or almost unique) salt. In other words, the hacker can crack it, but the process is so expe…

Engineering happened somewhere along the way. Something was too slow, or couldn't be finished that hour/day/sprint and compromises were made. Then those compromises were shipped.

Re: The False Allure of Hashing for Anonymization

#77

Post by js8 »

polskibus wrote: post #51
kevin_nisbet wrote: post #2

Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it. Let me know if you have any questions.

Please elaborate on whether hashing is a good pseudonimization strategy in context of GDPR guidelines.

I thought the blog post made it pretty clear that it isn't.

Re: The False Allure of Hashing for Anonymization

#78

Post by tribby »

Area12 wrote: post #74

I am not a crypto expert, but I thought that the idea was to produce a new more or less random salt for EACH password, store the salt with the hashed password, hashing using an expensive algorithm. Yes the hacker steals the salt with the hash, but now has to go to the trouble of brute forcing that ONE password with its UNIQUE (or almost unique) salt. In other words, the hacker can crack it, but the process is so expe…

this had me confused at first too, but I think the author's point is that if the initial data comes in a predictable form (e.g. an IP address that is x.x.x.x where x is 0-254, email addresses that are mostly short and ends in "gmail.com," etc), salts don't really save the hash from being brute forced, they just save the hash from being brute forced with a rainbow table. the author's post isn't about passwords, per se, but how the kind of datapoints we often hash for the sake of anonymization are really only pseudo-anonymous, or at least a lot weaker than people might expect for a string of x length.

that said, bcrypt, PBKDF2, and other time/work-based hashing solutions are still very good options for this.

Re: The False Allure of Hashing for Anonymization

#79

Post by rossy »

Area12 wrote: post #74

I am not a crypto expert, but I thought that the idea was to produce a new more or less random salt for EACH password, store the salt with the hashed password, hashing using an expensive algorithm. Yes the hacker steals the salt with the hash, but now has to go to the trouble of brute forcing that ONE password with its UNIQUE (or almost unique) salt. In other words, the hacker can crack it, but the process is so expe…

This is addressed in the article under "What if we make the hash slow?" You might be tempted to use salted hashes, but apparently this only works for protecting data that is supposed to be unpredictable, like passwords, and it's not too much of a setback if the data is easily predictable, like if it's a username or email (or presumably, an IP address or a timestamp.)

> Even with something like bcrypt at reasonable work factors, a database of 100,000 anonymous users would take less than a day on a single cpu core to test every bcrypted entry for the string “knisbet” and unmask my secret data.

Re: The False Allure of Hashing for Anonymization

#80

Post by your-nanny »

rossy wrote: post #79
Area12 wrote: post #74

I am not a crypto expert, but I thought that the idea was to produce a new more or less random salt for EACH password, store the salt with the hashed password, hashing using an expensive algorithm. Yes the hacker steals the salt with the hash, but now has to go to the trouble of brute forcing that ONE password with its UNIQUE (or almost unique) salt. In other words, the hacker can crack it, but the process is so expe…

This is addressed in the article under "What if we make the hash slow?" You might be tempted to use salted hashes, but apparently this only works for protecting data that is supposed to be unpredictable, like passwords, and it's not too much of a setback if the data is easily predictable, like if it's a username or email (or presumably, an IP address or a timestamp.) > Even with something like bcrypt at reasonable wo…

Would someone explain to me why this is true (or not).
Post reply on HN