Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The False Allure of Hashing for Anonymization

gravitational.com

Post reply on HN
Print view View original

31–40 of 106 posts

Re: The False Allure of Hashing for Anonymization

#31

Post by mjevans »

kevin_nisbet wrote: post #2

Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it. Let me know if you have any questions.

Technically a substitution lookup table (like is proposed at the end) is analogous to one-time-pad 'encryption'. In this case the 'pad' is only used within a single (extended time domain) context and is presumably selectively exposed and used only in a contexts where intercepting the context already reveals that data anyway.

Additional security could be added by making a session-unique identifier (not based on user, chronological, or external context data) and only having the master lookup table for user to sessions in an elevated security environment.

Re: The False Allure of Hashing for Anonymization

#32

Post by zAy0LfpBZLC8mAC »

I think another problem is that we even call any of that "anonymization". If you replace "foobar" with "1", you haven't anonymized anything. At best, you have pseudomized your data. Whether you use hashing or a secret mapping function, as long as identity within your dataset is preserved, what you are generating are pseudonyms.

Re: The False Allure of Hashing for Anonymization

#33

Post by mjevans »

madrox wrote: post #17
lolc wrote: post #15

The trouble is when we're holding on to the original data because we want the option to process it in new ways later on. The fundamental problem is that data correlates facts. Thus - as the article rightly points out - if you know some of the facts you can reconstruct identities. I find the distinction between information and exformation revealing: Information is the bits we gleaned from the data, exformation is the…

As someone who's had to work through the implications of GDPR lately, I think the future of user data is that you can't keep the option to "process it in new ways" later. Permissions are becoming opt-in instead of opt-out.

You probably can, but you need to be upfront about what you're collecting and the context that is being stored with it.

It MAY be more ethically permissible to degrade the context and preserve only the most valuable and least personally identifying data. (Such as saving only the actual search query and a local timestamp, but filtering out anything related to a recognized name that isn't famous)

Re: The False Allure of Hashing for Anonymization

#34

Post by zzzcpan »

> The way we’ve chosen to anonymize the data is by generating HMAC

You can also truncate the hash after the HMAC to mix the data of different users. It still would be useful for aggregate analytics, abuse protection, rate limiting, etc, but if each user shares an identifier with many others it would be harder to unmask them and make correlations.

Re: The False Allure of Hashing for Anonymization

#35

Post by stephengillie »

jacquesm wrote: post #22

The idea that data is a corporate asset has to die. Data is a corporate liability.

Like a million cogs in a warehouse, it costs money to store customer data. Storage costs can quickly overwhelm the value of the stored product. These costs can be lower for improper storage - not wrapping cogs can lead them to rust, and not encrypting data can lead it to being stolen. Financial documents should reflect the risks of storing data, to communicate this liability to shareholders and others.

"We are storing PII (Personally Identifiable Information) on 100 million Americans. A data leak could lead to significant material damages, from settlements to an impaired public reputation."

Re: The False Allure of Hashing for Anonymization

#36

Post by PeterisP »

SHA256 pretty much ensures that you have a unique hash for every value - and that's a feature you don't want for anonymization. So why not simply take the first few bytes of a SHA256, a small enough set to ensure that collisions not only might happen but will happen? I mean, that's a required feature to ensure anonymization, not just pseudonymization - if you can select a whole trail of events for ID #123 and be sure that these represent all the events for some (unknown) real user, then that by itself means that those events aren't anonymous, they're pseudonymous.

You can tweak the hash length so that whatever statistics you run out of the hashed data are meaningful (though not exact) despite the collisions, but that running a dictionary attack of plausible usernames returns an overwhelming amount of false positives.

Re: The False Allure of Hashing for Anonymization

#37

Post by merinowool »

sdenton4 wrote: post #10
kevin_nisbet wrote: post #2

Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it. Let me know if you have any questions.

So one of the issues here is using an externally visible ID (or a transformation of such) as an internal ID. Why not create a random int64 at account creation time which is invisibly linked to the public username (eg, email address). So now you've got a proper join key, you can restrict access to the map, and it's easy to delete the map entry when the user unsubscribes. (There can still be good reasons to apply one-w…

Then user emails you to ask what personal data of his you have on the server. Now you don't have a connection so you can't find it, but you have it. GDPR non compliance.

Re: The False Allure of Hashing for Anonymization

#38

Post by merinowool »

djhworld wrote: post #25

Where I work we've been debating about this a lot. I work with log data from CDNs, so user IP addresses get ingested. We use that information and correlate it with geoip services to determine stuff like the ISP being used. This is so we can evaluate CDN performance and also see how well ISPs are doing in serving content to the user. So it's essentially asking questions about network performance rather than at a macro…

How are you going to ask user for consent to process their IP this way?

Re: The False Allure of Hashing for Anonymization

#40

Post by dredmorbius »

andonisus wrote: post #39
jacquesm wrote: post #22

The idea that data is a corporate asset has to die. Data is a corporate liability.

Why?

https://www.irmi.com/term/insurance-definitions/electronic-d...

https://boingboing.net/2015/09/11/data-is-a-liability-not-an...

https://www.richie.fi/blog/data-is-a-liability.html

Post reply on HN