The False Allure of Hashing for Anonymization • Hacker News Board

Re: The False Allure of Hashing for Anonymization

#61

Post by merinowool » Mon, Apr 30, 2018, 10:32 PM UTC

unilynx wrote: post #50

JumpCrisscross wrote: post #45

Earlier quoted context omitted.

> If you can't connect it to the user in any way, it's no longer personal information Just because you can't connect it doesn't mean nobody else can.

http://www.privacy-regulation.eu/en/r26.htm ... account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments. The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an id…

It doesn't say that information cannot be _reasonably_ reconnected, but that you shouldn't be able to reconnect it at all.

I don't know how you have drawn that it shouldn't be NSA-proof from this text if it literally says "in such a manner that the data subject is not or no longer identifiable."

Re: The False Allure of Hashing for Anonymization

#62

Post by aidos » Mon, Apr 30, 2018, 10:34 PM UTC

I saw a case a few years ago where the management of a company I knew were worried that the sales team were covering their mistakes and lying about it to blame the (external) dev team's code. They asked me to take a look into it one morning.

At first glance there didn't seem to be a lot to go on. There was no auditing in the application itself so I focused on the nginx logs. It's amazing how clear of a picture you can create from ip addresses, user agent strings and accessed urls.

Within an hour I could say with a high degree of certainty that the story was something like:

    Sales rep makes mistake with record on Friday afternoon
    Monday morning - at home, late for work
    Receives call from another rep re mistake
    Logs in via mobile device to see the issue
    Logs in via desktop to fix broken record
    Arrives at work 1.5 hours later
    Claims dev team had broken the record for the weekend

There's a lot of information lurking in log files (let alone insecure dbs), and that's just the tip of the iceberg of what's stored these days. I dread to think how much personal information is stored in some of the bigger CRM apps these days.

Quite frankly I'm glad there's a push to start thinking about this stuff from the outset at the moment.

Re: The False Allure of Hashing for Anonymization

#63

Post by jwilliams » Mon, Apr 30, 2018, 11:01 PM UTC

lolc wrote: post #21

JTbane wrote: post #14

Surprising that no mention is made of rainbow tables or lookup tables. If you hash something that can easily be looked up in a table, it's obviously not anonymous. Passwords are stored as salted hashes for these obvious reasons...

The article explains very well how salted hashes don't help against username lookups.

In the case of salts, the article admits "Don’t get me wrong, this does make it significantly harder to attack a leaked database to unmask every user..."

So salts definitely do help. And if you chose your salt well (e.g. global fixed/rotating plus local/temporal) you significantly increase your protection compared to not using a salt at all.

Re: The False Allure of Hashing for Anonymization

#64

Post by ecnahc515 » Mon, Apr 30, 2018, 11:03 PM UTC

russnewcomer wrote: post #53

entee wrote: post #49

Earlier quoted context omitted.

Differential privacy and other formalized systems are a good choice, but if you never need to give the data back or present it as-such to the customer/inputer, you can get heuristic Pretty Good Anonymization if you understand the structure of your problem and how you're going to use it. For example taking your example of motor vehicle trips off the top of my head, in order the things that can ID you are: Driver's Lic…

But if I do need the original data back, say, the driver needs to produce an expense report with the hours, what would you do in that case? I have thoughts, but trying to bounce off of someone else.

Store the delta's + the identifying information somewhere else as a lookup table and use a random ID to join to it. Keep the PII database secured, offline, or whatever makes you feel best, and then if anyone needs direct correlation back to the end user, it is done through a different process that ensures higher access controls/auditing, etc.

Re: The False Allure of Hashing for Anonymization

#65

Post by JetSpiegel » Mon, Apr 30, 2018, 11:10 PM UTC

merinowool wrote: post #60

sdenton4 wrote: post #56

Earlier quoted context omitted.

It's your mapping, so you can easily gather up everything with the given marker and hand it back to them. You only throw away the key (and delete attached data) if the user deletes their account (and maybe after some additional time elapses, in case they change their mind or were hacked); it's the same process as GDPR per-user encryption key deletion.

If you throw away the key you still have the data but encrypted. There is no guarantee that in 5 years user data could be easily decrypted.

Well, the NSA slurps all Internet traffic, so by that definition, no encrypted communication is possible.

Re: The False Allure of Hashing for Anonymization

#66

Post by Thiez » Mon, Apr 30, 2018, 11:15 PM UTC

merinowool wrote: post #60

sdenton4 wrote: post #56

Earlier quoted context omitted.

It's your mapping, so you can easily gather up everything with the given marker and hand it back to them. You only throw away the key (and delete attached data) if the user deletes their account (and maybe after some additional time elapses, in case they change their mind or were hacked); it's the same process as GDPR per-user encryption key deletion.

If you throw away the key you still have the data but encrypted. There is no guarantee that in 5 years user data could be easily decrypted.

But there's no reason to believe that will be possible either. By that same reasoning it might be possible 'in 5 years' to recover the erased (and overwritten) data from the storage device, so you never can delete anything.

If you use something such as AES 256, which is approved for use to encrypt 'top secret' information by the NSA, and through some miracle it turns out that we can easily decrypt such data in 5 years, then I'm pretty sure you can argue in court that you were following best practices and had no reasonable way of predicting this encryption disaster.

Re: The False Allure of Hashing for Anonymization

#67

Post by sdenton4 » Mon, Apr 30, 2018, 11:35 PM UTC

merinowool wrote: post #60

sdenton4 wrote: post #56

Earlier quoted context omitted.

It's your mapping, so you can easily gather up everything with the given marker and hand it back to them. You only throw away the key (and delete attached data) if the user deletes their account (and maybe after some additional time elapses, in case they change their mind or were hacked); it's the same process as GDPR per-user encryption key deletion.

If you throw away the key you still have the data but encrypted. There is no guarantee that in 5 years user data could be easily decrypted.

'Key' here refers to the key in the mapping from external to internal userID. The whole point is that (as mentioned in a sibling comment) choosing an internal user ID uniformly at random is equivalent to a one-time pad; it's guaranteed non-decryptable, unless you invent a time machine...

Re: The False Allure of Hashing for Anonymization

#68

Post by [deleted] » Mon, Apr 30, 2018, 11:41 PM UTC

slooonz wrote: post #7

If you don’t require deterministic hashes (and deterministic hashes are bad for anonymization anyway) just hash data+randomBytes(16) (obviously, don't save randomBytes(16) anywhere). There you are, nobody can bruteforce your hashes. Even better, just replace your data with H(randomBytes(16)). Or a random UUID.

[deleted]

Re: The False Allure of Hashing for Anonymization

#69

Post by shabble » Tue, May 01, 2018, 12:02 AM UTC

GordonS wrote: post #48

JumpCrisscross wrote: post #45

Earlier quoted context omitted.

> If you can't connect it to the user in any way, it's no longer personal information Just because you can't connect it doesn't mean nobody else can.

For all practical purposes, a secure, one-way cryptographic hash is irreversible.

I'm thinking of a number between 1 and 100.

It's bcrypt hash is: '$2b$15$qUxzZ5ZF55lMuqiH9GMjQOHkNyee86qd2Vh2kQyF5P3U6JZJx9AEC'

I bet nobody could ever reverse this secure cryptographic hash to figure out what it could be... ;)

Re: The False Allure of Hashing for Anonymization

#70

Post by Terr_ » Tue, May 01, 2018, 12:23 AM UTC

Why not a two-step process, where you (A) generate a hash from fixed user details and (B) use that hash to access a lookup-table for the final UUID? This combines some strengths of both systems:

1. Outsiders can't determine an arbitrary UUID, even if they know the original user-details.

2. You can easily destroy a relationship (to limit correlation or to comply with laws like GDPR) by erasing the corresponding row in the lookup table.

3. Insiders can't directly go backwards from UUID to real-name, due to the hashing step. They would need to generate hashes for all the users, and hope that matches still exist in the lookup table.