The False Allure of Hashing for Anonymization • Hacker News Board

Re: The False Allure of Hashing for Anonymization

#101

Post by kevin_nisbet » Tue, May 01, 2018, 1:35 PM UTC

your-nanny wrote: post #99

kevin_nisbet wrote: post #83

Earlier quoted context omitted.

OP Here. With my understanding of bcrypt, it's an algorithm that's designed to be slow (and the implementation providing guarantee's to resist attempts to significantly speed it up), and the slowness is tuneable through a work factor. So usually you would target something like 200-500ms. Long enough to be slow if you have to make billions of guesses, but still fast enough that when someone enters their correct passwo…

I see, so assuming the salt is known (which I guess is a reasonable assumption in a large fraction of cases).

Yes, strictly speaking a salt is not a secret, and would generally be stored with the data you are salting.

If you change the semantics and make the salt a secret that is stored separately, it does make this difficult to attack, but the advice I was given is it would be better to use hmac, which is already designed to work this way based on storing a secret.

Re: The False Allure of Hashing for Anonymization

#102

Post by russnewcomer » Tue, May 01, 2018, 3:12 PM UTC

entee wrote: post #59

russnewcomer wrote: post #53

Earlier quoted context omitted.

But if I do need the original data back, say, the driver needs to produce an expense report with the hours, what would you do in that case? I have thoughts, but trying to bounce off of someone else.

If you need to provide the data back to the customer, then maybe the right answer is to follow the same standards as financial institutions and health companies do. In practice, that comes down to ensuring that no individual has access to the underlying data without extreme monitoring of how that data moves around and is used. This is a rather large burden though, so I can understand if that's too much for your use c…

That makes sense. The data set is going to be in the health area, and I'm less concerned about processes for the individuals in the organization having access (like what you've suggested) and more thinking about how to structure the data so we as an organization can't access it. Dealing with infectious disease, where there is personal benefit to not letting someone outside the care side know that you have a disease, but societal benefit to tracking trends, outbreaks, or hygiene around the disease. And figuring out how to structure the system so that if we were to sell, say, there wouldn't be this trove of information on who has what diseases, just who was a customer.

Thanks for your thoughts!

Re: The False Allure of Hashing for Anonymization

#103

Post by jonnycomputer » Tue, May 01, 2018, 3:15 PM UTC

kevin_nisbet wrote: post #101

your-nanny wrote: post #99

Earlier quoted context omitted.

I see, so assuming the salt is known (which I guess is a reasonable assumption in a large fraction of cases).

Yes, strictly speaking a salt is not a secret, and would generally be stored with the data you are salting. If you change the semantics and make the salt a secret that is stored separately, it does make this difficult to attack, but the advice I was given is it would be better to use hmac, which is already designed to work this way based on storing a secret.

Okay, that's what I thought. Under my current use-case, I think what I'm doing is quite adequate. But, your post is very relevant to what I think I'm going to need to do at some point soon, so thank you!

Re: The False Allure of Hashing for Anonymization

#104

Post by aflagatopamoon » Tue, May 01, 2018, 8:57 PM UTC

bglusman wrote: post #81

Area12 wrote: post #74

I am not a crypto expert, but I thought that the idea was to produce a new more or less random salt for EACH password, store the salt with the hashed password, hashing using an expensive algorithm. Yes the hacker steals the salt with the hash, but now has to go to the trouble of brute forcing that ONE password with its UNIQUE (or almost unique) salt. In other words, the hacker can crack it, but the process is so expe…

For the specific use case in question, what I've been doing for years is not just hashing the data, but hashing an internal secret AND the data. The secret isn't stored in the database anywhere (usually an env var but could be a secret in vault or other outside config), so our hashes are deterministic (and don't need a seperate salt for each one), but our hashes will never cooincide with another system's hashes. I di…

This is called a pepper:

https://en.wikipedia.org/wiki/Pepper_(cryptography)

If you read the article above, you'll see that you still need a salt, since users with very simple passwords will have the same hash: crack one, and you can crack the others for free.

Re: The False Allure of Hashing for Anonymization

#105

Post by oasisbob » Thu, May 03, 2018, 2:53 PM UTC

ryanworl wrote: post #47

unilynx wrote: post #46

Earlier quoted context omitted.

Wouldn't storing the first three octets of an IP address be enough for this kind of analysis? Or use the whois database and reduce the data to the first IP address of the network ?

I personally think just storing the autonomous system the IP originates from and never writing the IPs to disk at all would be advisable if the goal is purely which ISPs are delivering how many bytes to end users. Another benefit is the AS to IP mapping database is small enough to fit in memory without issue.

That's probably insufficient for the usecase. A single AS can advertise many different routes for different IP blocks that have dramatic geographic differences.

Re: The False Allure of Hashing for Anonymization

#106

Post by bglusman » Fri, May 11, 2018, 5:05 PM UTC

aflagatopamoon wrote: post #104

bglusman wrote: post #81

Earlier quoted context omitted.

For the specific use case in question, what I've been doing for years is not just hashing the data, but hashing an internal secret AND the data. The secret isn't stored in the database anywhere (usually an env var but could be a secret in vault or other outside config), so our hashes are deterministic (and don't need a seperate salt for each one), but our hashes will never cooincide with another system's hashes. I di…

This is called a pepper: https://en.wikipedia.org/wiki/Pepper_(cryptography) If you read the article above, you'll see that you still need a salt, since users with very simple passwords will have the same hash: crack one, and you can crack the others for free.

Yeah I should have been clear this wasn't for passwords at all, ever, this was/is only for other kinds of PII