Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You might not need to store plaintext email addresses

blog.klungo.no

Post reply on HN
Print view View original

21–30 of 181 posts

Re: You might not need to store plaintext email addresses

#21

Post by dmurray »

inopinatus wrote: post #12

This scheme struggles in the face of email address case folding. At the protocol level, email addresses are case-folded on the RHS but case-sensitive on the LHS. So it’s crucial that LHS case is preserved by delivery systems. Unfortunately most users then treat them as folded on both. So you can successfully verify one variant, store the downcased hash, and it’ll subsequently match but delivery bounces. Or, hash the…

You could store the hash of the downcased address plus a capitalization mask which tells you which letters to capitalize.

This works from a technical perspective, as letters with ambiguous capitalization (Turkish i, etc) aren't allowed in emails. It's a very minor privacy compromise: if a user has a very rare pattern of capitalization then an attacker with access to the database could identify their account. Negligible compared to the current standard.

Re: You might not need to store plaintext email addresses

#22

Post by inopinatus »

dmurray wrote: post #21
inopinatus wrote: post #12

This scheme struggles in the face of email address case folding. At the protocol level, email addresses are case-folded on the RHS but case-sensitive on the LHS. So it’s crucial that LHS case is preserved by delivery systems. Unfortunately most users then treat them as folded on both. So you can successfully verify one variant, store the downcased hash, and it’ll subsequently match but delivery bounces. Or, hash the…

You could store the hash of the downcased address plus a capitalization mask which tells you which letters to capitalize. This works from a technical perspective, as letters with ambiguous capitalization (Turkish i, etc) aren't allowed in emails. It's a very minor privacy compromise: if a user has a very rare pattern of capitalization then an attacker with access to the database could identify their account. Negligib…

It’s a neat idea but unfortunately RFC 6531 opened up the local part to most of UTF-8, so internationalised capitalisation is in the mix now.

Ultimately I’ll never advise delivering to email addresses other than the precise octets of the one already verified, and this means the gold standard is always folding for match and uniqueness, but delivery precisely as verified.

How about this: store the verified email address, but encrypted using the hash of the case-folded input as part of the key. The intention being, you had to have the matching folded form in hand to obtain the verified canonical form. For extra jollies, only decrypt it on the client. (cryptography warning: I write this as the idea comes to me and without any analysis of emergent properties, vulnerabilities etc)

Re: You might not need to store plaintext email addresses

#23

Post by tlarkworthy »

I think you can store emails but process them only in an anonymizing publicly auditable proxy, ensuring that downstream business services do not have plaintext access whilst still being able to send outbound emails whenever you want. I wrote about it recently: https://futurice.com/blog/trustworthy-services-from-cloud-pr...

The key is to grant cloudfunctions.functions.sourceCodeGet (Or AWS/Azure equivalent) on the edge so anybody can verify that your proxy is above board. End users just have to trust the Cloud Providers access controls, not the service providers word on implementation.

Re: You might not need to store plaintext email addresses

#24

Post by gpvos »

inopinatus wrote: post #12

This scheme struggles in the face of email address case folding. At the protocol level, email addresses are case-folded on the RHS but case-sensitive on the LHS. So it’s crucial that LHS case is preserved by delivery systems. Unfortunately most users then treat them as folded on both. So you can successfully verify one variant, store the downcased hash, and it’ll subsequently match but delivery bounces. Or, hash the…

True. But since so many websites (e.g. all aviation companies I've encountered) case-smash the LHS of the email address and can get away with it since all other email software has had to adapt, this is a rather minor concern by now.

Re: You might not need to store plaintext email addresses

#25

Post by laszlokorte »

inopinatus wrote: post #19
opk wrote: post #17

Earlier quoted context omitted.

No it doesn't, convert the case when generating the hash – think of it as part of the hash function. But leave the case unchanged from whatever the user entered for any steps that involve sending an e-mail to the address.

That is what produces the first failure mode described, viz. bouncing email, when delivering to a case-sensitive mailbox.

But wouldnt the user face this issue all the time if they have a case sensitive mailbox but type their own address in the wrong case? So the assumption may be that a user with a case sensitive mail box is used to typing in the correct address?

Re: You might not need to store plaintext email addresses

#27

Post by ThePhysicist »

In most cases, encrypting sensitive information like e-mail addresses with a memory-resident key (e.g. injected using tools like Vault) in the application layer is a better strategy, at least if you need asynchronous access to that information (e.g. to send out weekly update e-mails). Most of the data leaks in the past were caused by compromised or misconfigured databases, not by compromised application server code.

Also, within the EU I need to be able to proactively reach my users (e.g. to notify them about a data loss), so only storing hashes of e-mail addresses and hoping users will log in so that I can send them an e-mail won't work.

Re: You might not need to store plaintext email addresses

#28

Post by waheoo »

danielskogly wrote: post #11
cm2187 wrote: post #5

Good points. Though given how many emails have been leaked already, not sure sha256 with fixed salt achieves much. One can build a rainbow table with that salt fairly quickly. You might as well use bcrypt, scrypt and co.

Using something like bcrypt would definitively be better, but considering that the email is the identifier, I would have no way of retrieving the correct hash to check it against, so the salt must be fixed to allow for lookups. I'm currently using SHA512 with a fixed salt. If someone gains access to only the database and not the salt, the emails are well protected. If someone gains access to both, then it's true that…

Use the password as the salt.

Still need a fixed salt but that should help.

Re: You might not need to store plaintext email addresses

#29

Post by inopinatus »

laszlokorte wrote: post #25
inopinatus wrote: post #19

Earlier quoted context omitted.

That is what produces the first failure mode described, viz. bouncing email, when delivering to a case-sensitive mailbox.

But wouldnt the user face this issue all the time if they have a case sensitive mailbox but type their own address in the wrong case? So the assumption may be that a user with a case sensitive mail box is used to typing in the correct address?

No — many accounts aren’t created by the people that use them, and in any case we shouldn’t be relying on correct repeated string input and then blaming the user for a fulfilment process failure due to a typo from hours ago that we silently accepted at the time, and (worse) we can’t even distinguish between a capitalisation error and a discontinued recipient, even if it was previously verified.

As designers/developers, it’s our problem to solve.

Re: You might not need to store plaintext email addresses

#30

Post by aitchnyu »

ThePhysicist wrote: post #27

In most cases, encrypting sensitive information like e-mail addresses with a memory-resident key (e.g. injected using tools like Vault) in the application layer is a better strategy, at least if you need asynchronous access to that information (e.g. to send out weekly update e-mails). Most of the data leaks in the past were caused by compromised or misconfigured databases, not by compromised application server code.…

Is that generalizaion true for other forms of data theft too? I feel that way about some wrongly public s3 buckets and document leaks.
Post reply on HN