Live data from Hacker News

The Correct Way to Validate Email Addresses

hackernoon.com

331–340 of 405 posts

Re: The Correct Way to Validate Email Addresses

#331

Earlier quoted context omitted.

> 'Non-routable' as in a single label, or as in not resolvable. I don't think it is unreasonable to consider an address invalid when its domain part cannot be resolved. What exactly do you mean by "cannot be resolved"? > Checking for RFC1918 ranges means you don't try to send to another class of addresses that's never going to be received. But why check for it? Is that actually a common mistake people make? An attack…

"Cannot be resolved" means NXDOMAIN. Why assume email addresses only get checked in one place, and not all? Ten million a day was a milestone. I left that company over a year ago; it would astonish me to find that figure now exceeded by less than a factor of twenty. Granted these are mostly not signups. They are outgoing emails nonetheless, which makes the case germane despite that superficial distinction. Your propo…

> "Cannot be resolved" means NXDOMAIN.

OK, that at least shouldn't reject any valid addresses, so maybe ...

> Why assume email addresses only get checked in one place, and not all?

It's not an assumption, it's just a matter of simplicity and reliability.

> Ten million a day was a milestone. I left that company over a year ago; it would astonish me to find that figure now exceeded by less than a factor of twenty. Granted these are mostly not signups. They are outgoing emails nonetheless, which makes the case germane despite that superficial distinction.

Well, no clue how well common MTAs would cope with that, but 2500 mails per second could still be within the power of a single machine if it's designed for high performance. But regardless, I don't think that really matters: If you have a relatively low volume of emails, it's probably most efficient and secure to handle it all with one central outbound relay, if you need to send lots of emails, it obviously makes sense to distribute the load, but that shouldn't really otherwise change the strategy.

> Your proposed solution sounds pretty expensive in ops resource, to no obviously greater benefit than the rather simple (well under one dev-day) option we chose. You seem to feel yours is strongly preferable, but I still don't understand why.

What sounds expensive about it?

Whether there is any benefit to it: Well, depends on your goals and what exactly your solution actually does. I still don't understand why (or even how exactly) you do those RFC1918 checks, for example?! It seems like it's mostly a security measure? But then, it's not actually secure, it's essentially a race condition/TOCTTOU. Plus, it might even break valid email addresses.

Essentially, it's four factors why I think just delegating the validation of email addresses to the mail server is the best strategy:

1. Implementing your own checks risks introducing additional mistakes (which might lead to the rejection of valid addresses).

2. Implementing your own checks is additional work when you could just use the MTA which already knows how to do this (and which you have to install/configure/use anyway as soon as you want to actually use the email address), both for the initial implementation, and possibly for subsequent maintenance (if you just let your MTA do the work, only the MTA needs to be adapted to any changes in how emails get delivered, abstracting away the problem for any software that's supposed to be sending emails and isolating it from the lower layers).

3. My approach actually gives you the perfect result, in that it does not reject any valid addresses (assuming your MTA implements the RFCs correctly), and at the same time is perfectly secure against all possible abuses with weird addresses, which is impossible to achieve when you separate the check from the actual abuse scenario, and it's even kindof trivial to see that that is the case.

4. You have to have the infrastucture to deal with bounces anyhow, both because you ultimately cannot be sure an address actually exists unless you have successfully delivered an email to it, and because addresses that once existed might not exist anymore at a later time, so it's not like you can avoid that if only you validate your addresses better.

As for "ops resources": Assuming that any service that needs to send > 10 million mails per day will be deploying machines automatically anyway, what's so much more expensive about deploying the configuration of an additional network namespace? Writing that script certainly shouldn't be more than a day of work either, should it?

Re: The Correct Way to Validate Email Addresses

#332
post #28

If someone's valid email address [*\"32f2@13.31.43.11 they are up to no good and I don't want them as my customer. Also, according to the standard email addresses supposed to be case sensitive, since the username part refers to a unix user and unix is case sensitive. I work with a lot of email address lists originally collected on paper and of course noone knows that. So as bad as it sounds, part of my sanitation pro…

> If someone's valid email address

> [*\"32f2@13.31.43.11

> they are up to no good

Well, that one in particular, perhaps — but what about "sam & jill"@ourfamily.invalid? What about john(for sly010)@nowhere.invalid?

> Also, according to the standard email addresses supposed to be case sensitive, since the username part refers to a unix user and unix is case sensitive.

The local-part doesn't refer to a 'Unix user'; it is what it says on the tin: the local part. It's up to the receiving server to determine what it wants to do with case. It's the duty of other servers to preserve the email address as transmitted.

> I work with a lot of email address lists originally collected on paper and of course noone knows that. So as bad as it sounds, part of my sanitation process is to lowercase everything.

I might use the email address as given, and if that bounces try lowercasing. I don't know of anywhere that has case-sensitive names, but it's conceivable.

Re: The Correct Way to Validate Email Addresses

#333

Sorry, I disagree. There is absolutely is a correct way to lexically validate an e-mail address: namely, implement a parser for the syntax specified in whatever RFC is the up-to-date successor of RFC 822. There is such a thing as incorrect e-mail address syntax: namely, non-RFC-conforming syntax, whatever that is. You may reject that, and that's about it. Please don't reject RFC-conforming e-mail addresses.

I agree, with two specific exclusions: notations for IP addresses in square brackets, and comments. Just removing those two never-encountered-in-real-life syntaxes reduces the RFC regex down to nothing. Comments are not a real thing, RFC be damned. And square bracketed IP addresses are never encountered in the real world, especially with the requirements for PTR records to pass all the major providers' spam filters.…

> Comments are not a real thing, RFC be damned.

Comments are and should be a real thing. They are, for example, how to use service-specific email addresses when using a mail provider who does not treat + as a folder extension (jack+suzy@provider.invalid is a perfectly valid email address for Jack & Suzy).

FWIW, I tested recently, and Google do exactly the Right Thing™ with comments in email addresses: they deliver the mail.

Re: The Correct Way to Validate Email Addresses

#334

Earlier quoted context omitted.

Meet Time Warner Cable. Time Warner, which has an obscenity filter on their passwords. Enforced server-side! Bets on how long before those folks make the news for losing a hundred million unhashed passwords?

That is ridiculous. Why would you even filter passwords? It's not like it is public, unless you are planning on making it so.

My guess is that when you call them on the phone, they ask for your password to validate your identity. Which means it's stored in plain text in their database so that customer service can verify what you said is correct. Maybe they don't want their employees to have to be cursed at by customers.

I can't think of a good way for a business that has an online interface and frequently handles phone calls from customers to validate that they're talking to the correct person. Asking for other personal information can be used by an attacker to compromise multiple accounts via social engineering: http://www.wired.com/2012/08/apple-amazon-mat-honan-hacking/

Re: The Correct Way to Validate Email Addresses

#335
post #101

Correct way is run a regex for [wildcard @ wildcard . Wildcard] then send an opt-in email real-time as the user is typing additional info. If it bounces before the user finished the onboard form - alert them to the issue. If it gets validate, autologin the user. If bounces or there's zero response by the time the user completes the form, alert them, ask to type their email again without access to the reprior entry -…

> [wildcard @ wildcard . Wildcard]

That would invalidate ceo@cocacola, which is a legitimate address.

Re: The Correct Way to Validate Email Addresses

#336

Earlier quoted context omitted.

Even worse is rejecting my password because it has a + in it! Why do you as a business care what my random password generator spit out?? Scarier still is when it's a server-side response that rejects my password for its contents...

Meet Time Warner Cable. Time Warner, which has an obscenity filter on their passwords. Enforced server-side! Bets on how long before those folks make the news for losing a hundred million unhashed passwords?

Devil's Advocate: They could filter for obscenities before hashing.

Re: The Correct Way to Validate Email Addresses

#337
post #334

Earlier quoted context omitted.

That is ridiculous. Why would you even filter passwords? It's not like it is public, unless you are planning on making it so.

My guess is that when you call them on the phone, they ask for your password to validate your identity. Which means it's stored in plain text in their database so that customer service can verify what you said is correct. Maybe they don't want their employees to have to be cursed at by customers. I can't think of a good way for a business that has an online interface and frequently handles phone calls from customers…

Someone else in this thread mentioned a company that has customer service type in your password to open your account. So that would be a non-plaintext reason to insist on non-obscene passwords. But it's still terrible, because why the hell is customer service typing in your password.

Pretty much all organizations that allow phone authentication seem to be at risk of engineering attacks. The only ones that manage it send you something verifiable they can ask about like a credit card, and people who really care like the government just send an actual human to your house.

Re: The Correct Way to Validate Email Addresses

#338
post #334

Earlier quoted context omitted.

That is ridiculous. Why would you even filter passwords? It's not like it is public, unless you are planning on making it so.

My guess is that when you call them on the phone, they ask for your password to validate your identity. Which means it's stored in plain text in their database so that customer service can verify what you said is correct. Maybe they don't want their employees to have to be cursed at by customers. I can't think of a good way for a business that has an online interface and frequently handles phone calls from customers…

It doesn't have to be stored in plain text to validate it is correct. The phone operator could enter it into an authentication form to verify it is correct.

Re: The Correct Way to Validate Email Addresses

#339
post #301

Earlier quoted context omitted.

"Cannot be resolved" means NXDOMAIN. Why assume email addresses only get checked in one place, and not all? Ten million a day was a milestone. I left that company over a year ago; it would astonish me to find that figure now exceeded by less than a factor of twenty. Granted these are mostly not signups. They are outgoing emails nonetheless, which makes the case germane despite that superficial distinction. Your propo…

NXDOMAIN can be a temporary error. The SMTP queuing protocol is designed to be resilient against DNS failures, internet outages, routing problems, and temporary mail delivery issues.

> NXDOMAIN can be a temporary error.

Unless some DNS server is broken, it actually cannot. NXDOMAIN is an authoritative answer that tells you that the domain positively does not exist. Not to be confused with SERVFAIL, which you should get if the DNS resolver ran into a timeout or got an unintelligible response or whatever, NXDOMAIN should only occur if the authoritative nameserver of a parent zone explicitly says "I don't know this zone either locally nor do I have a delegation for it".

Re: The Correct Way to Validate Email Addresses

#340
post #334

Earlier quoted context omitted.

That is ridiculous. Why would you even filter passwords? It's not like it is public, unless you are planning on making it so.

My guess is that when you call them on the phone, they ask for your password to validate your identity. Which means it's stored in plain text in their database so that customer service can verify what you said is correct. Maybe they don't want their employees to have to be cursed at by customers. I can't think of a good way for a business that has an online interface and frequently handles phone calls from customers…

Customer care clicks a button to create a random temporary token associated with that account and user has to log on and read that token.
Post reply on HN