Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Correct Way to Validate Email Addresses

hackernoon.com

Post reply on HN
Print view View original

341–350 of 405 posts

Re: The Correct Way to Validate Email Addresses

#341

Post by zAy0LfpBZLC8mAC »

Bartweiss wrote: post #318
zAy0LfpBZLC8mAC wrote: post #312

Earlier quoted context omitted.

1. That wasn't the argument. 2. No, it still wouldn't make sense to reject existing email addresses if there is a method to figure out whether the email address you are being presented with actually exists instead of divining validity using some unreliable proxy. There is just absolutely no reason to ever reject an email address that you can successfully send emails to.

I mean, in this case there are two reasons. They're both bad reasons, but still. 1. "Our email provider won't send to them". That excuses OP's part in the thing, although now we need to ask why the email provider is being stupid. 2. "We don't do validation links, they cause too many lost users". I have serious problems with this, but from a pure-business standpoint they decided that rejecting valid emails loses fewer…

> 1. "Our email provider won't send to them". That excuses OP's part in the thing, although now we need to ask why the email provider is being stupid.

Nope, actually, it doesn't. If your reaction to noticing that some service that you are using is incompetent is to adopt the same incompetence, that doesn't excuse anything.

> 2. "We don't do validation links, they cause too many lost users". I have serious problems with this, but from a pure-business standpoint they decided that rejecting valid emails loses fewer users than using account confirmation.

> Number two is vaguely horrifying to me, but in terms of "new users gained" it probably works out.

Well, sure, it's as much a reason as "I don't like your nose!"

If they don't do verification emails, they might as well just not ask for an email address in the first place (or make it optional). Misguided "validation" doesn't help with most mistyped addresses anyhow.

Re: The Correct Way to Validate Email Addresses

#342

Post by sly010 »

zeveb wrote: post #332
sly010 wrote: post #28

If someone's valid email address [*\"32f2@13.31.43.11 they are up to no good and I don't want them as my customer. Also, according to the standard email addresses supposed to be case sensitive, since the username part refers to a unix user and unix is case sensitive. I work with a lot of email address lists originally collected on paper and of course noone knows that. So as bad as it sounds, part of my sanitation pro…

> If someone's valid email address > [*\"32f2@13.31.43.11 > they are up to no good Well, that one in particular, perhaps — but what about "sam & jill"@ourfamily.invalid? What about john(for sly010)@nowhere.invalid? > Also, according to the standard email addresses supposed to be case sensitive, since the username part refers to a unix user and unix is case sensitive. The local-part doesn't refer to a 'Unix user'; it…

I am all for doing things by standards, but to me this is de facto. No one uses case sensitive emails. If they did all sorts of weird things would happen: People couldn't log in, but other people could. There would be multiple accounts when there should be only one, etc. You would have to store every email in original format and in some canonical format for uniqueness, etc. It's a nightmare.

If I had to guess the main reason this apocalypse is not happening already is because by default mysql indexes are case insensitive, so dude@dude.com is the same as dude@DUDE.com, so people who use mysql never realize this being an issue.

Re: The Correct Way to Validate Email Addresses

#343

Post by Kadin »

jwecker wrote: post #57
kej wrote: post #24

Earlier quoted context omitted.

And that's a great philosophy until your email gets rejected by some service that picked a different "common class of email addresses" than you did. This is precisely why we have written standards.

Back when SMTP servers still had remnants of UUCP etc.- where the address could actually contain characters that specify intermediate servers to route to, I would have argued that front-end sanitization was important as, for example, html sanitization from end-users from a security perspective. However, IETF made lots of progress simplifying things- to the point where, at the very least, the standard tells us specifi…

I think there is a reasonable middle-ground for validating the domain side of an email address. There are RFCs on all this stuff; it's not just a total free-for-all. The RFCs just aren't nearly as strict as a lot of badly-designed validation regexps are, presumably because most people are unaware of the diversity of acceptable email addresses.

The currently operative RFC is 5322, specifically section 3.4.1: https://tools.ietf.org/html/rfc5322#section-3.4.1

There are some basic rules that you could safely apply to an address, which would prevent some attacks (buffer overflows, etc.) while also not blocking any legitimate addresses. E.g. limiting the overall length to 255 characters, for instance, could be defensible practice. There are also well-defined rules for validating the domain portion, since it has to be a routable address by definition.

What nobody ought to be doing is looking too hard at the string to the left of the @ symbol, because it's designed purely as instructions to the recipient server. Nobody else needs to care about it; only the receiving mailserver needs to actually parse that part of the address, in order to put the message into the right mailbox. From what I've seen, the vast majority of false-positive validation failures occur because people are looking at the mailbox portion of an email address when they have no business doing so.

Re: The Correct Way to Validate Email Addresses

#344

Post by zAy0LfpBZLC8mAC »

zeveb wrote: post #326
zAy0LfpBZLC8mAC wrote: post #73

Earlier quoted context omitted.

1. There are no "obviously invalid email address[es]". 2. Getting billed for sending emails? WTF?

> There are no "obviously invalid email address[es]". An email address which doesn't validate against the applicable RFC is ipso facto obviously invalid. Validating against the RFC is somewhat complex though.

> Validating against the RFC is somewhat complex though.

Hence, not obvious :-)

If there is one thing we can learn from all the misguided attempts at validating email addresses that we have seen over the years, it is that it is obviously not obvious which email addresses are syntactically invalid.

Re: The Correct Way to Validate Email Addresses

#345

Post by Sami_Lehtinen »

Benjamin_Dobell wrote: post #30

The number of websites that try reject my email address with a + in it, ugh! Surprisingly, the validation is often done 100% client-side anyway, and simply modifying the incorrect regex lets my email address through... If I wrecked havoc on your back-end, then it's your fault for sucking ;)

I've got # in my address and some systems fail with it too. It's totally valid. As example outlook.com won't allow sending email to addresses with # . Also some services reject short addresses for no reason. As example m@em.mm or so, that could be totally valid. You don't know it before checking for MX info and asking from the SMTP server if the address is valid in rcpt to: .

Re: The Correct Way to Validate Email Addresses

#346

Post by Bartweiss »

zAy0LfpBZLC8mAC wrote: post #341
Bartweiss wrote: post #318

Earlier quoted context omitted.

I mean, in this case there are two reasons. They're both bad reasons, but still. 1. "Our email provider won't send to them". That excuses OP's part in the thing, although now we need to ask why the email provider is being stupid. 2. "We don't do validation links, they cause too many lost users". I have serious problems with this, but from a pure-business standpoint they decided that rejecting valid emails loses fewer…

> 1. "Our email provider won't send to them". That excuses OP's part in the thing, although now we need to ask why the email provider is being stupid. Nope, actually, it doesn't. If your reaction to noticing that some service that you are using is incompetent is to adopt the same incompetence, that doesn't excuse anything. > 2. "We don't do validation links, they cause too many lost users". I have serious problems wi…

I oversold 'excuses'. Let's say "means the original error lies elsewhere". I also wonder who the hell they're using - who ever heard of an email service that bans domains for being likely misspellings?

The "it's as much a reason" I disagree with. Validating common typos will catch more errors than false positives, so you do get more users through your funnel than if you abandon it. "I don't like your nose" is a strict loss, this causes corrections to get real emails. So they'll still miss most typos, but it's a net gain compared to not doing it.

Of course, again, I don't endorse any of this. Decide if you're ok with bad emails, follow through on that decision, use verification, and get a not-incompetent email service.

Re: The Correct Way to Validate Email Addresses

#347

Post by Bartweiss »

billmalarky wrote: post #336
Bartweiss wrote: post #303

Earlier quoted context omitted.

Meet Time Warner Cable. Time Warner, which has an obscenity filter on their passwords. Enforced server-side! Bets on how long before those folks make the news for losing a hundred million unhashed passwords?

Devil's Advocate: They could filter for obscenities before hashing.

The concern isn't that there's no way to do it, it's that there's no reason to do it unless the plaintext is going to matter again in the future.

The best guess I've seen is that they might be hashing, but also having people read passwords to customer service reps. That would justify caring about rude plaintext, but it's also a terrible system.

Re: The Correct Way to Validate Email Addresses

#348

Post by throwanem »

zAy0LfpBZLC8mAC wrote: post #331
throwanem wrote: post #274

Earlier quoted context omitted.

"Cannot be resolved" means NXDOMAIN. Why assume email addresses only get checked in one place, and not all? Ten million a day was a milestone. I left that company over a year ago; it would astonish me to find that figure now exceeded by less than a factor of twenty. Granted these are mostly not signups. They are outgoing emails nonetheless, which makes the case germane despite that superficial distinction. Your propo…

> "Cannot be resolved" means NXDOMAIN. OK, that at least shouldn't reject any valid addresses, so maybe ... > Why assume email addresses only get checked in one place, and not all? It's not an assumption, it's just a matter of simplicity and reliability. > Ten million a day was a milestone. I left that company over a year ago; it would astonish me to find that figure now exceeded by less than a factor of twenty. Gran…

I may have erred in giving the impression that the application-level checks are the only line of defense here. They're not. The (bespoke) MTA underlying this product performs most if not all of these checks as well. I didn't really spend any time on that side of the business, so I might be wrong about that, but it would be something of a surprise. I do know our analytics needed to be able to cope usefully with an astonishing panoply of bogosity warnings that came back from the MTA, but I no longer recall exactly what they covered. And, in any case, it's nice when you can to tell the user "hey, this isn't deliverable" before it gets to the point of a bounce.

Checking whether an email address's domain-part is an RFC1918 IP is actually pretty easy. Split the address by '@'. The last piece is the domain part. Split it by '.'. If there are four pieces, all of which meaningfully cast to integers, treat it as an IP address. (Otherwise it's a domain name, which is fine as long as it has more than one part and isn't NXDOMAIN when the backend tries to resolve it.) If any part is negative or greater than 255, it's invalid. If it starts with [10] or [192, 168], or if it starts with [172] and the second part is between 16 and 31 inclusive, it's an RFC1918 address. Otherwise, it's fine.

Even with unit tests, that takes almost no time to write, and when your frontend and backend share a language as ours did, you can use the same logic both places. Node gives you a name resolver binding for free. I really can't imagine it being as quick to write, test, validate, and roll out a change to the MTA node manifest.

Re: The Correct Way to Validate Email Addresses

#349

Post by billmalarky »

Bartweiss wrote: post #347
billmalarky wrote: post #336

Earlier quoted context omitted.

Devil's Advocate: They could filter for obscenities before hashing.

The concern isn't that there's no way to do it, it's that there's no reason to do it unless the plaintext is going to matter again in the future. The best guess I've seen is that they might be hashing, but also having people read passwords to customer service reps. That would justify caring about rude plaintext, but it's also a terrible system.

Heh, to be fair, we are talking about a company with millions of customers that is consistently rated as "most hated" by consumers. I'd be surprised if a decent percentage of attempted passwords weren't "fuck time warner"...

Re: The Correct Way to Validate Email Addresses

#350

Post by dugluak »

burnbabyburn wrote: post #249

tl;dr send a confirmation email. it's 20 years that people suggest a cool new way to deal with email addresses, I don't even mind listening anymore! :)

I think its good to do some kind of validation upfront. It's too much to ask a user to go through whole process again for a silly typo.
Post reply on HN