The Correct Way to Validate Email Addresses • Hacker News Board

Re: The Correct Way to Validate Email Addresses

#271

Post by Freak_NL » Thu, Sep 08, 2016, 12:11 PM UTC

mgkimsal wrote: post #260

Freak_NL wrote: post #223

Earlier quoted context omitted.

Because the other monkeys will chew you out if you start doing it differently all of a sudden. Nobody knows exactly why we're doing it the way we are doing it, but it is complex, and changing it might break something. It's the five monkey experiment : http://johnstepper.com/2013/10/26/the-five-monkeys-experimen...

It's not always that. Let's use the "filtering chars from password" example above. You can't put some special chars in password field, and you want to change that so it's doing normal hashing where special chars don't matter. In a larger org, even changing a practice like that so that it "makes more sense" can have a big ripple effect. You have to * explain to someone else on the team who came up with the original pr…

At a small business convincing colleagues isn't that hard. At a larger corporation, getting the rest of the team on-board is the job of whoever is in charge of defining security policies and such. Not allowing characters present on all keyboards and input devices (!"@+'$) statistically increases the risk of people picking weaker passwords then possible, and he/she will guide that change through the proper processes, for example to make sure that any client software interacting with the backend is aware of the upcoming change. Same as with any security issue (e.g., the deprecation of SSLv3 ciphers in favour of newer TLS versions).

If a business can't handle a change like this, something is really broken in the development pipeline. Granted, that describes a lot of medium sized companies…

Re: The Correct Way to Validate Email Addresses

#272

Post by zAy0LfpBZLC8mAC » Thu, Sep 08, 2016, 12:17 PM UTC

throwanem wrote: post #263

zAy0LfpBZLC8mAC wrote: post #257

Earlier quoted context omitted.

> Depends on what you do with them, in the latter case. There could be an amplification attack there. Hu? How would that work? > Validating domain parts to a certain extent isn't a bad idea, at least as far as non-routable domain names and RFC1918 ranges go. What do you mean by "non-routable domain names" and what do you gain by checking for RFC1918 ranges? > I've seen this done (actually implemented some of it, in f…

'Non-routable' as in a single label, or as in not resolvable. I don't think it is unreasonable to consider an address invalid when its domain part cannot be resolved. Checking for RFC1918 ranges means you don't try to send to another class of addresses that's never going to be received. You would lose the bet. The product supported IPv6 from day one. That is a robust, if somewhat complex, solution for a relatively sm…

> 'Non-routable' as in a single label, or as in not resolvable. I don't think it is unreasonable to consider an address invalid when its domain part cannot be resolved.

What exactly do you mean by "cannot be resolved"?

> Checking for RFC1918 ranges means you don't try to send to another class of addresses that's never going to be received.

But why check for it? Is that actually a common mistake people make? An attacker could change the address after you checked it, so it's not going to help against attackers, is it?

> You would lose the bet. The product supported IPv6 from day one.

Good for you! :-)

> That is a robust, if somewhat complex, solution for a relatively small volume of mail. When you're sending ten million messages a day by the end of the first month, pushing everything into a single relay of any kind is asking for a lot of trouble.

Complex? Certainly less so than implementing validation yourself.

As for scalability: Well, yeah, as described that's more the setup for a company that's operating various different services, none of which has a high volume of outbound email (which would be most, even the best startups don't have ten million signups per day and don't send much email otherwise, and even that should actually still be managable with a single server).

But that's trivial to adapt without changing the general approach. First of all, obviously, you could just add more relay servers and have client servers select one randomly, that scales linearly. But if you really need to move massive amounts of email for one service, so that adding an additional relay hop for each email you send actually adds up to noticable costs, you can still use the same approach: Just put the MTA onto the same machine(s) that the service is running on, into its own network namespace (assuming Linux, analogous technology exists on other platforms), and firewall it off there so it cannot connect to your internal network. Potentially you can even just add blackhole routes for your internal networks/RFC1918 ranges, so you would not even need a stateful packet filter (though currently you might still need it due to IPv4 address shortage).

Re: The Correct Way to Validate Email Addresses

#273

Post by vkjv » Thu, Sep 08, 2016, 12:18 PM UTC

I like the approach of accepting anything that is a reasonable length with an`@`, but suggestinging corrections for possibly misspelled common domains.

https://github.com/mailcheck/mailcheck

If you really need to validate, the only way I know how is to send them an email and click a link to confirm.

Re: The Correct Way to Validate Email Addresses

#274

Post by throwanem » Thu, Sep 08, 2016, 12:23 PM UTC

zAy0LfpBZLC8mAC wrote: post #272

throwanem wrote: post #263

Earlier quoted context omitted.

'Non-routable' as in a single label, or as in not resolvable. I don't think it is unreasonable to consider an address invalid when its domain part cannot be resolved. Checking for RFC1918 ranges means you don't try to send to another class of addresses that's never going to be received. You would lose the bet. The product supported IPv6 from day one. That is a robust, if somewhat complex, solution for a relatively sm…

> 'Non-routable' as in a single label, or as in not resolvable. I don't think it is unreasonable to consider an address invalid when its domain part cannot be resolved. What exactly do you mean by "cannot be resolved"? > Checking for RFC1918 ranges means you don't try to send to another class of addresses that's never going to be received. But why check for it? Is that actually a common mistake people make? An attack…

"Cannot be resolved" means NXDOMAIN.

Why assume email addresses only get checked in one place, and not all?

Ten million a day was a milestone. I left that company over a year ago; it would astonish me to find that figure now exceeded by less than a factor of twenty. Granted these are mostly not signups. They are outgoing emails nonetheless, which makes the case germane despite that superficial distinction.

Your proposed solution sounds pretty expensive in ops resource, to no obviously greater benefit than the rather simple (well under one dev-day) option we chose. You seem to feel yours is strongly preferable, but I still don't understand why.

Re: The Correct Way to Validate Email Addresses

#275

Post by scaryclam » Thu, Sep 08, 2016, 12:40 PM UTC

zAy0LfpBZLC8mAC wrote: post #235

random_rr wrote: post #112

Earlier quoted context omitted.

You haven't worked in email, I presume. Yes, it does cost money to send emails at a large scale.

I am not sure what you are trying to say. Yes, of course, it costs money. Just like serving websites costs money. And just as checking the syntax of an email address costs money (CPUs cost money and need power to run!). Now, how is that an argument for checking the syntax of an email address to avoid sending a single email and seeing whether it bounces?

Sending a lot of bounced emails is going to get your mailservers reputation tanked, which is not an acceptable tradeoff for removing email validation.

Email validation is actually really important if you send even a medium amount of email. Testing by bouncing is not a very good idea, especially since there are a lot of ways to test the deliverability of the email before you try and send it.

That's not to say use a regex instead, you can do some simple checks on the domain etc or you can use one of the many services to check for you (for of a fee of course).

It's understandable not to know this unless you've run a medium - large site, so here's some best practices to follow: https://documentation.mailgun.com/best_practices.html#email-...

Re: The Correct Way to Validate Email Addresses

#276

Post by flagelate » Thu, Sep 08, 2016, 12:47 PM UTC

I literally just check for the presence of an @ with something on either side. Anything more, you're going to piss off a user at one point or another.

Re: The Correct Way to Validate Email Addresses

#277

Post by kasparsklavins » Thu, Sep 08, 2016, 12:54 PM UTC

creshal wrote: post #229

Freak_NL wrote: post #222

Earlier quoted context omitted.

> Scarier still is when it's a server-side response that rejects my password for its contents... Why? Assuming that you have established a secure connection with that server (i.e., HTTPS by means of TLS), then it is perfectly fine for the server to check, at the time you are setting the password , if your password confirms to the rules established. And when the password turns out to be suitable, it is okay for the pa…

> At no time is the plain text password stored anywhere Hopefully. When I see rules limiting passwords to 16 characters and disallowing SQL special characters, I'm having doubts.

My bank limits passwords at 15 characters. The best thing? There is no verification, it just cuts off. Have fun figuring out why you can not login anymore.

Re: The Correct Way to Validate Email Addresses

#278

Post by kasparsklavins » Thu, Sep 08, 2016, 12:58 PM UTC

girvo wrote: post #116

lstamour wrote: post #66

Earlier quoted context omitted.

Sadly at some places it's intentional so your password matches existing PIN systems or is "easy to remember" so you're not as easily locked out of your account. I wish in those cases there was a check box that says, "I know what I'm doing, leave me alone." Speaking of annoying validations, my name has a hyphen in it but you'd be surprised how often that's rejected with the familiar, "Please enter a valid last name."…

My partner has a hyphenated first and last name. So many systems refuse to accept that, and even regular humans struggle with understanding it!

Had this bug in a system I was working on. Pushed a patch in april.. and there still is an ongoing discussion whether it should be merged or not.

Re: The Correct Way to Validate Email Addresses

#279

Post by ihaveajob » Thu, Sep 08, 2016, 12:59 PM UTC

brianwawok wrote: post #166

pwg wrote: post #61

Earlier quoted context omitted.

Caring what characters are in the password heavily implies that the site is not hashing the plaintext password in any way, and scarier still, may just be storing the plaintext password as plain text. Why: Because if they were (at least) hashing it the output from the hash would be a binary string in which case they would have to be 8-bit clean through to the DB column where the hash output resided, and then there wou…

> Caring what characters are in the password heavily implies that the site is not hashing the plaintext password in any way, and scarier still, may just be storing the plaintext password as plain text. I don't think that is true at all. I may very well want to put a few simple rules I validate serverside, such as 1) No username in password 2) No email in password 3) No list of 100 most common passwords in password Al…

There's a related issue that I've seen in some banks, which prevents you from using passwords LONGER than a certain number of characters (usually 8 or 10). The only reason I can think this is happening is because they don't hash them and they need to fit in their database column. I took my money out of that bank the next day.

Re: The Correct Way to Validate Email Addresses

#280

Post by jones1618 » Thu, Sep 08, 2016, 1:01 PM UTC

chiefalchemist wrote: post #29

Great article. Wrong. Question. The better question: In it's current form, is an email addresses really the best way to do what it is that's trying to be accomplished? (Hint: It's a fax machine.) I mean, if I have have a phone number, why can't I have an email number? Okay, perhaps not the greatest example. But then again, if a phone number can be switched from one carrier to another, in the second decade of the 21st…

> Why can't I have an email number?

You can if you want (it is a valid email). But, if everyone had a numeric email it would be a lot harder to catch spam or know if you are sending to a bot. Sure, spammers can masquerade as a friendly name like yourmom313@domain.com now but it's easier to spot names you know than 471871731@domain.com.

> Why can't emails be carried over to other domains?

I like the idea in abstract. Practically speaking, though, this would not only require millions of emails to be purged from every ISP in the world it would also require some kind of creepy global email registry. That defies the decentralized spirit of the Web and would instantly become the Holy Grail of hacker targets.