Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In search of the perfect URL validation regex (2010)

mathiasbynens.be

Post reply on HN
Print view View original

21–30 of 67 posts

Re: In search of the perfect URL validation regex (2010)

#21

Post by likium »

Even if you built a URL validation regex that follows rfc3986[1] and rfc3987[2], you will still get user bug reports because web browsers follow a different standard.

For example, http://example.com./> , http:///example.com/> and https://en.wikipedia.org/wiki/Space (punctuation)> are classified as invalid urls in the blog, but they are accepted in the browser.

As the creator of cURL puts it, there is no URL standard[3].

[1]: https://www.ietf.org/rfc/rfc3986.txt

[2]: https://www.ietf.org/rfc/rfc3987.txt

[3]: https://daniel.haxx.se/blog/2016/05/11/my-url-isnt-your-url/

Re: In search of the perfect URL validation regex (2010)

#22

Post by lelandfe »

dmurray wrote: post #7

> I also don’t want to allow every possible technically valid URL — quite the opposite. Well, that should make things a lot easier. What does he mean here? The rest of the text doesn't make it clear to me, unless it's meant to be "every possibly valid HTTP, HTTPS, or FTP URL" which isn't exactly "the opposite".

The next paragraph might be that clarification, although I agree it isn't totally clear what he meant there:

> Assume that this regex will be used for a public URL shortener written in PHP, so URLs like http://localhost/, //foo.bar/, ://foo.bar/, data:text/plain;charset=utf-8,OHAI and tel:+1234567890 shouldn’t pass (even though they’re technically valid). Also, in this case I only want to allow the HTTP, HTTPS and FTP protocols.

Re: In search of the perfect URL validation regex (2010)

#23

Post by lelandfe »

gregsadetsky wrote: post #13

I was just struggling with this -- specifically, our users' "UX" expectation that entering "example.com" should work when asked for their website URL. Most URL validation rules/regex/librairies/etc. reject "example.com". However, if you head over to Stripe (for example), in the account settings, when asked for your company's URL, Stripe will accept "example.com", and assume " http:// " as the prefix (which yes, can h…

Address validators for online checkout are notoriously inaccurate, though they still help a lot. You just have to prompt the user, "Did you mean 123 Example St?"

I'd probably do the same for poorly formatted URLs. When the user hits Submit, a prompt appears saying, "Did you mean `https://example.com`?"

Re: In search of the perfect URL validation regex (2010)

#24

Post by Sephr »

likium wrote: post #21

Even if you built a URL validation regex that follows rfc3986[1] and rfc3987[2], you will still get user bug reports because web browsers follow a different standard. For example, http://example.com./ > , http:///example.com/ > and https://en.wikipedia.org/wiki/Space (punctuation) > are classified as invalid urls in the blog, but they are accepted in the browser. As the creator of cURL puts it, there is no URL standa…

There might not have been a generally accepted standard then, but there is now: https://url.spec.whatwg.org/

Re: In search of the perfect URL validation regex (2010)

#25

Post by axiosgunnar »

maciejgryka wrote: post #16

Using https://regex.help/ , I got this beauty which passes all the ones, which should pass. Obviously some room for improvement ;) But it works! ^(?:http(?:(?:://(?:(?:(?:code\.google\.com/events/#&product=browser|\-\.~_!\$&'\(\)\*\+,;=:%40:80%2f::::::@ex\.com|foo\.(?:bar/\?q=Test%20URL\-encoded%20stuff|com/(?:\(something\)\?after=parens|unicode_\(\)_in_parens|b_(?:\(wiki\)(?:_blah)?#cite\-1|b(?:_\(wiki\)_\(again\)|/…

> ⌘\.ws

I guess this is the regex equivalent of overfitting :)

Re: In search of the perfect URL validation regex (2010)

#26

Post by saghm »

axiosgunnar wrote: post #25
maciejgryka wrote: post #16

Using https://regex.help/ , I got this beauty which passes all the ones, which should pass. Obviously some room for improvement ;) But it works! ^(?:http(?:(?:://(?:(?:(?:code\.google\.com/events/#&product=browser|\-\.~_!\$&'\(\)\*\+,;=:%40:80%2f::::::@ex\.com|foo\.(?:bar/\?q=Test%20URL\-encoded%20stuff|com/(?:\(something\)\?after=parens|unicode_\(\)_in_parens|b_(?:\(wiki\)(?:_blah)?#cite\-1|b(?:_\(wiki\)_\(again\)|/…

> ⌘\.ws I guess this is the regex equivalent of overfitting :)

Yeah, not to mention "code.google.com" being right in there!

Re: In search of the perfect URL validation regex (2010)

#27

Post by axiosgunnar »

mpeg wrote: post #2

I was once failed on a technical interview, partly because on the coding test I was asked to write a url parser "from scratch, the way a browser would do it" and I explained it would take way too long to account for every edge case in the URL RFC but that I could do a quick and dirty approach for common urls. After I did this, the interviewer stopped me and told me in a negative way that he expected me to use a regex…

How do browsers parse URLs then?

Re: In search of the perfect URL validation regex (2010)

#28

Post by axiosgunnar »

codetrotter wrote: post #14
mercora wrote: post #4

Earlier quoted context omitted.

its not very likely this is whats happening here but i feel like this could be done on purpose to see how you act in this kind of situation. it kinda tells how you would act once you inevitably go into a conflict with colleagues arguing over stuff like that.

In that case I think the proper response should be: “I am very sure that browsers don’t do it that way. But let’s have a look.” And then pull up the source code for Chromium and Firefox. Assuming it’s not whiteboard only. And if they still insist even after the source of Chromium and FF has been consulted. Well then it’s time to leave. Don’t want to work with anyone like that.

How do browsers parse URLs then?

Re: In search of the perfect URL validation regex (2010)

#29

Post by maciejgryka »

axiosgunnar wrote: post #25
maciejgryka wrote: post #16

Using https://regex.help/ , I got this beauty which passes all the ones, which should pass. Obviously some room for improvement ;) But it works! ^(?:http(?:(?:://(?:(?:(?:code\.google\.com/events/#&product=browser|\-\.~_!\$&'\(\)\*\+,;=:%40:80%2f::::::@ex\.com|foo\.(?:bar/\?q=Test%20URL\-encoded%20stuff|com/(?:\(something\)\?after=parens|unicode_\(\)_in_parens|b_(?:\(wiki\)(?:_blah)?#cite\-1|b(?:_\(wiki\)_\(again\)|/…

> ⌘\.ws I guess this is the regex equivalent of overfitting :)

Yeah, grex (the library powering this) is really cool, but doesn’t generalize very well. I’m sure there are ways to improve it, but it’s not a trivial thing to do.

Re: In search of the perfect URL validation regex (2010)

#30

Post by Sephr »

axiosgunnar wrote: post #28
codetrotter wrote: post #14

Earlier quoted context omitted.

In that case I think the proper response should be: “I am very sure that browsers don’t do it that way. But let’s have a look.” And then pull up the source code for Chromium and Firefox. Assuming it’s not whiteboard only. And if they still insist even after the source of Chromium and FF has been consulted. Well then it’s time to leave. Don’t want to work with anyone like that.

How do browsers parse URLs then?

See https://chromium.googlesource.com/chromium/src/+/HEAD/url/#c...
Post reply on HN