Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In search of the perfect URL validation regex (2010)

mathiasbynens.be

Post reply on HN
Print view View original

51–60 of 67 posts

Re: In search of the perfect URL validation regex (2010)

#51

Post by gibsonf1 »

I use this:

u.checkURL = function (string) {

    if ($.type(string) === "string") {

        if (/^(https?|ftp):(\/\/(((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:)*@)?((\[(|(v[\da-f]{1,}\.(([a-z]|\d|-|\.|_|~)|[!\$&'\(\)\*\+,;=]|:)+))\])|((\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5]))|(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=])*)(:\d*)?)(\/(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|@)*)*|(\/((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|@)+(\/(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|@)*)*)?)|((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|@)+(\/(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|@)*)*)|((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|@)){0})(\?((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|@)|[\uE000-\uF8FF]|\/|\?)*)?(\#((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|@)|\/|\?)\*)?$/i.test(string)) {

            return true;

        } else {

            return false;

        }

    } else {

        return false;

    }

}

Re: In search of the perfect URL validation regex (2010)

#52

Post by tester756 »

azalemeth wrote: post #48

Honest question: there is a famous and very funny stack exchange answer on the topic of parsing html with a regex [1] that states that the problem is in general impossible and if if you find yourself doing this, something has gone wrong and you should re-evaluate your life choices / pray to Cthulu. So, does this apply to URLs? The fact that these regexes are....so huge...makes me think that something is fundamentally…

>So, does this apply to URLs? The fact that these regexes are....so huge...makes me think that something is fundamentally wrong

Yes, if your regex is above {.../50/100/...} characters, then write parser.

I struggle to understand why do people write those crazy regexes for emails, urls, html when probably in all popular technologies there are battle-tested parsers for those things.

Re: In search of the perfect URL validation regex (2010)

#53

Post by staticassertion »

jt2190 wrote: post #39
likium wrote: post #21

Even if you built a URL validation regex that follows rfc3986[1] and rfc3987[2], you will still get user bug reports because web browsers follow a different standard. For example, http://example.com./ > , http:///example.com/ > and https://en.wikipedia.org/wiki/Space (punctuation) > are classified as invalid urls in the blog, but they are accepted in the browser. As the creator of cURL puts it, there is no URL standa…

There's also a question of what we're really trying to validate, IMHO. All of these regex patterns will tell you that a string looks like a URL, but they won't actually tell you if: There's any web server listening at that particular URL; Whether that server has the resource in that location; If that server is reachable from where you want to fetch it; etc.

> All of these regex patterns will tell you that a string looks like a URL,

yeah that's it that's what they're trying to validate

Re: In search of the perfect URL validation regex (2010)

#54

Post by staticassertion »

tester756 wrote: post #52
azalemeth wrote: post #48

Honest question: there is a famous and very funny stack exchange answer on the topic of parsing html with a regex [1] that states that the problem is in general impossible and if if you find yourself doing this, something has gone wrong and you should re-evaluate your life choices / pray to Cthulu. So, does this apply to URLs? The fact that these regexes are....so huge...makes me think that something is fundamentally…

>So, does this apply to URLs? The fact that these regexes are....so huge...makes me think that something is fundamentally wrong Yes, if your regex is above {.../50/100/...} characters, then write parser. I struggle to understand why do people write those crazy regexes for emails, urls, html when probably in all popular technologies there are battle-tested parsers for those things.

Sometimes you're given an arbitrary bag of bytes with best-effort well-formed data. Regexes are gross but quite good for those cases where you need to try to rip out some bits from the data abyss.

Re: In search of the perfect URL validation regex (2010)

#55

Post by userbinator »

MildlySerious wrote: post #40
yyyk wrote: post #36

Earlier quoted context omitted.

http://example.com./ > is a valid URL, see for example: https://jdebp.uk/FGA/web-fully-qualified-domain-name.html

Tangentially, Youtube had a bug surface last year where adding that extra dot let you avoid all ads. Previous discussion[1] [1] https://news.ycombinator.com/item?id=23479435

This "bug", can definitely also be known as a feature ;-)

Re: In search of the perfect URL validation regex (2010)

#56

Post by louthy »

gibsonf1 wrote: post #51

I use this: u.checkURL = function (string) { if ($.type(string) === "string") { if (/^(https?|ftp):(\/\/(((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:)*@)?((\[(|(v[\da-f]{1,}\.(([a-z]|\d|-|\.|_|~)|[!\$&'\(\)\*\+,;=]|:)+))\])|((\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[…

    if(predicate) {
        return true;
    } else {
        return false;
    }
Can just be written;

    return predicate;
So, your code above can be:

    return &.type(string) === “string” && /regex/i.test(string);
Reg-exes like this are truly hideous though, they may as well be written in Brainfuck for all their lack of maintainability and readability.

I will never understand why regular expressions are considered the best tool for the job when it comes to parsing; they are far too terse, and do not declare intent in any way.

Software development is not just about communicating with the computer, it’s about communicating with other engineers, so we can work collaboratively. Regular expressions are the antithesis to that way of thinking

Re: In search of the perfect URL validation regex (2010)

#60

Post by [deleted] »

yyyk wrote: post #36
likium wrote: post #21

Even if you built a URL validation regex that follows rfc3986[1] and rfc3987[2], you will still get user bug reports because web browsers follow a different standard. For example, http://example.com./ > , http:///example.com/ > and https://en.wikipedia.org/wiki/Space (punctuation) > are classified as invalid urls in the blog, but they are accepted in the browser. As the creator of cURL puts it, there is no URL standa…

http://example.com./ > is a valid URL, see for example: https://jdebp.uk/FGA/web-fully-qualified-domain-name.html

[deleted]
Post reply on HN