In search of the perfect URL validation regex (2010) • Hacker News Board

Re: In search of the perfect URL validation regex (2010)

#41

Post by JadeNB » Sat, Sep 25, 2021, 9:31 PM UTC

anderskaseorg wrote: post #32

gregsadetsky wrote: post #13

I was just struggling with this -- specifically, our users' "UX" expectation that entering "example.com" should work when asked for their website URL. Most URL validation rules/regex/librairies/etc. reject "example.com". However, if you head over to Stripe (for example), in the account settings, when asked for your company's URL, Stripe will accept "example.com", and assume " http:// " as the prefix (which yes, can h…

Parse, don’t validate. If you need a heuristic that accepts non-URL strings as if they were valid URLs, you should convert those non-URL strings to valid URLs so the rest of your code can just deal with valid URLs. if (validateURL(url)) { return url; } else if (validateURL("http://" + url)) { return "http://" + url; } else { return null; }

I know we're not golfing, but it pains me to see that repetition in the middle. Mightn't we write

    if (!validateURL(url)) {
        url = "http://" + url;
        if (!validateURL(url)) {
            url = null;
        }
    }
    return url;

to snip a small probability of a bug?

Re: In search of the perfect URL validation regex (2010)

#42

Post by wolfgang42 » Sat, Sep 25, 2021, 9:35 PM UTC

axiosgunnar wrote: post #27

mpeg wrote: post #2

I was once failed on a technical interview, partly because on the coding test I was asked to write a url parser "from scratch, the way a browser would do it" and I explained it would take way too long to account for every edge case in the URL RFC but that I could do a quick and dirty approach for common urls. After I did this, the interviewer stopped me and told me in a negative way that he expected me to use a regex…

How do browsers parse URLs then?

Here’s a polyfill for the JS URL() interface which should give you a taste: https://github.com/zloirock/core-js/blob/272ac1b4515c5cfbf34... (I tried finding the one in Firefox but I couldn’t actually work out where it started, this one is much easier to follow)

TLDR: it’s a traditional parser—a big state machine that steps through the URL character by character and tokenizes it into the relevant pieces.

Re: In search of the perfect URL validation regex (2010)

#43

Post by JadeNB » Sat, Sep 25, 2021, 9:35 PM UTC

MaxBarraclough wrote: post #37

queuebert wrote: post #34

Uh oh, Regex is approaching sentience.

Every known sentient being is a finite state machine. Every finite state machine corresponds to a regular expression, and vice versa.

> Every known sentient being is a finite state machine.

I know this is just a cutesy slogan, but how could you possibly know whether a living creature is a finite state machine? What would it even mean? I know I don't respond identically to identical stimuli presented on different occasions ….

Re: In search of the perfect URL validation regex (2010)

#44

Post by dhsysusbsjsi » Sat, Sep 25, 2021, 9:41 PM UTC

MildlySerious wrote: post #40

yyyk wrote: post #36

Earlier quoted context omitted.

http://example.com./ > is a valid URL, see for example: https://jdebp.uk/FGA/web-fully-qualified-domain-name.html

Tangentially, Youtube had a bug surface last year where adding that extra dot let you avoid all ads. Previous discussion[1] [1] https://news.ycombinator.com/item?id=23479435

Also nearly every paywalled media site

Re: In search of the perfect URL validation regex (2010)

#45

Post by wolfgang42 » Sat, Sep 25, 2021, 9:43 PM UTC

JadeNB wrote: post #41

anderskaseorg wrote: post #32

Earlier quoted context omitted.

Parse, don’t validate. If you need a heuristic that accepts non-URL strings as if they were valid URLs, you should convert those non-URL strings to valid URLs so the rest of your code can just deal with valid URLs. if (validateURL(url)) { return url; } else if (validateURL("http://" + url)) { return "http://" + url; } else { return null; }

I know we're not golfing, but it pains me to see that repetition in the middle. Mightn't we write if (!validateURL(url)) { url = "http://" + url; if (!validateURL(url)) { url = null; } } return url; to snip a small probability of a bug?

I find that branchiness (and mutation of the variable) harder to follow. Personally, I’d just take “parse, don’t validate” to its logical conclusion and go for:

    const parseUrl = url => validateUrl(url) ? url : null;
    return parseUrl(url) || parseUrl('http://'+url) || null;

Re: In search of the perfect URL validation regex (2010)

#46

Post by throwamon » Sat, Sep 25, 2021, 9:47 PM UTC

JadeNB wrote: post #43

MaxBarraclough wrote: post #37

Earlier quoted context omitted.

Every known sentient being is a finite state machine. Every finite state machine corresponds to a regular expression, and vice versa.

> Every known sentient being is a finite state machine. I know this is just a cutesy slogan, but how could you possibly know whether a living creature is a finite state machine? What would it even mean? I know I don't respond identically to identical stimuli presented on different occasions ….

Obnoxious, I mean, trivial, answer: Just make "occasions" a variable. Assuming your lifetime is finite, you could simply assign each point in time to a value, and there you have it: a finite mapping from each moment to a state.

Re: In search of the perfect URL validation regex (2010)

#47

Post by MaxBarraclough » Sat, Sep 25, 2021, 10:09 PM UTC

JadeNB wrote: post #43

MaxBarraclough wrote: post #37

Earlier quoted context omitted.

Every known sentient being is a finite state machine. Every finite state machine corresponds to a regular expression, and vice versa.

> Every known sentient being is a finite state machine. I know this is just a cutesy slogan, but how could you possibly know whether a living creature is a finite state machine? What would it even mean? I know I don't respond identically to identical stimuli presented on different occasions ….

> I know this is just a cutesy slogan

Mostly, yes, but I do think there's a real point here as well.

> how could you possibly know whether a living creature is a finite state machine?

As I understand it, physicists don't really know whether the physical world has a finite number of states, or an infinite number. I think they tend to lean toward finite, though.

Even if it's infinite, I doubt it's of consequence. That is to say, I doubt that sentience depends on the physical possibility of an infinite number of states. (Of course, if it turns out the physical world only has a finite number of states, that demonstrates that sentience is compatible with the finite-states constraint.)

> What would it even mean?

Systems can be modelled as finite state machines. Sentient entities like people are extremely sophisticated systems, but that's just a matter of degree, not of category.

> I know I don't respond identically to identical stimuli presented on different occasions

Right, because you're in a different state. You'll never be in the same state twice. We don't need to resort to non-determinism.

Re: In search of the perfect URL validation regex (2010)

#48

Post by azalemeth » Sat, Sep 25, 2021, 10:46 PM UTC

Honest question: there is a famous and very funny stack exchange answer on the topic of parsing html with a regex [1] that states that the problem is in general impossible and if if you find yourself doing this, something has gone wrong and you should re-evaluate your life choices / pray to Cthulu.

So, does this apply to URLs? The fact that these regexes are....so huge...makes me think that something is fundamentally wrong. Are URLs describable in a Chomsky Type 3 grammar? Are they sufficiently regular that using a Regex is sensible? What do the actual browsers do?

[1] https://stackoverflow.com/questions/1732348/regex-match-open...

Re: In search of the perfect URL validation regex (2010)

#49

Post by MathMonkeyMan » Sat, Sep 25, 2021, 11:19 PM UTC

azalemeth wrote: post #48

Honest question: there is a famous and very funny stack exchange answer on the topic of parsing html with a regex [1] that states that the problem is in general impossible and if if you find yourself doing this, something has gone wrong and you should re-evaluate your life choices / pray to Cthulu. So, does this apply to URLs? The fact that these regexes are....so huge...makes me think that something is fundamentally…

I haven't looked at the BNF(s) for URIs lately, but I don't recall there being any recursion, so I wouldn't be surprised if the language were regular.

There was a Perl program that would take something like a BNF and barf out a gigantic regex (maybe with some maximum depth).

Re: In search of the perfect URL validation regex (2010)

#50

Post by aranchelk » Sat, Sep 25, 2021, 11:36 PM UTC

azalemeth wrote: post #48

Honest question: there is a famous and very funny stack exchange answer on the topic of parsing html with a regex [1] that states that the problem is in general impossible and if if you find yourself doing this, something has gone wrong and you should re-evaluate your life choices / pray to Cthulu. So, does this apply to URLs? The fact that these regexes are....so huge...makes me think that something is fundamentally…

Caveats: I know nothing of Chomsky Grammars, and I have only a passing familiarity with Cthulu, but IMO the real crux of the issue parsing html with regex (beyond all the “it’s hard”, “the spec is more complicated than you think”, “regex is impossible to read” etc.) is html is a recursive data structure, e.g. you can have a div, inside a div, inside a div ad infinitum. Regex, AFAIK, doesn’t allow you to describe recursion, so you’re left with regex plus supporting code. You’ll then have an impedance mismatch between the two.

URLs are not recursive structures, so I’d say the single hardest feature of html is not present.