In search of the perfect URL validation regex (2010) • Hacker News Board

Re: In search of the perfect URL validation regex (2010)

#11

Post by powersnail » Sat, Sep 25, 2021, 6:34 PM UTC

chrismorgan wrote: post #10

elif wrote: post #3

Earlier quoted context omitted.

how would you even "parse" a url with a regex? dynamically defined named subpatterns for each url parameter? I think the best i could do on paper with a regex is say "yup this is a url" or maybe "yup i can count the number of params" Unless it was a specific url with specific params?

Match groups so you can split it up into scheme, username, password, host, port, path, query, fragment. Not difficult to approximate, though for best results with diverse schemes you’d want an engine that allows repeated named groups, and I don’t know if any do (JavaScript and Python don’t).

Python's `regex` package does allow repeated named group.

Re: In search of the perfect URL validation regex (2010)

#12

Post by Sephr » Sat, Sep 25, 2021, 6:43 PM UTC

> Assume that this regex will be used for a public URL shortener written in PHP, so URLs like http://localhost/, //foo.bar/, ://foo.bar/, data:text/plain;charset=utf-8,OHAI and tel:+1234567890 shouldn’t pass (even though they’re technically valid)

At Transcend, we need to allow site owners to regulate any arbitrary network traffic, so our data flow input UI¹ was designed to detect all valid hosts (including local hosts, IDN, IPv6 literal addresses, etc) and URLs (host-relative, protocol-relative, and absolute). If the site owner inputs content that is not a valid host or URL, then we treat their input as a regex.

I came up with these simple utilities built on top of the URL interface standard² to detect all valid hosts & URLs:

• isValidHost: https://gist.github.com/eligrey/6549ad0a635fa07749238911b429...

Example valid inputs:

  host.example
  はじめよう.みんな (IDN domain; xn--p8j9a0d9c9a.xn--q9jyb4c)
  [::1] (IPv6 address)
  0xdeadbeef (IPv4 address; 222.173.190.239)
  123.456 (IPv4 address; 123.0.1.200)
  123456789 (IPv4 address; 7.91.205.21)
  localhost

• isValidURL (and isValidAbsoluteURL): https://gist.github.com/eligrey/443d51fab55864005ffb3873204b...

Example valid inputs to isValidURL:

  https://absolute-url.example
  //relative-protocol.example
  /relative-path-example

1. https://docs.transcend.io/docs/configuring-data-flows

2. https://developer.mozilla.org/en-US/docs/Web/API/URL

Re: In search of the perfect URL validation regex (2010)

#13

Post by gregsadetsky » Sat, Sep 25, 2021, 6:43 PM UTC

I was just struggling with this -- specifically, our users' "UX" expectation that entering "example.com" should work when asked for their website URL.

Most URL validation rules/regex/librairies/etc. reject "example.com". However, if you head over to Stripe (for example), in the account settings, when asked for your company's URL, Stripe will accept "example.com", and assume "http://" as the prefix (which yes, can have its own problems)

What's a good solution? I both want to validate URLs, but also let users enter "example.com". But if I simply do

    if(validateURL(url)) {
      return true;
    } else if(validateURL("http://" + url)) {
      return true;
    } else {
      return false;
    }

i.e. validate the given URL, and as a fallback, try to validate "http://" + the given url, that opens the door to weird, non-URLs strings being incorrectly validated...

Help :-)

Re: In search of the perfect URL validation regex (2010)

#14

Post by codetrotter » Sat, Sep 25, 2021, 7:00 PM UTC

mercora wrote: post #4

mpeg wrote: post #2

I was once failed on a technical interview, partly because on the coding test I was asked to write a url parser "from scratch, the way a browser would do it" and I explained it would take way too long to account for every edge case in the URL RFC but that I could do a quick and dirty approach for common urls. After I did this, the interviewer stopped me and told me in a negative way that he expected me to use a regex…

its not very likely this is whats happening here but i feel like this could be done on purpose to see how you act in this kind of situation. it kinda tells how you would act once you inevitably go into a conflict with colleagues arguing over stuff like that.

In that case I think the proper response should be: “I am very sure that browsers don’t do it that way. But let’s have a look.” And then pull up the source code for Chromium and Firefox. Assuming it’s not whiteboard only.

And if they still insist even after the source of Chromium and FF has been consulted. Well then it’s time to leave. Don’t want to work with anyone like that.

Re: In search of the perfect URL validation regex (2010)

#15

Post by tyingq » Sat, Sep 25, 2021, 7:05 PM UTC

elif wrote: post #3

mpeg wrote: post #2

I was once failed on a technical interview, partly because on the coding test I was asked to write a url parser "from scratch, the way a browser would do it" and I explained it would take way too long to account for every edge case in the URL RFC but that I could do a quick and dirty approach for common urls. After I did this, the interviewer stopped me and told me in a negative way that he expected me to use a regex…

how would you even "parse" a url with a regex? dynamically defined named subpatterns for each url parameter? I think the best i could do on paper with a regex is say "yup this is a url" or maybe "yup i can count the number of params" Unless it was a specific url with specific params?

I assume they meant "some regex implementation, including replace and/or match groups".

Like, for just the params part (yes, broken and simplistic):

  #!/usr/bin/perl
  $_="a=b&c=d&e=f&whatever=some thing";
  while (s/^([^&]*)=([^&]*)(&|$)//) {
    print "[$1] [$2]\n";
  }

Re: In search of the perfect URL validation regex (2010)

#16

Post by maciejgryka » Sat, Sep 25, 2021, 7:09 PM UTC

Using https://regex.help/, I got this beauty which passes all the ones, which should pass. Obviously some room for improvement ;) But it works!

  ^(?:http(?:(?:://(?:(?:(?:code\.google\.com/events/#&product=browser|\-\.~_!\$&'\(\)\*\+,;=:%40:80%2f::::::@ex\.com|foo\.(?:bar/\?q=Test%20URL\-encoded%20stuff|com/(?:\(something\)\?after=parens|unicode_\(\)_in_parens|b_(?:\(wiki\)(?:_blah)?#cite\-1|b(?:_\(wiki\)_\(again\)|/))))|uid(?::password@ex\.com(?::8080)?/|@ex\.com(?::8080)?/)|www\.ex\.com/wpstyle/\?p=364|223\.255\.255\.254|उदाहरण\.परीक्षा|1(?:42\.42\.1\.1/|337\.net)|مثال\.إختبار|df\.ws/123|a\.b\-c\.de|\.ws/䨹|⌘\.ws/|例子\.测试|j\.mp)|142\.42\.1\.1:8080/)|\.damowmow\.com/)|s://(?:www\.ex\.com/foo/\?bar=baz&inga=42&quux|foo_bar\.ex\.com/))|://(?:uid(?::password@ex\.com(?::8080)?|@ex\.com(?::8080)?)|foo\.com/b_b(?:_\(wiki\))?|⌘\.ws))|ftp://foo\.bar/baz)$

I had to replace some words with shorter ones to squeeze under 1000 char limit and there's no way to provide negative examples right now. Something to fix!

Re: In search of the perfect URL validation regex (2010)

#17

Post by Rebelgecko » Sat, Sep 25, 2021, 7:10 PM UTC

gregsadetsky wrote: post #13

I was just struggling with this -- specifically, our users' "UX" expectation that entering "example.com" should work when asked for their website URL. Most URL validation rules/regex/librairies/etc. reject "example.com". However, if you head over to Stripe (for example), in the account settings, when asked for your company's URL, Stripe will accept "example.com", and assume " http:// " as the prefix (which yes, can h…

This could potentially be abused, but you could actually try to resolve the DNS to determine if it's valid (could be weird for some cases like localhost or IP addresses). Or just do a "curl https://whatever.com" and see what happens (assuming that all of the websites are running a webserver, although idk if that is true in your situation)

Re: In search of the perfect URL validation regex (2010)

#18

Post by dilatedmind » Sat, Sep 25, 2021, 7:14 PM UTC

gregsadetsky wrote: post #13

I was just struggling with this -- specifically, our users' "UX" expectation that entering "example.com" should work when asked for their website URL. Most URL validation rules/regex/librairies/etc. reject "example.com". However, if you head over to Stripe (for example), in the account settings, when asked for your company's URL, Stripe will accept "example.com", and assume " http:// " as the prefix (which yes, can h…

i would suggest bias your implementation against false negatives. They can always come back and update it if it's wrong, and their url could just as easily be "valid" but incorrect, eg any typo in a domain name.

if it's really important, you could try making a request to the url and see if it loads, but that still doesn't validate its the url they intended to input.

might be cool to load the url with puppeteer and capture a screenshot of the page. if they can't recognize their own website, it's on them.

Re: In search of the perfect URL validation regex (2010)

#19

Post by dang » Sat, Sep 25, 2021, 7:16 PM UTC

Two past discussions, for the curious:

In search of the perfect URL validation regex - https://news.ycombinator.com/item?id=10019795 - Aug 2015 (77 comments)

In search of the perfect URL validation regex - https://news.ycombinator.com/item?id=7928968 - June 2014 (81 comments)

Re: In search of the perfect URL validation regex (2010)

#20

Post by mercora » Sat, Sep 25, 2021, 7:21 PM UTC

Sephr wrote: post #12

> Assume that this regex will be used for a public URL shortener written in PHP, so URLs like http://localhost/ , //foo.bar/, ://foo.bar/, data:text/plain;charset=utf-8,OHAI and tel:+1234567890 shouldn’t pass (even though they’re technically valid) At Transcend, we need to allow site owners to regulate any arbitrary network traffic, so our data flow input UI¹ was designed to detect all valid hosts (including local ho…

while not terribly important or outright not required this fails (treats urls as regex) for link-local addresses with device identifier (zone-id) applied like "[fe80::8caa:8cff:fe80:ff32%eth0]" although that would need to be fixed in the standard if its desired :)

i've found some reasoning[0] as to why its not supported with browsers in mind though.

[0] https://www.w3.org/Bugs/Public/show_bug.cgi?id=27234#c2