URLs: It's Complicated

> making this is a valid URL: https://!$%:)(*&^@www.netmeister.org/blog/urls.html

Uh, no. "%:)" is not nor is % allowed outside of that. (Although your browser will likely accept it)

> This includes spaces, and the following two URLs lead to the same file located in a directory that's named " ": > https://www.netmeister.org/blog/urls/ /f > https://www.netmeister.org/blog/urls/%20/f > Your client may automatically percent-encode the space, but e.g., curl(1) lets you send the raw space:

Uh, no. Just because one of your clients is wrong and some servers allow it doesn't mean it's allowed by the spec.

In fact, the HTTP/1.1 RFC defers to RFC2396 for the meaning of : which begin with a /.

What is ? A bunch of slash-delimited s.

What is ? A bunch of and maybe a semicolon.

What is ? , , or some special characters (not including space).

What is ? Letters, digits, and some special characters (not including space).

What is ? .

Most HTTP clients and servers are pretty forgiving about what they accept, because other people do broken stuff, like sending them literal spaces. But that doesn't mean it's "allowed", that doesn't mean every server allows it, and that doesn't mean it's a good idea.

> That is, if your web server supports (and has enabled) user directories, and you submit a request for "~username": [it does stuff]

Uh, no. If you're using Apache, that might be true. As you mentioned, this is implementation-defined (as are all pathnames).

> Now with all of this long discussion, let's go back to that silly URL from above: ... Now this really looks like the Buffalo buffalo equivalent of a URL.

Not really.

> Now we start to play silly tricks: "⁄ ⁄www.netmeister.org" uses the fraction slash characters

You are aware that URLs predate Unicode, right? Not to mention that Unicode lookalike characters are a Unicode (or UI) problem, not a URL problem?

> The next "https" now is the hostname component of the authority: a partially qualified hostname, that relies on /etc/hosts containing an entry pointing https to the right IP address.

Or on a search domain (which could be configured locally, or through GPO on Windows, or through DHCP!). Or maybe your resolver has a local zone for it. Or maybe ...

From source code of preferred ftp/http client, maybe this is helpful. Also suggest reading source code for djb's ftp server.

   Parse URL of form (per RFC 3986):
       ://[[:]@][:][/]
 
   XXX: this is not totally RFC 3986 compliant;  will have the
   leading `/' unless it's an ftp:// URL, as this makes things easier
   for file:// and http:// URLs.  ftp:// URLs have the `/' between the
   host and the URL-path removed, but any additional leading slashes
   in the URL-path are retained (because they imply that we should
   later do "CWD" with a null argument).
 
   Examples:
        input URL                       output path
        ---------                       -----------
       "http://host"                   "/"
       "http://host/"                  "/"
       "http://host/path"              "/path"
       "file://host/dir/file"          "dir/file"
       "ftp://host"                    ""
       "ftp://host/"                   ""
       "ftp://host//"                  "/"
       "ftp://host/dir/file"           "dir/file"
       "ftp://host//dir/file"          "/dir/file"
 
    If we are dealing with a classic `[user@]host:[path]'
    (urltype is CLASSIC_URL_T) then we have a raw directory
    name (not encoded in any way) and we can change
    directories in one step.
   
    If we are dealing with an `ftp://host/path' URL
    (urltype is FTP_URL_T), then RFC 3986 says we need to
    send a separate CWD command for each unescaped "/"
    in the path, and we have to interpret %hex escaping
    *after* we find the slashes.  It's possible to get
    empty components here, (from multiple adjacent
    slashes in the path) and RFC 3986 says that we should
    still do `CWD ' (with a null argument) in such cases.
   
    Many ftp servers don't support `CWD ', so if there's an
    error performing that command, bail out with a descriptive
    message.
   
    Examples:
                 
    host:                                dir="", urltype=CLASSIC_URL_T
                 logged in (to default directory)
    host:file                            dir=NULL, urltype=CLASSIC_URL_T
                 "RETR file"
    host:dir/                            dir="dir", urltype=CLASSIC_URL_T
                 "CWD dir", logged in
    ftp://host/                          dir="", urltype=FTP_URL_T
                 logged in (to default directory)
    ftp://host/dir/                      dir="dir", urltype=FTP_URL_T
                 "CWD dir", logged in
    ftp://host/file                      dir=NULL, urltype=FTP_URL_T
                 "RETR file"
    ftp://host//file                     dir="", urltype=FTP_URL_T
                 "CWD ", "RETR file"
    host:/file                           dir="/", urltype=CLASSIC_URL_T
                 "CWD /", "RETR file"
    ftp://host///file                    dir="/", urltype=FTP_URL_T
                 "CWD ", "CWD ", "RETR file"
    ftp://host/%2F/file                  dir="%2F", urltype=FTP_URL_T
                 "CWD /", "RETR file"
    ftp://host/foo/file                  dir="foo", urltype=FTP_URL_T
                 "CWD foo", "RETR file"
    ftp://host/foo/bar/file              dir="foo/bar"
                 "CWD foo", "CWD bar", "RETR file"
    ftp://host//foo/bar/file             dir="/foo/bar"
                 "CWD ", "CWD foo", "CWD bar", "RETR file"
    ftp://host/foo//bar/file             dir="foo//bar"
                 "CWD foo", "CWD ", "CWD bar", "RETR file"
    ftp://host/%2F/foo/bar/file          dir="%2F/foo/bar"
                 "CWD /", "CWD foo", "CWD bar", "RETR file"
    ftp://host/%2Ffoo/bar/file           dir="%2Ffoo/bar"
                 "CWD /foo", "CWD bar", "RETR file"
    ftp://host/%2Ffoo%2Fbar/file         dir="%2Ffoo%2Fbar"
                 "CWD /foo/bar", "RETR file"
    ftp://host/%2Ffoo%2Fbar%2Ffile       dir=NULL
                 "RETR /foo/bar/file"
   
    Note that we don't need `dir' after this point.
   
    The `CWD ' command (without a directory), which is required by   
    RFC 3986 to support the empty directory in the URL pathname (`//'),   
    conflicts with the server's conformance to RFC 959.

> If you go by shell sematics, that pokes around the home directory of the user running the FTP daemon; hopefully that doesn't actually work.

This is why FTP servers have default directories. They're the equivalent of user home directories. By the way, many FTP servers (especially historically) map FTP logins to real, local users.

> This smells like a security vulnerability for most setups.

How do you figure? Surely your sensitive files aren't world-readable... /s

The original filename is optional in gzip. It is not included in the response sent by, for example, Apache.

(There is a mandatory MTIME which is included, and an OS byte, but those only waste 5 bytes total. Far less than gzip will typically save.)

The spec is silent on length. 2000 bytes came from some web servers (old IIS comes to mind) that capped the URL at 2K or something close to that. So extra long URLS were problematic (and a lot of early web apps went nuts with parameters). So, max length is up to the implementer. All I know is that I've had to fix lots of code where someone assumed that 255 characters is all you'll ever need for a URL.