Live data from Hacker News

URLs: It's Complicated

netmeister.org

11–20 of 41 posts

Re: URLs: It's Complicated

#11
post #2

It seems like the colon is too ambiguous (is used as a protocol delimiter, delimiter for user/pass, delimiter for port). Reminds a little bit of Java labels where you can do this: public class Labels { public static void main(String args[]){ https://hn.ycombinator.com for(int i=0; i the https: is a label named https and everything after the colon is a comment so this is valid code.

Not in URLs, but related:

    Larry's 1st Law of Language Redesign: Everyone wants the colon
    Larry's 2nd Law of Language Redesign: Larry gets the colon
https://thelackthereof.org/Perl6_Colons

Re: URLs: It's Complicated

#12
post #3

It doesn’t seem complicated at all. Complicated to me means difficult to understand. This just involves reading the spec and it all seems pretty simple and consistent. Complicated doesn’t mean “new to me.” If I haven’t read a man page, that doesn’t mean the command is complicated.

Even browser developers have made mistakes as a result of the complexity of the spec, resulting in things like CVE-2018-6128 [0] happening.

[0] https://bugs.chromium.org/p/chromium/issues/detail?id=841105

Re: URLs: It's Complicated

#13
URLs are not complicated, unless you complicate them.

foo|foo -foo 's^foo^foo^'"">foo 2>>foo

is not a very good example for teaching the structure of the the command line.

Pick a better one.

It's simple.

Re: URLs: It's Complicated

#14
All extremely useful: the overview, the examples and the comments.

A few months ago while writing a bot/crawler I searched for hours for something like this, but I found only full specs or just bits and pieces scattered around that used different terminology and/or had different opinions.

In the end I didn't even clearly understand what should be the max total URL length (e.g. mixed opinions here https://stackoverflow.com/questions/417142/what-is-the-maxim... - come on, a xGiB long URL?) => most of the time 2000 bytes is mentioned but it's not 100% clear.

Writing a bot made me understand 1) why browsers are so complicated and 2) that the Internet is a mess (e.g. once I even found a page that used multiple character encodings...).

My personal opinion is that everything is too lax. Browsers try to be the best ones by implementing workarounds for stuff that does not have (yet) or does not comply to a spec => this way it can only end up in a mess. A simple example is the HTTP-header "Content-Encoding" ( https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Co... ) which I think should only indicate what kind of compression is being used, but I keep seeing in there stuff like "utf8"/"image/jpeg"/"base64"/"8bit"/"none"/"binary"/etc... and all those pages/files work perfectly in the browsers even if with those values they should actually be rejected... .

Re: URLs: It's Complicated

#15
post #2

It seems like the colon is too ambiguous (is used as a protocol delimiter, delimiter for user/pass, delimiter for port). Reminds a little bit of Java labels where you can do this: public class Labels { public static void main(String args[]){ https://hn.ycombinator.com for(int i=0; i the https: is a label named https and everything after the colon is a comment so this is valid code.

> It seems like the colon is too ambiguous (is used as a protocol delimiter, delimiter for user/pass, delimiter for port). and because that was still too boring they came up with ipv6

IIUC the IPv6 weirdness here is simply due to very unfortunate timing: IPv6 was being finalized at a time (first half of the 90s) when the Web (and with it URLs) was already nearly frozen but still not obviously important.

Re: URLs: It's Complicated

#16
post #14

All extremely useful: the overview, the examples and the comments. A few months ago while writing a bot/crawler I searched for hours for something like this, but I found only full specs or just bits and pieces scattered around that used different terminology and/or had different opinions. In the end I didn't even clearly understand what should be the max total URL length (e.g. mixed opinions here https://stackoverflo…

The use of Content-Encoding for compression is actually something of a historical wart: what was intended to be used for that purpose is Transfer-Encoding, but modern browsers don’t even send the TE header necessary to permit the HTTP server to use it (except for Transfer-Encoding: chunked which every HTTP 1.1 client must accept), even though some servers are perfectly capable of it and all but the most broken will at least ignore it. Things like 7bit, 8bit, binary, or quoted-printable are not supposed to be in the HTTP Content-Encoding header, either, but their presence is at least somewhat understandable as they are valid in the MIME Content-Transfer-Encoding header, and HTTP originally shares much of its infrastructure with MIME (think Content-Disposition: attachment).

I guess what I’m getting at here is that the blame for the C-E weirdness lies in large part on the browsers, which could’ve made a clean break and improved the semantics at the same time by using T-E, but instead chose to initiate a chicken-and-egg dilemma out of a desire to support broken HTTP servers from the last century.

(The intended semantics is that C-E, an “end-to-end” header, says “this resource genuinely exists in this encoded form”, while T-E, a “hop-to-hop” header, says “the origin or proxy server you’re using incidentally chose to encode this resource in this form”; this is why sometimes the wrong combination of hacks in the HTTP server and the Web browser will lead you to downloading a tar file when you expected a tar.gz file.)

The use of “gzip” as the compression is also a wart, because it’s “deflate” (which is what you want: DEFLATE compression with a checksum) with a useless decompressed filename (wat?) + decompressed mtime (double wat?) header stacked on top.

Re: URLs: It's Complicated

#17
Just to share a little more of the weirdness (discovered while reading a couple of the historical URL & URI RFCs several days ago):

Per the original spec, in FTP URLs,

- ftp://example.net/foo/bar will get you bar inside the foo directory inside the default directory of the FTP server at example.net (i.e. CWD foo, RETR bar);

- ftp://example.net//foo/bar will get you bar inside the foo directory inside the empty string directory inside the default directory of the FTP server at example.net (i.e. CWD, CWD foo, RETR bar; what do FTP servers even do with this?);

- and it’s ftp://example.net/%2Ffoo/bar that you must use if you want bar inside the foo directory inside the root directory of the FTP server at example.net (i.e. CWD /foo, RETR bar; %2F being the result of percent-encoding a slash character).

Re: URLs: It's Complicated

#19
post #3

It doesn’t seem complicated at all. Complicated to me means difficult to understand. This just involves reading the spec and it all seems pretty simple and consistent. Complicated doesn’t mean “new to me.” If I haven’t read a man page, that doesn’t mean the command is complicated.

If we're being sticklers for reading the docs... > com·pli·cat·ed | ˈkämpləˌkādəd | > adjective > 1 consisting of many interconnecting parts or elements

[deleted]

Re: URLs: It's Complicated

#20

Just to share a little more of the weirdness (discovered while reading a couple of the historical URL & URI RFCs several days ago): Per the original spec, in FTP URLs, - ftp://example.net/foo/bar will get you bar inside the foo directory inside the default directory of the FTP server at example.net ( i.e. CWD foo, RETR bar); - ftp://example.net//foo/bar will get you bar inside the foo directory inside the empty strin…

> what do FTP servers even do with this?

Pretty sure CWD by itself isn't even valid (at least RFC959 assumes it has an argument), and therefore // isn't valid in FTP URLs.

The %2Ffoo/bar is needed because of the fact that FTP CWD and RETR paths are system dependent (with, theoretically, system dependent path separators), but URLs are not, so the FTP client breaks the URL on / and sequentially executes CWD down the tree so that it doesn't need to know what it's connected to.

In other words: URL paths are not system paths, and it's a mistake to think of them as such.

(Alternate in other words: FTP is awful)

Post reply on HN