Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

URLs: It's Complicated

netmeister.org

Post reply on HN
Print view View original

31–40 of 41 posts

Re: URLs: It's Complicated

#31

Post by mananaysiempre »

jfrunyon wrote: post #20
mananaysiempre wrote: post #17

Just to share a little more of the weirdness (discovered while reading a couple of the historical URL & URI RFCs several days ago): Per the original spec, in FTP URLs, - ftp://example.net/foo/bar will get you bar inside the foo directory inside the default directory of the FTP server at example.net ( i.e. CWD foo, RETR bar); - ftp://example.net//foo/bar will get you bar inside the foo directory inside the empty strin…

> what do FTP servers even do with this? Pretty sure CWD by itself isn't even valid (at least RFC959 assumes it has an argument), and therefore // isn't valid in FTP URLs. The %2Ffoo/bar is needed because of the fact that FTP CWD and RETR paths are system dependent (with, theoretically, system dependent path separators), but URLs are not, so the FTP client breaks the URL on / and sequentially executes CWD down the tr…

> Pretty sure CWD by itself isn’t even valid [...] and therefore // isn’t valid in FTP URLs.

So, I looked it up carefully and it appears that (despite the promises in later RFCs such as 2396 and 3986) the current specification of the ftp scheme is still the ancient RFC 1738 which predates not only the URL / URI distinction but even the notion of relative URLs. In §3.2.2 https://tools.ietf.org/html/rfc1738#section-3.2.2> it specifically says that a null segment in the path should result in a “CWD ” command (i.e. CWD, space, null string argument) being sent to the FTP server, going against both the current RFC 959 and its predecessor 765 (apparently the earliest formal specification of FTP to include CWD) which require the argument to CWD to be non-null.

Thus apparently a conformant implementation of the ftp URL scheme cannot be a conformant implementation of an FTP client. Joy.

It still seems unlikely that Berners-Lee et al. would specifically call this case out if it were useless at the time... What were the servers that made this necessary, I wonder?

> FTP CWD and RETR paths are system dependent (with, theoretically, system dependent path separators), but URLs are not

Thank you, that’s the insight that I was missing. So a %2F inside an ftp URL component is just performing a (sanctioned) injection of the (supposedly UNIXy) server path syntax.

> FTP is awful

I’d go with “unbelievably ancient, with the attendant problems”, but yes. Funny how it still manages to be better than everything else (that I know) at transferring files by not multiplexing control and data onto the same TCP connection. (I think HTTP over QUIC can do this as well?)

Re: URLs: It's Complicated

#32

Post by chungy »

mananaysiempre wrote: post #15
jackewiehose wrote: post #8

Earlier quoted context omitted.

> It seems like the colon is too ambiguous (is used as a protocol delimiter, delimiter for user/pass, delimiter for port). and because that was still too boring they came up with ipv6

IIUC the IPv6 weirdness here is simply due to very unfortunate timing: IPv6 was being finalized at a time (first half of the 90s) when the Web (and with it URLs) was already nearly frozen but still not obviously important.

The colons also make IPv6 addresses unambiguous to IPv4 notation.

Re: URLs: It's Complicated

#33

Post by nayuki »

mananaysiempre wrote: post #16
zepearl wrote: post #14

All extremely useful: the overview, the examples and the comments. A few months ago while writing a bot/crawler I searched for hours for something like this, but I found only full specs or just bits and pieces scattered around that used different terminology and/or had different opinions. In the end I didn't even clearly understand what should be the max total URL length (e.g. mixed opinions here https://stackoverflo…

The use of Content-Encoding for compression is actually something of a historical wart: what was intended to be used for that purpose is Transfer-Encoding, but modern browsers don’t even send the TE header necessary to permit the HTTP server to use it (except for Transfer-Encoding: chunked which every HTTP 1.1 client must accept), even though some servers are perfectly capable of it and all but the most broken will a…

Even though HTTP DEFLATE saves ~20 bytes compared to GZIP, it itself is a wart because of some vendor misunderstandings. HTTP DEFLATE is actually DEFLATE data wrapped in a zlib container, not raw DEFLATE. See https://en.wikipedia.org/wiki/HTTP_compression#Problems_prev... ; https://stackoverflow.com/questions/3932117/handling-http-co...

Re: URLs: It's Complicated

#34

Post by masklinn »

indymike wrote: post #29
zepearl wrote: post #14

All extremely useful: the overview, the examples and the comments. A few months ago while writing a bot/crawler I searched for hours for something like this, but I found only full specs or just bits and pieces scattered around that used different terminology and/or had different opinions. In the end I didn't even clearly understand what should be the max total URL length (e.g. mixed opinions here https://stackoverflo…

The spec is silent on length. 2000 bytes came from some web servers (old IIS comes to mind) that capped the URL at 2K or something close to that. So extra long URLS were problematic (and a lot of early web apps went nuts with parameters). So, max length is up to the implementer. All I know is that I've had to fix lots of code where someone assumed that 255 characters is all you'll ever need for a URL.

> old IIS comes to mind

And msie for which it’s a hard limit not just a default.

Re: URLs: It's Complicated

#35

Post by cookiengineer »

mananaysiempre wrote: post #17

Just to share a little more of the weirdness (discovered while reading a couple of the historical URL & URI RFCs several days ago): Per the original spec, in FTP URLs, - ftp://example.net/foo/bar will get you bar inside the foo directory inside the default directory of the FTP server at example.net ( i.e. CWD foo, RETR bar); - ftp://example.net//foo/bar will get you bar inside the foo directory inside the empty strin…

I wanted to mention that in practice, most FTP server implementations are not unicode compatible and are very likely vulnerable to effective-power-like abuses of RTL/LTR switching characters as well.

Let alone that probably all server implementations on Windows seem to have been a fork of BSD's original ftpd at some point, which had an RCE vulnerability when the password exceeded the limited bytelength of 256 bytes iirc.

Even software like ProFTPd where vulnerable over 30 years later.

Just writing this to make a point to stay the fuck away from FTP, because software is heavily outdated in that space and never updated to fix issues. Use ssh/sftp, always.

Re: URLs: It's Complicated

#36

Post by tpetry »

indymike wrote: post #29
zepearl wrote: post #14

All extremely useful: the overview, the examples and the comments. A few months ago while writing a bot/crawler I searched for hours for something like this, but I found only full specs or just bits and pieces scattered around that used different terminology and/or had different opinions. In the end I didn't even clearly understand what should be the max total URL length (e.g. mixed opinions here https://stackoverflo…

The spec is silent on length. 2000 bytes came from some web servers (old IIS comes to mind) that capped the URL at 2K or something close to that. So extra long URLS were problematic (and a lot of early web apps went nuts with parameters). So, max length is up to the implementer. All I know is that I've had to fix lots of code where someone assumed that 255 characters is all you'll ever need for a URL.

255 characters is the default to a variable length string column in databases. So if a developer did not pay attention he just used the default which is in some cases to short for an url.

Re: URLs: It's Complicated

#37

Post by [deleted] »

mananaysiempre wrote: post #31
jfrunyon wrote: post #20

Earlier quoted context omitted.

> what do FTP servers even do with this? Pretty sure CWD by itself isn't even valid (at least RFC959 assumes it has an argument), and therefore // isn't valid in FTP URLs. The %2Ffoo/bar is needed because of the fact that FTP CWD and RETR paths are system dependent (with, theoretically, system dependent path separators), but URLs are not, so the FTP client breaks the URL on / and sequentially executes CWD down the tr…

> Pretty sure CWD by itself isn’t even valid [...] and therefore // isn’t valid in FTP URLs. So, I looked it up carefully and it appears that (despite the promises in later RFCs such as 2396 and 3986) the current specification of the ftp scheme is still the ancient RFC 1738 which predates not only the URL / URI distinction but even the notion of relative URLs. In §3.2.2 https://tools.ietf.org/html/rfc1738#section-3.2…

[deleted]

Re: URLs: It's Complicated

#38

Post by benibela »

nayuki wrote: post #33
mananaysiempre wrote: post #16

Earlier quoted context omitted.

The use of Content-Encoding for compression is actually something of a historical wart: what was intended to be used for that purpose is Transfer-Encoding, but modern browsers don’t even send the TE header necessary to permit the HTTP server to use it (except for Transfer-Encoding: chunked which every HTTP 1.1 client must accept), even though some servers are perfectly capable of it and all but the most broken will a…

Even though HTTP DEFLATE saves ~20 bytes compared to GZIP, it itself is a wart because of some vendor misunderstandings. HTTP DEFLATE is actually DEFLATE data wrapped in a zlib container, not raw DEFLATE. See https://en.wikipedia.org/wiki/HTTP_compression#Problems_prev... ; https://stackoverflow.com/questions/3932117/handling-http-co...

I just implemented decompression in my HTTP client this week

I could not test that part because both server I tried send raw deflate, without zlib container

Re: URLs: It's Complicated

#39

Post by mananaysiempre »

cookiengineer wrote: post #35
mananaysiempre wrote: post #17

Just to share a little more of the weirdness (discovered while reading a couple of the historical URL & URI RFCs several days ago): Per the original spec, in FTP URLs, - ftp://example.net/foo/bar will get you bar inside the foo directory inside the default directory of the FTP server at example.net ( i.e. CWD foo, RETR bar); - ftp://example.net//foo/bar will get you bar inside the foo directory inside the empty strin…

I wanted to mention that in practice, most FTP server implementations are not unicode compatible and are very likely vulnerable to effective-power-like abuses of RTL/LTR switching characters as well. Let alone that probably all server implementations on Windows seem to have been a fork of BSD's original ftpd at some point, which had an RCE vulnerability when the password exceeded the limited bytelength of 256 bytes i…

You know, in a fantasy world where standards of comparable complexity have equally good implementations I would much rather use Telnet and FTP over TLS (1.3) than SSH and SFTP. For all that they show their age they just seem to me to be cleaner designs.

I will have to concede, though, that FTP servers in the real world are surprisingly awful. Even the supposedly easy task of spinning up an anonymous read-only FTP server to serve the current directory for five minutes, all permissions and security be damned, is annoyingly non-trivial.

(Unrelated to that awfulness, does anyone know how to get active FTP to pass through SLIRP networking on Qemu?)

Re: URLs: It's Complicated

#40

Post by cookiengineer »

mananaysiempre wrote: post #39
cookiengineer wrote: post #35

Earlier quoted context omitted.

I wanted to mention that in practice, most FTP server implementations are not unicode compatible and are very likely vulnerable to effective-power-like abuses of RTL/LTR switching characters as well. Let alone that probably all server implementations on Windows seem to have been a fork of BSD's original ftpd at some point, which had an RCE vulnerability when the password exceeded the limited bytelength of 256 bytes i…

You know, in a fantasy world where standards of comparable complexity have equally good implementations I would much rather use Telnet and FTP over TLS (1.3) than SSH and SFTP. For all that they show their age they just seem to me to be cleaner designs. I will have to concede, though, that FTP servers in the real world are surprisingly awful. Even the supposedly easy task of spinning up an anonymous read-only FTP ser…

I totally agree with you in regards of complexity. The main issue behind a server's level of security is probably more related to using a memory safe language than we care to admit.

I have the feeling that way too many libraries and implementations written in C use a linter or any kind of mechanism to catch the obvious type errors.

Everyone loves typed languages, but nobody uses their obvious advantages in regards to security. Kinda ironic when you see a -Wall all over the place.

Post reply on HN