Live data from Hacker News

If your code accepts URIs as input, filter out “file://”

blog.steve.fi

101–110 of 161 posts

Re: If your code accepts URIs as input, filter out “file://”

#101
post #72

Earlier quoted context omitted.

It should be pointed out that while this was once accepted as gospel, it has been coming under a lot of fire lately. HTML, once arguably the flagship of this principle and its greatest success (I say "arguably" because you can also argue TCP), no longer works this way. HTML5 specifies how bad input should be handled, and if you accept that "how to process nominally bad input" as the "real" standard, HTML is now stric…

> HTML5 specifies how bad input should be handled, and if you accept that "how to process nominally bad input" as the "real" standard, HTML is now strict in what it accepts. HTML5 is a shining example of "be liberal in what you accept", and its improved documentation of how to handle bad input (note that bad input is still permitted!) greatly expands HTML's "be conservative in what you send". I think HTML5 is a perfe…

The "bad input" is, arguably, no longer bad input. The standard has been redefined to strictly specify what to do with that "bad" input, and if you don't handle it exactly as the standard specifies, it won't do what you "want" it to do.

That's not "being liberal in what you accept". Being liberal in what you expect is what we had before HTML 5, where the standard specified the "happy case" and the browsers were all "liberal in what they expect", in different ways. I am not stretching any definitions here or making anything up, because "liberal in what you accept" behaviors in the real world demonstrably work this way; everybody is liberal in different ways. It can hardly be otherwise; it isn't "being liberal in what you accept" if you accept exactly what the standard permits, after all. When liberality is permitted, what happens in practice is that out-of-spec input is handled in whatever the most convenient way for the local handler is, in the absence of any other considerations (such as deliberately trying to be compatible with the quirky internal details of the competition). Browsers leaked a lot about their internal differences if you observed how they tended to handle out-of-spec input. Thus a standard like HTML5 that clearly specifies how to handle all cases now is fundamentally not "liberal in what it accepts" anymore.

Instead, it is a rare, if not unique, example of a standard that has been rigidly specified after a couple of decades of seeing exactly how humans messed up the original standard. It is, nevertheless, now quite precise about what to do about the HTML you encounter. You aren't allowed to be "liberal", you're told exactly what to do.

Re: If your code accepts URIs as input, filter out “file://”

#102
post #35

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

> filtering out anything thats not letters, numbers, slash or dot is probably a good idea. This is highly non-trivial once you realize that the world speaks more than ASCII and things like http://www.xn--n3h.net exist.

I've had an international domain since 2006, and the sad truth is they still aren't widely supported 10 years later (the fuckyeahmarkdown website being a case in point). I don't think people are deliberately filtering out those characters - they just aren't aware that such names are even possible.

In the beginning I used to file bug reports whenever I encountered websites that couldn't handle my domain, but I eventually resigned myself to the fact that most people just don't care. Nowadays I don't even bother trying the unicode most of the time, and just use the punycode version instead.

Re: If your code accepts URIs as input, filter out “file://”

#103

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

You seriously want to disallow gopher://? C'mon, man!

[deleted]

Re: If your code accepts URIs as input, filter out “file://”

#104

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

Also make sure to fully resolve the DNS down to all possible IP addresses, and verify that they are all external to your network. And if you're on EC2, make sure nobody is hitting 169.254.169.254.

Really, there are so many gotchas around fetching user-supplied URLs that it's scary.

Re: If your code accepts URIs as input, filter out “file://”

#105

This is the confused deputy problem. The most general solution to this class of vulnerabilities, SELinux, has been largely ignored. Does SELinux need more work to "bring it to market", or is it just too complicated and needs to be simplified?

SELinux is not a solution to this class of vulnerabilities; it's a backup plan. The right way is to not have stupid APIs that are easy to do dangerous things with. Compare PHP's fopen wrappers with the requests Python package, for example.

Re: If your code accepts URIs as input, filter out “file://”

#106

Earlier quoted context omitted.

I only started seeing this weary/wary misspelling in recent years. They don't sound alike, and they don't really look alike. Did cell phone spellcheckers give rise to this one?

'weary' means tired of something, 'wary' means cautious of something.

Because our language is a mashup of multiple other languages, and thus the rules are inconsistent.

As for pronunciation of those two:

weary is pronounced like "ear" wary is pronounced like "air"

Re: If your code accepts URIs as input, filter out “file://”

#107

Don't write server code that opens URI's that come in as input, period. If you take URI's as input, do it only to turn them around and spit them out into some Javascript sent back to the same session. Whatever can or cannot be accessed this way is the browser's problem.

This works as long as your business model doesn't have opening user's URIs at its core, e.g. monitoring service.

Re: If your code accepts URIs as input, filter out “file://”

#109
post #101

Earlier quoted context omitted.

> HTML5 specifies how bad input should be handled, and if you accept that "how to process nominally bad input" as the "real" standard, HTML is now strict in what it accepts. HTML5 is a shining example of "be liberal in what you accept", and its improved documentation of how to handle bad input (note that bad input is still permitted!) greatly expands HTML's "be conservative in what you send". I think HTML5 is a perfe…

The "bad input" is, arguably, no longer bad input. The standard has been redefined to strictly specify what to do with that "bad" input, and if you don't handle it exactly as the standard specifies, it won't do what you "want" it to do. That's not "being liberal in what you accept". Being liberal in what you expect is what we had before HTML 5, where the standard specified the "happy case" and the browsers were all "…

> The "bad input" is, arguably, no longer bad input.

What? Yes it is! Defined behavior for invalid markup doesn't make that markup valid.

HTML5 doesn't refuse to accept anything that HTML 4 accepted. Defining behavior for invalid markup does not even impact "be liberal in what you accept", the scope of what is accepted hasn't changed. It affects "be conservative in what you send", in particular it more closely matches that half of the principle.

Re: If your code accepts URIs as input, filter out “file://”

#110

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

This is the correct answer and best practice. Be conservative in what you accept and liberal in what you produce.
Post reply on HN