Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If your code accepts URIs as input, filter out “file://”

blog.steve.fi

Post reply on HN
Print view View original

111–120 of 161 posts

Re: If your code accepts URIs as input, filter out “file://”

#111

Post by gsnedders »

oneplane wrote: post #65
ubernostrum wrote: post #53

Earlier quoted context omitted.

guarantee that the input is valid with a parser generator OK, that works really well... until you learn how much non-RFC-specified behavior is built in to web browsers. Simply building a parser to the RFC will leave you wide open to all sorts of nastiness! The is_safe_url() internal function in Django is a bit of a historical dive into things we've learned about how browsers interpret (or, arguably, misinterpret) var…

I do wonder; is there any browser that is actually full-RFC-specced? I checked a few (the mainstream desktop ones, but also links2 etc.), but so far they all seem to have glue to fix historical behavior.

Pretty much no, because it'd be practically useless. And I don't think anyone has the willingness to spend time or money on something that will essentially just be a toy.

There's been plenty of work on moving the standards so that there are actually implementations of them, instead of them being practically useless at best and misleading at worst (given doing input validation based on a spec that nobody actually implements is just outright dangerous), with HTML 5 and much of CSS 2.1 leading that charge (though CSS 2.1 still has massive blackholes, notably table layout remains largely undefined, though that is finally being worked on).

Re: If your code accepts URIs as input, filter out “file://”

#112

Post by idbehold »

byuu wrote: post #50

Also be very wary of ../ or possibly ..\ in URIs. Say you have http://example.org/download?file=release/software-1.0.zip so that you can log download statistics. If you just fetch GET["download"] and return it, you're gonna have a bad time if they try http://example.org/download?file=../../../../../etc/hosts (browsers strip it out automatically, but it's easy enough to type such a request into a telnet session.) And…

I've done something similar in node to ensure URIs like those cannot walk up past a base directory:

  function saferesolve(base, target) {
    var targetPath = '.' + path.posix.normalize('/' + target)
    return path.posix.resolve(base, targetPath)
  }

  saferesolve("./datasource", "a/b") === "./datasource/a/b"
  saferesolve("./datasource", "a/b/../c") === "./datasource/a/c"
  saferesolve("./datasource", "../..") === "./datasource"
  saferesolve("./datasource", "../../a/b") === "./datasource/a/b"
  saferesolve("./datasource", "../../a/b/..") === "./datasource/a"

Re: If your code accepts URIs as input, filter out “file://”

#113

Post by gsnedders »

ubernostrum wrote: post #53
pdkl95 wrote: post #33

Earlier quoted context omitted.

Can we please stop trying to enumerate badness [1]? When parsing input it is possible to define the set of valid input, not all possible invalid inputs. Also, anybody accepting input from an untrusted source (such as anything from a network or the user) that isn't verifying the data with a formal recognizer is doing it wrong[2]. Instead of writing another weird machine, guarantee that the input is valid with a parser…

guarantee that the input is valid with a parser generator OK, that works really well... until you learn how much non-RFC-specified behavior is built in to web browsers. Simply building a parser to the RFC will leave you wide open to all sorts of nastiness! The is_safe_url() internal function in Django is a bit of a historical dive into things we've learned about how browsers interpret (or, arguably, misinterpret) var…

Hopefully the URL spec (https://url.spec.whatwg.org) is helpful here in finding other potentially unsafe behaviours that browsers have, though given much of it seems to be dealing with the fact that urllib.urlparse doesn't match what browsers do in many, many ways it's probably of limited help. (Nobody really implements it yet; it's just an attempt at standardising rough intersection semantics of what browsers currently do. Eventually, however, it should suffice, once legacy browsers eventually die.)

Re: If your code accepts URIs as input, filter out “file://”

#114

Post by artursapek »

jerf wrote: post #72
artursapek wrote: post #24

Earlier quoted context omitted.

https://en.m.wikipedia.org/wiki/Robustness_principle

It should be pointed out that while this was once accepted as gospel, it has been coming under a lot of fire lately. HTML, once arguably the flagship of this principle and its greatest success (I say "arguably" because you can also argue TCP), no longer works this way. HTML5 specifies how bad input should be handled, and if you accept that "how to process nominally bad input" as the "real" standard, HTML is now stric…

I think web browsers are a better example of it. The HTML parsing/DOM tree system usually is pretty forgiving about missing/malformed tags, but still always returns a result rendered as if the HTML had been written to spec.

Re: If your code accepts URIs as input, filter out “file://”

#115

Post by gsnedders »

eyelidlessness wrote: post #109
jerf wrote: post #101

Earlier quoted context omitted.

The "bad input" is, arguably, no longer bad input. The standard has been redefined to strictly specify what to do with that "bad" input, and if you don't handle it exactly as the standard specifies, it won't do what you "want" it to do. That's not "being liberal in what you accept". Being liberal in what you expect is what we had before HTML 5, where the standard specified the "happy case" and the browsers were all "…

> The "bad input" is, arguably, no longer bad input. What? Yes it is! Defined behavior for invalid markup doesn't make that markup valid . HTML5 doesn't refuse to accept anything that HTML 4 accepted. Defining behavior for invalid markup does not even impact "be liberal in what you accept", the scope of what is accepted hasn't changed. It affects "be conservative in what you send", in particular it more closely match…

> HTML5 doesn't refuse to accept anything that HTML 4 accepted.

It does. It doesn't accept NET syntax, i.e., `p/This is contents of a p elements/`. (No browser ever supported this, but because HTML 4 is defined to be an SGML application and it's DTD allows NET syntax to be used, it is theoretically conforming HTML 4.)

Re: If your code accepts URIs as input, filter out “file://”

#116

Post by artursapek »

oldmanjay wrote: post #93
artursapek wrote: post #24

Earlier quoted context omitted.

https://en.m.wikipedia.org/wiki/Robustness_principle

No, be strict, fail fast, and report the errors. Robustness is not achieved by muddling through on a misinterpretation, it is achieved by working toward correctness.

I think "fail fast" can work well in a closed and controlled system but when accepting input from many other parties it's not as practical or desirable.

Re: If your code accepts URIs as input, filter out “file://”

#117

Post by gsnedders »

taneq wrote: post #70
artursapek wrote: post #24

Earlier quoted context omitted.

https://en.m.wikipedia.org/wiki/Robustness_principle

Didn't we decide in the aftermath of IE6 that this was a bad idea, and that we should be strict in both what we accept and what we emit?

No, we decided that what was important was interoperable implementations: it doesn't matter how you achieve that goal. What's needed is specs that define how to handle all input (it doesn't matter what the spec says: it can define how to handle every single last possible case as HTML5 does, or it can define a subset of inputs to trigger some fatal error handling as XML1.0 does) and sufficient test suites that implementers catch bugs in their code before it ships (and the web potentially starts relying on their quirks).

The problem with IE6 was the fact that it wasn't interoperable (in many cases, every implementation was conforming according to the spec, and there were frequently differences in behaviour in valid input that the spec didn't fully define) and the fact that it had lots of proprietary extensions (and being strict and disallowing any extensions makes it hard to extend formats in general in a non-proprietary way; one option is strict versioning but then you end up with a million if statements all over the implementation to alter behaviour depending on the version of the content).

Some of the worst issues with IE that took the longest for other browsers to match were things like table layout: IE quite closely matched NN4 having invested a lot in reverse-engineering that as the web depended on the NN4 behaviour in places; Gecko had rewritten all the table layout code from NN and didn't match its behaviour having been written according to the specs which scarcely define how to layout any table even today.

Re: If your code accepts URIs as input, filter out “file://”

#118

Post by Eridrus »

Eridrus wrote: post #17

This is a special case of a Server-Side Request Forget vulnerability. Validating schemes is part of the answer but not the whole answer because attackers can still forge requests to internal resources you have firewalled off from the internet. These were recently released to help people deal with these issues since the details can be finicky: http://blog.includesecurity.com/2016/08/safeurl-server-side-...

Should say "Server-Side Request Forgery", damn mobile.

Re: If your code accepts URIs as input, filter out “file://”

#119

Post by kuschku »

gsnedders wrote: post #113
ubernostrum wrote: post #53

Earlier quoted context omitted.

guarantee that the input is valid with a parser generator OK, that works really well... until you learn how much non-RFC-specified behavior is built in to web browsers. Simply building a parser to the RFC will leave you wide open to all sorts of nastiness! The is_safe_url() internal function in Django is a bit of a historical dive into things we've learned about how browsers interpret (or, arguably, misinterpret) var…

Hopefully the URL spec ( https://url.spec.whatwg.org ) is helpful here in finding other potentially unsafe behaviours that browsers have, though given much of it seems to be dealing with the fact that urllib.urlparse doesn't match what browsers do in many, many ways it's probably of limited help. (Nobody really implements it yet; it's just an attempt at standardising rough intersection semantics of what browsers curr…

That URL spec is just "this is what chrome does, everyone repeat that".

They’re unwilling to modify anything, or standardize anything, but just want to cement the current piece of shit that URL parsing it for the future.

Re: If your code accepts URIs as input, filter out “file://”

#120

Post by artursapek »

mootothemax wrote: post #37
artursapek wrote: post #30

Earlier quoted context omitted.

Right - you accept URIs. That's fairly liberal. > Be conservative in what you do However, you only handle specific schemes and ignore the rest.

Yeah, apologies, I was being pretty petty. My hopefully-better-expressed point is that it's easy to interpret the robustness principle in different ways, some of which lead to better code, and some of which... don't.

No worries. I think the philosophy is rooted in the fact that you can't control what other parties will send you; you can only control what you send in response. So that's the main thing to keep in mind.

It's sort of like the good life advice you hear occasionally: you can't control other peoples' actions; only your own. Emotional maturity, etc.

Post reply on HN