Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If your code accepts URIs as input, filter out “file://”

blog.steve.fi

Post reply on HN
Print view View original

51–60 of 161 posts

Re: If your code accepts URIs as input, filter out “file://”

#51

Post by kijin »

tokenizerrr wrote: post #44
mootothemax wrote: post #38

Earlier quoted context omitted.

>This is highly non-trivial once you realize that the world speaks more than ASCII and things like http://www.xn--n3h.net exist. I was under the impression that requests to and from the server still used ASCII? That is, the server would see a host header as this: Host: www.xn--n3h.net And not as this: Host: www.[snowman icon].net Anything else is a question of URL-encoding, which if not used would raise interesting b…

Right but how does the user submit it and what do you put in the href?

If a user copied the URL from the address bar, it will be correctly percent-encoded already.

You can put the same percent-encoded URL in the href attribute of a hyperlink. A properly encoded URL will not contain any character that requires escaping in an HTML context.

When a user clicks on that link, the browser will navigate to the percent-encoded URL but display the snowman icon in the address bar. If the user copies it, it will transparently turn back into the percent-encoded URL. All modern browsers do this.

Re: If your code accepts URIs as input, filter out “file://”

#52

Post by ptero »

pdkl95 wrote: post #33
hughperkins wrote: post #4

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

Can we please stop trying to enumerate badness [1]? When parsing input it is possible to define the set of valid input, not all possible invalid inputs. Also, anybody accepting input from an untrusted source (such as anything from a network or the user) that isn't verifying the data with a formal recognizer is doing it wrong[2]. Instead of writing another weird machine, guarantee that the input is valid with a parser…

I agree with the approach. However, specific examples of different badnesses are useful for testing the final product.

Re: If your code accepts URIs as input, filter out “file://”

#53

Post by ubernostrum »

pdkl95 wrote: post #33
hughperkins wrote: post #4

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

Can we please stop trying to enumerate badness [1]? When parsing input it is possible to define the set of valid input, not all possible invalid inputs. Also, anybody accepting input from an untrusted source (such as anything from a network or the user) that isn't verifying the data with a formal recognizer is doing it wrong[2]. Instead of writing another weird machine, guarantee that the input is valid with a parser…

guarantee that the input is valid with a parser generator

OK, that works really well... until you learn how much non-RFC-specified behavior is built in to web browsers. Simply building a parser to the RFC will leave you wide open to all sorts of nastiness!

The is_safe_url() internal function in Django is a bit of a historical dive into things we've learned about how browsers interpret (or, arguably, misinterpret) various types of oddball URLs:

https://github.com/django/django/blob/master/django/utils/ht...

Re: If your code accepts URIs as input, filter out “file://”

#54

Post by nitrogen »

philo23 wrote: post #21
milankragujevic wrote: post #15

Earlier quoted context omitted.

/etc != http :)

Ah yes, you're right. Totally mis-read the code. It'd still leave access to any files in the same (or sub) directory starting with http, which realistically would probably be none but still something to bear in mind.

Right, best to check the whole URL. Something like "http/../../../etc/passwd" might get through otherwise.

Re: If your code accepts URIs as input, filter out “file://”

#55

Post by rndgermandude »

greenleafjacob wrote: post #10

This is the confused deputy problem. The most general solution to this class of vulnerabilities, SELinux, has been largely ignored. Does SELinux need more work to "bring it to market", or is it just too complicated and needs to be simplified?

SELinux is more the stopgap measure when everything else failed already, or at least it should be and prevent the most harmful things like reading random stuff from /etc. It is not something I'd say of "I got SELinux, now I don't need to validate user input".

In the concrete example from the article, the process needs to access to /etc/hosts to do name resolutions, yet it should not send this information out to who knows who. How do you model that as a SELinux config? You cannot really. Unless you introduce dedicated (class of uncoupled, distinct, identifiable class of) processes acting as agents for resolving hosts with the help of /etc/hosts and whitelist them in SELinux... Which adds a whole lot of complexity. And you still have to make sure your new fancy agents cannot be tricked into giving up sensitive information.

So at the end of the day, you should do defense in depth which of course should include user input validation and probably SELinux as well.

Re: If your code accepts URIs as input, filter out “file://”

#56

Post by shocks »

byuu wrote: post #50

Also be very wary of ../ or possibly ..\ in URIs. Say you have http://example.org/download?file=release/software-1.0.zip so that you can log download statistics. If you just fetch GET["download"] and return it, you're gonna have a bad time if they try http://example.org/download?file=../../../../../etc/hosts (browsers strip it out automatically, but it's easy enough to type such a request into a telnet session.) And…

Also, you can't just find and replace ../

Consider: ....// which becomes ../

Re: If your code accepts URIs as input, filter out “file://”

#57

Post by kijin »

jwilk wrote: post #19

> Actually the actual output all newlines had been stripped. Not stripped, but replaced by spaces. Also, the linked image looks like /etc/passwd, not /etc/hosts. > Weird. Not weird. That's how whitespace in HTML works.

OP tested his hack on an online Markdown converter, so it probably has more to do with Markdown's treatment of whitespace than HTML's.

As it happens, Markdown ignores most non-consecutive newlines.

Re: If your code accepts URIs as input, filter out “file://”

#58

Post by konceptz »

Quick FYI. It's difficult to filter URIs with regex because they are context free grammars and belong to a larger language than regular expressions.

Recursive descent parsers are the way to go like uri objects in most languages.

Ex. URL url = URL.fromString(inputString); url.getScheme;

Re: If your code accepts URIs as input, filter out “file://”

#59

Post by detaro »

kijin wrote: post #57
jwilk wrote: post #19

> Actually the actual output all newlines had been stripped. Not stripped, but replaced by spaces. Also, the linked image looks like /etc/passwd, not /etc/hosts. > Weird. Not weird. That's how whitespace in HTML works.

OP tested his hack on an online Markdown converter, so it probably has more to do with Markdown's treatment of whitespace than HTML's. As it happens, Markdown ignores most non-consecutive newlines.

It's a converter taking HTML as input, converting it to Markdown.

Re: If your code accepts URIs as input, filter out “file://”

#60

Post by konceptz »

Grue3 wrote: post #12

I tried Python requests and Common Lisp drakma, and neither of them can handle "file://" URL schema. Which HTTP client libraries are actually vulnerable to this?

Not to nitpick but it's not really a vulnerability.

It's including the correct protocols for URIs.

Post reply on HN