Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If your code accepts URIs as input, filter out “file://”

blog.steve.fi

Post reply on HN
Print view View original

131–140 of 161 posts

Re: If your code accepts URIs as input, filter out “file://”

#131

Post by sgift »

kazinator wrote: post #90
pdkl95 wrote: post #33

Earlier quoted context omitted.

Can we please stop trying to enumerate badness [1]? When parsing input it is possible to define the set of valid input, not all possible invalid inputs. Also, anybody accepting input from an untrusted source (such as anything from a network or the user) that isn't verifying the data with a formal recognizer is doing it wrong[2]. Instead of writing another weird machine, guarantee that the input is valid with a parser…

Exception: passwords. Do not enumerate goodness when accepting a new password.

Password checking could be so easy ...

if(password.size() Update each year to stay ahead of faster computer speeds.

Re: If your code accepts URIs as input, filter out “file://”

#132

Post by gsnedders »

kuschku wrote: post #119
gsnedders wrote: post #113

Earlier quoted context omitted.

Hopefully the URL spec ( https://url.spec.whatwg.org ) is helpful here in finding other potentially unsafe behaviours that browsers have, though given much of it seems to be dealing with the fact that urllib.urlparse doesn't match what browsers do in many, many ways it's probably of limited help. (Nobody really implements it yet; it's just an attempt at standardising rough intersection semantics of what browsers curr…

That URL spec is just "this is what chrome does, everyone repeat that". They’re unwilling to modify anything, or standardize anything, but just want to cement the current piece of shit that URL parsing it for the future.

Given Chrome Canary currently fails a large number of tests, it seems like it's hardly just "this is what chrome does, everyone repeat that".

Re: If your code accepts URIs as input, filter out “file://”

#133

Post by rzzzt »

Symbiote wrote: post #96
germanier wrote: post #79

Earlier quoted context omitted.

I just tried doing that with a few domain names containing an umlaut (äöü) and every single time that letter was copied into the clipboard (even though behind the scenes at the request level it would have been encoded). This is what I expect as a regular user. They don't want to deal with encoded, unreadable URLs.

I tried with http://њњњ.срб , which Firefox copies correctly, but Chromium copies as http://xn--g2aaa.xn--90a3ac/ — not very useful. This is a different mechanism to the path part, where both Firefox and Chromium give https://ru.wikipedia.org/wiki/%D0%A0%D0%BE%D1%81%D1%81%D0%B8... rather than the readable https://ru.wikipedia.org/wiki/Россия

The two methods are punycode [1] and percent encoding [2].

[1] https://en.wikipedia.org/wiki/Punycode

[2] https://en.wikipedia.org/wiki/Percent-encoding

Re: If your code accepts URIs as input, filter out “file://”

#134

Post by tokenizerrr »

kijin wrote: post #51
tokenizerrr wrote: post #44

Earlier quoted context omitted.

Right but how does the user submit it and what do you put in the href?

If a user copied the URL from the address bar, it will be correctly percent-encoded already. You can put the same percent-encoded URL in the href attribute of a hyperlink. A properly encoded URL will not contain any character that requires escaping in an HTML context. When a user clicks on that link, the browser will navigate to the percent-encoded URL but display the snowman icon in the address bar. If the user copi…

If I'm a user writing an URL, then I will write it as it appears in the URL bar. That means you must be able to accept URLs that contain unicode.

Keep in mind that unicode isn't just for emojii. Plenty of languages use characters that are not in ascii.

Re: If your code accepts URIs as input, filter out “file://”

#135

Post by l_zzie »

zwily wrote: post #104
hughperkins wrote: post #4

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

Also make sure to fully resolve the DNS down to all possible IP addresses, and verify that they are all external to your network. And if you're on EC2, make sure nobody is hitting 169.254.169.254. Really, there are so many gotchas around fetching user-supplied URLs that it's scary.

Importantly, fetching DNS twice (once to check, another to download) is an incomplete solution, since DNS responses can change (cf "DNS rebinding").

Re: If your code accepts URIs as input, filter out “file://”

#136

Post by kuschku »

gsnedders wrote: post #132
kuschku wrote: post #119

Earlier quoted context omitted.

That URL spec is just "this is what chrome does, everyone repeat that". They’re unwilling to modify anything, or standardize anything, but just want to cement the current piece of shit that URL parsing it for the future.

Given Chrome Canary currently fails a large number of tests, it seems like it's hardly just "this is what chrome does, everyone repeat that".

Because the standard was changed to clean a bit of the stuff Google did up.

But WHATWG only changes standards to include more, never to include less.

Re: If your code accepts URIs as input, filter out “file://”

#137

Post by jcranmer »

kuschku wrote: post #119
gsnedders wrote: post #113

Earlier quoted context omitted.

Hopefully the URL spec ( https://url.spec.whatwg.org ) is helpful here in finding other potentially unsafe behaviours that browsers have, though given much of it seems to be dealing with the fact that urllib.urlparse doesn't match what browsers do in many, many ways it's probably of limited help. (Nobody really implements it yet; it's just an attempt at standardising rough intersection semantics of what browsers curr…

That URL spec is just "this is what chrome does, everyone repeat that". They’re unwilling to modify anything, or standardize anything, but just want to cement the current piece of shit that URL parsing it for the future.

WHATWG standards are generally formed by starting from what the 4 major browsers (Chrome, Firefox, IE (Edge), Safari) do. Anything that is done in common by all of them gets implemented no problem. It's when they all differ that the editor(s) tries to come up with more reasoned algorithms.

Re: If your code accepts URIs as input, filter out “file://”

#138

Post by kuschku »

jcranmer wrote: post #137
kuschku wrote: post #119

Earlier quoted context omitted.

That URL spec is just "this is what chrome does, everyone repeat that". They’re unwilling to modify anything, or standardize anything, but just want to cement the current piece of shit that URL parsing it for the future.

WHATWG standards are generally formed by starting from what the 4 major browsers (Chrome, Firefox, IE (Edge), Safari) do. Anything that is done in common by all of them gets implemented no problem. It's when they all differ that the editor(s) tries to come up with more reasoned algorithms.

Which is not really ideal.

Standards aren’t about documenting what is, but about defining what will be.

Re: If your code accepts URIs as input, filter out “file://”

#139

Post by dragonwriter »

jcranmer wrote: post #137
kuschku wrote: post #119

Earlier quoted context omitted.

That URL spec is just "this is what chrome does, everyone repeat that". They’re unwilling to modify anything, or standardize anything, but just want to cement the current piece of shit that URL parsing it for the future.

WHATWG standards are generally formed by starting from what the 4 major browsers (Chrome, Firefox, IE (Edge), Safari) do. Anything that is done in common by all of them gets implemented no problem. It's when they all differ that the editor(s) tries to come up with more reasoned algorithms.

> WHATWG standards are generally formed by starting from what the 4 major browsers (Chrome, Firefox, IE (Edge), Safari) do.

I thought WHATWG standards are formed by starting with what the four major browser vendors agree to do, not what they currently do (though usually at least one has an implementation before something gets proposed for standardization.)

Re: If your code accepts URIs as input, filter out “file://”

#140

Post by captn3m0 »

zwily wrote: post #104
hughperkins wrote: post #4

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

Also make sure to fully resolve the DNS down to all possible IP addresses, and verify that they are all external to your network. And if you're on EC2, make sure nobody is hitting 169.254.169.254. Really, there are so many gotchas around fetching user-supplied URLs that it's scary.

I remember this was exactly how a readability service (readability or instapaper or something similar, can't recall now) was attacked. The service allowed you to fetch internal urls and presented them formatted on your phone. A mixture of file:// and internal web urls allowed complete takeover.
Post reply on HN