Live data from Hacker News

If your code accepts URIs as input, filter out “file://”

blog.steve.fi

121–130 of 161 posts

Re: If your code accepts URIs as input, filter out “file://”

#121
post #33

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

Can we please stop trying to enumerate badness [1]? When parsing input it is possible to define the set of valid input, not all possible invalid inputs. Also, anybody accepting input from an untrusted source (such as anything from a network or the user) that isn't verifying the data with a formal recognizer is doing it wrong[2]. Instead of writing another weird machine, guarantee that the input is valid with a parser…

> Can we please stop trying to enumerate badness[1]?

No. Because we don't know what goodness looks like.

The world can be separated into good, bad and unknown. If you classify anything unknown as bad then anything new is DOA. People aren't going to add new things to the whitelist before they're popular which means they can never become popular. It's stasis.

But people do that anyway, which makes the good guys have to adopt the MO of the bad guys and make the new thing look like the existing thing. So everything uses HTTP and everything looks the same.

Which means everything is more complicated than it needs to be, because it has to pretend to be something else, which creates more attack surface.

And which means the whitelist is no longer meaningful because allow-http becomes equivalent to allow-everything.

It's like buying a car that can only drive to home and work on the theory that it will be safer. It will be at first, except that you can no longer go anywhere but home and work. But when enough people do that then everything (including the bad stuff) has to move to where people are allowed to go. Which puts you right back where you started except that now you have two problems.

Re: If your code accepts URIs as input, filter out “file://”

#122

Earlier quoted context omitted.

> The "bad input" is, arguably, no longer bad input. What? Yes it is! Defined behavior for invalid markup doesn't make that markup valid . HTML5 doesn't refuse to accept anything that HTML 4 accepted. Defining behavior for invalid markup does not even impact "be liberal in what you accept", the scope of what is accepted hasn't changed. It affects "be conservative in what you send", in particular it more closely match…

> HTML5 doesn't refuse to accept anything that HTML 4 accepted. It does. It doesn't accept NET syntax, i.e., `p/This is contents of a p elements/`. (No browser ever supported this, but because HTML 4 is defined to be an SGML application and it's DTD allows NET syntax to be used, it is theoretically conforming HTML 4.)

Ah good point, thanks for the correction.

Re: If your code accepts URIs as input, filter out “file://”

#123

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

Also, fully decode the input string before doing this processing to make sure you really find all those sequences. A simple, seemingly obvious step that a surprising amount of software neglects to do.

Re: If your code accepts URIs as input, filter out “file://”

#124
post #104

Wrong way around: only allow http:// and https:// (and generally filtering out anything thats not letters, numbers, slash or dot is probably a good idea. Remove any sequences of more than one slash or dot.

Also make sure to fully resolve the DNS down to all possible IP addresses, and verify that they are all external to your network. And if you're on EC2, make sure nobody is hitting 169.254.169.254. Really, there are so many gotchas around fetching user-supplied URLs that it's scary.

And be sure to check it again if there's a redirect. Don't let your URL handling library do this. Alternately send all your traffic through a proxy that can't talk into your network.

Re: If your code accepts URIs as input, filter out “file://”

#125

Earlier quoted context omitted.

You don't need elevated privileges to read /etc/passwd

That is correct. sebboh@namagiri:~$ ls -al /etc/passwd -rw-r--r-- 1 root root 1450 Jun 7 09:08 /etc/passwd So, there's this idea of running publicly facing services as a user which has less privileges than a normal interactive user. This user might be called 'nobody' or 'apache', etc. However, on your average distribution, /etc/passwd is accessible to the 'nobody' user or the 'apache' user. There are various work aro…

Since ages now: https://en.wikipedia.org/wiki/Shadow_password

Re: If your code accepts URIs as input, filter out “file://”

#126

Earlier quoted context omitted.

I just tried doing that with a few domain names containing an umlaut (äöü) and every single time that letter was copied into the clipboard (even though behind the scenes at the request level it would have been encoded). This is what I expect as a regular user. They don't want to deal with encoded, unreadable URLs.

I tried with http://њњњ.срб , which Firefox copies correctly, but Chromium copies as http://xn--g2aaa.xn--90a3ac/ — not very useful. This is a different mechanism to the path part, where both Firefox and Chromium give https://ru.wikipedia.org/wiki/%D0%A0%D0%BE%D1%81%D1%81%D0%B8... rather than the readable https://ru.wikipedia.org/wiki/Россия

If you select the entire URL bar, you get the encoded form. If you leave off the protocol, or just the h, you get it unencoded.

Re: If your code accepts URIs as input, filter out “file://”

#127
post #92

Earlier quoted context omitted.

I only started seeing this weary/wary misspelling in recent years. They don't sound alike, and they don't really look alike. Did cell phone spellcheckers give rise to this one?

> /ˈwɪəɹi/ > /wɛəɹ.i/ I'm not a native English speaker and I would never have guessed that they were not pronounced the same. (I'm still not even sure how it is pronounced as ɪ doesn't seem to exist in French and I always considered the examples I find were just "i"). I know English pronunciation is generally weird, but seriously how can you expect wary to sound like wear while weary sounds like something else?

"wear" is the word that's doing things wrong here.

Would "geary" and "gary" avoid the same mistake?

> seriously how can you expect

I can also question how you would expect two words that differ by their vowel to sound the same.

Re: If your code accepts URIs as input, filter out “file://”

#128
post #121
post #33

Earlier quoted context omitted.

Can we please stop trying to enumerate badness [1]? When parsing input it is possible to define the set of valid input, not all possible invalid inputs. Also, anybody accepting input from an untrusted source (such as anything from a network or the user) that isn't verifying the data with a formal recognizer is doing it wrong[2]. Instead of writing another weird machine, guarantee that the input is valid with a parser…

> Can we please stop trying to enumerate badness [1]? No. Because we don't know what goodness looks like. The world can be separated into good, bad and unknown. If you classify anything unknown as bad then anything new is DOA. People aren't going to add new things to the whitelist before they're popular which means they can never become popular. It's stasis. But people do that anyway, which makes the good guys have t…

> Because we don't know what goodness looks like.

You're writing the parser, so you define the set of acceptable input.

> The world can be separated into good, bad and unknown

The data your software receives as input can be separated into valid input that your software will correctly interpret, or invalid input that is either and error or an attack.

There shouldn't ever be any "unknown" input, as that would imply you don't know how your software parses its input. As the ccc talk in my previous [2] explains, this may be true if recognition of input is scattered across your software and thus hard to understand as a complete grammar. Thus the recommendation to put it all in one place using a parser generator (or whatever).

> If you classify anything unknown as bad then anything new is DOA.

Anything unknown is by definition not properly supported by the software you're writing.

Re: If your code accepts URIs as input, filter out “file://”

#129

Earlier quoted context omitted.

That is correct. sebboh@namagiri:~$ ls -al /etc/passwd -rw-r--r-- 1 root root 1450 Jun 7 09:08 /etc/passwd So, there's this idea of running publicly facing services as a user which has less privileges than a normal interactive user. This user might be called 'nobody' or 'apache', etc. However, on your average distribution, /etc/passwd is accessible to the 'nobody' user or the 'apache' user. There are various work aro…

Since ages now: https://en.wikipedia.org/wiki/Shadow_password

That doesn't mean you should allow your app to access `file://` ... (Nor allow it out of a chroot.)

Simply knowing some usernames on your system could provide an attacker with clues...

Re: If your code accepts URIs as input, filter out “file://”

#130

Earlier quoted context omitted.

> HTML5 doesn't refuse to accept anything that HTML 4 accepted. It does. It doesn't accept NET syntax, i.e., `p/This is contents of a p elements/`. (No browser ever supported this, but because HTML 4 is defined to be an SGML application and it's DTD allows NET syntax to be used, it is theoretically conforming HTML 4.)

Ah good point, thanks for the correction.

(There's also another load of SGML bits of syntax that browsers have never supported which HTML5 doesn't support. Indeed, HTML 4 has a whole section of such things: http://www.w3.org/TR/html4/appendix/notes.html#h-B.3.3)
Post reply on HN