Live data from Hacker News

Escaping user input is ridonkulously hard

codeofhonor.substack.com

21–30 of 88 posts

Re: Escaping user input is ridonkulously hard

#22
post #7

An alternate view: “string” is not a granular enough type, just like “bitfield” is not a type. Firstly, a string could be raw unknown bytes, verified UTF-8, or UCS-2 (or even UTF-16 or UCS-4), and you absolutely need to know which it is. But let’s assume that you’ve been a diligent programmer and filtered all that at the edges, and now have a sequence of Unicode code points (or possibly graphemes). You still need to…

> Firstly, a string could be raw unknown bytes, verified UTF-8, or UCS-2 (or even UTF-16 or UCS-4), and you absolutely need to know which it is. This is a language defect. If your language was invented in the 1960s it's an understandable defect, but it's still a defect. I do not want to write computer software with strings in a language that doesn't even have an actual string type rather than "Eh, maybe this is a str…

A type system isn't going to save you from users submitting all kinds of potentially different encodings. Which also depends on what kind of user input is being handled: Is it OS-provided UI? Is it something being sent to a service accessible on the internet? Is it from a CLI? Is it from a file? Context matters for the potential space of what kind of data you might be operating on, which could require different ways of either knowing what kind of data you have based on having more control over the input versus having to detect stuff (or be told, correctly) from highly arbitrary things like reading from a file. All of that is external to the type system, and requires doing something before you can tag it with the correct type. Some languages might attempt to detect this stuff for you, but that could potentially be considered a language defect if it's hard to detect what a string is without having other input telling you what that string contains, such as a header in an HTTP request saying that it's UTF-8.

Re: Escaping user input is ridonkulously hard

#24

Earlier quoted context omitted.

> Firstly, a string could be raw unknown bytes, verified UTF-8, or UCS-2 (or even UTF-16 or UCS-4), and you absolutely need to know which it is. This is a language defect. If your language was invented in the 1960s it's an understandable defect, but it's still a defect. I do not want to write computer software with strings in a language that doesn't even have an actual string type rather than "Eh, maybe this is a str…

A type system isn't going to save you from users submitting all kinds of potentially different encodings. Which also depends on what kind of user input is being handled: Is it OS-provided UI? Is it something being sent to a service accessible on the internet? Is it from a CLI? Is it from a file? Context matters for the potential space of what kind of data you might be operating on, which could require different ways…

"A type system isn't going to save you from users submitting all kinds of potentially different encodings."

Yes, it is, because you give that a type that indicates you don't know what the encoding is, like RawInput or something. You then can not pass this type to any other function that doesn't explicitly call for that type. If you have some function that accepts it, blindly casts it to UTF-8, and slams it out into a file, well, that's not the type system's fault [1].

Of course a type system won't prevent you from still just being wrong or writing bugs; nobody promises that, not even the formal methods advocates. But it will prevent you from just accidentally blindly shoveling it out somewhere it doesn't belong without ever examining it or thinking about it.

I think you may be believing in a popular myth about strong typing systems, that they are designed to somehow prevent bad data from coming in to your system at all. You correctly identify that as impossible. But what strong typing systems can do is force you to deal with the fact that bad data may be coming in. On the outside, you have the chaos of, say, a bag of bytes that may or may not be JSON. On the inside, you have a "type SomeStruct { int a; int b }". A strong type systems forces you to write some sort of adapting code between those two, and guarantees that the result of that adapting code will be only and exactly the type that comes out of that adapting code, no "whoops, sometimes this dynamic code just returns a string, or maybe a network socket, or who knows what". Nothing can prevent your HTTP API from receiving a JPG of an anime character instead of JSON specifying a user to delete, but a strong type system can make you deal with that immediately and fully, instead of garbage data of indeterminate type floating through the system for an indeterminate period of time.

[1]: Also note there are a lot of "strong type systems" in the world that still fail to take advantage of their own capabilities and let bare string types and such float around too much. There are reasons why libraries must support the lowest common denominator; a file is a series of bytes with no further constraints, so the lowest level API has no choice but to accept that, but higher level APIs should more often take more restricted types. That strong type systems can save you from this doesn't mean they all do. I have a number of wrapper types in various languages just to add these guarantees to my programs not provided by the underlying libraries, though I also have some code that just wraps the underlying libraries that can't help but correctly take raw bytes at the lowest level.

Re: Escaping user input is ridonkulously hard

#25
It really isn't. Proof: Most people who try, largely succeed. Those who do something silly like try to do it 100% manually generally rapidly realize that's not a good plan, and usually there is a not-very-hard way to encapsulate it somehow, since that's pretty much what our languages do, encapsulate things.

I'm not saying it's completely trivial or that there's never an issue here or there. What I'm saying is, it's on par with any of dozens of other issues in programming. Bugs happen, errors happen, but no more so than anyone else. A series of systems with slightly different encoding practices can also cause some headaches, but, again, these are on par with a number of other issues that can emerge in such systems, not especially bad. I've seen a lot of crappy code that gets this wrong at scale, written by programmers who don't really know or care what they're doing, but the same code was crap in a dozen other ways too, and generally screwed up even easier things as well.

Where you get the problems are, from largest to smallest, 1. People who don't realize it's an issue at all and concatenate everything and 2. People who have just been taught about it, and are doing a wrong thing, most often trying to filter on the way "in" instead of the way "out". ("Sanitize user input" delenda est. Stop saying it. It's wrong.) Which is also not an exceptional case, because again there are any number of things that have the exact same characteristics in the programming world.

I would expect "ridonkulously hard" to encompass something that even when tried is super hard and often a failure, and this isn't that case.

Re: Escaping user input is ridonkulously hard

#26
post #20

Earlier quoted context omitted.

There is no such thing as an "escaped string". Escaping is not a general concept, it is something that differs given the intended destination of that string. For example, "I am a %SYSTEM% person" is a perfectly fine escaped bash string, but an unsafe CMD string; it is also fine as a C# format string, but potentially unsafe as a C format string, depending on your actual implementation of printf; it is an escaped MSSQL…

Sure, you need a different type for each different form of escaping you want to track, but that doesn't make the idea unworkable. A type that says (say) "this is a string containing html PCDATA" is a useful thing to have.

The biggest problem appears once you start wanting to combine such strings.

Say a user inputs some raw text in a form that is intended to be the title of a button in a HTML form that will be sent in a JSON file to be stored in a SQL db. The expectation is that you can later retrieve this HTML snippet from the DB and display it on the screen.

You have to first escape the raw text from the user so that it can be safely used in HTML - so you will go from user_input_string to html_pcdata_escaped_ user_input_string. Then you compose a bit of HTML that contains the button and this part; let's say you store it in some HTML DOM object. Then you want to send this HTML object as JSON, so you have to know to convert html_pcdata_escaped_ user_input_string into json_string_escaped_user_input_string - but that loses type information which may hurt us later, so maybe we want to actually store it as json_string_escaped_html_pcdata_escaped_user_input_string. Then, if we want to use this as part of an SQL query string, by the same considerations, we want to put it in a mysql_like_filter_escaped_json_string_escaped_html_pcdata_escaped_user_input_string - which is getting really ugly, and easy to mess up.

Of course, the order of escaping matters, so an mysql_like_filter_escaped_json_string_escaped_html_pcdata_escaped_user_input_string and a json_string_escaped_mysql_like_filter_escaped_html_pcdata_escaped_user_input_string are different things that need to be decoded differently (of course, for SQL in particular we could use prepared queries instead).

Also, we can't ever concatenate this with any other string-like type until perhaps the final use point (such as sending a query string to the DB), since we need to remember which part of the string is escaped in which way, and for what types of uses it is safe (an HTML-escaped string may still contain SQLi or JSON injection).

The point is that even with proper types, this is not easy to manage or fix.

It also requires quite advanced type systems to be able to use these in normal contexts - say, you want to store several such strings with different provenances in a Map or Set or even List, without "forgetting" the provenance.

Re: Escaping user input is ridonkulously hard

#27
post #3
post #2

It's not, though. It's the easiest thing in the world: Just use a library that never emits unescaped content by default, or if you make a single-character typo. The problem is that most of the libraries aren't that.

Escaping isn't always a yes/no question. If someone enters "foo bar" into your frontend, should the backend only see "foo%20bar" ?

no, the backend has no reason to see `foo%20bar` - you escape when you're combining that string with other strings (ie into HTML, into a SQL query, etc.)

Re: Escaping user input is ridonkulously hard

#28
post #20

Earlier quoted context omitted.

Sure, you need a different type for each different form of escaping you want to track, but that doesn't make the idea unworkable. A type that says (say) "this is a string containing html PCDATA" is a useful thing to have.

The biggest problem appears once you start wanting to combine such strings. Say a user inputs some raw text in a form that is intended to be the title of a button in a HTML form that will be sent in a JSON file to be stored in a SQL db. The expectation is that you can later retrieve this HTML snippet from the DB and display it on the screen. You have to first escape the raw text from the user so that it can be safely…

As far as I'm concerned, "the point is" that what we're describing here -- keeping track of the type of each piece of data -- is the best way to think about this class of problems (as opposed to talking about "sanitising" or "safe data", for example).

It's then up to us to decide how to best make use of the type system of whatever language we end up implementing it in (or, indeed, to treat the ability to deal with this well as a requirement when we're choosing a language).

For me, effects like "we can't ever concatenate this with any other string-like type" are desirable features, not problems with this approach: either it's possible to convert both strings to a common form, or I shouldn't be trying to combine them.

Re: Escaping user input is ridonkulously hard

#29
post #12

Earlier quoted context omitted.

> it’s of no help to you if your templating engine generates content escaped for MSSQL when you’re not going to put it in MSSQL. Allow me to complain a bit about MSSQL. When you're escaping a LIKE expression for MSSQL, you must also escape the "[" character, since it's a wildcard for MSSQL (and nowhere else except AFAIK Sybase). When you're escaping a LIKE expression for other databases, you must not escape the "[" c…

> When you're escaping a LIKE expression for other databases, you must not escape the "[" character, since some databases reject escaping anything other than the % and _ wildcards. That is, your escaping code for a LIKE expression has to be database-specific, because MSSQL (and AFAIK Sybase, it seems both have a common ancestor) decided to be different. TBF you may need custom codepaths because defaults diverge as we…

> TBF you may need custom codepaths because defaults diverge as well, IIRC postgres and sqlite default to ESCAPE '\' while mssql and oracle default to ESCAPE '' (the latter being the actual spec behaviour).

The trick is to just avoid the default, and always use an explicit ESCAPE, which should work the same on every database (except mysql without NO_BACKSLASH_ESCAPES in which you also have to escape the backslash itself, otherwise it will escape the closing quote and get very confused, but that issue can be avoided by using a character other than backslash as the escape character).

Re: Escaping user input is ridonkulously hard

#30
post #28

Earlier quoted context omitted.

The biggest problem appears once you start wanting to combine such strings. Say a user inputs some raw text in a form that is intended to be the title of a button in a HTML form that will be sent in a JSON file to be stored in a SQL db. The expectation is that you can later retrieve this HTML snippet from the DB and display it on the screen. You have to first escape the raw text from the user so that it can be safely…

As far as I'm concerned, "the point is" that what we're describing here -- keeping track of the type of each piece of data -- is the best way to think about this class of problems (as opposed to talking about "sanitising" or "safe data", for example). It's then up to us to decide how to best make use of the type system of whatever language we end up implementing it in (or, indeed, to treat the ability to deal with th…

Sure - I'm just saying that the article is right that this problem is difficult, not easy, and that it doesn't get significantly easier if we accurately keep track.
Post reply on HN