Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Escaping user input is ridonkulously hard

codeofhonor.substack.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: Escaping user input is ridonkulously hard

#61

Post by jiggawatts »

simiones wrote: post #26
mjw1007 wrote: post #20

Earlier quoted context omitted.

Sure, you need a different type for each different form of escaping you want to track, but that doesn't make the idea unworkable. A type that says (say) "this is a string containing html PCDATA" is a useful thing to have.

The biggest problem appears once you start wanting to combine such strings. Say a user inputs some raw text in a form that is intended to be the title of a button in a HTML form that will be sent in a JSON file to be stored in a SQL db. The expectation is that you can later retrieve this HTML snippet from the DB and display it on the screen. You have to first escape the raw text from the user so that it can be safely…

> The point is that even with proper types, this is not easy to manage or fix.

In practice, in a typed language, nothing like this ever occurs, because the rule is just: "use string for everything, except the edge".

You're thinking of a type like: HtmlString>>

In practice the type that is "passed around" is almost always just "string", and this is converted at the last moment to a single destination format, such as HtmlString.

When writing to databases, there isn't even an escape step at all, because you use parametrised queries, right? Right!?

The database stores "string", not "DatabaseEscapedString".

This is similar to how instants in time ought to be handled. You store them as UTC and convert to the user's time zone at the last moment. You don't pass around some monstrosity that somehow keeps track of +10-5+3 in order to arrive at +7. That would be absurd. Instead you pass around the "Z" UTC timestamp and add +7 when needed.

Re: Escaping user input is ridonkulously hard

#62

Post by jiggawatts »

8n4vidtmkvmk wrote: post #55
evilDagmar wrote: post #40

Earlier quoted context omitted.

There's just a whole library of CVEs for people who attempt to escape things being sent to SQL. Use parameterized queries already.

every one says "just use parameterized queries" but they don't handle arrays which makes the idea rather useless.

Many database engines can handle arrays, or table-valued variables which are basically the same thing. Most ORMs will also abstract away arrays for you, so you as the developer never need to deal with escaping of data in arrays.

Re: Escaping user input is ridonkulously hard

#63

Post by jiggawatts »

evilDagmar wrote: post #41

I've always found it more useful to just discard user input that doesn't come in the format you're asking for, and bail on the entire operation. Like, if the user might be attempting something fishy, there's no reason to try and "clean it up" and have your program "do it's best" with the remainder. Throw an error back at the user and move on to the next query.

User: "My surname is O'Neill"

Server: HTTP/403, begone with you, foul SQL-injecting hacker!

Re: Escaping user input is ridonkulously hard

#64

Post by jiggawatts »

kayodelycaon wrote: post #23

Ruby on Rails pretty much handles this. Regular strings are always escaped in views. Only html_safe strings will emit html. For user input, you should always use the sanitize method instead of raw. :)

Razor pages in ASP will do this too: https://learn.microsoft.com/en-us/dotnet/api/system.web.ihtm...

Re: Escaping user input is ridonkulously hard

#65

Post by tyingq »

teddyh wrote: post #59
tyingq wrote: post #54

Earlier quoted context omitted.

Encoding was one example. But "string" is language dependent aside from encoding. Strong typing doesn't solve the inherent issue.

Strong typing never “solved” anything, except that is forces you to see the problem and solve it yourself, explicitly, instead of relying on weak typing to fudge the types for you.

Maybe. Ambiguity means some random bags of bytes pass as more than one type.

Re: Escaping user input is ridonkulously hard

#66

Post by tinus_hn »

teddyh wrote: post #7

An alternate view: “string” is not a granular enough type, just like “bitfield” is not a type. Firstly, a string could be raw unknown bytes, verified UTF-8, or UCS-2 (or even UTF-16 or UCS-4), and you absolutely need to know which it is. But let’s assume that you’ve been a diligent programmer and filtered all that at the edges, and now have a sequence of Unicode code points (or possibly graphemes). You still need to…

No. The way to solve this is to recognize where the problem lies. The problem does not lie with storing user input. The problem lies with improperly putting strings in other data.

So all you need to do, is to do that properly. Either you commit to using constructs like paramtrized queries instead of concatenizing strings and use the DOM to put together HTML the way you want, or you escape as you concatenate the strings.

Don’t store escaped strings, it’s a recipe for disaster.

Re: Escaping user input is ridonkulously hard

#67

Post by masklinn »

teddyh wrote: post #60
masklinn wrote: post #47

Earlier quoted context omitted.

> I can’t escape the string beforehand, since the escaping rules are all different. You’re still misunderstanding. You shouldn’t escape at any point, instead you should mark things as safe as early as possible. “Safe” almost always has a single context, you don’t care if it’s going to go somewhere else because it’s not safe for there. Anything that’s not marked as safe is then automatically considered unsafe and proc…

> It’s not an issue, because by default nothing is safe anywhere, so all those APIs should treat the injected data thus. No library does this, since it does not know what strings I send it with their literal meaning intended, and which strings I send it with their escape characters intended to be interpreted. The escape characters are part of the API of that library. The library does not accept “strings” as such, it…

> No library does this

Most modern templates do exactly that. Jinja certainly does.

> The library does not accept “strings” as such, it accepts “escaped” strings. And since my program deals with normal unescaped strings, I have to escape the strings before I send them to the API.

That’s the problem with the library. That is what needs to be fixed.

> I have a feeling that you have a different meaning of the word “escaped” than me.

Add “explicit” to the first occurrence if you don’t understand without it.

Re: Escaping user input is ridonkulously hard

#68

Post by masklinn »

neon_electro wrote: post #46
masklinn wrote: post #6

Earlier quoted context omitted.

> It's the easiest thing in the world: Just use a library that never emits unescaped content by default That doesn't make any sense? Escaping is a function of the consumer, not the producer. Hell, most of the problematic content doesn't come from a library to start with. And if your Markdown -> HTML converter produces escaped content... it's not a Markdown -> HTML converter, because the result is not HTML. More broad…

Is Rails' `html_safe` an example of what you're referring to? https://apidock.com/rails/String/html_safe

Might be, although it’s unclear how composition works.

Especially with Ruby having string interpolation.

Re: Escaping user input is ridonkulously hard

#69

Post by simiones »

jiggawatts wrote: post #61
simiones wrote: post #26

Earlier quoted context omitted.

The biggest problem appears once you start wanting to combine such strings. Say a user inputs some raw text in a form that is intended to be the title of a button in a HTML form that will be sent in a JSON file to be stored in a SQL db. The expectation is that you can later retrieve this HTML snippet from the DB and display it on the screen. You have to first escape the raw text from the user so that it can be safely…

> The point is that even with proper types, this is not easy to manage or fix. In practice, in a typed language, nothing like this ever occurs, because the rule is just: "use string for everything, except the edge". You're thinking of a type like: HtmlString >> In practice the type that is "passed around" is almost always just "string", and this is converted at the last moment to a single destination format, such as…

> In practice the type that is "passed around" is almost always just "string"

That's what happens in practice, of course. The GP was proposing something else, and I was explaining how complicated that gets.

> and this is converted at the last moment to a single destination format, such as HtmlString.

I explained before why this doesn't work unless we're talking about the final destination of this string. Otherwise, if that string is being taken through various encodings (say user input to JSON to sprintf format string to HTTP body), and if you need to combine safe and unsafe input, then what you're saying doesn't work anymore.

Here is a sketch of an example:

  userInput := read()
  jsonFormat := "{\"context\": \"%s\", \"input\": \"" + json.escape(userInput) + "\"}" //easy and safe
  finalJson := ""
  sprintf(finalJson, jsonFormat, "some context") 
  // oops - unsafe if original input was "%s"
  sprintf(finalJson, printf.escape(jsonFormat), "some context") 
  // oops - does the wrong thing - it will output "{\"context\": \"%s\", \"input\": \"%s"\"}
  
  //let's try the other way around?

  userInput := read()
  formatStr := "{\"context\": \"%s\", \"input\": \"" + printf.escape(userInput) + "\"}" //easy and safe
  finalJsonStr := ""
  sprintf(finalJson, formatStr, "some context") 
  // oops - unsafe if userInput was "safe-looking\", \"bypassAuth\": \"true\"}" 
  
  sprintf(finalJson, json.escape(formatStr), "some context") 
  // oops - does the wrong thing - it will output 
  // "\"{\\\"context\\\":\\\"some context\\\", \\\"input\\\": \\\"safe-looking\\\", \\\"bypassAuth\\\": \\\"true\\\"}\"" - that is, a JSON string instead of a JSON object
The only solution to get this to work is to keep the user input string entirely separate from any other string, and apply escaping to it individually at every level where it is used.

Additionally, you will need to remember what escaping has been applied to it, and in what order, so that it can be un-escaped back to the original value when needed.

Re: Escaping user input is ridonkulously hard

#70

Post by jerf »

tyingq wrote: post #31
jerf wrote: post #24

Earlier quoted context omitted.

"A type system isn't going to save you from users submitting all kinds of potentially different encodings." Yes, it is, because you give that a type that indicates you don't know what the encoding is, like RawInput or something. You then can not pass this type to any other function that doesn't explicitly call for that type. If you have some function that accepts it, blindly casts it to UTF-8, and slams it out into a…

>If you have some function that accepts it, blindly casts it to UTF-8 Unfortunately, if you interact with services you didn't write, you're usually back to getting "strings" of unknown encoding, and typically requirements that force some blind or semi-blind guessing.

Blind guessing is not related to the type system. Nobody has claimed type systems can solve that. What they can do is force you to guess, and make it clear where that is occurring.

This, again, goes back to a very broken understanding of types systems that I often see, and once held myself. The claim of type systems is not that they magically go out into the world and fix the external world to be well-typed; the claim is that it forces your code to deal with the conversion of the external world into a clean internal representation, and presumably, to have a clean error pathway when that fails. Dynamically-typed code will let you float along much more easily. Statically-typed code can still be written that way, but at least then it's poor statically-typed code. In some circles that sort of broken dynamic code is essentially idiomatic. (Though that is fading away as every year more programmers learn how bad an idea that is.)

Post reply on HN