Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Escaping user input is ridonkulously hard

codeofhonor.substack.com

Post reply on HN
Print view View original

11–20 of 88 posts

Re: Escaping user input is ridonkulously hard

#11

Post by nl »

phyzome wrote: post #8
masklinn wrote: post #6

Earlier quoted context omitted.

> It's the easiest thing in the world: Just use a library that never emits unescaped content by default That doesn't make any sense? Escaping is a function of the consumer, not the producer. Hell, most of the problematic content doesn't come from a library to start with. And if your Markdown -> HTML converter produces escaped content... it's not a Markdown -> HTML converter, because the result is not HTML. More broad…

I can give a more detailed response later, but... « Escaping is a function of the consumer, not the producer » This is incorrect. The producer emits something in a language , be it HTML or JSON or HTTP headers or whatever. Data must be encoded properly for that language. The consumer must then decode, of course, so in a sense it is the job of both. But the onus is really on the producer.

Too bad if your consumer has to interact with anything that could be malicious in any circumstances!

Consumers must properly escape any input.

Re: Escaping user input is ridonkulously hard

#12

Post by cesarb »

masklinn wrote: post #9
phyzome wrote: post #8

Earlier quoted context omitted.

I can give a more detailed response later, but... « Escaping is a function of the consumer, not the producer » This is incorrect. The producer emits something in a language , be it HTML or JSON or HTTP headers or whatever. Data must be encoded properly for that language. The consumer must then decode, of course, so in a sense it is the job of both. But the onus is really on the producer.

> This is incorrect. The producer emits something in a language, be it HTML or JSON or HTTP headers or whatever. Data must be encoded properly for that language. Which is the consumption side. When you send data to an HTML template engine, it’s escaped as input, meaning with the template engine as consumer, not with the template engine as producer. It may be a “pipeline” situation where the consumer also produces som…

> it’s of no help to you if your templating engine generates content escaped for MSSQL when you’re not going to put it in MSSQL.

Allow me to complain a bit about MSSQL.

When you're escaping a LIKE expression for MSSQL, you must also escape the "[" character, since it's a wildcard for MSSQL (and nowhere else except AFAIK Sybase). When you're escaping a LIKE expression for other databases, you must not escape the "[" character, since some databases reject escaping anything other than the % and _ wildcards. That is, your escaping code for a LIKE expression has to be database-specific, because MSSQL (and AFAIK Sybase, it seems both have a common ancestor) decided to be different.

Re: Escaping user input is ridonkulously hard

#13

Post by deredede »

nl wrote: post #11
phyzome wrote: post #8

Earlier quoted context omitted.

I can give a more detailed response later, but... « Escaping is a function of the consumer, not the producer » This is incorrect. The producer emits something in a language , be it HTML or JSON or HTTP headers or whatever. Data must be encoded properly for that language. The consumer must then decode, of course, so in a sense it is the job of both. But the onus is really on the producer.

Too bad if your consumer has to interact with anything that could be malicious in any circumstances! Consumers must properly escape any input.

That doesn't make sense to me and I agree with GP. If I consume HTML and I escape all HTML input I'm given, I'm utterly useless.

Now when I consume text and convert that text into HTML for further treatment, I'm producing HTML, and I must properly escape my input in that conversion. The escaping is only needed because I produce HTML. In fact the only time escaping can be done is when producing data, because if unescaped data is ever produced, the cat's out of the bag.

Edit: Actually think that producer/consumer is a wrong way to talk about this. Escaping only ever occurs at a boundary when transforming between formats (eg from "text string" to "html string") which is always both producer (of the new format) and consumer (of the old format). But it can always be thought of as a type cast, with possible type confusions when input and output formats share the same machine representation (eg string).

Re: Escaping user input is ridonkulously hard

#14

Post by rileymat2 »

nl wrote: post #11
phyzome wrote: post #8

Earlier quoted context omitted.

I can give a more detailed response later, but... « Escaping is a function of the consumer, not the producer » This is incorrect. The producer emits something in a language , be it HTML or JSON or HTTP headers or whatever. Data must be encoded properly for that language. The consumer must then decode, of course, so in a sense it is the job of both. But the onus is really on the producer.

Too bad if your consumer has to interact with anything that could be malicious in any circumstances! Consumers must properly escape any input.

I suspect you are having a vigorous debate about the ill defined “producer consumer” terminology and probably agree.

Re: Escaping user input is ridonkulously hard

#15

Post by masklinn »

cesarb wrote: post #12
masklinn wrote: post #9

Earlier quoted context omitted.

> This is incorrect. The producer emits something in a language, be it HTML or JSON or HTTP headers or whatever. Data must be encoded properly for that language. Which is the consumption side. When you send data to an HTML template engine, it’s escaped as input, meaning with the template engine as consumer, not with the template engine as producer. It may be a “pipeline” situation where the consumer also produces som…

> it’s of no help to you if your templating engine generates content escaped for MSSQL when you’re not going to put it in MSSQL. Allow me to complain a bit about MSSQL. When you're escaping a LIKE expression for MSSQL, you must also escape the "[" character, since it's a wildcard for MSSQL (and nowhere else except AFAIK Sybase). When you're escaping a LIKE expression for other databases, you must not escape the "[" c…

> When you're escaping a LIKE expression for other databases, you must not escape the "[" character, since some databases reject escaping anything other than the % and _ wildcards. That is, your escaping code for a LIKE expression has to be database-specific, because MSSQL (and AFAIK Sybase, it seems both have a common ancestor) decided to be different.

TBF you may need custom codepaths because defaults diverge as well, IIRC postgres and sqlite default to ESCAPE '\' while mssql and oracle default to ESCAPE '' (the latter being the actual spec behaviour).

So in Postgres and SQLite you must always escape your LIKE parameter, while in mssql and oracle that's not the case.

Re: Escaping user input is ridonkulously hard

#16

Post by masklinn »

deredede wrote: post #13
nl wrote: post #11

Earlier quoted context omitted.

Too bad if your consumer has to interact with anything that could be malicious in any circumstances! Consumers must properly escape any input.

That doesn't make sense to me and I agree with GP. If I consume HTML and I escape all HTML input I'm given, I'm utterly useless. Now when I consume text and convert that text into HTML for further treatment, I'm producing HTML, and I must properly escape my input in that conversion. The escaping is only needed because I produce HTML. In fact the only time escaping can be done is when producing data, because if unesca…

> That doesn't make sense to me and I agree with GP. If I consume HTML and I escape all HTML input I'm given, I'm utterly useless. [...] Now when I consume text and convert that text into HTML for further treatment, I'm producing HTML, and I must properly escape my input in that conversion.

Which is my point, it's the consumption side which defines what the escaping should be.

> Escaping only ever occurs at a boundary when transforming between formats (eg from "text string" to "html string") which is always both producer (of the new format) and consumer (of the old format).

A database interface is not a transformer / producer, needs escaping. Globbing is not a transformer either. Still needs escaping.

Re: Escaping user input is ridonkulously hard

#17

Post by iLoveOncall »

After reading the article I fail to see what is hard about escaping user input.

It seems like what the author means is that it's hard to think of all the places where user input should be escaped, but even then, if you use any modern framework, everything is escaped by default.

Re: Escaping user input is ridonkulously hard

#18

Post by tialaramex »

teddyh wrote: post #7

An alternate view: “string” is not a granular enough type, just like “bitfield” is not a type. Firstly, a string could be raw unknown bytes, verified UTF-8, or UCS-2 (or even UTF-16 or UCS-4), and you absolutely need to know which it is. But let’s assume that you’ve been a diligent programmer and filtered all that at the edges, and now have a sequence of Unicode code points (or possibly graphemes). You still need to…

> Firstly, a string could be raw unknown bytes, verified UTF-8, or UCS-2 (or even UTF-16 or UCS-4), and you absolutely need to know which it is.

This is a language defect. If your language was invented in the 1960s it's an understandable defect, but it's still a defect. I do not want to write computer software with strings in a language that doesn't even have an actual string type rather than "Eh, maybe this is a string or maybe it's just some random bytes, who cares".

Only in very low level software should it make a difference whether the string is in fact represented as UTF-8 or UTF-16 or whatever, but Rust shows that you can write software at a low level and still enforce type safety for strings.

I agree though that here once again the Right Thing™ is a strong type system. If I've got a Microsoft Graph username, a URL, an email address and a UUID, that's four types, those are not four strings with human names to distinguish them. We don't need to escape some or any of these types - in their context.

Re: Escaping user input is ridonkulously hard

#19

Post by simiones »

teddyh wrote: post #7

An alternate view: “string” is not a granular enough type, just like “bitfield” is not a type. Firstly, a string could be raw unknown bytes, verified UTF-8, or UCS-2 (or even UTF-16 or UCS-4), and you absolutely need to know which it is. But let’s assume that you’ve been a diligent programmer and filtered all that at the edges, and now have a sequence of Unicode code points (or possibly graphemes). You still need to…

There is no such thing as an "escaped string". Escaping is not a general concept, it is something that differs given the intended destination of that string. For example, "I am a %SYSTEM% person" is a perfectly fine escaped bash string, but an unsafe CMD string; it is also fine as a C# format string, but potentially unsafe as a C format string, depending on your actual implementation of printf; it is an escaped MSSQL filter string, but not an escaped PostgresSQL filter string.

Also, not all strings/texts should be thought of as Unicode code points/graphemes.

Re: Escaping user input is ridonkulously hard

#20

Post by mjw1007 »

simiones wrote: post #19
teddyh wrote: post #7

An alternate view: “string” is not a granular enough type, just like “bitfield” is not a type. Firstly, a string could be raw unknown bytes, verified UTF-8, or UCS-2 (or even UTF-16 or UCS-4), and you absolutely need to know which it is. But let’s assume that you’ve been a diligent programmer and filtered all that at the edges, and now have a sequence of Unicode code points (or possibly graphemes). You still need to…

There is no such thing as an "escaped string". Escaping is not a general concept, it is something that differs given the intended destination of that string. For example, "I am a %SYSTEM% person" is a perfectly fine escaped bash string, but an unsafe CMD string; it is also fine as a C# format string, but potentially unsafe as a C format string, depending on your actual implementation of printf; it is an escaped MSSQL…

Sure, you need a different type for each different form of escaping you want to track, but that doesn't make the idea unworkable.

A type that says (say) "this is a string containing html PCDATA" is a useful thing to have.

Post reply on HN