11–18 of 18 posts
The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?
Earlier quoted context omitted.
Wouldn’t "visually identical” tokens still be valid utf8?
It depends on the meaning of 'valid UTF-8'. They could be 'UTF-8 that is invalid for my application', but still be 'strings that are valid under the UTF-8 specification'.
The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?
The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?
If you skip this initial validation, you open yourself to security problems caused by differences in the undefined behavior of these various routines that are assuming valid input. As an example, imagine if your code for escaping HTML saw an invalid byte sequence immediately before a "<" and failed to escape it because of that, but then the HTML parser you pass the result to treats the invalid byte sequence differently and successfully sees and parses the "<" that should have been escaped.
Earlier quoted context omitted.
It depends on the meaning of 'valid UTF-8'. They could be 'UTF-8 that is invalid for my application', but still be 'strings that are valid under the UTF-8 specification'.
There is no such thing as "UTF-8 that is invalid for my application". Any properly encoded sequence of UTF-8 codepoints is valid UTF-8. Your application may have constraints on the codepoints that it should accept, but that does not suddenly make them improperly encoded. Using visually-similar lookalikes does not make a UTF-8 string invalid.
In fact, yes there is! Your application may vary. :)
The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?
https://www.brainonfire.net/blog/2022/04/11/what-is-parser-m...
The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?
e.g. the developer might either deny or allow /admin/, or /admin/.*
But if a given Unicode string [1] has two representations, then that check may not work as intended for one of them.
---
UTF-8 represents code points with 1-4 bytes, which are roughly similar to base 64 digits (because a continuation byte is 10xx_xxxx, it has 6 bits of freedom)
This opens up the possibility of "overlong encodings", which a UTF-8 validator must reject.
That is, you can do the equivalent of representing a number as "09" instead of "9" -- that's an overlong encoding. A decoder may understand "09" and "9" as both being the digit 9, but it's a different sequence of bytes.
So a naive decoding algorithm can accept a spelling of "admin" as 6, 7, 8, ... 20 bytes, not 5. There are many overlong encodings!
The only valid UTF-8 spelling has 5 bytes, but decoders that don't validate will "naturally" accept more (in fact I think I even wrote one of these :-/ )
In summary, if you don't do UTF-8 validation, then one layer doesn't reject the invalid representation, and another layer may decode it into a valid Unicode string.
Also, some regex engines work on UTF-8 encoded bytes, and some work on arrays of code points.
---
Again I'd be interested in more real examples.
[1] A sequence of "Unicode scalars" -- code points not in the surrogate range
Earlier quoted context omitted.
There is no such thing as "UTF-8 that is invalid for my application". Any properly encoded sequence of UTF-8 codepoints is valid UTF-8. Your application may have constraints on the codepoints that it should accept, but that does not suddenly make them improperly encoded. Using visually-similar lookalikes does not make a UTF-8 string invalid.
> There is no such thing as "UTF-8 that is invalid for my application". In fact, yes there is! Your application may vary. :)