How fast can you validate UTF-8 strings in JavaScript?
1–10 of 18 posts
Re: How fast can you validate UTF-8 strings in JavaScript?
#2The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?
Re: How fast can you validate UTF-8 strings in JavaScript?
#3The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?
As on example, you could mislead users by replacing valid codepoints (characters) with visually identical ones, for example in URLs.
Re: How fast can you validate UTF-8 strings in JavaScript?
#4Not tested is https://www.npmjs.com/package/utf-8-validate (1,624,873 weekly downloads) - which I use.
Re: How fast can you validate UTF-8 strings in JavaScript?
#5The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?
There are functions that will throw on invalid UTF-8. Perhaps not a full security risk but unexpected errors are always a danger.
Re: How fast can you validate UTF-8 strings in JavaScript?
#6Not tested is https://www.npmjs.com/package/utf-8-validate (1,624,873 weekly downloads) - which I use.
From https://github.com/websockets/utf-8-validate/tree/master/dep...
> This C++ library is part of the JavaScript package utf-8-validate. The utf-8-validate package is routinely downloaded more than a million times per week.
> If you are using Node JS (19.4.0 or better), you already have access to this function as buffer.isUtf8(input).
And at https://github.com/websockets/utf-8-validate/tree/master/dep...
> # Reference
> John Keiser, Daniel Lemire, [Validating UTF-8 In Less Than One Instruction Per Byte](https://arxiv.org/abs/2010.03090), Software: Practice & Experience 51 (5), 2021
Re: How fast can you validate UTF-8 strings in JavaScript?
#7The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?
As on example, you could mislead users by replacing valid codepoints (characters) with visually identical ones, for example in URLs.
Wouldn’t "visually identical” tokens still be valid utf8?
Re: How fast can you validate UTF-8 strings in JavaScript?
#8The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?
As on example, you could mislead users by replacing valid codepoints (characters) with visually identical ones, for example in URLs.
That's not what invalid UTF-8 means. It doesn't mean replacing codepoints with other codepoints, it means that the encoding of the codepoints is invalid. Since JavaScript uses UTF-16 strings, it's possible to create JavaScript strings that don't have a valid UTF-8 representation (I believe unpaired surrogates can do this). Also, if you are working with raw bytes, there is no guarantee that it has any encoding at all (this seems to be what the article is about). There is WTF-8 [0] that proposes a possible solution to the UTF-16 problem, but purely as an internal representation and not as an interchange format. I think the Rust standard library uses WTF-8 to represent Windows wide-strings, which allows ASCII and UTF-8 algorithms to work on them, while still being able to represent any UTF-16 string.
Re: How fast can you validate UTF-8 strings in JavaScript?
#9Earlier quoted context omitted.
As on example, you could mislead users by replacing valid codepoints (characters) with visually identical ones, for example in URLs.
Wouldn’t "visually identical” tokens still be valid utf8?
It depends on the meaning of 'valid UTF-8'. They could be 'UTF-8 that is invalid for my application', but still be 'strings that are valid under the UTF-8 specification'.
Re: How fast can you validate UTF-8 strings in JavaScript?
#10Looks like "...validate UTF-8 strings in Node.js" is a more accurate title, as valid8 runs on Node.js and the other uses a Node.js std library function. The TextDecoder method should work in the browser, but I don't see why you want to do this kind of validation in the browser.