Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Favorite Bugs: Invalid Surrogate Pairs

george.mand.is

Post reply on HN
Print view View original

41–50 of 53 posts

Re: My Favorite Bugs: Invalid Surrogate Pairs

#41

Post by ucarion »

chrismorgan wrote: post #18

A CRDT library working at the code unit level? Ouch. Of course that’s going to go wrong, it was inevitable. As for using extended grapheme clusters, it sounds a little bit iffy—maybe possible to use correctly, maybe not, because they’re not stable over time. That style of thing has created some fascinating bugs, like (a few years ago) index corruption in PostgreSQL due to collation changes. Unicode scalar values are…

> I do not agree that slice() should operate on extended grapheme clusters. Don’t lump the grapheme cluster/scalar value split in with the sins of UTF-16 and its unreliable code point/code unit split.

Maybe a simpler argument against this idea is that the definition of an extended grapheme cluster changes between versions of Unicode. The relevant standard is on its 47th revision (not all of which change extended grapheme clusters, but many do): https://www.unicode.org/reports/tr29/

Re: My Favorite Bugs: Invalid Surrogate Pairs

#42

Post by denzen »

I hit a similar problem in an application that was performing non-unicode-aware string operations on user-submitted text in a typescript codebase. The data couldn't be processed by an external service that was expecting valid Unicode. My fix was using toWellFormed: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Re: My Favorite Bugs: Invalid Surrogate Pairs

#43

Post by RedNifre »

agus4nas wrote: post #8

Great write-up. Do most modern languages handle invalid surrogates gracefully, or is it still a "good luck" situation depending on the runtime?

It depends on the language and/or used libraries. E.g. in Go, the problem does not exist, because it uses UTF-32; Rust uses UTF-8, but it makes sure that you can't cut a string between bytes that belong to the same character.

Fun Java/macos quirk: macos normalizes file names, so you can't have two files called ü in the same directory by writing ü as a single character and as composing characters. But unfortunately, this only happens on write, not on read, so if you type an ü on a German keyboard (produces a single character) into the Java source code file when writing a file name, the file will be saved with the decomposed name instead, but when trying to open the file, it will not be found when trying to open it with the single character name.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#45

Post by jandrese »

usrnm wrote: post #28
201984 wrote: post #24

Earlier quoted context omitted.

Why wouldn't 8 be enough? Surely 18,446,744,070,000,001,024 characters is enough for every writing system in the world.

Because that's not how Unicode works. It's not simply a table mapping numbers to all possible symbols. Unicode is full of special codepoints that have no meaning on their own, they serve as modifiers to other symbols and a single visible symbol can be formed by an arbitrary (in theory) long combimation of such codepoints. It doesn't matter how you encode it, it simply doesn't work as "codepoint -> symbol" and indexin…

I actually wonder if the combinatoral explosion of attempting to enumerate every possible character combination would exceed 2^64 bits. My intuition is that it might, and also such a system would be unworkably unwieldy. The size of the spec document would also suffer from the combinatoral explosion. Imagine a system that tries to encode a unique entry for every possible Zalgo character.

Also, literally nobody wants to use 64 bit values to encode ASCII values. Even in our world of insanely large storage that would be breathtakingly wasteful.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#46

Post by shawn_w »

chuckadams wrote: post #16
amluto wrote: post #9

Earlier quoted context omitted.

Modern string libraries largely use UTF-8 [0], and surrogates, regardless of whether they’re paired, are invalid in UTF-8. So, in a modern string library, as built in to most modern languages, you will not encounter surrogates except when translating between encodings. [0] But everyone disagrees as to what indexing a string means, so you need to make an actual choice if you want anything involving indexing to match a…

> surrogates, regardless of whether they’re paired, are invalid in UTF-8 Java did not get the memo. Since the char type is fixed at 16 bits, it uses surrogates to encode everything outside the BMP, regardless of the encoding.

If you use the string methods that work with code points instead of chars, you rarely if ever have to deal with surrogate pairs in Java.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#47

Post by chrismorgan »

Someone wrote: post #27
chrismorgan wrote: post #20

Earlier quoted context omitted.

> Unicode code points are 32 bit 21-bit, actually. It was supposed to be 32-bit, but UTF-16 caps out at 21-bit, so they lopped eleven bits of potential from Unicode (and UTF-8, so no more six-byte encoding). > at some point before Unicode No, in the early days of Unicode. > run length encodes Um… what? RLE is a data compression thing, UTF-16 has nothing to do with it.

>> Unicode code points are 32 bit > 21-bit, actually Less than that. https://en.wikipedia.org/wiki/Code_point#In_character_encodi... : “The Unicode code space is divided into seventeen planes (the basic multilingual plane, and 16 supplementary planes), each with 65,536 (= 2¹⁶) code points. Thus the total size of the Unicode code space is 17 × 65,536 = 1,114,112” That makes it log(1,114,112)/log(2) bit. That’s about 2…

Sorry, I was thinking of 0x1FFFFF as the end, but it’s 0x10FFFF. Forgetful.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#48

Post by mananaysiempre »

bombela wrote: post #39
Dwedit wrote: post #21

Windows allows unmatched surrogate pairs in filenames, invalid for UTF-16. Likewise, Linux allows invalid UTF-8 byte sequences in filenames. Because invalid UTF-16 strings could show up in places within Windows, someone made a UTF-8 variant called "WTF-8", which allows unmatched surrogate pairs to survive a round trip.

Indeed, Linux allows anything but "/" and "\0" in filenames. Those days its reasonable to refuse utf8 filenames. But one must always validate first!

> Indeed, Linux allows anything but "/" and "\0" in filenames.

For what it’s worth, NT allows any 16-bit quantity but L'\\' (0x005C) in filenames (even nulls); it’s the Win32 layer on top of it that imposes all the other weird restrictions and mappings.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#49

Post by chrismorgan »

rectang wrote: post #26
chrismorgan wrote: post #18

A CRDT library working at the code unit level? Ouch. Of course that’s going to go wrong, it was inevitable. As for using extended grapheme clusters, it sounds a little bit iffy—maybe possible to use correctly, maybe not, because they’re not stable over time. That style of thing has created some fascinating bugs, like (a few years ago) index corruption in PostgreSQL due to collation changes. Unicode scalar values are…

> I still can’t work out why it wasn’t obvious from the start that UCS-2 would never be enough) Surely certain people did know, but those people weren't in a position to do anything about it. Specifically, there were surely people who knew that because historical Chinese place names, Japanese nicknames, and so on, were not included in the original "Unicode" (it wasn't called UCS-2 yet) it was insufficient for complet…

There was never an adequate safety margin for anything but immediate (less than five year horizon) use—even at Unicode 1.1 it was more than half full, and they knew they weren’t done. And yet all kinds of major companies put all their eggs in that basket, and then doubled down with the monstrosity that is UTF-16, rather than backing out and going with UTF-8 instead, even though I strongly suspect it would have been easier for everyone involved in most cases, compared to the whole wchar shemozzle. Instead it took Windows twenty-five years to bridge the gap with a UTF-8 codepage (65001) that actually worked.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#50

Post by chrismorgan »

georgemandis wrote: post #29
chrismorgan wrote: post #23

Earlier quoted context omitted.

No, the language did not handle it fine. It allowed an invalid Unicode string to exist. This is basically a UTF-16 affliction— nothing that does UTF-16 validates, whereas almost everything that does UTF-8 does validate. encodeURIComponent deals with UTF-8, so of course it throws.

I'm realizing `encodeURIComponent` is actually part of the ECMA spec! I thought it was something provided by the browser like `window` or `document`. I withdraw my "the language handled it fine" comment, haha. Before I'd looked that up I was going to say: I feel like "don't allow an invalid Unicode string to exist all" feels like a separate/bigger problem to me from "handling it fine" when they do get created. To the…

In Rust, an invalid Unicode string simply cannot exist (* unless you use unsafe, but all bets are off then). An important part of this is that the code unit, the scalar value and the string are three different types (u8, char, str). Iteration must decide if it wants to go by code unit or by scalar value (… or by extended grapheme cluster, but that’s not provided in std).

JavaScript’s problems start with not having separate code unit or scalar value types. Sequences of UTF-16 code units, individual UTF-16 code units and scalar values all use the type string. (Code unit and scalar value also both use number in some contexts.)

The first step to fixing JavaScript’s bad semantics would be separating the code unit and scalar value types. If you did that… the changes required to support strict strings are perhaps surprisingly small. Even migrating to UTF-8 semantics is not very hard then.

Unfortunately, JavaScript seems very determined to do stupid things and allow stupid things and then do more stupid things with the stupid things it foolishly allowed.

Post reply on HN