21–30 of 53 posts
Because invalid UTF-16 strings could show up in places within Windows, someone made a UTF-8 variant called "WTF-8", which allows unmatched surrogate pairs to survive a round trip.
Damn, I’ve never really had to deal with Unicode all that much. Was already bad enough that instead of bytes, we have to worry about code points. Now even that isn’t enough? It would have been expensive, but all characters should have been fixed size 64bit values.
> It would have been expensive, but all characters should have been fixed size 64bit values. It would have been a non-starter, and then we'd all be dealing with Shift-JIS, BIG5, and FSM knows how many different codepages to this day. UTF-8 is about as elegant as it gets, though Java and JS still managed to fuck that up too (they both encode every codepoint outside the BMP as surrogate pairs in UTF-8)
I can’t comment on Java, but JS I know reasonably well and I can’t think of any place it uses CESU-8.
Great write-up. Do most modern languages handle invalid surrogates gracefully, or is it still a "good luck" situation depending on the runtime?
The language handled it fine. It will generally just show replacement characters (�) for combos that don't map to anything. It was really `encodeURIComponent` that didn't handle it gracefully. If you just type this into the console (surrogate pair for cowboy smiley face emoji), you see it encodes it ("%F0%9F%A4%A0"): encodeURIComponent("\uD83E\uDD20") If you give it an invalid surrogate pair, it will throw an actual…
Damn, I’ve never really had to deal with Unicode all that much. Was already bad enough that instead of bytes, we have to worry about code points. Now even that isn’t enough? It would have been expensive, but all characters should have been fixed size 64bit values.
> It would have been expensive, but all characters should have been fixed size 64bit values You're making the same mistake that numerous people made before you: thinking that it's as simple as using arrays of large enough numbers. First they thought that two bytes per symbol would be enough, then four. Spoiler alert: it wasn't. And eight won't work either.
A CRDT library working at the code unit level? Ouch. Of course that’s going to go wrong, it was inevitable. As for using extended grapheme clusters, it sounds a little bit iffy—maybe possible to use correctly, maybe not, because they’re not stable over time. That style of thing has created some fascinating bugs, like (a few years ago) index corruption in PostgreSQL due to collation changes. Unicode scalar values are…
Yeah, I think that's fair. I didn't really think this through as I was writing it.
I'm not even so sure "ending up with nonsense" here is the worst outcome. It might be unavoidable with this approach and if that had been the only problem this bug might have been less memorable.
The real problem—which I mention didn't articulate/emphasize particularly well—was that these invalid surrogate pairs were getting passed into `encodeURIComponent` somewhere deep in the stack and choking catastrophically on them. That was the "real" bug at the end of the day, but the invalid surrogate pairs and the way they were getting created on the way were a fun journey to untangle.
A CRDT library working at the code unit level? Ouch. Of course that’s going to go wrong, it was inevitable. As for using extended grapheme clusters, it sounds a little bit iffy—maybe possible to use correctly, maybe not, because they’re not stable over time. That style of thing has created some fascinating bugs, like (a few years ago) index corruption in PostgreSQL due to collation changes. Unicode scalar values are…
Surely certain people did know, but those people weren't in a position to do anything about it.
Specifically, there were surely people who knew that because historical Chinese place names, Japanese nicknames, and so on, were not included in the original "Unicode" (it wasn't called UCS-2 yet) it was insufficient for complete expression of Asian languages.
There were also many people who objected to Han unification, which is a different problem.
But all of these objections were discarded because of the overwhelming mandate for a fixed-width encoding. The original "Unicode" was conceived as a "16-bit" initiative. Its 16-bit-ness was an essential aspect of the design and the Unicode Consortium did what they had to do to fit all scripts and characters "in modern use" into 16 bits.
From the Wikipedia article on Han Unification[1]:
> Some of the controversy stems from the fact that the very decision of performing Han unification was made by the initial Unicode Consortium, which at the time was a consortium of North American companies and organizations (most of them in California), but included no East Asian government representatives. The initial design goal was to create a 16-bit standard, and Han unification was therefore a critical step for avoiding tens of thousands of character duplications.
In summary, Unicode code points (characters) are 32 bit. JavaScript manipulates Unicode in utf-16 for historical reasons, because at some point before Unicode, 16 bit was deemed enough (ucs-2). utf-16 run length encodes Unicode 32 codepoints into one or two code units. Splitting in a middle of a codepoints produces one invalid half string, and one semantically different half string. emojies are a sequence of Unicode…
> Unicode code points are 32 bit 21-bit, actually. It was supposed to be 32-bit, but UTF-16 caps out at 21-bit, so they lopped eleven bits of potential from Unicode (and UTF-8, so no more six-byte encoding). > at some point before Unicode No, in the early days of Unicode. > run length encodes Um… what? RLE is a data compression thing, UTF-16 has nothing to do with it.
> 21-bit, actually
Less than that. https://en.wikipedia.org/wiki/Code_point#In_character_encodi...:
“The Unicode code space is divided into seventeen planes (the basic multilingual plane, and 16 supplementary planes), each with 65,536 (= 2¹⁶) code points. Thus the total size of the Unicode code space is 17 × 65,536 = 1,114,112”
That makes it log(1,114,112)/log(2) bit. That’s about 20,09.
(https://www.unicode.org/versions/Unicode17.0.0/ assigns 159,801 of them to characters)
Earlier quoted context omitted.
> It would have been expensive, but all characters should have been fixed size 64bit values You're making the same mistake that numerous people made before you: thinking that it's as simple as using arrays of large enough numbers. First they thought that two bytes per symbol would be enough, then four. Spoiler alert: it wasn't. And eight won't work either.
Why wouldn't 8 be enough? Surely 18,446,744,070,000,001,024 characters is enough for every writing system in the world.
Earlier quoted context omitted.
The language handled it fine. It will generally just show replacement characters (�) for combos that don't map to anything. It was really `encodeURIComponent` that didn't handle it gracefully. If you just type this into the console (surrogate pair for cowboy smiley face emoji), you see it encodes it ("%F0%9F%A4%A0"): encodeURIComponent("\uD83E\uDD20") If you give it an invalid surrogate pair, it will throw an actual…
No, the language did not handle it fine. It allowed an invalid Unicode string to exist. This is basically a UTF-16 affliction— nothing that does UTF-16 validates, whereas almost everything that does UTF-8 does validate. encodeURIComponent deals with UTF-8, so of course it throws.
Before I'd looked that up I was going to say: I feel like "don't allow an invalid Unicode string to exist all" feels like a separate/bigger problem to me from "handling it fine" when they do get created. To the extent I can hand JavaScript an invalid combination of code units in a variety of other scenarios, returning a � felt fine.
e.g. // valid String.fromCodePoint(0xd83e, 0xdd20) // invalid, but "�" is ... fine? String.fromCodePoint(0xdd20, 0xd83e)
In summary, Unicode code points (characters) are 32 bit. JavaScript manipulates Unicode in utf-16 for historical reasons, because at some point before Unicode, 16 bit was deemed enough (ucs-2). utf-16 run length encodes Unicode 32 codepoints into one or two code units. Splitting in a middle of a codepoints produces one invalid half string, and one semantically different half string. emojies are a sequence of Unicode…
> Unicode code points are 32 bit 21-bit, actually. It was supposed to be 32-bit, but UTF-16 caps out at 21-bit, so they lopped eleven bits of potential from Unicode (and UTF-8, so no more six-byte encoding). > at some point before Unicode No, in the early days of Unicode. > run length encodes Um… what? RLE is a data compression thing, UTF-16 has nothing to do with it.
Although, conveniently this means that UTF-8 bytes 0xF8 through 0xFF are always nonsense so the third party Rust type `ColdString` uses leading bytes 0xF8 through 0xFF in its 8 bytes of representation to indicate "I am an inline UTF-8 string, but, the UTF-8 starts in the next byte with a total length of N bytes" where N = byte - 0xF8
This leaves the continuation marker bits alone so ColdString can use those in that front byte to indicate "I am not actually inline data, I'm a pointer, rotate me so these indicator bits are my LSB and zero out them out to make me a 4 byte aligned pointer".
Which leaves all other 8 bytes values for the valid UTF-8 strings, which all begin with either ASCII or a byte between 0xC2 and 0xF4 inclusive.