Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Favorite Bugs: Invalid Surrogate Pairs

george.mand.is

Post reply on HN
Print view View original

31–40 of 53 posts

Re: My Favorite Bugs: Invalid Surrogate Pairs

#31

Post by layer8 »

chrismorgan wrote: post #22
chuckadams wrote: post #13

Earlier quoted context omitted.

> It would have been expensive, but all characters should have been fixed size 64bit values. It would have been a non-starter, and then we'd all be dealing with Shift-JIS, BIG5, and FSM knows how many different codepages to this day. UTF-8 is about as elegant as it gets, though Java and JS still managed to fuck that up too (they both encode every codepoint outside the BMP as surrogate pairs in UTF-8)

> Java and JS […] both encode every codepoint outside the BMP as surrogate pairs in UTF-8 I can’t comment on Java, but JS I know reasonably well and I can’t think of any place it uses CESU-8.

Java doesn’t either.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#32

Post by tialaramex »

impure wrote: post #11

I had an emoji cut in half problem in Dart. I was a bit surprised because I thought substring operations worked on characters. It only caused an invalid Unicode symbol though so not too bad.

> I thought substring operations worked on characters

"character" turns out to be too vague an idea to correspond to some specific fact about the software. If you co-worker says his Uncle is "conservative" does he mean like "Believes Right To Work laws are a good idea" conservative or "Believes Joe Biden is a Communist" conservative ?

https://en.wikipedia.org/wiki/Character_(symbol) gives you some idea about this rabbit hole. Suffice to say, no, you can't have operations on "characters" until you've nailed down exactly what it was you meant by that.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#33

Post by dimes »

I have a similar, Unicode-related “favorite bug”.

We were expanding our product to a new language that used non-ASCII code points. Part of the system involved invoking binaries using text as input.

Locally, everything worked great. Once deployed, we got corrupted text output. As soon as we SSH’d on to the server to inspect, everything started working again.

It turns out that SSH servers can modify the LANG environment variable. The default value on our servers didn’t support Unicode, but LANG was updated as soon as we connected via ssh. It was a head scratcher for sure.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#35

Post by jcranmer »

rectang wrote: post #26
chrismorgan wrote: post #18

A CRDT library working at the code unit level? Ouch. Of course that’s going to go wrong, it was inevitable. As for using extended grapheme clusters, it sounds a little bit iffy—maybe possible to use correctly, maybe not, because they’re not stable over time. That style of thing has created some fascinating bugs, like (a few years ago) index corruption in PostgreSQL due to collation changes. Unicode scalar values are…

> I still can’t work out why it wasn’t obvious from the start that UCS-2 would never be enough) Surely certain people did know, but those people weren't in a position to do anything about it. Specifically, there were surely people who knew that because historical Chinese place names, Japanese nicknames, and so on, were not included in the original "Unicode" (it wasn't called UCS-2 yet) it was insufficient for complet…

Han unification predates Unicode by about a decade; most of the early work in Unicode largely consists of copy-pasting the Japanese and Chinese governments' standards for unified CJK ideographs. Indeed, read some of the early histories of Han unification (e.g., https://www.unicode.org/versions/Unicode16.0.0/core-spec/app...), and you'll notice that there's a lot of liasoning with East Asian technology groups in East Asian cities going on. I don't think any East Asian government representatives would have actually objected to Han unification!

It's also worth noting that the original goal of Unicode wasn't to be able to faithfully represent all text, but rather to faithfully represent existing character sets. Only later do you get the impetus to actually include everything, as people become a lot less tolerant of "computer can't actually represent " scenarios. Note too that a lot of the Han unification criticisms basically fall into the same bucket as, say, Medievalists, who want to preserve certain details of their source texts more faithfully than was the norm for computer systems in the 1980s.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#36

Post by layer8 »

chrismorgan wrote: post #18

A CRDT library working at the code unit level? Ouch. Of course that’s going to go wrong, it was inevitable. As for using extended grapheme clusters, it sounds a little bit iffy—maybe possible to use correctly, maybe not, because they’re not stable over time. That style of thing has created some fascinating bugs, like (a few years ago) index corruption in PostgreSQL due to collation changes. Unicode scalar values are…

> UTF-16 was unforced error (and I still can’t work out why it wasn’t obvious from the start that UCS-2 would never be enough).

ISO 10646 (“Universal Coded Character Set”) planned for 31-bit code points from the start (128 groups of 256 planes of 256 rows of 256 cells, with UCS-4 as a four-byte encoding), around 1989. Unicode, on the other hand, was a parallel effort initiated by Xerox and Apple a few years earlier, with more pragmatic aims, defining a 16-bit character set (but no encoding) that would allow round-tripping of existing character sets. For Unicode 1.1, it was decided to align it with ISO 10646 and make it coincide with the latter’s first plane (the BMP) and UCS-2. In Unicode 2.0, surrogate pairs and the UTF-16 encoding were introduced to allow future expansion to additional planes, in a way that would be compatible with existing implementations. Only with Unicode 3.1 in 2001, five years after Unicode 2.0 and ten years after Unicode 1.0, were actual characters assigned beyond the BMP.

History is complicated; aims, incentives, and constraints change over time.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#37

Post by bombela »

chrismorgan wrote: post #20
bombela wrote: post #15

In summary, Unicode code points (characters) are 32 bit. JavaScript manipulates Unicode in utf-16 for historical reasons, because at some point before Unicode, 16 bit was deemed enough (ucs-2). utf-16 run length encodes Unicode 32 codepoints into one or two code units. Splitting in a middle of a codepoints produces one invalid half string, and one semantically different half string. emojies are a sequence of Unicode…

> Unicode code points are 32 bit 21-bit, actually. It was supposed to be 32-bit, but UTF-16 caps out at 21-bit, so they lopped eleven bits of potential from Unicode (and UTF-8, so no more six-byte encoding). > at some point before Unicode No, in the early days of Unicode. > run length encodes Um… what? RLE is a data compression thing, UTF-16 has nothing to do with it.

If you are going to be pedantic, go all the way. 2^21 is 0 to 2_097_151. Unicode codepoint range is 0 to 1_114_111, slightly more than 2^20 (0 to 1_048_575).

I would argue that Unicode v2 onward; circa 1991 (Unicode Consortium and the ISO/IEC working together); is what anybody knows as Unicode with the 0 to 1_114_111 codepoints easily manipulated as a 32 bit value.

I meant variable length encoding, RLE encodes a number of successive repetition indeed.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#38

Post by bombela »

Someone wrote: post #27
chrismorgan wrote: post #20

Earlier quoted context omitted.

> Unicode code points are 32 bit 21-bit, actually. It was supposed to be 32-bit, but UTF-16 caps out at 21-bit, so they lopped eleven bits of potential from Unicode (and UTF-8, so no more six-byte encoding). > at some point before Unicode No, in the early days of Unicode. > run length encodes Um… what? RLE is a data compression thing, UTF-16 has nothing to do with it.

>> Unicode code points are 32 bit > 21-bit, actually Less than that. https://en.wikipedia.org/wiki/Code_point#In_character_encodi... : “The Unicode code space is divided into seventeen planes (the basic multilingual plane, and 16 supplementary planes), each with 65,536 (= 2¹⁶) code points. Thus the total size of the Unicode code space is 17 × 65,536 = 1,114,112” That makes it log(1,114,112)/log(2) bit. That’s about 2…

Don't know what you are being down voted (or my grand parent comment for that matter). You are very correct.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#39

Post by bombela »

Dwedit wrote: post #21

Windows allows unmatched surrogate pairs in filenames, invalid for UTF-16. Likewise, Linux allows invalid UTF-8 byte sequences in filenames. Because invalid UTF-16 strings could show up in places within Windows, someone made a UTF-8 variant called "WTF-8", which allows unmatched surrogate pairs to survive a round trip.

Indeed, Linux allows anything but "/" and "\0" in filenames. Those days its reasonable to refuse utf8 filenames. But one must always validate first!
Post reply on HN