My Favorite Bugs: Invalid Surrogate Pairs • Hacker News Board

Re: My Favorite Bugs: Invalid Surrogate Pairs

#51

Post by Dwedit » Sun, May 17, 2026, 1:24 PM UTC

mananaysiempre wrote: post #48

bombela wrote: post #39

Earlier quoted context omitted.

Indeed, Linux allows anything but "/" and "\0" in filenames. Those days its reasonable to refuse utf8 filenames. But one must always validate first!

> Indeed, Linux allows anything but "/" and "\0" in filenames. For what it’s worth, NT allows any 16-bit quantity but L'\\' (0x005C) in filenames (even nulls); it’s the Win32 layer on top of it that imposes all the other weird restrictions and mappings.

The NT Object Namespace itself indeed has no restrictions on filename characters except for "\", but once you reach a real filesystem like NTFS or FAT, the forbidden characters continue to be blocked.

https://projectzero.google/2016/02/the-definitive-guide-on-w...

Re: My Favorite Bugs: Invalid Surrogate Pairs

#52

Post by mweidner » Mon, May 18, 2026, 7:38 PM UTC

chrismorgan wrote: post #18

A CRDT library working at the code unit level? Ouch. Of course that’s going to go wrong, it was inevitable. As for using extended grapheme clusters, it sounds a little bit iffy—maybe possible to use correctly, maybe not, because they’re not stable over time. That style of thing has created some fascinating bugs, like (a few years ago) index corruption in PostgreSQL due to collation changes. Unicode scalar values are…

A CRDT that operates on code units should work out okay, because each grapheme cluster will always be inserted and deleted in a single edit - hence it should stick together in the text. (Some CRDTs actually can mess this up by interleaving concurrent-inserted code units, but Yjs avoids doing so.)

From the fix PR, I believe the issue in this case was with the insertion operations passed to the CRDT, not the CRDT itself. Specifically, Yjs's ProseMirror integration infers what text was inserted by diffing before and after states, instead of directly capturing user inputs (even though those are provided by ProseMirror transactions). The diff algorithm, lib0/diff, was not grapheme aware and hence could generate an inaccurate diff containing lone surrogates.

Operating on code units is convenient in JavaScript because then your CRDT's `length` matches the language's `String.length`, and likewise for indexed access.

Re: My Favorite Bugs: Invalid Surrogate Pairs

#53

Post by BobbyTables2 » Tue, May 19, 2026, 7:07 PM UTC

usrnm wrote: post #28

201984 wrote: post #24

Earlier quoted context omitted.

Why wouldn't 8 be enough? Surely 18,446,744,070,000,001,024 characters is enough for every writing system in the world.

Because that's not how Unicode works. It's not simply a table mapping numbers to all possible symbols. Unicode is full of special codepoints that have no meaning on their own, they serve as modifiers to other symbols and a single visible symbol can be formed by an arbitrary (in theory) long combimation of such codepoints. It doesn't matter how you encode it, it simply doesn't work as "codepoint -> symbol" and indexin…

Agreed, but it will take many generations for people to see characters in textual strings mainly as “code” instead of “data”.