Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

zaferbalkan.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#41

Post by wheybags »

Devasta wrote: post #13

Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake. The real solution is to force the entire world population to use the Rotokas language of Papua New Guinea.

I still think we should have forced everything into a 32-bit char, with no distinction between codepoints and grapheme clusters. One press on backspace removes one char. Address of char 7 is base+7x4. String length is byte length x 4. cat /dev/urandom is a valid string, it's the font's job to deal with unknown byte values, if you just want to process the text you dont need to care. Everything about text processing becomes super easy like in the old ascii only k&r c example code. I'm not 100% certain, but I don't think there's a widely used language that couldn't be represented by that.

Of course, you lose round trip ability with legacy encodings, which is why we have the mess that is unicode. Oh and silly things like unicode flag emojis wouldn't work, but honestly maybe that would be for the best. Oh well, it's too late now so I guess we just accept it.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#42

Post by p_ing »

ripe wrote: post #38
account42 wrote: post #25

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

Thank you for posting the WTF-16 document. Very relevant for OP. It's an old problem that people on different OS's need to access the same filesystem, particularly Windows clients versus UNIXy clients. While UNIX filesystems traditionally accept any sequence of bytes except slash and NUL, and treat "." and ".." specially, Windows filesystems have had many additional restrictions on valid filenames, e.g., a list of "r…

NTFS has the same two restrictions that many UNIX file systems have, NUL and slash.

The APIs, Win32 in the case of [1], have further restrictions. If you want you can use a different API/personality and write whatever value you'd like (sans NUL and /) provided said personality has no limits -- NTFS does no validation itself.

In practice, Win32 being the default personality makes said plethlora of restrictions true, but it isn't a "filesystem" limitation, rather an API restriction. A nuanced if unimportant difference.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#43

Post by wbl »

formerly_proven wrote: post #30
zombot wrote: post #28

Earlier quoted context omitted.

Microsoft never implements a standard, they only ever implement their own shit. Sometimes it's a close enough parody of a standard to fool superficial onlookers, but that's as close as you'll ever get.

Java, NT, .NET, "wide" C and C++ and a few others from the same time frame ended up with WTF-16 because surrogate pairs didn't exist when they were designed. They were designed with UCS-2, which is a fixed-length encoding. Unicode 2.0 then extended that to be variable length (16/32-bit) using surrogate pairs and that's where all the systems come from which don't validate surrogate pairs.

Where did they think Cuneiform was going to fit? Even with unihan the BMP was getting consumed fast.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#44

Post by jerf »

wheybags wrote: post #41
Devasta wrote: post #13

Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake. The real solution is to force the entire world population to use the Rotokas language of Papua New Guinea.

I still think we should have forced everything into a 32-bit char, with no distinction between codepoints and grapheme clusters. One press on backspace removes one char. Address of char 7 is base+7x4. String length is byte length x 4. cat /dev/urandom is a valid string, it's the font's job to deal with unknown byte values, if you just want to process the text you dont need to care. Everything about text processing be…

For better or worse, thanks to emoji Zero Width Joiner support [1], we're well on our way to there being more than 4 billion potential Unicode "characters". 4 billion is only 32 bits and you start spending a few bits here on hair style and a few bits there on skin color and a few bits on "misc" and then allow arbitrary combinations of them into composite families [2] and you can burn through 32-bits fairly quickly.

I don't think we're there yet. I think if someone did make a complete list of "valid" emoji right now, which for the sake of argument I'll call "formally defined in the Unicode standard", it would even on an absolute scale look like we're a long ways away from a full 32-bits of valid combinations. But you have to think of this on the log scale because this is about "bits" and those four-person families are already quite a long ways along to a full 32 bits. It wouldn't take much more customization, or the formal addition of more people in a group, to get there.

And someone who knows more about Unicode than I do may be able to establish that there are already in the standard ways to get to more than 32 bits' worth of data in a single standardized glyph; I certainly wouldn't bet much against that already being true.

(Personally, I'll go with "worse". In hindsight, we should probably have frozen Unicode into the original Docomo (and the other phone company that had them) emoji necessary for interoperability, and then created the emoji as an extension into Unicode. It seems like it would be useful to "support Unicode" without having to come with the complete understanding of what is increasingly the most complicated "language" in Unicode; forget doing good Arabic rendering or trying to understand an ideographic language, the emojis blow all that complexity away now. But here we are.)

[1]: https://unicode.org/emoji/charts/emoji-zwj-sequences.html

[2]: https://www.unicode.org/reports/tr51/#Multi_Person_Groupings

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#45

Post by extraduder_ire »

wheybags wrote: post #41
Devasta wrote: post #13

Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake. The real solution is to force the entire world population to use the Rotokas language of Papua New Guinea.

I still think we should have forced everything into a 32-bit char, with no distinction between codepoints and grapheme clusters. One press on backspace removes one char. Address of char 7 is base+7x4. String length is byte length x 4. cat /dev/urandom is a valid string, it's the font's job to deal with unknown byte values, if you just want to process the text you dont need to care. Everything about text processing be…

>Oh and silly things like unicode flag emojis wouldn't work, but honestly maybe that would be for the best.

Why not? They're just two (or more) characters from a special set next to each other that a font may combine. (and some ad-hoc ZWJ sequences) I don't think windows even ships a font that does that by default.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#46

Post by layer8 »

kwertzzz wrote: post #15
layer8 wrote: post #10

Earlier quoted context omitted.

You have the same, if not worse, issue on Linux with filenames that aren’t valid UTF-8 sequences. Not to mention that on Linux switching the locale may change the interpretation of filenames as characters, which isn’t the case with NTFS.

> Not to mention that on Linux switching the locale may change the interpretation of filenames as characters, which isn’t the case with NTFS. If you change the locale to an uninstalled one, then yes. But if the locale is installed, then I don't see a problem. echo $LANG # output: en_US.UTF-8 touch fusée.txt LANG=fr_FR.UTF-8 ls # output: 'fus'$'\303\251''e.txt' sudo locale-gen fr_FR.UTF-8 sudo update-locale LANG=fr_FR…

Yes, I mean locales like fr_FR.ISO-8859-15, ja_JP.SJIS or zh_CN.GBK.

While these probably aren’t used much anymore, it still means that your filenames can break just by setting an environment variable. Or issues like here: https://news.ycombinator.com/item?id=16992546

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#48

Post by layer8 »

account42 wrote: post #25

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

Historically, this is because Windows NT used UCS-2 [0]. Unicode only moved to beyond 65536 characters, and introduced the concept of surrogate pairs, with Unicode 2.0 in 1996.

[0] https://www.unicode.org/faq/utf_bom.html#utf16-11

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#49

Post by sumtechguy »

ripe wrote: post #38
account42 wrote: post #25

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

Thank you for posting the WTF-16 document. Very relevant for OP. It's an old problem that people on different OS's need to access the same filesystem, particularly Windows clients versus UNIXy clients. While UNIX filesystems traditionally accept any sequence of bytes except slash and NUL, and treat "." and ".." specially, Windows filesystems have had many additional restrictions on valid filenames, e.g., a list of "r…

If you want to subtly break a windows install you can use setCaseSensitiveInfo with fsutil. It turns on/off case sensitivity for a directory. There is also a similar set of options for samba shares which comes with interesting tradeoffs for speed of reading a directory list.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#50

Post by dwdz »

feldrim wrote: post #22
dwdz wrote: post #21

The script works just fine on real Linux, it creates 2048 files and ls command lists them all with different names. ls -l win32/ total 0 -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\277\237''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\267\213''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\240\220''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\274\273''.exe' -rw-r--r-- 1 dawid dawid 0…

Oh, great. Can you also share the locale? I'll write another Postscriptum section then.

    LANG=en_IE.UTF-8
    LANGUAGE=en_IE:en
    LC_CTYPE="en_IE.UTF-8"
    LC_NUMERIC="en_IE.UTF-8"
    LC_TIME="en_IE.UTF-8"
    LC_COLLATE="en_IE.UTF-8"
    LC_MONETARY="en_IE.UTF-8"
    LC_MESSAGES="en_IE.UTF-8"
    LC_PAPER="en_IE.UTF-8"
    LC_NAME="en_IE.UTF-8"
    LC_ADDRESS="en_IE.UTF-8"
    LC_TELEPHONE="en_IE.UTF-8"
    LC_MEASUREMENT="en_IE.UTF-8"
    LC_IDENTIFICATION="en_IE.UTF-8"
    LC_ALL=
Post reply on HN