I remember a long time ago I accidentally put some symbol in a folder name like ? in Windows had problems
Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read
51–60 of 65 posts
Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read
#52I remember that in Mac OS X times, sometime between OS X v10.1 and 10.4, a system upgrade caused a bunch of unicode named files to become inaccessible/untouchable (but still present with a directory listing). At the time I didn't have the skills to figure out what had happened. I'm still curious to know if it was an intended breaking change.
OS X (at the POSIX level) assumes UTF-8 and normalizes file names to decomposed form (NFD). If for example you `date >$'\xC3\xBC'` (i.e. ‘ü’), then the actual stored file name is `$'\x75\xCC\88'` (i.e. ‘ü’ — assuming HN or my browser don't normalize!) and both `cat $'\xC3\xBC'` and `cat $'\x75\xCC\88'` (or ‘ü’ or ‘ü’) both work.
Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read
#53The script works just fine on real Linux, it creates 2048 files and ls command lists them all with different names. ls -l win32/ total 0 -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\277\237''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\267\213''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\240\220''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\274\273''.exe' -rw-r--r-- 1 dawid dawid 0…
[deleted]
Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read
#54The script works just fine on real Linux, it creates 2048 files and ls command lists them all with different names. ls -l win32/ total 0 -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\277\237''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\267\213''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\240\220''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\274\273''.exe' -rw-r--r-- 1 dawid dawid 0…
[deleted]
Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read
#55I remember that in Mac OS X times, sometime between OS X v10.1 and 10.4, a system upgrade caused a bunch of unicode named files to become inaccessible/untouchable (but still present with a directory listing). At the time I didn't have the skills to figure out what had happened. I'm still curious to know if it was an intended breaking change.
OS X (at the POSIX level) assumes UTF-8 and normalizes file names to decomposed form (NFD). If for example you `date >$'\xC3\xBC'` (i.e. ‘ü’), then the actual stored file name is `$'\x75\xCC\88'` (i.e. ‘ü’ — assuming HN or my browser don't normalize!) and both `cat $'\xC3\xBC'` and `cat $'\x75\xCC\88'` (or ‘ü’ or ‘ü’) both work.
Unrelated to my question above, I think
Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read
#56Why does the Windows filesystem allow filenames with invalid strings? It seems obvious that attempts to create files with such filenames ought to be blocked.
It's mentioned in a comment here that the existing restrictions are due to Windows APIs and NTFS does not check file names in a restricted way. Therefore, if devs want to filter these out or not in the API, is another story.
Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read
#57Earlier quoted context omitted.
I still think we should have forced everything into a 32-bit char, with no distinction between codepoints and grapheme clusters. One press on backspace removes one char. Address of char 7 is base+7x4. String length is byte length x 4. cat /dev/urandom is a valid string, it's the font's job to deal with unknown byte values, if you just want to process the text you dont need to care. Everything about text processing be…
For better or worse, thanks to emoji Zero Width Joiner support [1], we're well on our way to there being more than 4 billion potential Unicode "characters". 4 billion is only 32 bits and you start spending a few bits here on hair style and a few bits there on skin color and a few bits on "misc" and then allow arbitrary combinations of them into composite families [2] and you can burn through 32-bits fairly quickly. I…
I forgot about "Zalgo". You can definitely get more than 4 billion glyphs, indeed a great deal more, by stacking on modifiers.
I can see an argument that that's not really a "valid" use case that we need to worry about too much, though. Emoticons are well on their way to having more possible fully legal, fully intended outputs that go beyond what 32 bits could specify.
Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read
#58Earlier quoted context omitted.
Java, NT, .NET, "wide" C and C++ and a few others from the same time frame ended up with WTF-16 because surrogate pairs didn't exist when they were designed. They were designed with UCS-2, which is a fixed-length encoding. Unicode 2.0 then extended that to be variable length (16/32-bit) using surrogate pairs and that's where all the systems come from which don't validate surrogate pairs.
Where did they think Cuneiform was going to fit? Even with unihan the BMP was getting consumed fast.
In the beginning Unicode was only meant to be used for extant languages afaik.
Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read
#59Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake. The real solution is to force the entire world population to use the Rotokas language of Papua New Guinea.
I still think we should have forced everything into a 32-bit char, with no distinction between codepoints and grapheme clusters. One press on backspace removes one char. Address of char 7 is base+7x4. String length is byte length x 4. cat /dev/urandom is a valid string, it's the font's job to deal with unknown byte values, if you just want to process the text you dont need to care. Everything about text processing be…
Grapheme clusters are locale dependent. Also, if you aren't allowing combining characters, then you are going to need lots of extra codepoints. In some languages, like Indian ones, vowels are combining characters. Or there are languages where multiple code points produce grapheme cluster, like Hangul syllables. You are going to need a lot more code points to represent all possible strings. Text processing is going to be much harder cause there a thousand different representations of Hangul character.
Also, backspace is locale dependent. In some languages, backspace removes the accent, which makes sense with combining characters, and other it removes the whole character. Which is going to be fun when whole syllable is code point.
Languages are hard, there is no way to make them simple.
Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read
#60Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16
Microsoft never implements a standard, they only ever implement their own shit. Sometimes it's a close enough parody of a standard to fool superficial onlookers, but that's as close as you'll ever get.
In this case, they correctly met a standard, and the standard changed.
If you look at OS-X there are similar issues. The Apple File System is case insensitive for a particular Unicode version.