Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

zaferbalkan.com

Post reply on HN
Print view View original

21–30 of 65 posts

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#21

Post by dwdz »

The script works just fine on real Linux, it creates 2048 files and ls command lists them all with different names.

    ls -l win32/
    total 0
    -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\277\237''.exe'
    -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\267\213''.exe'
    -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\240\220''.exe'
    -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\274\273''.exe'
    -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\251\205''.exe'
    -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\255\223''.exe'
    -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\272\257''.exe'
    -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\264\207''.exe'
    -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\261\246''.exe'
    -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\254\266''.exe'
    ...

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#22

Post by feldrim »

dwdz wrote: post #21

The script works just fine on real Linux, it creates 2048 files and ls command lists them all with different names. ls -l win32/ total 0 -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\277\237''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\267\213''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\240\220''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\274\273''.exe' -rw-r--r-- 1 dawid dawid 0…

Oh, great. Can you also share the locale? I'll write another Postscriptum section then.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#23

Post by mofeien »

Hi, thanks for the interesting submission!

I was a bit confused by the detour via utf-8 to arrive at the code points and had to look up UTF-8 encoding first to understand how they relate. Then I tried out the following

  candidate = chr(0xD800)
  candidate2 = bytes([0xED, 0xA0, 0x80]).decode('utf-8', errors='surrogatepass')
  print(candidate == candidate2) # True
and it seems that you could just iterate over code points directly with the `chr()` function.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#24

Post by feldrim »

mofeien wrote: post #23

Hi, thanks for the interesting submission! I was a bit confused by the detour via utf-8 to arrive at the code points and had to look up UTF-8 encoding first to understand how they relate. Then I tried out the following candidate = chr(0xD800) candidate2 = bytes([0xED, 0xA0, 0x80]).decode('utf-8', errors='surrogatepass') print(candidate == candidate2) # True and it seems that you could just iterate over code points di…

I f I remember correctly, I tried that but in order to cover the exact range I need, the high and low surrogates, I picked this way out of practicality. It was just easier.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#25

Post by account42 »

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings.

> Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000

Wrong. Windows uses WTF-16 [0] despite what the documentation says.

[0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#26

Post by rurban »

Devasta wrote: post #13

Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake. The real solution is to force the entire world population to use the Rotokas language of Papua New Guinea.

No, the real solution is to follow the unicode security guidelines for identifiers. Esp. on linux, where the silly garbage-in, garbage-out mantra doesn't fly with identifiers, because identifiers need to stay identifiable.

Apple HPFS did some things right. They did at least NFD. But linux insanities brought them back to -Whomoglyph attacks

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#27

Post by account42 »

feldrim wrote: post #6
n_plus_1_acc wrote: post #3

I think it's hilarious that the event viewer XML gets borked.

I am not 100% sure but mmc.exe has not been updated for years and it must be relying on WebBrowser control of Internet Explorer. Yes, IE is still alive in Windows. https://learn.microsoft.com/en-us/previous-versions/windows/...

And we should all be thankful for that. Just imagine if all those system tools were as "useful" as the modernized windows settings.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#28

Post by zombot »

account42 wrote: post #25

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

Microsoft never implements a standard, they only ever implement their own shit. Sometimes it's a close enough parody of a standard to fool superficial onlookers, but that's as close as you'll ever get.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#29

Post by zombot »

feldrim wrote: post #17

Hi all. OP here. I added a Postscriptum about the surrogte pairs and their status in Linux. I used WSL to access those files under Windows, and generated the same on Linux. You can see that behavior differs on the same file names: 1. On Windows, accessed by WSL 2. On Linux (WSL), using UTF-8 locale 3. On Linux (WSL), using POSIX locale The difference is weird for me as a user. I'd like to know about the decisions mad…

WSL is not Linux, despite whatever Microsoft says.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#30

Post by formerly_proven »

zombot wrote: post #28
account42 wrote: post #25

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

Microsoft never implements a standard, they only ever implement their own shit. Sometimes it's a close enough parody of a standard to fool superficial onlookers, but that's as close as you'll ever get.

Java, NT, .NET, "wide" C and C++ and a few others from the same time frame ended up with WTF-16 because surrogate pairs didn't exist when they were designed. They were designed with UCS-2, which is a fixed-length encoding. Unicode 2.0 then extended that to be variable length (16/32-bit) using surrogate pairs and that's where all the systems come from which don't validate surrogate pairs.
Post reply on HN