Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

zaferbalkan.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#31

Post by account42 »

formerly_proven wrote: post #30
zombot wrote: post #28

Earlier quoted context omitted.

Microsoft never implements a standard, they only ever implement their own shit. Sometimes it's a close enough parody of a standard to fool superficial onlookers, but that's as close as you'll ever get.

Java, NT, .NET, "wide" C and C++ and a few others from the same time frame ended up with WTF-16 because surrogate pairs didn't exist when they were designed. They were designed with UCS-2, which is a fixed-length encoding. Unicode 2.0 then extended that to be variable length (16/32-bit) using surrogate pairs and that's where all the systems come from which don't validate surrogate pairs.

"Wide" characters in C and C++ do not have a standardized width. On most non-Windows platforms they are actually four bytes so can cover the entirety of Unicode and then some in one wchar_t. Not that that's a more reasonable encoding for most applications than WTF-16.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#32

Post by chrismorgan »

account42 wrote: post #25

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

Nothing uses well-formed UTF-16. I don’t think I know of a single piece of software or library that uses 16-bit code units that validates them.

In practice, “UTF-16” means “potentially-ill-formed UTF-16”. It’s that simple.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#33

Post by p_ing »

zombot wrote: post #29
feldrim wrote: post #17

Hi all. OP here. I added a Postscriptum about the surrogte pairs and their status in Linux. I used WSL to access those files under Windows, and generated the same on Linux. You can see that behavior differs on the same file names: 1. On Windows, accessed by WSL 2. On Linux (WSL), using UTF-8 locale 3. On Linux (WSL), using POSIX locale The difference is weird for me as a user. I'd like to know about the decisions mad…

WSL is not Linux, despite whatever Microsoft says.

WSL is Linux -- it's an automatically managed VM with some special sauce for connectivity between the parent partition and guest.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#34

Post by FirmwareBurner »

zombot wrote: post #28
account42 wrote: post #25

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

Microsoft never implements a standard, they only ever implement their own shit. Sometimes it's a close enough parody of a standard to fool superficial onlookers, but that's as close as you'll ever get.

>Microsoft never implements a standard

Win32 ?

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#35

Post by anonymfus »

p_ing wrote: post #33
zombot wrote: post #29

Earlier quoted context omitted.

WSL is not Linux, despite whatever Microsoft says.

WSL is Linux -- it's an automatically managed VM with some special sauce for connectivity between the parent partition and guest.

WSL2 is what you describe, WSL1 is not.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#36

Post by p_ing »

FirmwareBurner wrote: post #34
zombot wrote: post #28

Earlier quoted context omitted.

Microsoft never implements a standard, they only ever implement their own shit. Sometimes it's a close enough parody of a standard to fool superficial onlookers, but that's as close as you'll ever get.

>Microsoft never implements a standard Win32 ?

Op likely meant other's standards, not ones they create themselves. Sort of like AD, even though AD is objectively better than the mishmash of Kerb + LDAP + policy implementations out there.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#37

Post by p_ing »

anonymfus wrote: post #35
p_ing wrote: post #33

Earlier quoted context omitted.

WSL is Linux -- it's an automatically managed VM with some special sauce for connectivity between the parent partition and guest.

WSL2 is what you describe, WSL1 is not.

Of course, why would we discuss WSLv1 in 2025? If we discuss "Windows" today, it's unlikely we mean Windows 2000.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#38

Post by ripe »

account42 wrote: post #25

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

Thank you for posting the WTF-16 document. Very relevant for OP.

It's an old problem that people on different OS's need to access the same filesystem, particularly Windows clients versus UNIXy clients.

While UNIX filesystems traditionally accept any sequence of bytes except slash and NUL, and treat "." and ".." specially, Windows filesystems have had many additional restrictions on valid filenames, e.g., a list of "reserved names" that should not be used, short and long names, and case-insensitive names [1].

The NetApp filer, a NAS storage appliance, runs a specialized OS called Data ONTAP, which maintains a special on-disk representation called WAFL to enable this. WAFL is famous for its copy-on-write representation for file contents. [3] But in practice, most people in the real world are affected by its treatment of filenames. It's useful to take a look at how it solves these problems.

The NAS presents the same set of files both as an NFS volume for UNIXy clients, as well as a CIFS volume for Windows clients. It does this by enhancing the directory entries with additional information and configuration features to satisfy both requirements. For typical problems and what features they offer to solve them, see their documentation page about naming files and paths [2].

[1] https://learn.microsoft.com/en-us/windows/win32/fileio/namin...

[2] https://docs.netapp.com/us-en/ontap/nfs-admin/multi-byte-fil...

[3] Dave Hitz, "File System Design for an NFS File Server Appliance", https://www.cs.princeton.edu/courses/archive/fall17/cos318/r...

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#39

Post by wheybags »

anonymfus wrote: post #35
p_ing wrote: post #33

Earlier quoted context omitted.

WSL is Linux -- it's an automatically managed VM with some special sauce for connectivity between the parent partition and guest.

WSL2 is what you describe, WSL1 is not.

WSL2 is what people use now though. Saying "WSL is not linux" (because wsl1 isn't) is pedantically analogous to saying "mac os is not a unix based os" (because mac os 9 and under isn't)

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#40

Post by p_ing »

feldrim wrote: post #17

Hi all. OP here. I added a Postscriptum about the surrogte pairs and their status in Linux. I used WSL to access those files under Windows, and generated the same on Linux. You can see that behavior differs on the same file names: 1. On Windows, accessed by WSL 2. On Linux (WSL), using UTF-8 locale 3. On Linux (WSL), using POSIX locale The difference is weird for me as a user. I'd like to know about the decisions mad…

Your subscript 2 applies to NTFS. The only characters NTFS does not allow are NUL and "/".

Beyond that, it is up to the API you're choosing to use to read the volume. Win32 has of course many more restrictions than POSIX would, but since Windows NT supports multiple personalities, you could still RW illegal Win32 characters under NT, e.g. with SFU.

Post reply on HN