Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

zaferbalkan.com

Post reply on HN
Print view View original

11–20 of 65 posts

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#11

Post by feldrim »

theiebrjfb wrote: post #8

Yet another reason to use Linux everywhere. It is 2025 and Windows (and probably Mac) users have to deal with weird Unicode filesystem issues. Good luck puting Chinese characters or emoticons into filenames. Ext4 filename has maximal length 255 characters. That is the only legacy limit you have to deal with as a Linux user. And even that can be avoided by using more modern filesystems. And we get filesystem level sna…

I see two points here. First, you did not read the article and did not see the footnote that these are valid in Linux as well.

Second, your comment shows you are lacking the knowledge on Linux as well. In Linux, as I have written in the foot note, accepts anything but 0x00 (null) and 0x2F (“/”). Other than that, all characters are valid paths. If you consider these a problem, I'd like to remind that the 2048 surrogate pairs is a really small subset of unrenderable combinations allowed in Linux.

Anyone are free to have their opinions but at least, before making bold claims, please do your due diligence.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#12

Post by feldrim »

layer8 wrote: post #9
feldrim wrote: post #7

Earlier quoted context omitted.

Yep. The quirk here is that the surrogates, that are merely enablers for other characters, can be paired with each other. With the absence of other valid characters, they are not enabling anything. One assumes there is a validation but it does not exist here.

There is no validation on the file system level because file names in NTFS are sequences of arbitrary 16-bit values, similar to how on Unix file systems, file names are sequences of arbitrary 8-bit values. Arguably the situation on Unix is worse, because there the interpretation and validity depends on the current locale.

Totally. These are design choices made by development teams. But as users, we "assume" all are readable until one day we learn that it does not work that way. Until I came across this issue, I assumed them to be all valid, renderable characters.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#14

Post by skissane »

feldrim wrote: post #11
theiebrjfb wrote: post #8

Yet another reason to use Linux everywhere. It is 2025 and Windows (and probably Mac) users have to deal with weird Unicode filesystem issues. Good luck puting Chinese characters or emoticons into filenames. Ext4 filename has maximal length 255 characters. That is the only legacy limit you have to deal with as a Linux user. And even that can be avoided by using more modern filesystems. And we get filesystem level sna…

I see two points here. First, you did not read the article and did not see the footnote that these are valid in Linux as well. Second, your comment shows you are lacking the knowledge on Linux as well. In Linux, as I have written in the foot note, accepts anything but 0x00 (null) and 0x2F (“/”). Other than that, all characters are valid paths. If you consider these a problem, I'd like to remind that the 2048 surrogat…

> In Linux, as I have written in the foot note, accepts anything but 0x00 (null) and 0x2F (“/”)

POSIX 2024 encourages (but doesn’t require) implementations to disallow newline in file names, returning EILSEQ if you try to create a new file or directory with a name containing a newline. Thus far Linux hasn’t adopted that recommendation, but I personally hope it does some day.

For backward compatibility, it would have to be a mount option. It could be done at VFS level so it applies to all filesystems.

Personally I would go even further and introduce a “require_sane_filenames” mount option, which would block you (at the VFS layer) from creating any file name containing invalid UTF-8 (including overlong sequences and UTF-8 encoded surrogates), C0 controls or (UTF-8 encoded) C1 controls.

Also I think it would be great if filesystems had a superblock bit that declared they only supported “sane filenames”. Then even accessing such a file would error because it would be a sign of filesystem corruption.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#15

Post by kwertzzz »

layer8 wrote: post #10
theiebrjfb wrote: post #8

Yet another reason to use Linux everywhere. It is 2025 and Windows (and probably Mac) users have to deal with weird Unicode filesystem issues. Good luck puting Chinese characters or emoticons into filenames. Ext4 filename has maximal length 255 characters. That is the only legacy limit you have to deal with as a Linux user. And even that can be avoided by using more modern filesystems. And we get filesystem level sna…

You have the same, if not worse, issue on Linux with filenames that aren’t valid UTF-8 sequences. Not to mention that on Linux switching the locale may change the interpretation of filenames as characters, which isn’t the case with NTFS.

> Not to mention that on Linux switching the locale may change the interpretation of filenames as characters, which isn’t the case with NTFS.

If you change the locale to an uninstalled one, then yes. But if the locale is installed, then I don't see a problem.

echo $LANG

# output: en_US.UTF-8

touch fusée.txt

LANG=fr_FR.UTF-8 ls

# output: 'fus'$'\303\251''e.txt'

sudo locale-gen fr_FR.UTF-8

sudo update-locale

LANG=fr_FR.UTF-8 ls

# output: fusée.txt

Are you maybe using non-UTF-8 locale?

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#16

Post by feldrim »

skissane wrote: post #14
feldrim wrote: post #11

Earlier quoted context omitted.

I see two points here. First, you did not read the article and did not see the footnote that these are valid in Linux as well. Second, your comment shows you are lacking the knowledge on Linux as well. In Linux, as I have written in the foot note, accepts anything but 0x00 (null) and 0x2F (“/”). Other than that, all characters are valid paths. If you consider these a problem, I'd like to remind that the 2048 surrogat…

> In Linux, as I have written in the foot note, accepts anything but 0x00 (null) and 0x2F (“/”) POSIX 2024 encourages (but doesn’t require) implementations to disallow newline in file names, returning EILSEQ if you try to create a new file or directory with a name containing a newline. Thus far Linux hasn’t adopted that recommendation, but I personally hope it does some day. For backward compatibility, it would have…

This I did not know. I know that ZFS has "utf8only" option, but not sure about others.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#17

Post by feldrim »

Hi all. OP here. I added a Postscriptum about the surrogte pairs and their status in Linux. I used WSL to access those files under Windows, and generated the same on Linux. You can see that behavior differs on the same file names:

1. On Windows, accessed by WSL

2. On Linux (WSL), using UTF-8 locale

3. On Linux (WSL), using POSIX locale

The difference is weird for me as a user. I'd like to know about the decisions made behind these. If anyone has information, please let me know.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#18

Post by formerly_proven »

feldrim wrote: post #17

Hi all. OP here. I added a Postscriptum about the surrogte pairs and their status in Linux. I used WSL to access those files under Windows, and generated the same on Linux. You can see that behavior differs on the same file names: 1. On Windows, accessed by WSL 2. On Linux (WSL), using UTF-8 locale 3. On Linux (WSL), using POSIX locale The difference is weird for me as a user. I'd like to know about the decisions mad…

The Linux section just seems to be artifacts of the WSL hacks, it has nothing to do with how Linux filenames function. Those are simply bags of bytes, the encoding only matters for displaying them, and isn't interpreted internally. ls failing to access the .exe is clearly a WSL filesystem issue and not a Linux / ls issue. You also can't set a UTF-16 locale because that's not what a locale is. UTF-16/32 vs SBCS and UTF-8 is the wide/narrow character distinction, which is a whole separate thing, different ABIs, different APIs.

Re: Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

#19

Post by feldrim »

formerly_proven wrote: post #18
feldrim wrote: post #17

Hi all. OP here. I added a Postscriptum about the surrogte pairs and their status in Linux. I used WSL to access those files under Windows, and generated the same on Linux. You can see that behavior differs on the same file names: 1. On Windows, accessed by WSL 2. On Linux (WSL), using UTF-8 locale 3. On Linux (WSL), using POSIX locale The difference is weird for me as a user. I'd like to know about the decisions mad…

The Linux section just seems to be artifacts of the WSL hacks, it has nothing to do with how Linux filenames function. Those are simply bags of bytes, the encoding only matters for displaying them, and isn't interpreted internally. ls failing to access the .exe is clearly a WSL filesystem issue and not a Linux / ls issue. You also can't set a UTF-16 locale because that's not what a locale is. UTF-16/32 vs SBCS and UT…

WSL-to-Windows, yes, it is due to translations. But within the WSL, not sure. I'll try to replicate them on a Ubuntu VM for comparison.
Post reply on HN