Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS does not normalize Unicode filenames

mjtsai.com

Post reply on HN
Print view View original

51–60 of 148 posts

Re: APFS does not normalize Unicode filenames

#51

Post by jstimpfle »

Ericson2314 wrote: post #20
ridiculous_fish wrote: post #14

Earlier quoted context omitted.

This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no... The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's b…

I'd argue: Use a langauge with generics and make the path type a ((de-)serializable) black box. You still need to make a decision in the end, but are much more free to change it.

This is not a language issue.

Re: APFS does not normalize Unicode filenames

#52

Post by Asooka »

jfim wrote: post #13
userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9). If the difference is not visible in your browser (it shouldn't), try copy-pasting those two filenames in a text editor, one of them is 10 characters long and one of them is 11 characters long.

It can get worse - imagine a program that normalizes all paths before handing them off to the fs. Then you go to delete the denormalized file, but it deletes the normalized one. With this change, the only safe way to handle paths would be to NOT normalize at all, but Apple are recommending the exact opposite. I predict a lot of confusion will arise from this change.

Re: APFS does not normalize Unicode filenames

#53

Post by Asooka »

sametmax wrote: post #40
userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

How do you normalize arab or chinese in a meaningful way for people speaking these languages ?

You don't, there is only one unicode representation of those characters.

Re: APFS does not normalize Unicode filenames

#54

Post by vbezhenar »

I'm not sure if normalization is good idea (generally because Unicode is complex beast and moving that complexity inside a kernel should be carefully weighted), but I'm sure that it doesn't solve any real problem. Characters "A" and "А" looks identical, unless you're missing Cyrillic font, but they won't be normalized, because they are completely different characters. There are many more other visually identical strings. So while normalization might solve some simple problems, it's not a complete solution, so filesystem might just treat names as byte arrays and let user solve his problems.

Re: APFS does not normalize Unicode filenames

#55

Post by vetinari »

std_throwaway wrote: post #45
Sami_Lehtinen wrote: post #44

Unicode isn't required to mess up things. Here's what baffled me for a while with NTFS. I'm pretty sure these issues are well known. http://www.sami-lehtinen.net/blog/linux-windows-ntfs-differe...

People should not use a non-compliant file system driver to create corrupted entries. NTFS mounts are for windows machines only.

What exactly makes a FS driver compliant or non-compliant? Is there a NTFS compliance test suite, that we can run against specific implementations?

If there isn't, accusations of non-compliance are just FUD.

Re: APFS does not normalize Unicode filenames

#56

Post by ben_bai »

kalleboo wrote: post #32

Linus Thorvalds will be happy to hear that http://www.cio.com/article/2868393/linus-torvalds-apples-hfs...

HFS+ can be configured at creation time to be case sensitive. I did so a while back. Worked perfectly except for one application which could not find it's files. So i had to create a container and Format it case in sensitive and intall the APP there...

Re: APFS does not normalize Unicode filenames

#59

Post by al2o3cr » 

    More generally, once APFS is deployed users can legitimately end up with
    multiple files in the same folder whose names only differ in normalization.
The initial message that starts this off seems to imply the opposite - instead, application developers should be normalizing the name before handing it to the filesystem. In that case, an application which allowed non-normalized naming would arguably have a bug.

Re: APFS does not normalize Unicode filenames

#60

Post by sametmax »

Asooka wrote: post #53
sametmax wrote: post #40

Earlier quoted context omitted.

How do you normalize arab or chinese in a meaningful way for people speaking these languages ?

You don't, there is only one unicode representation of those characters.

So what does mac do with those ?
Post reply on HN