Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS does not normalize Unicode filenames

mjtsai.com

Post reply on HN
Print view View original

91–100 of 148 posts

Re: APFS does not normalize Unicode filenames

#91

Post by therealmarv »

eridius wrote: post #24
therealmarv wrote: post #16

Is APFS still using Apple's style UTF-8 for e.g. Umlauts? I had a lot of trouble with rsync and also Samba later (filenames and folders hidden) when I discovered that Umlauts on HFS are different than Umlauts on e.g. Ext4.

What do you mean by "different"? Umlauts on HFS+ are still Umlauts anywhere else. The only real oddity of HFS+ (beyond the fact that it does normalization at all) is that it's not using NFD, it's using a variant of NFD based on an old version of Unicode (it has to be this way because the normalization tables must be immutable or there's compatibility issues when reading drives written to from different versions of th…

I don't know the technical details but the UTF-8 from a Mac is different from UTF-8 on Linux.

Copy "hällö" from Mac to Linux over samba -> works fine.

scp or rsync that same file again from Mac to Linux (without special conversion options) -> you will have two "hällö" there.

The first one is recognized by Samba, the second one is invisible.

Re: APFS does not normalize Unicode filenames

#92

Post by Ericson2314 »

jstimpfle wrote: post #51
Ericson2314 wrote: post #20

Earlier quoted context omitted.

I'd argue: Use a langauge with generics and make the path type a ((de-)serializable) black box. You still need to make a decision in the end, but are much more free to change it.

This is not a language issue.

I didn't say it was. I said generics would help one easily switch their choice on the different semantics, not avoid choosing at all.

Re: APFS does not normalize Unicode filenames

#93

Post by jstimpfle »

Ericson2314 wrote: post #92
jstimpfle wrote: post #51

Earlier quoted context omitted.

This is not a language issue.

I didn't say it was. I said generics would help one easily switch their choice on the different semantics, not avoid choosing at all.

Many, or most, languages already do that by representing filenames as native strings (which are usually abstract "Unicode" strings).

Re: APFS does not normalize Unicode filenames

#94

Post by acdha »

TazeTSchnitzel wrote: post #87
vbezhenar wrote: post #54

I'm not sure if normalization is good idea (generally because Unicode is complex beast and moving that complexity inside a kernel should be carefully weighted), but I'm sure that it doesn't solve any real problem. Characters "A" and "А" looks identical, unless you're missing Cyrillic font, but they won't be normalized, because they are completely different characters. There are many more other visually identical stri…

Confusable characters look similar or the same to humans. Canonically equivalent Unicode sequences look the same to machines . The latter is a much more significant problem, because it can wreak havoc with interoperability.

Very true - and it's amplified by inconsistency allowing problems to spread further before being noticed. At work I deal with a lot of Bag-It archives where we have a text manifest of checksums accompanying files on disk and this reliably bites users of tools when something (archive or network transfer tool, Git or SVN, transition to/from a Mac with HFS+, etc.) causes the encoding in the manifest not to match the local filesystem, and the confusion is amplified because some tools will handle normalization differences so the bug report is “why does tool A say this file is missing when Explorer/Finder and tool B say it's fine?”

Re: APFS does not normalize Unicode filenames

#95

Post by [deleted] »

xenadu02 wrote: post #10
QuercusMax wrote: post #4

This seems especially bad because US-based developers who don't test with unicode filenames might not come across this issue, leaving all their non-English-speaking customers broken. (Not that this excuses such developers in any way.) It also means that, yet again, every app will need to be updated for a new version of iOS. Makes me wonder how many apps will be left behind if not updated? Thousands? Hundreds of thous…

You should not be using anything other than UUIDs or integers for file names. Maintain your own mapping in a database or file. Using a network value or a value returned by an API is just asking for trouble. If a user names a file that will be hidden behind a URL the same advice applies. If not then the user can use any sequence of bytes they want and you shouldn't care.

[deleted]

Re: APFS does not normalize Unicode filenames

#96

Post by andybak »

spullara wrote: post #8
QuercusMax wrote: post #4

This seems especially bad because US-based developers who don't test with unicode filenames might not come across this issue, leaving all their non-English-speaking customers broken. (Not that this excuses such developers in any way.) It also means that, yet again, every app will need to be updated for a new version of iOS. Makes me wonder how many apps will be left behind if not updated? Thousands? Hundreds of thous…

Except filesystem experts like Dropbox, developers probably shouldn't be letting users name their files.

I'm not sure I understand. How can an app NOT allow me to name my files - unless you're talking about some iOS-like "hide the filesystem" silo.

Re: APFS does not normalize Unicode filenames

#97

Post by drdaeman »

rurban wrote: post #39
userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

No. Ever heard about http://websec.github.io/unicode-security-guide/ Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out. There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed…

(Offtopic) You may want to correct that blog post, the point about Japanese writing systems. "[...] Japanese using characters from Chinese (Kanji/Han), Katagana (modern japanase) and Hiregana (the old middle-age script used by women)" is really incorrect. Suggest to just say that modern Japanese uses both logographic (kanji, originated from Chinese hanzi[1]) and syllabic (kana) characters simultaneously, with two distinct types of kana (hiragana and katakana).

[1] In Unicode, they're generally unified into a single set, via process called "Han unification". So, unlike Greek "Α" vs Cyrillic "А", the "same" character that may even look slightly differently in Chinese vs Japanese (e.g. "海"), would have a single codepoint in Unicode. But that's another story...

Re: APFS does not normalize Unicode filenames

#98

Post by tyingq »

userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

>perhaps with one or two exceptions like '/' and \0

Aren't colons (:) an issue with MacOS as well? I think Finder and other userspace apps convert them to slashes. I suppose though, there's a case for the filesystem not caring about that.

Re: APFS does not normalize Unicode filenames

#99

Post by infogulch »

userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

If I were allowed one more distinction I would definitely add utf-8 only.

Re: APFS does not normalize Unicode filenames

#100

Post by dom0 »

std_throwaway wrote: post #45
Sami_Lehtinen wrote: post #44

Unicode isn't required to mess up things. Here's what baffled me for a while with NTFS. I'm pretty sure these issues are well known. http://www.sami-lehtinen.net/blog/linux-windows-ntfs-differe...

People should not use a non-compliant file system driver to create corrupted entries. NTFS mounts are for windows machines only.

This is not a bug in either, it's a configuration defect. NTFS-3g used with the correct options, namely windows_names, works exactly as expected, but is then not POSIX compliant anymore.
Post reply on HN