It doesn't really matter what they do, since filesystem naming is FUBAR and has been FUBAR pretty much since UNIXv1, and possibly even earlier than that outside the UNIX family.
APFS does not normalize Unicode filenames
101–110 of 148 posts
Re: APFS does not normalize Unicode filenames
#102Earlier quoted context omitted.
No. Ever heard about http://websec.github.io/unicode-security-guide/ Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out. There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed…
(Offtopic) You may want to correct that blog post, the point about Japanese writing systems. "[...] Japanese using characters from Chinese (Kanji/Han), Katagana (modern japanase) and Hiregana (the old middle-age script used by women)" is really incorrect. Suggest to just say that modern Japanese uses both logographic (kanji, originated from Chinese hanzi[1]) and syllabic (kana) characters simultaneously, with two dis…
Ugh… yeah, agreed; reading that was painful.
To reiterate:
• Chinese characters are called hànzì¹
• modern Japanese uses many hànzì, calling them kanji² instead
• katakana³ is used in modern Japanese to write words of foreign origin (loanwords) as well as onomatopoeia and is romanized as katakana, not katagana
• hiragana⁴ is used in modern Japanese to write okurigana⁵, particles, and certain words, and is romanized as hiragana, not hiregana
Here’s a typical (and extremely simple) sentence in modern Japanese that uses all three:
この文はサンプルです。
This is a sample sentence.
この: this (hiragana)
文: sentence (kanji)
は: particle (pronounced wa) (hiragana)
サンプル: sample (loanword; pronounced sanpuru) (katakana)
です: particle (pronounced desu) (hiragana)
――――――
¹ — https://en.wikipedia.org/wiki/Chinese_characters
² — https://en.wikipedia.org/wiki/Kanji
³ — https://en.wikipedia.org/wiki/Katakana
Re: APFS does not normalize Unicode filenames
#103Earlier quoted context omitted.
No. Ever heard about http://websec.github.io/unicode-security-guide/ Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out. There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed…
(Offtopic) You may want to correct that blog post, the point about Japanese writing systems. "[...] Japanese using characters from Chinese (Kanji/Han), Katagana (modern japanase) and Hiregana (the old middle-age script used by women)" is really incorrect. Suggest to just say that modern Japanese uses both logographic (kanji, originated from Chinese hanzi[1]) and syllabic (kana) characters simultaneously, with two dis…
Thanks, appreciated. I only have two "Learn Japanese" and "Learn Korean in x days" books for dummies.
Fixed
Re: APFS does not normalize Unicode filenames
#104This seems especially bad because US-based developers who don't test with unicode filenames might not come across this issue, leaving all their non-English-speaking customers broken. (Not that this excuses such developers in any way.) It also means that, yet again, every app will need to be updated for a new version of iOS. Makes me wonder how many apps will be left behind if not updated? Thousands? Hundreds of thous…
I'm going through apps on my phone and can hardly think what any of them would use Unicode filenames for. Say, a messenger might use user's nickname to name a history file — that would cause one-time loss of history, but not break the app. What else?
Something tells me practically no apps will be seriously affected.
Re: APFS does not normalize Unicode filenames
#105Don't Mac apps already have to deal with network and FAT32 drives? Or does macOS already normalize those?
Network drives are handled by the file sharing protocol, not the local file system. Fat32 is handled by a fat32 driver that does the correct thing according to fat32 rules.
Re: APFS does not normalize Unicode filenames
#106Earlier quoted context omitted.
Are you also against case sensitive file systems? Otherwise you can end up with two files that have the "same" name - eg. anne.jpg and Anne.jpg). Does normalization cover such a case?
I think, the problem with Unicode normalization is, input methods are (well, conceptually, at least) meant to produce text, not binary. If filesystems are using binary data for filenames, there can be a case when is really no way to address a file by typing its name, even if you can type in that language. This isn't an issue for case sensitivity or alike.
Programs often allow you to type arbitrary binary values using a keyboard, either by holding a keyboard control key and typing the numeric value, or prefixing it with '\'.
Re: APFS does not normalize Unicode filenames
#107I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…
No good has ever come from allowing BEL and DEL as part of filenames. I'm with David Wheeler: https://www.dwheeler.com/essays/fixing-unix-linux-filenames.... We need to limit filenames for the good of the entire system and the whole community. Filenames as byte strings may sound good, but nobody ever thinks of the costs and the scant benefits.
I disagree completely. What we need is to allow even slashes in the file names – not restrict (for what an end user may feel arbitrarily) even more writeable characters off file names, just so we can avoid fixing our replacing our tools or writing one line more in a shell script.
Re: APFS does not normalize Unicode filenames
#108Earlier quoted context omitted.
You should not be using anything other than UUIDs or integers for file names. Maintain your own mapping in a database or file. Using a network value or a value returned by an API is just asking for trouble. If a user names a file that will be hidden behind a URL the same advice applies. If not then the user can use any sequence of bytes they want and you shouldn't care.
It's poor UI to not name a file on the user's own machine, if they'll ever have to look at it. I'm counting things like web browser caches in this because OmniDiskSweeper/etc users will be seeing the large files you put in there.
It doesn't matter much on iOS, where users rarely if at all have to access the filesystem directly. It is very rare that the same files would be accessed and modified by more than a single application anyway.
Re: APFS does not normalize Unicode filenames
#109I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…
> Things are much easier for the file system if it can just treat names as bags of bytes. And much , much harder for applications if that "bag of bytes" is an invalid UTF-8 sequence. You will end up with an invalid string (or an exception), and trying to open that file will then fail. I'd really hope that Apple checks that the filenames are valid UTF-8 as they otherwise can end up with very interesting security bugs.
At least valid utf-8 isn't difficult for a filesystem to enforce.
Re: APFS does not normalize Unicode filenames
#110Linus Thorvalds will be happy to hear that http://www.cio.com/article/2868393/linus-torvalds-apples-hfs...
This is for iOS, where the app developer fully controls file naming within their sandbox. It is very unlikely that MacOS will fail to normalize because filenames there are presented directly to the user.