Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS does not normalize Unicode filenames

mjtsai.com

Post reply on HN
Print view View original

121–130 of 148 posts

Re: APFS does not normalize Unicode filenames

#121

Post by coldtea »

jfim wrote: post #13
userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9). If the difference is not visible in your browser (it shouldn't), try copy-pasting those two filenames in a text editor, one of them is 10 characters long and one of them is 11 characters long.

>It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9).

It's still good then. And several systems allow for that just fine, including VMS (uniqueness comes from more than the filename). This is more alike real world folders (which can have identical items, e.g. two copies of the same paper), and is also an excellent way to keep different document versions (keep the name the same, change the date shown).

Re: APFS does not normalize Unicode filenames

#122

Post by coldtea »

drdaeman wrote: post #90
hawski wrote: post #73

Earlier quoted context omitted.

Are you also against case sensitive file systems? Otherwise you can end up with two files that have the "same" name - eg. anne.jpg and Anne.jpg). Does normalization cover such a case?

I think, the problem with Unicode normalization is, input methods are (well, conceptually, at least) meant to produce text, not binary. If filesystems are using binary data for filenames, there can be a case when is really no way to address a file by typing its name, even if you can type in that language. This isn't an issue for case sensitivity or alike.

>If filesystems are using binary data for filenames, there can be a case when is really no way to address a file by typing its name, even if you can type in that language

You could do that trivially in UNIX since forever.

Re: APFS does not normalize Unicode filenames

#123

Post by coldtea »

YSFEJ4SWJUVU6 wrote: post #107
quotemstr wrote: post #46

Earlier quoted context omitted.

No good has ever come from allowing BEL and DEL as part of filenames. I'm with David Wheeler: https://www.dwheeler.com/essays/fixing-unix-linux-filenames.... We need to limit filenames for the good of the entire system and the whole community. Filenames as byte strings may sound good, but nobody ever thinks of the costs and the scant benefits.

I disagree completely. What we need is to allow even slashes in the file names – not restrict (for what an end user may feel arbitrarily) even more writeable characters off file names, just so we can avoid fixing our replacing our tools or writing one line more in a shell script.

Any solution you come up with must work with replacing existing software piecemeal, or it's no solution.

Re: APFS does not normalize Unicode filenames

#124

Post by jrochkind1 »

rurban wrote: post #50
jwilk wrote: post #48

Earlier quoted context omitted.

> No. Ever heard about http://websec.github.io/unicode-security-guide/ Which part of it is relevant to pathnames? > To the canonical form of course Do you mean NFC? > not the crazy Python 3 or Apple idea of NFD. In what circumstances does Python 3 normalize pathnames? > If it's encoded as bytes you get garbage in - garbage out. What do you mean by "encoded as bytes"? > http://perl11.org/blog/unicode-identifiers.html…

> Which part of it is relevant to pathnames? All parts regarding identifiers and names, if you see pathnames as publicly identifiable information. See below. > In what circumstances does Python 3 normalize pathnames? None. Python 3 normalizes identifiers to the shorter non-canonical NFKC form. Apple HFS+ normalizes pathnames to the longer D canonical form. Which is faster but takes more space. Usually space is more i…

> to the shorter non-canonical NFKC form

What definition or sense of 'canonical' do you mean? NFKC stands for "Compatibility Decomposition, followed by Canonical Composition", so it's canonical in some sense, right?

Re: APFS does not normalize Unicode filenames

#125

Post by jrochkind1 »

Asooka wrote: post #53
sametmax wrote: post #40

Earlier quoted context omitted.

How do you normalize arab or chinese in a meaningful way for people speaking these languages ?

You don't, there is only one unicode representation of those characters.

I don't believe that's literally true, some all legal and correct Japanese unicode sequences will be transformed under at least some of the unicode normalization forms, no?

Re: APFS does not normalize Unicode filenames

#126

Post by kalleboo »

djrogers wrote: post #105
kalleboo wrote: post #67

Don't Mac apps already have to deal with network and FAT32 drives? Or does macOS already normalize those?

Network drives are handled by the file sharing protocol, not the local file system. Fat32 is handled by a fat32 driver that does the correct thing according to fat32 rules.

And APFS drive will work according so APFS rules so I guess no worries

Re: APFS does not normalize Unicode filenames

#127

Post by msbarnett »

arm wrote: post #102
drdaeman wrote: post #97

Earlier quoted context omitted.

(Offtopic) You may want to correct that blog post, the point about Japanese writing systems. "[...] Japanese using characters from Chinese (Kanji/Han), Katagana (modern japanase) and Hiregana (the old middle-age script used by women)" is really incorrect. Suggest to just say that modern Japanese uses both logographic (kanji, originated from Chinese hanzi[1]) and syllabic (kana) characters simultaneously, with two dis…

Ugh… yeah, agreed; reading that was painful. To reiterate: • Chinese characters are called hànzì ¹ • modern Japanese uses many hànzì , calling them kanji ² instead • katakana³ is used in modern Japanese to write words of foreign origin (loanwords) as well as onomatopoeia and is romanized as katakana , not katagana • hiragana⁴ is used in modern Japanese to write okurigana⁵, particles, and certain words, and is romaniz…

> です: particle (pronounced desu) (hiragana)

(While we're being pedantic)

です isn't a particle, it's the imperfective (present/future) polite form of the copular verb (to be, in English). The only particle in that sentence is は.

And as a further side note, some words or names normally written in kanji or hiragana are ocassionally written in katakana for emphasis or to emphasize a foreign nature.

Re: APFS does not normalize Unicode filenames

#128

Post by srssays »

Asooka wrote: post #53
sametmax wrote: post #40

Earlier quoted context omitted.

How do you normalize arab or chinese in a meaningful way for people speaking these languages ?

You don't, there is only one unicode representation of those characters.

Not the case for Arabic. There's hundreds of ligature code points, for example ﰻ (kl), ﷵ (ṣl'm) and ﷳ (akbr). And of course ﷽

Re: APFS does not normalize Unicode filenames

#129

Post by slrz »

TazeTSchnitzel wrote: post #87
vbezhenar wrote: post #54

I'm not sure if normalization is good idea (generally because Unicode is complex beast and moving that complexity inside a kernel should be carefully weighted), but I'm sure that it doesn't solve any real problem. Characters "A" and "А" looks identical, unless you're missing Cyrillic font, but they won't be normalized, because they are completely different characters. There are many more other visually identical stri…

Confusable characters look similar or the same to humans. Canonically equivalent Unicode sequences look the same to machines . The latter is a much more significant problem, because it can wreak havoc with interoperability.

Canonically equivalent Unicode sequences look the same to machines.

Memcmp disagrees, as do the default equality operators of most programming languages in existence.

Re: APFS does not normalize Unicode filenames

#130

Post by slrz »

cjensen wrote: post #111
ben_bai wrote: post #56

Earlier quoted context omitted.

HFS+ can be configured at creation time to be case sensitive. I did so a while back. Worked perfectly except for one application which could not find it's files. So i had to create a container and Format it case in sensitive and intall the APP there...

Adobe apps are notoriously incompatible with case sensitivity. If your apps work, that's great. But if Apple switched to case sensitivity by default, it would break apps.

Apple breaks apps all the time on macOS.
Post reply on HN