Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

181–190 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#181

Post by cryptonector »

thaumasiotes wrote: post #166
cryptonector wrote: post #145

Earlier quoted context omitted.

No, just developers. Of course, the use of bytes for characters goes back a long time, to times when computers had small memories and disk (and other) storage capacities. And to even before then, to the days of telexes and typewriters. It's completely understandable. But UTF-8 is genius, which is why we use it. Incidentally, ASCII was actually a multi-byte codeset... since one could combine most lower-case characters…

I've always wondered what the concept of a "backspace" character was supposed to mean. (I tried printing it to erase a previously printed character, but that doesn't work.) I guess it's another relic of the idea that computer output goes to a printer rather than a display?

Yes! Terminal vendors had to explicitly support this in terminals, though maybe not just because of printers but because it's actually quite useful.

Note that BS only produces overstrike when followed by certain characters (which we might term "combining" for the fun of it), while most will just change the character at that location. A tty spinner is just |BS/BS-BS\BS|BS... with some delay between each -- no overstrike there.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#182

Post by cryptonector »

_urga wrote: post #170
cryptonector wrote: post #84

Earlier quoted context omitted.

Thank you for pointing this out. In particular ZFS has normalization-preserving/normalization-insensitive behavior, which is far superior to HFS+'s opinionated normalization-on-create (to a form that is different from the common input modes' output!).

Yes, HFS+ implements normalization-insensitive behavior through not being normalization-preserving, in the same way that some FATs might implement case-insensitive behavior through not being case-preserving. They didn't realize that you could have both features: normalization-preserving and normalization-insensitive.

I guess it was a very forgivable lack of imagination. When we came up with form-preserving/insensitive we were motivated in great part by the interop mess caused by HFS+ -- we might not have arrived there without that mess, though I'd like to believe that someone would eventually have reached this conclusion regardless.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#183

Post by cryptonector »

rtpg wrote: post #171
thaumasiotes wrote: post #165

Earlier quoted context omitted.

The filesystem can't be responsible for displaying anything, though. Displaying filenames is the job of the shell / window manager / etc. The filesystem is much better off handling filenames as a number of arbitrary bytes. Let people who want to put weird bytes in their filenames see ugly filenames along the lines of "\x00 Can you see this?"

OK so I move a file from my hard drive to a USB drive, and give it to a friend. Can they not read the filenames anymore because their shell/window manager is different?

Right!

Or worse, you speak multiple languages, or learn new ones, and need to... switch codesets? How do you then access your old files?!

We must switch to Unicode. Full stop. If there are imperfections in Unicode script support, then we must fix those, but otherwise we must adopt Unicode.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#184

Post by robjwells »

From the "What's new for developers" in macOS 10.13 High Sierra document [0], case-sensitive APFS can be normalisation-insensitive:

> APFS now supports an on-disk format change to allow for normalization-insensitive Case Sensitive volumes. This means that file names in either Unicode NFC or NFD will point to the same files.

Which means that both versions support normalisation-insensitivity.

(Edit: There is also a one-line mention of this in the iOS 11 document [1] but it doesn't say if it is the default.)

[0]: https://developer.apple.com/library/content/releasenotes/Mac...

[1]: https://developer.apple.com/library/content/releasenotes/Mac...

Re: APFS is not safe to use with names which have Unicode normalisation issues

#185

Post by tehabe »

cryptonector wrote: post #149
tehabe wrote: post #30

Personally I think the current Normalisation Form D is awful, storing an ü as two characters is really annoying and even bash can't really deal with it in the version Apple uses. I really hope APFS will fix this. But we'll see.

It's not bash, BTW, but the OS X input mode. A real irony there.

I used to compile my own bash from homebrew or alike and with that (newer) version, it worked w/o a problem.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#186

Post by rurban »

kps wrote: post #177
rurban wrote: post #173

Earlier quoted context omitted.

According to me not if you mixed Greek with Cyrillic, and not Math with Cyrillic. Unfortunately nobody cares about Unicode identifier security models. Garbage out is the most popular ideology.

I take the view that modern file names are human readable labels, not access keys. “10kΩ Резисторы” is Cyrillic, Latin, and Greek (U+2126 canonically maps to U+03A9). A proposal that disallows that but allows “10kΩ Resistors” is a political non-starter.

The Unicode consortium has published recommendations with various security levels everybody enjoys to ignore. Following even the weakest level would be a starter, but not in current filesystems community. This needs to start in a new OS or with high profile spoofing attacks, such as on GitHub.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#187

Post by kuon »

deong wrote: post #180
kuon wrote: post #179

Earlier quoted context omitted.

Higher level doesn't mean in app. What I mean is that unicode normalization is really hard, and it should be it's own module that can be used regardless of the fs. app->fopen->unicode normalization->APFS/HFS/FAT...

In that case, I agree. The problem here seems, from my understanding, to be that what Apple did was more like app -> CFFile -> unicode normalization -> fopen -> APFS which screws you because anyone can just call fopen on their own without using the core foundation libraries, leading to inconsistent states in the filesystem. You can be higher level than the filesystem, but only a little bit. You can't be higher level…

I guess the ZFS approach is the best one.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#188

Post by klodolph »

cryptonector wrote: post #175
klodolph wrote: post #164

Earlier quoted context omitted.

What about "bag of wchar_t" doesn't preserve normalization? Or am I not sure what you're trying to say? It's no more a "bag of code units" than Linux filesystems store a "bag of code units". Windows will barf back whatever wchar_t array you give it, just like Linux will barf back whatever char array you give it.

I'm saying "code unit" is the correct Unicode term for "8-bit bytes" (UTF-8) or "16-bit values" (UTF-16).

You're right that "code unit" is the correct Unicode term for parts of encoded Unicode strings, which is exactly why it's the wrong term in this case, because the whole point is the filenames don't have to be Unicode.

So I can take the byte string [0xFF, 0xFF, 0xFF] and use that as a filename on Linux, or I can take the sequence of 16-bit values [0xD800, 0xD800, 0xD800] and use that as a filename on NTFS. They're not made of code units because they're not Unicode strings.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#189

Post by robjwells »

robjwells wrote: post #184

From the "What's new for developers" in macOS 10.13 High Sierra document [0], case-sensitive APFS can be normalisation-insensitive: > APFS now supports an on-disk format change to allow for normalization-insensitive Case Sensitive volumes. This means that file names in either Unicode NFC or NFD will point to the same files. Which means that both versions support normalisation-insensitivity. (Edit: There is also a one…

The link to the What's New in iOS 11 page is wrong, it's here: https://developer.apple.com/library/content/releasenotes/Gen...

APFS is mentioned at the very end.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#190

Post by _urga »

cryptonector wrote: post #78

I've been saying this for years: http://cryptonector.com/2010/04/on-unicode-normalization-or-... (originally at blogs.sun.com, now blogs.oracle.com, though I can't find it there). The problem is that most input methods produce something close to NFC while HFS+ decomposes to something close to NFD. Which means that if you cut-n-paste non-ASCII Unicode names from a finder into any app that doesn't normalize, then you'l…

Thanks for the article, it is spot on.
Post reply on HN