Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

61–70 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#61

Post by loeg »

therealmarv wrote: post #40

So, can somebody answer this, is this right on APFS or not? rsync -rltv --iconv=utf-8-mac,utf-8 from_a to_b and similar sshfs: https://github.com/osxfuse/sshfs/issues/14#issuecomment-1859... I need to use that iconv options everytime when dealing with umlauts and mac linux communication. So do I still need that on APFS or not?

Per pilif[0], NFD filenames are converted to NFC during HFS+ to APFS conversion:

> BTW: On my 10.13 Beta 1 setup with an APFS converted boot drive, unless I manually create a NFD encoded file name on the command line (which you can't do accidentally), everything is NFC both in the UI and on the command line. This also applies to files I haven't touched since the conversion to APFS.

[0]: https://news.ycombinator.com/item?id=14496538

Re: APFS is not safe to use with names which have Unicode normalisation issues

#63

Post by vetinari »

klodolph wrote: post #53
vetinari wrote: post #48

Earlier quoted context omitted.

You are correct regarding the resource fork support in the fs and it's visibility, maybe I didn't express myself as exactly as I should. However, 99%+ of Mac users do use FAT-formatted USB sticks, ZIP files, or other fs/mechanism/whatever that does not support resource forks where the compatibility littering kicks in, so they, or the people they share their files with, will see ._$filename files too. On the other han…

Most files don't have resource forks though, so the ._$filename doesn't store resource fork data in most cases. It usually just stores metadata. I really doubt that 99%+ of Mac users use flash drives / zip files, though. I suspect that well more than 1% of users never use anything like that. A lot of people only have a single computer and share through e.g. just mailing files to people or using Dropbox, or not even t…

While resource forks are deprecated and on the retreat, extended attributes are a new hotness, used extensively (just download a file in the browser, and it will get com.apple.quarantine and com.apple.metadata:kMDItemWhereFroms). These are also shoveled into AppleDouble files. Actually, resource fork is just com.apple.ResourceFork extended attribute.

I'm not sure that there are more than 1% of mac users do not exchange files with other people, possibly on other platforms. What do they use the computer for, then? iPad would be more suitable then.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#64

Post by xoa »

pilif wrote: post #21

HFS+ was the only file system I know of that was doing Unicode normalisation and is certainly was the only one choosing NFD which encodes characters in a way that's impossible for someone to easily type in the UI. Short-term this will cause a lot of inconsistencies with applications using low-level APIs as the files currently existing will be NFD normalised, but any user-given path will very likely be more or less eq…

>HFS+ was the only file system I know of that was doing Unicode normalisation

I've seen something in a couple comments now to this effect, does nobody here do anything with ZFS at all? It's a pretty great filesystem on not just illumos but FreeBSD, Linux, and macOS, and it has full native normalization support (and for all 4 forms too). Normalizing is optional, but it's definitely there and with Macs I have been using NFD for compatibility purposes for 5 years now with no discernible performance problems in day-to-day use. I know ZFS isn't remotely a majority but I don't think it's that obscure either.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#65

Post by rbanffy »

klodolph wrote: post #45
acdha wrote: post #10

There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…

NTFS, by comparison, does the "bag of wchar_t" instead of the "bag of byte" for filenames.

> NTFS, by comparison, does the "bag of wchar_t"

I threw up a little.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#66

Post by bigin »

the_mitsuhiko wrote: post #42
bigin wrote: post #39

Earlier quoted context omitted.

HFS+ did perform normalization on a file system level and stored strings as utf-16 so it's not true for HFS+.

Pretty sure normalization was done in open() and not the fs. Normalization was performed even for fat32 sticks.

So you disagree with the linked blog post then? I mean this entire discussion is based on the premise that HFS+ performs normalization and APFS does not. The disrcepancy means that what used to happen automatically is now the responsibility of individual applications and the author hypothises that this means that shit will break.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#67

Post by the_mitsuhiko »

bigin wrote: post #66
the_mitsuhiko wrote: post #42

Earlier quoted context omitted.

Pretty sure normalization was done in open() and not the fs. Normalization was performed even for fat32 sticks.

So you disagree with the linked blog post then? I mean this entire discussion is based on the premise that HFS+ performs normalization and APFS does not. The disrcepancy means that what used to happen automatically is now the responsibility of individual applications and the author hypothises that this means that shit will break.

Not sure what you want me to say. In my mind the current situation with APFS is a massive pain and was not well thought through.

Previously normalization (while terrible) was at least consistent. Now different APIs have widely different behaviour and it's even worse on the application level. What used to be a annoyance has now become absolute chaos with no guidance from apple.

My understanding of the article is effectively: AFPS on macOS in the current form is unusable for non-English users and I strongly agree with that. While it might be partially usable, in particular if you upgrade from an earlier mac was performed, it has a ton of really terrible edge cases most users will quickly run into.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#68

Post by the_mitsuhiko »

acdha wrote: post #10

There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…

> I've had to write code to deal with normalization changes in archives and it's quite tedious.

You need to continue to write this on APFS. Nothing really changed in that regard. It's now just also possible to store denormalized filenames.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#69

Post by rurban »

vetinari wrote: post #3
foota wrote: post #2

Would be interesting to know why Apple made the decision not to normalize on a file system level then. Just an argument based on separation of concerns?

Normalizing on the file system level in the past was a mistake. Normalizing Unicode is not an easy task, it belongs into userspace and not into a kernel driver. Other systems consider filename a string of octets and interpretation what these octets mean is left for userspace to decide. For Linus Torvalds colorful opinions on HFS+, see here: https://plus.google.com/+JunioCHamano/posts/1Bpaj3e3Rru (in the comments).

Others have different opinions. Normalization of Unicode identifiers is necessary for identifiers to be identifiable. Mostly this is only done for domain names, I also do it for identifiers in my programming language, and consequently path names and directory entries should also identifiable. The garbage in - garbage out camp completely ignores all unicode security concerns and simply shifts blame to user-space. I applauded Apple to use normalization in HFS+, but the APFS integration is just horrible. 1st it makes unicode paths insecure again, and 2nd it's inconsistent.

NFD over NFC is simply trading data over CPU. NFC requires 3 passes over a string, NFD only 2, whilst NFD is usually a few bytes longer. Linus talking about NFD corrupting data is of course nonsense. He should stay technical and only rant about things he has an idea about. The python3 NFKC normalization format is nonsense, but not really important. NFD is fine, because faster.

Shifting interpretation of byte encodings from utf-8 to user-space is typical Linux non-sense, but he inherited the existing mess. Using utf-8 is miles better than bytes.

Case-insensitivity in HFS+ is of course legacy nonsense. This should have been to target to get rid of, not normalization.

The 2nd big security issue would be to forbid mixed scripts in a name (filename). I blogged about it here, and OP uses the same examples I used: http://perl11.org/blog/unicode-identifiers.html Invisible combining marks, right-to-left overwrites or similar / spoofs security problems are only fixable with normalization and more TR31 (mixed scripts, confusables), though confusables can only be handled in user-space, not the FS.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#70

Post by the_mitsuhiko »

therealmarv wrote: post #40

So, can somebody answer this, is this right on APFS or not? rsync -rltv --iconv=utf-8-mac,utf-8 from_a to_b and similar sshfs: https://github.com/osxfuse/sshfs/issues/14#issuecomment-1859... I need to use that iconv options everytime when dealing with umlauts and mac linux communication. So do I still need that on APFS or not?

> I need to use that iconv options everytime when dealing with umlauts and maclinux communication. So do I still need that on APFS or not?

Effectively you should most likely still do that, yes. Apple's apps more or less expect normalization.

Post reply on HN