Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS does not normalize Unicode filenames

mjtsai.com

Post reply on HN
Print view View original

71–80 of 148 posts

Re: APFS does not normalize Unicode filenames

#71

Post by madeofpalk »

astrange wrote: post #28
xenadu02 wrote: post #10

Earlier quoted context omitted.

You should not be using anything other than UUIDs or integers for file names. Maintain your own mapping in a database or file. Using a network value or a value returned by an API is just asking for trouble. If a user names a file that will be hidden behind a URL the same advice applies. If not then the user can use any sequence of bytes they want and you shouldn't care.

It's poor UI to not name a file on the user's own machine, if they'll ever have to look at it. I'm counting things like web browser caches in this because OmniDiskSweeper/etc users will be seeing the large files you put in there.

Most importantly, APFS is first being rolled out to iOS where this isn't as much of a problem because the user never really sees the filesystem anyway.

Re: APFS does not normalize Unicode filenames

#72

Post by hawski »

userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

This sounds like DJB's Netstring [1].

"hello world!" encoded as "12:hello world!,"

An empty string as "0:,"

[1] https://en.wikipedia.org/wiki/Netstring

Re: APFS does not normalize Unicode filenames

#73

Post by hawski »

jfim wrote: post #13
userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9). If the difference is not visible in your browser (it shouldn't), try copy-pasting those two filenames in a text editor, one of them is 10 characters long and one of them is 11 characters long.

Are you also against case sensitive file systems? Otherwise you can end up with two files that have the "same" name - eg. anne.jpg and Anne.jpg). Does normalization cover such a case?

Re: APFS does not normalize Unicode filenames

#74

Post by cesarb »

This used to be a pain point with git, when some developers were using MacOS and the repository had file names with accents; to git, it looked like the files had been renamed. Some time later, git added the "core.precomposeunicode" option to work around this problem.

Re: APFS does not normalize Unicode filenames

#75

Post by hawski »

rurban wrote: post #39
userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

No. Ever heard about http://websec.github.io/unicode-security-guide/ Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out. There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed…

For average user "documents" and "Documents" is the same thing. Are paths also normalized to upper or lower case? Or file system should be case insensitive?

Re: APFS does not normalize Unicode filenames

#76

Post by _jal »

xenadu02 wrote: post #10
QuercusMax wrote: post #4

This seems especially bad because US-based developers who don't test with unicode filenames might not come across this issue, leaving all their non-English-speaking customers broken. (Not that this excuses such developers in any way.) It also means that, yet again, every app will need to be updated for a new version of iOS. Makes me wonder how many apps will be left behind if not updated? Thousands? Hundreds of thous…

You should not be using anything other than UUIDs or integers for file names. Maintain your own mapping in a database or file. Using a network value or a value returned by an API is just asking for trouble. If a user names a file that will be hidden behind a URL the same advice applies. If not then the user can use any sequence of bytes they want and you shouldn't care.

Assuming your application is the only app that will ever interact with a file is only correct in very specific circumstances. This is terrible advice in the general case.

Re: APFS does not normalize Unicode filenames

#77

Post by ttflee »

jstimpfle wrote: post #49
jfim wrote: post #13

Earlier quoted context omitted.

It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9). If the difference is not visible in your browser (it shouldn't), try copy-pasting those two filenames in a text editor, one of them is 10 characters long and one of them is 11 characters long.

The problem exists already on the visual level, since there are pairs of distinct Unicode characters that look very much alike. I think the only thing you can really do about that is restricting to ASCII minus control characters. Probably many programmers would be willing to accept that, but non-technical users not. The next best thing would be to enforce a canonical (code-point?) encoding. But given the complexities…

https://turnoff.us/geek/user-space-election/

Re: APFS does not normalize Unicode filenames

#78

Post by asdfaoeu »

silvestrov wrote: post #69
userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

> Things are much easier for the file system if it can just treat names as bags of bytes. And much , much harder for applications if that "bag of bytes" is an invalid UTF-8 sequence. You will end up with an invalid string (or an exception), and trying to open that file will then fail. I'd really hope that Apple checks that the filenames are valid UTF-8 as they otherwise can end up with very interesting security bugs.

You can handle that fine provided you don't assume that.

> I'd really hope that Apple checks that the filenames are valid UTF-8 as they otherwise can end up with very interesting security bugs.

I think that's a bit pessimistic as Linux has run for year like that without any real issues.

Re: APFS does not normalize Unicode filenames

#79

Post by kevingadd »

hawski wrote: post #75
rurban wrote: post #39

Earlier quoted context omitted.

No. Ever heard about http://websec.github.io/unicode-security-guide/ Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out. There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed…

For average user "documents" and "Documents" is the same thing. Are paths also normalized to upper or lower case? Or file system should be case insensitive?

Case insensitivity is something best addressed in userspace because:

1) History has shown that doing it at the filesystem level causes significant compatibility issues

2) Case insensitivity can present visually confusing results

3) Round-tripping between cases sometimes is not possible without loss of information

4) Some languages don't have a meaningful definition of 'case insensitive'

5) Some languages would demand a more comprehensive approach (i.e. being able to search using either kana or kanji for Japanese filenames)

So for things like case insensitivity (or #5 above) it's best to handle them at the application (or, ideally, OS user space library) level, taking advantage of information like system locale. Low-level APIs and system services can use case-sensitive filenames.

This is different from the question of whether to normalize filenames, because filenames are still visually unambiguous (at least most of the time) in UI, logs, and debuggers.

Incidentally, Windows is proof of this technique's advantages because NTFS is a case-sensitive file system while Win32 is case-insensitive. IIRC the Linux subsystem for Windows takes advantage of this.

Post reply on HN