Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS does not normalize Unicode filenames

mjtsai.com

Post reply on HN
Print view View original

131–140 of 148 posts

Re: APFS does not normalize Unicode filenames

#131

Post by TazeTSchnitzel »

slrz wrote: post #129
TazeTSchnitzel wrote: post #87

Earlier quoted context omitted.

Confusable characters look similar or the same to humans. Canonically equivalent Unicode sequences look the same to machines . The latter is a much more significant problem, because it can wreak havoc with interoperability.

Canonically equivalent Unicode sequences look the same to machines. Memcmp disagrees, as do the default equality operators of most programming languages in existence.

Sure, but normalisation can nonetheless happen automatically and implicitly in many places.

Re: APFS does not normalize Unicode filenames

#132

Post by __david__ »

comex wrote: post #118
hamstergene wrote: post #104

Earlier quoted context omitted.

I'm going through apps on my phone and can hardly think what any of them would use Unicode filenames for. Say, a messenger might use user's nickname to name a history file — that would cause one-time loss of history, but not break the app. What else? Something tells me practically no apps will be seriously affected.

Any app that stores files in iCloud Drive has a fully user-accessible chunk of filesystem; it has to give the files meaningful names and properly handle files being renamed, added, etc. from the iCloud Drive app or file picker. So they have to deal with arbitrary filenames; on the other hand, for the same reason, they can't maintain a master list of files somewhere which would break, but have to check the actual dire…

> they can't maintain a master list of files somewhere which would break, but have to check the actual directory contents instead.

They should be doing that anyway. Anything else is just begging to get out of sync at some point.

Re: APFS does not normalize Unicode filenames

#133

Post by __david__ »

ben_bai wrote: post #56
kalleboo wrote: post #32

Linus Thorvalds will be happy to hear that http://www.cio.com/article/2868393/linus-torvalds-apples-hfs...

HFS+ can be configured at creation time to be case sensitive. I did so a while back. Worked perfectly except for one application which could not find it's files. So i had to create a container and Format it case in sensitive and intall the APP there...

I used to format my whole disk as case-insensitive but enough poorly written apps broke (ok, mostly Steam) that I now just have a case sensitive partition for my source code and /usr/local.

Re: APFS does not normalize Unicode filenames

#134

Post by Ericson2314 »

jstimpfle wrote: post #93
Ericson2314 wrote: post #92

Earlier quoted context omitted.

I didn't say it was. I said generics would help one easily switch their choice on the different semantics, not avoid choosing at all.

Many, or most, languages already do that by representing filenames as native strings (which are usually abstract "Unicode" strings).

I want to swap the path type used by the FS implementation. This is totally distinct from a langauge providing some string type which while abstract is nevertheless fixed and unchangable.

Re: APFS does not normalize Unicode filenames

#135

Post by snuxoll »

drdaeman wrote: post #90
hawski wrote: post #73

Earlier quoted context omitted.

Are you also against case sensitive file systems? Otherwise you can end up with two files that have the "same" name - eg. anne.jpg and Anne.jpg). Does normalization cover such a case?

I think, the problem with Unicode normalization is, input methods are (well, conceptually, at least) meant to produce text, not binary. If filesystems are using binary data for filenames, there can be a case when is really no way to address a file by typing its name, even if you can type in that language. This isn't an issue for case sensitivity or alike.

Filesystems themselves treating file names and paths at bits of binary data is the right way to do it, normalization shouldn't be handled directly by the filesystem as it adds complexity and duplication of effort.

Ideally, the VFS layer should be handling all this garbage.

Re: APFS does not normalize Unicode filenames

#136

Post by [deleted] »

userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

[deleted]

Re: APFS does not normalize Unicode filenames

#137

Post by [deleted] »

QuercusMax wrote: post #4

This seems especially bad because US-based developers who don't test with unicode filenames might not come across this issue, leaving all their non-English-speaking customers broken. (Not that this excuses such developers in any way.) It also means that, yet again, every app will need to be updated for a new version of iOS. Makes me wonder how many apps will be left behind if not updated? Thousands? Hundreds of thous…

[deleted]

Re: APFS does not normalize Unicode filenames

#138

Post by [deleted] »

xenadu02 wrote: post #10
QuercusMax wrote: post #4

This seems especially bad because US-based developers who don't test with unicode filenames might not come across this issue, leaving all their non-English-speaking customers broken. (Not that this excuses such developers in any way.) It also means that, yet again, every app will need to be updated for a new version of iOS. Makes me wonder how many apps will be left behind if not updated? Thousands? Hundreds of thous…

You should not be using anything other than UUIDs or integers for file names. Maintain your own mapping in a database or file. Using a network value or a value returned by an API is just asking for trouble. If a user names a file that will be hidden behind a URL the same advice applies. If not then the user can use any sequence of bytes they want and you shouldn't care.

[deleted]

Re: APFS does not normalize Unicode filenames

#139

Post by [deleted] »

spullara wrote: post #8
QuercusMax wrote: post #4

This seems especially bad because US-based developers who don't test with unicode filenames might not come across this issue, leaving all their non-English-speaking customers broken. (Not that this excuses such developers in any way.) It also means that, yet again, every app will need to be updated for a new version of iOS. Makes me wonder how many apps will be left behind if not updated? Thousands? Hundreds of thous…

Except filesystem experts like Dropbox, developers probably shouldn't be letting users name their files.

[deleted]
Post reply on HN