Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

171–180 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#171

Post by rtpg »

thaumasiotes wrote: post #165
rtpg wrote: post #160

Earlier quoted context omitted.

the problem that arrives there almost instantly is that people want to see a list of filenames. If you're ever going to do some sort of "displaying" of data, you cannot store it as bytes. You need to know what characters things are supposed to be presented as. You could imagine not settling on a specific encoding, but you must know the encoding. Unless your plan is to show a list of numbers to users.

The filesystem can't be responsible for displaying anything, though. Displaying filenames is the job of the shell / window manager / etc. The filesystem is much better off handling filenames as a number of arbitrary bytes. Let people who want to put weird bytes in their filenames see ugly filenames along the lines of "\x00 Can you see this?"

OK so I move a file from my hard drive to a USB drive, and give it to a friend.

Can they not read the filenames anymore because their shell/window manager is different?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#172

Post by adrianratnapala »

cryptonector wrote: post #135
derefr wrote: post #125

Earlier quoted context omitted.

In other words, like most RDBMSes, ZFS differentiates "field type" from "collation." "Field type" in an RDBMS controls what can be written (e.g. "valid UTF8 strings"), what will be read back (e.g. the use of the Unicode replacement character), and what special values like NULL will cast to. An RDBMS field's collation controls how values in the field will compare for equality, and what will happen when you sort on tha…

Well, readdir() in ZFS does not produce outputs in any order, since directories are just plain hash tables. And the n-i string comparison function produces a boolean if I remember correctly, not a trinary. So it doesn't define a collation. But it could define a collation, that's true. The problem with moving sorting into the kernel is that you now need to have the collations there (English? French? something else? "U…

Indeed!

Moreover, like it should be EBADIDEA.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#173

Post by rurban »

kps wrote: post #163
rurban wrote: post #73

Earlier quoted context omitted.

The biggest actual problem is Greek vs Cyrillic. They appear the same, but use different encodings. Any filename (directory entry as identifier) must forbid mixed scripts. See TR31 http://www.unicode.org/reports/tr31/ Combining marks or RTL switching tricks are the other popular spoofs.

> Any filename (directory entry as identifier) must forbid mixed scripts. So I'm not allowed to name my files “A=πr²” or “10kΩ Резисторы”?

According to me not if you mixed Greek with Cyrillic, and not Math with Cyrillic. Unfortunately nobody cares about Unicode identifier security models. Garbage out is the most popular ideology.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#174

Post by lurker456 »

konstmonst wrote: post #97
deathanatos wrote: post #75

Earlier quoted context omitted.

> Should filesystems be making up for our Unicode shortcomings? Absolutely, yes. File names are text by their very definition; that we've been treating them as "bags of bytes" is a historical tragedy. At the very least, file names need to be displayed, as text, to the user, so they should be stored as text , that is in some well-defined encoding, and yes, it should be the job of the filesystem driver / kernel to enfo…

But what is text? Not everyone wants to use unicode. It is dependent of the platform, the region, the OS and on many other different things like LC_* variables on linux. Why should a filesystem depend on those too?

RDBMS's have gone through the same journey. First it was hardcoded, now for many of them we can specify the codepage or UTF encoding and collation of a DB, some even offer it per table or column, and so on.

Filesystems should do the same. Either pick one way and stick to it (expose it as UTF8, with a sensible normalization and collation) or offer several options that can be specified when creating a volume and do on the fly conversions for clients that need it.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#175

Post by cryptonector »

klodolph wrote: post #164
cryptonector wrote: post #96

Earlier quoted context omitted.

"Bag of code units" would be a better description (if it's correct anyways). This is not very good. You want normalization-preserving/insensitive behavior instead.

What about "bag of wchar_t" doesn't preserve normalization? Or am I not sure what you're trying to say? It's no more a "bag of code units" than Linux filesystems store a "bag of code units". Windows will barf back whatever wchar_t array you give it, just like Linux will barf back whatever char array you give it.

I'm saying "code unit" is the correct Unicode term for "8-bit bytes" (UTF-8) or "16-bit values" (UTF-16).

Re: APFS is not safe to use with names which have Unicode normalisation issues

#176

Post by cryptonector »

thaumasiotes wrote: post #165
rtpg wrote: post #160

Earlier quoted context omitted.

the problem that arrives there almost instantly is that people want to see a list of filenames. If you're ever going to do some sort of "displaying" of data, you cannot store it as bytes. You need to know what characters things are supposed to be presented as. You could imagine not settling on a specific encoding, but you must know the encoding. Unless your plan is to show a list of numbers to users.

The filesystem can't be responsible for displaying anything, though. Displaying filenames is the job of the shell / window manager / etc. The filesystem is much better off handling filenames as a number of arbitrary bytes. Let people who want to put weird bytes in their filenames see ugly filenames along the lines of "\x00 Can you see this?"

Software will need to know what codeset and encoding those arbitrary bytes are in!

Re: APFS is not safe to use with names which have Unicode normalisation issues

#177

Post by kps »

rurban wrote: post #173
kps wrote: post #163

Earlier quoted context omitted.

> Any filename (directory entry as identifier) must forbid mixed scripts. So I'm not allowed to name my files “A=πr²” or “10kΩ Резисторы”?

According to me not if you mixed Greek with Cyrillic, and not Math with Cyrillic. Unfortunately nobody cares about Unicode identifier security models. Garbage out is the most popular ideology.

I take the view that modern file names are human readable labels, not access keys.

“10kΩ Резисторы” is Cyrillic, Latin, and Greek (U+2126 canonically maps to U+03A9). A proposal that disallows that but allows “10kΩ Resistors” is a political non-starter.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#178

Post by kstrauser »

cryptonector wrote: post #95
mchanson wrote: post #23

Earlier quoted context omitted.

"without warning"? Not really. This has been discussed to death on ATP last year. Also, in regards to the headline, there are tens or maybe over a hundred million non-english speakers using iOS already running APFS...

Did anyone mention normalization-preserving/normalization-insensitive behavior? That's what ZFS does, and it's great!

(BTW, I found your comments interesting, even as I laughed at how often you were making them. Thanks for contributing, and sorry for getting us both flagged (for IMHO the goofiest possible reason).)

Re: APFS is not safe to use with names which have Unicode normalisation issues

#179

Post by kuon »

deong wrote: post #122
kuon wrote: post #6

I'm happy that the filesystem treats name as sequences of bytes. Normalization should happen at a higher level.

"Happening at a higher level" is a reasonable solution, if it happens consistently , no matter which higher level you're using. If you have 18 different functions to open a file, and 11 of them normalize and 7 don't, then you're screwed before you even get out the door. Programmers simply aren't capable of getting this right in a consistent way if asked to solve the problem application by application.

Higher level doesn't mean in app.

What I mean is that unicode normalization is really hard, and it should be it's own module that can be used regardless of the fs.

app->fopen->unicode normalization->APFS/HFS/FAT...

Re: APFS is not safe to use with names which have Unicode normalisation issues

#180

Post by deong »

kuon wrote: post #179
deong wrote: post #122

Earlier quoted context omitted.

"Happening at a higher level" is a reasonable solution, if it happens consistently , no matter which higher level you're using. If you have 18 different functions to open a file, and 11 of them normalize and 7 don't, then you're screwed before you even get out the door. Programmers simply aren't capable of getting this right in a consistent way if asked to solve the problem application by application.

Higher level doesn't mean in app. What I mean is that unicode normalization is really hard, and it should be it's own module that can be used regardless of the fs. app->fopen->unicode normalization->APFS/HFS/FAT...

In that case, I agree. The problem here seems, from my understanding, to be that what Apple did was more like

    app -> CFFile -> unicode normalization -> fopen -> APFS
which screws you because anyone can just call fopen on their own without using the core foundation libraries, leading to inconsistent states in the filesystem. You can be higher level than the filesystem, but only a little bit. You can't be higher level than some API that developers will regularly use (unless you do like ZFS and normalize at lookup rather than create).
Post reply on HN