Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

161–170 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#161

Post by cryptonector »

chipotle_coyote wrote: post #152
cryptonector wrote: post #85

Earlier quoted context omitted.

A terrible mistake. APFS should have been normalization-preserving/normalization-insensitive.

Quoting from https://developer.apple.com/library/content/documentation/Fi... "The case-insensitive variant of APFS is normalization-preserving, but not normalization-sensitive."

"The case-sensitive variant of APFS is both normalization-preserving and normalization-sensitive."

It should just always be form-preserving and form-INsensitive.

There is absolutely no reason to want the combination they give. It makes me suspect that they didn't bother separating case- and form-insensitivity, although I can't imagine why one wouldn't. It seems like.. a mistake. Someone misunderstood.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#162

Post by cryptonector »

rtpg wrote: post #160
konstmonst wrote: post #93

Earlier quoted context omitted.

I also think encoding doesn't belong into a file system. Let the names be arrays of bytes and leave the encoding to the people that use it, be it utf-8, utf-16 or something entirely different.

the problem that arrives there almost instantly is that people want to see a list of filenames. If you're ever going to do some sort of "displaying" of data, you cannot store it as bytes. You need to know what characters things are supposed to be presented as. You could imagine not settling on a specific encoding, but you must know the encoding. Unless your plan is to show a list of numbers to users.

Right. And it's just rather difficult for the kernel to know the user-land LC_CTYPE setting, so it doesn't, so it doesn't know the encoding of any string, so the best thing to do is assume UTF-8. If you want some other encoding, then libc's syscall stubs will have to do codeset conversions, or if not then you have to make sure that you only use that codeset everywhere.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#163

Post by kps »

rurban wrote: post #73
Piskvorrr wrote: post #29

Earlier quoted context omitted.

Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é ), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has…

The biggest actual problem is Greek vs Cyrillic. They appear the same, but use different encodings. Any filename (directory entry as identifier) must forbid mixed scripts. See TR31 http://www.unicode.org/reports/tr31/ Combining marks or RTL switching tricks are the other popular spoofs.

  > Any filename (directory entry as identifier) must forbid mixed scripts.
So I'm not allowed to name my files “A=πr²” or “10kΩ Резисторы”?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#164

Post by klodolph »

cryptonector wrote: post #96
klodolph wrote: post #45

Earlier quoted context omitted.

NTFS, by comparison, does the "bag of wchar_t" instead of the "bag of byte" for filenames.

"Bag of code units" would be a better description (if it's correct anyways). This is not very good. You want normalization-preserving/insensitive behavior instead.

What about "bag of wchar_t" doesn't preserve normalization? Or am I not sure what you're trying to say?

It's no more a "bag of code units" than Linux filesystems store a "bag of code units". Windows will barf back whatever wchar_t array you give it, just like Linux will barf back whatever char array you give it.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#165

Post by thaumasiotes »

rtpg wrote: post #160
konstmonst wrote: post #93

Earlier quoted context omitted.

I also think encoding doesn't belong into a file system. Let the names be arrays of bytes and leave the encoding to the people that use it, be it utf-8, utf-16 or something entirely different.

the problem that arrives there almost instantly is that people want to see a list of filenames. If you're ever going to do some sort of "displaying" of data, you cannot store it as bytes. You need to know what characters things are supposed to be presented as. You could imagine not settling on a specific encoding, but you must know the encoding. Unless your plan is to show a list of numbers to users.

The filesystem can't be responsible for displaying anything, though. Displaying filenames is the job of the shell / window manager / etc.

The filesystem is much better off handling filenames as a number of arbitrary bytes. Let people who want to put weird bytes in their filenames see ugly filenames along the lines of "\x00 Can you see this?"

Re: APFS is not safe to use with names which have Unicode normalisation issues

#166

Post by thaumasiotes »

cryptonector wrote: post #145
jrimbault wrote: post #18

Earlier quoted context omitted.

I'm just being sarcastic: should we also blame the computers of old for not being able to handle more than 255 characters ?

No, just developers. Of course, the use of bytes for characters goes back a long time, to times when computers had small memories and disk (and other) storage capacities. And to even before then, to the days of telexes and typewriters. It's completely understandable. But UTF-8 is genius, which is why we use it. Incidentally, ASCII was actually a multi-byte codeset... since one could combine most lower-case characters…

I've always wondered what the concept of a "backspace" character was supposed to mean. (I tried printing it to erase a previously printed character, but that doesn't work.)

I guess it's another relic of the idea that computer output goes to a printer rather than a display?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#167

Post by Piskvorrr »

thaumasiotes wrote: post #166
cryptonector wrote: post #145

Earlier quoted context omitted.

No, just developers. Of course, the use of bytes for characters goes back a long time, to times when computers had small memories and disk (and other) storage capacities. And to even before then, to the days of telexes and typewriters. It's completely understandable. But UTF-8 is genius, which is why we use it. Incidentally, ASCII was actually a multi-byte codeset... since one could combine most lower-case characters…

I've always wondered what the concept of a "backspace" character was supposed to mean. (I tried printing it to erase a previously printed character, but that doesn't work.) I guess it's another relic of the idea that computer output goes to a printer rather than a display?

Indeed it is. Or rather, output goes to a remote printer. Morse code - teletype (backspace appears here) - mainframe - ASCII - Unicode (you are here).

Re: APFS is not safe to use with names which have Unicode normalisation issues

#168

Post by Piskvorrr »

cryptonector wrote: post #146
rurban wrote: post #73

Earlier quoted context omitted.

The biggest actual problem is Greek vs Cyrillic. They appear the same, but use different encodings. Any filename (directory entry as identifier) must forbid mixed scripts. See TR31 http://www.unicode.org/reports/tr31/ Combining marks or RTL switching tricks are the other popular spoofs.

Confusables are a real problem. Some Cyrillic and Greek letters even look like Latin letters. And in some fonts '1' (one) and 'l' (ell) look the same even in Latin. Dealing with these is much harder than with normalization forms. And confusables are a source of serious security headaches.

1Il|!

Re: APFS is not safe to use with names which have Unicode normalisation issues

#169

Post by _urga »

For tips on working with different filesystems, especially filesystems that have different approaches to normalization, see an article I wrote:

https://nodejs.org/en/docs/guides/working-with-different-fil...

The gist is that normalization should only ever be used for comparison (if needed, e.g. "do these two files have filenames that would look the same to user"), and never for changing data (filenames are user data and should be stored verbatim without normalization). HFS+ should never have used normalization in the first place. You can think of normalization as essentially a lossy hash function (you cannot get back to HFS+ NFD form once you normalize it to NFC - HFS+ NFD and NFD proper are not the same thing - many people don't realize this). Using normalization for anything other than temporary comparison leads to data loss.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#170

Post by _urga »

cryptonector wrote: post #84
xoa wrote: post #64

Earlier quoted context omitted.

> HFS+ was the only file system I know of that was doing Unicode normalisation I've seen something in a couple comments now to this effect, does nobody here do anything with ZFS at all? It's a pretty great filesystem on not just illumos but FreeBSD, Linux, and macOS, and it has full native normalization support (and for all 4 forms too). Normalizing is optional, but it's definitely there and with Macs I have been usi…

Thank you for pointing this out. In particular ZFS has normalization-preserving/normalization-insensitive behavior, which is far superior to HFS+'s opinionated normalization-on-create (to a form that is different from the common input modes' output!).

Yes, HFS+ implements normalization-insensitive behavior through not being normalization-preserving, in the same way that some FATs might implement case-insensitive behavior through not being case-preserving. They didn't realize that you could have both features: normalization-preserving and normalization-insensitive.
Post reply on HN