Do "most non-English" languages have normalization issues? At least CJK users do not.
Even English has things like "reëlect" instead of "reelect" and so on. http://www.newyorker.com/culture/culture-desk/the-curse-of-t...
APFS is not safe to use with names which have Unicode normalisation issues
151–160 of 191 posts
Re: APFS is not safe to use with names which have Unicode normalisation issues
#152Earlier quoted context omitted.
So it is correct then, the file system doesn't concern itself with normalization. File names are stored internally as utf-8 strings which are just a sequence of bytes.
A terrible mistake. APFS should have been normalization-preserving/normalization-insensitive.
Quoting from https://developer.apple.com/library/content/documentation/Fi...
"The case-insensitive variant of APFS is normalization-preserving, but not normalization-sensitive."
Re: APFS is not safe to use with names which have Unicode normalisation issues
#153i guess we'll see a new option on the finder "show non-normalized files" next to "show hidden files" pretty soon. i don't see how they could automatically solve all the potential mismatch issues otherwise.
ls | od -c?
Re: APFS is not safe to use with names which have Unicode normalisation issues
#154Earlier quoted context omitted.
Normalization is very expensive and does not belong at the FS level. This kills performance for some classes of applications. The comparison with other filesystems does not hold since applications for other OSs have always been developed with no normalization at FS level, and hence it was done by the applications, or through the use of high-level OS APIs. Mac applications, on the other hand, expect it to be the respo…
Normalization is very expensive and does not belong at the FS level. This kills performance for some classes of applications. Why? It's done once per open, not once per I/O.
In HFS+ it's even only done once per-create, so pretty cheap.
In ZFS it's once per-open()/stat()/and so on. But still, not at all on readdir(), and anyways, it's highly optimized. For an all ASCII filename the slow path is never taken, and for a mostly ASCII filename the slow path is only taken for non-ASCII codepoints that are followed by combining codepoints (that check is itself a slower-than-the-fast path, but still faster than the slowest path).
Basically, it's noise.
Re: APFS is not safe to use with names which have Unicode normalisation issues
#155Earlier quoted context omitted.
> Should filesystems be making up for our Unicode shortcomings? Absolutely, yes. File names are text by their very definition; that we've been treating them as "bags of bytes" is a historical tragedy. At the very least, file names need to be displayed, as text, to the user, so they should be stored as text , that is in some well-defined encoding, and yes, it should be the job of the filesystem driver / kernel to enfo…
But what is text? Not everyone wants to use unicode. It is dependent of the platform, the region, the OS and on many other different things like LC_* variables on linux. Why should a filesystem depend on those too?
Unicode is meant to replace all the other codesets. It's time to move on from them, except for compatibility reasons.
Anyways, it's possible to allow UTF-8 and non-UTF-8 on the same filesystem, and still provide form-preserving/insensitive behavior... ZFS does it.
Re: APFS is not safe to use with names which have Unicode normalisation issues
#156Earlier quoted context omitted.
So it is correct then, the file system doesn't concern itself with normalization. File names are stored internally as utf-8 strings which are just a sequence of bytes.
A terrible mistake. APFS should have been normalization-preserving/normalization-insensitive.
[deleted]
Re: APFS is not safe to use with names which have Unicode normalisation issues
#157Earlier quoted context omitted.
A terrible mistake. APFS should have been normalization-preserving/normalization-insensitive.
Quoting from https://developer.apple.com/library/content/documentation/Fi... "The case-insensitive variant of APFS is normalization-preserving, but not normalization-sensitive."
[deleted]
Re: APFS is not safe to use with names which have Unicode normalisation issues
#158[deleted]
Re: APFS is not safe to use with names which have Unicode normalisation issues
#159There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…
> everyone else chose the bag of bytes approach.
Sequence of bytes. "spiroagnew.txt" and "xtt.growapenis" are not the same filename.
Re: APFS is not safe to use with names which have Unicode normalisation issues
#160I've been saying this for years: http://cryptonector.com/2010/04/on-unicode-normalization-or-... (originally at blogs.sun.com, now blogs.oracle.com, though I can't find it there). The problem is that most input methods produce something close to NFC while HFS+ decomposes to something close to NFD. Which means that if you cut-n-paste non-ASCII Unicode names from a finder into any app that doesn't normalize, then you'l…
I also think encoding doesn't belong into a file system. Let the names be arrays of bytes and leave the encoding to the people that use it, be it utf-8, utf-16 or something entirely different.
the problem that arrives there almost instantly is that people want to see a list of filenames.
If you're ever going to do some sort of "displaying" of data, you cannot store it as bytes. You need to know what characters things are supposed to be presented as.
You could imagine not settling on a specific encoding, but you must know the encoding. Unless your plan is to show a list of numbers to users.