Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hell that is filename encoding (2016)

beets.io

Post reply on HN
Print view View original

71–80 of 121 posts

Re: The hell that is filename encoding (2016)

#71

Post by kalleboo »

Someone wrote: post #57
loeg wrote: post #13

Earlier quoted context omitted.

Not only are they normalized unicode, they're normalized decomposed , and not only that, but slightly non-standard (does not conform to standard Unicode "NFD" form). (Or at least, this was the case with HFS. I haven't followed APFS closely enough to say for that.)

HFS intended to store name entries in “US display order”, but it had a bug in sorting. https://developer.apple.com/legacy/library/technotes/tn/tn11... : ”HFS uses 31-byte strings to store file names. HFS does not store any kind of script information with the file name to indicate how it should be interpreted. File names are compared and sorted using a routine that assumes a Roman script, wreaking havoc for names that…

For non-Macheads who are confused, this refers to the original HFS from 1985, which was replaced with HFS Plus in 1998:

> HFS Plus uses up to 255 Unicode characters to store file names. Allowing up to 255 characters makes it easier to have very descriptive names. Long names are especially useful when the name is computer-generated (such as Java class names).

I have to admit I let out a laugh at Apple's reference to Java class names...

Re: The hell that is filename encoding (2016)

#72

Post by icebraining »

ensiferum wrote: post #46
jabl wrote: post #39

Funny(???) warstory: 1. Back in the days, we were using a Linux NFS server, with NFSv3, and out-of-the-box locale was iso-8859-1 (latin1). Life was good, except for occasional problems with people with strange non-latin1 names, or documents with non-latin1 names etc. 2. At some point, we switch to using UTF-8 by default. Telling users to use convmv to rename their files when they are ready to switch to the new defaul…

Useless non-sense like this is the main reason why i desperately want to move away from software engineering.

Careful, though. Plain human bureaucracy can be just as bad - and often harder to debug :)

Re: The hell that is filename encoding (2016)

#73

Post by gpvos »

dvlsg wrote: post #55
mehrdadn wrote: post #52

Earlier quoted context omitted.

Only in some special cases, not in general.

Really? I use forward slashes all the time in windows 10. I don't think I've run into a problem yet.

They work everywhere, except on the command line or file dialogue windows.

Re: The hell that is filename encoding (2016)

#74

Post by wazoox »

jabl wrote: post #39

Funny(???) warstory: 1. Back in the days, we were using a Linux NFS server, with NFSv3, and out-of-the-box locale was iso-8859-1 (latin1). Life was good, except for occasional problems with people with strange non-latin1 names, or documents with non-latin1 names etc. 2. At some point, we switch to using UTF-8 by default. Telling users to use convmv to rename their files when they are ready to switch to the new defaul…

Ah yes, had the same fun problem at a customer's facility last week. Moving 350 TB of data from an old DDP storage server to a Linux one. Mounting with CIFS (no other option available), an copying using "cp -a".

The file names look OK after the copy on the Linux machine. However, when exporting the directory through Samba, the Macs Finder doesn't display files with accents in the names (though they appear correctly with "ls", weird...).

So the user copies the files again, using the Finder. Now I have files with exactly the same name (uhhhhh???):

# ls -l Mmo-1. -rw-rw-rw- 1 root root 8417218 6 sept. 2013 Mémo-1.aif -rwxr--r-- 1 test test 8417218 6 sept. 2013 Mémo-1.aif -rw-rw-rw- 1 root root 363175 6 sept. 2013 Mémo-1.m4a -rwxr--r-- 1 test test 363175 6 sept. 2013 Mémo-1.m4a

Yes, it looks like two files have exactly the same name, but actually they're different: one as "é" encoded as 0xCC81, and the other one (the "good one") as 0xC3A9. Why is that? Why does one work with the Finder, and the other doesn't? who knows.

Re: The hell that is filename encoding (2016)

#75

Post by gpvos »

cryptonector wrote: post #15
netheril96 wrote: post #5

It misses a even more complex, I'd say insane, encoding problem: on HFS+ (or even APFS now?) filenames are unicode normalized.

HFS+'s use of NFD made even more insane by the fact that OS X's input modes prefer to produce NFC anyways. And besides, so do other OSes' input modes, so in any heterogeneous system this is a nightmare. This is why ZFS does form-insensitive directory lookups (and hashing)[0] rather than normalize-on-CREATE! I'm so glad ZFS got it right, and can stand as a model for all. (I implemented none of that functionality, thou…

Many people rail against case-insensitive lookups like on Windows. How is this different?

Re: The hell that is filename encoding (2016)

#76

Post by throwaway7767 »

IBM's backup software TSM/Spectrum Protect messes this up as well.

If the machine has a UTF-8 encoding (like, say, every modern system), it will try to treat filenames as valid UTF-8 strings and fail to back up files which don't fulfill that assumption. The "solution" is to run the TSM software with a single-byte locale like en_US.

I've seen a number of shops that were silently missing files from backup from old systems because of this problem.

Re: The hell that is filename encoding (2016)

#77

Post by blattimwind »

throwaway7767 wrote: post #76

IBM's backup software TSM/Spectrum Protect messes this up as well. If the machine has a UTF-8 encoding (like, say, every modern system), it will try to treat filenames as valid UTF-8 strings and fail to back up files which don't fulfill that assumption. The "solution" is to run the TSM software with a single-byte locale like en_US. I've seen a number of shops that were silently missing files from backup from old syst…

I don't think any backup software actually can do the right thing(tm). Some might preserve (or attempt so, anyway) binary representation, others attempt to preserve unicode codepoint-space representation...

... most do neither, but rather do ${complex thing emerging from combination of implementation details of runtime and backup tool, impossible to reproduce in any other runtime, likely platform- and environment dependent; the same backup likely restores in different ways on different machines, and the same source files create different backups on different machines; creating a backup on one machine and restoring it on another does not generally result in the same files; and I have not yet mentioned what might happen if you mount the same source file system from different platforms, because results might vary a lot; also, we are only talking about paths here, not any of the other plethora of things that can and will be different between any element in OSxFSxEnv}.

Re: The hell that is filename encoding (2016)

#78

Post by lmm »

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

What are you trying to achieve? If you want to store data for your own application, you'd use something higher-level than a filesystem (e.g. a database); those exist and offer sane semantics. The only reason to interact with the OS-level filesystem is to use it to communicate with other programs, in which case the insanity is necessary: if you want to e.g. read files created by other programs, you have to be prepared to deal with malformed names, because other programs will create files with malformed names.

Re: The hell that is filename encoding (2016)

#79

Post by lmm »

gpvos wrote: post #75
cryptonector wrote: post #15

Earlier quoted context omitted.

HFS+'s use of NFD made even more insane by the fact that OS X's input modes prefer to produce NFC anyways. And besides, so do other OSes' input modes, so in any heterogeneous system this is a nightmare. This is why ZFS does form-insensitive directory lookups (and hashing)[0] rather than normalize-on-CREATE! I'm so glad ZFS got it right, and can stand as a model for all. (I implemented none of that functionality, thou…

Many people rail against case-insensitive lookups like on Windows. How is this different?

It isn't. Case-insensitive lookups are the right thing; unix people oppose them out of tribalism rather than anything else.

Re: The hell that is filename encoding (2016)

#80

Post by ryl00 »

wongarsu wrote: post #58
zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

What is sane path handling? Alone on windows, the maximum path length is 260 characters, except when you use extended-length paths which have a 4 character prefix and a maximum length of 32,767 characters. A sane API for reading files probably converts your paths to extended-length paths. But if you do the same thing for writing files your users start calling you insane again, because most of Windows (including Windo…

There's support for >260 char path lengths now without the \\?\ prefix, but I think you need to modify the registry, plus the application needs to opt into the new behavior via a setting in its manifest.
Post reply on HN