Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hell that is filename encoding (2016)

beets.io

Post reply on HN
Print view View original

81–90 of 121 posts

Re: The hell that is filename encoding (2016)

#81

Post by CodesInChaos »

brazzy wrote: post #67
Hermel wrote: post #40

Earlier quoted context omitted.

I think Java does that. Thanks to its built-in libraries, you can write code that works on multiplie platforms with different file systems. However, you still need to know yourself which characters are legal in which system (for example, ‘:’ is not a legal character in file names on windows, but it is on OS X.

At least since Java 7, the API has a concept of a filesystem which can create filesystem-specific Path objects and, if it encounters illegal characters, throws an InvalidPathException that tells you which character was illegal.

Sounds tricky to get right, considering the filesystem can change in the middle of a path.

Re: The hell that is filename encoding (2016)

#82

Post by Avamander »

0x0 wrote: post #43
jabl wrote: post #39

Funny(???) warstory: 1. Back in the days, we were using a Linux NFS server, with NFSv3, and out-of-the-box locale was iso-8859-1 (latin1). Life was good, except for occasional problems with people with strange non-latin1 names, or documents with non-latin1 names etc. 2. At some point, we switch to using UTF-8 by default. Telling users to use convmv to rename their files when they are ready to switch to the new defaul…

A similar thing happens with Java's file and directory APIs on Linux. IIRC in Java filenames are Strings. If your vm is configured with utf8 as "file.encoding" and you have non-utf8-compliant filenames on your filesystem, those files are completely inaccessible to Java!

This is kind-of why I think there should be heavy push towards not supporting any other encoding than UTF-8.

Re: The hell that is filename encoding (2016)

#83

Post by brazzy »

CodesInChaos wrote: post #81
brazzy wrote: post #67

Earlier quoted context omitted.

At least since Java 7, the API has a concept of a filesystem which can create filesystem-specific Path objects and, if it encounters illegal characters, throws an InvalidPathException that tells you which character was illegal.

Sounds tricky to get right, considering the filesystem can change in the middle of a path.

I haven't worked extensively with the API, but I believe that it represents filesystem instances accessible to the JVM, not abstract technical filesystems.

So in the case of a Unix filesystem tree with various mount points, the FileSystem object would throw the exception iff the path you're trying to construct is illegal for the actual filesystem configuration in the system.

Re: The hell that is filename encoding (2016)

#84

Post by linsomniac »

I found a similar problem with my backups (Ugh. :-)

A year ago I went on a trip and some combination of the humidity, the travel, and the 6 year old Thinkpad resulted in my laptop not booting.

I had been experimenting with Borg to backup the system, and so I tried using Borg to restore the latest copy onto the new laptop. Turns out that I have a bunch of files on my laptop that have names with weird characters in them: rips of my CD collection. I couldn't find any combination of settings and environment and locale that would allow borg to recover or skip these files and recover everything else.

Now, I had 2-3 other copies of the data (my pre-borg backups, the original SSD which was still readable, a few other rsync copies), so it wasn't a big deal.

But, as always, test your recoveries!

Re: The hell that is filename encoding (2016)

#85

Post by throwaway7767 »

blattimwind wrote: post #77
throwaway7767 wrote: post #76

IBM's backup software TSM/Spectrum Protect messes this up as well. If the machine has a UTF-8 encoding (like, say, every modern system), it will try to treat filenames as valid UTF-8 strings and fail to back up files which don't fulfill that assumption. The "solution" is to run the TSM software with a single-byte locale like en_US. I've seen a number of shops that were silently missing files from backup from old syst…

I don't think any backup software actually can do the right thing(tm). Some might preserve (or attempt so, anyway) binary representation, others attempt to preserve unicode codepoint-space representation... ... most do neither, but rather do ${complex thing emerging from combination of implementation details of runtime and backup tool, impossible to reproduce in any other runtime, likely platform- and environment dep…

> I don't think any backup software actually can do the right thing(tm).

Sure it can. In this case, I'd say treating the filename as a bag of bytes is the correct way to go, as that's the way the OS treats them. Translating filenames between character sets should not be part of a backup systems job.

There are valid setups where different software on the same machine might be running with different character sets for legacy reasons. In that case there is no correct way to handle the filenames as text. But treating it as a bag-of-bytes will always work consistently.

Also, the one purpose of a backup system is to back up the files on the filesystem. If it can't back up some files that the OS considers valid, it's the backup software that failed.

Re: The hell that is filename encoding (2016)

#86

Post by throwaway7767 »

wazoox wrote: post #74
jabl wrote: post #39

Funny(???) warstory: 1. Back in the days, we were using a Linux NFS server, with NFSv3, and out-of-the-box locale was iso-8859-1 (latin1). Life was good, except for occasional problems with people with strange non-latin1 names, or documents with non-latin1 names etc. 2. At some point, we switch to using UTF-8 by default. Telling users to use convmv to rename their files when they are ready to switch to the new defaul…

Ah yes, had the same fun problem at a customer's facility last week. Moving 350 TB of data from an old DDP storage server to a Linux one. Mounting with CIFS (no other option available), an copying using "cp -a". The file names look OK after the copy on the Linux machine. However, when exporting the directory through Samba, the Macs Finder doesn't display files with accents in the names (though they appear correctly w…

Most likely it's different normalization. I've seen this before with Mac systems.

Renaming the files to use NFKC normalization fixed it. In python, you could loop through the files and do something like:

  os.rename(originalfilename, unicodedata.normalize('NFKC', originalfilename.decode('utf8')))
EDIT: You'll probably need to do this on a non-Mac system, linux for example should work.

Re: The hell that is filename encoding (2016)

#87

Post by leeter »

masklinn wrote: post #64
burfog wrote: post #47

Earlier quoted context omitted.

Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated. The kernel generally uses the 16-bit equivalent of the old Pascal string, that being a 16-bit count of 16-bit pieces of UTF-16 data. This allows a 16-bit NUL to get into various places that make the Win32 API choke.

> Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated. I usually call that ucs2-plus-surrogates, to make it clear that you may encounter unpaired surrogates, and thus invalid paths if you assume proper UTF-16.

This is actually kinda painful with NTFS as NTFS doesn't really care what's in a path other than the directory separators it's all binary. This means that different applications using different Unicode normalization will result in odd things happening. To me the right answer is NTFS should normalize all paths the same way internally but they have yet to implement it because it would break legacy systems that have un-normalized paths (I assume).

Re: The hell that is filename encoding (2016)

#88

Post by draw_down »

lifthrasiir wrote: post #2

Rust `std::path` [1] has two representations under the hood for Windows (UTF-16 plus lone surrogates) and non-Windows (bytes) for the exactly same reason. Paths are not strings nor texts. [1] https://doc.rust-lang.org/stable/std/path/

Yeah, this seems like the least frustrating way to do it, rather than trying to express one kind of path in terms of the other kind.

Re: The hell that is filename encoding (2016)

#89

Post by nialo »

I work on another music file management system, my personal special hell is playlist files. An m3u playlist file is just a new-line separated list of file paths, which can be relative or absolute, and potentially encoded in whatever locale is set on the users computer. Some fun issues:

* Windows and Mac filesystems are generally case-insensitive, so some users will have the file names in the playlist file in one case and the actual file names on disk in another format * Sometimes file paths cross between two different filesystems, because one is mounted in the other with a USB drive or over CIFS or similar. Sometimes these two different filesystems have different case sensitivities * There's no way to know how the playlist file was encoded * HFS+ normalizes file paths to Unicode NFD, but there's no guarantee that the paths in a playlist file will be normalized. Also, sometimes users generate an m3u file on a Windows system and expect it to just work on a Mac. Also, the filesystem nesting problem with network or USB mounts can happen this way too.

Re: The hell that is filename encoding (2016)

#90

Post by freeone3000 »

leeter wrote: post #87
masklinn wrote: post #64

Earlier quoted context omitted.

> Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated. I usually call that ucs2-plus-surrogates, to make it clear that you may encounter unpaired surrogates, and thus invalid paths if you assume proper UTF-16.

This is actually kinda painful with NTFS as NTFS doesn't really care what's in a path other than the directory separators it's all binary. This means that different applications using different Unicode normalization will result in odd things happening. To me the right answer is NTFS should normalize all paths the same way internally but they have yet to implement it because it would break legacy systems that have un-…

Precisely.

Programs that internally use ShiftJIS, for instance, would stop functioning on UTF-16 enforced compatability or normalization. They're currently "broken" (as in, operating incorrectly) but in a way that works.

Post reply on HN