Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hell that is filename encoding (2016)

beets.io

Post reply on HN
Print view View original

91–100 of 121 posts

Re: The hell that is filename encoding (2016)

#91

Post by yepguy »

linsomniac wrote: post #84

I found a similar problem with my backups (Ugh. :-) A year ago I went on a trip and some combination of the humidity, the travel, and the 6 year old Thinkpad resulted in my laptop not booting. I had been experimenting with Borg to backup the system, and so I tried using Borg to restore the latest copy onto the new laptop. Turns out that I have a bunch of files on my laptop that have names with weird characters in the…

I think you could have mounted your borg backup as a FUSE filesystem, and then used rsync to restore your files.

Re: The hell that is filename encoding (2016)

#92

Post by leeter »

freeone3000 wrote: post #90
leeter wrote: post #87

Earlier quoted context omitted.

This is actually kinda painful with NTFS as NTFS doesn't really care what's in a path other than the directory separators it's all binary. This means that different applications using different Unicode normalization will result in odd things happening. To me the right answer is NTFS should normalize all paths the same way internally but they have yet to implement it because it would break legacy systems that have un-…

Precisely. Programs that internally use ShiftJIS, for instance, would stop functioning on UTF-16 enforced compatability or normalization. They're currently "broken" (as in, operating incorrectly) but in a way that works.

Can you explain why this would be the case? In theory this shouldn't be an issue because any ShiftJIS conversion to Unicode should be reversible.

Re: The hell that is filename encoding (2016)

#93

Post by creeble »

nialo wrote: post #89

I work on another music file management system, my personal special hell is playlist files. An m3u playlist file is just a new-line separated list of file paths, which can be relative or absolute, and potentially encoded in whatever locale is set on the users computer. Some fun issues: * Windows and Mac filesystems are generally case-insensitive, so some users will have the file names in the playlist file in one case…

Sounds a lot like my life a couple of years ago (and intermittently since). I don't get the bug reports any more because I think customer service has learned that file name problems can be fixed by renaming the files. Not fun for the user, but a sure fix.

Ya know what kind of file names work virtually everywhere? ASCII ones.

Re: The hell that is filename encoding (2016)

#94

Post by liotier »

quietbritishjim wrote: post #31

> on Windows, paths are fundamentally text They were back when there were less than 2^16 characters in the Unicode standard. Back then each two-byte word in a filename corresponded exactly with a Unicode code point. Now there are more than 2^16 but well under 2^32, Windows uses UTF-16 in filenames. That is, Unicode code points above 2^15 are obtained by a pair of special Unicode code points in the range 2^15-2^16 cal…

> file names that are supposed to be in one encoding (UTF16) but contain invalid data for that encoding

You just described my music collection, aggregated over two decades and haphazard successive migrations. I've given up salvaging the corrupted names in an automated manner...

Re: The hell that is filename encoding (2016)

#96

Post by TazeTSchnitzel »

leeter wrote: post #92
freeone3000 wrote: post #90

Earlier quoted context omitted.

Precisely. Programs that internally use ShiftJIS, for instance, would stop functioning on UTF-16 enforced compatability or normalization. They're currently "broken" (as in, operating incorrectly) but in a way that works.

Can you explain why this would be the case? In theory this shouldn't be an issue because any ShiftJIS conversion to Unicode should be reversible.

I think it's not a perfect round-trip due to differences in how the two standards encode certain characters. You will get a correct conversion either way, but the result of a round-trip might not be bit-identical.

Re: The hell that is filename encoding (2016)

#97

Post by dwheeler »

I wrote an essay a while ago about fixing Unix/Linux filenames here: https://www.dwheeler.com/essays/fixing-unix-linux-filenames....

This is a big disconnect between "what most users expect" and "what systems actually do". Usually generally expect that filenames are sequences of characters - and today almost everyone expects that they must be in UTF-8 on a Unix-like system. That is not, of course, what most systems actually do.

Re: The hell that is filename encoding (2016)

#98

Post by rspeer »

leeter wrote: post #92
freeone3000 wrote: post #90

Earlier quoted context omitted.

Precisely. Programs that internally use ShiftJIS, for instance, would stop functioning on UTF-16 enforced compatability or normalization. They're currently "broken" (as in, operating incorrectly) but in a way that works.

Can you explain why this would be the case? In theory this shouldn't be an issue because any ShiftJIS conversion to Unicode should be reversible.

It should be reversible, sure. Just let me know when you find a standardized, correct, one-to-one mapping between JIS codepoints and Unicode codepoints. Also make sure it hasn't changed in the lifetime of the oldest software that anyone is using.

Re: The hell that is filename encoding (2016)

#99

Post by tialaramex »

lifthrasiir wrote: post #41
cesarb wrote: post #9

Earlier quoted context omitted.

Emphasis on the "plus lone surrogates" part. Like on Unix, Windows does not require a path to be valid Unicode. That is, on Windows, paths are fundamentally sequences of 16-bit words, just like on Unix paths are fundamentally sequences of 8-bit bytes. On neither system are paths fundamentally text.

The story then goes on further: every NTFS volume contains a special file named `$UpCase` that has a uppercase mapping for all possible 16-bit words, resulting in an 128 KiB table. This approach has an upside for backward and forward compatibility... unless you eventually need a case mapping for non-BMP characters or complex mapping that expands to multiple characters.

I should briefly explain why this is here:

NTFS is (usually) case preserving but not case-sensitive. So the OS needs to be able to tell whether EXAMPLE.TXT and example.txt are the "same" name, which means it needs case conversion.

Not everybody agrees about how this conversion should work. The most famous example is Turkish, but there are others. So there's an actual choice to make here.

If Windows baked this into the core OS, they might get pushback in countries where their (presumably American) defaults were culturally unacceptable.

If they made it configurable at the OS level, everything would seem fine until, say, a German tries to access a USB drive with files from a Turk on them and some files don't work correctly, or the disk just can't be mounted at all.

So, they have to bake it into each NTFS filesystem.

Re: The hell that is filename encoding (2016)

#100

Post by klodolph »

Hermel wrote: post #40
zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

I think Java does that. Thanks to its built-in libraries, you can write code that works on multiplie platforms with different file systems. However, you still need to know yourself which characters are legal in which system (for example, ‘:’ is not a legal character in file names on windows, but it is on OS X.

":" is or is not a legal character in filenames on macOS depending on which API you use!

Try saving a file or renaming a file to contain ":" using the GUI. It will not work. However, "/" is fine.

Try saving a file or renaming a file to contain "/" using the CLI. It will not work. However, ":" is fine.

A ":" in the CLI is translated to "/" in the GUI and vice versa. ":" was the directory separator in Mac OS 9 and earlier, which explains the behavior. The actual on-disk filenames will contain "/", which is translated to ":" for the POSIX API. This is for HFS and HFS+, not sure how APFS changes things.

Post reply on HN