Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hell that is filename encoding (2016)

beets.io

Post reply on HN
Print view View original

41–50 of 121 posts

Re: The hell that is filename encoding (2016)

#41

Post by lifthrasiir »

cesarb wrote: post #9
lifthrasiir wrote: post #2

Rust `std::path` [1] has two representations under the hood for Windows (UTF-16 plus lone surrogates) and non-Windows (bytes) for the exactly same reason. Paths are not strings nor texts. [1] https://doc.rust-lang.org/stable/std/path/

Emphasis on the "plus lone surrogates" part. Like on Unix, Windows does not require a path to be valid Unicode. That is, on Windows, paths are fundamentally sequences of 16-bit words, just like on Unix paths are fundamentally sequences of 8-bit bytes. On neither system are paths fundamentally text.

The story then goes on further: every NTFS volume contains a special file named `$UpCase` that has a uppercase mapping for all possible 16-bit words, resulting in an 128 KiB table. This approach has an upside for backward and forward compatibility... unless you eventually need a case mapping for non-BMP characters or complex mapping that expands to multiple characters.

Re: The hell that is filename encoding (2016)

#42

Post by mehrdadn »

mook wrote: post #38
mehrdadn wrote: post #33

Earlier quoted context omitted.

As far as I can tell, file systems are inherently broken by design; this isn't an implementation issue. For example, the notion of finding a file by its path is just riddled with race conditions. If you create files /a/b/c and /a/b/d, are c and d necessarily in the same directory? Not really, because someone could have moved around the parent directories in between. But we conveniently assume paths stay the same... e…

Isn't that why openat() exists? Of course, that isn't used nearly as much because it's annoying to have to do things that way, but it seems like the sort of thing if you need it. The "open things by paths" thing, IIRC, is part of the reason Windows doesn't like to let you delete open files by default.

Yeah, and Windows has NtOpenFile(OBJECT_ATTRIBUTES*) to let you specify a parent directory, but these are only at the syscall level, not at the standard application API level (Win32 or C APIs don't allow it). The entire model exposed to normal applications has this problem.

Re: The hell that is filename encoding (2016)

#43

Post by 0x0 »

jabl wrote: post #39

Funny(???) warstory: 1. Back in the days, we were using a Linux NFS server, with NFSv3, and out-of-the-box locale was iso-8859-1 (latin1). Life was good, except for occasional problems with people with strange non-latin1 names, or documents with non-latin1 names etc. 2. At some point, we switch to using UTF-8 by default. Telling users to use convmv to rename their files when they are ready to switch to the new defaul…

A similar thing happens with Java's file and directory APIs on Linux. IIRC in Java filenames are Strings. If your vm is configured with utf8 as "file.encoding" and you have non-utf8-compliant filenames on your filesystem, those files are completely inaccessible to Java!

Re: The hell that is filename encoding (2016)

#45

Post by ktpsns »

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

The uncomfortable generic answer to this: https://xkcd.com/927/

Re: The hell that is filename encoding (2016)

#46

Post by ensiferum »

jabl wrote: post #39

Funny(???) warstory: 1. Back in the days, we were using a Linux NFS server, with NFSv3, and out-of-the-box locale was iso-8859-1 (latin1). Life was good, except for occasional problems with people with strange non-latin1 names, or documents with non-latin1 names etc. 2. At some point, we switch to using UTF-8 by default. Telling users to use convmv to rename their files when they are ready to switch to the new defaul…

Useless non-sense like this is the main reason why i desperately want to move away from software engineering.

Re: The hell that is filename encoding (2016)

#47

Post by burfog »

chungy wrote: post #34
quietbritishjim wrote: post #31

> on Windows, paths are fundamentally text They were back when there were less than 2^16 characters in the Unicode standard. Back then each two-byte word in a filename corresponded exactly with a Unicode code point. Now there are more than 2^16 but well under 2^32, Windows uses UTF-16 in filenames. That is, Unicode code points above 2^15 are obtained by a pair of special Unicode code points in the range 2^15-2^16 cal…

> Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16). NTFS (and Windows as a whole) does not use UTF-16, it uses UCS-2. It is a subtle difference, but surrogate pairs didn't exist in UCS-2.

Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated.

The kernel generally uses the 16-bit equivalent of the old Pascal string, that being a 16-bit count of 16-bit pieces of UTF-16 data. This allows a 16-bit NUL to get into various places that make the Win32 API choke.

Re: The hell that is filename encoding (2016)

#49

Post by mattbierner »

The built-in file system libraries for many languages are total footguns. Using strings as paths is a great example. I’ve had a few recent bugs around case sensitive vs case insensitive file systems because a lot of code assumes that when pathA != pathB then it must be dealing with two different resources. Not to mention the classic “doesn’t work on windows” problem: newPath = pathA + “/“ + pathB

Re: The hell that is filename encoding (2016)

#50

Post by lostmsu »

mattbierner wrote: post #49

The built-in file system libraries for many languages are total footguns. Using strings as paths is a great example. I’ve had a few recent bugs around case sensitive vs case insensitive file systems because a lot of code assumes that when pathA != pathB then it must be dealing with two different resources. Not to mention the classic “doesn’t work on windows” problem: newPath = pathA + “/“ + pathB

AFAIK, Windows understands / as a directory separator.
Post reply on HN