Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hell that is filename encoding (2016)

beets.io

Post reply on HN
Print view View original

31–40 of 121 posts

Re: The hell that is filename encoding (2016)

#31

Post by quietbritishjim »

> on Windows, paths are fundamentally text

They were back when there were less than 2^16 characters in the Unicode standard. Back then each two-byte word in a filename corresponded exactly with a Unicode code point.

Now there are more than 2^16 but well under 2^32, Windows uses UTF-16 in filenames. That is, Unicode code points above 2^15 are obtained by a pair of special Unicode code points in the range 2^15-2^16 called surrogates; surrogate pairs need to be collapsed into a single code point when decoding the file name. Surrogates are exactly those things that Python uses on Linux to hide bytes that are not valid UTF-8. Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16).

In summary, on Windows, you end up with effectively the same situation as Linux: file names that are supposed to be in one encoding (UTF16) but contain invalid data for that encoding.

Re: The hell that is filename encoding (2016)

#32

Post by tcoff91 »

cryptonector wrote: post #25
lowbloodsugar wrote: post #23

"NTFS allows any sequence of 16-bit values for name encoding (file names, stream names, index names, etc.) except 0x0000. This means UTF-16 code units are supported, but the file system does not check whether a sequence is valid UTF-16 (it allows any sequence of short values, not restricted to those in the Unicode standard). " - from wikipedia NTFS page [1] So if you assume that NTFS filename is valid UTF-16 and conv…

Doesn't it (or Windows) also disallow the path component separator character(s) ('/' and '\')? Unix and alike disallow NULs and /, for obvious reasons.

There are a number of characters like path separators that cannot be part of a file name on windows. However I am not sure if this is enforced by the OS APIs or by NTFS itself. It is entirely possible that NTFS could allow something that higher layers don’t.

Re: The hell that is filename encoding (2016)

#33

Post by mehrdadn »

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

As far as I can tell, file systems are inherently broken by design; this isn't an implementation issue. For example, the notion of finding a file by its path is just riddled with race conditions. If you create files /a/b/c and /a/b/d, are c and d necessarily in the same directory? Not really, because someone could have moved around the parent directories in between. But we conveniently assume paths stay the same... except, of course, when they change. Now try actually formalizing what exactly that means from a global (multi-program) perspective!

Re: The hell that is filename encoding (2016)

#34

Post by chungy »

quietbritishjim wrote: post #31

> on Windows, paths are fundamentally text They were back when there were less than 2^16 characters in the Unicode standard. Back then each two-byte word in a filename corresponded exactly with a Unicode code point. Now there are more than 2^16 but well under 2^32, Windows uses UTF-16 in filenames. That is, Unicode code points above 2^15 are obtained by a pair of special Unicode code points in the range 2^15-2^16 cal…

> Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16).

NTFS (and Windows as a whole) does not use UTF-16, it uses UCS-2. It is a subtle difference, but surrogate pairs didn't exist in UCS-2.

Re: The hell that is filename encoding (2016)

#35

Post by quietbritishjim »

hackits wrote: post #8
gear54rus wrote: post #6

Earlier quoted context omitted.

Honest question, why is that feature complex? What is the problem in looking at last part after dot?

Because Microsoft made their new office extension .xml if that doesn't make your head spin i don't know what else will.

Is this in very recent Office versions? Or are you talking about the XML format used by Office 2007 upwards? Because that does still use different file extensions for different programs, obtained by adding an 'x' on their old binary format extensions (.docx rather than .doc, .xlsx rather than .xls, ...). They're zip archives of XML files rather than individual XML files so .xml wouldn't make any sense.

I got the impression the problem in the comment above was too many extensions rather than too few. For example, you have use .docm etc rather than .docx if your document contains macros otherwise Word will refuse to open it (this is a security feature to prevent document viruses). But it sounds like there are many others.

Re: The hell that is filename encoding (2016)

#36

Post by Moru »

lowbloodsugar wrote: post #23

"NTFS allows any sequence of 16-bit values for name encoding (file names, stream names, index names, etc.) except 0x0000. This means UTF-16 code units are supported, but the file system does not check whether a sequence is valid UTF-16 (it allows any sequence of short values, not restricted to those in the Unicode standard). " - from wikipedia NTFS page [1] So if you assume that NTFS filename is valid UTF-16 and conv…

There was a time when some of our customers had lots of problems with gigantic files on their drives that was impossible to delete with windows explorer. I would come home to them and help them delete the files with the command line using filename*.ext to catch them. My guess was that the filename had some protected characters that windows explorer didn't allow. Don't remember how they ended up with the files but most likely some download program and someone having a laugh :-)

Re: The hell that is filename encoding (2016)

#37

Post by quietbritishjim »

chungy wrote: post #34
quietbritishjim wrote: post #31

> on Windows, paths are fundamentally text They were back when there were less than 2^16 characters in the Unicode standard. Back then each two-byte word in a filename corresponded exactly with a Unicode code point. Now there are more than 2^16 but well under 2^32, Windows uses UTF-16 in filenames. That is, Unicode code points above 2^15 are obtained by a pair of special Unicode code points in the range 2^15-2^16 cal…

> Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16). NTFS (and Windows as a whole) does not use UTF-16, it uses UCS-2. It is a subtle difference, but surrogate pairs didn't exist in UCS-2.

Indeed. I should have written something more like "in places Windows appears to accept UTF-16". But there is certainly a little truth to it: if you pass a surrogate pair to e.g. the text of a Windows label using an appropriate Unicode font, I believe it will show its UTF-16 interpretation.

Re: The hell that is filename encoding (2016)

#38

Post by mook »

mehrdadn wrote: post #33
zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

As far as I can tell, file systems are inherently broken by design; this isn't an implementation issue. For example, the notion of finding a file by its path is just riddled with race conditions. If you create files /a/b/c and /a/b/d, are c and d necessarily in the same directory? Not really, because someone could have moved around the parent directories in between. But we conveniently assume paths stay the same... e…

Isn't that why openat() exists? Of course, that isn't used nearly as much because it's annoying to have to do things that way, but it seems like the sort of thing if you need it.

The "open things by paths" thing, IIRC, is part of the reason Windows doesn't like to let you delete open files by default.

Re: The hell that is filename encoding (2016)

#39

Post by jabl »

Funny(???) warstory:

1. Back in the days, we were using a Linux NFS server, with NFSv3, and out-of-the-box locale was iso-8859-1 (latin1). Life was good, except for occasional problems with people with strange non-latin1 names, or documents with non-latin1 names etc.

2. At some point, we switch to using UTF-8 by default. Telling users to use convmv to rename their files when they are ready to switch to the new defaults. Most people ignored this, of course, but files with now invalid utf-8 were mostly fine, just with the occasional "?"'s in the names.

3. Switch to NFSv4. Invisible to end users. NFSv4 per se requires that paths are UTF-8 encoded, but in practice the Linux NFS server and client just pass along a bag of bytes, so invalid UTF-8 just worked as fine as it did previously.

4. Switch from a Linux NFS server to a netapp.

5. User complains that files are missing. Initial comparison with the old Linux NFS server, which was still online, shows no problems. Problem occurs only on user workstation, not on admin box which has both the old Linux NFS and netapp directory trees mounted. Investigation on users workstation shows that in some cases lots of files appear to be missing, including ones which plain ASCII names.

- Turns out that the admin box had the netapp mounted with NFSv3, and thus everything appeared Ok there, including the rsync from Linux NFS -> netapp in the first place.

- However, when mounted using NFSv4, netapp follows the spec and does not like non-utf8 paths. Does it report an error then? Hell no, the NFS READDIR (READDIRPLUS?) message reply just stops returning directory entries when it hits the first one with invalid UTF-8. And thus you get a partial directory listing. GAAAH!

- So the solution was to run convmv centrally (from the admin box which had the netapp mounted with NFSv3) for the entire directory tree which had been moved.

Re: The hell that is filename encoding (2016)

#40

Post by Hermel »

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

I think Java does that. Thanks to its built-in libraries, you can write code that works on multiplie platforms with different file systems. However, you still need to know yourself which characters are legal in which system (for example, ‘:’ is not a legal character in file names on windows, but it is on OS X.
Post reply on HN