Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hell that is filename encoding (2016)

beets.io

Post reply on HN
Print view View original

21–30 of 121 posts

Re: The hell that is filename encoding (2016)

#21

Post by cryptonector »

cryptonector wrote: post #16
loeg wrote: post #13

Earlier quoted context omitted.

Not only are they normalized unicode, they're normalized decomposed , and not only that, but slightly non-standard (does not conform to standard Unicode "NFD" form). (Or at least, this was the case with HFS. I haven't followed APFS closely enough to say for that.)

NFD hadn't been standardized at the time. IIUC the reason they did this is that they wanted directories to be canonically ordered on disk, and they thought decomposition would naturally yield better results than pre-composition. I'm not sure that's right, and frankly I don't care either, because the most important thing to note is that input methods (especially for European languages) by and large produce NFC, and mo…

I should add that because different locales have different collations, it's not that important that directories be order by name. It's good enough that directories be somewhat ordered, and even that they not be at all. GUIs will almost always let you sort by name and/or date, and the same goes for ls(1), so, really, it doesn't matter at all.

IMO it was a terrible mistake to normalize to NFD on create. Normalizing to NFC on create would still have been a mistake, but a lesser one.

Re: The hell that is filename encoding (2016)

#22

Post by cryptonector »

chungy wrote: post #18
zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

ZFS has a `utf8only` property that constrains filenames to be UTF-8 only... :)

It does more than that! It also does form-insensitive directory lookups.

Re: The hell that is filename encoding (2016)

#23

Post by lowbloodsugar »

"NTFS allows any sequence of 16-bit values for name encoding (file names, stream names, index names, etc.) except 0x0000. This means UTF-16 code units are supported, but the file system does not check whether a sequence is valid UTF-16 (it allows any sequence of short values, not restricted to those in the Unicode standard). "

- from wikipedia NTFS page [1]

So if you assume that NTFS filename is valid UTF-16 and convert it to UTF-8 there might be a problem. Basically they can be any sequence of 16-bit values.

  [1] https://en.wikipedia.org/wiki/NTFS

Re: The hell that is filename encoding (2016)

#24

Post by cryptonector »

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

ZFS handles internationalization about as well as could be hoped for. You can forbid non-UTF-8 strings (note that ZFS doesn't know if some string that is valid UTF-8 is actually encoded in UTF-8 -- it might not be), and ZFS does form-insensitive directory lookups, so if you copy some normalized-to-NFD files from OS X, it will work out fine.

Making filesystems codeset-aware is not worth the trouble. It's best instead to just use UTF-8 locales everywhere. If you need to deal with other codesets, convert as needed, but don't use non-UTF-8 locales.

On ZFS with a fast ZIL fsync()/sync() function a lot like write barriers, which is what we really need. Actually, what we really need is for all filesystem operations to be available with async system calls, write barriers included.

Re: The hell that is filename encoding (2016)

#25

Post by cryptonector »

lowbloodsugar wrote: post #23

"NTFS allows any sequence of 16-bit values for name encoding (file names, stream names, index names, etc.) except 0x0000. This means UTF-16 code units are supported, but the file system does not check whether a sequence is valid UTF-16 (it allows any sequence of short values, not restricted to those in the Unicode standard). " - from wikipedia NTFS page [1] So if you assume that NTFS filename is valid UTF-16 and conv…

Doesn't it (or Windows) also disallow the path component separator character(s) ('/' and '\')?

Unix and alike disallow NULs and /, for obvious reasons.

Re: The hell that is filename encoding (2016)

#26

Post by heeen2 »

gear54rus wrote: post #19
hackits wrote: post #8

Earlier quoted context omitted.

Because Microsoft made their new office extension .xml if that doesn't make your head spin i don't know what else will.

I mean, this is only adding another entry to your array of extensions for the specific icon. It's also not a deal-braker if it doesn't work. It really is easy, I don't see what's the problem.

Maybe because it is used for all kinds of office documents. You'd have to look at the contents to see if it's a spreadsheet, text document, presentation or not even a non office pain XML file

Re: The hell that is filename encoding (2016)

#29

Post by netheril96 »

cryptonector wrote: post #14

I wrote about this eons ago: https://cryptonector.com/2006/12/filesystem-i18n/ and https://cryptonector.com/2010/04/on-unicode-normalization-or... -- these might still be available on https://blogs.oracle.com/ , though these are from my days at Sun. TL;DR, basically, the lack of ability to tag strings in the system call API with codesets means that UTF-8 is the only plausible answer, and the ends (C library system ca…

You cannot use UTF-8 locales on Windows though.

Re: The hell that is filename encoding (2016)

#30

Post by chungy »

cryptonector wrote: post #22
chungy wrote: post #18

Earlier quoted context omitted.

ZFS has a `utf8only` property that constrains filenames to be UTF-8 only... :)

It does more than that! It also does form-insensitive directory lookups.

Only if you set the normalization parameter. Otherwise it'll guarantee only that a file name is UTF8-strict and not that there aren't two files named "é" with different normalization.

Honestly, normalization should probably always be set. It gets way more confusing than case-sensitivity (which can also be changed on ZFS!) already is.

Post reply on HN