Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hell that is filename encoding (2016)

beets.io

Post reply on HN
Print view View original

101–110 of 121 posts

Re: The hell that is filename encoding (2016)

#101

Post by cryptonector »

gpvos wrote: post #75
cryptonector wrote: post #15

Earlier quoted context omitted.

HFS+'s use of NFD made even more insane by the fact that OS X's input modes prefer to produce NFC anyways. And besides, so do other OSes' input modes, so in any heterogeneous system this is a nightmare. This is why ZFS does form-insensitive directory lookups (and hashing)[0] rather than normalize-on-CREATE! I'm so glad ZFS got it right, and can stand as a model for all. (I implemented none of that functionality, thou…

Many people rail against case-insensitive lookups like on Windows. How is this different?

People can see case. They cannot see form.

Re: The hell that is filename encoding (2016)

#102

Post by fiedzia »

jstimpfle wrote: post #69
zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

The simple thing to go about this is to make assumptions that do not always hold technically. And to be ok when these assumptions break - in this case you simply can't deliver your promises anymore. Another way to put this is "shit in, shit out". So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file. That's what I would do if I wrote a music tagger, at least. When someone…

> So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file.

The problem with invalid filenames is that doing anything at all with them might be impossible (without escaping of fixing). Your output (gui or terminal) most likely uses utf8, you can't pass invalid filename to it, so you can't even display it. Also in most situation you can't just ignore something. Imagine a text editor where user tries to open invalid file, how do you "ignore" that?

Many (if not most) application that process filenames do rely on those being valid text at least in some codepaths and they simply cannot work otherwise. The proper solution is to fail and let the user fix the problem.

Re: The hell that is filename encoding (2016)

#103

Post by cryptonector »

tcoff91 wrote: post #32
cryptonector wrote: post #25

Earlier quoted context omitted.

Doesn't it (or Windows) also disallow the path component separator character(s) ('/' and '\')? Unix and alike disallow NULs and /, for obvious reasons.

There are a number of characters like path separators that cannot be part of a file name on windows. However I am not sure if this is enforced by the OS APIs or by NTFS itself. It is entirely possible that NTFS could allow something that higher layers don’t.

If the kernel (and SMB, and...) imposes these constraints, it's fine for the filesystem to not also impose the same constraints on file naming.

Re: The hell that is filename encoding (2016)

#104

Post by fiedzia »

lmm wrote: post #78
zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

What are you trying to achieve? If you want to store data for your own application, you'd use something higher-level than a filesystem (e.g. a database); those exist and offer sane semantics. The only reason to interact with the OS-level filesystem is to use it to communicate with other programs, in which case the insanity is necessary: if you want to e.g. read files created by other programs, you have to be prepared…

> you'd use something higher-level than a filesystem (e.g. a database); those exist and offer sane semantics

In many cases they offer the exact same semantics (damn you, mysql). Also they store data in files, and those files have to be named, so... back to square one.

Re: The hell that is filename encoding (2016)

#105

Post by creeble »

tom_ wrote: post #62
creeble wrote: post #7

Earlier quoted context omitted.

They're not strings nor texts right up until the point you need to display them to users.

Same applies to many things, but that doesn’t make those things strings. Numbers might be another example.

My point is that you _always_ have to convert path names to strings to display them to users, but you don't always know how because you don't always know the (sometimes implied) encoding.

Re: The hell that is filename encoding (2016)

#106

Post by lmm »

fiedzia wrote: post #104
lmm wrote: post #78

Earlier quoted context omitted.

What are you trying to achieve? If you want to store data for your own application, you'd use something higher-level than a filesystem (e.g. a database); those exist and offer sane semantics. The only reason to interact with the OS-level filesystem is to use it to communicate with other programs, in which case the insanity is necessary: if you want to e.g. read files created by other programs, you have to be prepared…

> you'd use something higher-level than a filesystem (e.g. a database); those exist and offer sane semantics In many cases they offer the exact same semantics (damn you, mysql). Also they store data in files, and those files have to be named, so... back to square one.

> In many cases they offer the exact same semantics (damn you, mysql).

I mean bad databases exist, sure, but the solution to that is to not use those.

> Also they store data in files, and those files have to be named, so... back to square one.

Not really - the database developers have handled the fsync, buffering and what-have-you for you, so you don't have to deal with them. (And FWIW serious databases generally offer the option of storing data on raw partitions).

Re: The hell that is filename encoding (2016)

#107

Post by cryptonector »

netheril96 wrote: post #29
cryptonector wrote: post #14

I wrote about this eons ago: https://cryptonector.com/2006/12/filesystem-i18n/ and https://cryptonector.com/2010/04/on-unicode-normalization-or... -- these might still be available on https://blogs.oracle.com/ , though these are from my days at Sun. TL;DR, basically, the lack of ability to tag strings in the system call API with codesets means that UTF-8 is the only plausible answer, and the ends (C library system ca…

You cannot use UTF-8 locales on Windows though.

That's OK. On Unix use UTF-8. On Windows use Unicode, and let apps use UTF-8 or UTF-16 as appropriate -- the kernel/NTFS make it right.

Re: The hell that is filename encoding (2016)

#108

Post by jstimpfle »

fiedzia wrote: post #102
jstimpfle wrote: post #69

Earlier quoted context omitted.

The simple thing to go about this is to make assumptions that do not always hold technically. And to be ok when these assumptions break - in this case you simply can't deliver your promises anymore. Another way to put this is "shit in, shit out". So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file. That's what I would do if I wrote a music tagger, at least. When someone…

> So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file. The problem with invalid filenames is that doing anything at all with them might be impossible (without escaping of fixing). Your output (gui or terminal) most likely uses utf8, you can't pass invalid filename to it, so you can't even display it. Also in most situation you can't just ignore something. Imagine a text…

> Many (if not most) applications that process filenames do rely on those being valid text at least in some codepaths and they simply cannot work otherwise. The proper solution is to fail and let the user fix the problem.

Yep, that's what I meant. You need to do what is appropriate to the situation. Ignoring / warning are only two possible handling strategies. In many situations, straight-out failing is another valid one.

A file copy program should just not care and copy the darn thing. A text editor, basically the same, but maybe issue a warning.

Different tasks have different requirements, like the filename being text, the filename not containing spaces, etc. Given that these requirements are somewhat arbitrary, I think it's a fine choice to just not put any non-technical constraints in the guts.

There's an expression for it: Mechanism, not policy. You can always add policy on top, be it in the VFS layer, or as additional programs / classes of programs.

So many potential bugs would be easily fixed if the shell glob ('*') was more configurable.

Re: The hell that is filename encoding (2016)

#109

Post by beagle3 »

mehrdadn wrote: post #52
lostmsu wrote: post #50

Earlier quoted context omitted.

AFAIK, Windows understands / as a directory separator.

Only in some special cases, not in general.

On the contrary. It accepts '/' as a path separator for filenames in every API call. The special case that doesn't is command line parsing (cmd.exe and a few others)

Re: The hell that is filename encoding (2016)

#110

Post by beagle3 »

netheril96 wrote: post #29
cryptonector wrote: post #14

I wrote about this eons ago: https://cryptonector.com/2006/12/filesystem-i18n/ and https://cryptonector.com/2010/04/on-unicode-normalization-or... -- these might still be available on https://blogs.oracle.com/ , though these are from my days at Sun. TL;DR, basically, the lack of ability to tag strings in the system call API with codesets means that UTF-8 is the only plausible answer, and the ends (C library system ca…

You cannot use UTF-8 locales on Windows though.

    chcp 65001
and bob's your uncle.
Post reply on HN