The hell that is filename encoding (2016) • Hacker News Board

Re: The hell that is filename encoding (2016)

#111

Post by dataflow » Fri, May 04, 2018, 8:16 PM UTC

beagle3 wrote: post #109

mehrdadn wrote: post #52

Earlier quoted context omitted.

Only in some special cases, not in general.

On the contrary. It accepts '/' as a path separator for filenames in every API call. The special case that doesn't is command line parsing (cmd.exe and a few others)

> On the contrary. It accepts '/' as a path separator for filenames in every API call. The special case that doesn't is command line parsing (cmd.exe and a few others)

No...

  assert( PathIsRoot(TEXT("C:\\")));
  assert(!PathIsRoot(TEXT("C:/" )));

Also, I believe you meant directory separator, not path separator.

Please don't be so tempted to take an antagonistic position and confidently declare other people wrong when you cannot possibly support your position in full... I see this very commonly on HN and I cannot tell you how extremely frustrating it is for those trying to help. It sucks away all the energy and enthusiasm we have for trying to help people get accurate information (meaning we might not even have the energy to bother to respond), and on top of that, you risk disseminating incorrect information. In this case, you simply could not have tried all the wide variety of Windows APIs, so at the very least, maybe say "in my experience" if something is only based on your experience.

Re: The hell that is filename encoding (2016)

#112

Post by shadytrees » Fri, May 04, 2018, 8:39 PM UTC

And a slow black snake slithers out of your computer's USB ports, made of billowed smoke and unrealized dreams. It sticks its tongue out; "typesssss," it whispers, "typessssss."

Re: The hell that is filename encoding (2016)

#113

Post by freeone3000 » Fri, May 04, 2018, 9:27 PM UTC

leeter wrote: post #92

freeone3000 wrote: post #90

Earlier quoted context omitted.

Precisely. Programs that internally use ShiftJIS, for instance, would stop functioning on UTF-16 enforced compatability or normalization. They're currently "broken" (as in, operating incorrectly) but in a way that works.

Can you explain why this would be the case? In theory this shouldn't be an issue because any ShiftJIS conversion to Unicode should be reversible.

The apps I'm thinking of write bytes, not strings. If you enforced UTF-16 compatibility, they'd have to say what encoding they're using (they don't) or convert it themselves (they don't) - and changing either of these would require at least an application recompile.

The reason they currently work is because bytes out == bytes in, so they can read the files they create, despite what mojibake the user sees.

Re: The hell that is filename encoding (2016)

#114

Post by tinus_hn » Sat, May 05, 2018, 12:56 PM UTC

wongarsu wrote: post #58

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

What is sane path handling? Alone on windows, the maximum path length is 260 characters, except when you use extended-length paths which have a 4 character prefix and a maximum length of 32,767 characters. A sane API for reading files probably converts your paths to extended-length paths. But if you do the same thing for writing files your users start calling you insane again, because most of Windows (including Windo…

You can use subst to open these paths. The better option is to not create them in the first place though.

Re: The hell that is filename encoding (2016)

#115

Post by JdeBP » Sun, May 06, 2018, 9:39 AM UTC

tialaramex wrote: post #99

lifthrasiir wrote: post #41

Earlier quoted context omitted.

The story then goes on further: every NTFS volume contains a special file named `$UpCase` that has a uppercase mapping for all possible 16-bit words, resulting in an 128 KiB table. This approach has an upside for backward and forward compatibility... unless you eventually need a case mapping for non-BMP characters or complex mapping that expands to multiple characters.

I should briefly explain why this is here: NTFS is (usually) case preserving but not case-sensitive. So the OS needs to be able to tell whether EXAMPLE.TXT and example.txt are the "same" name, which means it needs case conversion. Not everybody agrees about how this conversion should work. The most famous example is Turkish, but there are others. So there's an actual choice to make here. If Windows baked this into th…

HPFS had a similar system some years before.

* http://www.edm2.com/index.php/Inside_the_High_Performance_Fi...

Re: The hell that is filename encoding (2016)

#116

Post by JdeBP » Sun, May 06, 2018, 9:49 AM UTC

creeble wrote: post #93

nialo wrote: post #89

I work on another music file management system, my personal special hell is playlist files. An m3u playlist file is just a new-line separated list of file paths, which can be relative or absolute, and potentially encoded in whatever locale is set on the users computer. Some fun issues: * Windows and Mac filesystems are generally case-insensitive, so some users will have the file names in the playlist file in one case…

Sounds a lot like my life a couple of years ago (and intermittently since). I don't get the bug reports any more because I think customer service has learned that file name problems can be fixed by renaming the files . Not fun for the user, but a sure fix. Ya know what kind of file names work virtually everywhere? ASCII ones.

This is a mis-use of ASCII. After all, the colon, asterisk, forward slash, question mark, backward slash, and NUL characters are all in ASCII, yet they are far from things that "work virtually everywhere". And that isn't even considering the open and close square bracket and semi-colon characters which are also not anywhere near portable to the extent of "working virtually everywhere".

The kind of file names that do work "virtually everywhere" are not ASCII, but rather are those who only use characters from the POSIX Portable Filename Character Set, which at 65 characters is just over half the size of ASCII (which has 128 characters).

* http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_...

Re: The hell that is filename encoding (2016)

#117

Post by JdeBP » Sun, May 06, 2018, 9:56 AM UTC

hossbeast wrote: post #10

In which the author takes a long and winding path to what most of us already know, "paths are fundamentally bytes".

... except on the operating systems, discussed at length in this very same discussion, where paths are fundamentally 16-bit words.

Re: The hell that is filename encoding (2016)

#118

Post by JdeBP » Sun, May 06, 2018, 10:01 AM UTC

cryptonector wrote: post #17

cyberferret wrote: post #3

Just dealing with file extensions is enough of a head spin. We stopped trying to differentiate between .xls, .xlsx, *.xlst... etc. to show an Excel icon for a file uploaded to our SaaS and just went with a generic file icon in the end.

Only on Windows.

On the contrary, if one is on Windows this stuff is actually much easier to handle, as the operating system supplies out of the box a large, extensible and maintained by installed applications, table of extensions, with an indirect mapping (via an intermediate) to display icons supplied in that very same table.

The ASSOC .DOCX command shows an example of the first leg of the indirect mapping.

Re: The hell that is filename encoding (2016)

#119

Post by JdeBP » Sun, May 06, 2018, 10:07 AM UTC

jstimpfle wrote: post #69

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

The simple thing to go about this is to make assumptions that do not always hold technically. And to be ok when these assumptions break - in this case you simply can't deliver your promises anymore. Another way to put this is "shit in, shit out". So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file. That's what I would do if I wrote a music tagger, at least. When someone…

Actually, these Unix-alike conventions of filenames just being NUL-terminated sequences of bytes/16-bit words come from the same thinking that gave us the concepts of files just being length-counted sequences of bytes that it was up to applications softwares to interpret and impose structure upon.

They were reactions to the more structured access methods of the day.

Re: The hell that is filename encoding (2016)

#120

Post by JdeBP » Sun, May 06, 2018, 10:17 AM UTC

netheril96 wrote: post #29

cryptonector wrote: post #14

I wrote about this eons ago: https://cryptonector.com/2006/12/filesystem-i18n/ and https://cryptonector.com/2010/04/on-unicode-normalization-or... -- these might still be available on https://blogs.oracle.com/ , though these are from my days at Sun. TL;DR, basically, the lack of ability to tag strings in the system call API with codesets means that UTF-8 is the only plausible answer, and the ends (C library system ca…

You cannot use UTF-8 locales on Windows though.

That's alright. On Windows, cryptonector's assertion about a lack of tagging API calls with code sets is wrong to begin with and the reasoning thus does not apply. All of the ...A() API calls are implicitly tagged with the current code page, after all.