Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hell that is filename encoding (2016)

beets.io

Post reply on HN
Print view View original

61–70 of 121 posts

Re: The hell that is filename encoding (2016)

#61

Post by majewsky »

ktpsns wrote: post #45
zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

The uncomfortable generic answer to this: https://xkcd.com/927/

It even lists "character encodings" in the title as an example.

Re: The hell that is filename encoding (2016)

#62

Post by tom_ »

creeble wrote: post #7
lifthrasiir wrote: post #2

Rust `std::path` [1] has two representations under the hood for Windows (UTF-16 plus lone surrogates) and non-Windows (bytes) for the exactly same reason. Paths are not strings nor texts. [1] https://doc.rust-lang.org/stable/std/path/

They're not strings nor texts right up until the point you need to display them to users.

Same applies to many things, but that doesn’t make those things strings. Numbers might be another example.

Re: The hell that is filename encoding (2016)

#64

Post by masklinn »

burfog wrote: post #47
chungy wrote: post #34

Earlier quoted context omitted.

> Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16). NTFS (and Windows as a whole) does not use UTF-16, it uses UCS-2. It is a subtle difference, but surrogate pairs didn't exist in UCS-2.

Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated. The kernel generally uses the 16-bit equivalent of the old Pascal string, that being a 16-bit count of 16-bit pieces of UTF-16 data. This allows a 16-bit NUL to get into various places that make the Win32 API choke.

> Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated.

I usually call that ucs2-plus-surrogates, to make it clear that you may encounter unpaired surrogates, and thus invalid paths if you assume proper UTF-16.

Re: The hell that is filename encoding (2016)

#65

Post by majewsky »

cyberferret wrote: post #3

Just dealing with file extensions is enough of a head spin. We stopped trying to differentiate between .xls, .xlsx, *.xlst... etc. to show an Excel icon for a file uploaded to our SaaS and just went with a generic file icon in the end.

If this is a head spin, then go take a stroll on a Debian package mirror for what kinds of awesome ^W bizarre file extension combos they've come up with. This one is my favorite: https://twitter.com/stefanmajewsky/status/928983817825665025

Re: The hell that is filename encoding (2016)

#66

Post by Piskvorrr »

tcoff91 wrote: post #32
cryptonector wrote: post #25

Earlier quoted context omitted.

Doesn't it (or Windows) also disallow the path component separator character(s) ('/' and '\')? Unix and alike disallow NULs and /, for obvious reasons.

There are a number of characters like path separators that cannot be part of a file name on windows. However I am not sure if this is enforced by the OS APIs or by NTFS itself. It is entirely possible that NTFS could allow something that higher layers don’t.

You could break NTFS into accepting this. Fun things happen, for a specific definition of "fun".

Re: The hell that is filename encoding (2016)

#67

Post by brazzy »

Hermel wrote: post #40
zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

I think Java does that. Thanks to its built-in libraries, you can write code that works on multiplie platforms with different file systems. However, you still need to know yourself which characters are legal in which system (for example, ‘:’ is not a legal character in file names on windows, but it is on OS X.

At least since Java 7, the API has a concept of a filesystem which can create filesystem-specific Path objects and, if it encounters illegal characters, throws an InvalidPathException that tells you which character was illegal.

Re: The hell that is filename encoding (2016)

#68

Post by sametmax »

quietbritishjim wrote: post #31

> on Windows, paths are fundamentally text They were back when there were less than 2^16 characters in the Unicode standard. Back then each two-byte word in a filename corresponded exactly with a Unicode code point. Now there are more than 2^16 but well under 2^32, Windows uses UTF-16 in filenames. That is, Unicode code points above 2^15 are obtained by a pair of special Unicode code points in the range 2^15-2^16 cal…

Funnily, because of the 2/3 transition, Python became a language obsessed with encoding correctness. Hence the team spent a great deal improving the situation, from version to version during the last 10 years.

For a fantastic read (from Victor Stinner himself) on all the work done and get how twisted this get when you want to be cross plateform and abstract it:

https://vstinner.github.io/python37-new-utf8-mode.html

Windows and Linux FS encoding, of course, are at the center of the challenge.

It's also the reason why we now have a __fspath__ protocol that allows any object to be converted to a file system path instead of having pathlib.Path inheriting from string.

Re: The hell that is filename encoding (2016)

#69

Post by jstimpfle »

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

The simple thing to go about this is to make assumptions that do not always hold technically. And to be ok when these assumptions break - in this case you simply can't deliver your promises anymore.

Another way to put this is "shit in, shit out".

So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file. That's what I would do if I wrote a music tagger, at least.

When someone else modifies a directory tree at the same time, we're bound to run into problems. That's just how it is. Actually there are synchronization facilities (e.g. flock()) for some of these types of problems. But these are seldom used because they lead to other problems.

I think they knew all that in the 70s, and just chose to be pragmatic about it.

Re: The hell that is filename encoding (2016)

#70

Post by SSLy »

oblio wrote: post #51
mattbierner wrote: post #49

The built-in file system libraries for many languages are total footguns. Using strings as paths is a great example. I’ve had a few recent bugs around case sensitive vs case insensitive file systems because a lot of code assumes that when pathA != pathB then it must be dealing with two different resources. Not to mention the classic “doesn’t work on windows” problem: newPath = pathA + “/“ + pathB

Most decent stdlibs have a path.join or an path.separator you can use instead. Just that people don't bother and hardcode...

IIRC java's stdlib converts `/` to the mentioned separator
Post reply on HN