Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hell that is filename encoding (2016)

beets.io

Post reply on HN
Print view View original

51–60 of 121 posts

Re: The hell that is filename encoding (2016)

#51

Post by oblio »

mattbierner wrote: post #49

The built-in file system libraries for many languages are total footguns. Using strings as paths is a great example. I’ve had a few recent bugs around case sensitive vs case insensitive file systems because a lot of code assumes that when pathA != pathB then it must be dealing with two different resources. Not to mention the classic “doesn’t work on windows” problem: newPath = pathA + “/“ + pathB

Most decent stdlibs have a path.join or an path.separator you can use instead. Just that people don't bother and hardcode...

Re: The hell that is filename encoding (2016)

#52

Post by mehrdadn »

lostmsu wrote: post #50
mattbierner wrote: post #49

The built-in file system libraries for many languages are total footguns. Using strings as paths is a great example. I’ve had a few recent bugs around case sensitive vs case insensitive file systems because a lot of code assumes that when pathA != pathB then it must be dealing with two different resources. Not to mention the classic “doesn’t work on windows” problem: newPath = pathA + “/“ + pathB

AFAIK, Windows understands / as a directory separator.

Only in some special cases, not in general.

Re: The hell that is filename encoding (2016)

#53

Post by lostmsu »

mehrdadn wrote: post #42
mook wrote: post #38

Earlier quoted context omitted.

Isn't that why openat() exists? Of course, that isn't used nearly as much because it's annoying to have to do things that way, but it seems like the sort of thing if you need it. The "open things by paths" thing, IIRC, is part of the reason Windows doesn't like to let you delete open files by default.

Yeah, and Windows has NtOpenFile(OBJECT_ATTRIBUTES*) to let you specify a parent directory, but these are only at the syscall level, not at the standard application API level (Win32 or C APIs don't allow it). The entire model exposed to normal applications has this problem.

I though Windows won't let you move or rename a directory, if it has open files.

Re: The hell that is filename encoding (2016)

#54

Post by mehrdadn »

lostmsu wrote: post #53
mehrdadn wrote: post #42

Earlier quoted context omitted.

Yeah, and Windows has NtOpenFile(OBJECT_ATTRIBUTES*) to let you specify a parent directory, but these are only at the syscall level, not at the standard application API level (Win32 or C APIs don't allow it). The entire model exposed to normal applications has this problem.

I though Windows won't let you move or rename a directory, if it has open files.

Almost, but not quite. I just tried it to confirm: Windows lets you delete parent directories if they're reparse points. (I don't believe the deletion prevention was attempting to solve this problem in the first place, so this isn't really a bug or anything.)

Re: The hell that is filename encoding (2016)

#56

Post by dataflow »

dvlsg wrote: post #55
mehrdadn wrote: post #52

Earlier quoted context omitted.

Only in some special cases, not in general.

Really? I use forward slashes all the time in windows 10. I don't think I've run into a problem yet.

Yes. One huge problem is slashes are also used for command-line switches. That can entirely change the meaning of your commands.

For example, compare these two in the command prompt:

  start /Windows/Notepad.exe
  start \Windows\Notepad.exe
I wouldn't blame this on poor parsing or other silly things though. I actually think it makes sense to use slashes for switches, because they are invalid filename characters, whereas dashes are valid, and hence get ambiguous (hence the need for "--" in *nix). I think it would've made more sense to disallow slashes as directory separators entirely, to avoid this for good.

Re: The hell that is filename encoding (2016)

#57

Post by Someone »

loeg wrote: post #13
netheril96 wrote: post #5

It misses a even more complex, I'd say insane, encoding problem: on HFS+ (or even APFS now?) filenames are unicode normalized.

Not only are they normalized unicode, they're normalized decomposed , and not only that, but slightly non-standard (does not conform to standard Unicode "NFD" form). (Or at least, this was the case with HFS. I haven't followed APFS closely enough to say for that.)

HFS intended to store name entries in “US display order”, but it had a bug in sorting. https://developer.apple.com/legacy/library/technotes/tn/tn11...:

”HFS uses 31-byte strings to store file names. HFS does not store any kind of script information with the file name to indicate how it should be interpreted. File names are compared and sorted using a routine that assumes a Roman script, wreaking havoc for names that use some other script (such as Japanese). Worse, this algorithm is buggy, even for Roman scripts. The Finder and other applications interpret the file name based on the script system in use at runtime.”

The bug (or part of it) was that some punctuation sorted before everything else.

Re: The hell that is filename encoding (2016)

#58

Post by wongarsu »

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

What is sane path handling?

Alone on windows, the maximum path length is 260 characters, except when you use extended-length paths which have a 4 character prefix and a maximum length of 32,767 characters. A sane API for reading files probably converts your paths to extended-length paths. But if you do the same thing for writing files your users start calling you insane again, because most of Windows (including Windows Explorer) can't open extended-length paths. So you would be creating files that only select software can even open, and which the user can't browse without third party software.

(An easier to ignore fun fact is that NTSF and Windows also support case sensitive names if you set the right flags in the file APIs. But nobody uses that, so it's probably save to ignore that (until somebody mounts EXT3 partitions in windows...))

Re: The hell that is filename encoding (2016)

#59

Post by majewsky »

wongarsu wrote: post #58
zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

What is sane path handling? Alone on windows, the maximum path length is 260 characters, except when you use extended-length paths which have a 4 character prefix and a maximum length of 32,767 characters. A sane API for reading files probably converts your paths to extended-length paths. But if you do the same thing for writing files your users start calling you insane again, because most of Windows (including Windo…

> So you would be creating files that only select software can even open, and which the user can't browse without third party software.

iOS suggests that people are ok with this. /s

Re: The hell that is filename encoding (2016)

#60

Post by Valmar »

VilniusHacker wrote: post #27

Why support Windows!? We should support Linux only and make everyone else conform. Also Mac won't support more than 64 characters.

Ummm... problem is, Windows is used by a ton of people.

Break their workflow, and said peoples will hate you.

So... UTF-16 in Windows Hell, and UTF-8 everywhere else that is actually sane.

Post reply on HN