Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hell that is filename encoding (2016)

beets.io

Post reply on HN
Print view View original

11–20 of 121 posts

Re: The hell that is filename encoding (2016)

#11

Post by zaptheimpaler »

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites with UX from the 90s.

Has anyone tried this? Is it possible with FUSE? I would love to hear from people who know about this stuff - what are the obstacles? Or do you think FSs are fine the way they are?

Re: The hell that is filename encoding (2016)

#12

Post by deathanatos »

hossbeast wrote: post #10

In which the author takes a long and winding path to what most of us already know, "paths are fundamentally bytes".

… but also, sort-of, but not really, text: they get displayed to the user, they get input from the user, and they get emitted in logs, messages, etc. All as text. And that rub between where they're bytes but they should have been text, that's the problem and that's the complexity.

Re: The hell that is filename encoding (2016)

#13

Post by loeg »

netheril96 wrote: post #5

It misses a even more complex, I'd say insane, encoding problem: on HFS+ (or even APFS now?) filenames are unicode normalized.

Not only are they normalized unicode, they're normalized decomposed, and not only that, but slightly non-standard (does not conform to standard Unicode "NFD" form). (Or at least, this was the case with HFS. I haven't followed APFS closely enough to say for that.)

Re: The hell that is filename encoding (2016)

#14

Post by cryptonector »

I wrote about this eons ago: https://cryptonector.com/2006/12/filesystem-i18n/ and https://cryptonector.com/2010/04/on-unicode-normalization-or... -- these might still be available on https://blogs.oracle.com/, though these are from my days at Sun.

TL;DR, basically, the lack of ability to tag strings in the system call API with codesets means that UTF-8 is the only plausible answer, and the ends (C library system call stubs, filesystems) have to apply whatever codeset conversions. But there's practically zero chance of C library system call stubs (and related functions) performing codeset conversions (can you imagine readdir(3) doing it?), which means that the only reasonable answer is to use UTF-8 locales and be done.

Even shorter: just use UTF-8 locales and be done.

Re: The hell that is filename encoding (2016)

#15

Post by cryptonector »

netheril96 wrote: post #5

It misses a even more complex, I'd say insane, encoding problem: on HFS+ (or even APFS now?) filenames are unicode normalized.

HFS+'s use of NFD made even more insane by the fact that OS X's input modes prefer to produce NFC anyways. And besides, so do other OSes' input modes, so in any heterogeneous system this is a nightmare.

This is why ZFS does form-insensitive directory lookups (and hashing)[0] rather than normalize-on-CREATE! I'm so glad ZFS got it right, and can stand as a model for all. (I implemented none of that functionality, though I code-reviewed some of it, specifically the u8_* functions in Solaris/Illumos, but I remember it took some doing to convince others that this was the correct approach.)

[0] https://cryptonector.com/2006/12/filesystem-i18n/ [1] https://cryptonector.com/2010/04/on-unicode-normalization-or...

Re: The hell that is filename encoding (2016)

#16

Post by cryptonector »

loeg wrote: post #13
netheril96 wrote: post #5

It misses a even more complex, I'd say insane, encoding problem: on HFS+ (or even APFS now?) filenames are unicode normalized.

Not only are they normalized unicode, they're normalized decomposed , and not only that, but slightly non-standard (does not conform to standard Unicode "NFD" form). (Or at least, this was the case with HFS. I haven't followed APFS closely enough to say for that.)

NFD hadn't been standardized at the time.

IIUC the reason they did this is that they wanted directories to be canonically ordered on disk, and they thought decomposition would naturally yield better results than pre-composition. I'm not sure that's right, and frankly I don't care either, because the most important thing to note is that input methods (especially for European languages) by and large produce NFC, and most application software does no normalization at all, so disagreements as to form cause problems[0][1].

[0] https://cryptonector.com/2010/04/on-unicode-normalization-or... [1] https://cryptonector.com/2006/12/filesystem-i18n/

Re: The hell that is filename encoding (2016)

#17

Post by cryptonector »

cyberferret wrote: post #3

Just dealing with file extensions is enough of a head spin. We stopped trying to differentiate between .xls, .xlsx, *.xlst... etc. to show an Excel icon for a file uploaded to our SaaS and just went with a generic file icon in the end.

Only on Windows.

Re: The hell that is filename encoding (2016)

#18

Post by chungy »

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

ZFS has a `utf8only` property that constrains filenames to be UTF-8 only... :)

Re: The hell that is filename encoding (2016)

#19

Post by gear54rus »

hackits wrote: post #8
gear54rus wrote: post #6

Earlier quoted context omitted.

Honest question, why is that feature complex? What is the problem in looking at last part after dot?

Because Microsoft made their new office extension .xml if that doesn't make your head spin i don't know what else will.

I mean, this is only adding another entry to your array of extensions for the specific icon. It's also not a deal-braker if it doesn't work. It really is easy, I don't see what's the problem.

Re: The hell that is filename encoding (2016)

#20

Post by ManuelKiessling »

zaptheimpaler wrote: post #11

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites…

sqlite?
Post reply on HN