Unicode Normalization Forms: When ö ≠ ö • Hacker News Board

Re: Unicode Normalization Forms: When ö ≠ ö

#21

Post by stefan_ » Fri, Dec 31, 2021, 9:13 PM UTC

misnome wrote: post #3

Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.

Sure but at some point you might want to create a file and frequently using user input or filter files using some user provided query string, the kind of use cases that unicode normalization was invented for. So the whole "opaque blob of bytes" filesystem handling is nice if all you want is to not silently corrupt files, but it is very obviously not even covering 10% of normal use cases. Rust isn't being super smart, it just has its hands thrown up in the air.

Re: Unicode Normalization Forms: When ö ≠ ö

#22

Post by 1over137 » Fri, Dec 31, 2021, 9:14 PM UTC

GlitchMr wrote: post #13

tialaramex wrote: post #9

Earlier quoted context omitted.

In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters . Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you…

Filenames in HFS+ filesystem (an old filesystem used by Mac OS X) are normalized with a proprietary variant of NFD - this is a filesystem feature. APFS removed this feature.

>APFS removed this feature.

And then brought it back. It normalizes now.

Re: Unicode Normalization Forms: When ö ≠ ö

#23

Post by nixpulvis » Fri, Dec 31, 2021, 9:15 PM UTC

Half normal isn't normal. That said, I personally try to avoid unicode in filenames (and caps too) for similar reasons.

Re: Unicode Normalization Forms: When ö ≠ ö

#24

Post by lilyball » Fri, Dec 31, 2021, 9:16 PM UTC

GlitchMr wrote: post #13

tialaramex wrote: post #9

Earlier quoted context omitted.

In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters . Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you…

Filenames in HFS+ filesystem (an old filesystem used by Mac OS X) are normalized with a proprietary variant of NFD - this is a filesystem feature. APFS removed this feature.

By “proprietary variant” you mean “publicly documented variant” which IIRC is just the normalization tables frozen in time from an early version of Unicode (the idea being that updating your OS shouldn’t change the rules about what filenames are valid).

As for APFS, it ~~doesn’t~~didn’t normalize but I believe it still requires UTF-8. And the OS will normalize filenames at a higher level. EDIT: they added native normalization. At least for iOS, I didn’t dig enough to check it macOS is doing native normalizing or is just normalization-insensitive.

Re: Unicode Normalization Forms: When ö ≠ ö

#25

Post by rob_c » Fri, Dec 31, 2021, 9:17 PM UTC

jodrellblank wrote: post #14

mjevans wrote: post #8

Earlier quoted context omitted.

Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.

The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one…

WORD, Word WoRD....

Sorry to say I tend to use case sensitivity as a filter for me offering support to other developers. I'm not willing to find time for people who can't get their head around "turn on/off caps lock". You don't do it in professional writeups or applications (and I hope not in a CV) so don't pollute my filesystems or codebases with that madness.

Re: Unicode Normalization Forms: When ö ≠ ö

#26

Post by im3w1l » Fri, Dec 31, 2021, 9:17 PM UTC

laurent92 wrote: post #12

mjevans wrote: post #8

Earlier quoted context omitted.

Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.

So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? And John.McCauley@yahoo.fr and john.mccauley@yahoo.fr being attributed to two different people ;)

> So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories?

Case (in)sensitivity for filenames is a non-issue in my experience. Never had problems with either convention. As for emails, I do think insensitivity was the right choice.

Re: Unicode Normalization Forms: When ö ≠ ö

#27

Post by rob_c » Fri, Dec 31, 2021, 9:19 PM UTC

rzzzt wrote: post #15

dataflow wrote: post #10

Earlier quoted context omitted.

I'm not sure this problem even has a "right" solution. > Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes. What do you do when the input isn't a dumb pile of bytes, but actual text? (Like from a text box the user typed into?)

Maintain a table that maps the original file name to random-generated one that doesn't hit these gotchas.

And place the files in chunks, and... Wait I think we're getting close to reinventing block storage again ;)

Re: Unicode Normalization Forms: When ö ≠ ö

#28

Post by dataflow » Fri, Dec 31, 2021, 9:24 PM UTC

rzzzt wrote: post #15

dataflow wrote: post #10

Earlier quoted context omitted.

I'm not sure this problem even has a "right" solution. > Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes. What do you do when the input isn't a dumb pile of bytes, but actual text? (Like from a text box the user typed into?)

Maintain a table that maps the original file name to random-generated one that doesn't hit these gotchas.

I'm afraid I don't follow. Who maintains this table and who consumes it? What if they're different entities? How do you prevent it from going out of sync with the file system when the user renames a file? Are you inventing your own file system here? How do you deal with existing file systems?

Re: Unicode Normalization Forms: When ö ≠ ö

#29

Post by B-Con » Fri, Dec 31, 2021, 9:27 PM UTC

guerrilla wrote: post #2

> But here, normalization caused this issue. Nope, the lack of normalization on both accounts by the SMB server caused the issue. It could have normalized before emitting but it definitely should have normalized on receiving for comparison.

I think that in the ls->read workflow, Nextcloud shouldn't normalize the response from SMB and should issue back to SMB whatever SMB returned to Nextcloud.

Re: Unicode Normalization Forms: When ö ≠ ö

#30

Post by 0x0 » Fri, Dec 31, 2021, 9:32 PM UTC

Java is terrible in this regard, as most file APIs use "java.lang.String" to identify the filename, which most of the time depends on the system property "file.encoding". With the result that there will be files that you can never read from a java application if the filename encoding does not match the java file.encoding encoding.