Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Normalization Forms: When ö ≠ ö

blog.opencore.ch

Post reply on HN
Print view View original

11–20 of 144 posts

Re: Unicode Normalization Forms: When ö ≠ ö

#11

Post by atoav »

tialaramex wrote: post #9
misnome wrote: post #3

Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.

In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters . Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you…

This is what I found quite refreshing about Rust — instead of choosing one of the following:

  A) The programmer is a almighty god who knows everything, we just expose him to the raw thing
  
  B) The programmer is a immature toddler who cannot be trusted, so we handle things for them
What Rust does is more among the lines of "you might already know this, but anyways here is a reminder that you, the programmer need to take some decision about this".

Re: Unicode Normalization Forms: When ö ≠ ö

#12

Post by laurent92 »

mjevans wrote: post #8
pavlov wrote: post #4

Earlier quoted context omitted.

The most common desktop file systems are case-insensitive, which complicates the picture.

Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.

So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? And John.McCauley@yahoo.fr and john.mccauley@yahoo.fr being attributed to two different people ;)

Re: Unicode Normalization Forms: When ö ≠ ö

#13

Post by GlitchMr »

tialaramex wrote: post #9
misnome wrote: post #3

Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.

In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters . Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you…

Filenames in HFS+ filesystem (an old filesystem used by Mac OS X) are normalized with a proprietary variant of NFD - this is a filesystem feature. APFS removed this feature.

Re: Unicode Normalization Forms: When ö ≠ ö

#14

Post by jodrellblank »

mjevans wrote: post #8
pavlov wrote: post #4

Earlier quoted context omitted.

The most common desktop file systems are case-insensitive, which complicates the picture.

Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.

The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one in water-based ink, or whatever compares with behind the scenes implementation details like combining form vs single character.

I have just been arguing the same thing in far too much detail in this thread: https://news.ycombinator.com/item?id=29722019

Re: Unicode Normalization Forms: When ö ≠ ö

#15

Post by rzzzt »

dataflow wrote: post #10
Pxtl wrote: post #6

Earlier quoted context omitted.

Still, it looks like the right thing to do is let the filesystem do the filesystem's job. The filesystem should be normalizing unicode and enforceing the case-insensitivity and whatnot, but just the filesystem. Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes.

I'm not sure this problem even has a "right" solution. > Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes. What do you do when the input isn't a dumb pile of bytes, but actual text? (Like from a text box the user typed into?)

Maintain a table that maps the original file name to random-generated one that doesn't hit these gotchas.

Re: Unicode Normalization Forms: When ö ≠ ö

#16

Post by zarzavat »

arka2147483647 wrote: post #7
misnome wrote: post #3

Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.

That works for programmers, but not for users. There could be several files with the same name, buth with different encodings. Worse, depending on how your terminal encodes user input, some of them migth not be typable.

From the users perspective I don't want any normalisation at all. It's good as long as you only have one file system but as soon as you get multiple file systems with conflicting rules (which includes transferring files to other people) it becomes hell. Unfortunately we are stuck with that hell.

Re: Unicode Normalization Forms: When ö ≠ ö

#17

Post by justaguy37 »

laurent92 wrote: post #12
mjevans wrote: post #8

Earlier quoted context omitted.

Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.

So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? And John.McCauley@yahoo.fr and john.mccauley@yahoo.fr being attributed to two different people ;)

can we just say no to capital letters? (or lowercase?)

do capital letters have a good enough usage case to justify their continued existence?

Re: Unicode Normalization Forms: When ö ≠ ö

#18

Post by colejohnson66 »

justaguy37 wrote: post #17
laurent92 wrote: post #12

Earlier quoted context omitted.

So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? And John.McCauley@yahoo.fr and john.mccauley@yahoo.fr being attributed to two different people ;)

can we just say no to capital letters? (or lowercase?) do capital letters have a good enough usage case to justify their continued existence?

You are free to stop using capital letters, but good luck getting everyone to go along. Capitals have been around for centuries (they’re older than the printing press) and aren’t going anywhere.

Re: Unicode Normalization Forms: When ö ≠ ö

#19

Post by skymt »

jodrellblank wrote: post #14
mjevans wrote: post #8

Earlier quoted context omitted.

Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.

The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one…

There are a couple arguments against case-insensitive filesystems I think are strong. The first is simply compatibility with existing case-sensitive systems. The second is that case is locale-dependent, so a pair of names could be equivalent or not depending on the device's locale.

I don't think I've seen any good argument against normalization, though.

Re: Unicode Normalization Forms: When ö ≠ ö

#20

Post by vgel »

laurent92 wrote: post #12
mjevans wrote: post #8

Earlier quoted context omitted.

Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.

So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? And John.McCauley@yahoo.fr and john.mccauley@yahoo.fr being attributed to two different people ;)

First one: yes, though good UI should prevent it from happening unless the user really intended it (for example I have ~/Documents symlinked into Dropbox, so ~/documents could be local-only documents)

Second one: no, emails are not filenames, and more generally distinguishability is more important for identifiers. In cases where identifiers like emails need to be mapped to filenames, like caches, they should be normalized.

Post reply on HN