Unicode Normalization Forms: When ö ≠ ö
blog.opencore.ch
Unicode Normalization Forms: When ö ≠ ö
1–10 of 144 posts
Re: Unicode Normalization Forms: When ö ≠ ö
#2> But here, normalization caused this issue.
Nope, the lack of normalization on both accounts by the SMB server caused the issue. It could have normalized before emitting but it definitely should have normalized on receiving for comparison.
Re: Unicode Normalization Forms: When ö ≠ ö
#3Why isn’t the answer just “Don’t unicode normalise the file name”?
I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.
Re: Unicode Normalization Forms: When ö ≠ ö
#4Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.
The most common desktop file systems are case-insensitive, which complicates the picture.
Re: Unicode Normalization Forms: When ö ≠ ö
#5> But here, normalization caused this issue. Nope, the lack of normalization on both accounts by the SMB server caused the issue. It could have normalized before emitting but it definitely should have normalized on receiving for comparison.
At least it should perform validation and reject the NFD form and force the client to normalize to NFC?
Re: Unicode Normalization Forms: When ö ≠ ö
#6Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.
The most common desktop file systems are case-insensitive, which complicates the picture.
Still, it looks like the right thing to do is let the filesystem do the filesystem's job. The filesystem should be normalizing unicode and enforceing the case-insensitivity and whatnot, but just the filesystem. Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes.
Re: Unicode Normalization Forms: When ö ≠ ö
#7Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.
That works for programmers, but not for users. There could be several files with the same name, buth with different encodings. Worse, depending on how your terminal encodes user input, some of them migth not be typable.
Re: Unicode Normalization Forms: When ö ≠ ö
#8Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.
The most common desktop file systems are case-insensitive, which complicates the picture.
Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.
Re: Unicode Normalization Forms: When ö ≠ ö
#9Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.
In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters.
Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you could think of as UTF-16 code units but they don't promise to encode UTF-16.
Probably the files have human readable names, but, maybe not. If you're accepting command line file names it's not crazy to insist on human readable (thus, Unicode) names, but if you process arbitrary input files you didn't create, particularly files you just found by looking around on disks unsupervised - you need to accept that utter gibberish is inevitable sooner or later and you must cope with that successfully.
Rust's OSStr variants match this reality.
Re: Unicode Normalization Forms: When ö ≠ ö
#10Earlier quoted context omitted.
The most common desktop file systems are case-insensitive, which complicates the picture.
Still, it looks like the right thing to do is let the filesystem do the filesystem's job. The filesystem should be normalizing unicode and enforceing the case-insensitivity and whatnot, but just the filesystem. Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes.
I'm not sure this problem even has a "right" solution.
> Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes.
What do you do when the input isn't a dumb pile of bytes, but actual text? (Like from a text box the user typed into?)