Unicode Normalization Forms: When ö ≠ ö • Hacker News Board

Re: Unicode Normalization Forms: When ö ≠ ö

#61

Post by BlueTemplar » Fri, Dec 31, 2021, 10:36 PM UTC

int_19h wrote: post #44

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

You can already map Unicode ranges to "code pages" of sorts, so how would that help? Thing is, people who are not linguists do want to mix languages. It's very common in some cultures to intersperse the native language with English. But even if not, if the language in question uses a non-Latin alphabet, there are often bits and pieces of data that have to be written down in Latin. So that "most of us" perspective is…

Yeah, Greek alphabet is used a lot in sciences. It's really annoying that we're only starting to get proper support now. (Including on keyboards : http://norme-azerty.fr/en/ )

Re: Unicode Normalization Forms: When ö ≠ ö

#62

Post by jerf » Fri, Dec 31, 2021, 10:39 PM UTC

david-gpu wrote: post #56

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

> Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us we would be better off being able to a assert that a piece of text was German, or sanskrit, not a jumble of both. Presumably the person who wrote it speaks a single language. Just because something is not useful to them, it doesn't mean it is not useful…

I think in that case the idea would either be that you should then have an array of strings, each of which may have its own language set, or that the string should be labelled as "containing Latin and Cyrillic", but still not able to include arbitrary other characters from Unicode. And multi-lingual text still generally breaks on words... Kilobytes of Latin text with a single Cyrillic character in the middle of a word is very suspicious, in a way that kilobytes of Latin text with a single Cyrillic word isn't.

Of course you'd always need an "unrestricted" string (to speak to the rest of the system if necessary), but there are very few natural strings out there in the world that consist of half-a-dozen languages just mishmashed together. Those exceptions can be treated as exceptions.

Re: Unicode Normalization Forms: When ö ≠ ö

#63

Post by wisty » Fri, Dec 31, 2021, 10:42 PM UTC

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

What's a word? (A quick test - how many words were in the previous sentence, maybe 3 or 4 depending on whether the 's is part of a word; so can we talk about Jóhannesson's foreign policy?).

It's hard enough to know what a letter is in unicode. Breaking things into words is just another massive headache.

Re: Unicode Normalization Forms: When ö ≠ ö

#64

Post by Someone » Fri, Dec 31, 2021, 10:44 PM UTC

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

That doesn’t make sense to me. Even disregarding cases where people mix languages (how do you write a dictionary? If the answer is “just create a data structure combining multiple strings”, shouldn’t we standardize how to do that?), all languages share thousands of symbols such as currency symbols, mathematical symbols, Greek and Hebrew alphabets (to be used in math books written in the language), etc. So, even languages such as Greek and English share way more symbols than that they have unique ones.

Re: Unicode Normalization Forms: When ö ≠ ö

#65

Post by deadbunny » Fri, Dec 31, 2021, 10:45 PM UTC

tim-- wrote: post #35

im3w1l wrote: post #26

Earlier quoted context omitted.

> So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? Case (in)sensitivity for filenames is a non-issue in my experience. Never had problems with either convention. As for emails, I do think insensitivity was the right choice.

The RFC states that email addresses are case sensitive. The local-part of a mailbox MUST BE treated as case sensitive. Section 2.4 RFC 2821, https://www.ietf.org/rfc/rfc2821.txt

My guess would be that the local part of an email address would usually map to a directory on case sensitive filesystems...

Re: Unicode Normalization Forms: When ö ≠ ö

#66

Post by matja » Fri, Dec 31, 2021, 10:46 PM UTC

GlitchMr wrote: post #13

tialaramex wrote: post #9

Earlier quoted context omitted.

In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters . Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you…

Filenames in HFS+ filesystem (an old filesystem used by Mac OS X) are normalized with a proprietary variant of NFD - this is a filesystem feature. APFS removed this feature.

ZFS can support normalization also:

    $ echo test > $'\xc3\xb6'
    $ cat $'\x6f\xcc\x88'
    cat: ö: No such file or directory

    $ zfs create -o normalization=formD pool/dataset
    $ echo test > $'\xc3\xb6'
    $ cat $'\x6f\xcc\x88'
    test

Re: Unicode Normalization Forms: When ö ≠ ö

#67

Post by LAC-Tech » Fri, Dec 31, 2021, 10:51 PM UTC

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Sprinkling English with foreign words is really, really common. I'm in New Zealand and people do it all the time. And even in the states, right? Don't want two different strings because someone writes an English sentence about how much they love jalapeño.

Re: Unicode Normalization Forms: When ö ≠ ö

#68

Post by chrismorgan » Fri, Dec 31, 2021, 11:03 PM UTC

A fun related issue that could occur: applying NFD to a string can make it longer, so a sanitiser that limits file names to 255 UTF-16 code units but doesn’t first normalise to NFD could fail on HFS+.

This could occur on systems that normalise to NFC as well: NFC lengthens some strings, e.g. 𝅗𝅥 (U+1D15E MUSICAL SYMBOL HALF NOTE) normalises to 𝅗𝅥 (U+1D157 MUSICAL SYMBOL VOID NOTEHEAD, U+1D165 MUSICAL SYMBOL COMBINING STEM) in both NFC and NFD (similar happens in various Indic scripts, pointed Hebrew, and the isolated case of U+2ADC FORKING which is special for reasons UAX #15 explains), but I don’t think there are any file systems that actually normalise to NFC? (APFS prefers NFC, but doesn’t normalise at the file system level.)

The remaining concern would be that NFC could take more UTF-8 code units than NFD despite adding a character, but in practice this doesn’t occur (checked on NormalizationTest-3.2.0.txt).

Re: Unicode Normalization Forms: When ö ≠ ö

#69

Post by david-gpu » Fri, Dec 31, 2021, 11:14 PM UTC

jerf wrote: post #62

david-gpu wrote: post #56

Earlier quoted context omitted.

> Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us we would be better off being able to a assert that a piece of text was German, or sanskrit, not a jumble of both. Presumably the person who wrote it speaks a single language. Just because something is not useful to them, it doesn't mean it is not useful…

I think in that case the idea would either be that you should then have an array of strings, each of which may have its own language set, or that the string should be labelled as "containing Latin and Cyrillic", but still not able to include arbitrary other characters from Unicode. And multi-lingual text still generally breaks on words ... Kilobytes of Latin text with a single Cyrillic character in the middle of a wo…

How would that look like to an end-user? Do I need to tell my browser which scripts are contained in my emails? What would happen if I start typing in a different script?

> there are very few natural strings out there in the world that consist of half-a-dozen languages just mishmashed together

...in your experience. What is an almost-absurd exception to you, is every day life to others.

Re: Unicode Normalization Forms: When ö ≠ ö

#70

Post by teej » Fri, Dec 31, 2021, 11:17 PM UTC

LAC-Tech wrote: post #67

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Sprinkling English with foreign words is really, really common. I'm in New Zealand and people do it all the time. And even in the states, right? Don't want two different strings because someone writes an English sentence about how much they love jalapeño.

It’s very common in the online messaging I’ve seen in English, Spanish, Chinese, and Russian.