"If you vote for my nutburger glyph my kid drew for a kindergarten assignment, I'll vote for the chicken scratching you noticed in the barnyard dust."
Unicode Normalization Forms: When ö ≠ ö
91–100 of 144 posts
Re: Unicode Normalization Forms: When ö ≠ ö
#92Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…
Sprinkling English with foreign words is really, really common. I'm in New Zealand and people do it all the time. And even in the states, right? Don't want two different strings because someone writes an English sentence about how much they love jalapeño.
Re: Unicode Normalization Forms: When ö ≠ ö
#93Earlier quoted context omitted.
I'm not a linguist and that will probably be readily apparent. The word jalapeño leaves me wondering how distinct a boundary a language can possess or how one can sort out which language an individual word belongs to outside the context of the rest of the text or speech. In English, jalapeño is correctly spelled with or without the eñe (and AFAIK the letter doesn't have a name in English, you have to use the Spanish…
Words like angst or ersatz are English words borrowed from German. The German words are written identically (except capitalisation), but the meaning of the English word is much more specific than the German "original". Meanwhile the word "Blitz" has completely district meanings in English and German. In English it's a sudden concerted effort, in German it's lightning. Despite the English word originating from German,…
Re: Unicode Normalization Forms: When ö ≠ ö
#94Earlier quoted context omitted.
In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…
Funnily base64 suffers from a related issue that the likes of base58 correct : l and I or O and 0 looking similar or even identical depending on the font !
Re: Unicode Normalization Forms: When ö ≠ ö
#95Earlier quoted context omitted.
> So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? Case (in)sensitivity for filenames is a non-issue in my experience. Never had problems with either convention. As for emails, I do think insensitivity was the right choice.
The RFC states that email addresses are case sensitive. The local-part of a mailbox MUST BE treated as case sensitive. Section 2.4 RFC 2821, https://www.ietf.org/rfc/rfc2821.txt
Re: Unicode Normalization Forms: When ö ≠ ö
#96Earlier quoted context omitted.
What's "crazy" about the letter? It's a standard letter of several European alphabets.
Nothing crazy about the "letter", but it is crazy that there are multiple different ways to encode the "letter".
Re: Unicode Normalization Forms: When ö ≠ ö
#97Earlier quoted context omitted.
In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…
We had to ETL .csv data that must have originated in SQLServer. The utf-16 fact about Windows was apparently unknown to my predecessor. Who wrote some nasty c-language binary to copy the data, knock the upper byte off of each character ahead, and save the now ASCII text to a new file of the mysql load. The encoding='utf-16' argument was all that was needed. For want of a nail. . .
The "maybe strip null bytes" code lived for years "just in case" after I fixed that because people couldn't believe that's all that was ever "wrong" with the data.
Re: Unicode Normalization Forms: When ö ≠ ö
#98Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…
If a programming language allows text processing but can't even properly compare unicode text, it is buggy and needs to be fixed. If an operating system can't deal with unicode, it's buggy and needs to be fixed.
Re: Unicode Normalization Forms: When ö ≠ ö
#99Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…
In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…
Re: Unicode Normalization Forms: When ö ≠ ö
#100Earlier quoted context omitted.
Normalisation is expressly done with the composition of version 3.1 for compatibility: see https://www.unicode.org/reports/tr15/#Versioning >. IF that’s what HFS+ does, then “proprietary variant” is wrong. And if not, I’m curious what it does differently. (On the use of version 3.1, note that in practice version 3.2 is used, correcting one typo: see https://www.unicode.org/versions/corrigendum3.html >.) I find a few…
The `filename-sanitizer` library you have linked has the following comment. # FIXME: improve HFS+ handling, because it does not use the standard NFD. It's # close, but it's not exactly the same thing. 'hfs+': (255, 'characters', 'utf-16', 'NFD'), I wonder what does that mean...
> The characters with codes in the range u+2000 through u+2FFF are punctuation, symbols, dingbats, arrows, box drawing, etc. The u+24xx block, for example, has single characters for things like "(a)". The characters in this range are not fully decomposed; they are left unchanged in HFS Plus strings. This allows strings in Mac OS encodings to be converted to Unicode and back without loss of information. This is not unnatural since a user would not necessarily expect a dingbat "(a)" to be equivalent to the three character sequence "(", "a", ")" in a file name.
> The characters in the range u+F900 through u+FAFF are CJK compatibility ideographs, and are not decomposed in HFS Plus strings.
The bit about the u+24xx block is misleading, the decomposition of the characters I looked at there (such as ⒜) are compatibility canonicalizations. However the CJK compatibility ideographs is a working example. U+F902 (車) decomposes to U+8ECA (車) regardless of normalization form but the technote says these must not be decomposed.