Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Normalization Forms: When ö ≠ ö

blog.opencore.ch

Post reply on HN
Print view View original

71–80 of 144 posts

Re: Unicode Normalization Forms: When ö ≠ ö

#71

Post by DangitBobby »

jodrellblank wrote: post #14
mjevans wrote: post #8

Earlier quoted context omitted.

Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.

The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one…

> word WORD Word and woRD as the same thing

I don't know about anyone else, but I read WORD as someone yelling, Word as designating/specifying a "word" with some importance, and woRD as the mocking Spongebob meme. I absolutely don't read "case insensitive" and I don't think filesystems should either.

Re: Unicode Normalization Forms: When ö ≠ ö

#72

Post by cgriswald »

LAC-Tech wrote: post #67
hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Sprinkling English with foreign words is really, really common. I'm in New Zealand and people do it all the time. And even in the states, right? Don't want two different strings because someone writes an English sentence about how much they love jalapeño.

I'm not a linguist and that will probably be readily apparent. The word jalapeño leaves me wondering how distinct a boundary a language can possess or how one can sort out which language an individual word belongs to outside the context of the rest of the text or speech.

In English, jalapeño is correctly spelled with or without the eñe (and AFAIK the letter doesn't have a name in English, you have to use the Spanish name). So, there's an English word that doesn't use the letters assigned to the English alphabet. How do we place the word? Well, obviously English borrowed the word from Spanish, so it's a Spanish word. Well, no, it's only the Spanish adjectivization of Nahautl words used to name the place called Xalapa...

Re: Unicode Normalization Forms: When ö ≠ ö

#73

Post by josefx »

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

> being able to mix arbitrary languages into a single String object

Unless I missed something that is impossible with Unicode. Mixing multiple languages would require a way to specify the language used for case conversion, sorting and font rendering settings mid string and I don't think that Unicode has that. For example try to write a program that correctly uppercase a single string containing both an English i and a Turkish i in your favorite Unicode supporting language, the code point is the same for both, and you generally only get to specify one language globally or per function call.

Re: Unicode Normalization Forms: When ö ≠ ö

#74

Post by yongjik »

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Reminds me of the good old days with EUC-KR, KSC 5601, and all those different encoding schemes I've successfully repressed in my memory for years. Yes, you could probably assert that a piece of string was either Korean or English but never anything else... because the system was incapable of representing it.

I'm not exactly sure how a code page is supposed to help us here. Developers have trouble supporting multiple languages when they're all in the Unicode Standard. Supporting code pages for languages they've never heard of? Not a chance.

Re: Unicode Normalization Forms: When ö ≠ ö

#75

Post by aliceryhl »

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

I mix languages all the time.

Re: Unicode Normalization Forms: When ö ≠ ö

#76

Post by wongarsu »

cgriswald wrote: post #72
LAC-Tech wrote: post #67

Earlier quoted context omitted.

Sprinkling English with foreign words is really, really common. I'm in New Zealand and people do it all the time. And even in the states, right? Don't want two different strings because someone writes an English sentence about how much they love jalapeño.

I'm not a linguist and that will probably be readily apparent. The word jalapeño leaves me wondering how distinct a boundary a language can possess or how one can sort out which language an individual word belongs to outside the context of the rest of the text or speech. In English, jalapeño is correctly spelled with or without the eñe (and AFAIK the letter doesn't have a name in English, you have to use the Spanish…

Words like angst or ersatz are English words borrowed from German. The German words are written identically (except capitalisation), but the meaning of the English word is much more specific than the German "original". Meanwhile the word "Blitz" has completely district meanings in English and German. In English it's a sudden concerted effort, in German it's lightning. Despite the English word originating from German, they don't share a meaning at all

Re: Unicode Normalization Forms: When ö ≠ ö

#77

Post by jodrellblank »

DangitBobby wrote: post #71
jodrellblank wrote: post #14

Earlier quoted context omitted.

The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one…

> word WORD Word and woRD as the same thing I don't know about anyone else, but I read WORD as someone yelling, Word as designating/specifying a "word" with some importance, and woRD as the mocking Spongebob meme. I absolutely don't read "case insensitive" and I don't think filesystems should either.

You read DOG as someone yelling ‘dog’, not as a different word to ‘dog’. And Dog as a significant dog, not a significant something else.

Imagine if you could only search for ‘dog’ if you had to specify whether the author yelled it or not before you could find it.

Re: Unicode Normalization Forms: When ö ≠ ö

#78

Post by nine_k »

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Where I work and communicate, mixing 2, 3, and sometimes 4 writing systems is pretty normal; I have 3 keyboard layouts on my phone (Latin that covers English and occasional Spanish, Cyrillic, and Japanese).

In any case, there are emoji which are expected to be a part of text.

On one hand, it would be great to separate areas of different encodings inside a string. But character codes are already such separators.

Two things need to go though: the assumption of linear-time index-based access to characters in a string, and the custom to compare strings as byte arrays.

The first us already gone from several advanced string implementations. The second is harder: e.g. Linux filesystems support Unicode by being encoding-agnostic and handling names as byte sequences. Reworking that would be hard if practical at all.

Re: Unicode Normalization Forms: When ö ≠ ö

#79

Post by wongarsu »

josefx wrote: post #73
hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

> being able to mix arbitrary languages into a single String object Unless I missed something that is impossible with Unicode. Mixing multiple languages would require a way to specify the language used for case conversion, sorting and font rendering settings mid string and I don't think that Unicode has that. For example try to write a program that correctly uppercase a single string containing both an English i and…

You can write a string with words from multiple languages, you just can't easily modify it with operations like case conversion. But sorting shouldn't depend on the origin language anyway, it depends on the language of the reader. All words in an English dictionary are sorted in "English" order

Re: Unicode Normalization Forms: When ö ≠ ö

#80

Post by jodrellblank »

rob_c wrote: post #25
jodrellblank wrote: post #14

Earlier quoted context omitted.

The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one…

WORD, Word WoRD.... Sorry to say I tend to use case sensitivity as a filter for me offering support to other developers. I'm not willing to find time for people who can't get their head around "turn on/off caps lock". You don't do it in professional writeups or applications (and I hope not in a CV) so don't pollute my filesystems or codebases with that madness.

I’m not talking about caps lock. I can get my head around case sensitivity, I can use it, it’s worse, I don’t want to have to use it anymore than I want to use filesystem permissions in octal even though I can. Having tools take chmod u+r is easier and doesn’t change the filesystem at all.
Post reply on HN