Live data from Hacker News

I Can’t Write My Name in Unicode

modelviewculture.com

271–280 of 377 posts

Re: I Can’t Write My Name in Unicode

#271
post #230

Earlier quoted context omitted.

In Haskell: isLower $ toUpper 'ß' is True. I wonder how many security holes this unexpected behaviour causes.

.Net seems to do the same thing, Javascript (according to jsfiddle) as well. So maybe this is more widespread than I thought (again - I have never seen that character in the wild)? Java (as in Try Clojure) seems to do the 'expected' SS thing. Trying the golang playground I get even worse: fmt.Println(strings.ToUpper("ßẞ")) returns ßẞ (yeah, unchanged?) So, while I agree that you're technically correct (ẞ exists!) I d…

I think this is more related to the fact that there aren't many sane libraries implementing unicode and locales -- so you'll get either some c lib/c++ lib, system lib, java lib -- or an actual new implementation that's actually been done "seriously" -- as part of being able to say: "Yes, X does actually support unicode strings.".

Python3 got a lot of flac for the decision to break away from it's byte sequences, to it's a unicode string. But I think that was the right choice. I still understand why people writing software that only cared about network, on-the-wire, pretend-to-be-text type strings.

Then again, based on some other comments here, apparently there are still some dark corners:

    Python 3.2.3 (default, Feb 20 2013, 14:44:27) 
    [GCC 4.7.2] on linux2
    >>> s="Åßẞ"
    >>> s == s.upper().lower()
    False
    >>> s.lower()
    'åßß'
However, to complicate things:

    Python 3.4.2 (default, Dec 27 2014, 13:16:08)
    [GCC 4.9.2] on linux
    >>> s="Åßẞ"
    >>> s.lower()
    'åßß'
    >>> s.lower().upper()
    'ÅSSSS'
    >>> s == s.lower().upper()
    False
    >>> s.lower().upper() == 'ÅSSSS'
    True
    >>> 'SS'.lower()
    'ss'
    >>> 'ß'.lower()
    'ß'
    >>> 'ß'.lower().upper()
    'SS'
    >>> 'ß'.lower().upper().lower()
    'ss'
So that's fun.

Re: I Can’t Write My Name in Unicode

#272

Earlier quoted context omitted.

> For me, many of these problems are more of an input issue, than an encoding issue. I think you've hit the nail on the head here. I'm a native English speaker, so I may in fact be making bad assumptions here, but I think the biggest issue here is that people conflate text input systems with text encoding systems. Unicode is all about representing written text in a way that computers can understand. But the way that…

They're not unrelated though. You have to have a way to get from your input format to the finished product in a consistent way, and the glyph set you design has a large bearing on that. You can't solve it completely with AI, because then you just have an AI interpretation of human language, not human language. A language like Korean written in Hangul would need to create individual glyphs from smaller ones through th…

Interestingly enough, one early Apple II-era input method for Chinese would generate the characters on the fly (since the hardware at the time couldn't handle storing, searching and rendering massive fonts) meaning it could generate partial Chinese characters or ones that didn't actually exist.

http://en.wikipedia.org/wiki/Cangjie_input_method#Early_Cang...

In the Wikipedia article it even shows an example of a rare character that's not encoded in Unicode but which can be represented using this method.

Re: I Can’t Write My Name in Unicode

#274

Even though I'm not a native english speaker and couldn't write my name in ASCI, I really despise Unicode. Its broken technically and a setback socially. Unicode itself is such a unfathomably huge project that it's impossible to do it right, too many languages, too many weird writing systems, and too many ways to do mathematical notation on paper that can't be expressed. Just look at the code pages, they are an utter…

Why do you think English is the best candidate for the universal language, how do you define simplicity? First of all, pronunciation and spelling are almost unrelated and you have to learn them separately. That results in really different accents throughout the world. Even if you look at AmE and BrE, they differ much at the word level. Which one you want to choose? Besides, personally I find English really ambiguous and density of idioms in average text repelling, although that's only a subjective opinion.

Usage of Latin alphabet in English seems like it's on plus, but there's at least one language that uses that simple alphabet better.

> Besides. UTF-8 is broken because it doesn't allow for constant time random character and length counting.

And why you'd want that? And how do you define length? Are you a troll?

Re: I Can’t Write My Name in Unicode

#275
post #271

Earlier quoted context omitted.

.Net seems to do the same thing, Javascript (according to jsfiddle) as well. So maybe this is more widespread than I thought (again - I have never seen that character in the wild)? Java (as in Try Clojure) seems to do the 'expected' SS thing. Trying the golang playground I get even worse: fmt.Println(strings.ToUpper("ßẞ")) returns ßẞ (yeah, unchanged?) So, while I agree that you're technically correct (ẞ exists!) I d…

I think this is more related to the fact that there aren't many sane libraries implementing unicode and locales -- so you'll get either some c lib/c++ lib, system lib, java lib -- or an actual new implementation that's actually been done "seriously" -- as part of being able to say: "Yes, X does actually support unicode strings.". Python3 got a lot of flac for the decision to break away from it's byte sequences, to it…

FWIW,

- 3.2 is considered broken with a narrow unicode build (although it doesn't matter here)

- .lower and .upper are primarily for display purposes

- .casefold is for caseless matching

Re: I Can’t Write My Name in Unicode

#276

The argument that author makes is "every letter in English alphabet is represented, why not every letter/grapheme in Bengali/Tamil/Telugu/Name your language" argument is specious at best

Except that the whole purpose of Unicode is to create a character encoding that "enables people around the world to use computers in any language" - taken from the Unicode Consortium website. Bengali is also not an obscure language. It is the 10th most spoken language in the world and the national language of Bangladesh.

Re: I Can’t Write My Name in Unicode

#277
post #276

The argument that author makes is "every letter in English alphabet is represented, why not every letter/grapheme in Bengali/Tamil/Telugu/Name your language" argument is specious at best

Except that the whole purpose of Unicode is to create a character encoding that "enables people around the world to use computers in any language" - taken from the Unicode Consortium website. Bengali is also not an obscure language. It is the 10th most spoken language in the world and the national language of Bangladesh.

And besides, the argument isn't "Why aren't all Bengali characters represented when all English characters are?" The argument is "Why aren't all Bengali characters represented when a pile of poo is?"

Re: I Can’t Write My Name in Unicode

#278
post #248

I see many comments about Han unification being a bad idea but I am not seeing any reason why it was such a bad idea. I am from a CJK country and I find it makes a lot of sense. Most commonly used characters should be considered identical regardless of whether it is used in Chinese Japanese Korean or Vietnamese. Sure there are some characters that are rendered improperly depending on your font but I don't think that…

[deleted]

Re: I Can’t Write My Name in Unicode

#279
Re: the Han Unification debate that's going on in parallel here,

I think CJK unification makes sense from the Unicode point of view (although if they had to choose again after the expansion beyond 16-bit I doubt they'd bother with the political headache). The problem stems from the fact that only a few high-level text storage formats (HTML, MS Word, etc) have a way to mark what language some text is in. There's no way to include both Japanese and Chinese in a git commit log, or a comment on Hacker News.

Sure you can say "that's just the problem of the software developer!" but that's what we said about supporting different character sets before Unicode, and we abandoned that attitude. Hacker News is never going to add that feature on their own.

What's needed is either a "language switch" glyph in Unicode (like the right-to-left/left-to-right ones) or a layer ontop of Unicode which does implement one that gets universally implemented in OSes and rendering systems.

Re: I Can’t Write My Name in Unicode

#280

Earlier quoted context omitted.

Yeah, I thought that "No native English speaker would ever think to try “Greco Unification”" was a poor argument. In seems like a reasonable idea.

I think their argument was: The characters look the same (e.g. Russian's first character and the English A) but have different meanings. So in this example if you searched for the English word "Eat" that is also a completely legal Russian word (E, A, and T, exist in English and Russian), however it means nothing remotely similar. I don't know if they're right or wrong. I am just saying that might be the point they we…

Quite a few English and Russian Cyrillic letters unify just fine. E and A unify, and have identical lowercase forms, e and a. They don't really have different meanings, no more so than the letters E and A in English and French. T is more interesting: it has the same phonetic sound, but a different lowercase appearance: t in English, т in Russian. In this case, unification would be pretty terrible.

For simple alphabet-type languages, the basic rule should be: if the uppercase and lowercase look the same, then unify mercilessly. P (English) and Р (Russian) should unify even though they represent different consonants. But not V (English) and В (Russian): they sound the same, but have totally different graphemes. On the other hand, unifying B (English) and В (Russian) does not make sense: the lowercase forms look different: b (English) and в (Russian).

Sounds like the major problem with Unicode (and the author's complaints) was always where to draw the line. Han unification went too far and included too many characters that look different. With other languages, some common combinable characters were forced into diacritic representation rather than getting their own code points. To me, the first problem seems way more serious.

Post reply on HN