Live data from Hacker News

I Can’t Write My Name in Unicode

modelviewculture.com

201–210 of 377 posts

Re: I Can’t Write My Name in Unicode

#201
post #8

I came in expecting to read an article bemoaning some niche language and playing the diversity card. I was not disappointed, but as I kept reading, the author made some very good points. I don't really care that the organization is run by white men who speak English, because frankly the entire computing industry and telecommunications industry is based on that. I'm not going to argue about the original sin there, bec…

Han unification makes things really hard for programmers. You end up with code that tries to guess what language a string is in to pick out which character set should be used! It is an absolute nightmare and a horrid idea.

Well, if you find Hiragana/Katakana it's Japanese, if you find Chữ Nôm it's Vietnamese. Otherwise it's Chinese (Well, given the definition of "language" is very hard in the context of Chinese).

From a purly theoritical perspecitve the Han unification looks like a great idea. Image the horror of normalisations if it didn't happen. ; and greek questionmark would have been a joke in comparison.

Re: I Can’t Write My Name in Unicode

#202

Earlier quoted context omitted.

I'm coming from a German background and I sympathize with the author. German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. That's not related to composing on a keyboard. In fact, although I'm German I'm using the US keyboard layout and HAD to compose these characters now. But I wouldn't need to and the result is a single codepoint…

> German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. German does not consider "ä", "ö" and "ü" letters. Our alphabet has 26 letters none of which are the ones you mentioned. In fact, if you go back in History it becomes even clearer that those letters used to be ligatures in writing. They still are collated as the basic letters…

One by one:

I'm confused by your use of 'our' and 'we'. It seems you're trying to write from the general point of view of a German, answering .. a German?

Are umlauts letters? Yes. [1] [2] Maybe not the best source, but please provide a better one if you disagree so that I can actually understand where you're coming from.

I understand - I hope? - composition. And I tend to agree that it shouldn't matter much if the input just works. If I press a key labeled ü and that letter shows up on the screen, I shouldn't really care if that is one codepoint or a composition of two (or more). I do think that the history you mention is an indicator that supports the author's argument. There IS a codepoint for ü (painful to type..). For 'legacy reasons' perhaps. And it feels to me that non-ASCII characters - for legacy reasons or whatever - have better support than the ones he is complaining about, if they originate in western Europe/in my home country.

Typewriters and umlauts:

http://i.ebayimg.com/00/s/Mzk2WDQwMA==/$T2eC16N,!)sE9swmYlFP...

(basically I searched for old typewriter models, 'Adler Schreibmaschinen' results in lots of hits like that). Note the separate umlaut keys. And these are typewriters from .. the 60s? Maybe?)

1: https://de.wikipedia.org/wiki/Alphabet 2: https://de.wikipedia.org/wiki/Deutsches_Alphabet

Re: I Can’t Write My Name in Unicode

#203
post #4

I wonder if the author has submitted a proposal to get the missing glyph for their name added. You don't need to be a member of the consortium to propose adding a missing glyph/updating the standard. The point of the committee as I understand it isn't to be an expert in all forms of writing, but to take the recommendations from scholars/experts and get a working implementation, though more diverse representation of l…

> I wonder if the author has submitted a proposal to get the missing glyph for their name added.

Probably not. If he did, he wouldn't be able to rant about the injustices of the White Man, now, would he?

Re: I Can’t Write My Name in Unicode

#204

Earlier quoted context omitted.

Imagine if the letter Q had been left out of Unicode's Latin alphabet. The argument against it is that it can be written with a capital O combined with a comma. (That's going to play hell with naive sorting algorithms, of course, but oh well.) Oh, and also imagine your name is Quentin.

> Imagine if the letter Q had been left out of Unicode's Latin alphabet. To properly write my european last name I have to press between 2 and 4 different simultaneous keys, depending on the system. Han unification is beyond misguided, but combining characters is not the problem.

When Chinese linguists came up with hanyu pinyin, they specifically wanted to pick up Latin characters (1) for Chinese phonetics, so that Chinese phonetic writing could use what we'd call "white men's writing system".

Now, they did use the letter Q for the sound tɕʰ that was formerly often romanized as "ch". It is not really a "k" as Q is in English.

Are people now saying that hanyu pinyin should use a different coding to English, because it would be more "respectable" for non-English languages to have their own code points even if the character has same roots and appearance? That is absolutely pointless. The whole idea of using Q for tɕʰ is that you can use the same letter, same coding, same symbol as in English.

(1) OK they did add ü to the mix, although that is usually only used in romanization in linguistics or textbooks, and regular pinyin just replaces it with u.

Re: I Can’t Write My Name in Unicode

#205
post #157

Earlier quoted context omitted.

AFAIK it was a historical attempt to save on encoding space back when Unicode had a 16bit fixed width and could only support up to 65K characters. Now that Unicode has expanded out, I am confused why anyone still defends this practice.

That's just one of the reasons people had (and you can't really expect everyone involved to have totally congruent reasons). But how about "g"? Do you really believe the two common variants (one storey/two stories) should have separate code points? What about German vs. French vs. Danish vs. etc.? All different "g"? Why? And if not, what is the core difference?

The Cyrillic R (looks like P) and and Greek Roh (also looks like P) and the Latin P are not unified either. Although I think unifying some Greek and Cyrillic letters would have made sense.

Re: I Can’t Write My Name in Unicode

#207
post #77
post #72

Earlier quoted context omitted.

I don't understand Bengali at all. I'm trying to understand your second sentence though. When you say "no one writes that way", do you mean nobody hits the keys for letter, followed by vowel-silencing diacritic, followed by another vowel? Or do you mean the glyph that results from that combination of keystrokes doesn't match how a Bengali speaker would write it on paper? If it's the latter, isn't that an issue for th…

It has to do with how the text is rendered. For example, if you see the Bengali text on page 3 of this PDF: http://www.unicode.org/L2/L2004/04252-khanda-ta-review.pdf it is unreadable and incorrect Bengali. ;-)

On page 3 I see two different pieces of Bengali. One is text, and the other is an image. I assume you're referring to the text? What makes it wrong? And what software are you using to view the PDF? If it's wrong, it's quite possible that the software you're using doesn't render it correctly, rather than the document actually being wrong.

Re: I Can’t Write My Name in Unicode

#208
post #194

Earlier quoted context omitted.

I'm coming from a German background and I sympathize with the author. German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. That's not related to composing on a keyboard. In fact, although I'm German I'm using the US keyboard layout and HAD to compose these characters now. But I wouldn't need to and the result is a single codepoint…

8 letters actually. 'ẞ' was added quite a while later.

Agreed, it exists. But then again, most systems in use today (as far as I'm aware) would turn a ß into SS, not ẞ.

Actually I think I've never seen a ẞ in use, ever. Not once.

Now I'm running around testing 'Try $programmingLanguage' services on the net. Try Clojure for example:

> (.toUpperCase "ß") "SS"

Re: I Can’t Write My Name in Unicode

#209
post #170

Earlier quoted context omitted.

Not to say there aren’t problems with CJK unification. It’s difficult or impossible to represent old family names or newly coined characters without some means of composing characters from radicals—even if you do want “precomposed” characters a majority of the time, as is the case with, say, é (00E9) versus é (0065 0301). As far as these characters being “the same”, I think it’s better to say they’re analogues . A i…

> I would actually support the author’s “Greco Unification” strawman if it could be done in a principled way A 5 year old tech note by a Unicode Consortium member at http://www.unicode.org/notes/tn26 gives 7 reasons "why the Latin, Greek, and Cyrillic scripts have been separately encoded, rather than being encoded as a single script", and 12 reasons why the Han script was unified. Edit: These 7 reasons make a good ca…

Very good points. That’s why I said “if it could be done in a principled way”—I don’t think it can.

Re: I Can’t Write My Name in Unicode

#210

Earlier quoted context omitted.

> For me, many of these problems are more of an input issue, than an encoding issue. I think you've hit the nail on the head here. I'm a native English speaker, so I may in fact be making bad assumptions here, but I think the biggest issue here is that people conflate text input systems with text encoding systems. Unicode is all about representing written text in a way that computers can understand. But the way that…

They're not unrelated though. You have to have a way to get from your input format to the finished product in a consistent way, and the glyph set you design has a large bearing on that. You can't solve it completely with AI, because then you just have an AI interpretation of human language, not human language. A language like Korean written in Hangul would need to create individual glyphs from smaller ones through th…

Why does Unicode need to represent a half-written Japanese word? If it's half-written, you're still in the process of writing it, and this is entirely the domain of your text input system.

Which is to say, there is absolutely no need for the text input system to represent all stages of input as Unicode. It is free to represent the input however it chooses to do so, and only produce Unicode when each written unit is "committed", so to speak. To demonstrate why this is true, take the example of a handwriting recognition input system. It's obviously impossible to represent a half-written character in Unicode. It's a drawing! When the text input system is confident it knows what character is being drawn, then it can convert that into text and "commit" it to the document (or the text field, or whatever you're typing in).

But there's nothing special about drawing. You can have fancy text input systems with a keyboard that have intermediate input stages that represent half-written glyphs/words. In fact, that's basically what the OS X text input system does. I'm not a Japanese speaker, so I actually don't know whether all the intermediate forms that text input in the Japanese input systems (there's multiple of them) go through have Unicode representations, but the text input system certainly has a distinction between text that is being authored and text that has been "committed" to the document (which is to say, glyphs that are in their final form and will not be changed by subsequent typed letters). And I'm pretty sure the software that you're typing in doesn't know about the typed characters until it's "committed".

Edit: In fact, you can even see this input system at work in the US English keyboard. In OS X, with the US English keyboard, if you type Option-E, it draws the ACCUTE ACCENT glyph (´) with a yellow background. This is a transitional input form, because it's waiting for the user to type another character. If the user types a vowel, it produces the appropriate combined character (e.g. if the user types "e" it produces "é"). If the user types something else, it produces U+00B4 ACCUTE ACCENT followed by the typed character.

Post reply on HN