I Can’t Write My Name in Unicode • Hacker News Board

Re: I Can’t Write My Name in Unicode

#291

Post by ptaipale » Wed, Mar 18, 2015, 6:39 AM UTC

toufka wrote: post #220

ptaipale wrote: post #179

Earlier quoted context omitted.

I think not. I do want to be able to say both People's Republic of China 中华人民共和国 and Republic of China 中華民國 in the same text, and if I had to choose rendering either 国+华 or 國+華 then it wouldn't work.

Curious, as I'm not sure when that would actually happen in real life (in Chinese). Generally in mainland China , the ROC would always be rendered with 国, even officially [1]. And in Taiwan the PRC would be rendered with 國 [2]. It gets a bit weirder in Japanese where the word is distinctly not the same - one is a traditional version (proper noun) of the other and you could imagine a text using both (William vs Wilhel…

That is true, mainland China writes "中华民国".

But I still do want to be able to write texts that are like this discussion: mainly in English, but contain fragments in Chinese, and so that I can use both the traditional and simplified characters.

And it also makes total sense to me that 日本 is Japan, both in Japanese and Chinese, using the exact same Unicode characters.

Re: I Can’t Write My Name in Unicode

#292

Post by olgeni » Wed, Mar 18, 2015, 6:41 AM UTC

I searched for "white" and of course I found a few, as expected.

Re: I Can’t Write My Name in Unicode

#293

Post by wodenokoto » Wed, Mar 18, 2015, 6:55 AM UTC

microcolonel wrote: post #34

“Whatever path we take, it’s imperative that the writing system of the 21st century be driven by the needs of the people using it. In the end, a non-native speaker – even one who is fluent in the language – cannot truly speak on behalf the monolingual, native speaker.” Not sure how the author can simultaneously say this, while criticizing the CJK unification, which makes total sense, and has never been a point of con…

I think it is a problem when 2 characters that are written differently share the same code point.

Having 大 be the same code point for Chinese, Japanese and Korean is to me as obvious as having the same A for English and Italian. We agree on this point.

But some of the unified characters listed on Wikipedia ought to have different code points because they are no longer written the same across languages.

https://en.wikipedia.org/wiki/Han_unification#Examples_of_la...

This is like having a and α share the same code point.

Re: I Can’t Write My Name in Unicode

#294

Post by ptaipale » Wed, Mar 18, 2015, 6:58 AM UTC

darklajid wrote: post #202

the_mitsuhiko wrote: post #192

Earlier quoted context omitted.

> German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. German does not consider "ä", "ö" and "ü" letters. Our alphabet has 26 letters none of which are the ones you mentioned. In fact, if you go back in History it becomes even clearer that those letters used to be ligatures in writing. They still are collated as the basic letters…

One by one: I'm confused by your use of 'our' and 'we'. It seems you're trying to write from the general point of view of a German, answering .. a German? Are umlauts letters? Yes. [1] [2] Maybe not the best source, but please provide a better one if you disagree so that I can actually understand where you're coming from. I understand - I hope? - composition. And I tend to agree that it shouldn't matter much if the i…

I am not entirely sure if Germans count umlauts as distinct characters or modified versions of the base character. And maybe it is not so important; they still do deserve their own code points.

Note BTW that in e.g. Swedish and German alphabets, there are some overlapping non-ASCII characters (ä, ö) and some that are distinct to each language (å, ü). It is important that the Swedish ä and German ä are rendered to the same code point and same representation in files; this way I can use a computer localised for Swedish and type German text. Only when I need to type ü I need to compose it from ¨ and u, while ä and ö are right on the keyboard.

The German alphabetical order supports the idea that umlauts are not so distinct from their bases: it is

AÄBCDEFGHIJKLMNOÖPQRSßTUÜVWXYZ while the Swedish/Finnish one is ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ

This has the obvious impacts on sorting order.

BTW, traditionally Swedish/Finnish did not distinguish between V and W in sorting, thus a correct sorting order would be

Vasa

Westerlund

Vinberg

Vårdö

- the W drops right in the middle, it's just an older way to write V. And Vå... is at the end of section V, while Va... is at the start.

Re: I Can’t Write My Name in Unicode

#295

Post by enupten » Wed, Mar 18, 2015, 7:26 AM UTC

I couldn't agree with you more on this!

But there is more to it when it comes to India, it is mostly that "nobody really cares".

It is sad that while India is develops, it is rapidly leaving its many languages behind when it comes to the computer; indeed, while Chinese/Arabic keyboards are extremely common, one would be hard pressed to find Hindi, "Nagari", (let alone Bengali) keyboards in India. This despite the nauseous linguistic jingoism in the country's political history. Indeed there has been, sadly very little to show, and it has infact, gotten worse past independence, as with many things; see, http://www.columbia.edu/cu/mesaas/faculty/directory/pollock_...

I would not be surprised, at this rate, if India turns into a monoculture, two centuries hence.

P.S: Speaking of representation, the script the OP talks of, belongs to the set of phonetically accurate scripts; one which while being populated mostly by Indic scripts, is given - quite disgracefully - the name "abugida", after a lone Semitic script - "Geez" - from Ethiopia - which ironically is probably derived from one of the Indic ones. Systematic biases are far too apparent in Indology.

Re: I Can’t Write My Name in Unicode

#296

Post by snogglethorpe » Wed, Mar 18, 2015, 7:31 AM UTC

zhemao wrote: post #247

kijin wrote: post #239

Earlier quoted context omitted.

Which is exactly why 飞 and 飛 are encoded separately. I don't see any problem with that.

Yes, but other characters that also look different are merged. Here's an example: http://www.tofugu.com/2012/04/04/the-sorry-state-of-japanese... That's the character for "cold". If you showed me (a Chinese speaker) the Japanese or Korean variant, I would have no idea what it meant.

As far as I can see, Unicode has mostly settled down into a sort of "good enough" state: characters that have sufficiently different renderings have gotten separate "variant" codepoints for each rendering, while characters that are very similar (even if not completely identical as commonly written) are still only present as unified codepoints.

I've no idea if these variant codepoints are actually supposed to show up in user files, or are intended mainly for the use of font rendering systems, etc... the whole thing seems a bit of a mess, even if the information is technically present.

Judging from unicode.com, "冷" does seem have separate codepoints: 冷 (chinese/unified), and 冷 (japanese z-variant). However my browser renders both as similar characters. Similarly, on my phone, the same character gets input whether using a Chinese or a Japanese input method, and both get rendered using the Japanese rendering (it's a Japanese phone) which makes Chinese text look a little funny.

An interesting example is "晩" / "晚", which has one more stroke in the Japanese variant, but it's situated in a location which makes both variants look pretty much identical (and in small bitmapped fonts, they are identical). Nonetheless, Unicode includes codepoints for both...

Re: I Can’t Write My Name in Unicode

#297

Post by PaulAJ » Wed, Mar 18, 2015, 7:37 AM UTC

eridius wrote: post #217

PaulAJ wrote: post #133

Earlier quoted context omitted.

Its not an AI issue, just a small matter of having lots of rules. Moreover this is not just an issue for non-Western languages: the character â (lower case "a" with a circumflex) can be represented either as a single code-point U+00E2 or as an "a" combined with a "^". Furthermore Unicode implementations are required to evaluate these two versions as being equal in string comparisons, so if you search for the combined…

> Unicode implementations are required to evaluate these two versions as being equal in string comparisons What do you mean by "required"? There's different forms of string equality. It's plausible to have string equality that compares the actual codepoint sequence, vs string equality that compares NFC or NFD forms, and there's string equality that compares NFKC or NFKD forms. And heck, there's also comparing strings…

Its part of the Unicode standard. See http://en.wikipedia.org/wiki/Unicode_equivalence for details.

(OK, so "required" might be overstating it; you are perfectly free to write a program that doesn't conform to the standard. But most people will consider that a bug unless there is a good reason for it)

Re: I Can’t Write My Name in Unicode

#298

Post by sirseal » Wed, Mar 18, 2015, 7:40 AM UTC

Let's use UTF-16: http://en.wikipedia.org/wiki/UTF-16. Problem solved.

Re: I Can’t Write My Name in Unicode

#299

Post by deerpig » Wed, Mar 18, 2015, 7:52 AM UTC

eloisant wrote: post #191

microcolonel wrote: post #34

“Whatever path we take, it’s imperative that the writing system of the 21st century be driven by the needs of the people using it. In the end, a non-native speaker – even one who is fluent in the language – cannot truly speak on behalf the monolingual, native speaker.” Not sure how the author can simultaneously say this, while criticizing the CJK unification, which makes total sense, and has never been a point of con…

Actually yes, the CJK unification is a problem for many people, including me when I want to read Japanese on a phone bought in Europe. Example 1. Typically, any time you want to mix the 2 languages you're getting in trouble. Let's say you write a textbook for Chinese people to learn Japanese as a second language. Or a research article in Japanese citing old Chinese literature. In your text, you'll have to mark specif…

I run into this problem quite often. Has anyone proposed creating an alias for each character that indicated that it was chinese, japanese, or korean? In this way you could mix words within documents and when it is rendered each language could be represented by different fonts? The same could be used for variations in chinese -- so that if you used a Hong Kong Cantonese or Taiwanese namespace or alias it would use traditional characters, and if using a Mandarin, or Singapore Chinese namespace it would use simplified characters.

Re: I Can’t Write My Name in Unicode

#300

Post by eridius » Wed, Mar 18, 2015, 8:03 AM UTC

PaulAJ wrote: post #297

eridius wrote: post #217

Earlier quoted context omitted.

> Unicode implementations are required to evaluate these two versions as being equal in string comparisons What do you mean by "required"? There's different forms of string equality. It's plausible to have string equality that compares the actual codepoint sequence, vs string equality that compares NFC or NFD forms, and there's string equality that compares NFKC or NFKD forms. And heck, there's also comparing strings…

Its part of the Unicode standard. See http://en.wikipedia.org/wiki/Unicode_equivalence for details. (OK, so "required" might be overstating it; you are perfectly free to write a program that doesn't conform to the standard. But most people will consider that a bug unless there is a good reason for it)

Unicode defines equivalence relations, yes. But nowhere does is a program that uses Unicode required to use a equivalence relation whenever it wishes to compare two strings. It probably should use one, but there are various reasons why it might want strict equality for certain operations.