I Can’t Write My Name in Unicode • Hacker News Board

Re: I Can’t Write My Name in Unicode

#331

Post by fenomas » Wed, Mar 18, 2015, 1:48 PM UTC

microcolonel wrote: post #34

“Whatever path we take, it’s imperative that the writing system of the 21st century be driven by the needs of the people using it. In the end, a non-native speaker – even one who is fluent in the language – cannot truly speak on behalf the monolingual, native speaker.” Not sure how the author can simultaneously say this, while criticizing the CJK unification, which makes total sense, and has never been a point of con…

You're oversimplifying this.

Take characters like 今 or 骨. Whether the C/J/K variants are the "same characters" or not is semantic, but philosophy aside they are written differently in each language. That is to say, if you are localizing an app into a CJK language, your app would not be working correctly if it displayed Japanese text with a Chinese 今, or vice versa. If you showed it to a user you'd get a bug report.

But such characters have only one codepoint, so unless you have language metadata for your text, you can't render it correctly (which is half of was so bad about the pre-unicode days!).

(Note: Unicode later added "variation selectors", which as far as I know solve the problems mentioned above. I don't know why the aren't in widespread use, or perhaps they are and I'm misunderstanding something.)

Re: I Can’t Write My Name in Unicode

#332

Post by krdln » Wed, Mar 18, 2015, 1:49 PM UTC

ticking wrote: post #321

krdln wrote: post #274

Earlier quoted context omitted.

Why do you think English is the best candidate for the universal language, how do you define simplicity? First of all, pronunciation and spelling are almost unrelated and you have to learn them separately. That results in really different accents throughout the world. Even if you look at AmE and BrE, they differ much at the word level. Which one you want to choose? Besides, personally I find English really ambiguous…

English is the best candidate because it has the second largest user base (1.2 Billion vs 1.3 Billion for Mandarin), http://en.wikipedia.org/wiki/List_of_languages_by_total_numb... and is twice as spoken as the third most popular language Spanish. (0.55 Billion) If I got to pick the universal language, it would be Lojban (a few hundred speakers), but that is not a realistic goal, teaching the other 6 Billion people a…

Even if 1.2 billion seems a lot, that's still a small fraction of a world's population. So every choice of a universal language would force majority of a world to learn new one. So that's why I think winning popularity contest is a poor argument and we shouldn't look at that and focus on things like simplicity (which I don't find in English), speed of learning, consistency, expressiveness etc. I'd be happy to use Lojban (it's easier for machines too, I guess) or any other invented language. If I had to pick one from popular ones, I'd like Spanish more than English.

I was asking what are your specific usecases, which forbid you to treat UTF-8 string as a black box blob of bytes? If dealing with international code, you'd rather want to use predefined functions. If you want to limit yourself to ASCII, just do it and simply don't touch bytes >= 0x80.

And what is a character? Do you mean graphemes or codepoints? Or something else? Few years before I was thinking like you – that calculating length is a useful feature. But most often when you think about your usecase, you realise either that you don't need length or you need some other kind of length: like monospace-width, rendered-width or some kind of entropy-based amount of information. Twitter is the only case I know, where you want to really count "characters". And I find it really silly: eg. Japanese tweet vs. English tweet.

Re: I Can’t Write My Name in Unicode

#333

Post by [deleted] » Wed, Mar 18, 2015, 1:51 PM UTC

[deleted]

Re: I Can’t Write My Name in Unicode

#334

Post by chris_wot » Wed, Mar 18, 2015, 1:54 PM UTC

EdiX wrote: post #84

bbreier wrote: post #13

Earlier quoted context omitted.

The author's explanation of what characters Chinese, Japanese, and Korean share is very limited. All three languages use Chinese characters in written language to varying extents, and in some cases the differences begin significantly less than a century ago. Though there are cases where the same Chinese character represented in Japanese writing is different from how it is represented in Traditional Chinese writing (i…

> If they are talking about unifying these alphabets, then they are out of their minds. AFAIK the author is just discussing han unification: http://en.wikipedia.org/wiki/Han_unification

According to Wikipedia, this is being coordinated by the Ideographic Rapporteur Group, and "the working members of the IRG are either appointed by member governments, or are invited experts from other countries. IRG members include Mainland China, Hong Kong, Macau, Taipei Computer Association, Singapore, Japan, South Korea, North Korea, Vietnam and United States."

So this criticism of English speakers seems pretty unfounded! And his concerns about unification is being driven by a diverse group of experts in a variety of countries - so not sure why the concern?

Re: I Can’t Write My Name in Unicode

#335

Post by chris_wot » Wed, Mar 18, 2015, 2:05 PM UTC

iso8859-1 wrote: post #256

wbkang wrote: post #248

I see many comments about Han unification being a bad idea but I am not seeing any reason why it was such a bad idea. I am from a CJK country and I find it makes a lot of sense. Most commonly used characters should be considered identical regardless of whether it is used in Chinese Japanese Korean or Vietnamese. Sure there are some characters that are rendered improperly depending on your font but I don't think that…

Is it really depending on the font, or is it depending on some language metadata? Having it depend on the font seems stupid, since a font ideally be able to represent a languages which use a script encodable using Unicode.

You have the world's greatest username and I was wondering when you would turn up!

Re: I Can’t Write My Name in Unicode

#336

Post by frivoal » Wed, Mar 18, 2015, 2:07 PM UTC

systoll wrote: post #283

TazeTSchnitzel wrote: post #237

Earlier quoted context omitted.

> Ä and Æ are more different from eachother than the characters in Chinese and Japanese which have been merged. > They are used for the same thing, they share etymology, but they are not the same letter. That second quote applies equally to Ä and Æ.

It applies to Ä and Æ... which is what the parent said. It doesn't apply to 気 and 气 and 氣. Those are all the same thing.

気　气 and 氣 are actually not merged by han unification, and could be described as similar to similar to Ä and Æ: shared etymology, same meaning, some languages decided to use a simpler character because the old one was too complicated to write.

The variations on characters which have been merged are usually even closer than that. More like the single or double storey "a", or the single or double loop "g".

Re: I Can’t Write My Name in Unicode

#337

Post by jdmichal » Wed, Mar 18, 2015, 2:15 PM UTC

elFarto wrote: post #308

jdmichal wrote: post #288

Earlier quoted context omitted.

To be fair, proper codepoint processing is a pain even in Java, which was created back when Unicode was in 16-bit mode. Now that it's extended to 32-bits, proper Unicode string looping looks something like this: for(int i = 0; i

Actually, that's not correct, and it's the exact same mistake I made when using that API. codePointAt returns the codepoint at index i, where i is measured in 16-bit chars, which means you could index into the middle of a surrogate pair. The correct version is: for (int i = 0; i Java 8 seems to have acquired a codePoints() method on the CharSequence interface which seems to do the same thing. But this just adds to th…

I think you missed the part where `i` is not incremented in the for statement, but inside the loop using `Character.charCount`, which returns the number of `char` necessary to represent the code point. If there's something wrong with this, my unit tests have never brought it up, and I am always sure to test with multi-`char` codepoints.

Re: I Can’t Write My Name in Unicode

#338

Post by chris_wot » Wed, Mar 18, 2015, 2:18 PM UTC

pc2g4d wrote: post #178

qntm wrote: post #17

> He proudly announces that there are ‘no fewer than 147 Indian dialects’ – a pathetically inaccurate count. (Today, India has 57 non-endangered and 172 endangered languages, each with multiple dialects – not even counting the many more that have died out in the century since My Fair Lady took place) So, how many were there really? At the time, I mean.

I believe the number of "dialects" named in My Fair Lady can be largely explained by the lack of clear distinction between language and dialect over the years. From [1]: "There is no universally accepted criterion for distinguishing a language from a dialect. A number of rough measures exist, sometimes leading to contradictory results. The distinction is therefore subjective and depends on the user's frame of referen…

It also doesn't make sense to criticize the Unicode Consortium for an inaccurate quote from a playwright who wrote a play a century ago.

Re: I Can’t Write My Name in Unicode

#339

Post by frivoal » Wed, Mar 18, 2015, 2:32 PM UTC

zhemao wrote: post #247

kijin wrote: post #239

Earlier quoted context omitted.

Which is exactly why 飞 and 飛 are encoded separately. I don't see any problem with that.

Yes, but other characters that also look different are merged. Here's an example: http://www.tofugu.com/2012/04/04/the-sorry-state-of-japanese... That's the character for "cold". If you showed me (a Chinese speaker) the Japanese or Korean variant, I would have no idea what it meant.

The fact that some characters are debatable does not change the fact that han unification is a good idea. In a few cases you can disagree, but not unifying at all would be madness.

As for this particular character (cold), both variants are familiar to Japanese readers, with the one described as Japanese in your link being the one you'd typically see in print, while the other one is common in handwriting, and nobody in Japan would treat these two as different. From a Japanese point of view, this is definitely the kind of thing you change by switching fonts.

This pdf is the official list of basic chinese characters, published by the Japanese governement. Look on page 9, it shows both variants of this characters in hand writing.

http://www.bunka.go.jp/kokugo_nihongo/pdf/jouyoukanjihyou_h2...

The fact that one of the variants in not familiar to Chinese readers complicates the issue, but there are at least reasonable reasons to argue that this is one, not two, characters.

I think it is possible to argue that han unification was not done very well and that the UC made too many classification mistakes (although I personally think it is generally not that bad), but I don't think arguing that unification is a bad thing entirely has legs.

Re: I Can’t Write My Name in Unicode

#340

Post by fenomas » Wed, Mar 18, 2015, 2:50 PM UTC

SiVal wrote: post #311

mbessey wrote: post #261

Earlier quoted context omitted.

I respectfully disagree. If Japanese ideograms and Chinese ideograms actually used different code points (i.e. no "Han unification"), then the problem wouldn't exist - the phone could trivially use a Japanese font for Japanese text, and a Chinese font for Chinese text.

No. Using different code points for the same character used in different languages creates big problems. It would be like having different code points for 'A' depending on whether it was used in English, Spanish, German, etc. If you somehow ended up writing "color" with both 'o' characters from the Spanish ABCs and the others from the English ABCs, you'd have a real mess when it came to sorting, searching, name match…

Sure, having no unification at all would be bad, but the issue is with the gray zone. Some characters are written identically in each CJK language, but among those that aren't the amount of difference varies widely. The trouble is that Unicode leaves separate codepoints for each version that somebody, somewhere decided were "different enough" (even when they are the same character historically and linguistically) but merges many characters with (consistent, well-defined) differences because somebody felt they were close enough for horseshoes. People often think that characters were only merged if they were linguistically the same, but that's not the case.

Also, comparisons like "different ABCs for English and Spanish" are spurious and unhelpful. If you could tell an English "b" from a Spanish one by looking at it, the comparison would be sound.