Live data from Hacker News

I Can’t Write My Name in Unicode

modelviewculture.com

321–330 of 377 posts

Re: I Can’t Write My Name in Unicode

#321
post #274

Even though I'm not a native english speaker and couldn't write my name in ASCI, I really despise Unicode. Its broken technically and a setback socially. Unicode itself is such a unfathomably huge project that it's impossible to do it right, too many languages, too many weird writing systems, and too many ways to do mathematical notation on paper that can't be expressed. Just look at the code pages, they are an utter…

Why do you think English is the best candidate for the universal language, how do you define simplicity? First of all, pronunciation and spelling are almost unrelated and you have to learn them separately. That results in really different accents throughout the world. Even if you look at AmE and BrE, they differ much at the word level. Which one you want to choose? Besides, personally I find English really ambiguous…

English is the best candidate because it has the second largest user base (1.2 Billion vs 1.3 Billion for Mandarin), http://en.wikipedia.org/wiki/List_of_languages_by_total_numb... and is twice as spoken as the third most popular language Spanish. (0.55 Billion)

If I got to pick the universal language, it would be Lojban (a few hundred speakers), but that is not a realistic goal, teaching the other 6 Billion people a language that is already spoken by 1/7th of the population is at least plausible.

> Why would you want that...

Why would, you not want that?! Many popular programming languages are based on array indexing through pointer arithmetic, having a variable width encoding there is a horrible idea, because you have to iterate through the text to get to an index.

Length is the number of characters, which is just the number of bytes in ASCI, but has to be calculated by looking at every character in UTF-8.

Re: I Can’t Write My Name in Unicode

#322
post #313

Earlier quoted context omitted.

Is Bengali your first language? A better question is, Are there any native Bengali speakers creating character set standards in Bangladesh or India? If not, why not? If so, did they omit your character? I ask, because although you prefer to follow the orthodox pattern of blaming white racism for your grievance du jour, the policy of the Unicode Technical Committee for years has been to use the national standards crea…

I mostly agree with your point, but note that the author is male (well, the name is a commonly male one). It's a bit telling that folks in the software industry[1] seem to assume that techies are male (a priori), but those who write articles of this kind are female. Not blaming you for it, but it's something you should try to be conscious about and fix. [1] I've been guilty of this myself, though usually in cases whe…

I had a female coworker by that name, so your assumption that I just assume that people who write articles like this are female and need to have my consciousness raised to "fix" my unconscious sexism is something you should try to be more conscious of and try to fix.

However, I clearly do need to question my assumption that since this was a female name before, it's a female name now, so I should change it to "he/she".

Re: I Can’t Write My Name in Unicode

#323

I don't understand; I don't feel like character combination using the zero width joiner is on the same level as 13375p34k. It looks like the character just doesn't have a separate code-point, but is instead a composite, but still technically "reachable" from within Unicode, no?

I agree.

Most devanagari glyphs don't have their own codepoint. Marathi/Hindi/Sanskrit (which use devanagari) have a bunch of basic consonants and some vowels (which can appear independently or as modifiers). All the glyphs are logically formed by mixing the two, so the glyph for "foo" would be the consonant for "f"[1] plus the vowel modifier for "oo". When typing this in Unicode, you would do the same, type फ then ू, creating फू.

It gets interesting when we get to consonant clusters. As mentioned in [1], the consonants have a schwa by default, so the consonant for s followed by the consonant for k with a vowel modifier for the "y" sound would not be "sky", but instead something like "səky" (suh-ky).

So how do we write these? We can do this in two ways. One way is to use the no-vowel modifier, which looks like a straight tail[3] (on स, the consonant for "s", or "sə", the tail looks like स्), and follow that by the other consonant. So "sky" would be स् कै [2]. This method is rarely used, and the straight-tail is only used when you want to end a word with a consonant[4].

The more common way of doing multiple consonants are by using glyphs known as consonant clusters or conjuncts[5]. For "sky", we get स्कै, which is a partial glyph for स stuck to the glyph for कै. For most clusters you can take the known "partial" form of the first glyph and stick it to the full form of the second glyph, but there are tons of exceptions, eg द+द=द्ध, ह+म=ह्म (the second character was broken), and whatnot. See http://en.wikipedia.org/wiki/Devanagari#Biconsonantal_conjun... if you want a full table.

There aren't individual Unicode codepoints for this, not even codepoints for the straight-tail form of the consonants. I typed स्कै as स्+कै which was itself typed as स + ् + क + ै. This isn't an irregular occurrence either, consonant clusters (with a vowel modifier!) are pretty common in these languages[6].

I personally don't see anything wrong with having to use combining characters to get a single glyph. If it's possible and logical for it to be broken up that way, it's fine. With this trick, it's possible to represent Devanagari as a 128-codepoint block (http://en.wikipedia.org/wiki/Devanagari_%28Unicode_block%29), including a lot of the archaic stuff. It's either that, or you make a characters for every combined glyph there is, which is a lot[7]. One could argue that things like o-umlaut get their own codepoint, but स्क doesn't, but o-umlaut is one of maybe 10 such characters for a given European language, whereas स्क is one of around 700 (and that number is being conservative with the "usefulness" of the glyph, see [7]).

The article is never quite clear about which glyph Aditya finds lacking for his name (sadly, I don't know Bengali so I can't figure it out), but from the comments it seems like it it's something which can be inputted in Unicode, just not as a single character. That's okay, I guess. And if it's not showing up properly, that's a fault of the font. (And if it's hard to input, a fault of the keyboard).

It becomes a Unicode problem when:

- There is no way to input the glyph as a set of unicode code points, or - The input method for the glyph as a set of unicode code points can also mean and look like something else given a context (fonts can only implement one, so it's not fair to blame them)

[1]: well, fə, since the consonants are schwa'd by default. Pronounced "fuh" (ish)

[2]: the space is intentional here so that I can type this without it becoming something else, but in written form you wouldn't have the space. Also it's not exactly "sky", but close enough.

[3]: called paimodi ("broken foot") in Marathi

[4]: which is pretty rare in these languages. In some cases however, words that end with consonant-vowel combinations do get pronounced as if they end with a consonant (http://en.wikipedia.org/wiki/Schwa_deletion_in_Indo-Aryan_la...), but they're still written as if they ended with a vowel (this is true for my own name too, the schwa at the end is dropped). Generally words ending with consonants are only seen in onomatopoeia and whatnot.

[5]: called jodakshar ("joined word") in Marathi

[6]: Almost as common as having two side by side consonants in English. We like vowels, so it's a bit less common, but still common.

[7]: technically infinite, though that table of 700-odd biconsonantal conjuncts would contain all the common ones (assuming we still have the vowel modifying diacritics as separate codepoints), and adding a few triconsonantal conjuncts would represent almost all of what you need to write Marathi/Hindi/Sanskrit words. It wouldn't let you represent all sounds though, unless you still have the ् modifier, in which case why not just use that in the first place?

Re: I Can’t Write My Name in Unicode

#324
post #301

Earlier quoted context omitted.

Was British rule actually a net negative, in retrospect? Have there been studies done using objective criteria (not emotional) over counties that were colonies versus ones that weren't? I suppose you can't really quantify the value of people that were destroyed by colonization, but you can look at the current population. Also I just gotta wonder: suppose European or other relatively simple-to-encode languages didn't…

[deleted]

Although I agree with a lot of what you said, I perceive the last part, your post-scriptum, to be a little bit distorted. The Renaissance had many sources. One of them was the massive intellectual immigration from the crumbling Byzantine Empire, thus serving for much of intellectual works as a bridge over the time (from antiquity to enlightened medieval period) and space (near east to all over the Europe). And it was mostly Greek in its cultural background.

Re: I Can’t Write My Name in Unicode

#325
post #313

Earlier quoted context omitted.

Is Bengali your first language? While one can make the case that ত্য is simply "'to' - 'o' + 'ya' = 'to'"[0][1], it's rather confusing mental acrobatics, and it doesn't reflect either how the writing system is taught, or how native speakers use it and think of it on a day-to-day basis. If anything, your comment makes a stronger argument for consolidating ই and ি (they are literally the same letter and phoneme, but wr…

Is Bengali your first language? A better question is, Are there any native Bengali speakers creating character set standards in Bangladesh or India? If not, why not? If so, did they omit your character? I ask, because although you prefer to follow the orthodox pattern of blaming white racism for your grievance du jour, the policy of the Unicode Technical Committee for years has been to use the national standards crea…

> I had a female coworker by that name

Oh, sorry about that. Not sure if you're joking about the assumption of assumptions, but asking people to take note of their behavior based on something that they _might_ have assumed is not dangerous. Assuming gender roles is. Apologies for making that assumption, but IMO it's a rather harmless one so I don't see anything to fix about it :P

Re: I Can’t Write My Name in Unicode

#326
post #48

Not sure if the l33tspeak analogy is fully justified. In case of the "missing" letter (called khanda-ta in Bengali) for the Bengali equivalent of "suddenly", historically, it has been a derivative of the ta-halant form (ত + ্ + ‍ ). As the language evolved, khanda-ta became a grapheme of its own, and Unicode 4.1 did encode it as a distinct grapheme. A nicely written review of the discussions around the addition can b…

> I could write the author's name fine: আদিত্য Author here. Well, yes and no. The jophola at the end is not actually given its own codepoint[0]. The best analogy I can give is to a ligature in English[1]. The Bengali fonts that you have installed happen to render it as a jophola, the way some fonts happen to render "ff" as "ff" but that's not the same thing as saying that it actually is a jophola (according to the Uni…

(I can't read Bengali, so I'm not entirely sure what the johphola is, but I'm trying to relate this to Devanagari -- if my analogy is mistaken or if you don't know Devanagari, let me know)

I don't see anything discriminatory about not giving glyphs their own codepoints. Devanagari has tons of glyphs which are logically broken up into consonants, modifying diacritics, and bare vowels. Do we really need separate codepoints for these[1] when they can be made by combinations of these[2]?

I mentioned this as a reply to another comment, but it's only a Unicode problem if:

- There is no way to write the glyph as a combination of code points - There is a way to write the glyph as a combination of code points, but the same combination could mean something else (not counting any rendering mistakes, the question is about if Unicode defines it to mean something else)

If it's hard to input, that's the fault of the input method. If it doesn't combine right, it's the fault of the font.

[1]: http://en.wikipedia.org/wiki/Devanagari#Biconsonantal_conjun... [2]: http://en.wikipedia.org/wiki/Devanagari_%28Unicode_block%29

Re: I Can’t Write My Name in Unicode

#327
post #253

Earlier quoted context omitted.

Nor is it unreasonable to "unify" Latin, Greek and Cyrilic: Cyrillic ПФ vs Greek ΠΦ Cyrillic АВ vs Latin AB Obviously using ω for w (as he does) is stupid, but his reducto-ad-absurdum is not particularly absurd.

Cyrillic ПФ vs Greek ΠΦ? Here's Cyrillic lower case: пф Here's Greek lower case: πφ in some fonts the pi would be rendered with a longer bar on top, but you just showed why it's a bad idea: I would want to be able to discuss Greek in Russian on a forum, but this would not be possible because all the glyphs in lowercase would look Russian

The forms 𝜙 and 𝜑 of "lowercase phi" to have different codepoints makes perfect sense to me. That doesn’t mean that upper-case variants of these can’t share a codepoint. As presented elsewhere in this thread, "X.toUpper().toLower()" doesn’t have to be "X". The same holds for "B → b" and "B → β" depending on the context. It’s just that the savings from such a unification would be far smaller due to the smaller sizes of the relevant alphabets.

Re: I Can’t Write My Name in Unicode

#328
post #313

Earlier quoted context omitted.

Is Bengali your first language? While one can make the case that ত্য is simply "'to' - 'o' + 'ya' = 'to'"[0][1], it's rather confusing mental acrobatics, and it doesn't reflect either how the writing system is taught, or how native speakers use it and think of it on a day-to-day basis. If anything, your comment makes a stronger argument for consolidating ই and ি (they are literally the same letter and phoneme, but wr…

Is Bengali your first language? A better question is, Are there any native Bengali speakers creating character set standards in Bangladesh or India? If not, why not? If so, did they omit your character? I ask, because although you prefer to follow the orthodox pattern of blaming white racism for your grievance du jour, the policy of the Unicode Technical Committee for years has been to use the national standards crea…

> The complaint in this silly article about tiny Klingon being included before a complete Bengali is precisely because getting Bengali right was more complex and far more important.

This is factually incorrect. It seems you missed both the factual point about the Klingon script in the article as well as the broader point which that detail was meant to illustrate.

> although you prefer to follow the orthodox pattern of blaming white racism for your grievance du jour, the policy of the Unicode Technical Committee for years has been to use the national standards created by the national standards bodies where these scripts are most used as their most important input.

There's a huge difference between piggybacking off of a decades-old proposed scheme which was never widely adopted even in its country of origin, and which was created under a very different set of constraints than Unicode, and which was created to address a very different set of goals than Unicode, versus making native speakers an active and equal part of the actual decision-making process.

Rather than trying to shoehorn the article into a familiar pattern which doesn't actually fit ("orthodox pattern of blaming white racism for your grievance du jour"), please take note that the argument in the article is more nuanced than you're giving it credit for.

Re: I Can’t Write My Name in Unicode

#329

Earlier quoted context omitted.

Unicode is not "forced" unto the world either. It's just a good way to do the thing it does, but it doesn't happen without contribution from those who are impacted.

The word is "unicode". "Uni" as in "united". Wikipedia doesn't claim to be the One True Encyclopedia, but with Unicode it's literally right there in the name.

That is a stretch.

Re: I Can’t Write My Name in Unicode

#330

This article is imbued with its own form of curious racism. In particular, I became suspicious of its motives at the line: > "It took half a century to replace the English-only ASCII with Unicode, and even that was only made possible with an encoding that explicitly maintains compatibility with ASCII, allowing English speakers to continue ignoring other languages." ASCII-compatibilty was essential to ensure the adopt…

It's sort of hilarious that he said that, given that English speakers were able to encode every character in 7-bit ASCII. The issues around standardising characters was because non-English characters were being squabbled about between the French, Russians and a whole bunch of other non-English countries.

In essence, he's not understood that really ASCII was used as the base for Unicode because it was widely used. In fact. It's actually ISO8859-1 that has been used because of its wide coverage of a variety of languages, far more than any of the other 8859-x character sets.

I cannot speak to anything else he's said, aside from saying that trying to encode all the world's languages is bloody hard.

Even when a limited number of countries try to nut out a standard for 128 characters, it's a nightmare. And don't forget that they were competing with EBCDIC.

I wrote about it here:

http://randomtechnicalstuff.blogspot.com.au/2009/05/unicode-...

Post reply on HN