Earlier quoted context omitted.
This is like complaining that Wikipedia has only 35K Bengali articles, while English has over 4.7M! Why is Wikipedia not doing more to translate the 4.7M articles into Bengali? It does claim to be the world's encyclopedia!
Wikipedia is not forced unto the world as everyone's only source of knowledge. Unicode on the other hand is the only way many people have to input and see text in their native language. When one group proposes being the ultimate solution to everyone's problems, and then pushes their standard forward as such, complaints about inadequacies in the solution presented are perfectly fair and valid.
I Can’t Write My Name in Unicode
181–190 of 377 posts
Re: I Can’t Write My Name in Unicode
#182Earlier quoted context omitted.
I think the numbers are somewhat disputed. The People's Linguistic Survey of India says there are at least 780, with ~220 having died out in the last half century.[1] The Anthropological Survey of India reported 325 languages.[2] The discrepancies are made particularly tricky because of the somewhat ambiguous distinction between languages and dialects. [1] http://blogs.reuters.com/india/2013/09/07/india-speaks-780-l.…
So, this person thinks there are 229 languages in India, but actually there's a minimum of 325, possibly twice that?
[1] http://en.wikipedia.org/wiki/Languages_of_India#Inventories
Re: I Can’t Write My Name in Unicode
#183Earlier quoted context omitted.
> Is Bengali your first language? Yes. > [...] it's rather confusing mental acrobatics, and it doesn't reflect either how the writing system is taught, or how native speakers use it and think of it on a day-to-day basis. Mental acrobatics are part-and-parcel of the language, either in digital or non-digital form. If I were to spell out your name aloud, I would end with "ত-এ য-ফলা", which doesn't really say anything a…
> My point in the original comment (and to some extent in the preceding one) was to emphasize that a lot of these issues are at the input method level - we should not have to think about encoding as long as it accurately and unambiguously represent whatever we want it to represent. I might be sympathetic to this, except that keyboard layouts and input (esp. on mobile devices) is an even bigger mess and even more frag…
Re: I Can’t Write My Name in Unicode
#184Earlier quoted context omitted.
> Is Bengali your first language? Yes. > [...] it's rather confusing mental acrobatics, and it doesn't reflect either how the writing system is taught, or how native speakers use it and think of it on a day-to-day basis. Mental acrobatics are part-and-parcel of the language, either in digital or non-digital form. If I were to spell out your name aloud, I would end with "ত-এ য-ফলা", which doesn't really say anything a…
> My point in the original comment (and to some extent in the preceding one) was to emphasize that a lot of these issues are at the input method level - we should not have to think about encoding as long as it accurately and unambiguously represent whatever we want it to represent. I might be sympathetic to this, except that keyboard layouts and input (esp. on mobile devices) is an even bigger mess and even more frag…
Encoding was in a similar place 10-15 years ago. Almost every publisher in Bengali had their own encoding, font, and keyboard layout - the bigger ones built their own in-house systems, while the smaller ones used systems that were built or maintained by very small operators. To make things even more complicated, these systems needed a very specific combination of operating system and page layout software to work. Now the situation is quite better with most publishers switching to Unicode, at least for public facing content.
With input methods, I expect to see at least some consolidation - I don't necessarily think we need standards here, but there will clear leaders that emerge. Yes, keyboard layouts are influenced by Unicode code-points, but only in a specific context. Usually when people who already have experience with computers start to type in Bengali (or any other Indic language), they use a phonetic keyboard, which is influenced mostly by the QWERTY layout. Then, if they write a significant amount, they find that the phonetic input is not very efficient (typing kha everytime to get খ is painful), and they switch to a system where there's a one-to-one mapping between commonly used characters and keys. This does tend to have a relationship between defined codepoints and keys, but that's probably because the defined codepoints cover the basic characters in the script (so in your case, ্য would need to have a separate key, which I think is fine). There will still be awkward gestures, but that's again, a part of adjusting to the new medium. No one bats an eyelid when hitting "enter" to get a newline - but when we learn to write on paper, we never encounter the concept of a carriage-return.
> I have literally never once heard the jophola referred to as a viram and a য
Interesting - I guess we have somewhat different mental models. For me, I did think of jophola as a "hoshonto + jo", possibly because of the "jo" connection, and this was true even before I started to mess around with computers or Unicode. I always thought about jophola as a "yuktakshar", and if it's a "yuktakshar", I always mentally broke it down to its constituents.
> [...] especially when looking at some incredibly obscure and/or antiquated modifiers that are given their own characters
I think those exist because of backwards compatibility reasons. For Bengali I think Unicode made the right choice to start with the minimum number of code points (based on what ISCII had at that time). As others have pointed out elsewhere in the thread - it is an evolving standard, and additions are possible. Khanda-ta did get accepted, and contrary to what many think, non-consortium members can provide their input (for example, I am acknowledged in the khanda-ta document I linked to earlier, and all I did was participate in the mailing list and provide my suggestions and some evidence).
> Out of curiosity, where in Bengal are you from?
কলকাতা
Re: I Can’t Write My Name in Unicode
#185Earlier quoted context omitted.
> > It seems like you're trying to single out combining pairs as "less legitimate" when they're extensively used in the standard. > I'm saying that Unicode only does it in English where it makes semantic sense to a native English speaker. Well, combining characters almost never come up in English. The best I can think of would be the use of cedillas, diaereses, and acute accents in words like façade, coördinate and…
I'm coming from a German background and I sympathize with the author. German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. That's not related to composing on a keyboard. In fact, although I'm German I'm using the US keyboard layout and HAD to compose these characters now. But I wouldn't need to and the result is a single codepoint…
Doesn't have to be though, it'd be perfectly correct for an IME to generate multiple codepoints. IIRC, that's what you'd get if you typed those in a filename on OSX then asked for the native file path, as HFS+ stores filenames in NFD. Meanwhile Safari does (used to do?) the opposite, text is automatically NFC'd before sending. Things get interesting when you don't expect it and don't do unicode-equivalent comparisons.
Re: I Can’t Write My Name in Unicode
#186Earlier quoted context omitted.
> > It seems like you're trying to single out combining pairs as "less legitimate" when they're extensively used in the standard. > I'm saying that Unicode only does it in English where it makes semantic sense to a native English speaker. Well, combining characters almost never come up in English. The best I can think of would be the use of cedillas, diaereses, and acute accents in words like façade, coördinate and…
I'm coming from a German background and I sympathize with the author. German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. That's not related to composing on a keyboard. In fact, although I'm German I'm using the US keyboard layout and HAD to compose these characters now. But I wouldn't need to and the result is a single codepoint…
The reason they have two representations is for backwards compatibility with previous character encoding standards, but the unicode standard is more complex because of this (it needs to specify more equivalences for normalization). I guess for languages which were not previously covered by any standards, the unicode consortium tries to represent things "as uniquely as possible".
Re: I Can’t Write My Name in Unicode
#187“Whatever path we take, it’s imperative that the writing system of the 21st century be driven by the needs of the people using it. In the end, a non-native speaker – even one who is fluent in the language – cannot truly speak on behalf the monolingual, native speaker.” Not sure how the author can simultaneously say this, while criticizing the CJK unification, which makes total sense, and has never been a point of con…
I agree, but it seems tricky like it would be tricky to strike exactly the right balance between unifying too much and too little. The arguments for Han unification could have just as well been applied to unifying the Nordic languages – the Swedish Ä is really exactly the same letter as Danish/Norwegian Æ (except that Swedish words never ever use the latter and presumably Danish words never use the former), so it cou…
By the way, Unicode is about scripts, not languages. If we started distinguishing by language, we might need to start remembering that china doesn't have a single language. Duplicating all those characters again to cover Mandarin and Cantonese and Wu and Hakka and Hokkien?
Should we have a different letter for Æ in Bokmål and Nynorsk? How about Riskmål?
The equivalent of Æ and Ä in Han scripts have not been unified. The equivalent of a in Futura and a in Verdana have. There's tons of chinese characters, so a few of them are bound to be borderline and maybe contentious. But overall, Han unification was unavoidable for a project like Unicode.
Re: I Can’t Write My Name in Unicode
#188Earlier quoted context omitted.
> Unicode makes extensive use of combining characters for european languages, for example to produce diacritics: ìǒ or even for flag emoji. But it doesn't, for example say that a lowercase "b" is simply "a lowercase 'l' followed by an 'o' followed by an invisible joiner", because no native English speaker thinks of the character "b" as even remotely related to "lo" when reading and writing. > It seems like you're try…
> > It seems like you're trying to single out combining pairs as "less legitimate" when they're extensively used in the standard. > I'm saying that Unicode only does it in English where it makes semantic sense to a native English speaker. Well, combining characters almost never come up in English. The best I can think of would be the use of cedillas, diaereses, and acute accents in words like façade, coördinate and…
In Spanish "rr" has never been considered as a single letter. "Ch" and "ll" used to be, but not anymore. Ñ is, of course.
Re: I Can’t Write My Name in Unicode
#189Earlier quoted context omitted.
Unicode makes extensive use of combining characters for european languages, for example to produce diacritics: ìǒ or even for flag emoji. A correct rendering system will properly combine those, and if it doesn't then that's a flaw in the implementation, not the standard. It seems like you're trying to single out combining pairs as "less legitimate" when they're extensively used in the standard.
> Unicode makes extensive use of combining characters for european languages, for example to produce diacritics: ìǒ or even for flag emoji. But it doesn't, for example say that a lowercase "b" is simply "a lowercase 'l' followed by an 'o' followed by an invisible joiner", because no native English speaker thinks of the character "b" as even remotely related to "lo" when reading and writing. > It seems like you're try…
As long as the typeface/font used can display the character/combination reasonably, and I can input reasonably it doesn't matter so much how it's used...
Now, having to type in 2-3 characters to get there, that's a different story, and one that should involve better input devices in most cases.
Re: I Can’t Write My Name in Unicode
#190Earlier quoted context omitted.
They're not unrelated though. You have to have a way to get from your input format to the finished product in a consistent way, and the glyph set you design has a large bearing on that. You can't solve it completely with AI, because then you just have an AI interpretation of human language, not human language. A language like Korean written in Hangul would need to create individual glyphs from smaller ones through th…
Its not an AI issue, just a small matter of having lots of rules. Moreover this is not just an issue for non-Western languages: the character â (lower case "a" with a circumflex) can be represented either as a single code-point U+00E2 or as an "a" combined with a "^". Furthermore Unicode implementations are required to evaluate these two versions as being equal in string comparisons, so if you search for the combined…