Please, if you're going to post text to a Gist at least use the .md extension: https://gist.github.com/4151124
Which enables an even more readable layout with gist.io http://gist.io/4151124
Why we can't process Emoji anymore
101–110 of 162 posts
Re: Why we can't process Emoji anymore
#102Earlier quoted context omitted.
Code points aren't letters. Consider the following sequence of code points: U+0041 U+0308 [edit: corrected sequence] That equals this european letter: Ä Two code points, one letter. MAGIC! You can also get the same-looking letter with a single code point using U+00C4 (unicode likes redundancy). Not all languages have letters. Not all languages that have letters represent each one with a single code point. Please thin…
> Two code points, one letter. Yes I under stand there are million ways to display the same shape using various unicode. But how does that make code point counting impossible? AND if you explictly using COMBINING DIAERESIS instead of single U+00C4, counting diaeresis separately is wrong somehow? Why don't we make a law stating that both ae and æ is single letter?
I am responding to your earlier post which announced that UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Hopefully now you understand that just taking the number of UCS2 bytes and dividing by 2 does not give you the number of letters.
Just in case you don't, let's walk through it again.
UCS-16 big-endian represenation of Ä:
0x00 0x41 0x03 0x08
Another UCS-16 big-endian representation of Ä:
0x00 0xc4
If you look at the number of bytes, the first example has 4. It represents one letter. The second example has 2. It also represents one letter. Conclusion: UCS2 does not "count unicode characters faster than UTF8." You still have to look at every byte to see how many letters you have, same as in UTF-8.
Do you grasp this? If not, maybe you are one of those "ascii-centric ignorant morons" I keep hearing so much about.
Re: Why we can't process Emoji anymore
#103Earlier quoted context omitted.
You've moved the goalposts: u'\U00021613' This is a UTF-32 code unit, not a UTF-16 code unit. Even UTF-32 doesn't help when you have combining characters. I suggest you read dietrichepp's post again, he's talking about Normalization Form D.
Okay, if it's an explicit combining character what's wrong with explicit character part counting? You know normalized form is the norm , right?
There are four different normalized forms in Unicode. Maybe you should enlighten us about which one you're talking about.
Or just stop embarrassing yourself.
Re: Why we can't process Emoji anymore
#104Earlier quoted context omitted.
UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6
Issues like this are why I hate internationalization. If it was simple as making everything Unicode and it Just Working, it would be possible. But the number of difficulties and problems I've seen have made me decide -- and tell everyone I know -- to avoid dealing with internationalization if you value your sanity. Issues discussed here: * Different incompatible variable-length encodings * Broken implementations * Ch…
I'll take a few of those questions.
> If I split() a string, does each piece get its own BOM?
Conceptually, each piece is a sequence of code points. The BOM stuff only comes into play when you turn it into an external encoding. And frankly, I would much rather use UTF-8, explicitly specify that the encoding is UTF-8, and not have to worry about adding a BOM.
> If a chr(0x20) is part of a multi-byte escape sequence, does it count as a space when I use .split()?
In valid UTF-8, all bytes in multibyte characters will have the high bit set. A space can only be represented as the 0x20 byte, and an 0x20 byte can only be a space. If you've got malformed input, then that's a whole other can of worms.
> Is it possible for a zero byte to be part of a multibyte sequence representing a character? How does this work with C API's that expect zero-terminated strings?
In UTF-8, the answer is no. In other multibyte encodings (e.g. UTF-16), you should not expect to be able to treat it at all like ASCII.
> If you're using UTF-16, what endianness is used? Is it the same as the machine endianness, or fixed? What operations cause endian conversion?
When reading external text, you can detect this from the BOM -- byte order, after all, is why you have a byte order marker. When converting from your internal format to UTF-16, you pick whatever is most convenient.
> Should my C programs handle the possibility that sizeof(char) != 1? Or at least check for this case and spit out a warning or error?
I don't know any popular non-embedded platform on which sizeof(char) != 1. That said, it can't hurt to get it Right.
> What automated tools exist to remove BOM's or change accented characters into regular ones, if other automated tools don't accept Unicode?
In Python, there's a library called "unidecode" which does a pretty good job of punching Unicode text until it turns into ASCII.
Re: Why we can't process Emoji anymore
#105A couple of reasons why it makes sense for V8 and other vendors to use UCS2: - The spec says UCS2 or UTF16. Those are the only options. - UCS2 allows random access to characters, UTF-16 does not. - Remember how the JS engines were fighting for speed on arbitrary benchmarks, and nobody cared about anything else for 5 years? UCS2 helps string benchmarks be fast! - Changing from UCS2 to UTF-16 might "break the web", som…
UCS2 allows random access to characters, UTF-16 does not.
I'm not sure if that's really true. On IBM's site, they define 3 levels of UCS-2, only one of which excludes "combining characters" (really code points).
http://pic.dhe.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%...
If you have combining characters, then you can't simply take the number of bytes and divide by 2 to get the number of letters. If you don't have combining characters, then you have something which isn't terribly useful except for European languages (I think?)
Maybe someone more familiar with the implementation can describe which path they actually went down for this... given what I've heard so far, I'm not optimistic.
Re: Why we can't process Emoji anymore
#106Earlier quoted context omitted.
> Two code points, one letter. Yes I under stand there are million ways to display the same shape using various unicode. But how does that make code point counting impossible? AND if you explictly using COMBINING DIAERESIS instead of single U+00C4, counting diaeresis separately is wrong somehow? Why don't we make a law stating that both ae and æ is single letter?
I am responding to your earlier post which announced that UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Hopefully now you understand that just taking the number of UCS2 bytes and dividing by 2 does not give you the number of letters. Just in case you don't, let's walk through it again. UCS-16 big-endian represenation of Ä: 0x00 0x41 0x03 0x08 Another UCS-16 big-endian repr…
Name one Unicode implementation which shows utf16 `0x00 0x41 0x03 0x08` as length 1.
U+4100 U+0803 is two code points by defintion. Thus length == 2.
Re: Why we can't process Emoji anymore
#107Earlier quoted context omitted.
Okay, if it's an explicit combining character what's wrong with explicit character part counting? You know normalized form is the norm , right?
There are four different normalized forms in Unicode. Maybe you should enlighten us about which one you're talking about. Or just stop embarrassing yourself.
Reading all of your comments, so you are suggesting a Unicode object should not have len() or substring() ?
A standard like that is totally not embarrassing.
Re: Why we can't process Emoji anymore
#108Sometimes you need to know about encodings, even if you're just a consumer. Putting just one non 7-bit character in your SMS message will silently change its encoding from 7-bit (160 chars) to 8-bit (140 chars) or even 16 bit (70 chars) which might make the phone split it into many chunks. The resulting chunks are billed as separate messages.
On iOS, using any non-basic Latin character in SMS makes it switch to 16 bit, even when there is no reason for that to happen. It's a thing that most foreign language speakers must live with. By doing this full of excuses write-up, this guy wasted a substantial amount of time that he could have spent better researching the issue. Your consumer doesn't care that Emoji is this much or that much bits, it doesn't matter…
This was an internal email: https://medium.com/tech-talk/1aff50f34fc
Re: Why we can't process Emoji anymore
#109Earlier quoted context omitted.
Safari uses UTF-16, not UCS-2. I believe this is true of other browsers as well. Otherwise this would render the replacement char, but it doesn't, it renders correctly: javascript:var x = '𝌆';document.write(x);
Well, a JS string is just a series of UTF-16 code-units (per ES5, there is no impl choice here), so there isn't really any encoding pre-se (and isn't necessarily a UTF-16 string, per the spec's definition thereof, as lone surrogates are valid). The fact that that works is more a testament to the the DOM being UTF-16 than JS. (On the other hand, I'm sure you knew that. But probably there are people reading your commen…
[deleted]
Re: Why we can't process Emoji anymore
#110Earlier quoted context omitted.
I am responding to your earlier post which announced that UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Hopefully now you understand that just taking the number of UCS2 bytes and dividing by 2 does not give you the number of letters. Just in case you don't, let's walk through it again. UCS-16 big-endian represenation of Ä: 0x00 0x41 0x03 0x08 Another UCS-16 big-endian repr…
Name one Unicode implementation which shows utf16 `0x00 0x41 0x03 0x08` as length 1. U+4100 U+0803 is two code points by defintion. Thus length == 2.