101–110 of 230 posts
Here’s a slightly more in-depth blog post on the many issues this causes, and how to avoid them in JavaScript: http://mathiasbynens.be/notes/javascript-unicode Some of these problems are briefly mentioned in the above post, too.
Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4…
ECMAScript 6 fixes that, mostly. See http://mathiasbynens.be/notes/javascript-unicode for details.
E.g.:
var decomp="noël";
var precomp="noël";
console.log(decomp.split(""));
console.log(precomp.split(""));
console.log(decomp.localeCompare(precomp));
["n", "o", "e", "̈", "l"]
["n", "o", "ë", "l"]
0
Note: In Chrome when comparing (e.g. sorting) a lot of strings String.prototype.localeCompare is much slower than using a pre-composed Intl.Collator instance (because internally localeCompare creates a new collator for each call). Using Intl.Collator rediced startup time of my http://greattuneplayer.jit.su/ immensely. node.js currently has no support for Intl.*. It probably will be a compile time option for 0.12.
Perl seems to pass nearly all the tests (including uppercasing baffle): $ perl -E 'use utf8; binmode STDOUT, ":utf8"; say uc("baffle");' BAFFLE The only failure I can see is that it treats "no el" as 5 characters (so reports length as 5 and reversing places the accent on the wrong character). That's documented here: http://perldoc.perl.org/perluniintro.html#Handling-Unicode "Note that Perl considers grapheme clusters t…
my $length = 0; $length++ while $dec =~ /\X/g;
A nitpick from the article >This spells trouble for languages using UTF-16 encodings (Java, C#, JavaScript). if they were using UTF-16, this wouldn't be a problem as UTF-16 can be used to perfectly well encode code points outside of the BMP (at the cost of losing ability for O(1) access to specific code points of course. If you need to know what the n-th code point is, you have to scan the string until the n-th posit…
.NET uses 16-bit characters, but you can use the System.Globalization.StringInfo class to iterate through a string one Unicode character at a time, index into strings by Unicode character, etc. The API's a bit awkward, but it works.
Earlier quoted context omitted.
>In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go. I'm not sure what "internally using unicode" means. Pyhon's internal representation of strings has changed a lot. It hasn't even been stable in Python 3. Now they are apparently using an internal representation that varies depending on the "widest" character stored. The only solution that isn't driving me ins…
In Python 3, you don't care about what they use internally. You don't need to. If you want to work with strings, you work with strings. If you want to work with bytes, you work with bytes. If you want to convert bytes into strings (maybe because it's user input that you want to work with), then you tell Python what encoding these bytes are in and you have it create a string for you. You don't care what Python uses in…
"because their string API is correct"
Apparently they have a bug in their UTF-7 parser that can lead to invalid unicode strings. Don't know if it's already fixed.
Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4…
What does it gets right????? It's all broken as nearly everything else!
It's sad 99% comments there are “oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!”
The reality is there's 1-2 languages that are trying to make it correct from the beginning (perl6, I'm looking at you). It's 2013 and if language can compose bytes to code points everyone declares a win, sticks "full unicode support" label to it and continues to str[5:9].
”But I've got UnicodeUtils!” — it won't help. People just don't want or cannot write it correctly. Word is not [a-z]. Not [[:alpha:]] either. And not [insert regex here]. You cannot reverse by reversing codepoint list. And you cannot reverse by reversing grapheme list. And string length is hard to compute and then it doesn't any sense. And indexing string doesn't make any sense and it's far away from O(1)
Earlier quoted context omitted.
>In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go. I'm not sure what "internally using unicode" means. Pyhon's internal representation of strings has changed a lot. It hasn't even been stable in Python 3. Now they are apparently using an internal representation that varies depending on the "widest" character stored. The only solution that isn't driving me ins…
In Python 3, you don't care about what they use internally. You don't need to. If you want to work with strings, you work with strings. If you want to work with bytes, you work with bytes. If you want to convert bytes into strings (maybe because it's user input that you want to work with), then you tell Python what encoding these bytes are in and you have it create a string for you. You don't care what Python uses in…
I do need to know and I always care. My requirements may be different than those of most others because I write text analysis code and I need to optimize the hell out of every single step. I shiver at the thought that any representation could be chosen for me automatically.
Of course, nothing is stopping me from simply using the bytes type instead of str, but clearly the Python community has decided to go down a road I feel is entirely wrong so I'm not coming along.
>I know that just using UTF-8 everywhere would be cool, but that's not how the world works for various reasons. One is that UTF-8 is a variable length encoding which has some performance issues for some operations (like getting the length of the string. Or finding the n-th character).
I'm bound to live in a variable length character world unless I decide to use 4 byte code points everywhere, which is prohibitive in terms of memory usage. Memory usage is absolutely critical. Iterating over a few characters now and then to count them is almost always negligible.
The need to index into a string to find the nth character only comes up when I know what I'm looking for. Things like parsing syntax or protocol headers come to mind, and they are always ASCII. I don't remember a situation where I needed to know the nth character of some arbitrary piece of text and repeat that operation in a tight loop.
If one day I find myself in such a situation I will have to convert to an array of code points anyway.
Earlier quoted context omitted.
.NET uses 16-bit characters, but you can use the System.Globalization.StringInfo class to iterate through a string one Unicode character at a time, index into strings by Unicode character, etc. The API's a bit awkward, but it works.
One unicode character at a time or one unicode codepoint at a time? (see character composition)