51–60 of 230 posts
Do people really need to reverse strings in the real world? I don't think I've ever written code to do that outside of homework assignments and interviews.
Substrings exhibit similar problems and those are used quite often. It's just that in this case the effect of seeing it fail is a little more dramatic (i.e., l̈ – which doesn't even seem to render properly here).
Earlier quoted context omitted.
Use UTF8, no endian issues. Thats yet another reason why UTF16 and UTF32 are broken.
language will not store unicode string internally with UTF8. Yes, we use it as input and output, but in memory, utf8 is terrible for random access characters. endian is only an issue (normally) for input and output, not really an issue for internal storage. especially when using UTF16 and UTF32 you know exactly the size of items.
Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4 bytes per character)
In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go.
But: Due to http://en.wikipedia.org/wiki/Han_unification being locked into Unicode with a language might not be feasible for everybody - especially in Asian regions, Unicode isn't yet as widely spread and you still need to deal with regional encodings, mainly because even with the huge character set of Unicode, we still can't reliably write in every language.
Ruby 1.9 and later helps here by having many, many string types (as many as it knows encodings), which can't be assigned to each other without conversion.
This allows you to still have an internal character set for your application and doing encoding/decoding at i/o time, but you're not stuck with unicode if that's not feasible for your use-case.
People hate this though because it seems to interfere with their otherwise perfectly fine workflow ("why can't I assign this "string" I got from a user to this string variable here??"), but it's actually preventing data corruption (once strings of multiple encodings are mixed up, it's often impossible to un-mix them, if they have the same characer width).
I don't know how good the library support for the various Unicode encodings is in Ruby though. According to the article, there still is trouble with correctly doing case transformations and reversing them.
Which brings me to another point: Some of the stuff you do with strings isn't just dependent on string encoding, but also locale.
Uppercasing rules for example depend on locale, so you need to keep that into account too. And, of course, deal with cases when you don't know the locale the string was in (encoding is hard enough and most of the cases undetectable - but locales - next to impossible).
I laugh at people who constantly tell me that this isn't hard and that "it's just strings".
>This spells trouble for languages using UTF-16 encodings (Java, C#, JavaScript).
if they were using UTF-16, this wouldn't be a problem as UTF-16 can be used to perfectly well encode code points outside of the BMP (at the cost of losing ability for O(1) access to specific code points of course. If you need to know what the n-th code point is, you have to scan the string until the n-th position).
They are, however, using UCS-2 which can't. If you use a library that knows about UCS-2 to work on strings encoded in UTF-16, then you will get broken characters, your counts will be off and case transformations might fail.
Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception)
Do people really need to reverse strings in the real world? I don't think I've ever written code to do that outside of homework assignments and interviews.
Earlier quoted context omitted.
May be not reversing, but trimming a Unicode string to certain character count is a close relative and it is a very common operation.
What do you use it for? Unless you have a monospaced font the number of characters do not mean much. So unless you are implementing command line tools or text editors it should not be that common.
The problem with text (that Unicode solves only partially) is that text representation, being a representation of human thought, in inherently ambiguous and imprecise. Some examples: (1) A == A but A != Α. The last letter is not uppercase "a", but uppercase "α". Most of the time, the difference is important, but sometimes humans want to ignore it (imagine you can't find an entry in a database since it contains Α that…
I think (2) is an issue with Unicode specifically. They should have specified Turkish alphabet to use ı and a diacritic to make the dotted one. That would have made (in this case) capitalization locale-independent.