He has posted some alternatives too
http://github.com/candlerb/string19/blob/master/alternatives...
The first suggestion seems like the logical solution to me however I don't need to deal with this stuff on a day-to-day basis...
A rant about Ruby 1.9 String encoding
51–55 of 55 posts
Re: A rant about Ruby 1.9 String encoding
#52My sup (on ruby 1.9) just crashed on me because I dared to try to write a mail with UTF8 in it.
This is not a trivial screwup! This is the sort of screwup that should make everybody who's using that wretched platform think thrice before continuing to use it.
Re: A rant about Ruby 1.9 String encoding
#53I believe that Python, Java, C#, Objective-C and Javascript all have the same basic approach to this problem. The Ruby way is better for handling some Japan-specific problems. But that's at the cost of making life harder and less predictable for everyone else.
It's a pretty straightforward tradeoff. Of course people who are not Japanese will naturally be upset to pay a cost in complexity for a feature of benefit primarily to a programmers from a single country. Non-Japanese Ruby programmers will just have to decide whether their solidarity with Japanese programmers outweighs their personal and collective inconvenience.
Re: A rant about Ruby 1.9 String encoding
#54Earlier quoted context omitted.
Practically speaking, what data are you talking about? Which characters don't map? The mapping here looks pretty complete, except for "DBCS LEAD BYTE". Is that a problem? Other non-round trip mappings look like it's because different vendors have different meanings for that character. That's not Unicode's fault. http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOW...
If there's ~398 byte combinations that can be translated only one way, but not the other, you're still losing data. (The big one that always gets pointed out? '\' and '¥'.)
"(The big one that always gets pointed out? '\' and '¥’.)”
The point there though is that Japanese DON’T want \ and ¥ to map properly. They want \ and ¥ to be considered the same. So Unicode isn’t losing information, it’s forcing a distinction the Japanese don’t want to be able to make.
Re: A rant about Ruby 1.9 String encoding
#55Rants about strings and character sets that contain words of the following spirit are usually neither correct nor worth of any further thought: > It's a +String+ for crying out loud! What other > language requires you to understand this > level of complexity just to work with strings?! clearly the author lives in his ivory tower of english language environments where he is able to say that he "switched to UTF-8" with…
I liked your response a lot. Thank you. And I was almost persuaded by it. In fact I was persuaded for about 5 minutes after reading it. But at the last second, a thought occurred to me: if there's a deficiency in Unicode that prevents its use for Japanese, isn't the right solution to just fix that deficiency?
I mean, Unicode is meant to be an abstract representation of glyphs, separate from any encoding, that works for all of Earth's languages. It's tailor made to be a programming language's internal representation of a string. This is its raison d'etre.
So it seems to me that #2 is definitely The Right Way™ and that if there's some problem with Unicode that has kept Ruby from adopting it, they should have worked on fixing it, rather than breaking Ruby. OK, "break" is probably too strong a word for the state of Ruby 1.9. And in the real world, fixing an international politicized standard like Unicode is probably impossible. So I can see that this pragmatic solution might have been the only one available. But still, it seems wrong to me.
Out of curiosity, what exactly is the deficiency in Unicode that caused Matz to go with option 3? I presume there are epic flamewars all over the internet about this issue, but I just haven't been paying close enough attention.