A rant about Ruby 1.9 String encoding
github.com
A rant about Ruby 1.9 String encoding
1–10 of 55 posts
Re: A rant about Ruby 1.9 String encoding
#2[deleted]
Re: A rant about Ruby 1.9 String encoding
#3I agree. I'd say a good 5% of development time on a new Ruby 1.9 project of mine has been spent dealing with strings. I've taken to the idea that as long as _everything_ is UTF-8, then I'll be okay, but good luck enforcing that! Especially since the whole word seems to default to ASCII, while actually _using_ multi-byte chars anyway. I think I've got it now, but no, not really. It just works on my development machine. On my server, where it actually counts, I'm getting garbage where I should be getting an accent. I've spend two days now just trying to figure out how to debug something like that!
Re: A rant about Ruby 1.9 String encoding
#4AOL. 1.9's encoding support feels thoroughly wrong to me.
Re: A rant about Ruby 1.9 String encoding
#5Extrapolating, the same rant applies to all duck-typed languages: the number of possible outcomes for a=b+c explodes depending on types and contents of a,b and c, therefore his first assumption about one dimensional space is incorrect.
This is why most C++ teams prohibit their members to overload operators.
Re: A rant about Ruby 1.9 String encoding
#6It seems like this would be a problem on Python as well, no?
Re: A rant about Ruby 1.9 String encoding
#7It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?
Re: A rant about Ruby 1.9 String encoding
#8I know nothing about the author, but there are some statements made that suggest that the author hasn't had to deal with the wild-and-woolly reality of encodings out there in a lot of extant data. One only wishes that all data were UTF-8.
What Ruby 1.9 gets absolutely right is that its String implementation is completely encoding agnostic (by which I specifically mean that it doesn't force your data to be encoded in a particular way). There are encodings for which there is no safe UTF-8 roundtrip (you can successfully convert the data to UTF-8 nicely, but when you convert back to UTF-8 to that encoding, you won't get the original input back; you'll get a slightly different output).
Rubyists in Japan don't have the luxury of dealing with Unicode all the time; they still get lots of data in ShiftJIS and other encodings. (The same is true of Rubyists elsewhere, but since US-ASCII is a proper subset of UTF-8, most folks don't know the difference; Win1252 is a pain in the ass, though.) If you have to do ANY work with older data formats, you curse languages that force you to use UTF-8 all the time instead of letting you work with the native data.
Most developers don't think about i18n nearly enough in any case; there's a lot more to worry about that simply using Unicode doesn't solve for you. Even the developers of Ruby have to worry about the fact that LATIN SMALL LETTER E WITH ACUTE (U+00E9) is the same as LATIN SMALL LETTER E (U+0065) COMBINING ACUTE ACCENT (U+0301); it doesn't begin to address the capitalization of 'ß' ('SS', which isn't necessarily reversible) or that in Turkish 'ı' capitalizes to 'I', but 'i' capitalizes to 'İ'. Don't EVEN get me started on number formatting...
EDIT: Added the last paragraph.
Re: A rant about Ruby 1.9 String encoding
#9It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?
The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.")
Loss of text data is bad.
Re: A rant about Ruby 1.9 String encoding
#10It seems like this would be a problem on Python as well, no?
Python "cheats" by converting everything to Unicode internally. It seems like a simple solution, but it's not a solution since not everything can be converted safely back to the original encoding.