It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?
[deleted]
A rant about Ruby 1.9 String encoding
11–20 of 55 posts
Re: A rant about Ruby 1.9 String encoding
#12It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?
Is there a technical reason why Ruby doesn't do this?
More likely it's political. Matz (the creator of Ruby), and many of its early contributors, are said not to like Unicode.
Re: A rant about Ruby 1.9 String encoding
#13It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?
I'm not a Ruby-head, so maybe I don't have the necessary context to understand what's going on, but:
As far as I'm concerned, the only time I care about encodings is when I'm trying to do I/O. I'll always have to choose an encoding when engaging in I/O, explicitly or implicitly. While I'm shuffling strings around in memory, I expect it to behave like a sequence of unicode code points. If the implementation wants to allow multiple implementations for performance reasons (e.g. because reading UTF-8 from file, converting it to UTF-16, processing it, converting it back to UTF-8 to send it over the network is inefficient) then that's fine by me. I'll still want to tell it what encoding that byte stream from which I'm writing is, and what that stream I'm writing to expects.
If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings. That was a bad idea in C and with the complexity that unicode adds, this is an even worse idea nowadays.
Re: A rant about Ruby 1.9 String encoding
#14Dealing with string encoding is sometimes a PITA. But I think as English speakers, we are usually sheltered from the problem because most programming languages are English centric. I'd like to hear opinions from people who don't speak English.
If you think the way 1.8 handles (or doesn't handle) encoding is just fine, try things you typically do, but with a different language.
For example:
ruby -ryaml -e'p YAML.dump("こんにちは！")'
IMO, not having an encoding associated with some text sucks if you're a non-english speaker.
Re: A rant about Ruby 1.9 String encoding
#15It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?
The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.") Loss of text data is bad.
That page says there are duplicates in CP-932 because of different vendor variants. If those characters are otherwise entirely identical, calling it a single encoding seems wrong. Wouldn't you just have a CP-932-IBM and a CP-932-NEC encoding?
Re: A rant about Ruby 1.9 String encoding
#16It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?
Is there a technical reason why Ruby doesn't do this? More likely it's political. Matz (the creator of Ruby), and many of its early contributors, are said not to like Unicode.
That doesn't jive with what I know of Matz. Japan (and much of Asia) adopted Unicode much slower because both UTF-8 and UTF-16 are less efficient than many of the native encodings for representing much of what they need; there was definitely some cultural tone-deafness in the Unicode community when it did Han unification (which may be a good idea, but could have been handled better).
Matz (and the people he works with who use Ruby to get their jobs done) needs access to data that's Not Unicode. Painfully Not Unicode as in it doesn't necessarily round-trip.
Re: A rant about Ruby 1.9 String encoding
#17Earlier quoted context omitted.
The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.") Loss of text data is bad.
That page says there are duplicates in CP-932 because of different vendor variants. If those characters are otherwise entirely identical, calling it a single encoding seems wrong. Wouldn't you just have a CP-932-IBM and a CP-932-NEC encoding?
Because someone unified two similar encodings into CP-932/ShiftJIS. That someone who unified them wasn't IBM or NEC (both of whom made competing systems and had made different encoding choices, but whose choices were mostly compatible).
Rules for dealing with legacy encodings: 1. They make no sense. 2. If you think they make sense, remember that you weren't there so refer to rule 1.
Re: A rant about Ruby 1.9 String encoding
#18It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?
The performance is better if you use an encoding with a fixed byte-length character-set.
Re: A rant about Ruby 1.9 String encoding
#19It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?
The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.") Loss of text data is bad.
General question: Why didn't the UTF-8 boys and girls make it safe in the first place? This doesn't sound like rocket science. "This character maps to that character, this character to that one." I don't understand how we have unicode snowmen, but we can't safely round trip characters.
Re: A rant about Ruby 1.9 String encoding
#20It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?
Java stores all strings in UTF-16. It probably seemed like a good idea at the time but has led to problems later. There is now a huge body of broken code out there which assumes that each 16-bit char value represents a complete and separate Unicode character.