Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A rant about Ruby 1.9 String encoding

github.com

Post reply on HN
Print view View original

21–30 of 55 posts

Re: A rant about Ruby 1.9 String encoding

#21

Post by ominous_prime »

halostatue wrote: post #10
AdamN wrote: post #6

It seems like this would be a problem on Python as well, no?

Python "cheats" by converting everything to Unicode internally. It seems like a simple solution, but it's not a solution since not everything can be converted safely back to the original encoding.

Python 3 doesn't use an internal encoding, it uses unicode. Python 2 can use unicode strings if you specify them. I'm not familiar enough with ruby to comapare the two, but I don't see a problem with how python handles it. You sometimes need to know about your string encoding, which is just a fact of modern programming.

[EDIT] I think I missed my point slightly. Python 3 doesn't change the encoding of strings, it decodes them to unicode. You can encode the string back to the original encoding without loss.

Re: A rant about Ruby 1.9 String encoding

#22

Post by DougBTX »

pmjordan wrote: post #13
alextgordon wrote: post #7

It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?

I'm not a Ruby-head, so maybe I don't have the necessary context to understand what's going on, but: As far as I'm concerned, the only time I care about encodings is when I'm trying to do I/O. I'll always have to choose an encoding when engaging in I/O, explicitly or implicitly. While I'm shuffling strings around in memory, I expect it to behave like a sequence of unicode code points. If the implementation wants to a…

If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings.

My understanding was just the opposite: now that strings are associated with encodings, he can no longer assume that a1 + a2 results in a string with the same encoding as a1 and a2, since a1 and a2 can have different encodings.

Re: A rant about Ruby 1.9 String encoding

#23

Post by tolmasky »

pmjordan wrote: post #13
alextgordon wrote: post #7

It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?

I'm not a Ruby-head, so maybe I don't have the necessary context to understand what's going on, but: As far as I'm concerned, the only time I care about encodings is when I'm trying to do I/O. I'll always have to choose an encoding when engaging in I/O, explicitly or implicitly. While I'm shuffling strings around in memory, I expect it to behave like a sequence of unicode code points. If the implementation wants to a…

I also thought the exact same thing, but I know that encoding stuff is so complex that I may be missing something. For what its worth, I believe Cocoa does things this way: a string is an abstract object of unicode code points, when its time to read or write you must choose an encoding.

Re: A rant about Ruby 1.9 String encoding

#24

Post by wooster »

alextgordon wrote: post #7

It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?

> Is there a technical reason why Ruby doesn't do this?

Han Unification. Gory details here: http://en.wikipedia.org/wiki/Han_unification

Re: A rant about Ruby 1.9 String encoding

#25

Post by alextgordon »

halostatue wrote: post #9
alextgordon wrote: post #7

It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?

The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.") Loss of text data is bad.

Are these encodings common enough that they're worth supporting in the basic String class? Surely another class could be provided to handle these edge cases, they don't have to be natively handled.

There's no need for any data loss to occur - the String class would merely not support converting from non round-trippable encodings.

Re: A rant about Ruby 1.9 String encoding

#26

Post by flogic »

3pt14159 wrote: post #19
halostatue wrote: post #9

Earlier quoted context omitted.

The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.") Loss of text data is bad.

General question: Why didn't the UTF-8 boys and girls make it safe in the first place? This doesn't sound like rocket science. "This character maps to that character, this character to that one." I don't understand how we have unicode snowmen, but we can't safely round trip characters.

There isn't a good idea that a standards body hasn't fucked up. Maybe it's because these problems are harder than they seem. Or maybe they're just plain screwed up. Or maybe it's Adobe.

Re: A rant about Ruby 1.9 String encoding

#27

Post by bhiggins »

halostatue wrote: post #9
alextgordon wrote: post #7

It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?

The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.") Loss of text data is bad.

Practically speaking, what data are you talking about? Which characters don't map? The mapping here looks pretty complete, except for "DBCS LEAD BYTE". Is that a problem? Other non-round trip mappings look like it's because different vendors have different meanings for that character. That's not Unicode's fault.

http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOW...

Re: A rant about Ruby 1.9 String encoding

#28

Post by halostatue »

3pt14159 wrote: post #19
halostatue wrote: post #9

Earlier quoted context omitted.

The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.") Loss of text data is bad.

General question: Why didn't the UTF-8 boys and girls make it safe in the first place? This doesn't sound like rocket science. "This character maps to that character, this character to that one." I don't understand how we have unicode snowmen, but we can't safely round trip characters.

Some of it has to do with Han Unification (http://en.wikipedia.org/wiki/Han_unification).

Mostly, though, it's because some of these characters are overloaded. If you've got a Windows system, go into the DOS window and type "chcp 932" (you may need the Japanese language files installed). When you type '\', you'll get '¥' (making "C:\Program Files\" look like "C:¥Program Files¥").

In the systems where what become CP932 were first used, the backslash wasn't necessary in Japanese, so that character point was used to encode the yen symbol. Other systems used the backslash, so it was encoded as a different point. When JIS unified the existing Japanese code pages, it couldn't very well go back in time to change all that old data, so it merged the two encodings on many things. So, there's only one Unicode codepoint for the yen glyph ¥, but in this one encoding there's two different characters for it.

This is the most blatant example of a problem with Unicode transcoding, but as far as I know, it's not the only one.

See http://www.mail-archive.com/linux-utf8@nl.linux.org/msg02337... for what could be done, but probably won't.

Re: A rant about Ruby 1.9 String encoding

#29

Post by halostatue »

ominous_prime wrote: post #21
halostatue wrote: post #10

Earlier quoted context omitted.

Python "cheats" by converting everything to Unicode internally. It seems like a simple solution, but it's not a solution since not everything can be converted safely back to the original encoding.

Python 3 doesn't use an internal encoding, it uses unicode. Python 2 can use unicode strings if you specify them. I'm not familiar enough with ruby to comapare the two, but I don't see a problem with how python handles it. You sometimes need to know about your string encoding, which is just a fact of modern programming. [EDIT] I think I missed my point slightly. Python 3 doesn't change the encoding of strings, it dec…

If a CP932 '\' is interpreted as '¥', but is exported as CP932 '¥', there's data loss. Unless Python 3 keeps the original data around when it converts to Unicode (probably UTF-8 or UTF-16), there will be data loss in those cases. It's unavoidable.

Re: A rant about Ruby 1.9 String encoding

#30

Post by halostatue »

bhiggins wrote: post #27
halostatue wrote: post #9

Earlier quoted context omitted.

The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.") Loss of text data is bad.

Practically speaking, what data are you talking about? Which characters don't map? The mapping here looks pretty complete, except for "DBCS LEAD BYTE". Is that a problem? Other non-round trip mappings look like it's because different vendors have different meanings for that character. That's not Unicode's fault. http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOW...

If there's ~398 byte combinations that can be translated only one way, but not the other, you're still losing data. (The big one that always gets pointed out? '\' and '¥'.)
Post reply on HN