Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A rant about Ruby 1.9 String encoding

github.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: A rant about Ruby 1.9 String encoding

#31

Post by halostatue »

nradov wrote: post #20
alextgordon wrote: post #7

It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?

Java stores all strings in UTF-16. It probably seemed like a good idea at the time but has led to problems later. There is now a huge body of broken code out there which assumes that each 16-bit char value represents a complete and separate Unicode character.

Um. Wasn't Java's original choice UCS-2 (same as NTFS originally)? UTF-16 has surrogate characters and the length will be one or two 16-bit values depending on whether it's a surrogate. UCS-2 was a fixed 16-bit character size, long since deemed a mistake.

Re: A rant about Ruby 1.9 String encoding

#32

Post by ominous_prime »

halostatue wrote: post #29
ominous_prime wrote: post #21

Earlier quoted context omitted.

Python 3 doesn't use an internal encoding, it uses unicode. Python 2 can use unicode strings if you specify them. I'm not familiar enough with ruby to comapare the two, but I don't see a problem with how python handles it. You sometimes need to know about your string encoding, which is just a fact of modern programming. [EDIT] I think I missed my point slightly. Python 3 doesn't change the encoding of strings, it dec…

If a CP932 '\' is interpreted as '¥', but is exported as CP932 '¥', there's data loss. Unless Python 3 keeps the original data around when it converts to Unicode (probably UTF-8 or UTF-16), there will be data loss in those cases. It's unavoidable.

UTF-\d is not unicode. Unicode isn't an encoding, it's the decoded representation of a character encoding.

    [edit] clarified example
    >>> s = u'\xa5' # shiftjis decoding of \
    >>> print s.encode('shiftjis')
    \
    >>> print s.encode('utf-8')
    ¥

Re: A rant about Ruby 1.9 String encoding

#33

Post by wooster »

halostatue wrote: post #8

I know nothing about the author, but there are some statements made that suggest that the author hasn't had to deal with the wild-and-woolly reality of encodings out there in a lot of extant data. One only wishes that all data were UTF-8. What Ruby 1.9 gets absolutely right is that its String implementation is completely encoding agnostic (by which I specifically mean that it doesn't force your data to be encoded in…

All good points. However, there are more issues with dealing with strings in the wild than just round-tripping between encodings.

Many of the issues I've dealt with when data mining involved mixed encodings within the same document, documents labelled with the wrong encoding in the metadata, and documents with no encoding information. There's only so much you can do as far as sniffing character sets and languages to avoid mojibake and other, more subtle, problems.

For my purposes, converting to Unicode and losing round-tripping is only a minor concern, whereas dealing with non-Unicode encodings is often a source of major problems.

So, personally, having worked both in languages that deal with strings by converting them internally to Unicode, and ones that treat them as encoding-tagged byte streams, I definitely favor the ones that deal with them as Unicode. But, my purposes aren't everyone's, and I'm not convinced there's a paradigm that would suit both usage patterns.

Re: A rant about Ruby 1.9 String encoding

#34

Post by nradov »

halostatue wrote: post #31
nradov wrote: post #20

Earlier quoted context omitted.

Java stores all strings in UTF-16. It probably seemed like a good idea at the time but has led to problems later. There is now a huge body of broken code out there which assumes that each 16-bit char value represents a complete and separate Unicode character.

Um. Wasn't Java's original choice UCS-2 (same as NTFS originally)? UTF-16 has surrogate characters and the length will be one or two 16-bit values depending on whether it's a surrogate. UCS-2 was a fixed 16-bit character size, long since deemed a mistake.

You're right. Java originally used UCS-2 and later switched to UTF-16. http://java.sun.com/docs/books/jls/first_edition/html/3.doc.... The problem is that dealing with surrogate pairs is so awkward that most developers don't even bother.

Re: A rant about Ruby 1.9 String encoding

#35

Post by pvg »

halostatue wrote: post #8

I know nothing about the author, but there are some statements made that suggest that the author hasn't had to deal with the wild-and-woolly reality of encodings out there in a lot of extant data. One only wishes that all data were UTF-8. What Ruby 1.9 gets absolutely right is that its String implementation is completely encoding agnostic (by which I specifically mean that it doesn't force your data to be encoded in…

The roundtrip thing is an edge-case that doesn't really justify inflicting the non-deterministic pain on everyone. Python 3 and Java have taken the 'one true internal encoding' path and while hardly free of warts, it's an approach that is practically saner. The alternative is making some people's hell everyone's hell, forever.

Re: A rant about Ruby 1.9 String encoding

#36

Post by papachito »

tenderlove wrote: post #14

Dealing with string encoding is sometimes a PITA. But I think as English speakers, we are usually sheltered from the problem because most programming languages are English centric. I'd like to hear opinions from people who don't speak English. If you think the way 1.8 handles (or doesn't handle) encoding is just fine, try things you typically do, but with a different language. For example: ruby -ryaml -e'p YAML.dump(…

> I'd like to hear opinions from people who don't speak English.

I'm sure people who don't speak English will read that and answer you straight away ;)

Re: A rant about Ruby 1.9 String encoding

#37

Post by harpastum »

This "rant" is just a part of an effort by the author to catalog the behavior of strings in ruby 1.9. You can see the whole project here: http://github.com/candlerb/string19

The main (runnable) documentation file is here: http://github.com/candlerb/string19/blob/master/string19.rb

Re: A rant about Ruby 1.9 String encoding

#38

Post by pmjordan »

DougBTX wrote: post #22
pmjordan wrote: post #13

Earlier quoted context omitted.

I'm not a Ruby-head, so maybe I don't have the necessary context to understand what's going on, but: As far as I'm concerned, the only time I care about encodings is when I'm trying to do I/O. I'll always have to choose an encoding when engaging in I/O, explicitly or implicitly. While I'm shuffling strings around in memory, I expect it to behave like a sequence of unicode code points. If the implementation wants to a…

If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings. My understanding was just the opposite: now that strings are associated with encodings, he can no longer assume that a1 + a2 results in a string with the same encoding as a1 and a2, since a1 and a2 can have different encodings.

The implication is that the programmer must know what is used internally, though, because things will go horribly wrong otherwise. That's a sign of a very leaky abstraction - if you need to know the format and behaviour of the underlying byte stream, you'd probably be better off using the byte stream directly, the abstraction is just introducing uncertainty where it should be isolating from such concerns.

Re: A rant about Ruby 1.9 String encoding

#39

Post by mtarnovan »

Having given up porting a medium-sized Rails application to Ruby 1.9 just last week I though I'd share some of my experience. Ruby 1.9's String implementation is fine, but will require lots of changing to existing code. Rails 2.3.5 is definitely _not_ ready for Ruby 1.9 and UTF-8. For example some regexps in helpers are not encoding aware, rack 1.0.1 which is required by 2.3.5 doesn't play nicely with encodings and many more small and annoying problems (template encoding for example; the app I was porting mostly uses HAML which supports Ruby 1.9's string encodings in the latest versions; but no luck for the few erb templates)

All in all, this is a huge transition which will take a while to propagate through the hole Rails stack.

Re: A rant about Ruby 1.9 String encoding

#40

Post by ochuuzu1 »

wedesoft wrote: post #18
alextgordon wrote: post #7

It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?

The performance is better if you use an encoding with a fixed byte-length character-set.

No it isn't.
Post reply on HN