Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

1–10 of 230 posts

Re: The string type is broken

#2

Post by Pitarou »

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3.

Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

Re: The string type is broken

#3

Post by andrewcooke »

Pitarou wrote: post #2

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

[edit:] ah, ok, so on python 3 it depends how it's constructed.

[originally i had a post here saying i couldn't get the noel to work on 3.3.2]

Re: The string type is broken

#5

Post by unwind »

Pitarou wrote: post #2

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

I also doubt the validity of the upper-casing, it feels like in an internationalization/localization context, converting a string to all upper case is not a valid thing to be doing.

Not all languages (or even characters) have a well-defined upper-case versions of their glyphs.

Even if they all did, I would expect the interpretation by a (human) reader to vary culturally.

Re: The string type is broken

#6

Post by judofyr »

In many languages it's difficult fixing the string type without breaking existing code. In Ruby: String#upcase only handles ASCII (by spec), #length counts codepoints, #reverse reverses codepoints.

You can use UnicodeUtils if you need "full" Unicode support:

    >> UnicodeUtils.upcase("baffle")
    => "BAFFLE"
    >> graphemes = UnicodeUtils.each_grapheme("noe\u0308l").to_a
    >> graphemes.reverse.join
    => "lëon"
    >> graphemes.size
    => 4
    >> graphemes[0, 3]
    => "noë"

Re: The string type is broken

#8

Post by judofyr »

unwind wrote: post #5
Pitarou wrote: post #2

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

I also doubt the validity of the upper-casing, it feels like in an internationalization/localization context, converting a string to all upper case is not a valid thing to be doing. Not all languages (or even characters) have a well-defined upper-case versions of their glyphs. Even if they all did, I would expect the interpretation by a (human) reader to vary culturally.

The Unicode standard includes uppercase rules. If you're already representing strings using Unicode codepoints, why not follow the whole Unicode standard?

EDIT: And yes, different languages can have different uppercasing rules. There's still a standard: https://github.com/lang/unicode_utils/blob/master/data/CaseF...

Re: The string type is broken

#9

Post by shioyama »

judofyr wrote: post #7
shioyama wrote: post #4

Happy to see that ruby (2.0 at least) passes all the tests except the "baffle" one. Edit: sadly, it doesn't.

"noe\u0308l".size # => 5 ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-darwin13.0.0]

Ah, oops. I read the post too quickly, now I see the problem.

Re: The string type is broken

#10

Post by XorNot »

unwind wrote: post #5
Pitarou wrote: post #2

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

I also doubt the validity of the upper-casing, it feels like in an internationalization/localization context, converting a string to all upper case is not a valid thing to be doing. Not all languages (or even characters) have a well-defined upper-case versions of their glyphs. Even if they all did, I would expect the interpretation by a (human) reader to vary culturally.

The usual goal is to apply a consistent transform though, to smooth out interpretation differences - i.e. when looking for command input I either lowercase or uppercase things to smooth over the fact that "yes" "YES" "Yes" are all completely valid ways of saying the same thing with those characters.

If there's only one way of expressing the thing - i.e. a single chinese character - then it would be valid to do nothing. It's just in english "y" and "Y" might change context, but as far as computer input is generally concerned they are the same thing.

Post reply on HN