Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

221–230 of 230 posts

Re: The string type is broken

#221

Post by e12e »

lelf wrote: post #128
wbond wrote: post #115

Earlier quoted context omitted.

Can you provide some examples of Python 3 getting strings wrong? Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition ( http://docs.python.org/3.3/library/unicodedata.html ) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on http://docs.python.org/3.3/howto/…

Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”. PS: Python 3.3.2 (default, Nov 27 2013, 20:04:48) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'öo̧'[1:] '̈o̧' And sorry, those new regexes don't even support \X (grapheme matching) Edit: python version

But what is that sequence (I know the unicode sequence is listed below -- but is it some wierd edge-case)? — because if I manually compose/type those (and a few other characters) everything seems to work fine:

    [edit: Python 3.2.3]
    [edit: [GCC 4.7.2] on linux2]

    >>> 'öo̧'[1:] #copy-paste
    'o̧'
    >>> 'öo̧'[::-1] # "reverse" also breaks
    '̧oö'
    #But for Japanese:
    >>> '日本語'[1:]
    '本語'
    >>> '日本語'[:-1]
    '日本'
    >>> '日本語'[-1:]
    '語'
    >>> '日本語'[::-1]
    '語本日'
    # And Norwegian
    >>> 'æåø'[::-1]
    'øåæ'
    # And a few "French" characters (in this case
    # manually typed as alt+~+e, etc
    >>> 'ẽêèe'[::-1]
    'eèêẽ'
    # And crucially for your example, typed as
    # alt+"+o
    >>> 'öo'[::-1]
    'oö'
So is your initial example some kind of unicode-without-bom(b) or something?

[edit2: I gather, that working with "pre-composed" characters work, and working with "de-composed" ones break. Which, while expected, is a little sad, I agree.]

Re: The string type is broken

#222

Post by enko »

ddebernardy wrote: post #218

This is misinformation. OP's strings are just wrong... >> "\u0308" => "̈" >> "\u00eb" => "ë" >> "noe\u0308l" => "noël" >> "no\u00ebl" => "noël" His noël examples work just fine if you don't copy/paste the string he posts, and instead type them in like I just did. If anything, languages are reporting correct reverses and length, since he's really manipulating 5 characters rather than four.

Congratulations, you've discovered unicode composition!

  2.0.0p247 :045 > Unicode::compose("e\u0308").unpack('U').first.to_s(16)
   => "eb" 
  2.0.0p247 :046 > Unicode::compose("\u00eb").unpack('U').first.to_s(16)
   => "eb" 
  2.0.0p247 :047 > Unicode::decompose("e\u0308").unpack('U').first.to_s(16)
   => "65" 
  2.0.0p247 :048 > Unicode::decompose("\u00eb").unpack('U').first.to_s(16)
   => "65"
I presume the ones you pasted in were changed by the browser. His examples are not wrong at all, indeed how can a string be "wrong"?

Re: The string type is broken

#223

Post by aerique »

duncan_bayne wrote: post #211

I'd have hoped Common Lisp would fare well here, but SBCL (1.1.11 on 64-bit Linux Mint 15) is pretty broken. My results: string: noël, reversed: l̈eon, first 3 chars: noe, length: 5 string: 😸😾, reversed: 😾😸, first 1 char: 😸, length: 2 string: baffle, upcase: BAfflE string: noël, equals precomposed: NIL Edited : GNU CLISP 2.49 produces identical results.

I was somewhat disappointed as well. I wrote some tests here: http://paste.lisp.org/display/140280

Perhaps playing around with different internal representations as pointed out by sedachv (https://news.ycombinator.com/item?id=6811407) would work but the initial, naive string usage doesn't work.

While I expected the default usage to work correctly in Common Lisp.

Re: The string type is broken

#224

Post by unwind »

judofyr wrote: post #8
unwind wrote: post #5

Earlier quoted context omitted.

I also doubt the validity of the upper-casing, it feels like in an internationalization/localization context, converting a string to all upper case is not a valid thing to be doing. Not all languages (or even characters) have a well-defined upper-case versions of their glyphs. Even if they all did, I would expect the interpretation by a (human) reader to vary culturally.

The Unicode standard includes uppercase rules. If you're already representing strings using Unicode codepoints, why not follow the whole Unicode standard? EDIT: And yes, different languages can have different uppercasing rules. There's still a standard: https://github.com/lang/unicode_utils/blob/master/data/CaseF...

Thanks, I was not aware of that.

I guess "uppercase this string" goes from being a tiny loop to a big ... thing based on a lot of hardcoded knowledge, which in turn might indicate that it's not a very simple operation any more.

Re: The string type is broken

#225

Post by kvb »

masklinn wrote: post #112
panzi wrote: post #105

Earlier quoted context omitted.

One unicode character at a time or one unicode codepoint at a time? (see character composition)

Codepoint. Java 5 also added new string APIs for this. IIRC, Cocoa is one of the very few frameworks/languages/whatever which provides APIs for manipulating and iterating on grapheme clusters out of the box. And provides a page explaining some of the unicode concepts and how they map to NSString: https://developer.apple.com/library/mac/documentation/Cocoa/...

This is incorrect - see the definition of a text element in the remarks of http://msdn.microsoft.com/en-us/library/system.globalization....

Re: The string type is broken

#226

Post by judk »

fauigerzigerk wrote: post #172
berdario wrote: post #156

Earlier quoted context omitted.

I strongly disagree and I'd like to know what do you think the "right thing" would be I agree that only using UTF-8 would be the right thing, but only if you don't want to have "array of codepoints"... the problem is: every language, and every developer expect to be able to have random access to codepoints in their strings... there're some weird exceptions, like Haskell Data.Text (I think that's due to haskell lazine…

>I agree that only using UTF-8 would be the right thing, but only if you don't want to have "array of codepoints" Then we agree entirely. I want all strings to be UTF-8. Period. What I said about an array of codepoints was that I would create one seperately from the string if I ever had a requirement to access individual code point positions repeatedly in a tight loop. >the problem is: every language, and every devel…

Sounds like you want to use Go. Feels like Python, but technically correct implementations of concepts.

Re: The string type is broken

#227

Post by masklinn »

kvb wrote: post #225
masklinn wrote: post #112

Earlier quoted context omitted.

Codepoint. Java 5 also added new string APIs for this. IIRC, Cocoa is one of the very few frameworks/languages/whatever which provides APIs for manipulating and iterating on grapheme clusters out of the box. And provides a page explaining some of the unicode concepts and how they map to NSString: https://developer.apple.com/library/mac/documentation/Cocoa/...

This is incorrect - see the definition of a text element in the remarks of http://msdn.microsoft.com/en-us/library/system.globalization... .

Thanks for the info.

Re: The string type is broken

#228

Post by ddebernardy »

enko wrote: post #222
ddebernardy wrote: post #218

This is misinformation. OP's strings are just wrong... >> "\u0308" => "̈" >> "\u00eb" => "ë" >> "noe\u0308l" => "noël" >> "no\u00ebl" => "noël" His noël examples work just fine if you don't copy/paste the string he posts, and instead type them in like I just did. If anything, languages are reporting correct reverses and length, since he's really manipulating 5 characters rather than four.

Congratulations, you've discovered unicode composition! 2.0.0p247 :045 > Unicode::compose("e\u0308").unpack('U').first.to_s(16) => "eb" 2.0.0p247 :046 > Unicode::compose("\u00eb").unpack('U').first.to_s(16) => "eb" 2.0.0p247 :047 > Unicode::decompose("e\u0308").unpack('U').first.to_s(16) => "65" 2.0.0p247 :048 > Unicode::decompose("\u00eb").unpack('U').first.to_s(16) => "65" I presume the ones you pasted in were chan…

His example is "wrong" in the sense that you cannot reasonably complain that "noe¨l" gets reversed to "l¨eon" and put the "¨" part on top of the "l" when it does — which seems entirely correct. Or for that matter, that the string's length is 5 when there are indeed 5 characters.

As for being changed by the browser, the latter (or rather the OS) copied what there was, and the OS pasted it verbatim insofar as I can tell.

Re: The string type is broken

#229

Post by vfclists »

Honestly I think 'you' computer programmers love useless challenges too much. Why can't you adopt lessons from Q?

If it isn't easy to get some languages working with Unicode properly then fix the languages and leave Unicode alone. Remove all the language characteristics that makes working with Unicode difficult. If Unicode will not go to the language then the language must go to Unicode, or opt out of the computer era, or die!!

KISS!!

Re: The string type is broken

#230

Post by llimllib »

deathanatos wrote: post #220
llimllib wrote: post #116

Earlier quoted context omitted.

Python 3 gets that one, but python 2.7 doesn't: $ python3 -c 'print("baffle".upper())' BAFFLE $ python -c 'print "baffle".upper()' BAfflE $ python -c 'print u"baffle".upper()' BAϬ„E

It's interesting that you get BAFFLE for the first one. I get the same result in both 3 and 2. Note first that the reason you get "BAϬ„E" is a bit of garbarge-in garbage-out. Strangely, the interpreter isn't rejecting that with the typical "SyntaxError: Non-ASCII character in file" error; instead, it appears to be assuming ISO-8859-1, and then performing .upper(). You can fix that: python2 -c '# coding: utf-8 print…

Interesting.

I am on python 3.3, but I don't know if it's the updated interpreter that fixes the bug.

Post reply on HN