Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

151–160 of 230 posts

Re: The string type is broken

#151

Post by wbond »

lelf wrote: post #141
wbond wrote: post #137

Earlier quoted context omitted.

Yes, I did, and you did not provide a single example. You just said "“oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!". Taking the time to actually prove your point it useful. However, your recent example seems to be running fine on Python 3.3. You did not include any version info in your example output. Python 3.3.0 (default, Mar 11 2013, 00:32:12) [GCC 4.7.2] o…

3.3.2. No, it is not. Use 'o\u0308o\u0327'

Oh, I see the issue here. You are expecting the string class to function via graphemes rather than characters. It should be possible to implement grapheme support since character support is there, but I imagine the reverse is not true.

A little googling turned this up. https://mail.python.org/pipermail/python-ideas/2013-July/021...

Re: The string type is broken

#152

Post by cpeterso »

lelf wrote: post #76
judofyr wrote: post #6

In many languages it's difficult fixing the string type without breaking existing code. In Ruby: String#upcase only handles ASCII (by spec), #length counts codepoints, #reverse reverses codepoints. You can use UnicodeUtils if you need "full" Unicode support: >> UnicodeUtils.upcase("baffle") => "BAFFLE" >> graphemes = UnicodeUtils.each_grapheme("noe\u0308l").to_a >> graphemes.reverse.join => "lëon" >> graphemes.size =>…

> String#upcase only handles ASCII (by spec) Bad for Ruby > You can use UnicodeUtils if you need "full" Unicode support: Oh, sure Betty:~ lelf$ ruby -r unicode_utils/u -e 'puts UnicodeUtils.each_grapheme("A‮͜CB‬D").to_a.reverse.join' D‬BC͜‮A So, "full" (it's not) Unicode support won't help you if you have little idea about what you're doing (like indexing stringه҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈…

Strange. Chrome indents those characters into a >-shaped "flock formation", but Firefox renders them as a vertical column.

Re: The string type is broken

#153

Post by brihat »

This article is mostly written from a European language perspective. For Indian scripts, storing combining characters as a separate code points is the right thing to do.

For example, कि (ki) is composed of क and ि When I'm writing this in an editor, say, I typed ku (कु) instead of ki (कि) and I press backspace, I indeed want to see क rather than deleting the whole "कि".

Re: The string type is broken

#154

Post by kbenson »

AsymetricCom wrote: post #148

Now, look over here! When I substitute this context with that, ka-pow! now it's an array of characters! Big deal. I don't understand what the point of this article is when it shows the shortcomings of half a dozen different string implementations in random languages. Yes, if you don't understand the language, then your assumptions about how it works may be wrong. Big surprise, that doesn't mean every string implement…

Unicode is a standard. It says how to act in these circumstances. Calling out incorrect unicode implementations is useful. You shouldn't have to worry about inconsistent behavior between different languages that purport to support unicode strings. That's the point of a standard.

Re: The string type is broken

#155

Post by jmah »

jbert wrote: post #40

Perl seems to pass nearly all the tests (including uppercasing baffle): $ perl -E 'use utf8; binmode STDOUT, ":utf8"; say uc("baffle");' BAFFLE The only failure I can see is that it treats "no el" as 5 characters (so reports length as 5 and reversing places the accent on the wrong character). That's documented here: http://perldoc.perl.org/perluniintro.html#Handling-Unicode "Note that Perl considers grapheme clusters t…

Also Cocoa's NSString:

  [@"baffle" uppercaseString]; // @"BAFFLE"

Re: The string type is broken

#156

Post by berdario »

fauigerzigerk wrote: post #139
wbond wrote: post #121

Earlier quoted context omitted.

So in your one, specific, performance-limited situation, Python 3's implementation of unicode doesn't work for you. Mostly because you are trying to optimize based on implementation details. I don't see how this equates to a general purpose language failing at strings, especially when the language isn't particularly focused on performance and optimization. And if memory usage is of concern, I would certainly think an…

>I don't see how this equates to a general purpose language failing at strings And I don't see where I said it did. I used to favor a dual Python/C++ strategy, but Python's multithreading limitations and the decisions around unicode have convinced me to move on. It's not like anything has gotten worse in Python 3, it's just that there has been a major change and the opportunity to do the right thing was missed. I hap…

I strongly disagree

and I'd like to know what do you think the "right thing" would be

I agree that only using UTF-8 would be the right thing, but only if you don't want to have "array of codepoints"... the problem is: every language, and every developer expect to be able to have random access to codepoints in their strings...

there're some weird exceptions, like Haskell Data.Text (I think that's due to haskell laziness)

would you prefer to have O(n) indexing and slicing of strings... or you'd prefer to get rid of these operations altogheter?

if the latter, what'd you prefer to do? force the developers to use .find() and handle such things manually... or create some compatibility string type restricted to non composable codepoints?

Getting an implementation out to see it used in the wild might be an interesting endeavor... probably it'd be easier to do in a language that allows you to customize it's reader/parser... like some lisp... clojure

Re: The string type is broken

#157

Post by brihat »

The author intentionally chooses decomposed form. Indeed all of them work with Python 3. Here:

    Python 3.3.2+ (default, Oct  9 2013, 14:50:09) 
    [GCC 4.8.1] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> noel="noël"
    >>> noel[::-1]       # reverse
    'lëon'
    >>> noel[0:3]        # first three characters 
    'noë'
    >>> len(noel)        # length
    4
The point is, defining what is a character based on how it is displayed is flawed. Just precompose the string ifg you want and carry on. Like I said in my other comment, making automatic conversion of decomposed -> precomposed wrecks havoc with Indian languages.

Re: The string type is broken

#158

Post by berdario »

panzi wrote: post #107
pilif wrote: post #95

Earlier quoted context omitted.

In Python 3, you don't care about what they use internally. You don't need to. If you want to work with strings, you work with strings. If you want to work with bytes, you work with bytes. If you want to convert bytes into strings (maybe because it's user input that you want to work with), then you tell Python what encoding these bytes are in and you have it create a string for you. You don't care what Python uses in…

Agreed with most of it except: "because their string API is correct" Apparently they have a bug in their UTF-7 parser that can lead to invalid unicode strings. Don't know if it's already fixed.

It was a bug in the decoding: it raised an unexpected exception, nothing that couldn't be worked around with a check (afaik it didn't crash the interpreter)

and it has been fixed since more than 1 month, just 2 days after it was reported

http://bugs.python.org/issue19279

Let's avoid spreading fud, shall we? :)

Re: The string type is broken

#160

Post by gsg »

DannoHung wrote: post #138

Is it possible that Unicode is actually a bunch of horseshit? If literally nobody gets the spec right, then maybe the spec is wrong.

Unfortunately, general purpose text is not a clean simple thing that you can model nicely. Unicode is a mess because the problem it tries to solve is messy.

Even if you could somehow come up with something obviously better, getting any new standard adopted widely enough to be useful would be a formidable, if not insurmountable, challenge. It's less pain to keep using Unicode and try to deal with the worst of the damage.

Post reply on HN