Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

11–20 of 230 posts

Re: The string type is broken

#11

Post by DougBTX »

unwind wrote: post #5
Pitarou wrote: post #2

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

I also doubt the validity of the upper-casing, it feels like in an internationalization/localization context, converting a string to all upper case is not a valid thing to be doing. Not all languages (or even characters) have a well-defined upper-case versions of their glyphs. Even if they all did, I would expect the interpretation by a (human) reader to vary culturally.

Here are the Unicode rules, which do consider localization: ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt

It would be interesting to know how often these rules are actually used though...

Re: The string type is broken

#12

Post by mercurial »

andrewcooke wrote: post #3
Pitarou wrote: post #2

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

[edit:] ah, ok, so on python 3 it depends how it's constructed. [originally i had a post here saying i couldn't get the noel to work on 3.3.2]

Seems to work fine on 3.3.3 (Linux)

  Python 3.3.3 (default, Nov 23 2013, 09:49:26)
  [GCC 4.8.2] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> a = "noël"
  >>> len(a)
  4
  >>> a[::-1]
  'lëon'

Re: The string type is broken

#13

Post by tomp »

The problem with text (that Unicode solves only partially) is that text representation, being a representation of human thought, in inherently ambiguous and imprecise.

Some examples:

(1) A == A but A != Α. The last letter is not uppercase "a", but uppercase "α". Most of the time, the difference is important, but sometimes humans want to ignore it (imagine you can't find an entry in a database since it contains Α that looks just like A). Google gives different autocomplete suggestions for A and Α. Is this outcome expected? is it desired?

(2) The Turkish alphabet is mostly the same as the Latin alphabet, except for the letter "i", which exists in two variants: dotless ı and dotted i (as in Latin). For the sake of consistency, this distinction is kept in the upper case as well: dotless I (as in Latin) and dotted İ. We can see that not even the uppercase lowercase transformation is defined for text independently of language.

These are just two examples of problems with text processing that arise even before all the problems with Unicode (combining characters, ligatures, double-width characters, ...) and without considering all the conventions and exceptions that exist in richer (mostly Asian) alphabets.

Re: The string type is broken

#14

Post by judofyr »

mercurial wrote: post #12
andrewcooke wrote: post #3

Earlier quoted context omitted.

[edit:] ah, ok, so on python 3 it depends how it's constructed. [originally i had a post here saying i couldn't get the noel to work on 3.3.2]

Seems to work fine on 3.3.3 (Linux) Python 3.3.3 (default, Nov 23 2013, 09:49:26) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> a = "noël" >>> len(a) 4 >>> a[::-1] 'lëon'

This is the test case: https://eval.in/73766

    print(len("noe\u0308l"))

Re: The string type is broken

#15

Post by masklinn »

I'm not sure I agree with the title, although I do agree with just about all of the content:

* a string type is probably a good idea to bundle the subtleties of unicode, a plain array or list (whether it's of bytes or of codepoints) won't cut it: standard array operations are incorrect/invalid on unicode streams

* the vast majority of string types are broken anyway, as even in the best case they're codepoint arrays (possibly with a smart implementation). The bad cases are just code unit arrays, which break before you even reach fine points of unicode manipulation

And then, you've got the issue that a lot of unicode manipulation is locale-dependent, which most languages either ignore completely or fuck up (or half and half, for extra fun)

Re: The string type is broken

#16

Post by lmm »

I think the mistake here is seeing a string as an extension of an array or vector. What I would prefer is a string type that didn't support all the operations of vectors. The length of a string is not inherently a meaningful question (and for the cases where it is, what you want is something like a vector of grapheme clusters - which is a useful type to have, but not so useful that every string in your program should incur the overhead of creating such a thing); likewise reversing and splitting are operations that simply shouldn't be allowed for your "fast path, undecoded string" type.

Re: The string type is broken

#18

Post by drdaeman »

That's because in a truly sane languages there should be a distinction between data type and its implementation.

Then it would be not "string" type, that's broken, but an implementation of "string" type.

Re: The string type is broken

#19

Post by taeric »

tomp wrote: post #13

The problem with text (that Unicode solves only partially) is that text representation, being a representation of human thought, in inherently ambiguous and imprecise. Some examples: (1) A == A but A != Α. The last letter is not uppercase "a", but uppercase "α". Most of the time, the difference is important, but sometimes humans want to ignore it (imagine you can't find an entry in a database since it contains Α that…

I think your first assertion can be strengthened even further. It isn't like this is unique to letters that look the same. That is, sometimes WORD != WORD. Consider a few common words. Time? As in Time of day? As in how long you have? An interesting combination of the two? Day? As in a marker on the calendar? Just the time when the sun is out? Then we get into names. Imagine the joy of having to find someone named "Brad" that isn't famous. From a city named Atlanta, but not the one in GA. (If you really want some fun, consider the joy that is abbreviations. Dr?)

Re: The string type is broken

#20

Post by ygra »

tomp wrote: post #13

The problem with text (that Unicode solves only partially) is that text representation, being a representation of human thought, in inherently ambiguous and imprecise. Some examples: (1) A == A but A != Α. The last letter is not uppercase "a", but uppercase "α". Most of the time, the difference is important, but sometimes humans want to ignore it (imagine you can't find an entry in a database since it contains Α that…

Homoglyphs vary sometimes with text styles, though. So Α doesn't always have to look like A. Or, more to the point, while T and т might look alike, T and т often do not (the latter of which often looks like m). So even as humans we need to keep track of the script at times.
Post reply on HN