Live data from Hacker News

The string type is broken

mortoray.com

21–30 of 230 posts

Re: The string type is broken

#21
post #14

Earlier quoted context omitted.

Seems to work fine on 3.3.3 (Linux) Python 3.3.3 (default, Nov 23 2013, 09:49:26) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> a = "noël" >>> len(a) 4 >>> a[::-1] 'lëon'

This is the test case: https://eval.in/73766 print(len("noe\u0308l"))

Ah, I see, the decomposed case indeed doesn't work as well.

Re: The string type is broken

#22
post #2

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

[edit:] ah, ok, so on python 3 it depends how it's constructed. [originally i had a post here saying i couldn't get the noel to work on 3.3.2]

The test case is decomposed.

Re: The string type is broken

#24

I would argue that it's a unicode problem. `U+0308` shouldn't exist in the first place as a unicode character. That's why we have `U+00EB` ('LATIN SMALL LETTER E WITH DIAERESIS'), etc.

Not all combination of base and combining characters exist in a precomposed form, since a base character can have an infinite number of combining characters tacked onto it.

If anything should not exist, it's U+00EB, which is a convenience, compatibility and (space) optimisation codepoint.

Re: The string type is broken

#25

I would argue that it's a unicode problem. `U+0308` shouldn't exist in the first place as a unicode character. That's why we have `U+00EB` ('LATIN SMALL LETTER E WITH DIAERESIS'), etc.

Uhm, nope. Definitely not. All the precomposed letters only exist because of compatibility with legacy character sets. There are also some languages that routinely use more than one stacked diacritic on letters and encoding every possible precomposed variant would be at least a little bit silly.

Re: The string type is broken

#26

I would argue that it's a unicode problem. `U+0308` shouldn't exist in the first place as a unicode character. That's why we have `U+00EB` ('LATIN SMALL LETTER E WITH DIAERESIS'), etc.

I would argue the opposite. Combining characters are a general (and thus preferable) solution to diacritics, so precombined codepoints should not have been included in Unicode.

Re: The string type is broken

#27
post #13

The problem with text (that Unicode solves only partially) is that text representation, being a representation of human thought, in inherently ambiguous and imprecise. Some examples: (1) A == A but A != Α. The last letter is not uppercase "a", but uppercase "α". Most of the time, the difference is important, but sometimes humans want to ignore it (imagine you can't find an entry in a database since it contains Α that…

I think (2) is an issue with Unicode specifically. They should have specified Turkish alphabet to use ı and a diacritic to make the dotted one. That would have made (in this case) capitalization locale-independent.

Re: The string type is broken

#28
post #23

Do people really need to reverse strings in the real world? I don't think I've ever written code to do that outside of homework assignments and interviews.

The same mechanism that is used for reversing a string can be very useful though.

Think in the lines of python's:

>>> 'abcd'[:-1]

'abc'

Re: The string type is broken

#29
post #19
post #13

The problem with text (that Unicode solves only partially) is that text representation, being a representation of human thought, in inherently ambiguous and imprecise. Some examples: (1) A == A but A != Α. The last letter is not uppercase "a", but uppercase "α". Most of the time, the difference is important, but sometimes humans want to ignore it (imagine you can't find an entry in a database since it contains Α that…

I think your first assertion can be strengthened even further. It isn't like this is unique to letters that look the same. That is, sometimes WORD != WORD. Consider a few common words. Time? As in Time of day? As in how long you have? An interesting combination of the two? Day? As in a marker on the calendar? Just the time when the sun is out? Then we get into names. Imagine the joy of having to find someone named "B…

Except these are all well outside the ambit of what programmers usually think of as text processing, so they won't try to solve them using the same tools.

More to the point, they sound hard, so people won't be so quick to claim they've solved them.

On the other hand, case-insensitive string matching sounds easy, even if it's actually somewhat difficult due to the language dependencies mentioned above, so people will claim to have a general solution that fails the first time it's faced with i up-casing to İ instead of I, or the fact the German 'ß' up-cases to 'SS' as opposed to any single character. (Unicode does contain 'ẞ', a single-character capital 'ß', which occurs in the real world but is vanishingly rare. As far as modern German speakers are concerned, the capital form of 'ß' is 'SS'.)

http://en.wikipedia.org/wiki/Capital_%E1%BA%9E

http://opentype.info/blog/2013/11/18/capital-sharp-s-design-...

http://blogs.msdn.com/b/michkap/archive/2009/07/28/9850675.a...

http://www.personal.psu.edu/ejp10/blogs/gotunicode/2008/07/a...

Re: The string type is broken

#30
post #23

Do people really need to reverse strings in the real world? I don't think I've ever written code to do that outside of homework assignments and interviews.

Had such a case a few months back. Strings of single-byte characters are Endian-agnostic but multi-byte character encoding is affected by Endianness. To cope with it I read the sequence as single byte, then reversed, then changed the encoding to proper encoding and reversed again. The data came from a binary dump where I only needed a section that contained a few strings.

I admit it's dirty but it was throwaway code for an isolated case.

Edit: eh, guys, as I stated the string came from a binary dump. I didn't get to choose the encoding, it came from ROM in an embedded system with a different Endianness. I had to figure out a way to make it human readable.

Post reply on HN