Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

211–220 of 230 posts

Re: The string type is broken

#211

Post by duncan_bayne »

I'd have hoped Common Lisp would fare well here, but SBCL (1.1.11 on 64-bit Linux Mint 15) is pretty broken. My results:

string: noël, reversed: l̈eon, first 3 chars: noe, length: 5

string: 😸😾, reversed: 😾😸, first 1 char: 😸, length: 2

string: baffle, upcase: BAfflE

string: noël, equals precomposed: NIL

Edited: GNU CLISP 2.49 produces identical results.

Re: The string type is broken

#212

Post by jheriko »

zokier wrote: post #124
jheriko wrote: post #93

Earlier quoted context omitted.

isn't this solving the wrong side of the problem? how about not having to think about such things at all and just accepting that uppercase/lowercase conversion is never going to be language agnostic. thats futureproof and powerful, rather than extra thinking and work...

Most likely case-changes need to be locale-aware, that is true. But still I think minimizing number of locale-specifics is a reasonable goal and in that light I dislike the common usage of turkish i as a example because it is such a obviously fixable (if legacy stuff wasn't concern) flaw in Unicode rather than fundamental issue.

You are right, everything should be as easy as possible. This is a good philosophy for design in general...

Re: The string type is broken

#213

Post by [deleted] »

pilif wrote: post #88
jeltz wrote: post #77

Earlier quoted context omitted.

> Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception) Most non-JVM languages[1] actually use UTF-8 as the internal encoding so they should not suffer from this. Python 3 does not use UTF-16 either, it selects an encoding based on the contents of the string. http://www.python.org/dev/peps/pep-0393/ 1. I think .NET too uses UCS-2 or UTF-16, but I am not a Windows…

.NET uses UCS-2 because the Windows API uses UCS-2 (so when you use Visual Studio out of the box, you will get UCS-2). ECMAScript (JS) uses UCS-2 because that's all there was when the spec was written. Other scripting languages I know for certain are - PHP doesn't care and treats strings as arrays of bytes. All the str functions operate on these byte arrays and thus happily destroy your strings if they are encoded as…

[deleted]

Re: The string type is broken

#214

Post by jeltz »

berdario wrote: post #164
jeltz wrote: post #77

Earlier quoted context omitted.

> Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception) Most non-JVM languages[1] actually use UTF-8 as the internal encoding so they should not suffer from this. Python 3 does not use UTF-16 either, it selects an encoding based on the contents of the string. http://www.python.org/dev/peps/pep-0393/ 1. I think .NET too uses UCS-2 or UTF-16, but I am not a Windows…

Python Ruby >1.8 lets you choose the encoding .NET UCS2/UTF-16 (I know the difference, imho if the stdlib has a .size, .length or .count that works on code units instead of code points it's broken... thus I'll mention only UCS2 from now on) Java UCS2 Clojure UCS2 Scala UCS2 QT UCS2 Haskell String UCS4 Haskell Data.Text UTF-16 (yes, not a naive UCS-2) Rust UCS4 (last time I checked) Javascript UCS2 Dart UCS2 PHP Unico…

Perl, Rust, Go and Vala. I take back the "most" part though. It seems like there are many popular solutions.

Erlang uses Unicode code points and also binaries.

Re: The string type is broken

#215

Post by jeorgun »

I think the specific case of ligatures isn't a failure in strings per se, but a failure in Unicode in that it includes them in the first place. What "fi".upper() (or whatever) should do is kind of ambiguous. The following doesn't really seem appropriate:

  "fi".upper().lower() #=> "fi"
But obviously nor does

   "fi".upper() #=> "fi"
In Turkish (which distinguishes between dotted and dotless 'i'), this issue exists already:

   "ı".upper().lower() #=> "i"
This case couldn't (so far as I know) be fixed by any string library without breaking Unicode compatibility, so it seems slightly disingenuous to call it an issue with strings.

Re: The string type is broken

#217

Post by aroman »

afandian wrote: post #113
antocv wrote: post #89

Earlier quoted context omitted.

Awesome way to exercise the brain.

Interesting. I had no problem reading that (except for 'arsing' which I thought I'd misread). The ability of the brain to pattern-match upside down is amazing.

Fascinatingly, I read your comment first, then tried to read the upside down post — the only word I had trouble with was arsing.

Sight reading is really fascinating.

Re: The string type is broken

#218

Post by ddebernardy »

This is misinformation. OP's strings are just wrong...

    >> "\u0308"
    => "̈"
    >> "\u00eb"
    => "ë"
    >> "noe\u0308l"
    => "noël"
    >> "no\u00ebl"
    => "noël"
His noël examples work just fine if you don't copy/paste the string he posts, and instead type them in like I just did.

If anything, languages are reporting correct reverses and length, since he's really manipulating 5 characters rather than four.

Re: The string type is broken

#219

Post by wazoox »

Tom Christiansen (of Perl fame) made a much, much thorough analysis of Unicode problems in his OSCON 2011 presentation: http://www.oscon.com/oscon2012/public/schedule/detail/24252

Here are the slides: http://training.perl.com/OSCON2011/gbu/gbu.pdf

The site seems down ATM, but Internet Archive has it: https://web.archive.org/web/20121224081332/http://98.245.80....

Re: The string type is broken

#220

Post by deathanatos »

llimllib wrote: post #116
jbert wrote: post #40

Perl seems to pass nearly all the tests (including uppercasing baffle): $ perl -E 'use utf8; binmode STDOUT, ":utf8"; say uc("baffle");' BAFFLE The only failure I can see is that it treats "no el" as 5 characters (so reports length as 5 and reversing places the accent on the wrong character). That's documented here: http://perldoc.perl.org/perluniintro.html#Handling-Unicode "Note that Perl considers grapheme clusters t…

Python 3 gets that one, but python 2.7 doesn't: $ python3 -c 'print("baffle".upper())' BAFFLE $ python -c 'print "baffle".upper()' BAfflE $ python -c 'print u"baffle".upper()' BAϬ„E

It's interesting that you get BAFFLE for the first one. I get the same result in both 3 and 2.

Note first that the reason you get "BAϬ„E" is a bit of garbarge-in garbage-out. Strangely, the interpreter isn't rejecting that with the typical "SyntaxError: Non-ASCII character in file" error; instead, it appears to be assuming ISO-8859-1, and then performing .upper(). You can fix that:

    python2 -c '# coding: utf-8
    print u"baffle".upper()'
(Note, of course, that the #coding needs to match your terminals encoding, which is likely UTF-8, but it isn't guaranteed.)

That, for me, prints "BAfflE" in both Python 2 and 3 (adjusting for 3 by adding parens around print, and removing the u prefix on the literal.) I'm on Python 3.2, so perhaps 3.3 does better. (I'm behind on updates, but last I did update, Gentoo stable was still on 3.2.)

Post reply on HN