Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

111–120 of 230 posts

Re: The string type is broken

#111

Post by Dewie »

rayiner wrote: post #56

This is why the U.S. dominates the software world. Back when everyone was figuring out how to express their languages, we had the option to punt on complexity and just use ASCII.

It seems rather that it is the other way around - the US dominated (and still does) the computer industry, and so ASCII, the English-centered character set, became the standard. ASCII is good enough (you might lose some accents on certain characters in certain words and such, but nothing much) for English but has no consideration for any other characters that might be used in other languages.

If Turkey was the dominant country in IT, I don't see why they wouldn't do the same thing only for their own alphabet; include all the characters of their alphabet (latin alphabet plus a few more), plus some more common characters used in math etc.

Re: The string type is broken

#112

Post by masklinn »

panzi wrote: post #105
kvb wrote: post #100

Earlier quoted context omitted.

.NET uses 16-bit characters, but you can use the System.Globalization.StringInfo class to iterate through a string one Unicode character at a time, index into strings by Unicode character, etc. The API's a bit awkward, but it works.

One unicode character at a time or one unicode codepoint at a time? (see character composition)

Codepoint. Java 5 also added new string APIs for this.

IIRC, Cocoa is one of the very few frameworks/languages/whatever which provides APIs for manipulating and iterating on grapheme clusters out of the box. And provides a page explaining some of the unicode concepts and how they map to NSString: https://developer.apple.com/library/mac/documentation/Cocoa/...

Re: The string type is broken

#113

Post by afandian »

antocv wrote: post #89
edent wrote: post #33

˙ƃuᴉuɐǝɯ ⅋ 'spɹoʍ 'sɥdʎlƃ 'sɹǝʇɔɐɹɐɥɔ uǝǝʍʇǝq 'ɹǝʌǝʍoɥ 'ǝɔuǝɹǝɟɟᴉp ɐ sᴉ ǝɹǝɥ┴ ˙ʇxǝʇ ɥʇᴉʍ punoɹɐ ƃuᴉsɹɐ oʇ sǝɯoɔ ʇᴉ uǝɥʍ sǝᴉʇᴉlᴉqᴉssod ƃuᴉʇsǝɹǝʇuᴉ ǝɯos sɹǝɟɟo ǝpoɔᴉu∩

Awesome way to exercise the brain.

Interesting. I had no problem reading that (except for 'arsing' which I thought I'd misread). The ability of the brain to pattern-match upside down is amazing.

Re: The string type is broken

#114

Post by drdaeman »

riffraff wrote: post #80
agravier wrote: post #39

Earlier quoted context omitted.

I agree, it seems like a much saner thing to do. Now that you make me think of that, I do not know many instances of this. I just could think of https://github.com/clojure-numerics/core.matrix upon which I stumbled recently. Do you have other example of efforts to separate a type from its implementations?

doesn't every statically typed imperative language do this, and recommend it?

Not to my knowledge.

C++ and Java are statically typed and they, as far as I know, don't have distinction between string interface and implementation, just a standard string type. You can't make your own string implementation and make others (given that - would it exist - they use standard string interface) transparently accept them instead of language's standard string implementation.

Even Haskell (with standard Prelude) doesn't have a readily available and widely accepted typeclass for strings. As String is just an alias to [Char], if library writer used that, they won't accept, say, Data.Text (I know, it's a bit distinct thing, but...)

Re: The string type is broken

#115

Post by wbond »

lelf wrote: post #108
pilif wrote: post #54

Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4…

> Python 3 gets so much of this right What does it gets right????? It's all broken as nearly everything else! It's sad 99% comments there are “oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!” The reality is there's 1-2 languages that are trying to make it correct from the beginning (perl6, I'm looking at you). It's 2013 and if language can compose bytes to code p…

Can you provide some examples of Python 3 getting strings wrong?

Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition (http://docs.python.org/3.3/library/unicodedata.html) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on http://docs.python.org/3.3/howto/regex.html.

If you want to compare strings you should really normalize them first, which is where unicodedata comes in. In my programming situations it would be wrong to conflate different decomposition of the same unicode string. Why is this? Because other software you interact with uses encodings and the UTF-8 encoding of two different decompositions if different. I've run into this with UTF-8 filenames on OS X when working with Subversion.

Re: The string type is broken

#116

Post by llimllib »

jbert wrote: post #40

Perl seems to pass nearly all the tests (including uppercasing baffle): $ perl -E 'use utf8; binmode STDOUT, ":utf8"; say uc("baffle");' BAFFLE The only failure I can see is that it treats "no el" as 5 characters (so reports length as 5 and reversing places the accent on the wrong character). That's documented here: http://perldoc.perl.org/perluniintro.html#Handling-Unicode "Note that Perl considers grapheme clusters t…

Python 3 gets that one, but python 2.7 doesn't:

    $ python3 -c 'print("baffle".upper())'
    BAFFLE
    $ python -c 'print "baffle".upper()'
    BAfflE
    $ python -c 'print u"baffle".upper()'
    BAϬ„E

Re: The string type is broken

#117

Post by wsc981 »

exDM69 wrote: post #92
pilif wrote: post #54

Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4…

> Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). One of the biggest things that I feel Python gets right with the string type is that strings are immutable . It makes a lot of things easier. It really makes sense to have a good string type for small strings, stored in unicode. Immutability makes everything s…

Isn't the string type immutable in many (most?) other languages as well? In Objective-C the default is an immutable string (though optionally

one can create mutable strings as well). Lua also uses immutable strings. In Java and C# I think the situation is the same, since if you want

to use high performance string manipulation, you'll generally resort some form of StringBuilder helper class.

Re: The string type is broken

#118

Post by hrktb »

this_user wrote: post #35

The string type isn't broken. If anything these "X is broken" posts are broken. Taking one special case, finding problems with that case and deducing that the whole concept must therefore be discarded is just silly. Strings work fine for the vast majority of use cases. No technology is free of flaws and engineering decisions are almost always based on weighting the pros and cons and choosing a solution that on balanc…

The point is not to reduce the number of options (everyone going back to arrays of characters) but to put the spotlight on some problems where going a level lower could help a lot.

> *Strings work fine for the vast majority of use cases

In the CKJ space (a third of the population ?) strings are "broken" in the vast majority of use cases (really, things like what format you should accept for a telephone number). It get exponentially dirty as you try more complex manipulations, and I think these are interesting problems. It helps discussing them from time to time.

Re: The string type is broken

#119

Post by masklinn »

pilif wrote: post #88
jeltz wrote: post #77

Earlier quoted context omitted.

> Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception) Most non-JVM languages[1] actually use UTF-8 as the internal encoding so they should not suffer from this. Python 3 does not use UTF-16 either, it selects an encoding based on the contents of the string. http://www.python.org/dev/peps/pep-0393/ 1. I think .NET too uses UCS-2 or UTF-16, but I am not a Windows…

.NET uses UCS-2 because the Windows API uses UCS-2 (so when you use Visual Studio out of the box, you will get UCS-2). ECMAScript (JS) uses UCS-2 because that's all there was when the spec was written. Other scripting languages I know for certain are - PHP doesn't care and treats strings as arrays of bytes. All the str functions operate on these byte arrays and thus happily destroy your strings if they are encoded as…

> - Python It's Python < 3.3 (the Flexible String Represrntation was introduced in 3.3), there's a byte array type (str in P2, bytes in P3) and a string type (unicode/str), which may be UCS2 ("narrow" builds, the default) or UCS4 ("wide" builds, set by many linux distros)

Re: The string type is broken

#120

Post by notJim »

preek wrote: post #87
jhull wrote: post #84

Earlier quoted context omitted.

what are these characters printing here?

It's an awesome little gadget - looks like one character, but is a really big messy bunch of bytes: "\xD9\x87\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\ x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD 2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\ xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xB F\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\ xBF\x…

Ugh your post is totally breaking the page layout!
Post reply on HN