Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

181–190 of 230 posts

Re: The string type is broken

#181

Post by revelation »

I seem to have rather little use for the cases the author presents here. If I'm working with strings, they are either of the debug or internal variant, where even basic ASCII would suffice, or I get them from somewhere and don't touch them at all, just pass them around.

But what I absolutely need in a language is to have a very very clear seperation between strings and byte arrays, or raw data, and ideally a way to transform between the two. C# gets this right with its byte and string types, the framework uses them correctly, and there is the wonderful Encoding namespace to interchange the two. Python 2.7 is the absolte worst, it's apparently impossible to get anything done with raw data and not run into some obscure 'ASCII codec can't handle octet 128' whatever exception (reminds you why we have strict typing: magic is fucking annoying).

Re: The string type is broken

#182

Post by wbond »

lelf wrote: post #128
wbond wrote: post #115

Earlier quoted context omitted.

Can you provide some examples of Python 3 getting strings wrong? Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition ( http://docs.python.org/3.3/library/unicodedata.html ) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on http://docs.python.org/3.3/howto/…

Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”. PS: Python 3.3.2 (default, Nov 27 2013, 20:04:48) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'öo̧'[1:] '̈o̧' And sorry, those new regexes don't even support \X (grapheme matching) Edit: python version

You got me curious about grapheme matching in Python with regex. It looks like it is not in the stdlib yet with 3.3. However, it you install https://pypi.python.org/pypi/regex and then replace:

    import re
with

    import regex as re
Then if you want to get into using graphemes slicing, you could use something like:

    import regex as re
    
    decomposed_str = 'o\u0308o\u0327'
    graphemes = re.findall('(\\X)', decomposed_str)
    sub_graphemes = grapheme[1:]
    decomposed_substr = ''.join(sub_graphemes)

Re: The string type is broken

#183

Post by pitterpatter »

berdario wrote: post #164
jeltz wrote: post #77

Earlier quoted context omitted.

> Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception) Most non-JVM languages[1] actually use UTF-8 as the internal encoding so they should not suffer from this. Python 3 does not use UTF-16 either, it selects an encoding based on the contents of the string. http://www.python.org/dev/peps/pep-0393/ 1. I think .NET too uses UCS-2 or UTF-16, but I am not a Windows…

Python Ruby >1.8 lets you choose the encoding .NET UCS2/UTF-16 (I know the difference, imho if the stdlib has a .size, .length or .count that works on code units instead of code points it's broken... thus I'll mention only UCS2 from now on) Java UCS2 Clojure UCS2 Scala UCS2 QT UCS2 Haskell String UCS4 Haskell Data.Text UTF-16 (yes, not a naive UCS-2) Rust UCS4 (last time I checked) Javascript UCS2 Dart UCS2 PHP Unico…

> Rust UCS4 (last time I checked)

Rust chars are 32bit Unicode codepoints. But strings themselves are utf-8. That is the string type, ~str, is basically just ~[u8], a vector of bytes and not ~[char].

`.len()` [O(1)] gives you byte length while `.char_len()` [O(n)] gives you the number of codepoints.

So strings in rust are just vectors of bytes with the invariant that it's valid utf-8.

Re: The string type is broken

#184

Post by masklinn »

berdario wrote: post #177
fauigerzigerk wrote: post #172

Earlier quoted context omitted.

>I agree that only using UTF-8 would be the right thing, but only if you don't want to have "array of codepoints" Then we agree entirely. I want all strings to be UTF-8. Period. What I said about an array of codepoints was that I would create one seperately from the string if I ever had a requirement to access individual code point positions repeatedly in a tight loop. >the problem is: every language, and every devel…

> If by random access you mean constant time access then those developers would be very disappointed to learn that they cannot do that in Java, C#, C++, JavaScript or Python, unless they happen to know that their string cannot possibly contain any characters outside the ASCII or BMP range. Actually, you can in Python... and obviously most developers ignore such issues [citation needed] My point is that most developer…

> Actually, you can in Python...

No, you can only get random access on codepoints which will break text as soon as combining characters are involved. Even if you normalize everything beforehand (which most people don't do) as not all possible combinations have precomposed forms.

Unicode makes random access useless at anything other than destroying text.

> but a good stdlib would obviously help immensely in this regard

Which is extremely rare, and which Python does not have.

Re: The string type is broken

#185

Post by chrisoverzero »

zokier wrote: post #133
ygra wrote: post #86

Earlier quoted context omitted.

Uppercasing and lowercasing is inherently lossy. E.g. the German ß becomes SS when uppercased, yet there is no way to know whether SS should be lowercased to ss or ß again. That's a reason why those things should be used, if at all, only as display transformations. Same goes for ligatures, but even those actually shouldn't be applied automatically, depending on the language. E.g. in German ligatures cannot span sylla…

I feel like I should learn German only so that I would be able to comment on the ß issue every time a Unicode thread pops up. From my uninformed point of view it is not really clear if ß should really be handled as a separate character/grapheme, or just as a ligature in rendering phase and stored as 'ss'. Or even if current-day orthography should be held at such a sacrosanct position that it shouldn't be changed to s…

As "ß" vs. "ss" changes pronunciation of preceding vowels, I can't see how it could be anything other than its own letter.

* "Fuß" ("foot") roughly rhymes with "loose."

* "Fluss" ("river") roughly rhymes with… um, nothing I can think of. It has the vowel sound of "look" and "book," at least as pronounced in the American Northeast.

Since the orthographic reform of 1996, this has become a big deal.

Re: The string type is broken

#186

Post by reflectiv »

meepmorp wrote: post #163
wsc981 wrote: post #117

Earlier quoted context omitted.

Isn't the string type immutable in many (most?) other languages as well? In Objective-C the default is an immutable string (though optionally one can create mutable strings as well). Lua also uses immutable strings. In Java and C# I think the situation is the same, since if you want to use high performance string manipulation, you'll generally resort some form of StringBuilder helper class.

I believe strings are mutable in Ruby.

They are...

    s = "hello"
    s

Re: The string type is broken

#187

Post by AndyKelley »

I don't think the solution to this problem is to make our string classes more complicated. I think it's to make our languages and character sets less complicated. I can't believe that multiple codepoints being used to generate a single glyph made it into the Unicode spec. That breaks a bunch of extremely useful abstractions. I think it is reasonable to expect human languages to be made up of distinct glypths that do not interfere with each other. Any language that does not is too complicated to be worth supporting. Let it die.

Re: The string type is broken

#188

Post by FreeFull »

NkVczPkybiXICG wrote: post #150

All of these examples work in Haskell's canonical text library, 'text'! It's the only language I know of that works.

The reversal of the decomposed noël doesn't produce the right result. Converting baffle to uppercase does do the right thing though, and the rest works as expected.

Re: The string type is broken

#189

Post by rverghes »

Out of curiosity, why only have one string type? We don't do the same for numbers. Many languages don't have "number", they have int, float, long, etc.

Instead of just String, maybe we should have ASCIIString, UTF8String, and UTF16String.

Re: The string type is broken

#190

Post by itsybitsycoder »

baddox wrote: post #175

A big problem here is a lack of clear definitions for various concepts like "character," "reversed string," "upper case," etc. The author briefly recognizes this, but brushes it off with statements like "I generally expect that..." and "I assume most people would not be happy with the current result." I think these hand-wavings aren't helpful. Short of extensive surveying, which is bound to be controversial no matter…

"If I saw that "ffl" ligature, how would I know it's a ligature and not some single unrelated character in another language?"

Because the name of that character is "Latin Small Ligature ffl". Knowing to capitalize ffl as FFL doesn't require a word list any more than knowing to capitalize "ffl" does.

Post reply on HN