Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

131–140 of 230 posts

Re: The string type is broken

#131

Post by asveikau »

TorKlingberg wrote: post #78
XorNot wrote: post #10

Earlier quoted context omitted.

The usual goal is to apply a consistent transform though, to smooth out interpretation differences - i.e. when looking for command input I either lowercase or uppercase things to smooth over the fact that "yes" "YES" "Yes" are all completely valid ways of saying the same thing with those characters. If there's only one way of expressing the thing - i.e. a single chinese character - then it would be valid to do nothin…

For that use case it is better to compare case insensitively with "yes" instead of converting the input to lower case first.

If you can compare case-insensitively then you (or the library you call into) must be aware of case and you face the exact same problems. It's a pretty good thought exercise to attempt writing your own Unicode-aware case insensitive string compare. A lot of people call into libraries for this stuff without realizing how complex the problem gets.

Re: The string type is broken

#132

Post by klrr »

I hope Haskell Prime solves this. In Haskell String is literally a list of characters. This causes some overhead and leads to bad performance. Of course we got Text and for binary data you can use ByteString, but it's a bit of pain compared to having a real string type by default.

Re: The string type is broken

#133

Post by zokier »

ygra wrote: post #86
delinka wrote: post #79

Now let's take the lower case of "BAFFLE" - should we get "baffle" or should the string class/function/wtfe attempt to recognize that a ligature can replace "ffl" and return to us "baffle"? More generally, should the string library ever attempt to replace letter with ligatures? Should this be yet another option? And as I type this, another issue manifests: the spelling correction can't even recognize baffle as a properly…

Uppercasing and lowercasing is inherently lossy. E.g. the German ß becomes SS when uppercased, yet there is no way to know whether SS should be lowercased to ss or ß again. That's a reason why those things should be used, if at all, only as display transformations. Same goes for ligatures, but even those actually shouldn't be applied automatically, depending on the language. E.g. in German ligatures cannot span sylla…

I feel like I should learn German only so that I would be able to comment on the ß issue every time a Unicode thread pops up. From my uninformed point of view it is not really clear if ß should really be handled as a separate character/grapheme, or just as a ligature in rendering phase and stored as 'ss'. Or even if current-day orthography should be held at such a sacrosanct position that it shouldn't be changed to save significant amount of collective effort.

Re: The string type is broken

#134

Post by frezik »

Dewie wrote: post #111
rayiner wrote: post #56

This is why the U.S. dominates the software world. Back when everyone was figuring out how to express their languages, we had the option to punt on complexity and just use ASCII.

It seems rather that it is the other way around - the US dominated (and still does) the computer industry, and so ASCII, the English-centered character set, became the standard. ASCII is good enough (you might lose some accents on certain characters in certain words and such, but nothing much) for English but has no consideration for any other characters that might be used in other languages. If Turkey was the domina…

The OP probably needed to clarify. English having a simple alphabet gave the US a leg up on personal computing compared to the KJC countries. It's only one contributing factor, though.

Re: The string type is broken

#135

Post by SideburnsOfDoom »

wsc981 wrote: post #117
exDM69 wrote: post #92

Earlier quoted context omitted.

> Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). One of the biggest things that I feel Python gets right with the string type is that strings are immutable . It makes a lot of things easier. It really makes sense to have a good string type for small strings, stored in unicode. Immutability makes everything s…

Isn't the string type immutable in many (most?) other languages as well? In Objective-C the default is an immutable string (though optionally one can create mutable strings as well). Lua also uses immutable strings. In Java and C# I think the situation is the same, since if you want to use high performance string manipulation, you'll generally resort some form of StringBuilder helper class.

Correct, C# and .Net have an immutable string class and a mutable StringBuilder helper class.

Re: The string type is broken

#136

Post by rayiner »

adamtj wrote: post #126
rayiner wrote: post #56

This is why the U.S. dominates the software world. Back when everyone was figuring out how to express their languages, we had the option to punt on complexity and just use ASCII.

ASCII doesn't make the U.S. special. ASCII is special because it's from the U.S. Lots of people speak languages that trivially fit in 8 bits with no real "figuring out" to do. Before Unicode, we all had our different codepages or encodings. Including the U.S. The U.S. is pretty central to computing. Because of that, and because ASCII only uses 7 bits, some other 8-bit cultures use it as a subset for their native 8-bi…

Many languages fit into 8 bits, but English is particularly simple in its alphabet. Even many of the European languages that can fit in 8 bits have things like accented characters that complicates things somewhat.

Of course this isn't to say English is simple overall. Just that it's complexities lie elsewhere, and it's simplicities lie in an area that made it particularly simple for early computer systems to process.

Re: The string type is broken

#137

Post by wbond »

lelf wrote: post #128
wbond wrote: post #115

Earlier quoted context omitted.

Can you provide some examples of Python 3 getting strings wrong? Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition ( http://docs.python.org/3.3/library/unicodedata.html ) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on http://docs.python.org/3.3/howto/…

Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”. PS: Python 3.3.2 (default, Nov 27 2013, 20:04:48) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'öo̧'[1:] '̈o̧' And sorry, those new regexes don't even support \X (grapheme matching) Edit: python version

Yes, I did, and you did not provide a single example. You just said "“oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!".

Taking the time to actually prove your point it useful. However, your recent example seems to be running fine on Python 3.3. You did not include any version info in your example output.

    Python 3.3.0 (default, Mar 11 2013, 00:32:12) 
    [GCC 4.7.2] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> "öo̧"[1:]
    'o̧'
    >>>
I haven't run across any situations where Python 3.3 is doing wrong, which is why I am asking for some examples.

Re: The string type is broken

#139

Post by fauigerzigerk »

wbond wrote: post #121
fauigerzigerk wrote: post #109

Earlier quoted context omitted.

>In Python 3, you don't care about what they use internally. You don't need to. I do need to know and I always care. My requirements may be different than those of most others because I write text analysis code and I need to optimize the hell out of every single step. I shiver at the thought that any representation could be chosen for me automatically. Of course, nothing is stopping me from simply using the bytes typ…

So in your one, specific, performance-limited situation, Python 3's implementation of unicode doesn't work for you. Mostly because you are trying to optimize based on implementation details. I don't see how this equates to a general purpose language failing at strings, especially when the language isn't particularly focused on performance and optimization. And if memory usage is of concern, I would certainly think an…

>I don't see how this equates to a general purpose language failing at strings

And I don't see where I said it did.

I used to favor a dual Python/C++ strategy, but Python's multithreading limitations and the decisions around unicode have convinced me to move on. It's not like anything has gotten worse in Python 3, it's just that there has been a major change and the opportunity to do the right thing was missed.

I happen to think that UTF-8 everywhere is the right way to go, not just for my particular requirements, but for all applications, because it reduces overall complexity.

Re: The string type is broken

#140

Post by al2o3cr »

jerf wrote: post #65
jeltz wrote: post #50

Earlier quoted context omitted.

What do you use it for? Unless you have a monospaced font the number of characters do not mean much. So unless you are implementing command line tools or text editors it should not be that common.

I have a database field limited to 100 "characters" [1]. The user sent me a form submission with 150. I need to do something to resolve that. This is incredibly common. Truncation to a defined size is routine. [1]: I'm leaving "characters" undefined here, because no matter what Unicode-aware definition you apply here, you've got trouble.

"I have a database field limited to 100 "characters"."

Well there's your problem right there...

"The user sent me a form submission with 150. I need to do something to resolve that."

Any software that defines "do something" here as "silently discard 1/3rd of the user input" is software I'm going to throw in the trash. If you must have fixed-length fields, surely telling the user "much characters, wow overflow" is better than just chopping the input.

Post reply on HN