Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

171–180 of 230 posts

Re: The string type is broken

#171

Post by gcr »

For the record, Racket gets the "baffle" example right:

    racket@> (string-upcase "baffle")
    "BAFFLE"
It also passes all of the author's other tests (except for the ones involving combining diacritics, but racket includes built-in functions for normalizing such strings so you can work with them)

Re: The string type is broken

#172

Post by fauigerzigerk »

berdario wrote: post #156
fauigerzigerk wrote: post #139

Earlier quoted context omitted.

>I don't see how this equates to a general purpose language failing at strings And I don't see where I said it did. I used to favor a dual Python/C++ strategy, but Python's multithreading limitations and the decisions around unicode have convinced me to move on. It's not like anything has gotten worse in Python 3, it's just that there has been a major change and the opportunity to do the right thing was missed. I hap…

I strongly disagree and I'd like to know what do you think the "right thing" would be I agree that only using UTF-8 would be the right thing, but only if you don't want to have "array of codepoints"... the problem is: every language, and every developer expect to be able to have random access to codepoints in their strings... there're some weird exceptions, like Haskell Data.Text (I think that's due to haskell lazine…

>I agree that only using UTF-8 would be the right thing, but only if you don't want to have "array of codepoints"

Then we agree entirely. I want all strings to be UTF-8. Period. What I said about an array of codepoints was that I would create one seperately from the string if I ever had a requirement to access individual code point positions repeatedly in a tight loop.

>the problem is: every language, and every developer expect to be able to have random access to codepoints in their strings

If by random access you mean constant time access then those developers would be very disappointed to learn that they cannot do that in Java, C#, C++, JavaScript or Python, unless they happen to know that their string cannot possibly contain any characters outside the ASCII or BMP range.

>would you prefer to have O(n) indexing and slicing of strings

I would leave indexing/slicing operators in place and make sure everyone knows that it works with bytes not codepoints. In addition to that I would provide an O(n) function to access the nth codepoint as part of the standard library.

Re: The string type is broken

#173

Post by sesqu »

rayiner wrote: post #136
adamtj wrote: post #126

Earlier quoted context omitted.

ASCII doesn't make the U.S. special. ASCII is special because it's from the U.S. Lots of people speak languages that trivially fit in 8 bits with no real "figuring out" to do. Before Unicode, we all had our different codepages or encodings. Including the U.S. The U.S. is pretty central to computing. Because of that, and because ASCII only uses 7 bits, some other 8-bit cultures use it as a subset for their native 8-bi…

Many languages fit into 8 bits, but English is particularly simple in its alphabet. Even many of the European languages that can fit in 8 bits have things like accented characters that complicates things somewhat. Of course this isn't to say English is simple overall. Just that it's complexities lie elsewhere, and it's simplicities lie in an area that made it particularly simple for early computer systems to process.

Meanwhile, many of those languages with accented characters have no use for letters like z.

It's not really worth mentioning the alphabet when talking about unique features of English.

Re: The string type is broken

#174

Post by reginaldjcooper »

jerf wrote: post #65
jeltz wrote: post #50

Earlier quoted context omitted.

What do you use it for? Unless you have a monospaced font the number of characters do not mean much. So unless you are implementing command line tools or text editors it should not be that common.

I have a database field limited to 100 "characters" [1]. The user sent me a form submission with 150. I need to do something to resolve that. This is incredibly common. Truncation to a defined size is routine. [1]: I'm leaving "characters" undefined here, because no matter what Unicode-aware definition you apply here, you've got trouble.

This is a good real-world example and the response is an armchair programmer informing you that you are doing it wrong. The internet is rife with know-it-alls. "Just do X." Well, I cannot because I am contractually obligated to write the software as specified and not cowboy up and do whatever I like.

Maybe someone decided 100 characters was a reasonable cutoff and that field is not important enough to reject (read: increase bounce rate) on if someone manages to send too much.

Maybe the 100 characters is a short string generated from an unrestricted long string and cached on a separate server.

Re: The string type is broken

#175

Post by baddox »

A big problem here is a lack of clear definitions for various concepts like "character," "reversed string," "upper case," etc. The author briefly recognizes this, but brushes it off with statements like "I generally expect that..." and "I assume most people would not be happy with the current result."

I think these hand-wavings aren't helpful. Short of extensive surveying, which is bound to be controversial no matter what the result, talking about "general expectations" is a purely subjective notion, and not a good way to evaluate the actions of cold, soulless silicon that is just following orders.

Like the author, I also consider myself a mostly reasonable person, yet is might come up with very different expectations. If I saw that "ffl" ligature, how would I know it's a ligature and not some single unrelated character in another language? You might respond "but it's clearly part of the word 'baffle' and should be capitalized thusly." But would you suggest that string libraries ship with word lists and perform contextual analysis to determine how to perform string operations? Surely that's a fool's errand, not to mention that it would inevitably produce unexpected results.

Re: The string type is broken

#176

Post by jfim »

brihat wrote: post #157

The author intentionally chooses decomposed form. Indeed all of them work with Python 3. Here: Python 3.3.2+ (default, Oct 9 2013, 14:50:09) [GCC 4.8.1] on linux Type "help", "copyright", "credits" or "license" for more information. >>> noel="noël" >>> noel[::-1] # reverse 'lëon' >>> noel[0:3] # first three characters 'noë' >>> len(noel) # length 4 The point is, defining what is a character based on how it is display…

Works as expected too in Scala, although it might be because the terminal does normalization.

scala> val noel = "Noël" noel: String = Noël

scala> noel.reverse res0: String = lëoN

scala> noel.take(3) res1: String = Noë

scala> noel.length res2: Int = 4

scala> import java.text.Normalizer

val nfdNoel = Normalizer.normalize(noel, Normalizer.Form.NFD) import java.text.Normalizer

scala> nfdNoel: String = Noël

scala> nfdNoel.length res3: Int = 5

scala> nfdNoel.reverse res4: String = l̈eoN

scala> nfdNoel.take(3) res5: String = Noe

The problem with an array of characters, as he mentions, is that it doesn't work properly in many use cases. If your array of characters stores 16 bit codepoints, it breaks with the 32 bit codepoints (Java got bit hard by that, where a char used to be a character prior to the introduction of surrogate pairs in Unicode); if it stores 32 bit codepoints, then it's pretty wasteful in most cases, which is exactly why you'd want a string type that handles storage of series of characters in an optimal fashion.

Re: The string type is broken

#177

Post by berdario »

fauigerzigerk wrote: post #172
berdario wrote: post #156

Earlier quoted context omitted.

I strongly disagree and I'd like to know what do you think the "right thing" would be I agree that only using UTF-8 would be the right thing, but only if you don't want to have "array of codepoints"... the problem is: every language, and every developer expect to be able to have random access to codepoints in their strings... there're some weird exceptions, like Haskell Data.Text (I think that's due to haskell lazine…

>I agree that only using UTF-8 would be the right thing, but only if you don't want to have "array of codepoints" Then we agree entirely. I want all strings to be UTF-8. Period. What I said about an array of codepoints was that I would create one seperately from the string if I ever had a requirement to access individual code point positions repeatedly in a tight loop. >the problem is: every language, and every devel…

> If by random access you mean constant time access then those developers would be very disappointed to learn that they cannot do that in Java, C#, C++, JavaScript or Python, unless they happen to know that their string cannot possibly contain any characters outside the ASCII or BMP range.

Actually, you can in Python... and obviously most developers ignore such issues [citation needed]

My point is that most developers don't know these details, a lot of idioms are ingrained... get them to work with string types properly won't be easy (but a good stdlib would obviously help immensely in this regard)

> I would leave indexing/slicing operators in place and make sure everyone knows that it works with bytes not codepoints. In addition to that I would provide an O(n) function to access the nth codepoint as part of the standard library.

Ok, so with your proposal an hypothetical slicing method on a String class in a java-like language would have this signature?

byte[] slice(int start, int end);

I've been fancying the idea of writing a custom String type/protocol for clojure that deals with the shortcoming of Java's strings... I'll probably have a try with your idea as well :)

Re: The string type is broken

#178

Post by Groxx »

meepmorp wrote: post #168
rayiner wrote: post #136

Earlier quoted context omitted.

Many languages fit into 8 bits, but English is particularly simple in its alphabet. Even many of the European languages that can fit in 8 bits have things like accented characters that complicates things somewhat. Of course this isn't to say English is simple overall. Just that it's complexities lie elsewhere, and it's simplicities lie in an area that made it particularly simple for early computer systems to process.

> Even many of the European languages that can fit in 8 bits have things like accented characters that complicates things somewhat. I don't see your point here, with respect to English orthography making computer implementation easier. How exactly does not needing representations for accented characters make anything easier?

It has fewer characters (don't need one for each accent, possibly exceeding 8 bits otherwise) and/or no variable width characters. Also capitalization rules are trivial.

Not that I'm claiming English is unique here, just convenient, and many languages can't claim that.

Re: The string type is broken

#179

Post by implicit »

What we really should be doing is doing away with broken nomenclature.

What does the "length" of a string even mean? A database will tell you it has to do with storage. A nontechnical person will say it's the number of symbols. A visual designer might say that it has to do with onscreen width when rasterized in a particular way. None of these people are obviously right or wrong.

It's very useful to be able to count the number of glyphs in a string, or the number of unicode codepoints, or bytes, or pixels when rasterized in a particular way, but "length" isn't clear enough to unambiguously refer to any of them. Any meaning you try to ascribe to the "length" operation is going to be wrong to someone.

Re: The string type is broken

#180

Post by ygra »

zokier wrote: post #133
ygra wrote: post #86

Earlier quoted context omitted.

Uppercasing and lowercasing is inherently lossy. E.g. the German ß becomes SS when uppercased, yet there is no way to know whether SS should be lowercased to ss or ß again. That's a reason why those things should be used, if at all, only as display transformations. Same goes for ligatures, but even those actually shouldn't be applied automatically, depending on the language. E.g. in German ligatures cannot span sylla…

I feel like I should learn German only so that I would be able to comment on the ß issue every time a Unicode thread pops up. From my uninformed point of view it is not really clear if ß should really be handled as a separate character/grapheme, or just as a ligature in rendering phase and stored as 'ss'. Or even if current-day orthography should be held at such a sacrosanct position that it shouldn't be changed to s…

The point is, while it may have started out as a ligature (of either ſs or ſz, no one really knows for sure), it has long become a letter in its own right. You cannot treat it like a display-only ligature without throwing away information, e.g. the difference between Maße (measurements) and Masse (mass). People in Switzerland made a conscious decision not to use ß anymore, but that's not the case in other countries where the language is used.
Post reply on HN