Live data from Hacker News

The string type is broken

mortoray.com

91–100 of 230 posts

Re: The string type is broken

#91
post #87
post #84

Earlier quoted context omitted.

what are these characters printing here?

It's an awesome little gadget - looks like one character, but is a really big messy bunch of bytes: "\xD9\x87\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\ x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD 2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\ xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xB F\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\ xBF\x…

funny that an ad comes up when you search that character on google. http://imgur.com/Y5vkbHF

Re: The string type is broken

#92
post #54

Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4…

> Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below).

One of the biggest things that I feel Python gets right with the string type is that strings are immutable. It makes a lot of things easier.

It really makes sense to have a good string type for small strings, stored in unicode. Immutability makes everything simpler.

The string type is not a good fit for handling large amounts of text. There are trade offs for efficiency that have to be made to create a handy string type. It really makes sense to have a separate "bytes" type or some kind of StringBuffer for doing big text operations.

Re: The string type is broken

#93
post #27
post #13

The problem with text (that Unicode solves only partially) is that text representation, being a representation of human thought, in inherently ambiguous and imprecise. Some examples: (1) A == A but A != Α. The last letter is not uppercase "a", but uppercase "α". Most of the time, the difference is important, but sometimes humans want to ignore it (imagine you can't find an entry in a database since it contains Α that…

I think (2) is an issue with Unicode specifically. They should have specified Turkish alphabet to use ı and a diacritic to make the dotted one. That would have made (in this case) capitalization locale-independent.

isn't this solving the wrong side of the problem? how about not having to think about such things at all and just accepting that uppercase/lowercase conversion is never going to be language agnostic.

thats futureproof and powerful, rather than extra thinking and work...

Re: The string type is broken

#94
post #54

Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4…

>In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go.

I'm not sure what "internally using unicode" means. Pyhon's internal representation of strings has changed a lot. It hasn't even been stable in Python 3. Now they are apparently using an internal representation that varies depending on the "widest" character stored.

The only solution that isn't driving me insane is to use UTF-8 everywhere. The Python 3 unicode situation is actually the main reason why I'm not using Python much these days.

Re: The string type is broken

#95
post #54

Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4…

>In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go. I'm not sure what "internally using unicode" means. Pyhon's internal representation of strings has changed a lot. It hasn't even been stable in Python 3. Now they are apparently using an internal representation that varies depending on the "widest" character stored. The only solution that isn't driving me ins…

In Python 3, you don't care about what they use internally. You don't need to.

If you want to work with strings, you work with strings. If you want to work with bytes, you work with bytes. If you want to convert bytes into strings (maybe because it's user input that you want to work with), then you tell Python what encoding these bytes are in and you have it create a string for you. You don't care what Python uses internally, because their string API is correct and correctly works on characters.

That noël example of the original article consists of 4 characters in Python 3 which is exactly what you want.

I know that just using UTF-8 everywhere would be cool, but that's not how the world works for various reasons. One is that UTF-8 is a variable length encoding which has some performance issues for some operations (like getting the length of the string. Or finding the n-th character).

UTF-8 also isn't widely used by current operating systems (Mac OS and Windows use UCS-2). It's also not what's used by way too many legacy systems still around.

So as long as the data you work with likely isn't in UTF-8, the encoding and decoding steps will be needed if you want to be correct. Otherwise, you risk mixing strings in different encodings together which is an irrecoverable error (aside of using heuristics based on the language of the content).

Re: The string type is broken

#96
For Go: the for-range loop iterates 5 times, reversed (manually, using the resulting runes) is l̈eon, utf8.RuneCount is 5. The blog has just recently been talking about text normalization[1] via a library, but it isn't built into the core.

[1] http://blog.golang.org/normalization

Re: The string type is broken

#97
post #29
post #19

Earlier quoted context omitted.

I think your first assertion can be strengthened even further. It isn't like this is unique to letters that look the same. That is, sometimes WORD != WORD. Consider a few common words. Time? As in Time of day? As in how long you have? An interesting combination of the two? Day? As in a marker on the calendar? Just the time when the sun is out? Then we get into names. Imagine the joy of having to find someone named "B…

Except these are all well outside the ambit of what programmers usually think of as text processing, so they won't try to solve them using the same tools. More to the point, they sound hard , so people won't be so quick to claim they've solved them. On the other hand, case-insensitive string matching sounds easy , even if it's actually somewhat difficult due to the language dependencies mentioned above, so people wil…

Right, I do not disagree. I just feel better treating them the same. That is, both are actually easy and reliable so long as you realize you have to make some gross simplifications. And most of the time your life will be much easier if you start with the gross simplifications and try to expand beyond them only when necessary. (This is also why I'm loathe to try programming in unicode...)

Re: The string type is broken

#99

Logically equivalent doesn't mean equivalent for computers. While you can't define why reverse of “noël“ is “lëon“ by set of rules that computer can follow, computer just can't know.

That's why the article says, it's better to have a bare-bytes data structure, than a broken string type.

Re: The string type is broken

#100
post #57

A nitpick from the article >This spells trouble for languages using UTF-16 encodings (Java, C#, JavaScript). if they were using UTF-16, this wouldn't be a problem as UTF-16 can be used to perfectly well encode code points outside of the BMP (at the cost of losing ability for O(1) access to specific code points of course. If you need to know what the n-th code point is, you have to scan the string until the n-th posit…

.NET uses 16-bit characters, but you can use the System.Globalization.StringInfo class to iterate through a string one Unicode character at a time, index into strings by Unicode character, etc. The API's a bit awkward, but it works.
Post reply on HN