Earlier quoted context omitted.
what are these characters printing here?
It's an awesome little gadget - looks like one character, but is a really big messy bunch of bytes: "\xD9\x87\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\ x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD 2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\ xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xB F\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\ xBF\x…
The string type is broken
91–100 of 230 posts
Re: The string type is broken
#92Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4…
One of the biggest things that I feel Python gets right with the string type is that strings are immutable. It makes a lot of things easier.
It really makes sense to have a good string type for small strings, stored in unicode. Immutability makes everything simpler.
The string type is not a good fit for handling large amounts of text. There are trade offs for efficiency that have to be made to create a handy string type. It really makes sense to have a separate "bytes" type or some kind of StringBuffer for doing big text operations.
Re: The string type is broken
#93The problem with text (that Unicode solves only partially) is that text representation, being a representation of human thought, in inherently ambiguous and imprecise. Some examples: (1) A == A but A != Α. The last letter is not uppercase "a", but uppercase "α". Most of the time, the difference is important, but sometimes humans want to ignore it (imagine you can't find an entry in a database since it contains Α that…
I think (2) is an issue with Unicode specifically. They should have specified Turkish alphabet to use ı and a diacritic to make the dotted one. That would have made (in this case) capitalization locale-independent.
thats futureproof and powerful, rather than extra thinking and work...
Re: The string type is broken
#94Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4…
I'm not sure what "internally using unicode" means. Pyhon's internal representation of strings has changed a lot. It hasn't even been stable in Python 3. Now they are apparently using an internal representation that varies depending on the "widest" character stored.
The only solution that isn't driving me insane is to use UTF-8 everywhere. The Python 3 unicode situation is actually the main reason why I'm not using Python much these days.
Re: The string type is broken
#95Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4…
>In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go. I'm not sure what "internally using unicode" means. Pyhon's internal representation of strings has changed a lot. It hasn't even been stable in Python 3. Now they are apparently using an internal representation that varies depending on the "widest" character stored. The only solution that isn't driving me ins…
If you want to work with strings, you work with strings. If you want to work with bytes, you work with bytes. If you want to convert bytes into strings (maybe because it's user input that you want to work with), then you tell Python what encoding these bytes are in and you have it create a string for you. You don't care what Python uses internally, because their string API is correct and correctly works on characters.
That noël example of the original article consists of 4 characters in Python 3 which is exactly what you want.
I know that just using UTF-8 everywhere would be cool, but that's not how the world works for various reasons. One is that UTF-8 is a variable length encoding which has some performance issues for some operations (like getting the length of the string. Or finding the n-th character).
UTF-8 also isn't widely used by current operating systems (Mac OS and Windows use UCS-2). It's also not what's used by way too many legacy systems still around.
So as long as the data you work with likely isn't in UTF-8, the encoding and decoding steps will be needed if you want to be correct. Otherwise, you risk mixing strings in different encodings together which is an irrecoverable error (aside of using heuristics based on the language of the content).
Re: The string type is broken
#96Re: The string type is broken
#97Earlier quoted context omitted.
I think your first assertion can be strengthened even further. It isn't like this is unique to letters that look the same. That is, sometimes WORD != WORD. Consider a few common words. Time? As in Time of day? As in how long you have? An interesting combination of the two? Day? As in a marker on the calendar? Just the time when the sun is out? Then we get into names. Imagine the joy of having to find someone named "B…
Except these are all well outside the ambit of what programmers usually think of as text processing, so they won't try to solve them using the same tools. More to the point, they sound hard , so people won't be so quick to claim they've solved them. On the other hand, case-insensitive string matching sounds easy , even if it's actually somewhat difficult due to the language dependencies mentioned above, so people wil…
Re: The string type is broken
#98Re: The string type is broken
#99Logically equivalent doesn't mean equivalent for computers. While you can't define why reverse of “noël“ is “lëon“ by set of rules that computer can follow, computer just can't know.
Re: The string type is broken
#100A nitpick from the article >This spells trouble for languages using UTF-16 encodings (Java, C#, JavaScript). if they were using UTF-16, this wouldn't be a problem as UTF-16 can be used to perfectly well encode code points outside of the BMP (at the cost of losing ability for O(1) access to specific code points of course. If you need to know what the n-th code point is, you have to scan the string until the n-th posit…