Strings – Dive Into Python 3
getpython3.com
Strings – Dive Into Python 3
1–10 of 32 posts
Re: Strings – Dive Into Python 3
#2Apparently, all I know about strings is correct.
Re: Strings – Dive Into Python 3
#3Apparently, all I know about strings is correct.
My thought exactly too haha
Re: Strings – Dive Into Python 3
#4TL;DR: think of string as tuple of numbers. some are bytes, some are integers. if you want to transform those numbers into a particular encoding (e.g. UTF-8, CP-1252) then that's a different story.
EDIT: I know the article went all out about character abstraction, that why i said "some are bytes, some are integers"
Re: Strings – Dive Into Python 3
#5TL;DR: think of string as tuple of numbers. some are bytes, some are integers. if you want to transform those numbers into a particular encoding (e.g. UTF-8, CP-1252) then that's a different story. EDIT: I know the article went all out about character abstraction, that why i said "some are bytes, some are integers"
This is entirely the wrong take-away message from this article. The point is that strings are not sequences of numbers, but are, rather sequences of characters. Characters are abstracted from the underlying byte representation which is unimportant when dealing with strings.
For situations where a concrete byte representation is needed, you can get one by encoding the string.
Re: Strings – Dive Into Python 3
#6TL;DR: think of string as tuple of numbers. some are bytes, some are integers. if you want to transform those numbers into a particular encoding (e.g. UTF-8, CP-1252) then that's a different story. EDIT: I know the article went all out about character abstraction, that why i said "some are bytes, some are integers"
This is entirely the wrong take-away message from this article. The point is that strings are not sequences of numbers, but are, rather sequences of characters . Characters are abstracted from the underlying byte representation which is unimportant when dealing with strings. For situations where a concrete byte representation is needed, you can get one by encoding the string.
Even this definition can get hairy, though. What is a character? Is 'á' one character or two? Most human beings would say one, but in actuality I formed it with an 'a' (U+0061) and a combining acute accent (U+0301): Two separate code points. But you can also get the same result with 'á' (U+00E1); this is not true of all combining character combinations.
In the past, I've had to deal with horrible mashups of fixed-byte-length columns in flat text files with UTF-8 bolted onto it. In Java, no less. Trying to figure out how to deal with all the edge cases (how do you truncate a string when the boundary is between a "normal" character and a combining character?) was an endless parade of the bizarre. Strings are hard, fundamentally.
Re: Strings – Dive Into Python 3
#7TL;DR: think of string as tuple of numbers. some are bytes, some are integers. if you want to transform those numbers into a particular encoding (e.g. UTF-8, CP-1252) then that's a different story. EDIT: I know the article went all out about character abstraction, that why i said "some are bytes, some are integers"
This is entirely the wrong take-away message from this article. The point is that strings are not sequences of numbers, but are, rather sequences of characters . Characters are abstracted from the underlying byte representation which is unimportant when dealing with strings. For situations where a concrete byte representation is needed, you can get one by encoding the string.
Indeed. I think Python 3 is very explicit with that distinction as well. You can have either text, which is in Unicode, or you have data which are arbitrary bytes. Sure, those bytes can represent text by interpreting them with a specific encoding, but you have to convert between one and the other explicitly to make it work. A very nice thing after the debacle in Python 2 where bytestrings in UTF-8 locales on Unix-likes happen to almost work in many cases, just to break horribly in other environments.
That being said, there are a lot of inaccuracies and even wrong things in that article, which saddens me.
Re: Strings – Dive Into Python 3
#8TL;DR: think of string as tuple of numbers. some are bytes, some are integers. if you want to transform those numbers into a particular encoding (e.g. UTF-8, CP-1252) then that's a different story. EDIT: I know the article went all out about character abstraction, that why i said "some are bytes, some are integers"
This is entirely the wrong take-away message from this article. The point is that strings are not sequences of numbers, but are, rather sequences of characters . Characters are abstracted from the underlying byte representation which is unimportant when dealing with strings. For situations where a concrete byte representation is needed, you can get one by encoding the string.
If you think of a string as a sequence of code points (integers) you get a correct but inefficient model (UTF-32).
Character is a context in which we read integers and currently we don't use more than a couple hundred thousand of those.
UTF-8 is a data-compression technique taking advantage of the fact that smaller code points are used more often, and the largest ones (which require 5+ bytes, because any compression algorithm expands some inputs) are, currently, not standardized and effectively never used.
Re: Strings – Dive Into Python 3
#9Earlier quoted context omitted.
This is entirely the wrong take-away message from this article. The point is that strings are not sequences of numbers, but are, rather sequences of characters . Characters are abstracted from the underlying byte representation which is unimportant when dealing with strings. For situations where a concrete byte representation is needed, you can get one by encoding the string.
Even this definition can get hairy, though. What is a character? Is 'á' one character or two? Most human beings would say one, but in actuality I formed it with an 'a' (U+0061) and a combining acute accent (U+0301): Two separate code points. But you can also get the same result with 'á' (U+00E1); this is not true of all combining character combinations. In the past, I've had to deal with horrible mashups of fixed-byt…
Yes. It's much better to think of strings as a sequence of Unicode code points.
Re: Strings – Dive Into Python 3
#10Earlier quoted context omitted.
This is entirely the wrong take-away message from this article. The point is that strings are not sequences of numbers, but are, rather sequences of characters . Characters are abstracted from the underlying byte representation which is unimportant when dealing with strings. For situations where a concrete byte representation is needed, you can get one by encoding the string.
Even this definition can get hairy, though. What is a character? Is 'á' one character or two? Most human beings would say one, but in actuality I formed it with an 'a' (U+0061) and a combining acute accent (U+0301): Two separate code points. But you can also get the same result with 'á' (U+00E1); this is not true of all combining character combinations. In the past, I've had to deal with horrible mashups of fixed-byt…
In that case you “only” have to know what you're actually after: Either a grapheme (a character in the human sense) or a code point (a character in the Unicode sense) – well, and then there is the code unit which can be a “character” in the programming language sense but it's best not to go there, lest you want to fall into plenty of traps.
As long as you only wander around one of those levels (grapheme, code point, code unit, byte) all is (fairly) easy, but once you deal with multiple levels mistakes almost invariably creep in and you start treating code points as graphemes or code units as code points, etc. Fun source of all kinds of bugs :-)
So yes, text in general is hard. And Han, Hangul and the Japanese scripts are probably among the easiest scripts to support in software :-)