Why I'm Making Python 2.8

To be perfectly clear: bytes (b'') is not a string. Again: bytes is NOT a string. It is an array of octets, aka bytes, aka unsigned 8 bit integers. NOT characters. NOT a string.

If you are dealing with bytes that are encoded representations of a string, then you have to know what encoding they use to decode them and treat them as strings.

I'm pretty sure the people complaining here about how python2 str only supports ascii and they couldn't paste their smartquotes were bitten either by windows or unnecessarily bad unicode/str interactions due to python not just hardcoding utf-8 auto-conversion. That is the only sane thing to do (Your locale isn't *.UTF-8? Well sucks to be you. By now even the Japanese and Chinese seem to slowly have come around to the utf-8 bandwagon, and they had better reasons then most).

I might be wrong, but I can see basically 3 non-idiotic ways to do text in a programming language:

1. arrays of utf-8 bytes (Rust, Go). Python was close to that already and then messed it up. Indexing indexes into bytes O(1).

Upsides:

- efficient: most text you're going to get is already utf-8 and the rest should be converted on ingress/egress; html/css/most code will be represented fairly efficiently even if the body text is mostly say, Chinese; you can do a lot of text processing by just working on the ascii range (e.g. CSV parsing).

- sane: no BOM, no 32 bit encoding of 21 bit quantities etc; unix-compatible

Downsides: - can't efficiently access individual logical characters or know the fixed-font width of the text

- normalization is kinda nasty (concatenation etc.), in practice people just tend to ignore that

- hard to constrain to only valid utf-8 without significant downsides

- maybe not that beginner friendly

2. use some non-array type that doesn't allow for indexing (e.g. ropes), probably using (mostly) utf-8 for internal encoding.

3. arrays of logical characters. That means you need to make up fake characters to handle graphemes that are not directly representable as a single pre-composed code point in unicode. The upside is that this has beginner friendly semantics in a sense and allows indexing on what's meaningful in the domain (graphemes). The downside is that I can't see how to do this with a lot of complexity and some nasty gotchas. This seems to be what perl6 does https://design.perl6.org/S15.html#NFG

Examples of A in this discussion:

- Disagreements about the degree of library support for 3.x

- '' the ease/value of porting from 2 to 3

- '' the rate of industry adoption of 3 for new projects

- '' the degree to which people are driven away from Python entirely because of the version situation

Examples of B in this discussion:

- Claims of paternalism on the part of GVR/ the PSF in pushing 3

- Claims of unreasonable/emotional attachment to 2 by partisan devs

- Claims of willful distortion of facts by both sides (see A)

- Claims that the writing is on the wall for 2, because usage of 3 is supposedly accelerating

- Claims that the writing is on the wall for 3, because it's supposedly taken too long to drive not enough adoption

It seems outrageous to suggest that the differences between 2 and 3 are anywhere near as significant as the differences between, say political conservatism and liberalism, and yet the level of partisanship seems nearly the same. How did it come to be like this?

Unicode in Python 2 was fundamentally broken in that whether it had UTF-16 semantics or UTF-32 semantic depended on how the interpreter was compiled. That's a terrible, terrible idea. However, they could have fixed it by sticking to one option: UTF-16 (which provided compatibility with some interesting things that Python interoperated with like Cocoa and, via Jython, Java).

UTF-16 is a sad legacy mistake, but APIs providing Unicode operations of any kind can be build on top. So UTF-16 is a mistake to begin with, but it's not a blocker for supporting all of Unicode and features targeted at the needs of all writing systems and languages. Java, Windows, the Web Platform (including JS) show that proper i18n can be built on top of the bad but backward-compatible 16-bit code unit foundation.

Now, the _even_ sadder part of Python 3 is that if you decide that UTF-16 is a mistake and want to fix it, UTF-32 is the naive and wrong solution. When a Unicode newbie is told about surrogates, they think that UTF-32 is the answer. But then they waste memory and cache line space (and, if dynamically omitting leading zeros on a per-string basis, the compute and copy cost of promoting to different unit width when adding one emoji). And once the damage is done, someone points out that grapheme clusters are a thing, so they still didn't get O(1) indexing to user-perceived units.

The enlightened thing, of course, is to do what Rust does: use UTF-8 and use iterators on top for accessing pieces larger than a code unit (code point, grapheme cluster). (To my taste, Swift strings are too magic and DWIM-y. At least back when I read the Swift book, it didn't even explain the underlying representation. With Rust, the representation is very explicitly known.)

"UTF-16 sucks" is what Python 3 got right. That UTF-32 (with dynamic leading zero omission on a per-string basis) is the answer is what Python 3 got very, very wrong. The correct answers are either UTF-8 (for a new language like Rust) or holding the nose and making stuff work on top UTF-16 (Java, JavaScript) without breaking old programs.

So then you might have to add code to work around these issues, code that is needed in neither pure py2 or pure py3 -- hence making it a weird detour.

In general, it is more compelling to use `unicode_literals` when back-porting new or existing Python 3 code to Python 2/3 than when porting existing Python 2 code to 2/3 [1]

[1] http://python-future.org/unicode_literals.html