How Python 3 Should Have Worked (2012)

This misses the point of why Python 3 was invented: Unicode.

Python 2's string handling is broken in the presence of unicode characters, often leading to subtle errors that wouldn't cause exceptions until far away from the place where the error was introduced, and oftentimes didn't produce exceptions at all, just wrong data. Strings were defined as sequences of bytes, and then provided a .decode method to convert them to a unicode object that stores them as a sequence of codepoints. The problem was that a large number of libraries (including all of Aaron's that I've looked at) used str as their internal string type, which meant they were storing a sequence of bytes in an arbitrary encoding but not storing the encoding along with it. If you pass such a library a string in a different encoding, it will happily store it, manipulate it, and concatenate it with other strings. If you pass such a library multiple strings in multiple encodings (like, for example, if you're pulling data from multiple webpages), you will get garbage data that can't be decoded in any codec.

Python 3 changes this so that str stores unicode codepoints and there's a separate 'bytes' type for uninterpreted bytes, and you are supposed to decode your bytes into strings at system boundaries. This is recommended software engineering practice for anyone who builds large systems that have to interact with foreign-language text; however, a large number of Python developers work in English-only environments where anything they receive will automatically be ASCII. They've never tried to track down subtly broken encoding issues; for them, the decode step is extra busywork that seems pointless.

The reason the Python2->3 transition has been so painful is that it involves a whole language ecosystem fixing bugs in their software, but the bugs are subtle enough that the vast majority of people doing the work will never have encountered them.

You can't just use the "from __future__ import python3_unicode" support because this is a change to the semantics of an existing language feature. In Python2, a string is a sequence of bytes. In Python3, a string is a sequence of unicode codepoints. What happens when a Python3 program calls a Python2 library with a string object? Do you try to auto-convert the strings? You can't, really, because strings in Python2 don't specify their encoding; you have no way of knowing which codec the Python2 library meant, because chances are they didn't think about it.

The other major change in Python 3 - iterators everywhere - is similar, and it's a recognition that an increasingly large proportion of the programming ecosystem lives in a world where async operation is important and many concurrent activities may be happening at once. And I'm really glad to see Python willing to take on these challenges even with 5 years of short-term pain, because it shows a commitment to keeping Python relevant for the issues that 21st-century programmers will face. An increasing number of software platforms will have to deal with non-English text; an increasing number will need to handle concurrent, event-based environments. Without these changes Python would basically cede these areas to languages like Go or Javascript that have their unicode story straight and are well-adapted to async programming.

They aren't. And there are automatic tools for converting between them (2to3 and 3to2), along with those __future__ imports that Aaron mentioned: doing "from __future__ import print_function, unicode_literals, absolute_import, division" would give you most of the Python 3 syntax changes in Python 2. The "everything expects bytes" to "everything expects text" change is the biggest hurdle for a lot of projects.

Let's assume that you're right, that the main reason for Python 3 was Unicode. I've dealt with Unicode both with and without language support (in Python and elsewhere), and this is a pretty important feature. I think that the world in general, and the Python world in particular, is better off with Unicode support in the built-in str class.

That said, other languages -- most recently and notably, Ruby -- have managed to make the Unicode transition without leaving much of the existing user base behind. I realize that Python is often used in larger and slower-moving organizations than Ruby, and also has a more conservative philosophy. Even so, perhaps they could have announced Python 2.8, identical to 2.7 in every way except that strings are now Unicode. Then, after 1-2 years of everyone getting their Unicode house in order, we could have moved onto 2.9 or 3.0, and/or used the "from future" syntax.

There is no perfect solution, but it seems to me that by breaking so many things at once, we're in a situation where everyone wants to upgrade, but no one sees the overwhelming benefit from doing so, and thus puts it off even further.

Ok but... you could do unicode in python 2, it just wasn't ideal. The problem was, python 3 doesn't actually solve most peoples actual day to day problems.

Here are real problems with python:

* It's slow (excluding pypy)

* The C interface sucks (compared to something like Lua) and holds back language progress

* It can't handle multicore well outside of multiprocess hacks (which are sold as "the right way" -- bullshit. Sometimes threads are useful).

* Lambdas/closures are unnecessarily limited (I don't buy the whitespace/syntax argument -- look at how Boo works. You can do this just fine while keeping it pythonic).

* Explicit "self" is stupid and most people hate it. Javascript and Ruby are comparable languages, and neither of them need this while still having the exact same flexibility as python.

* (Down somewhere near the bottom:) strings should probably be unicode by default.

Python 3 doesn't solve any of the first five major problems, and the last problem can be worked around in python 2.

You've correctly identified problems with python 2, but I think you're incorrectly giving them more weight than they deserve. Most people just don't run into those issues, and don't care, and that's why python 3 is dead in the water -- because it doesn't solve the real pain points of python enough to make people want to upgrade.

Insofar as C interfaces go, look at cffi.