Making Python 3 more attractive

This post seems to conflate language and implementation. IMO, Python 3 the language has tons of improvements and no regressions. The grief on the internet about Python 3 makes me seriously wonder how many people who don't like Python 3 have actually tried it yet. (There are legitimate critiques of Python 3, but they're few on the ground and none are presented here.)

The suggestions in this post are mostly changes to the implementation (i.e., make it go faster), not the language itself. While CPython 2.7 and CPython 3.4 (implementations) surely have interesting implementational differences that don't boil down to just language changes, I'm not aware of them.

You've probably read this over and over, but for me there were 2 reasons why Python 3 wasn't a top priority: 1) lack of support in some libraries --- this got a lot better; 2) lack of support on Google App Engine.

On the other hand, I didn't find the new features in Python 3 appealing enough to make me fight the above drawbacks.

Last but not least, while I use most of the tools I've developed in Python on a daily basis, they are just that: tools meant to make my life nicer.

If you need better performance you'd be looking at PyPy instead of CPython. If Python 3 runs faster on PyPy as well that could make a difference.

It's not about a need for new features. It's about basic quality control. I recently ported a medium-sized production system (the back end of "sitetruth.com") from Python 2 to Python 3. It took about a month. Not because of the language changes, but because several major third-party packages were discontinued for Python 3, and their replacements were buggy.

Specifically:

- Python 3 forces you to use CPickle instead of the Python version of Pickle. In some multi-thread/multiprocess situations, CPickle has some memory allocation error, Python's memory becomes corrupted, and things go downhill to a crash. The Python version is fine. I submitted a bug report, but nothing will happen unless I come up with a simple test case, which is hard. Meanwhile I found out how to use the Python version, which works, despite Python 3, and am using that.

- PyMySQL (a "drop in replacement" for MySQLdb) originally didn't implement LOAD DATA LOCAL. When it was implemented, it wasn't tested for large data loads. I kept getting random database disconnects, until I figured out that it was trying to send the entire bulk data load as one 16MB MySQL connection packet. This only works if you configure insanely big buffers in your MySQL server. There's no reason to send a packet that big; LOAD DATA LOCAL will use multiple packets when necessary. It was just a lame default.

- HTML parsing uses different packages under Python 3. The HTML5parser/BS4 combination blows up on some bad HTML, usually involving misplaced items that belong in the HEAD section. The HTML5 parser, obeying the HTML5 spec for tolerating bad HTML, tries to add to the tree being produced at points other than after the last item. BS4 is buggy in that area. I wrote a function to check and fix defective BS4 trees, came up with a simple test case, and submitted a bug report. I have a workaround for now.

- Python 3 finally has TLS support in SSL. (That's also been backported to Python 2.7.9). SSL cert checking is now on by default. It doesn't work for certain sites, including "verisign.com". This is because of a complicated interaction between a cross-signed root certificate Verisign created, a feature of OpenSSL, and how the Python "ssl" package calls OpenSSL. It took weeks of work to get that fixed. Because it's a core Python package, it will remain broken until the next release of Python, 3.5, whenever that happens.

- Running FCGI/WSGI programs from Apache requires a different package than with Python 2. There are 11 different packages and versions of packages for doing this. The Python documentation recommended one that hadn't been updated since 2007, and its SVN repository was gone. There are six forks of it on Github, three of them abandoned. I finally found a derivative version from which much of the unnecessary stuff had been stripped out, and it worked.

- Python's installer program, "pip3", doesn't know which packages work under Python 3, and tried to install a version of one of them that only worked with Python 2.5-2.6. You have to know to install "dnspython3", not "dnspython", for example.

These are all bugs that should have been found by now, and would have if Python 3 had a more substantial user base. We're six years into Python 3. I shouldn't be finding beta-version bugs like these at this late date.

Python's Little Tin God's position on third-party library problems is that it's not Python's problem. His fanboys follow along. (Comment on comp.lang.python: "You have found yet another poorly-maintained package which is not at all the responsibility of Python 3. Why are you discussing it as if Python 3 is at fault?") As a result, PyPi (Python's third-party package list) has no quality control. Perl's CPAN has reviews, testing, and hosts the actual packages. Most of Go's key packages are well-exercised within Google and maintained there. PyPi is just a link farm.

That's why Python 3 isn't getting used. It's not a need for new features. It's that Python 3 doesn't work out of the box. Its supporters are in heavy denial about this.

The way to make Python 3 more attractive is to backport the best bits to Python 2.8...