Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Porting to Python 3 Redux

lucumr.pocoo.org

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Porting to Python 3 Redux

#11

Post by JulianWasTaken »

kmike84 wrote: post #9
JulianWasTaken wrote: post #5

Earlier quoted context omitted.

> There is also an advice for encoding __repr__ and __str__ results to utf8 under Python 2.x in the article; this is fine (other approaches are not better), but it has some non-obvious consequences (like breaking REPL in some setups) that developers should be aware of, see http://kmike.ru/python-with-strings-attached/ I don't see `__repr__` mentioned there, but `__repr__` should basically always be ascii (which a qui…

You're right that __repr__ was not mentioned, my bad. I think `.encode(locale.getpreferredencoding())` is awful because this changes string encoding from run to run, and because locale.getpreferredencoding() could be different (and is different by default e.g. in Cyrillic Windows XP) from both `sys.stdout.encoding` (used for printing) and `sys.getdefaultencoding()` (used for implicit type conversions).

Good point. Honestly I'm careful about calling str on random objects which I know are doing this. But yeah, I guess that's probably a good enough reason to pick an encoding and go with it, which `utf-8` is a good of a choice as any.

Re: Porting to Python 3 Redux

#12

Post by wallunit »

the_mitsuhiko wrote: post #7
wallunit wrote: post #4

> If you have a C module written on top of the Python C API: shoot yourself. There is no tooling available for that yet from what I know and so much stuff changed. I don't agree with that one. I have added Python 3 support two years ago to the Python bindings for libssh2 and it was straight forward. First of all it is still C and therefore you don't have to care about the syntax changed in Python. Just add some #if P…

That again depends on how much you do with strings and integers and how many modules you construct. PyInt is gone, PyUnicode is now PyStr, module construction uses a vastly different system and on 3.x you want to support the stable ABI which looks a bit different.

Except for the module creation you can easily add some very simple compatibility macros. I don't see how that would be different from your _compat module. However module creation can't be abstracted into a uniform macro in fact, because of it requires to define a PyModuleDef struct and the the modlue's init function got a return value in Python 3. But I'm fine with using some #if PY_MAJOR_VERSION >= 3 here.

After all you have to deal with way less compatibility issues, in extension modules than in actual python code. And if needed you can always do a simple version switch. You don't have to care about changes in the syntax of Python. You also don't have to care about changes of the __*__ magic method, because of you don't call them directly, and when defining classes you use slots for stuff like that.

Re: Porting to Python 3 Redux

#13

Post by the_mitsuhiko »

wallunit wrote: post #12
the_mitsuhiko wrote: post #7

Earlier quoted context omitted.

That again depends on how much you do with strings and integers and how many modules you construct. PyInt is gone, PyUnicode is now PyStr, module construction uses a vastly different system and on 3.x you want to support the stable ABI which looks a bit different.

Except for the module creation you can easily add some very simple compatibility macros. I don't see how that would be different from your _compat module. However module creation can't be abstracted into a uniform macro in fact, because of it requires to define a PyModuleDef struct and the the modlue's init function got a return value in Python 3. But I'm fine with using some #if PY_MAJOR_VERSION >= 3 here. After all…

Fair enough. As I said you can probably get around with some macros. To the best of my knowledge no such thing currently exists and what markupsafe does is not particularly nice.

Re: Porting to Python 3 Redux

#14

Post by mynegation »

I also found that maintaining single code base for both 2 and 3 is the only sane way. Running 2to3 during build is just too intrusive.

I liked the approach of 'six', but it is not shipped as a system module. Having something like that as a default system module in Python 2.6, 2.7, and 3.x would go a long way towards adoption of Python 3.

I found that I end up either using six or implementing some subset of it if I do not want to introduce the dependency.

Re: Porting to Python 3 Redux

#15

Post by bdarnell »

This is a good writeup. On Tornado I went through a similar transition from 2to3 to a single codebase. As long as you can drop Python 2.5 support you can probably avoid 2to3, but if you do need it I wrote some tools to make it less painful: http://bdarnell.github.io/blog/2012/03/13/cross-python-devel...

Re: Porting to Python 3 Redux

#16

Post by wiredfool »

wallunit wrote: post #4

> If you have a C module written on top of the Python C API: shoot yourself. There is no tooling available for that yet from what I know and so much stuff changed. I don't agree with that one. I have added Python 3 support two years ago to the Python bindings for libssh2 and it was straight forward. First of all it is still C and therefore you don't have to care about the syntax changed in Python. Just add some #if P…

My experience is that at least with Py 2.6, 2.7, 3.2, and 3.3, it's not all that bad. I'm helping maintain Pillow, a PIL fork, and the commit to add Python 3 support touched a lot of things, but it wasn't that complicated. We've got a py3k.h file that has some ifdefs in it, all the print statements got changed to functions, and there's a few other bits and pieces.

Prior to Pycon, I wasn't really ready for python 3, now I'm missing it in my main codebase (which is on 2.7).

Re: Porting to Python 3 Redux

#17

Post by kmike84 »

the_mitsuhiko wrote: post #10
kmike84 wrote: post #8

Earlier quoted context omitted.

In case of NLTK unicode_literals ("unicode by default") fixed a lot of bugs and made other bugs visible, so mileage may vary :) Could you give an example of cases where unicode_literals is a terrible idea? 3.2 is important for newcomer experience IMHO; it is very common for people starting with Python to use 3.x version and wonder why the code doesn't work. It's a pity high-profile packages are dropping 3.2 support,…

So will latin1 `__str__` on Russian locales. So will ASCII `__str__` on any locale that is not ASCII compatible. You can't expect the impossible. In regards to cases where unicode_literals is a terrible idea is any piece of code that then suddenly gets a unicode string which does not expect it. Because unicode coercion in 2.x spreads like a cancer you might not see the failure until someone uses your API. I still hav…

Are there non-ascii compatible encodings that are default in any OSes? With ascii-incompatible system/terminal encoding a lot of software will stop working. Strange things happen, but this looks like a theoretical issue, and ascii looks safe. In Python 2.x __str__ of all standard container types are ASCII-only (even if elements has non-ascii __str__), and __repr__ of standard objects is also ASCII-only as far as I can tell. ASCII-only is an option, and it is not uncommon and relatively safe (but it has its own issues of course).

It was exactly this unicode_literals property (turning everything into unicode) that helped to reveal bugs :) For example, models were trained on bytestrings under Python 2.x, and nobody remembers what was the encoding of the text models were trained on. This was unnoticed for several years because instead of raising an exception functions just handled some egde cases (e.g. unicode punctuation) in a suboptimal way. This leads to almost correct results, but with less accuracy/precision/recall. After changing to "unicode everywhere" the issue became visible.

The issue was not with cancer-like turning text into unicode, issue was with the code that works with text and doesn't support unicode. Python 2.x standard library has such APIs, and this causes troubles, but I don't see how it is a bug in the code that works with text and returns unicode.

What I'm writing are common words and a standard "unicode mantra", but anyways.

We could say "programmers should just handle encodings properly, and unicode_literals have nothing to do with this", but this doesn't always work. "Unicode everywhere" makes some code changes necessary, but some of these changes reveal real bugs.

Another story: I took 2 different courses from 2 different top-notch universities at coursera.org where instructors gave us starter code (written in Python 2.x) for programming assignments. The code was not bad, but there were many cases of incorrect encodings handling in most of the provided files (such errors that would be impossible in Python 3.x) - this was the code that was supposed to teach students something (including Python programming).

What I like about unicode_literals is that it makes things more consistent and easier (at least for me) to reason about: if variable is unicode under 3.x, it is unicode in 2.x, the same applies to bytestrings. In cases where different behaviour is necessary (e.g. because of non-unicode API in 2.x stdlib), explicit str("foo") is used; otherwise code is written in Python 3.x and works with the same semantics under Python 2.x.

Just curious, what newcomer issues are you talking about, and who do you mean by "newcomers"?

Re: Porting to Python 3 Redux

#18

Post by rogerbinns »

wallunit wrote: post #4

> If you have a C module written on top of the Python C API: shoot yourself. There is no tooling available for that yet from what I know and so much stuff changed. I don't agree with that one. I have added Python 3 support two years ago to the Python bindings for libssh2 and it was straight forward. First of all it is still C and therefore you don't have to care about the syntax changed in Python. Just add some #if P…

My C module is APSW - a python wrapper around SQLite - https://code.google.com/p/apsw/

It supports every version of Python from 2.3 onwards with the exception of 3.0. I provide binaries for Windows and astonishingly people are still downloading the 2.3 version.

As you stated, most of the work is done by feeding the C preprocessor relevant information - https://code.google.com/p/apsw/source/browse/src/pyutil.c

It did take me considerably longer to make my test suite work. This is because I have 99.6% code coverage, and it exercises a lot of edge/error conditions. The test suite code (in Python) is written to run under both Python 2 and 3 as is and has to use some of the similar tricks with exec as the article mentioned. Fun challenges are constructing invalid UTF8 sequences in all Python versions and that sort of thing.

Re: Porting to Python 3 Redux

#19

Post by hoodoof »

So Armin what did not come across from your blog post is how you feel about Python 3.

Until now you have been seen as one of the people holding out strongly against it. Where are you at now? Are you going to move to Python 3?

What's the future for Jinja2 now? Are you enthused about maintaining it? If it's headed for the deadpool please let us know as we can move to technologies that are going to have a future.

What the future for Flask?

What are your current thoughts on Python 3?

Post reply on HN