Live data from Hacker News

Using both Python 2 and 3 in Windows

spapas.github.io

31–40 of 44 posts

Re: Using both Python 2 and 3 in Windows

#31
post #10

Earlier quoted context omitted.

I write things in Python 2. No mac comes with Python 3 installed. So, if I want to distribute simple scripts, I can either send a file and say "run this", or I can give a long list of instructions to install Python 3. Similarly, many linux distributions still don't, or only very recently, started distributing Python 3. I don't know of any place which would have Python 3 and not Python 2. So, the question is, why shou…

Only running things that are installed by default is an odd concept. Sure, it's an extra step and you have a "similar enough" language sitting in the base OS. But I never stick to vi because vim isn't installed and I don't stick to sh because bash isn't installed etc; Sometimes the improvements are worth-while. But the need for "upgrade" is not arbitrary, I wouldn't claim that you need to upgrade to maintain support…

Installing python 3 on my machine is easy.

Now if I distribute a piece of software to 10,000 users, I need to explain to all of them how to install Python 3. That is much more painful.

Re: Using both Python 2 and 3 in Windows

#32
post #2

You know what? I appreciate the authors drill-down into the "how" of this issue but I'd like to focus on the "why". Why, has it taken so long to get off of python2.x- noted I understand that a few libraries cough twisted cough have been holding projects back that depend on them. But I haven't wrote a single block of code in half a decade that wasn't python 2.x and python 3.x compatible. At what point will we exclusiv…

The real reason is that the Python community has been TOO NICE. NodeJS and Ruby broke everything. What did they do ? They said "move your butt" and gave a small windows of time. Everybody moved. Python ? We gave 10 years. Created tools. Tutorials. A way to write Python 2/3 compatible programs. Then gave 5 more years. The result ? Nobody moved for 9 years and spread FUD on how hard it is. But the reality is: most peop…

I have a Python project with 28KLOC of Python, 22KLOC for a Python/C extension, and 26KLOC for the tests, so about 4x larger.

It took me about 6-8 weeks to port.

Part of it was because I had to change the API. I had an API with a method like "X.to_string(fmt)" where the format specifier could be something like 'csv' for text and 'csv.gz' for gzip-compressed text.

I had to split that out into a "X.to_string()" and "X.to_bytes()" variants.

I also deal with formats which almost always only use ASCII but where people end up putting in Latin-1 or UTF-8. The API was something like "X.get_field(name)", where name was a byte string, and it returned a byte string.

(GIGO and it's your responsibility to deal with your byte strings.)

In Python 3, X had to acquire an "encoding" and "errors" parameter, so it could know the expected encoding for the entire record. But to make it work, in practice each field could have a different encoding, so I ended up also adding a "X.get_field_as_bytes(name)". The 'name' can be a Unicode or byte string.

These changes to return a Unicode string from a byte string ended up adding a lot of code of the form:

  try:
    field = field_bytes.decode(self.encoding, self.encoding_errors)
  except UnicodeDecodeError as error:
    die("Cannot decode field %r (%r): %s" %
        (field_name, field_bytes, error)
All of these new branches then meant adding unit tests to ensure coverage.

Parts of the code supported passing in a byte string, Unicode string containing only ASCII characters, or buffer object. There's no easy way to handle that in the API, so I eventually dug into Python's own source code to copy how Modules/binascii.c supports it.

I also had to switch all my code to use the new buffer API. Only to find I missed a PyBuffer_Release(), which caused a slow memory leak.

Then the performance in Python 3 was slower than Python 2, so I pushed more of the Python-level code into a C extension.

I'm not convinced the new API is all that much better. It's certainly harder to implement and therefore maintain, and I don't have all that many users.

But I can say that "you can convert that in 2 days" feels like an optimistic statement. The test suite contains over 2000 byte string constants, and even at 5 changes per minute that one change alone would take 6 hours.

Admittedly, part of the reason it took 6-8 weeks was because the result works under both Python 2.7 and Python 3.5+. It is not so easy to make compatible Python/C extensions. But as literally all of my users were using Python 2.x, I wasn't going to drop support for at least Python 2.7.

(I learned a few months ago that one potential customer was still using Python 2.6 on their compute cluster. They have since upgraded to Python 2.7. I warned them about 2020.)

Re: Using both Python 2 and 3 in Windows

#33

Earlier quoted context omitted.

Don't know how updated http://py3readiness.org/ is but according to it 346 of the top 360 libs support py3. Interesting missing libs for me is supervisor and ansible.

Ansible does support python 3: $ python Python 3.6.3 (default, Oct 6 2017, 08:44:35) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import ansible As of supervisor, nobody embeds it in a python program. It's something you apt-get install or yum install, not pip install. So it doesn't matter.

It didn't work for me when I tried it out; there was key functionality broken for various tasks I was using. I'm sure it will be fixed eventually, but there's still some work to be done IME.

Re: Using both Python 2 and 3 in Windows

#34
My approach to using both py2 and py3 in windows: Use 2 folders of WinPython (completely portable) for py2 and py3. A shell script which changes all the necessary env vars from 2->3, 3->2. Just double-click on the shell script when you want to switch and you're done.

Re: Using both Python 2 and 3 in Windows

#35
post #34

My approach to using both py2 and py3 in windows: Use 2 folders of WinPython (completely portable) for py2 and py3. A shell script which changes all the necessary env vars from 2->3, 3->2. Just double-click on the shell script when you want to switch and you're done.

This was my earlier approach before I moved to using 3 for most things. At that time, I tried moving env vars to 3->2 and using the Windows Launcher and haven’t looked back. It makes everything work so much better. This article actually does a fairly insufficient job of explaining it. The official docs on the Windows Launcher are worth working through. Being able to use #! in my scripts and work seamlessly with ‘py’ commands in terminal is superior in every way.

Re: Using both Python 2 and 3 in Windows

#36

Earlier quoted context omitted.

Don't know how updated http://py3readiness.org/ is but according to it 346 of the top 360 libs support py3. Interesting missing libs for me is supervisor and ansible.

Ansible does support python 3: $ python Python 3.6.3 (default, Oct 6 2017, 08:44:35) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import ansible As of supervisor, nobody embeds it in a python program. It's something you apt-get install or yum install, not pip install. So it doesn't matter.

> As of supervisor, ... it doesn't matter.

General sentiment I see on the internet for people asking for assistance. Normally followed with "why don't you write a systemd service for it?"

Re: Using both Python 2 and 3 in Windows

#37
post #10

Earlier quoted context omitted.

Only running things that are installed by default is an odd concept. Sure, it's an extra step and you have a "similar enough" language sitting in the base OS. But I never stick to vi because vim isn't installed and I don't stick to sh because bash isn't installed etc; Sometimes the improvements are worth-while. But the need for "upgrade" is not arbitrary, I wouldn't claim that you need to upgrade to maintain support…

Installing python 3 on my machine is easy. Now if I distribute a piece of software to 10,000 users, I need to explain to all of them how to install Python 3. That is much more painful.

Don't people generally just use some sort of package manager to distribute scripts and their dependencies? I don't use Mac so maybe they're way behind the curve in this.

Re: Using both Python 2 and 3 in Windows

#38
post #32

Earlier quoted context omitted.

The real reason is that the Python community has been TOO NICE. NodeJS and Ruby broke everything. What did they do ? They said "move your butt" and gave a small windows of time. Everybody moved. Python ? We gave 10 years. Created tools. Tutorials. A way to write Python 2/3 compatible programs. Then gave 5 more years. The result ? Nobody moved for 9 years and spread FUD on how hard it is. But the reality is: most peop…

I have a Python project with 28KLOC of Python, 22KLOC for a Python/C extension, and 26KLOC for the tests, so about 4x larger. It took me about 6-8 weeks to port. Part of it was because I had to change the API. I had an API with a method like "X.to_string(fmt)" where the format specifier could be something like 'csv' for text and 'csv.gz' for gzip-compressed text. I had to split that out into a "X.to_string()" and "X.…

You took my "port 20K LOC python code" to "take 70K LOC of mixed of Python and C and make it compatible with 2 and 3". That's an entirely different story, and is definitely the average project.

Re: Using both Python 2 and 3 in Windows

#39

Earlier quoted context omitted.

Ansible does support python 3: $ python Python 3.6.3 (default, Oct 6 2017, 08:44:35) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import ansible As of supervisor, nobody embeds it in a python program. It's something you apt-get install or yum install, not pip install. So it doesn't matter.

> As of supervisor, ... it doesn't matter. General sentiment I see on the internet for people asking for assistance. Normally followed with "why don't you write a systemd service for it?"

Well, systemd was useful when you had 1000 start up systems on linux if you wanted the same easy one for all.

But indeed, now with systemd becoming ubiquitous, it's a better solution unless you target also windows.

But honestly, when I need something to handle my process, I just attach them to my crossbar.io instance since I have it running anyway. This way it's crossplatform and easy my deployment.

Re: Using both Python 2 and 3 in Windows

#40
post #37

Earlier quoted context omitted.

Installing python 3 on my machine is easy. Now if I distribute a piece of software to 10,000 users, I need to explain to all of them how to install Python 3. That is much more painful.

Don't people generally just use some sort of package manager to distribute scripts and their dependencies ? I don't use Mac so maybe they're way behind the curve in this.

Most Linux package managers need root, so that doesn't help if I need to help a random user of a machine.
Post reply on HN