Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on the Python packaging ecosystem

pradyunsg.me

Post reply on HN
Print view View original

71–80 of 121 posts

Re: Thoughts on the Python packaging ecosystem

#71

Post by jujube3 »

chaxor wrote: post #5

Packaging and Nvidia are pretty much the main reasons python has degraded in status for the past 7 years. It started with the added and unnecessary confusion from conda (and silly things like pythonxy), but has really escalated through extra stupidities like poetry. Much of the features that may be enticing for these extra packages should have pushed the developers to improve the standard tools in python (pip). The o…

Python packaging and versioning has been a shitshow for far more than 7 years. The language has existed since the early 1990s and the philosophy about backwards compatibility and packaging hasn't changed.

Re: Thoughts on the Python packaging ecosystem

#72

Post by jujube3 »

rerdavies wrote: post #46

Asking as a complete newcomer to Python programming... Why doesn't somebody just fix pip? Or is pipenv just understood to be the fix?

Pip alone is not enough.

Python makes backwards incompatible changes to the language all the time, even in dot releases. This means that you can't just install everything into some shared directories since some code would need version X of Python and some would need version Y. So you need virtualenv or something like that.

But virtualenv isn't enough either because most actually useful Python code calls into C libraries (Pandas, Tensorflow, Matplotlib, etc.) And those need to be managed by something like RPM, deb, or even docker.

This is leaving aside implementation issues. For example, many python dependency management tools are incredibly slow and there are some differences of opinion on how to express dependencies.

Re: Thoughts on the Python packaging ecosystem

#73

Post by 411111111111111 »

AlphaSite wrote: post #63
divbzero wrote: post #26

Earlier quoted context omitted.

To me the natural choice would be pip which, as OP points out, has the advantage of shipping with Python by default. We could remedy the shortcomings of pip by incorporating the most successful features from the N alternatives that exist today. The npm / yarn fork that formed in 2016 is the closest analogy I can think of. My impression is that npm improved on its biggest weaknesses ( e.g. a lock file) and has remaine…

Even if pip was a little better I still wouldn’t use it, because Poetry solves the problem so well you’d need to pull it out of my cold hands. It solves not only the install package problem, but locking, venvs, project structure and packaging in a well integrated solution with a fantastic UX. If poetry (or similar) were renamed to Pip and it were to be included in the stdlib id obviously switch. It doesn’t solve the…

Everytime you install a package through poetry, you're also running pip code.

You didn't even understand the point they made: the need for Pipenv and poetry would pretty much go away if pip added support for a proper lockfile and venvs. And that's the only correct choices as pip is already pythons package manager.

Re: Thoughts on the Python packaging ecosystem

#74

Post by Epa095 »

jujube3 wrote: post #72
rerdavies wrote: post #46

Asking as a complete newcomer to Python programming... Why doesn't somebody just fix pip? Or is pipenv just understood to be the fix?

Pip alone is not enough. Python makes backwards incompatible changes to the language all the time, even in dot releases. This means that you can't just install everything into some shared directories since some code would need version X of Python and some would need version Y. So you need virtualenv or something like that. But virtualenv isn't enough either because most actually useful Python code calls into C librar…

I don't understand your paragraph about C libraries. With wheels (which pip can install) there is no problem packing, and linking to, compiled C libraries.

Re: Thoughts on the Python packaging ecosystem

#75

Post by meandmycode »

AlphaSite wrote: post #63
divbzero wrote: post #26

Earlier quoted context omitted.

To me the natural choice would be pip which, as OP points out, has the advantage of shipping with Python by default. We could remedy the shortcomings of pip by incorporating the most successful features from the N alternatives that exist today. The npm / yarn fork that formed in 2016 is the closest analogy I can think of. My impression is that npm improved on its biggest weaknesses ( e.g. a lock file) and has remaine…

Even if pip was a little better I still wouldn’t use it, because Poetry solves the problem so well you’d need to pull it out of my cold hands. It solves not only the install package problem, but locking, venvs, project structure and packaging in a well integrated solution with a fantastic UX. If poetry (or similar) were renamed to Pip and it were to be included in the stdlib id obviously switch. It doesn’t solve the…

Poetry ended up being mostly a disappointment having used it for the last 18 months. First time installs were eventually an hour long to resolve, and lock files were platform specific, making using them for reproducibility almost useless. The wider python ecosystem is frankly painful to productionize even today and it really does need some focus and governance.

Re: Thoughts on the Python packaging ecosystem

#76

Post by at_a_remove »

KyeRussell wrote: post #30
ram_rar wrote: post #15

I get a lot of flak for saying this, but I do hope python committers and psf members take a hard look at perl and its demise. Perl had a very similar problem of having N equivalent choices for doing the same thing and python for better or worse is heading in the same direction. In the last few yrs of my professional career many projects I worked, were on are migrating python -> golang cuz frankly many people were fed…

These days I don’t think I’m being too brash in saying that anyone dealing with Python 2 to Python 3 transition stuff is so deep in the realm of enterprise abandonware that it’s not worth paying it much attention. Not necessarily including you here! But as someone that went through the transition, I’m certainly not going to deny how much of a shit show it was. And I get a feeling that the two situations were/are caus…

It is, sadly, not always Enterprise abandonware.

The industry I am in is extremely conservative about its upgrades and changes. Between that, our custom software interfacing with the 800 pound gorilla for our market, and a few other things, we're behind.

Now, this software depends on a very specific version of ArcGIS. Desktop, not Pro. Not just version 10. Not just version 10.2. But version 10.2.1.

If you go look at ESRI's page for Python and 10.2.1, it says, in large and bold letters (which is something ESRI doesn't do in its documentation very often), NOT TO UPGRADE OR CHANGE PYTHON VERSIONS, EVEN A TINY BIT.

And that version is 2.7. I couldn't even get pip working when I wanted to install a very old version of lxml I wanted.

I guess what I am getting at is, as a language becomes successful and has market penetration, it also seeps down, way far down, in the chain of dependencies. It's the price of success, essentially. I think language maintainers should really pay more attention to that. Just as you know how a program ends up sticking around longer than you originally thought, so too do versions of your language. It is in some ways akin to trying to explain to da Vinci that very far in the future, some of his artworks will adorn clothing and coffee mugs: the good stuff just gets to places you would never dream.

Re: Thoughts on the Python packaging ecosystem

#77

Post by pronlover723 »

asix66 wrote: post #54
pronlover723 wrote: post #49

For me, before I even get to packaging, I hit the install/environment issue with python. Python, by default, wants to be installed at a system level and wants libraries/packages to be at a system level. That shit has to stop. The default needs to be project local installs. Node might have issues but one thing they got right is defaulting to project local installs vs python where you need various incantations to get o…

You can do this now by creating a Python virtual environment[0]. Then you can package your project with a requirements file and some instructions on its use. I use Python VENV often, and it works really well. [0] https://realpython.com/python-virtual-environments-a-primer/

Apparently you missed the point. local needs to be the default. Not some magic incantation

In node

    cd projectFoo
    # do stuff, node uses packages local to projectFoo
    cd ../projectBar
    # do stuff, node uses packages local to projectBar
There is no "activation", the default is it just works.

Installing packages local to the project is also the default.

Re: Thoughts on the Python packaging ecosystem

#78

Post by bombolo »

411111111111111 wrote: post #73
AlphaSite wrote: post #63

Earlier quoted context omitted.

Even if pip was a little better I still wouldn’t use it, because Poetry solves the problem so well you’d need to pull it out of my cold hands. It solves not only the install package problem, but locking, venvs, project structure and packaging in a well integrated solution with a fantastic UX. If poetry (or similar) were renamed to Pip and it were to be included in the stdlib id obviously switch. It doesn’t solve the…

Everytime you install a package through poetry, you're also running pip code. You didn't even understand the point they made: the need for Pipenv and poetry would pretty much go away if pip added support for a proper lockfile and venvs. And that's the only correct choices as pip is already pythons package manager.

What's the problem with first doing the venv thing and then running pip?

Re: Thoughts on the Python packaging ecosystem

#79

Post by ivoflipse »

at_a_remove wrote: post #76
KyeRussell wrote: post #30

Earlier quoted context omitted.

These days I don’t think I’m being too brash in saying that anyone dealing with Python 2 to Python 3 transition stuff is so deep in the realm of enterprise abandonware that it’s not worth paying it much attention. Not necessarily including you here! But as someone that went through the transition, I’m certainly not going to deny how much of a shit show it was. And I get a feeling that the two situations were/are caus…

It is, sadly, not always Enterprise abandonware. The industry I am in is extremely conservative about its upgrades and changes. Between that, our custom software interfacing with the 800 pound gorilla for our market, and a few other things, we're behind. Now, this software depends on a very specific version of ArcGIS. Desktop, not Pro. Not just version 10. Not just version 10.2. But version 10.2.1. If you go look at…

You're situation is related to this: https://pypackaging-native.github.io/key-issues/native-depen... Where it is incredibly hard to align various pieces of software spanning OSes, architectures, programming languages. The result is that you have to compile the whole universe for them to work together. In your case that universe is stuck on 2.7 and it will require a large effort to get everything up compatible with a recent Python version

Re: Thoughts on the Python packaging ecosystem

#80

Post by woolion »

Very good summary of the situation, I hope something good will come out of it.

I've used pipenv, pyenv, poetry and settled for my own use-cases on just pip with virtualenvwrapper. So any program goes into its dedicated folder with requirements and virtualenv. However there's something that I've never managed to get working. You have say a 3.8 virtualenv, and package file mentions python should be >=3.8. But when calling python -m build, the build process creates a new environment using the system python. So if you are on an old system that is on Python 3.7, the resulting package cannot be installed because of the inconsistency between declared python version and the one that has been used. It may be a very stupid problem, but I haven't been able to find any documentation for it. Any search engine is just too happy to throw any 'python packaging result' before anything so specific. Maybe I should use one other tool for that? But there's no clear (default?) path documented.

Post reply on HN