Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on the Python packaging ecosystem

pradyunsg.me

Post reply on HN
Print view View original

81–90 of 121 posts

Re: Thoughts on the Python packaging ecosystem

#81

Post by dmr_92 »

meandmycode wrote: post #75
AlphaSite wrote: post #63

Earlier quoted context omitted.

Even if pip was a little better I still wouldn’t use it, because Poetry solves the problem so well you’d need to pull it out of my cold hands. It solves not only the install package problem, but locking, venvs, project structure and packaging in a well integrated solution with a fantastic UX. If poetry (or similar) were renamed to Pip and it were to be included in the stdlib id obviously switch. It doesn’t solve the…

Poetry ended up being mostly a disappointment having used it for the last 18 months. First time installs were eventually an hour long to resolve, and lock files were platform specific, making using them for reproducibility almost useless. The wider python ecosystem is frankly painful to productionize even today and it really does need some focus and governance.

Are you sure the lockfiles are platform-specific?

I spent a long time migrating a project to use poetry. One of the reasons I opted for poetry over others was that the lockfile retained all of the environment markers in the packaging metadata, so that the lockfile could support multiple interpreters and interpreter versions, multiple platforms, etc.

Re: Thoughts on the Python packaging ecosystem

#82

Post by numbsafari »

bombolo wrote: post #78
411111111111111 wrote: post #73

Earlier quoted context omitted.

Everytime you install a package through poetry, you're also running pip code. You didn't even understand the point they made: the need for Pipenv and poetry would pretty much go away if pip added support for a proper lockfile and venvs. And that's the only correct choices as pip is already pythons package manager.

What's the problem with first doing the venv thing and then running pip?

Any multi step process is guaranteed to produce random assortments of workflow tools to manage those steps which will seek to replace the original process as the one true process.

If there two steps, one part of the community will insist they belong in: a makefile, bash script, python script, lambda network service, bazel, pants, scons, terraform, ansible, npm …

Re: Thoughts on the Python packaging ecosystem

#83

Post by numbsafari »

ivoflipse wrote: post #50
pronlover723 wrote: post #49

For me, before I even get to packaging, I hit the install/environment issue with python. Python, by default, wants to be installed at a system level and wants libraries/packages to be at a system level. That shit has to stop. The default needs to be project local installs. Node might have issues but one thing they got right is defaulting to project local installs vs python where you need various incantations to get o…

Lucky for you, there's a new PEP proposing just that: https://discuss.python.org/t/pep-704-require-virtual-environ... Written by the author of this blog post no less! However, you'll find that as with all packaging discussions there are people opposing it, because their workflow doesn't match yours and they don't want to change how they work. We, as a community, need a way to resolve such stalemates or I fear we won'…

My solution to this problem has been … to stop using python as a universal tool for all things and instead use other tools purpose built for those purposes.

Unfortunately (or not) that basically means no more python. Because it is definitely a “Jack of all trades, master of none”.

Focusing on programming languages that just want to be programming languages and not also a system service makes life so much better. You end up using languages that produce self-contained, easily shippable binaries, or languages with easily embeddable runtimes, instead of trying to write code that has to somehow survive in a “diverse ecosystem”, which generally makes it overly bloated and brittle as it grows so many appendages to solve so many orthogonal incompatibilities it comes to resemble enterprise open source…

Re: Thoughts on the Python packaging ecosystem

#84

Post by numbsafari »

earthboundkid wrote: post #53
ivoflipse wrote: post #50

Earlier quoted context omitted.

Lucky for you, there's a new PEP proposing just that: https://discuss.python.org/t/pep-704-require-virtual-environ... Written by the author of this blog post no less! However, you'll find that as with all packaging discussions there are people opposing it, because their workflow doesn't match yours and they don't want to change how they work. We, as a community, need a way to resolve such stalemates or I fear we won'…

We need some person, a nice person, a benevolent person, who could some how tell other people what to do, dictate it if you will, and it would be best if they could keep up this job for the rest of his or her life. We’ll call them, the Friendly Language Uncle.

I’d prefer a crazy aunt to a creepy uncle… but maybe let’s leave family out of it altogether.

Re: Thoughts on the Python packaging ecosystem

#85

Post by ivoflipse »

woolion wrote: post #80

Very good summary of the situation, I hope something good will come out of it. I've used pipenv, pyenv, poetry and settled for my own use-cases on just pip with virtualenvwrapper. So any program goes into its dedicated folder with requirements and virtualenv. However there's something that I've never managed to get working. You have say a 3.8 virtualenv, and package file mentions python should be >=3.8. But when call…

You could look into cibuildwheel, to have more control over what Python versions are used to build a wheel, but as the name implies not a great solution for running it locally.

Otherwise you'll have to be rather specific about what Python interpreter to use. For example using the python launcher for Linux (https://github.com/brettcannon/python-launcher).

Looking through the docs of pdm, Hatch or Poetry, I can't really find a definitive answer if they will use the Python version you specify. They all at least need to be able to locate the correct Python version in order to do so. It would be great if managing Python versions was something that came along with Python, so that these tools could rely on it more.

Re: Thoughts on the Python packaging ecosystem

#86

Post by ivoflipse »

meandmycode wrote: post #75
AlphaSite wrote: post #63

Earlier quoted context omitted.

Even if pip was a little better I still wouldn’t use it, because Poetry solves the problem so well you’d need to pull it out of my cold hands. It solves not only the install package problem, but locking, venvs, project structure and packaging in a well integrated solution with a fantastic UX. If poetry (or similar) were renamed to Pip and it were to be included in the stdlib id obviously switch. It doesn’t solve the…

Poetry ended up being mostly a disappointment having used it for the last 18 months. First time installs were eventually an hour long to resolve, and lock files were platform specific, making using them for reproducibility almost useless. The wider python ecosystem is frankly painful to productionize even today and it really does need some focus and governance.

If you use C-extensions or platform specific packages like pywin32, how would.you expect them to NOT be platform specific? Would you want it to only allow pure Python packages?

Re: Thoughts on the Python packaging ecosystem

#87

Post by Too »

I'm convinced 99% of all issues with pip can be solved with 3 simple aliases. It already has everything else, it just the UX that isn't opinionated enough. There wouldn't be a need for Pipenv, Poetry or yet another project-definition or lockfile format, if only these few fundamental commands with sane defaults were included out of the box.

1. Installation. Just install everything in a bloody application-specific virtualenv by default and use the existing lockfile mechanism by default.

    pyinstall='python -m venv .pip_modules && ./.pip_modules/bin/activate && pip install --editable ./ --constraints=constraints.txt'
2. Lockfiles. Encourage and always always use the constraints.txt-file. Maybe take the oppurtunity to rename it to lock.pip instead.

    pyfreeze='./pip_modules/bin/pip freeze > lock.pip'
3. Running. Remembering to activate a virtualenv is too easy to forget and leads to mode-confusion. We need an equivalent of npm run.

    pyrun='./pip_modules/bin/$1'
Done. That took 5 minutes tops. Now if you were to do this just a little more seriously its of course a bit more effort but still within the realms of weeks/months, not years. It seems they are more busy bikeshedding forever about the build-backends or whatever else that only very few packagers cares about. The packaging of course has its issues but they are all minor compared to preserving end-users sanity. The UX is the #1 issue that impacts millions of users daily, it can and just need to be fixed yesterday.

Re: Thoughts on the Python packaging ecosystem

#88

Post by rerdavies »

ehutch79 wrote: post #61
rerdavies wrote: post #46

Asking as a complete newcomer to Python programming... Why doesn't somebody just fix pip? Or is pipenv just understood to be the fix?

There’s nothing really wrong with pip. It doesn’t have a lot of features, but its position isn’t one to have those features. It’s simple ina good way. I don’t need to futz with a bunch of stuff I don’t need to get it going

Keras/tensorflow has some version installed, pip install fails.

Being able to install multiple versions of the same module seems like a fairly basic feature for a packaging manager. And it would seem to me that having a shim executable to examine a profile of some kind for an app at startup, and load/install the correct version of python doesn't seem impossible.

Re: Thoughts on the Python packaging ecosystem

#89

Post by odiroot »

woodruffw wrote: post #6

This is a great writeup by a central figure in Python packaging, and gets to the core of one of Python packaging's biggest strengths (and weaknesses): the PyPA is primarily an "open tent," with mostly independent (but somewhat standards-driven) development within it. Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at…

> some prescriptions about the one good tool to use for 99.9% of use cases

And we've always had such tool, the humble "pip". Now it even works great with pyproject.toml. Plus built-in "virtulalenv" if you need isolation.

Re: Thoughts on the Python packaging ecosystem

#90

Post by bombolo »

numbsafari wrote: post #82
bombolo wrote: post #78

Earlier quoted context omitted.

What's the problem with first doing the venv thing and then running pip?

Any multi step process is guaranteed to produce random assortments of workflow tools to manage those steps which will seek to replace the original process as the one true process. If there two steps, one part of the community will insist they belong in: a makefile, bash script, python script, lambda network service, bazel, pants, scons, terraform, ansible, npm …

So we should have 1 single shell command to do literally everything?
Post reply on HN