Live data from Hacker News

Python Has Too Many Package Managers

dublog.net

121–130 of 170 posts

Re: Python Has Too Many Package Managers

#121
post #63

Earlier quoted context omitted.

Same. Additionally I use pyenv and pyenv-virtualenv to manage multiple Python versions. Different venvs pointing to different Python versions etc, but the core tools are still pip, venv and pip-tools. (I do not use conda) I publish a Python SDK with about 85k monthly downloads according to PyPI. I make sure to run tests (unit tests, type checking, integration tests) against all currently supported minor versions of P…

I think pyenv finally solved the versioning problem for me. Additionally, being able to set a Python version or virtual environment per-directory with "pyenv local" which has eliminated having to remember which venv I was using on a project directory, or remembering which convention for venvs I used for a project.

And "pyenv shell" is also handy for temporarily changing the default Python version of your current shell session.

Or changing the global default using "pyenv global" - for example I currently have this set to "3.12.3", even though I usually test the SDKs I build against the oldest version I have to support first - the latest 3.8.X

Re: Python Has Too Many Package Managers

#122
post #93

The people who say "just use pip and venv" don't understand the issue. Distutils has been ripped out of Python core, setuptools is somewhat deprecated but not really. Just don't call setup.py directly. Or use flit. Or perhaps pyproject.toml? If the latter, flit, poetry and the 100 other frontends all have a different syntax. Would you like to copy external data into the staging area while using flit? You are out of l…

> The people who say "just use pip and venv" don't understand the issue.

The people saying that say that because it works for them and it solved their problems.

> setuptools is somewhat deprecated but not really

setuptools remains the default and most popular backend (the thing that builds your package). What was deprecated was calling it directly, instead you use pip or build or any other frontend.

Why? One reason is because it exposed lots of non-standard eccentricities that users would then start to rely on, breaking the effort of standardisation.

> Or perhaps pyproject.toml? If the latter, flit, poetry and the 100 other frontends all have a different syntax.

They all use toml syntax, they all use the same build dependency standard, poetry does not use PEP 440 configuring version dependencies, but it does read PEP 440 dependencies from wheel metadata fine. I hope one day the version dependency spec can be updated with everything that has been learnt, from both Poetry and the rest of the ecosystem, but Python packaging is a slow moving beast.

> Should you use pip? Well, the latest hotness are the search-unfriendly named modules "build" and "install", which create a completely isolated environment.

pip does this for you though, I'm not sure why most end users need to know "build" or "install" exist.

> Which is the officially supported tool that will stay supported and not ripped out like distutils?

I'm not sure why you think any of these are "the officially supported tool", they are an attempt by various PyPA members to minimally and correctly implement the standards, but PyPA is a loose collection of volunteers.

Whereas, CPython core dev have a pretty clear arms length attitude towards all of packaging. This for me feels like the real contention in the Python packaging world.

> Questions over questions. And all that for creating a simple zip archive of a directory, which for some reason demands gigantic frameworks.

Well the people using pip and venv didn't have any of these questions, they just got on with it because it solved everything for them.

Likely you want your package to figure something out as it’s being built, or have interesting metadata, or include binaries, or something else not on the happy path, and you want your build tool to "just work" even though chances are you’re doing something not that simple or what the tool creator thought. Otherwise, if you really just need a “simple zip archive of a directory” just write out the directory and call python -m zipfile.

There’s nothing wrong with lots of people needing specific use cases, but it does mean all these build tools need to add lots and lots of options to accommodate them all, and before you know it you're complaining of a "gigantic framework" (even though I wouldn't describe either setuptools or hatchling like that).

Re: Python Has Too Many Package Managers

#123

I use pip. I plan to continue using pip. If I need an isolated environment, I use conda, but then I install everything with pip. If I need to guarantee versions I pip freeze. There's a lot of cruft and desire for a one-size-fits-all solution but the base tools are probably good enough. My setup is not the one-size-fits-all solution but it works for me, and my team, and lots of other teams. Beware anyone who tells you…

This is what I do essentially. I make a new conda env for each project and use pip or conda install. What if I have a new project that needs components from two projects? Sometimes there will be impossible to solve dependencies when trying to use both components. Its not feasible to dive into each dependency within each dependency to figure out how to resolve them. Rust's package manager, cargo, is able to handle thi…

[deleted]

Re: Python Has Too Many Package Managers

#124

I use pip. I plan to continue using pip. If I need an isolated environment, I use conda, but then I install everything with pip. If I need to guarantee versions I pip freeze. There's a lot of cruft and desire for a one-size-fits-all solution but the base tools are probably good enough. My setup is not the one-size-fits-all solution but it works for me, and my team, and lots of other teams. Beware anyone who tells you…

I do the same. I see that using conda outside data science is frowned upon, but in my experience, is the closest I can get to a docker container. Freezing the python version and its dependencies in a conda env has saved me a lot of trouble when coming back to old projects.

Re: Python Has Too Many Package Managers

#125

Poetry isn't perfect but I'm happy to have it. It installs most packages without issue and effortlessly handles multiple versions of Python on the same system. In my experience, Poetry works much better than, say... ... npm.

I like poetry but I’m not sure the Python ecosystem is disciplined enough for it, all too often I get stuck trying to resolve dependencies between packages which should be compatible but the dependencies weren’t defined well

Re: Python Has Too Many Package Managers

#126
post #93

The people who say "just use pip and venv" don't understand the issue. Distutils has been ripped out of Python core, setuptools is somewhat deprecated but not really. Just don't call setup.py directly. Or use flit. Or perhaps pyproject.toml? If the latter, flit, poetry and the 100 other frontends all have a different syntax. Would you like to copy external data into the staging area while using flit? You are out of l…

> The people who say "just use pip and venv" don't understand the issue. The people saying that say that because it works for them and it solved their problems. > setuptools is somewhat deprecated but not really setuptools remains the default and most popular backend (the thing that builds your package). What was deprecated was calling it directly, instead you use pip or build or any other frontend. Why? One reason i…

> pip does this for you though, I'm not sure why most end users need to know "build" or "install" exist.

Because at least at one point in time "build" was marketed as the future, it even appeared in a setuptools warning message. This developer spends several paragraphs on the issue:

https://gregoryszorc.com/blog/2023/10/30/my-user-experience-...

If it is no longer the future, so be it. Pip seemed to force isolated installs by default for a while and then backed off again.

As for the volunteer question: All OSS package managers are maintained by volunteers and I do not know of any other ecosystem that prompted an xkcd comic about the packaging situation.

Re: Python Has Too Many Package Managers

#127

Earlier quoted context omitted.

We analyze firmware and binaries, amongst other things at my day job. Reconstructing dependency graphs is a necessary part of our day to day. Assuming dependency graphs cant have cycles has shot us in the foot. There are absolutely real circular dependencies in assets in the wild.

Original author here... wait, what? I believe pip (or at least older versions of pip) makes it possible to have cycles. Are you analyzing dep graphs made with older versions of pip?

Not pip, but we ran into an older version of glibc that had real circular deps. Sorry I don’t remember off the top of my head which one.

Re: Python Has Too Many Package Managers

#128
post #43

I use Python but I’m not a professional Python programmer. I use pip and venv, with requirements.txt. It copies Python binary and some of the shared standard libraries from the system, installs the required packages in sites-packages, and it always works fine. Am I missing anything major by not using conda and Poetry?

That's generally fine until the build breaks due to a transitive dependency several levels down releasing a breaking change. While requirements.txt may hold enough information to build the project now, it is not guaranteed to work in the future -- you would need lockfiles for that guarantee. I've seen projects sort of emulating this by using "requirements.in" to list direct program dependencies, and auto-generating "…

I thought pip freeze > requirements.txt lists direct dependencies, as well as the transitive dependencies, and includes exact version numbers. From system, it copies the relevant binaries.

In the future, pip install requirements.txt will install the exact same set of packages at those pinned versions. Unless the system components drastically change, the build is deterministic and project should run in the future.

Sure, in the future if you change the version number for one package, the build might break, but that’s expected.

It looks like the lock files specify the dependency graph, hashes of each package, URLs, and metadata about the system, so they do better when a package is updated.

Re: Python Has Too Many Package Managers

#129
post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

Pulling the latest version without freezing is how you Break Stuff. Every python developer seems to be addicted to breaking changes.

Nuking and recreating, freezing, etc are all easy minor tasks but nonetheless add a few lines each of code.

Auditing for security issues would probably need some other package or other.

And with virtualenv, you can have an activated environment. It's stateful and your scripts have to then take into account state. Poetry run doesn't have that issue.

That's enough minor annoyances to make me not want to ever use anything but poetry plus the audit plugin and freeze-wheel if needed.

Re: Python Has Too Many Package Managers

#130

Earlier quoted context omitted.

I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking changes in the dependencies. The python ecosystem does not stand still and seems quite happy to introduce breaking changes in non-major versions. To my mind, there are 3 ways to make sure your python project of today will work in 2 years time: 1. Have no depen…

IMHO the problem with python package management isn't python package management. The biggest problem I've hit is when python depends on non-python - `pip install` failing because I need to `apt install libsomething-dev` is the big one, but also managing multiple versions of python itself (I count this as a dependency on non-python because the python3 binary isn't written in python - if it was, then pip+virtualenv cou…

If we could reboot all of computer science, I'd want one unified cross platform package manager with the same market takeover level as Git, and all languages designed to be aware of it.

No file system overlay stacking or anything, no file system hierarchy of different folders for different types of resources, just python style search paths and virtual environments, and no global environment allowed, ever.

But Python development is pretty nice at the moment so I can't complain.

Post reply on HN