Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Has Too Many Package Managers

dublog.net

Post reply on HN
Print view View original

131–140 of 170 posts

Re: Python Has Too Many Package Managers

#131

Post by slt2021 »

jamesboehmer wrote: post #113
slt2021 wrote: post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

pip is so easy, but unfortunately i've found that if you add package signatures to requirements.txt, pip chokes on it when installing it later. And subdependencies aren't always named perfectly, e.g. they might specify ~=1.4, and a subdependency that what was once 1.4.0 is now 1.4.27, and incompatible or compromised. conda is so heavyweight installing whole pre-approved builds. and the command line options I find ext…

wouldn't `pip freeze > requirements.txt` solve your problem? It will list everything currently installed, including transitive dependencies at currently installed and working versions

Re: Python Has Too Many Package Managers

#132

Post by slt2021 »

abdullahkhalids wrote: post #68
slt2021 wrote: post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

I use the same workflow, and works for me very well. It would be cool, though, to have a wrapper around pip that adds any packages I install to the requirements file. That way I don't have to do it manually.

I actually would love something like `pip generate-reqs` where pip goes through every .py file in my app and looks at whats being imported and writes these libraries to requirements.txt, together with all transitive dependencies from the currently installed `pip list`

Re: Python Has Too Many Package Managers

#133

Post by eternityforest »

slt2021 wrote: post #51
rangerelf wrote: post #41

Earlier quoted context omitted.

I've read "they're too/so/very fragile" so many times, but nobody has given me an example of how or why they consider them fragile, what steps do I need to do to break them. Myself? I consider them ephemeral: I create my Makefile with a 'venv' target to delete/rebuild the virtual environment in case of changes. Some do take longer to rebuild, but with a global pip cache it takes much less time to rebuild once it's be…

Can I ask you a question, what do you think is better approach: 1) publish packages as wheels or 2) publish apps as docker images (or docker-compose files or helm charts)? Why some People prefer 1 to 2? I think 2 is more “production friendly” and universal across other languages and stacks (same approach used for java js ruby etc)

Wheels can be production friendly if you know that you're only ever going to care about one specific OS family, and your app is pure python+stuff available on PyPi.

For internal use, where you can just pretend everything other than Debian based systems don't exist, it's ok, but I definitely see the value of docker, even though I like Snap a lot more.

I'm surprised nobody has ever made a "distro" where all the packages had a thin python wrapper and installed from a private Pip repository, so you could package your whole app and all dependencies with it.

Re: Python Has Too Many Package Managers

#135

Post by klxamb »

p1esk wrote: post #21

Looks like Conda is still the best package/env manager for ML engineers.

First I need to create enough space for my comment:

/root/miniconda3/build/_placehold_placehold_placehold_placehold_placehold_pplacehold_placehold_placehold_placehold_placehold_p ...

Seriously, conda can be pretty annoying for packagers. I don't like the Jinja configuration stuff, I don't like the chaotic build output (see above), sometimes libraries are leaking in from the system, for some undocumented issues it takes ages to search for an answer.

One everything is set up, things tend to work, apart from the slow solver. I also do not like the way that conda embeds itself in the Windows installation.

It is a bit overhyped, like all things in the "scientific" ecosystem.

Re: Python Has Too Many Package Managers

#136

Post by LarsDu88 »

eesmith wrote: post #112
LarsDu88 wrote: post #89

Earlier quoted context omitted.

I go over the downsides in the article, but there is nothing fundamentally wrong with using pip and venv. It's just that if you've ever worked in other programming ecosystems, it would be immediately apparent that things could be significantly simpler and more reproducible. Why does python need virtual environments? Why isn't it that you simply are in the correct environment when you're in your project folder? Why ar…

> Why isn't it that you simply are in the correct environment when you're in your project folder? I typically have several environments for my main project. Mostly I use a Python 3.12 version with the full set of optional third-party packages installed, but I also have a "clean" version with none of them installed so I can test error handling, as well as a Python 3.10 version as that is the oldest supported version f…

Sounds like you may want to check out Hatch

Re: Python Has Too Many Package Managers

#137

Post by MrJohz »

yjftsjthsd-h wrote: post #96
colemannerd wrote: post #88

Earlier quoted context omitted.

BUT importantly... it REALLY does not work well for lots of teams. For me, this setup has caused production outages multiple times across multiple teams. Maybe the root python ecosystem should learn and adopt from other ecosystems that have figured out complex deployment in a much easier way.

> For me, this setup has caused production outages multiple times across multiple teams How does that happen?

I've seen people use that setup but not freeze the dependencies and so have errors in production that didn't exist in development and waste days trying to figure out what was going on.

I've seen people use this setup and then struggle to deploy in different environments, especially when a dependency updates and no longer works correctly on a particular device, or where there are differences in behaviour or packaging or something in two different machines.

I've seen people accidentally install packages locally and not add them to the requirements file (especially when they're less experienced with Python), and cause outages by having the application crash on startup.

I've seen people freeze the dependency list and then have excess dependencies floating around because they couldn't differentiate between dependencies that were being used, and dependencies that were previously transitively installed and no longer needed. This doesn't necessarily cause outages, but does slow everything down over time, either in continual package maintenance or in downloading excess packages.

Most of the time, when I've seen teams use this sort of "simple" packaging process, they end up writing a bunch of scripts to facilitate it (because it's rarely so simple in practice). I have seen these scripts fail in almost every possible way. Often this happens in a development environment or before production, but I've seen production issues here as well.

To be clear, I think there are some situations where .venv and requirements.txt really are all you need. But I don't think going down that route removes complexity or makes things easier. Instead, it means you need to manage that essential complexity yourself. There are sometimes advantages to that, and reasons why it might make sense to take that option, but they are relatively rare. And given that pip/venv are right now the most official way of handling packaging in Python, that raises a massive red flag for the entire ecosystem.

Re: Python Has Too Many Package Managers

#138

Post by MrJohz »

poikroequ wrote: post #84

Poetry isn't perfect but I'm happy to have it. It installs most packages without issue and effortlessly handles multiple versions of Python on the same system. In my experience, Poetry works much better than, say... ... npm.

Can you describe what issues you've had with NPM that you haven't had with Poetry? In principle, they should be doing the same thing - both use a per-package environment and a lockfile, allow you to run commands in the environment, ensure the checked-in dependencies remain in sync with the local dependencies, etc. The main difference is that Python's ecosystem only allows one version of a dependency to exist at a time, but Node's ecosystem can handle multiple coexisting dependencies.

Re: Python Has Too Many Package Managers

#139

Post by eesmith »

LarsDu88 wrote: post #136
eesmith wrote: post #112

Earlier quoted context omitted.

> Why isn't it that you simply are in the correct environment when you're in your project folder? I typically have several environments for my main project. Mostly I use a Python 3.12 version with the full set of optional third-party packages installed, but I also have a "clean" version with none of them installed so I can test error handling, as well as a Python 3.10 version as that is the oldest supported version f…

Sounds like you may want to check out Hatch

How would that help? At the end I'm still using setuptools to build the extension, right?

So I'm stuck with the same core issues, but with a different front end?

To add to the complexity, I distribute my software as source, and I am wary of telling my customers to install another package in order to run the commands to install my package.

Re: Python Has Too Many Package Managers

#140

Post by odie5533 »

mih wrote: post #107
sealedservant wrote: post #18

I concur and I also think that there are too many build backends. pdm is my current favorite package manager. It is fully PEP-compliant and the lockfile generation is nice. I wouldn't call hatch a package manager because I don't think it can make lockfiles. uv is on my radar but it doesn't look ready for primetime yet. I saw they are building out a package management API with commands such as `uv add` and `uv remove`…

Another user of pdm here for professional projects. It sure is more standards compliant than poetry. Support for in-project venvs and integration of configs for packages such as pytest is quite useful. When evaluating package managers, poetry for sure was a contender. However listening to others experiences regarding poetry developers introducing breaking changes that could potentially cause the CI pipeline made it a…

Poetry supports in-project venvs. You can enable it globally.

https://python-poetry.org/docs/configuration/#virtualenvsin-...

Post reply on HN