Looks like Conda is still the best package/env manager for ML engineers.
Python Has Too Many Package Managers
21–30 of 170 posts
Re: Python Has Too Many Package Managers
#22I use pip. I plan to continue using pip. If I need an isolated environment, I use conda, but then I install everything with pip. If I need to guarantee versions I pip freeze.
There's a lot of cruft and desire for a one-size-fits-all solution but the base tools are probably good enough. My setup is not the one-size-fits-all solution but it works for me, and my team, and lots of other teams.
Beware anyone who tells you that thirty years of tooling doesn't have a solution to the problem you're facing and you need this new shiny thing.*
*Playing with shiny things is fun and should not be discouraged, but must not be mandated either.
Re: Python Has Too Many Package Managers
#23I see a lot of package managers I never head of, am a happy venv & pip user.
One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft.
This is similar approach to “zero-based budgeting”. It forces you to carefully pick your dependencies and think about what you carry.
I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs.
You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade
If you dont do that, you can quickly go down the javascript’s path of bloated node_modules.
Can people explain why venv&pip is a bad solution that doesnt work for them they have to resort to other package managers?
Even venv is not really required if you dockerize your python apps, which you will have to do anyways at deploy time
Re: Python Has Too Many Package Managers
#24Happily using pip, venv, and pip-tools for every project and still finding them more than suitable. They might not have the marketing budget or pizazz of others, but if you're looking for effective and boring tools that get the job done so you can solve more interesting problems they work just fine.
Re: Python Has Too Many Package Managers
#25I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…
Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile.
Nuking a whole venv when you mess up isn't really efficient.
They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try.
Want to convert the packages in your venv back to .tar.gz or wheels? There's no way to do that either.
Re: Python Has Too Many Package Managers
#26I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…
Pip is mediocre for vendoring non-Python or binary dependencies. Wheels are OK, but sometimes there are still problems.
Conda solves this pain point by being a general purpose package manager.
Poetry is for the more reproducible builds people. It's a little better, but it's not great enough to get people to switch.
Re: Python Has Too Many Package Managers
#27Looks like Conda is still the best package/env manager for ML engineers.
Unfortunately I'm still waiting for "Solving environment". It's been 8 years. Please send help. I'm so hungry.
(Yes, I know about mamba.)
Re: Python Has Too Many Package Managers
#28After all, the motto is "there's more than one way to do it"!
Maybe it's just a joke, but for those that don't know that is Perl's motto and even better because there's the venerable, wonderful CPAN [0] which is the de facto package repository for the language. [0] https://www.cpan.org/
It's certainly a joke, because the Python original motto was "there's only one way to do it", and the current one is "there's only one (obvious) way to do it".
The Python's motto was created as an obvious reference to the Perl's one, purposefully negating it.
Re: Python Has Too Many Package Managers
#29I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…
Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile. Nuking a whole venv when you mess up isn't really efficient. They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broke…
> There's no concept of caching venv packages elsewhere
A venv's pip shares a systemwide cache for downloads.
Re: Python Has Too Many Package Managers
#30Use Rye. It wasn't abandoned it ownership was transferred.
Rye uses other pretty standard stuff under the hood, tools that follow PEPs, its just a front end that is sane. uv is fast as well. It downloads the pinned version of standalone Python, it keeps everything in its own venv and theres very little messing/tweaking of the environment.
It is messy, although its getting better. I doubt everything will ever standardise to one tool however.