Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Has Too Many Package Managers

dublog.net

Post reply on HN
Print view View original

111–120 of 170 posts

Re: Python Has Too Many Package Managers

#111

Post by hadlock »

Scaevolus wrote: post #39
sealedservant wrote: post #25

Earlier quoted context omitted.

Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile. Nuking a whole venv when you mess up isn't really efficient. They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broke…

Pip will cache the downloaded package tarballs to make repeated installs faster at least, but that's little help when you have a dozen venvs with multiple gigabytes of pytorch.

Time to containerize your pytorch app

Re: Python Has Too Many Package Managers

#112

Post by eesmith »

LarsDu88 wrote: post #89
slt2021 wrote: post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

I go over the downsides in the article, but there is nothing fundamentally wrong with using pip and venv. It's just that if you've ever worked in other programming ecosystems, it would be immediately apparent that things could be significantly simpler and more reproducible. Why does python need virtual environments? Why isn't it that you simply are in the correct environment when you're in your project folder? Why ar…

> Why isn't it that you simply are in the correct environment when you're in your project folder?

I typically have several environments for my main project. Mostly I use a Python 3.12 version with the full set of optional third-party packages installed, but I also have a "clean" version with none of them installed so I can test error handling, as well as a Python 3.10 version as that is the oldest supported version for one of my third-party optional dependencies, plus a 3.8 version since I still support that version in my core system.

In that way I can quickly test code against the typical deployment cases before committing.

The project has about 500K lines of C code, mostly in one Python/C extension which takes over a minute to fully rebuild, but most changes are to Python code or other bits of Python/C or Cython code which are faster to rebuild, so I do editable builds for most of my work for the quick edit cycle.

I don't know how to nominate one of these environments as "correct".

(The full test suite against the various permutations of Python versions and 10 or so optional components is a mess. I've given up on testing all combinations as it wasn't proving worthwhile.)

Re: Python Has Too Many Package Managers

#113

Post by jamesboehmer »

slt2021 wrote: post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

pip is so easy, but unfortunately i've found that if you add package signatures to requirements.txt, pip chokes on it when installing it later. And subdependencies aren't always named perfectly, e.g. they might specify ~=1.4, and a subdependency that what was once 1.4.0 is now 1.4.27, and incompatible or compromised.

conda is so heavyweight installing whole pre-approved builds. and the command line options I find extremely frustrating.

I need supply chain security and perfectly reproducible builds, so poetry was the only real option.

Re: Python Has Too Many Package Managers

#114

Post by darepublic »

slt2021 wrote: post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

I also use venv and pip and I'm happy with the solution. I don't get the knock on JavaScript node modules though. Why wouldn't you want to list out your dependencies and their versions? It can be greatly helpful and I don't see how it contributes to bloat

Re: Python Has Too Many Package Managers

#115

Post by p1esk »

gillesjacobs wrote: post #85
p1esk wrote: post #21

Looks like Conda is still the best package/env manager for ML engineers.

ML engineer, I am up to 3 irreversibly broken environments after simply adding a package. Conda is inexcusably slow and but even mamba can save it. Every conda project I have ever was later switched to poetry. Here's a fun challenge: try to determine channel priority in your conda env, go ahead try. Conda is the single worst packaging tool I have ever used.

OK, would you say poetry is the best for ML engineers?

Re: Python Has Too Many Package Managers

#117

Post by spiralk »

wantsanagent wrote: post #22

I use pip. I plan to continue using pip. If I need an isolated environment, I use conda, but then I install everything with pip. If I need to guarantee versions I pip freeze. There's a lot of cruft and desire for a one-size-fits-all solution but the base tools are probably good enough. My setup is not the one-size-fits-all solution but it works for me, and my team, and lots of other teams. Beware anyone who tells you…

This is what I do essentially. I make a new conda env for each project and use pip or conda install. What if I have a new project that needs components from two projects? Sometimes there will be impossible to solve dependencies when trying to use both components. Its not feasible to dive into each dependency within each dependency to figure out how to resolve them.

Rust's package manager, cargo, is able to handle this by allowing multiple versions of libraries to be installed in a single environment. Why can't python do that? How can one solve this with conda/pip or any currently available python tool? I've given up and decided to use websockets between different python processes from different environments.

Re: Python Has Too Many Package Managers

#118

Post by eesmith »

artexy wrote: post #93

The people who say "just use pip and venv" don't understand the issue. Distutils has been ripped out of Python core, setuptools is somewhat deprecated but not really. Just don't call setup.py directly. Or use flit. Or perhaps pyproject.toml? If the latter, flit, poetry and the 100 other frontends all have a different syntax. Would you like to copy external data into the staging area while using flit? You are out of l…

And if you want to build a Python/C extension, perhaps with a codegen step as well, then most of these modern tools throw their hands in the air.

I had to hack my setup.py because PEP 517's fascination for completely isolated environment mean even a single typo fix in a Python comment meant a full re-compile of all 500KLOC in my main Python/C extension.

Setuptools creates temporary directories for the .o files, and passes them to the build command. I had to override that to revert to the old behavior:

  class build_subclass(build):
      def finalize_options(self):
        self.build_lib = self.build_temp = self.build_platlib = None
        super().finalize_options()
You might like my minimal wheel builder at https://news.ycombinator.com/item?id=38111355 . No dependencies, no framework.

Re: Python Has Too Many Package Managers

#119

Post by spapas82 »

sealedservant wrote: post #25
slt2021 wrote: post #23

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from s…

Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile. Nuking a whole venv when you mess up isn't really efficient. They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broke…

> They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try.

This isn't 100% true. If you carefully use the same python version and the same path for your venv and python then copying the venv between computers works perfectly. I've done it many times, actually last time I changed my computer I copied over all my projects and venvs and everything worked flawlessly.

Post reply on HN